2024-11-04 12:26:28.579 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:26:35.768 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:26:35.768 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:26:35.808 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:26:35.809 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:26:35.809 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:26:35.994 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:26:38.154 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2024-11-04 12:26:38.157 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2024-11-04 12:26:39.259 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT11.489030748S 2024-11-04 12:26:39.368 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:26:39.577 INFO 16260 --- [main,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:26:39.577 INFO 16260 --- [main,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:26:39.681 INFO 16260 --- [main,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:26:39.766 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:26:39.766 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:26:39.766 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:26:39.766 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:26:39.766 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:26:39.767 INFO 16260 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : The following 1 profile is active: "composite" 2024-11-04 12:26:39.779 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:26:47.668 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2024-11-04 12:26:47.668 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2024-11-04 12:26:47.882 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 186 ms. Found 4 JPA repository interfaces. 2024-11-04 12:26:47.894 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2024-11-04 12:26:47.894 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2024-11-04 12:26:48.129 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 234 ms. Found 16 JPA repository interfaces. 2024-11-04 12:26:48.130 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2024-11-04 12:26:48.130 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2024-11-04 12:26:49.660 INFO 16260 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1529 ms. Found 157 JPA repository interfaces. 2024-11-04 12:26:53.286 INFO 16260 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=0f233f8c-7d1e-3396-84b4-7fa06161d9e7 2024-11-04 12:26:55.425 INFO 16260 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'commonSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.server.common.config.security.conf.CommonSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$b8f2385a] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2024-11-04 12:26:55.448 INFO 16260 --- [main,,] .o.s.c.c.s.c.CommonSecurityConfiguration : Method-level access control enabled. 2024-11-04 12:26:56.538 INFO 16260 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] driver class=com.oceanbase.jdbc.Driver 2024-11-04 12:26:56.538 INFO 16260 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] username=root@ocp_meta 2024-11-04 12:26:56.538 INFO 16260 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] jdbc url=jdbc:oceanbase://10.46.128.24:2883/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2024-11-04 12:26:56.548 INFO 16260 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Starting... 2024-11-04 12:26:56.630 INFO 16260 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Start completed. 2024-11-04 12:26:56.647 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2024-11-04 12:26:56.647 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2024-11-04 12:26:56.724 INFO 16260 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=10.46.128.24:8090, name=a1e196d99e 2024-11-04 12:26:56.884 INFO 16260 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 10.46.128.24:8090 2024-11-04 12:26:58.321 INFO 16260 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8090 (http) 2024-11-04 12:26:58.357 INFO 16260 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2024-11-04 12:26:58.357 INFO 16260 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.83] 2024-11-04 12:26:58.637 INFO 16260 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2024-11-04 12:26:58.637 INFO 16260 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 18858 ms 2024-11-04 12:26:59.957 INFO 16260 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2024-11-04 12:27:00.507 INFO 16260 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.15.Final 2024-11-04 12:27:01.696 INFO 16260 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2024-11-04 12:27:06.336 INFO 16260 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2024-11-04 12:27:08.332 WARN 16260 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2024-11-04 12:27:09.765 WARN 16260 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2024-11-04 12:27:09.765 WARN 16260 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2024-11-04 12:27:14.672 INFO 16260 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2024-11-04 12:27:14.788 INFO 16260 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2024-11-04 12:27:17.808 INFO 16260 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2024-11-04 12:27:18.668 INFO 16260 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2024-11-04 12:27:18.682 INFO 16260 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2024-11-04 12:27:18.682 INFO 16260 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] username=root@ocp_monitor 2024-11-04 12:27:18.682 INFO 16260 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] jdbc url=jdbc:oceanbase://10.46.128.24:2883/monitor_database?useUnicode=true&characterEncoding=utf-8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=convertToNull 2024-11-04 12:27:18.682 INFO 16260 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Starting... 2024-11-04 12:27:18.703 INFO 16260 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Start completed. 2024-11-04 12:27:18.703 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2024-11-04 12:27:18.703 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2024-11-04 12:27:18.756 INFO 16260 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] driver class=com.oceanbase.jdbc.Driver 2024-11-04 12:27:18.756 INFO 16260 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] username=root@ocp_meta 2024-11-04 12:27:18.756 INFO 16260 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] jdbc url=jdbc:oceanbase://10.46.128.24:2883/meta_database?useUnicode=true&characterEncoding=UTF8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2024-11-04 12:27:18.796 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2024-11-04 12:27:18.796 INFO 16260 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2024-11-04 12:27:19.777 INFO 16260 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2024-11-04 12:27:23.311 INFO 16260 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2024-11-04 12:27:23.980 INFO 16260 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2024-11-04 12:27:23.981 INFO 16260 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2024-11-04 12:27:45.821 INFO 16260 --- [main,,] org.reflections.Reflections : Reflections took 9118 ms to scan 81 urls, producing 1062 keys and 7488 values 2024-11-04 12:27:55.506 INFO 16260 --- [main,,] org.reflections.Reflections : Reflections took 9578 ms to scan 81 urls, producing 1062 keys and 7488 values 2024-11-04 12:28:07.151 INFO 16260 --- [main,,] org.reflections.Reflections : Reflections took 11618 ms to scan 81 urls, producing 985 keys and 4895 values 2024-11-04 12:28:07.190 INFO 16260 --- [main,,] c.o.o.s.c.analyzer.OcpStartUpChecker : audit event is disable 2024-11-04 12:28:17.262 INFO 16260 --- [main,,] org.reflections.Reflections : Reflections took 9494 ms to scan 81 urls, producing 985 keys and 4895 values 2024-11-04 12:28:17.289 INFO 16260 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116021] DistributedEventManager scanned event classes: [class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent, class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent, class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent, class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent, class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent] 2024-11-04 12:28:17.291 INFO 16260 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent failed, error: null 2024-11-04 12:28:17.291 INFO 16260 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent failed, error: null 2024-11-04 12:28:17.291 INFO 16260 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent failed, error: null 2024-11-04 12:28:17.291 INFO 16260 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116022] DistributedEventManager scanned event class map: {refresh ocp cluster cache=class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, reset-all-obproxy-id=class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, refresh-config-server-cache=class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, tenant-renamed=class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent, invalidate-ob-version-cache=class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent} 2024-11-04 12:28:17.321 INFO 16260 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116001] DistributedEventManager started on a1e196d99e 2024-11-04 12:28:17.513 INFO 16260 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2024-11-04 12:28:20.694 INFO 16260 --- [main,,] c.o.o.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2024-11-04 12:28:22.313 INFO 16260 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2024-11-04 12:28:22.581 INFO 16260 --- [main,,] c.o.o.a.service.TaskRepositoryService : create task success, task=TaskInfo(id=1, uuid=354af545-dee8-4634-bc44-2ef0a637b63a, taskType=AlarmProcess, target=null, parameters=null, intervalSeconds=10, status=WAITING, executorName=10.46.128.24:16260, createAt=null, startAt=null, endAt=null, heartbeatAt=null, result=null) 2024-11-04 12:28:22.581 INFO 16260 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : create task done, taskType=AlarmProcess, expectCount=1, createCount=1 2024-11-04 12:28:22.581 INFO 16260 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize global tasks done 2024-11-04 12:28:22.581 INFO 16260 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize done 2024-11-04 12:28:23.878 INFO 16260 --- [main,,] c.o.o.alarm.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=ob_log_clog_write_error,ob_log_right_to_die,ob_host_net_send_percent_over_threshold,binlog_host_storage_usage_over_threshold,oblb_etcd_leader_absent,ob_tenant_compaction_error,ob_log_easy_hold_timeout,odp_instance_expanse_failed,binlog_host_cpu_usage_over_threshold,ob_log_decent_voting,ob_cluster_frozen_version_delta_over_threshold,binlog_node_status_abnormal,ob_log_commit_error,obproxy_memory_peak,host_operation,ob_log_mysql_request_full,ob_log_timer_cost_much,ob_log_worker_thread_full,host_disk_readonly,node_file_root_usage,rootservice_log_alarm,ob_log_disk_almost_full,obdns_etcd_cannot_connect_to_leader,oms_sync_failed,binlog_service_conversion_delay,ob_cluster_sync_delay_time_too_long,arbitration_service_unavailable,ob_cluster_merge_timeout,ob_common_unexpected_internal_error,partition_create_failed,ob_cluster_recyclebin_disk_used_over_threshold,ob_log_mem_alloc_error,ob_rootservice_stop_fail,base_backup_timeout,refresh_location_cache_failed,host_ntp_offset_too_large,ob_log_error_lib_direct_alloc,ob_host_invalid_disk_exists,ob_cluster_no_frozen,ocp_collect_metric_failure_rate_high,ob_log_distributed_strong_read,ob_tenant_cpu_usage_over_threshold,ob_log_fail_check_localserver,host_agent_open_fd_count_over_threshold,ob_cluster_sync_failed,oblb_ip_forward_invalid,backup_process_dead,oblb_keepalived_service_inactive,oblb_process_dead,ob_tenant500_storage_short_meta_mem_hold_high,ob_host_install_disk_percent_over_threshold,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,binlog_instance_operation_info,obagent_dead,ob_server_stop_fail,ob_server_cannot_connect_arbitration,ob_log_rpc_error,ob_server_sstable_percent_over_threshold,oms_sync_delay,ob_host_monitordb_disconnected,oblb_keepalived_become_master,os_tsar_traffic_drop,ob_log_unit_migrate_error,ob_log_reply_memstore_failed,host_agent_version_not_same,ob_log_global_trans_version_delay,ob_tenant_partition_leader_absent,ob_cluster_operation_info,ob_cluster_merge_error,ob_tenant_name_mismatch,backup 2024-11-04 12:28:25.171 INFO 16260 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table name=metric_data_daily, createTime=1730694377 2024-11-04 12:28:25.171 INFO 16260 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Set table switch time, time=1730694377 2024-11-04 12:28:26.602 INFO 16260 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:28:26.860 INFO 16260 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:28:26.958 INFO 16260 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:28:27.054 INFO 16260 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:28:27.160 INFO 16260 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:28:27.400 INFO 16260 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:28:27.410 INFO 16260 --- [main,,] c.o.o.m.query.LocalMetricQueryService : Monitor router init, username=system, connectTimeoutMs=3000, readTimeoutMs=120000 2024-11-04 12:28:27.957 INFO 16260 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Init consumer, consumerInfo=ShardingConsumerEntity(id=1, module=alarm-detect-by-rule, name=10.46.128.24:8090, state=NEW, enabled=true, heartbeatTime=2024-11-04T12:28:27.947+08:00, createTime=null) 2024-11-04 12:28:27.958 INFO 16260 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Init consumer, consumerInfo=ShardingConsumerEntity(id=2, module=alarm-detect-by-target, name=10.46.128.24:8090, state=NEW, enabled=true, heartbeatTime=2024-11-04T12:28:27.947+08:00, createTime=null) 2024-11-04 12:28:27.991 INFO 16260 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByTargetIdShardingStrategy@4c0302f8 2024-11-04 12:28:27.991 INFO 16260 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByRuleShardingStrategy@3ac3615d 2024-11-04 12:28:28.000 INFO 16260 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2024-11-04 12:28:28.000 INFO 16260 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2024-11-04 12:28:28.423 INFO 16260 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances start. 2024-11-04 12:28:28.479 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2024-11-04 12:28:28.479 INFO 16260 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2024-11-04 12:28:28.481 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2024-11-04 12:28:28.497 INFO 16260 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2024-11-04 12:28:28.499 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2024-11-04 12:28:28.499 INFO 16260 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2024-11-04 12:28:28.501 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.query.global.timeout.millis = 10000 2024-11-04 12:28:28.501 INFO 16260 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set queryGlobalTimeoutMillis=10000 2024-11-04 12:28:28.502 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2024-11-04 12:28:28.502 INFO 16260 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2024-11-04 12:28:28.504 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 10000 2024-11-04 12:28:28.504 INFO 16260 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=10000 2024-11-04 12:28:28.505 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2024-11-04 12:28:28.505 INFO 16260 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2024-11-04 12:28:28.507 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 2000 2024-11-04 12:28:28.510 INFO 16260 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=2000 2024-11-04 12:28:28.512 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2024-11-04 12:28:28.512 INFO 16260 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2024-11-04 12:28:28.513 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2024-11-04 12:28:28.513 INFO 16260 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2024-11-04 12:28:28.514 INFO 16260 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2024-11-04 12:28:28.514 INFO 16260 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2024-11-04 12:28:28.551 INFO 16260 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances end. 2024-11-04 12:28:30.995 INFO 16260 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2024-11-04 12:28:30.995 INFO 16260 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2024-11-04 12:28:31.043 INFO 16260 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2024-11-04 12:28:31.124 INFO 16260 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2024-11-04 12:28:31.495 INFO 16260 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/dba/obtest/software 2024-11-04 12:28:32.857 INFO 16260 --- [main,,] c.o.o.s.common.config.BeanConfiguration : begin set auto discover false. 2024-11-04 12:28:32.862 INFO 16260 --- [main,,] c.o.o.s.common.config.BeanConfiguration : set auto discover false success. 2024-11-04 12:28:34.005 INFO 16260 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2024-11-04 12:28:34.005 INFO 16260 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2024-11-04 12:28:34.005 INFO 16260 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2024-11-04 12:28:34.005 INFO 16260 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2024-11-04 12:28:34.015 INFO 16260 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-rule, name=10.46.128.24:8090, consumerId=1, resources=[ShardingSlice(consumerId=1, type=DEFAULT, resourceIds=[274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 336, 337, 362, 363, 505, 506, 507, 508, 509])] 2024-11-04 12:28:37.825 INFO 16260 --- [main,,] c.o.o.alarm.schedule.AlarmScheduledJobs : AlarmSchedules construct done 2024-11-04 12:28:38.870 INFO 16260 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2024-11-04T12:28:37.888+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system- admin-, subscribedJudgerVersion 2024-11-04 12:28:38.880 INFO 16260 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2024-11-04 12:28:38.880 INFO 16260 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2024-11-04T12:28:37.888+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system- admin-, subscribedJudgerVersion 2024-11-04 12:28:38.880 INFO 16260 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2024-11-04 12:28:38.881 INFO 16260 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2024-11-04 12:28:38.946 INFO 16260 --- [main,,] c.o.o.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/home/dba/ocp-server/alarm 2024-11-04 12:28:39.206 INFO 16260 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2024-11-04 12:28:39.285 INFO 16260 --- [main,,] c.o.o.alarm.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=10.46.128.24:16260 2024-11-04 12:28:39.395 INFO 16260 --- [main,,] c.o.o.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=169 2024-11-04 12:28:39.473 INFO 16260 --- [main,,] c.o.o.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=169 2024-11-04 12:28:39.502 INFO 16260 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2024-11-04 12:28:39.573 INFO 16260 --- [main,,] c.o.ocp.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2024-11-04 12:28:39.937 INFO 16260 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 10.46.128.24:8090 2024-11-04 12:28:39.951 INFO 16260 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2024-11-04 12:28:40.050 WARN 16260 --- [pool-11-thread-1,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:40.050 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2024-11-04 12:28:42.420 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2024-11-04 12:28:42.474 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2024-11-04 12:28:42.510 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:28:42.524 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:28:43.052 WARN 16260 --- [pool-11-thread-1,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:43.052 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2024-11-04 12:28:44.455 INFO 16260 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2024-11-04 12:28:44.538 INFO 16260 --- [main,,] c.o.o.s.c.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2024-11-04 12:28:44.952 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:44.956 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:44.958 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:44.961 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:44.963 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:44.965 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:44.967 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:44.970 WARN 16260 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2024-11-04 12:28:45.882 INFO 16260 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Validate and set keys, publicKey=null, valid=false 2024-11-04 12:28:46.028 INFO 16260 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Generate new rsa key, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYo+dfEA+ti3lp8yywLmeYA2IPgY0aLvYoP9drFzPTopFyNoOCSB6O+ieFk01zZw4wQLLUUF8WZV+FgS62Eyk2sfwbrRfiMS6z6SpCHJYTsrB99sefX8HeUFDXENz1L0GfutE734bJc06TI5vqzy8aUTtxcCW/mBZbRT4bp5Df5wIDAQAB 2024-11-04 12:28:46.041 INFO 16260 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.sensitive-data.encryption.public-key, previousValue=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYo+dfEA+ti3lp8yywLmeYA2IPgY0aLvYoP9drFzPTopFyNoOCSB6O+ieFk01zZw4wQLLUUF8WZV+FgS62Eyk2sfwbrRfiMS6z6SpCHJYTsrB99sefX8HeUFDXENz1L0GfutE734bJc06TI5vqzy8aUTtxcCW/mBZbRT4bp5Df5wIDAQAB, updatedValue=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYo+dfEA+ti3lp8yywLmeYA2IPgY0aLvYoP9drFzPTopFyNoOCSB6O+ieFk01zZw4wQLLUUF8WZV+FgS62Eyk2sfwbrRfiMS6z6SpCHJYTsrB99sefX8HeUFDXENz1L0GfutE734bJc06TI5vqzy8aUTtxcCW/mBZbRT4bp5Df5wIDAQAB, runningValue= 2024-11-04 12:28:46.049 INFO 16260 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.sensitive-data.encryption.private-key, previousValue=MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBANij518QD62LeWnzLLAuZ5gDYg+BjRou9ig/12sXM9OikXI2g4JIHo76J4WTTXNnDjBAstRQXxZlX4WBLrYTKTax/ButF+IxLrPpKkIclhOysH32x59fwd5QUNcQ3PUvQZ+60TvfhslzTpMjm+rPLxpRO3FwJb+YFltFPhunkN/nAgMBAAECgYAXL4OFswxagIDgC/yOZokDxejqy+y0Dq6dpCeD+Jm9Xrq+RINSoDFVEATG0PsLDM6B74q0Gl3y4aqVeTvslKRmWaWDojKgy49GIWFKPrqalJqG/kwDhFgikP2CYjioV1+bTVOVI+HWuWfsW+z63lYWrOcCs78EfUvUeVYFz7aP4QJBAPuJivhDjeRtnSTxiHqEMzFFPl6y4ss/Mt46hRbuh3clsAtrEUIIcrzwZwIDltF+UnayJrmmkkIt5oro7CJmxr8CQQDce9y05JJEbhpRyJ2aAlqV0VAPr6SNE9PFl8Q56V8LsUNg5Sn6T059xYtlzFiKr4C38n6oteqIMPNDCao8z5jZAkBMs1Q8/9x0mzgf28bSFR3HGGY8Wi/t6geI3C9ZHsMQAd38nVlV2V2jX2sJoi6LEwnyR8CnO7O9mmGCWiQnGU/hAkEArOLe/GUioE/piWmS0xm3aDS536Ltl8YNT6oirvpkkemMmOT5FpKrTInjNY6XeyAFAbg/QxFqdyJTDaT9fTOXeQJBAJaX2OdH2pd51tXiQDMKY52Fr7VFzvq9Hh19jhYKfNeZyOtH3SY4PB1rTsjgYqIkPUKrP9XDiqO5noR5mhPlt4Y=, updatedValue=MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBANij518QD62LeWnzLLAuZ5gDYg+BjRou9ig/12sXM9OikXI2g4JIHo76J4WTTXNnDjBAstRQXxZlX4WBLrYTKTax/ButF+IxLrPpKkIclhOysH32x59fwd5QUNcQ3PUvQZ+60TvfhslzTpMjm+rPLxpRO3FwJb+YFltFPhunkN/nAgMBAAECgYAXL4OFswxagIDgC/yOZokDxejqy+y0Dq6dpCeD+Jm9Xrq+RINSoDFVEATG0PsLDM6B74q0Gl3y4aqVeTvslKRmWaWDojKgy49GIWFKPrqalJqG/kwDhFgikP2CYjioV1+bTVOVI+HWuWfsW+z63lYWrOcCs78EfUvUeVYFz7aP4QJBAPuJivhDjeRtnSTxiHqEMzFFPl6y4ss/Mt46hRbuh3clsAtrEUIIcrzwZwIDltF+UnayJrmmkkIt5oro7CJmxr8CQQDce9y05JJEbhpRyJ2aAlqV0VAPr6SNE9PFl8Q56V8LsUNg5Sn6T059xYtlzFiKr4C38n6oteqIMPNDCao8z5jZAkBMs1Q8/9x0mzgf28bSFR3HGGY8Wi/t6geI3C9ZHsMQAd38nVlV2V2jX2sJoi6LEwnyR8CnO7O9mmGCWiQnGU/hAkEArOLe/GUioE/piWmS0xm3aDS536Ltl8YNT6oirvpkkemMmOT5FpKrTInjNY6XeyAFAbg/QxFqdyJTDaT9fTOXeQJBAJaX2OdH2pd51tXiQDMKY52Fr7VFzvq9Hh19jhYKfNeZyOtH3SY4PB1rTsjgYqIkPUKrP9XDiqO5noR5mhPlt4Y=, runningValue= 2024-11-04 12:28:46.049 INFO 16260 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Refresh and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYo+dfEA+ti3lp8yywLmeYA2IPgY0aLvYoP9drFzPTopFyNoOCSB6O+ieFk01zZw4wQLLUUF8WZV+FgS62Eyk2sfwbrRfiMS6z6SpCHJYTsrB99sefX8HeUFDXENz1L0GfutE734bJc06TI5vqzy8aUTtxcCW/mBZbRT4bp5Df5wIDAQAB 2024-11-04 12:28:46.053 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 gets the leadership... 2024-11-04 12:28:46.147 INFO 16260 --- [main,,] c.o.o.a.c.AlarmConfiguration : alarm configuration set from system properties 2024-11-04 12:28:47.324 INFO 16260 --- [main,,] c.o.o.m.promethues.InnerMetricContainer : init inner metric container success, configToMetricMap = {InnerMetricContainer.InnerMetricConfig(scrapeType=second)=[system_cpu_count, jvm_threads_states_threads, jvm_gc_memory_promoted_bytes_total, jvm_memory_committed_bytes, ocp_monitor_collect_request_errors_total, ocp_meta_db_health, jvm_gc_pause_seconds_max, ocp_monitor_collect_request_duration_ms_count, system_cpu_usage, process_threads, http_server_requests_seconds_count, jvm_gc_memory_allocated_bytes_total, jvm_threads_peak_threads, ocp_alarm_detect_duration_ms_count, jvm_memory_used_bytes, process_cpu_usage, jvm_memory_max_bytes, jvm_gc_max_data_size_bytes, http_server_requests_seconds_sum, jvm_gc_pause_seconds_sum, ocp_alarm_detect_duration_ms_sum, jvm_threads_live_threads, system_load_average_1m, jvm_gc_pause_seconds_count, jvm_gc_overhead_percent]} 2024-11-04 12:28:47.345 INFO 16260 --- [main,,] c.o.o.s.c.config.MetricConfiguration : Monitor sharding strategy is monitor 2024-11-04 12:28:48.275 INFO 16260 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2024-11-04 12:28:49.053 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:28:49.107 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:28:49.368 INFO 16260 --- [main,,] c.o.o.s.c.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/home/dba/obtest/ocp/ocp-server/lib 2024-11-04 12:28:49.368 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2024-11-04 12:28:49.421 ERROR 16260 --- [main,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.IllegalArgumentException occurred with code error.file.meta.not.found, and args [built-in, obproxyd.sh] 2024-11-04 12:28:49.421 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Script:obproxyd.sh not exist. 2024-11-04 12:28:49.421 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2024-11-04 12:28:49.484 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=1 2024-11-04 12:28:49.593 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=1, name=obproxyd.sh 2024-11-04 12:28:49.621 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-libs-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:28:49.756 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=oceanbase-ce-libs-4.3.3.1-101000012024102216.el7.x86_64.rpm, objectId=2 2024-11-04 12:28:49.783 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=2, name=oceanbase-ce-libs-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:28:49.791 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-utils-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:28:49.814 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=oceanbase-ce-utils-4.3.3.1-101000012024102216.el7.x86_64.rpm, objectId=3 2024-11-04 12:28:52.107 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:28:52.109 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:28:52.371 INFO 16260 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Init consumer, consumerInfo=ShardingConsumerEntity(id=3, module=metric-collect, name=10.46.128.24:8090, state=NEW, enabled=true, heartbeatTime=2024-11-04T12:28:52.367+08:00, createTime=null) 2024-11-04 12:28:52.373 INFO 16260 --- [sharding-metric-collect1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.monitor.sharding.MonitorShardingStrategy@c81e8a2 2024-11-04 12:28:52.374 INFO 16260 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2024-11-04 12:28:55.109 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:28:55.110 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:28:55.374 INFO 16260 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2024-11-04 12:28:55.461 INFO 16260 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2024-11-04 12:28:58.110 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:28:58.112 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:28:58.381 INFO 16260 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2024-11-04 12:28:58.382 INFO 16260 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.s.c.config.MetricConfiguration : Consumer had rebalanced. 2024-11-04 12:28:58.632 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=3, name=oceanbase-ce-utils-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:28:58.636 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 12:28:58.661 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 12:29:00.233 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=4, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 12:29:00.238 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.2-20241012145836.el7.aarch64.rpm 2024-11-04 12:29:00.265 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.aarch64.rpm, objectId=5 2024-11-04 12:29:01.113 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:01.114 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:01.354 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=5, name=ocp-agent-ce-4.3.2-20241012145836.el7.aarch64.rpm 2024-11-04 12:29:01.358 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:obproxy-ce-4.3.2.0-26.el7.x86_64.rpm 2024-11-04 12:29:01.384 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=obproxy-ce-4.3.2.0-26.el7.x86_64.rpm, objectId=6 2024-11-04 12:29:04.114 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:04.116 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:07.116 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:07.118 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:09.699 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=6, name=obproxy-ce-4.3.2.0-26.el7.x86_64.rpm 2024-11-04 12:29:09.704 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:29:09.736 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=oceanbase-ce-4.3.3.1-101000012024102216.el7.x86_64.rpm, objectId=7 2024-11-04 12:29:10.118 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:10.120 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:13.120 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:13.122 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:14.434 INFO 16260 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=7, name=oceanbase-ce-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:29:14.440 INFO 16260 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Save matched ocp-agent version, version=4.3.2-20241012145836 2024-11-04 12:29:14.450 INFO 16260 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.agent.version.full, previousValue=4.3.2-20241012145836, updatedValue=4.3.2-20241012145836, runningValue= 2024-11-04 12:29:14.647 INFO 16260 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanDistributedEvent":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":1,"tableName":"distributed_event"},"cleanDistributedEventReception":{"batchIntervalMillis":0,"batchSize":0,"condition":"event_id in (select id from distributed_event where expire_time <= date_sub(curdate(), interval 30 day))","priority":0,"tableName":"distributed_event_reception"},"cleanStatusTransition":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":0,"tableName":"ops_status_transition"}} 2024-11-04 12:29:15.154 INFO 16260 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2024-11-04 12:29:15.420 INFO 16260 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2024-11-04 12:29:15.422 INFO 16260 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2024-11-04 12:29:16.123 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:16.127 INFO 16260 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:17.522 INFO 16260 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2024-11-04 12:29:17.662 INFO 16260 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2024-11-04 12:29:17.664 INFO 16260 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694600, interval=60 2024-11-04 12:29:17.665 INFO 16260 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:29:17.667 INFO 16260 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2024-11-04 12:29:17.770 INFO 16260 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ob_cluster_server_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_data_async=1, metric_hour_data=1098, metric_daily_data=1098} 2024-11-04 12:29:18.027 INFO 16260 --- [main,,] .r.DefaultUserResourceManagerServiceImpl : not UPGRADE, skip create default user resource 2024-11-04 12:29:18.478 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2024-11-04 12:29:18.489 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2024-11-04 12:29:18.492 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:29:18.493 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:29:18.494 INFO 16260 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2024-11-04 12:29:18.563 INFO 16260 --- [main,,] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Create task, req=CreateTaskRequest(name=Clean expired log files on schedule, creatorId=200, groupId=2, scheduleType=SCHEDULE, scheduleRule=0 0 0 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, templateName=BackupScheduleCleanLogTemplateFactory, comments=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}, enable=true) 2024-11-04 12:29:18.665 INFO 16260 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : Backup init, create system schedule task, task=TaskDefinition{id=1000000, name=Clean expired log files on schedule, creatorId=200, templateName=BackupScheduleCleanLogTemplateFactory, scheduleType=SCHEDULE, scheduleRule=0 0 0 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, nextRunTime=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}}. 2024-11-04 12:29:18.670 INFO 16260 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2024-11-04 12:29:18.677 INFO 16260 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2024-11-04 12:29:18.893 INFO 16260 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2024-11-04 12:29:18.910 INFO 16260 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2024-11-04 12:29:19.127 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:19.131 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:19.423 INFO 16260 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2024-11-04 12:29:20.223 INFO 16260 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=STATELESS. 2024-11-04 12:29:20.248 INFO 16260 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Internal auth configured. 2024-11-04 12:29:20.250 INFO 16260 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Access key auth configured. 2024-11-04 12:29:20.256 INFO 16260 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Basic auth enabled. 2024-11-04 12:29:20.849 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.server.common.config.security.conf.StatelessSecurityConfiguration$$Lambda$2076/498091335@6ef943e5 with [org.springframework.security.web.session.DisableEncodeUrlFilter@1b17cb95, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@4e2cdbf7, org.springframework.security.web.context.SecurityContextPersistenceFilter@3da0dd5c, org.springframework.security.web.header.HeaderWriterFilter@3f3d5059, org.springframework.security.web.authentication.logout.LogoutFilter@43af8bac, com.oceanbase.ocp.server.common.config.security.accesskey.ContentCacheRequestFilter@39baed0, com.oceanbase.ocp.server.common.config.security.filter.InternalAuthenticationFilter@27d9232d, com.oceanbase.ocp.server.common.config.security.accesskey.AccessKeyAuthenticationFilter@2371f20, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@2a6c8ff2, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@5ae28313, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@23ac79c5, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@68be1d3f, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@745f5941, org.springframework.security.web.session.SessionManagementFilter@47d31f96, org.springframework.security.web.access.ExceptionTranslationFilter@86caba, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@10c2c2d] 2024-11-04 12:29:21.473 INFO 16260 --- [main,,] o.o.core.config.InitializationService : Initializing OpenSAML using the Java Services API 2024-11-04 12:29:21.644 INFO 16260 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmlenc#ripemd160 2024-11-04 12:29:21.652 INFO 16260 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160 2024-11-04 12:29:21.668 INFO 16260 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#rsa-ripemd160 2024-11-04 12:29:22.131 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:22.133 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:23.360 INFO 16260 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=IF_REQUIRED. 2024-11-04 12:29:23.364 INFO 16260 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : CSRF configured. 2024-11-04 12:29:23.941 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@32217c0f, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@2fd868a4, org.springframework.security.web.context.SecurityContextPersistenceFilter@473da825, org.springframework.security.web.header.HeaderWriterFilter@7affb8a6, org.springframework.security.web.csrf.CsrfFilter@34602645, org.springframework.security.web.authentication.logout.LogoutFilter@35cbea6a, org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter@97ced7, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter@2bf5fb30, com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter@74ed7975, org.springframework.security.oauth2.client.web.OAuth2LoginAuthenticationFilter@6efa19d6, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationFilter@1cb8eac0, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@30ec6e34, com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter@914ad22, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@5c80ccdf, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@106d193c, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@1750a91f, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@65e579ec, org.springframework.security.web.session.SessionManagementFilter@4f18fbb2, org.springframework.security.web.access.ExceptionTranslationFilter@7af5be40, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@3739758a] 2024-11-04 12:29:24.151 INFO 16260 --- [main,,] c.o.o.s.c.config.SpringScheduleConfig : Spring scheduled pool size is 30. 2024-11-04 12:29:24.177 INFO 16260 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : storage exporter register: collectEnable=true, periodMinutes=180 2024-11-04 12:29:24.177 INFO 16260 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : register backup exporter, periodMinutes=180 2024-11-04 12:29:24.181 INFO 16260 --- [main,,] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Create task, req=CreateTaskRequest(name=Collect backup files size on schedule, creatorId=200, groupId=2, scheduleType=SCHEDULE, scheduleRule=0 0 */3 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, templateName=BackupScheduleCollectTemplateFactory, comments=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}, enable=true) 2024-11-04 12:29:24.192 INFO 16260 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : Backup init, create system schedule task, task=TaskDefinition{id=1000001, name=Collect backup files size on schedule, creatorId=200, templateName=BackupScheduleCollectTemplateFactory, scheduleType=SCHEDULE, scheduleRule=0 0 */3 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, nextRunTime=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}}. 2024-11-04 12:29:24.567 INFO 16260 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2024-11-04 12:29:24.931 WARN 16260 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2024-11-04 12:29:25.133 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:25.135 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:26.048 INFO 16260 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2024-11-04 12:29:28.136 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:28.138 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:30.504 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/login']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/login'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/docs/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/docs/**'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/favicon.ico']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/favicon.ico'] 2024-11-04 12:29:30.505 WARN 16260 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/pre_login_loading']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:29:30.505 INFO 16260 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/pre_login_loading'] 2024-11-04 12:29:31.139 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:31.142 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:31.654 INFO 16260 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2024-11-04 12:29:32.102 INFO 16260 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8090 (http) with context path '' 2024-11-04 12:29:32.463 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Ocp scheduled task specified config name, methodName=collectHighRiskSqlAuditStat, configName=collectHighSqlAuditStat 2024-11-04 12:29:32.464 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=4 2024-11-04 12:29:32.464 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectSlowSqlForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.464 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectTransStatForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.464 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectHighRiskSqlAuditStat, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.464 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=oasSqlCuring, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.464 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2024-11-04 12:29:32.464 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogClusters, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=30, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogInstances, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=14 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=alarmDetectOnce, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshDetectorFactory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshGlobalSilencer, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshActiveAlarmStatus, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=resolveAlarmsOfAbsentTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshBackupAlarmStat, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmSummary, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAggregators, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshHostAlarmInfo, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=removeInactiveAlarmEvents, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=fixAlarmTarget, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTaskManager, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=20, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmInMemory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp multi cluster mode disabled. 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp vpd mode disabled. 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=partitionDaemon, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskRecords, conf=ScheduledConf{initialDelay=0, cron='0 20 5 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkOdpClient, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectOdpMetrics, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshOcpConfig, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskLog, conf=ScheduledConf{initialDelay=0, cron='0 10 3 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTmpLogFiles, conf=ScheduledConf{initialDelay=0, cron='0 20 4 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.465 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanupDruidCache, conf=ScheduledConf{initialDelay=600, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=7 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostAgentStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAgentConfig, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=10, fixedRate=0, timeUnit=MINUTES, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=getHostAgentAlarms, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=1, fixedRate=0, timeUnit=MINUTES, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostVersion, conf=ScheduledConf{initialDelay=30, cron='null', fixedDelay=4, fixedRate=0, timeUnit=MINUTES, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostClockDiff, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkCpuBindCore, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkObproxyServers, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectObproxyMetrics, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkStandbyClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=120, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterSystemEvent, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllClusterCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllArbitrationService, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=300, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=5 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncAllTenantInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllStandbyTenant, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMajorCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMiniCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.466 INFO 16260 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllTenantServiceNameConflict, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:29:32.517 INFO 16260 --- [ocp-schedules-4,5a146dc54c56b8d9,a54972678f2098ed] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:29:32.517 INFO 16260 --- [ocp-schedules-1,fc3fd25532479bb5,d31eb95ce639bd5b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:29:32.517 INFO 16260 --- [ocp-schedules-10,c77854f714758047,9f66fed9611033d2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:29:32.520 INFO 16260 --- [ocp-schedules-13,8a13a24166bd87a7,b8272ee2819da863] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:29:32.521 INFO 16260 --- [ocp-schedules-22,1c79c0daeb891057,16ced100bfc6c946] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:29:32.526 ERROR 16260 --- [ocp-schedules-22,1c79c0daeb891057,16ced100bfc6c946] com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set 2024-11-04 12:29:32.526 INFO 16260 --- [ocp-schedules-28,9309c5c1385abd66,5fce72fb916f613e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:29:32.527 INFO 16260 --- [ocp-schedules-25,64e791871c0186db,a412024003fd3ac0] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 12:29:32.528 INFO 16260 --- [ocp-schedules-17,aae596c5af530164,c8390cb0cbc5f8be] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:29:32.532 INFO 16260 --- [ocp-schedules-17,aae596c5af530164,c8390cb0cbc5f8be] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:29:32.532 INFO 16260 --- [ocp-schedules-22,1c79c0daeb891057,16ced100bfc6c946] com.alibaba.druid.pool.DruidDataSource : {dataSource-7} inited 2024-11-04 12:29:32.540 INFO 16260 --- [ocp-schedules-8,d9b84471f08e3944,ed99b1aadcf1e46e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 12:29:32.541 INFO 16260 --- [ocp-schedules-12,332c021a9070f04e,b06bdf4324c80073] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:29:32.543 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:29:32.543 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:29:32.547 INFO 16260 --- [ocp-schedules-20,e095253411645709,1c2f0e95f6041979] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2024-11-04 12:29:32.549 INFO 16260 --- [ocp-schedules-23,d02e1dcb57235c5e,4b477e00c4a548ce] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 12:29:32.550 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2024-11-04 12:29:32.553 ERROR 16260 --- [ocp-schedules-10,c77854f714758047,9f66fed9611033d2] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterInfo(ClusterSchedules.java:77) ... 15 common frames omitted 2024-11-04 12:29:32.553 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : Active profiles: [composite] 2024-11-04 12:29:32.554 INFO 16260 --- [ocp-schedules-10,1c5d32fe78d9f4ea,d8ccb603252421fd] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:29:32.555 INFO 16260 --- [ocp-schedules-2,8f64da0417450ee0,3e5dcea6fceae8e6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:29:32.556 INFO 16260 --- [ocp-schedules-19,b4bf9c99a5f60a49,ccfc7cdbd0340454] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 12:29:32.556 INFO 16260 --- [ocp-schedules-28,21d6466fb440fcba,f6127dbbc0f87194] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:29:32.556 INFO 16260 --- [ocp-schedules-16,c1d6e08fd1aef1de,4c664bdeb53910d3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:29:32.556 INFO 16260 --- [ocp-schedules-8,d901c18e3cba6e87,a449776adfbca06b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 12:29:32.556 INFO 16260 --- [ocp-schedules-1,21bc9f9aec2288e1,fb20bfe98de7f191] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:29:32.556 INFO 16260 --- [ocp-schedules-4,72d0034b87c3acb3,bb4f50b75fb0d682] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:29:32.560 INFO 16260 --- [ocp-schedules-4,72d0034b87c3acb3,bb4f50b75fb0d682] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:29:32.560 INFO 16260 --- [ocp-schedules-1,21bc9f9aec2288e1,fb20bfe98de7f191] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:29:32.560 ERROR 16260 --- [ocp-schedules-16,c1d6e08fd1aef1de,4c664bdeb53910d3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) ... 15 common frames omitted 2024-11-04 12:29:32.560 INFO 16260 --- [ocp-schedules-8,d901c18e3cba6e87,a449776adfbca06b] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 12:29:32.568 INFO 16260 --- [ocp-schedules-21,7764834c26e4f487,099823b7c13372ff] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2024-11-04 12:29:32.568 INFO 16260 --- [ocp-schedules-21,7764834c26e4f487,099823b7c13372ff] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2024-11-04 12:29:32.573 INFO 16260 --- [ocp-schedules-26,ac201374d64d9705,42258f2e1dc8518d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:29:32.572+08:00, clusters=[] 2024-11-04 12:29:32.577 WARN 16260 --- [ocp-async-3,f3db53f89660106d,a7bfbb55b9b4b3ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:29:32.577 INFO 16260 --- [ocp-schedules-9,87a3311cd95d2721,91aba3d0130f7403] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:29:32.572+08:00, clusters=[] 2024-11-04 12:29:32.578 INFO 16260 --- [ocp-schedules-5,5a3601e152b13be6,eb26e40a8fe8a9b5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:29:32.578 INFO 16260 --- [ocp-schedules-27,42f4a8003b90ec6a,33bd3aa56e2b7ad5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : agent.log.query.http.timeout=60s 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : backup.dbname.prefix=backup 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : display-parse-failed-sql=true 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : enable-sql.parse-service=true 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file=/home/dba/ocp-server/log/ocp.log 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.clean-history-on-start=true 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-history=100 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-size=100MB 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.name=/home/dba/obtest/logs/ocp-server.log 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.total-size-cap=10GB 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.web=INFO 2024-11-04 12:29:32.578 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(16260){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %.-2048m%n%wEx 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 16260 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %.-2048m%n%wEx 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : mgragent.log.level=info 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : monagent.log.level=info 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.hidden.parameter.whitelist=__min_full_resource_pool_memory,_rowsets_max_rows,_ob_max_thread_num,_enable_defensive_check,_bloom_filter_ratio,_io_read_batch_size,_io_read_redundant_limit_percentage,_enable_kv_feature 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.enabled=true 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.run.internal=5m 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.holder.capacity=2000 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2024-11-04 12:29:32.579 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.timeout.millis=10000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.ob.connection.mode=direct 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.connect.timeout.ms=10000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.socket.timeout.ms=10000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.connect.timeout.ms=20000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.socket.timeout.ms=20000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.print.sql=true 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.query.global.timeout.millis=10000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2024-11-04 12:29:32.581 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.anomaly.event.analyzer-sql-cpu-percent-limit=15 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.config={"enabled":true, "analyzeRangeMin" : 5, "offsetMin": 1, "cpuPercentLimit" : 90} 2024-11-04 12:29:32.582 INFO 16260 --- [ocp-schedules-23,d02e1dcb57235c5e,4b477e00c4a548ce] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 12:29:32.582 INFO 16260 --- [ocp-schedules-8,d901c18e3cba6e87,a449776adfbca06b] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.enabled=true 2024-11-04 12:29:32.582 INFO 16260 --- [ocp-schedules-23,d02e1dcb57235c5e,4b477e00c4a548ce] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2024-11-04 12:29:32.582 INFO 16260 --- [ocp-schedules-23,d02e1dcb57235c5e,4b477e00c4a548ce] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2024-11-04 12:29:32.582 INFO 16260 --- [ocp-schedules-1,21bc9f9aec2288e1,fb20bfe98de7f191] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:29:32.582 INFO 16260 --- [ocp-schedules-1,21bc9f9aec2288e1,fb20bfe98de7f191] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.collect.snapshot.status=active 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.snapshot.maxsize=1GB 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.version.full= 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.auto.resolved.alarm-types=ob_tenant_task_timeout,ob_host_task_timeout,standby_tenant_sync_delay_too_long 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2024-11-04 12:29:32.582 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-expired-seconds=600 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.enabled=true 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event-readable-cluster-auth=true 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2024-11-04 12:29:32.583 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric.only-cache=true 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=6 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=60 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.enabled=true 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.enabled=false 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2024-11-04 12:29:32.584 INFO 16260 --- [ocp-schedules-29,68476480a99e3377,e848196acbb401d9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:29:32.584 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.password=null 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.username=null 2024-11-04 12:29:32.585 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.audit.enabled=true 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.enabled=false 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-location= 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-name= 2024-11-04 12:29:32.585 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2024-11-04 12:29:32.585 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.obs.support.enabled=false 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2024-11-04 12:29:32.585 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2024-11-04 12:29:32.585 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2024-11-04 12:29:32.586 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2024-11-04 12:29:32.586 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2024-11-04 12:29:32.586 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2024-11-04 12:29:32.586 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.build.version.check=false 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ce.supported.ob.regex=^oceanbase-actiondb-(\d+\.){2}(\d+)(\.\d+)?-([\d.]+)\.([a-zA-Z0-9]+)\.(x86_64|aarch64)(\.rpm)?$ 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.check.host.http.read.timeout=5000 2024-11-04 12:29:32.588 INFO 16260 --- [ocp-schedules-20,e095253411645709,1c2f0e95f6041979] c.o.o.m.service.SeriesIdCleanService : HostIpSize=0, clusterSize=0, obproxyClusterSize=0 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2024-11-04 12:29:32.588 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"ob.cluster.parameter.template.built-in.default-2-2-77.name"}] 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.site.url= 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.use-request-address=true 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.cache.refresh.interval=60 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.connect-timeout=2000 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.socket-timeout=30000 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.initialSize=10 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.keepAlive=true 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxActive=100 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxWait=2000 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.minIdle=2 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.name=config-server-connect-pool 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.testWhileIdle=true 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.validationQuery=SELECT 1 FROM DUAL 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.event.snapshot.maxage-days=90 2024-11-04 12:29:32.589 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.has.signature.keywords.list=PGP,signatures 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.built-in.dir=/home/dba/obtest/ocp/ocp-server/lib 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.dir=/home/dba/obtest/software 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.enabled=true 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.invalid.keywords.list=NOT OK 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.enable=true 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.clockdiff.timeout.millis=20000 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.account.system.active=false 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth=composite 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2024-11-04 12:29:32.590 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/sensitiveDataKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate|ObRootServiceUnregister|GetObTenantInfoByServiceNames|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,performance/ash/reports/.*/data 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObRootServiceUnregister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,/api/v2/monitor/metric/scan.* 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=1 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.max-attempts=60 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obproxy,ocp_admin,monitor 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.sso-login-url= 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.cleanup.enabled=false 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.days=15 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2024-11-04 12:29:32.591 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2024-11-04 12:29:32.592 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.tmp.dir=/home/dba/obtest/ocp/logs/ocp 2024-11-04 12:29:32.592 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.max-download-speed=1024 2024-11-04 12:29:32.592 INFO 16260 --- [ocp-async-2,4aca698873f0d438,642ae32d52f7bdc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:29:32.592 INFO 16260 --- [ocp-schedules-3,af48437a19d2d428,0b4f8c08588cc1d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:29:32.592 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.query.http.read.timeout=70000 2024-11-04 12:29:32.592 INFO 16260 --- [ocp-async-7,c43c39c348959ae7,0350369e5066bf7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:29:32.592 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.cluster.name= 2024-11-04 12:29:32.592 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.disconnect.alarm.minutes=2 2024-11-04 12:29:32.592 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.async-compute.enabled=true 2024-11-04 12:29:32.592 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.collect.interval.second=15 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async-compute.query.minute.enable=false 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.query.min.time.buffer.second=0 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.write.queue.capacity=5000000 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.enabled=true 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.request.timeout={"second_connect_timeout":1000,"second_read_timeout":1000,"second_request_timeout":2000,"minute_connect_timeout":10000,"minute_read_timeout":30000,"minute_request_timeout":40000} 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.custom.collectItem.sql.function.whiteList=count,time_to_usec,timestampdiff,coalesce,sum,max,avg,min,round,unix_timestamp,abs 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ob_cluster_server_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_data_async":1,"metric_hour_data":1098,"metric_daily_data":1098} 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.table-switch-time= 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.deletable.metricNames=process_count|app=HOST|name=observer,process_count|app=HOST|name=obproxy,process_cpu_percent|app=HOST|name=observer,process_cpu_percent|app=HOST|name=obproxy,process_exists|app=HOST|name=observer,process_exists|app=HOST|name=obproxy,process_fd_count|app=HOST|name=observer,process_fd_count|app=HOST|name=obproxy,process_memory_rss_bytes|app=HOST|name=observer,process_memory_rss_bytes|app=HOST|name=obproxy,process_thread_count|app=HOST|name=observer,process_thread_count|app=HOST|name=obproxy,process_uptime_seconds|app=HOST|name=observer,process_uptime_seconds|app=HOST|name=obproxy,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_latch_get_total,ob_latch_miss_total,ob_latch_sleep_total,ob_latch_immediate_get_total,ob_latch_immediate_miss_total,ob_latch_spin_get_total,ob_latch_wait_seconds_total,ob_tenant_context_memory_hold_bytes,ob_tenant_context_memory_used_bytes,ob_tenant_context_memory_count 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=30 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=30 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=180 2024-11-04 12:29:32.595 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.logproxy.exporters=/metrics/logproxy 2024-11-04 12:29:32.596 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:29:32.596 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.archive.downgrade.enabled=true 2024-11-04 12:29:32.596 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.interval.factor=2 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.service.type=LOCAL 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.exporters=/metrics/ob/extra,/metrics/obcloud/extra 2024-11-04 12:29:32.596 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.write.queue.capacity=5000000 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.prometheus.service.discovery.enabled=true 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy,/metrics/node/oblb,/metrics/node/obdns,/metrics/logproxy 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.write.queue.capacity=5000000 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.server.exporter.auth.enabled=true 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.ConnectionTimeout=2000 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.maximumPoolSize=200 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.minimumIdle=10 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.poolName=monitor-connect-pool 2024-11-04 12:29:32.596 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.host=10.46.128.24 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.password=****** 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.port=2883 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.public-key=null 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.series.scan.batch-size=2000 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.username=root@ocp_monitor 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.multicloud.monitordb.password-secret-key=****** 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist= 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.database.name.pattern=^[a-z][a-z_0-9]{1,127}$ 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.enable=false 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.standby.replication.role.privilege.objects=DBA_OB_TENANTS,DBA_OB_LS_HISTORY,GV$OB_UNITS,GV$OB_PARAMETERS,GV$OB_LOG_STAT,DBA_OB_LS,DBA_OB_ACCESS_POINT 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.switchover.check.to.primary.wait-period-seconds=3 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2024-11-04 12:29:32.597 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.clean-tmp-rpm=true 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obcloud.api.address= 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obproxy.query-with-ob-connections=true 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.cgroup.minimum-ob-version=4.0.0.0 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.default.os.user=admin 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2024-11-04 12:29:32.598 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.host.arm-lse-supported=true 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2024-11-04 12:29:32.600 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2024-11-04 12:29:32.601 INFO 16260 --- [ocp-schedules-2,8f64da0417450ee0,3e5dcea6fceae8e6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.prohibit.obsharding.operation=true 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.skip-set-proxyro-password=****** 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.backup.status.refresh.interval=300000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.performance.status.refresh.interval=300000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2024-11-04 12:29:32.601 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.count-sql.slow-sql-explicitly=true 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.oas.stat.enabled=true 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.password=****** 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.public.key=****** 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=0 2024-11-04 12:29:32.602 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable.in.147x-meta-cluster=false 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":1000} 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=False 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2024-11-04 12:29:32.603 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2},"alarm":true} 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2}, "alarm":true, "sysTenantEnabled":false} 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2024-11-04 12:29:32.604 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=True 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2024-11-04 12:29:32.605 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.rows.limit=1000000 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table.limit=32 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table4.limit=20 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=5d 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2024-11-04 12:29:32.606 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=7d 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=10m 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled=false 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled.in.147x-meta-cluster=true 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql.query-parallel=0 2024-11-04 12:29:32.607 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-multiply-of-plan-comparison={"0":10, "1":5, "10":2, "100":1.5, "1000":1.2} 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.from-header=false 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.header-candidates=X-Forwarded-For 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.private-key=****** 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.public-key=****** 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.event.included={sql: {}, daily_merge:{}, election:{}, physical_restore:{}, observice:{}, observer:{}, clog_restore:{}, warmup:{}, log_archive:{}, archive_log:{}, minor_merge:{}, ddl:{"create_partition", "batch_create_partition", "add_replica", "remove_member", "batch_remove_member", "batch_modify_quorum", "batch_change_replica", "migrate_replica", "remove_replica", "rebuild_replica"}, freeze:{"do minor freeze fail"}, storage:{"add_replica failed", "migrate_replica failed", "reubild_replica failed", "change_replica failed", "log is new enough, turn off rebuild flag", "remove_replica failed", "restore follower replica failed", "restore_replica failed", "batch change_replica failed", "copy global index failed", "restore_standby_replica failed", "rebuild_replica failed"}} 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.readonly.mode.enabled=false 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.site.url=http://10.46.128.24:8090 2024-11-04 12:29:32.608 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.standby.tenant.delay.threshold=5000 2024-11-04 12:29:32.613 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2024-11-04 12:29:32.613 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2024-11-04 12:29:32.613 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.minimum-timeout-secs=-1 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.prohibit.skip.names=[] 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.language=zh-CN 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.timezone= 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.event.excluded={balancer: {"start_sql_bkgd_task","finish_sql_bkgd_task"}, schema_recycler: {}} 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tag.usage.max.count=5000 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.prohibit.skip.names=["Upgrade OB cluster"] 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.fixedDelay=10 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.initialDelay=60 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.fixedDelay=60 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.initialDelay=60 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.fixedDelay=60 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.initialDelay=60 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.fixedDelay=10 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.initialDelay=60 2024-11-04 12:29:32.614 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.fixedDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.fixedDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.fixedDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.fixedDelay=20 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.fixedDelay=180 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.fixedDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.fixedDelay=10 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.fixedDelay=600 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.fixedDelay=600 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.fixedDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.initialDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogClusters.fixedDelay=30 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogInstances.fixedDelay=60 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.fixedRate=30 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.initialDelay=120 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.fixedRate=30 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.initialDelay=120 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.fixedRate=30 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.initialDelay=120 2024-11-04 12:29:32.615 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.fixedRate=30 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.initialDelay=120 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.fixedDelay=4 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.initialDelay=30 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.timeUnit=MINUTES 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.fixedDelay=1 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.initialDelay=2 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.timeUnit=MINUTES 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.fixedDelay=10 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.initialDelay=2 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.timeUnit=MINUTES 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkOdpClient.fixedDelay=60 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkVpcConnect.fixedDelay=5 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskLog.cron=0 10 3 * * * 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskRecords.cron=0 20 5 * * * 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTmpLogFiles.cron=0 20 4 * * * 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.fixedDelay=300 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.initialDelay=600 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.collectOdpMetrics.fixedDelay=60 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.ocpReplicationDaemon.fixedDelay=5 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.fixedDelay=600 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.initialDelay=120 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.refreshOcpConfig.fixedDelay=180 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkAllArbitrationService.fixedRate=30 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterRootServer.fixedRate=300 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterStatus.fixedRate=30 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkStandbyClusterInfo.fixedRate=30 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.collectAllClusterCompaction.fixedDelay=60 2024-11-04 12:29:32.616 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterInfo.fixedRate=30 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterSystemEvent.fixedDelay=600 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncRootServer.fixedDelay=120 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllStandbyTenant.fixedRate=30 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllTenantServiceNameConflict.fixedDelay=60 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMajorCompaction.fixedDelay=60 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMiniCompaction.fixedDelay=300 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.syncAllTenantInfo.fixedDelay=60 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.cron=10/30 * * * * * 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.initialDelay=120 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.cron=10/30 * * * * * 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.initialDelay=120 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.fixedDelay=300 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.initialDelay=120 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectSlowSqlForAlarm.fixedDelay=60 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectTransStatForAlarm.fixedDelay=60 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.oasSqlCuring.fixedDelay=60 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.pool.coreSize=30 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.mysql=^[a-z][a-z_0-9]{1,127}$ 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version=4.3.2 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version.full=4.3.2-20241012145836 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.context-path=/ 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.tracking.enabled=1 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.syslog-level=INFO 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.work-thread-num=8 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2024-11-04 12:29:32.617 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.address=0.0.0.0 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.port=8090 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.name=SESSION 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.same-site=null 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.secure=null 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.timeout=30m 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.enabled=false 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-alias=null 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-password=null 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store= 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-password=****** 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2024-11-04 12:29:32.618 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.max-page-size=5000 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxActive=200 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.ConnectionTimeout=2000 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.maximumPoolSize=200 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.minimumIdle=10 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.poolName=metadb-connect-pool 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.enabled=false 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-exclude=null 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2024-11-04 12:29:32.619 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2024-11-04 12:29:32.620 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2024-11-04 12:29:32.620 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2024-11-04 12:29:32.620 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2024-11-04 12:29:32.620 INFO 16260 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ****************************************************************************** 2024-11-04 12:29:32.626 INFO 16260 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : Started OcpCeServerApplication in 257.32 seconds (JVM running for 258.569) 2024-11-04 12:29:32.629 INFO 16260 --- [ocp-schedules-3,af48437a19d2d428,0b4f8c08588cc1d0] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:29:32.635 INFO 16260 --- [ocp-schedules-22,1c79c0daeb891057,16ced100bfc6c946] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:29:32.635 INFO 16260 --- [ocp-schedules-22,1c79c0daeb891057,16ced100bfc6c946] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2024-11-04 12:29:32.644 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:29:32.651 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:29:32.651 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:29:32.651 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:29:32.651 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:29:32.651 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:29:32.653 INFO 16260 --- [ocp-schedules-17,aae596c5af530164,c8390cb0cbc5f8be] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:29:32.653 INFO 16260 --- [ocp-schedules-17,aae596c5af530164,c8390cb0cbc5f8be] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:29:32.658 INFO 16260 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : Setup sensitive data utils, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYo+dfEA+ti3lp8yywLmeYA2IPgY0aLvYoP9drFzPTopFyNoOCSB6O+ieFk01zZw4wQLLUUF8WZV+FgS62Eyk2sfwbrRfiMS6z6SpCHJYTsrB99sefX8HeUFDXENz1L0GfutE734bJc06TI5vqzy8aUTtxcCW/mBZbRT4bp5Df5wIDAQAB 2024-11-04 12:29:32.661 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_dynamic_sql 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2024-11-04 12:29:32.661 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat_v2 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_server_event 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_async 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2024-11-04 12:29:32.662 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2024-11-04 12:29:32.665 INFO 16260 --- [ocp-schedules-20,e095253411645709,1c2f0e95f6041979] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2024-11-04 12:29:32.665 INFO 16260 --- [ocp-schedules-20,e095253411645709,1c2f0e95f6041979] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2024-11-04 12:29:32.665 INFO 16260 --- [ocp-schedules-20,e095253411645709,1c2f0e95f6041979] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2024-11-04 12:29:32.667 INFO 16260 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2024-11-04 12:29:32.668 ERROR 16260 --- [ocp-async-9,a04bdfc8d9af0c65,2847bc9ca97fd8a9] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [task not exist., 999989] 2024-11-04 12:29:32.668 INFO 16260 --- [ocp-async-9,a04bdfc8d9af0c65,2847bc9ca97fd8a9] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Task not found, task id :999989 2024-11-04 12:29:32.669 INFO 16260 --- [ocp-async-9,a04bdfc8d9af0c65,2847bc9ca97fd8a9] c.o.o.s.c.l.ApplicationStartedListener : Begin to init proxyro password config in sys tenant 2024-11-04 12:29:32.672 INFO 16260 --- [ocp-async-9,a04bdfc8d9af0c65,2847bc9ca97fd8a9] c.o.o.s.c.l.ApplicationStartedListener : Success to init proxyro password config in sys tenant 2024-11-04 12:29:32.681 INFO 16260 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP application uuid is 0083958465584847af344ffec220e32d. 2024-11-04 12:29:32.681 INFO 16260 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP boot ok! 2024-11-04 12:29:32.684 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT4M17.37883952S 2024-11-04 12:29:32.684 INFO 16260 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2024-11-04 12:29:32.686 INFO 16260 --- [ocp-async-3,0d310bfd096bc16a,32843c184565cf93] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [] 2024-11-04 12:29:32.686 INFO 16260 --- [ocp-async-3,0d310bfd096bc16a,32843c184565cf93] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [] 2024-11-04 12:29:32.689 INFO 16260 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2024-11-04 12:29:32.689 INFO 16260 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : SetProxyIdAfterUpgrade created. 2024-11-04 12:29:32.700 WARN 16260 --- [main,,] c.o.o.o.i.ObproxyOperationServiceImpl : No obproxy needs to set proxy_id, input param is empty. 2024-11-04 12:29:32.700 INFO 16260 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : Submit task null success, assign proxy_id for obproxies []. 2024-11-04 12:29:32.701 INFO 16260 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2024-11-04 12:29:32.706 INFO 16260 --- [ocp-schedules-17,aae596c5af530164,c8390cb0cbc5f8be] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:29:32.706 INFO 16260 --- [ocp-schedules-17,aae596c5af530164,c8390cb0cbc5f8be] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:29:33.036 INFO 16260 --- [ocp-async-10,a5f68b8a08b5f626,04b8d8a225554915] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2024-11-04 12:29:33.051 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:29:33.200 INFO 16260 --- [ocp-async-10,a5f68b8a08b5f626,04b8d8a225554915] c.o.o.s.common.config.TaskEngineRunner : Startup subtask coordinator. subtask=592 2024-11-04 12:29:33.200 INFO 16260 --- [ocp-async-10,a5f68b8a08b5f626,04b8d8a225554915] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2024-11-04 12:29:33.481 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.885519209S 2024-11-04 12:29:33.481 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:29:33.555 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:29:33.555 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:29:33.585 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:29:33.662 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:29:33.662 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:29:33.662 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:29:33.662 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:29:33.662 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:29:33.662 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:29:33.671 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:29:33.791 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] o.s.boot.SpringApplication : Started application in 1.206 seconds (JVM running for 259.734) 2024-11-04 12:29:33.792 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.206886297S 2024-11-04 12:29:34.142 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:34.143 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:34.249 INFO 16260 --- [ocp-schedules-15,3e6b6ffd159bfab9,befded5785768d67] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:29:34.536 INFO 16260 --- [ocp-async-2,808f7f780101ae04,4170704eb1199490] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:29:34.537 WARN 16260 --- [ocp-async-6,fd2cf81adbd896b8,5de0effaa654b9b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:29:34.626 INFO 16260 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:29:34.647 INFO 16260 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:29:34.665 INFO 16260 --- [prometheus-minute-schedule-1,ca0890a29aeebdb0,72b0d82d46ff0160] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694600, interval=60 2024-11-04 12:29:34.665 INFO 16260 --- [prometheus-hour-schedule-1,1534fb88c092f480,14490c24ea14c289] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:29:36.536 INFO 16260 --- [ocp-async-2,abf1ed8ba09ce441,1559492f8b16eec8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:29:36.537 WARN 16260 --- [ocp-async-1,5859e81516eb3d9c,f364e0bde8ee30fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:29:37.144 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:37.146 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:37.664 INFO 16260 --- [prometheus-minute-schedule-1,64310750483fdacf,ac2a4facde665dfa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694600, interval=60 2024-11-04 12:29:38.536 INFO 16260 --- [ocp-async-2,1c1c17c6038d8a2d,c311cee587ec8b78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:29:38.538 WARN 16260 --- [ocp-async-3,465ee2c0877a0753,52878dd79941ceaa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:29:40.004 INFO 16260 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:29:40.004 INFO 16260 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:29:40.146 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:40.148 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:40.536 INFO 16260 --- [ocp-async-7,bf710f4911654c44,43ec177927bd4dc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:29:40.538 WARN 16260 --- [ocp-async-8,db15edbfc19e6cef,19d1c9ce875d9cff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:29:40.664 INFO 16260 --- [prometheus-minute-schedule-1,68971c6a9b73c8b8,6e9c2206caaa9432] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694600, interval=60 2024-11-04 12:29:42.535 INFO 16260 --- [ocp-async-1,cdea56b13346e1ba,a3748ad46dd461a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:29:42.537 INFO 16260 --- [ocp-async-5,1df59cca1af2996a,be382abe6e6b8832] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:29:42.538 WARN 16260 --- [ocp-async-3,5f7bf05638cfd2ac,29352feb54ba645f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:29:42.556 INFO 16260 --- [ocp-schedules-1,5df1029b126a7346,92b62ba5830b1a4e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:29:43.148 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:43.149 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:29:43.664 INFO 16260 --- [prometheus-minute-schedule-1,081e8d8fd2b33ccd,18fc5d47b0b797e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694600, interval=60 2024-11-04 12:29:44.537 INFO 16260 --- [ocp-async-9,c784c217dd9d421d,6877519cdfc7c1d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:29:44.538 WARN 16260 --- [ocp-async-8,d99e6c76410f9e77,9042dc00be3ba095] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:29:45.462 INFO 16260 --- [sql-rollup-daemon,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=ocp_monitor, username=root, address=10.46.128.24, port=2883, database=monitor_database) 2024-11-04 12:29:45.462 INFO 16260 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2883, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2024-11-04 12:29:45.478 INFO 16260 --- [obsdk-init-thread-1,04f92563c4c76f7b,64d384868e81fa38] com.alibaba.druid.pool.DruidDataSource : {dataSource-8,ob-connector-datasource_direct_ocp_monitor_root_10.46.128.24_2883} inited 2024-11-04 12:29:45.478 INFO 16260 --- [obsdk-init-thread-1,04f92563c4c76f7b,64d384868e81fa38] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2883, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2024-11-04 12:29:45.479 INFO 16260 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2024-11-04 12:29:45.494 INFO 16260 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:29:45.496 INFO 16260 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:29:45.499 INFO 16260 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:29:45.500 INFO 16260 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:29:45.501 INFO 16260 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:29:45.506 INFO 16260 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2024-11-04 12:29:46.149 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:29:46.150 INFO 16260 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:30:03.803 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:30:03.803 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:30:03.803 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:30:03.803 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:30:03.804 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:30:03.804 INFO 21062 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : The following 1 profile is active: "composite" 2024-11-04 12:30:03.815 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:30:11.203 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2024-11-04 12:30:11.204 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2024-11-04 12:30:11.474 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 234 ms. Found 4 JPA repository interfaces. 2024-11-04 12:30:11.489 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2024-11-04 12:30:11.489 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2024-11-04 12:30:11.765 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 275 ms. Found 16 JPA repository interfaces. 2024-11-04 12:30:11.767 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2024-11-04 12:30:11.767 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2024-11-04 12:30:13.452 INFO 21062 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1685 ms. Found 157 JPA repository interfaces. 2024-11-04 12:30:18.042 INFO 21062 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=0f233f8c-7d1e-3396-84b4-7fa06161d9e7 2024-11-04 12:30:19.922 INFO 21062 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'commonSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.server.common.config.security.conf.CommonSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$31e2907] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2024-11-04 12:30:19.950 INFO 21062 --- [main,,] .o.s.c.c.s.c.CommonSecurityConfiguration : Method-level access control enabled. 2024-11-04 12:30:20.938 INFO 21062 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] driver class=com.oceanbase.jdbc.Driver 2024-11-04 12:30:20.938 INFO 21062 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] username=root@ocp_meta 2024-11-04 12:30:20.938 INFO 21062 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] jdbc url=jdbc:oceanbase://10.46.128.24:2883/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2024-11-04 12:30:20.946 INFO 21062 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Starting... 2024-11-04 12:30:21.015 INFO 21062 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Start completed. 2024-11-04 12:30:21.027 INFO 21062 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2024-11-04 12:30:21.027 INFO 21062 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2024-11-04 12:30:21.071 INFO 21062 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=10.46.128.24:8090, name=a1e196d99e 2024-11-04 12:30:21.139 WARN 21062 --- [main,,] c.o.o.c.d.server.DefaultServerUpdater : Deleted expired server: DistributedServer{id=10.46.128.24:8090, name=a1e196d99e, expireInSeconds=30, createTime=2024-11-04T12:26:56+08:00, updateTime=2024-11-04T12:29:41+08:00, expireTime=2024-11-04T12:30:11+08:00, isActive=false} 2024-11-04 12:30:21.166 INFO 21062 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 10.46.128.24:8090 2024-11-04 12:30:22.639 INFO 21062 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8090 (http) 2024-11-04 12:30:22.678 INFO 21062 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2024-11-04 12:30:22.678 INFO 21062 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.83] 2024-11-04 12:30:22.966 INFO 21062 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2024-11-04 12:30:22.966 INFO 21062 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 19151 ms 2024-11-04 12:30:24.310 INFO 21062 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2024-11-04 12:30:24.870 INFO 21062 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.15.Final 2024-11-04 12:30:26.101 INFO 21062 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2024-11-04 12:30:30.684 INFO 21062 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2024-11-04 12:30:32.638 WARN 21062 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2024-11-04 12:30:34.054 WARN 21062 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2024-11-04 12:30:34.054 WARN 21062 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2024-11-04 12:30:39.072 INFO 21062 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2024-11-04 12:30:39.163 INFO 21062 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2024-11-04 12:30:41.598 INFO 21062 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2024-11-04 12:30:42.464 INFO 21062 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2024-11-04 12:30:42.476 INFO 21062 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2024-11-04 12:30:42.476 INFO 21062 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] username=root@ocp_monitor 2024-11-04 12:30:42.476 INFO 21062 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] jdbc url=jdbc:oceanbase://10.46.128.24:2883/monitor_database?useUnicode=true&characterEncoding=utf-8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=convertToNull 2024-11-04 12:30:42.476 INFO 21062 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Starting... 2024-11-04 12:30:42.484 INFO 21062 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Start completed. 2024-11-04 12:30:42.484 INFO 21062 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2024-11-04 12:30:42.484 INFO 21062 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2024-11-04 12:30:42.530 INFO 21062 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] driver class=com.oceanbase.jdbc.Driver 2024-11-04 12:30:42.530 INFO 21062 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] username=root@ocp_meta 2024-11-04 12:30:42.530 INFO 21062 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] jdbc url=jdbc:oceanbase://10.46.128.24:2883/meta_database?useUnicode=true&characterEncoding=UTF8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2024-11-04 12:30:42.570 INFO 21062 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2024-11-04 12:30:42.570 INFO 21062 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2024-11-04 12:30:43.602 INFO 21062 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2024-11-04 12:30:47.494 INFO 21062 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2024-11-04 12:30:48.081 INFO 21062 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2024-11-04 12:30:48.082 INFO 21062 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2024-11-04 12:31:11.938 INFO 21062 --- [main,,] org.reflections.Reflections : Reflections took 12151 ms to scan 81 urls, producing 1062 keys and 7488 values 2024-11-04 12:31:22.460 INFO 21062 --- [main,,] org.reflections.Reflections : Reflections took 10427 ms to scan 81 urls, producing 1062 keys and 7488 values 2024-11-04 12:31:33.563 INFO 21062 --- [main,,] org.reflections.Reflections : Reflections took 11074 ms to scan 81 urls, producing 985 keys and 4895 values 2024-11-04 12:31:33.602 INFO 21062 --- [main,,] c.o.o.s.c.analyzer.OcpStartUpChecker : audit event is disable 2024-11-04 12:31:41.487 INFO 21062 --- [main,,] org.reflections.Reflections : Reflections took 7323 ms to scan 81 urls, producing 985 keys and 4895 values 2024-11-04 12:31:41.514 INFO 21062 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116021] DistributedEventManager scanned event classes: [class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent, class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent, class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent, class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent, class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent] 2024-11-04 12:31:41.516 INFO 21062 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent failed, error: null 2024-11-04 12:31:41.516 INFO 21062 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent failed, error: null 2024-11-04 12:31:41.516 INFO 21062 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent failed, error: null 2024-11-04 12:31:41.516 INFO 21062 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116022] DistributedEventManager scanned event class map: {refresh ocp cluster cache=class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, reset-all-obproxy-id=class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, refresh-config-server-cache=class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, tenant-renamed=class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent, invalidate-ob-version-cache=class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent} 2024-11-04 12:31:41.548 INFO 21062 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116001] DistributedEventManager started on a1e196d99e 2024-11-04 12:31:41.736 INFO 21062 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2024-11-04 12:31:45.040 INFO 21062 --- [main,,] c.o.o.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2024-11-04 12:31:46.054 INFO 21062 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2024-11-04 12:31:46.091 INFO 21062 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize global tasks done 2024-11-04 12:31:46.091 INFO 21062 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize done 2024-11-04 12:31:47.338 INFO 21062 --- [main,,] c.o.o.alarm.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=ob_log_clog_write_error,ob_log_right_to_die,ob_host_net_send_percent_over_threshold,binlog_host_storage_usage_over_threshold,oblb_etcd_leader_absent,ob_tenant_compaction_error,ob_log_easy_hold_timeout,odp_instance_expanse_failed,binlog_host_cpu_usage_over_threshold,ob_log_decent_voting,ob_cluster_frozen_version_delta_over_threshold,binlog_node_status_abnormal,ob_log_commit_error,obproxy_memory_peak,host_operation,ob_log_mysql_request_full,ob_log_timer_cost_much,ob_log_worker_thread_full,host_disk_readonly,node_file_root_usage,rootservice_log_alarm,ob_log_disk_almost_full,obdns_etcd_cannot_connect_to_leader,oms_sync_failed,binlog_service_conversion_delay,ob_cluster_sync_delay_time_too_long,arbitration_service_unavailable,ob_cluster_merge_timeout,ob_common_unexpected_internal_error,partition_create_failed,ob_cluster_recyclebin_disk_used_over_threshold,ob_log_mem_alloc_error,ob_rootservice_stop_fail,base_backup_timeout,refresh_location_cache_failed,host_ntp_offset_too_large,ob_log_error_lib_direct_alloc,ob_host_invalid_disk_exists,ob_cluster_no_frozen,ocp_collect_metric_failure_rate_high,ob_log_distributed_strong_read,ob_tenant_cpu_usage_over_threshold,ob_log_fail_check_localserver,host_agent_open_fd_count_over_threshold,ob_cluster_sync_failed,oblb_ip_forward_invalid,backup_process_dead,oblb_keepalived_service_inactive,oblb_process_dead,ob_tenant500_storage_short_meta_mem_hold_high,ob_host_install_disk_percent_over_threshold,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,binlog_instance_operation_info,obagent_dead,ob_server_stop_fail,ob_server_cannot_connect_arbitration,ob_log_rpc_error,ob_server_sstable_percent_over_threshold,oms_sync_delay,ob_host_monitordb_disconnected,oblb_keepalived_become_master,os_tsar_traffic_drop,ob_log_unit_migrate_error,ob_log_reply_memstore_failed,host_agent_version_not_same,ob_log_global_trans_version_delay,ob_tenant_partition_leader_absent,ob_cluster_operation_info,ob_cluster_merge_error,ob_tenant_name_mismatch,backup 2024-11-04 12:31:48.622 INFO 21062 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table switch time is 1730694377 2024-11-04 12:31:49.898 INFO 21062 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:31:49.942 INFO 21062 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:31:50.040 INFO 21062 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:31:50.052 INFO 21062 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:31:50.149 INFO 21062 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:31:50.180 INFO 21062 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:31:50.189 INFO 21062 --- [main,,] c.o.o.m.query.LocalMetricQueryService : Monitor router init, username=system, connectTimeoutMs=3000, readTimeoutMs=120000 2024-11-04 12:31:51.113 INFO 21062 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances start. 2024-11-04 12:31:51.145 INFO 21062 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances end. 2024-11-04 12:31:51.169 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2024-11-04 12:31:51.170 INFO 21062 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2024-11-04 12:31:51.172 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2024-11-04 12:31:51.184 INFO 21062 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2024-11-04 12:31:51.185 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2024-11-04 12:31:51.186 INFO 21062 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2024-11-04 12:31:51.187 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.query.global.timeout.millis = 10000 2024-11-04 12:31:51.187 INFO 21062 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set queryGlobalTimeoutMillis=10000 2024-11-04 12:31:51.188 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2024-11-04 12:31:51.188 INFO 21062 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2024-11-04 12:31:51.189 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 10000 2024-11-04 12:31:51.189 INFO 21062 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=10000 2024-11-04 12:31:51.191 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2024-11-04 12:31:51.191 INFO 21062 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2024-11-04 12:31:51.192 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 2000 2024-11-04 12:31:51.195 INFO 21062 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=2000 2024-11-04 12:31:51.196 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2024-11-04 12:31:51.196 INFO 21062 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2024-11-04 12:31:51.198 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2024-11-04 12:31:51.198 INFO 21062 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2024-11-04 12:31:51.200 INFO 21062 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2024-11-04 12:31:51.200 INFO 21062 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2024-11-04 12:31:51.430 INFO 21062 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=2, heartbeatTime=2024-11-04T12:29:43+08:00, now=2024-11-04T12:31:51.429+08:00 2024-11-04 12:31:51.430 INFO 21062 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=1, heartbeatTime=2024-11-04T12:29:43+08:00, now=2024-11-04T12:31:51.429+08:00 2024-11-04 12:31:51.617 INFO 21062 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-rule, name=10.46.128.24:8090, consumerId=1, resources=[ShardingSlice(consumerId=1, type=DEFAULT, resourceIds=[274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 336, 337, 362, 363, 505, 506, 507, 508, 509])] 2024-11-04 12:31:53.790 INFO 21062 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/dba/obtest/software 2024-11-04 12:31:54.618 INFO 21062 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from OFFLINE to NEW. 2024-11-04 12:31:54.619 INFO 21062 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from OFFLINE to NEW. 2024-11-04 12:31:54.969 INFO 21062 --- [main,,] c.o.o.s.common.config.BeanConfiguration : begin set auto discover false. 2024-11-04 12:31:55.006 INFO 21062 --- [main,,] c.o.o.s.common.config.BeanConfiguration : set auto discover false success. 2024-11-04 12:31:56.634 INFO 21062 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByTargetIdShardingStrategy@2de0a3c7 2024-11-04 12:31:56.636 INFO 21062 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=1, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByRuleShardingStrategy@7a684ccd 2024-11-04 12:31:56.638 INFO 21062 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2024-11-04 12:31:56.638 INFO 21062 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2024-11-04 12:31:57.628 INFO 21062 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2024-11-04 12:31:57.630 INFO 21062 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2024-11-04 12:31:57.664 INFO 21062 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2024-11-04 12:31:57.779 INFO 21062 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2024-11-04 12:32:00.252 INFO 21062 --- [main,,] c.o.o.alarm.schedule.AlarmScheduledJobs : AlarmSchedules construct done 2024-11-04 12:32:00.638 INFO 21062 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2024-11-04 12:32:00.638 INFO 21062 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2024-11-04 12:32:00.638 INFO 21062 --- [heartbeat-alarm-detect-by-rule-10.46.128.24:80901,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2024-11-04 12:32:00.638 INFO 21062 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2024-11-04 12:32:01.089 INFO 21062 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2024-11-04T12:32:00.344+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system- admin-, subscribedJudgerVersion 2024-11-04 12:32:01.097 INFO 21062 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2024-11-04 12:32:01.097 INFO 21062 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2024-11-04T12:32:00.344+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system- admin-, subscribedJudgerVersion 2024-11-04 12:32:01.097 INFO 21062 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2024-11-04 12:32:01.098 INFO 21062 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2024-11-04 12:32:01.164 INFO 21062 --- [main,,] c.o.o.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/home/dba/ocp-server/alarm 2024-11-04 12:32:01.418 INFO 21062 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2024-11-04 12:32:01.495 INFO 21062 --- [main,,] c.o.o.alarm.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=10.46.128.24:21062 2024-11-04 12:32:01.604 INFO 21062 --- [main,,] c.o.o.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=169 2024-11-04 12:32:01.681 INFO 21062 --- [main,,] c.o.o.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=169 2024-11-04 12:32:01.709 INFO 21062 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2024-11-04 12:32:01.781 INFO 21062 --- [main,,] c.o.ocp.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2024-11-04 12:32:02.145 INFO 21062 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 10.46.128.24:8090 2024-11-04 12:32:02.159 INFO 21062 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2024-11-04 12:32:02.234 WARN 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2024-11-04 12:32:02.234 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2024-11-04 12:32:04.669 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2024-11-04 12:32:04.721 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2024-11-04 12:32:04.755 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:32:04.769 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:32:05.236 WARN 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2024-11-04 12:32:05.236 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2024-11-04 12:32:06.441 INFO 21062 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2024-11-04 12:32:06.504 INFO 21062 --- [main,,] c.o.o.s.c.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2024-11-04 12:32:07.153 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=CMDB_SYNCER, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.155 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=MC_MANAGER, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.156 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PERF_SQL_ROLLUP, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.158 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.159 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=RATE_LIMIT, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.160 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.161 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.163 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=TASK_MANAGER, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:28:44+08:00, updateTime=2024-11-04T12:29:44+08:00, expireTime=null, isLeader=false} 2024-11-04 12:32:07.164 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.166 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.168 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.170 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.172 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.175 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.177 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.179 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2024-11-04 12:32:07.680 INFO 21062 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Validate and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYo+dfEA+ti3lp8yywLmeYA2IPgY0aLvYoP9drFzPTopFyNoOCSB6O+ieFk01zZw4wQLLUUF8WZV+FgS62Eyk2sfwbrRfiMS6z6SpCHJYTsrB99sefX8HeUFDXENz1L0GfutE734bJc06TI5vqzy8aUTtxcCW/mBZbRT4bp5Df5wIDAQAB, valid=true 2024-11-04 12:32:07.680 INFO 21062 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Sensitive data encryption key is legal. 2024-11-04 12:32:07.749 INFO 21062 --- [main,,] c.o.o.a.c.AlarmConfiguration : alarm configuration set from system properties 2024-11-04 12:32:08.236 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 gets the leadership... 2024-11-04 12:32:08.648 INFO 21062 --- [main,,] c.o.o.m.promethues.InnerMetricContainer : init inner metric container success, configToMetricMap = {InnerMetricContainer.InnerMetricConfig(scrapeType=second)=[system_cpu_count, jvm_threads_states_threads, jvm_gc_memory_promoted_bytes_total, jvm_memory_committed_bytes, ocp_monitor_collect_request_errors_total, ocp_meta_db_health, jvm_gc_pause_seconds_max, ocp_monitor_collect_request_duration_ms_count, system_cpu_usage, process_threads, http_server_requests_seconds_count, jvm_gc_memory_allocated_bytes_total, jvm_threads_peak_threads, ocp_alarm_detect_duration_ms_count, jvm_memory_used_bytes, process_cpu_usage, jvm_memory_max_bytes, jvm_gc_max_data_size_bytes, http_server_requests_seconds_sum, jvm_gc_pause_seconds_sum, ocp_alarm_detect_duration_ms_sum, jvm_threads_live_threads, system_load_average_1m, jvm_gc_pause_seconds_count, jvm_gc_overhead_percent]} 2024-11-04 12:32:08.662 INFO 21062 --- [main,,] c.o.o.s.c.config.MetricConfiguration : Monitor sharding strategy is monitor 2024-11-04 12:32:09.463 INFO 21062 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2024-11-04 12:32:10.515 INFO 21062 --- [main,,] c.o.o.s.c.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/home/dba/obtest/ocp/ocp-server/lib 2024-11-04 12:32:10.516 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2024-11-04 12:32:10.555 INFO 21062 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object, bucket=built-in, name=obproxyd.sh, objectId=1 2024-11-04 12:32:10.583 INFO 21062 --- [main,,] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2024-11-04 12:32:10.630 INFO 21062 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/dba/obtest/software/built-in/obproxyd.sh.sha1sum 2024-11-04 12:32:10.630 INFO 21062 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/dba/obtest/software/built-in/obproxyd.sh 2024-11-04 12:32:10.630 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2024-11-04 12:32:10.671 INFO 21062 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=8 2024-11-04 12:32:10.702 INFO 21062 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=8, name=obproxyd.sh 2024-11-04 12:32:10.718 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-libs-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:32:10.749 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-libs-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:32:10.749 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-utils-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:32:10.756 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-utils-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:32:10.756 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 12:32:10.762 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 12:32:10.762 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.2-20241012145836.el7.aarch64.rpm 2024-11-04 12:32:10.769 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.2-20241012145836.el7.aarch64.rpm 2024-11-04 12:32:10.769 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:obproxy-ce-4.3.2.0-26.el7.x86_64.rpm 2024-11-04 12:32:10.775 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=obproxy-ce-4.3.2.0-26.el7.x86_64.rpm 2024-11-04 12:32:10.775 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:32:10.781 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-4.3.3.1-101000012024102216.el7.x86_64.rpm 2024-11-04 12:32:10.783 INFO 21062 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Save matched ocp-agent version, version=4.3.2-20241012145836 2024-11-04 12:32:10.793 INFO 21062 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.agent.version.full, previousValue=4.3.2-20241012145836, updatedValue=4.3.2-20241012145836, runningValue=4.3.2-20241012145836 2024-11-04 12:32:10.960 INFO 21062 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanDistributedEvent":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":1,"tableName":"distributed_event"},"cleanDistributedEventReception":{"batchIntervalMillis":0,"batchSize":0,"condition":"event_id in (select id from distributed_event where expire_time <= date_sub(curdate(), interval 30 day))","priority":0,"tableName":"distributed_event_reception"},"cleanStatusTransition":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":0,"tableName":"ops_status_transition"}} 2024-11-04 12:32:11.236 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:11.249 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:11.510 INFO 21062 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2024-11-04 12:32:11.797 INFO 21062 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2024-11-04 12:32:11.800 INFO 21062 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2024-11-04 12:32:13.683 INFO 21062 --- [sharding-metric-collect1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.monitor.sharding.MonitorShardingStrategy@20fc4e41 2024-11-04 12:32:13.685 INFO 21062 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2024-11-04 12:32:13.901 INFO 21062 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2024-11-04 12:32:13.980 INFO 21062 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2024-11-04 12:32:13.983 INFO 21062 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:13.983 INFO 21062 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:32:13.985 INFO 21062 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2024-11-04 12:32:14.071 INFO 21062 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ob_cluster_server_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_data_async=1, metric_hour_data=1098, metric_daily_data=1098} 2024-11-04 12:32:14.249 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:14.251 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:14.275 INFO 21062 --- [main,,] .r.DefaultUserResourceManagerServiceImpl : not UPGRADE, skip create default user resource 2024-11-04 12:32:14.596 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2024-11-04 12:32:14.604 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2024-11-04 12:32:14.607 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:32:14.608 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2024-11-04 12:32:14.608 INFO 21062 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2024-11-04 12:32:14.630 INFO 21062 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2024-11-04 12:32:14.636 INFO 21062 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2024-11-04 12:32:14.692 INFO 21062 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2024-11-04 12:32:14.706 INFO 21062 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2024-11-04 12:32:15.169 INFO 21062 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2024-11-04 12:32:15.779 INFO 21062 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=STATELESS. 2024-11-04 12:32:15.798 INFO 21062 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Internal auth configured. 2024-11-04 12:32:15.800 INFO 21062 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Access key auth configured. 2024-11-04 12:32:15.805 INFO 21062 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Basic auth enabled. 2024-11-04 12:32:16.391 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.server.common.config.security.conf.StatelessSecurityConfiguration$$Lambda$2103/826692830@1a9d362b with [org.springframework.security.web.session.DisableEncodeUrlFilter@360d5f46, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@1f8e37f2, org.springframework.security.web.context.SecurityContextPersistenceFilter@23272aed, org.springframework.security.web.header.HeaderWriterFilter@1340392b, org.springframework.security.web.authentication.logout.LogoutFilter@3b446df5, com.oceanbase.ocp.server.common.config.security.accesskey.ContentCacheRequestFilter@6de51db4, com.oceanbase.ocp.server.common.config.security.filter.InternalAuthenticationFilter@1eba524d, com.oceanbase.ocp.server.common.config.security.accesskey.AccessKeyAuthenticationFilter@4211c9b3, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@2c4e57a4, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@4528bd0a, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@5c8ab2bb, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@2d1b1750, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@6903e724, org.springframework.security.web.session.SessionManagementFilter@6cae43e9, org.springframework.security.web.access.ExceptionTranslationFilter@7008b710, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@4178d1b7] 2024-11-04 12:32:16.686 INFO 21062 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2024-11-04 12:32:16.724 INFO 21062 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2024-11-04 12:32:16.997 INFO 21062 --- [main,,] o.o.core.config.InitializationService : Initializing OpenSAML using the Java Services API 2024-11-04 12:32:17.197 INFO 21062 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmlenc#ripemd160 2024-11-04 12:32:17.208 INFO 21062 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160 2024-11-04 12:32:17.228 INFO 21062 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#rsa-ripemd160 2024-11-04 12:32:17.251 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:17.253 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:18.939 INFO 21062 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=IF_REQUIRED. 2024-11-04 12:32:18.942 INFO 21062 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : CSRF configured. 2024-11-04 12:32:19.444 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@62dec58b, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@443f0e8a, org.springframework.security.web.context.SecurityContextPersistenceFilter@6c5643da, org.springframework.security.web.header.HeaderWriterFilter@5856fa36, org.springframework.security.web.csrf.CsrfFilter@19d7afd9, org.springframework.security.web.authentication.logout.LogoutFilter@651de64c, org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter@3bf1fef2, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter@2cdfd44e, com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter@4ebac569, org.springframework.security.oauth2.client.web.OAuth2LoginAuthenticationFilter@108a20aa, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationFilter@7aee0c28, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@26d06c1e, com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter@323607d2, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@43ccdb26, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@7161e248, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@1e092118, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@ec11440, org.springframework.security.web.session.SessionManagementFilter@7cdb3f82, org.springframework.security.web.access.ExceptionTranslationFilter@6e9e769e, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@47341cfa] 2024-11-04 12:32:19.694 INFO 21062 --- [main,,] c.o.o.s.c.config.SpringScheduleConfig : Spring scheduled pool size is 30. 2024-11-04 12:32:19.697 INFO 21062 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2024-11-04 12:32:19.697 INFO 21062 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.s.c.config.MetricConfiguration : Consumer had rebalanced. 2024-11-04 12:32:19.732 INFO 21062 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : storage exporter register: collectEnable=true, periodMinutes=180 2024-11-04 12:32:19.732 INFO 21062 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : register backup exporter, periodMinutes=180 2024-11-04 12:32:20.179 INFO 21062 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2024-11-04 12:32:20.253 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:20.255 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:20.710 WARN 21062 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2024-11-04 12:32:22.132 INFO 21062 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2024-11-04 12:32:23.256 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:23.261 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:26.261 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:26.262 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:26.863 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.863 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/login']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/login'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/docs/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/docs/**'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/favicon.ico']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/favicon.ico'] 2024-11-04 12:32:26.864 WARN 21062 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/pre_login_loading']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2024-11-04 12:32:26.864 INFO 21062 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/pre_login_loading'] 2024-11-04 12:32:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9fc20a781a5f3daa,7b73596f2ed26691] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:26.985 INFO 21062 --- [prometheus-hour-schedule-1,12d5efd9a7bb36ff,943e37a68e2a40bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:32:27.748 INFO 21062 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2024-11-04 12:32:28.179 INFO 21062 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8090 (http) with context path '' 2024-11-04 12:32:28.460 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Ocp scheduled task specified config name, methodName=collectHighRiskSqlAuditStat, configName=collectHighSqlAuditStat 2024-11-04 12:32:28.460 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=4 2024-11-04 12:32:28.460 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectSlowSqlForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectTransStatForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectHighRiskSqlAuditStat, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=oasSqlCuring, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogClusters, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=30, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogInstances, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=14 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshBackupAlarmStat, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmSummary, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAggregators, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshHostAlarmInfo, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=removeInactiveAlarmEvents, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=fixAlarmTarget, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTaskManager, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=20, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmInMemory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshDetectorFactory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=alarmDetectOnce, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshGlobalSilencer, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshActiveAlarmStatus, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=resolveAlarmsOfAbsentTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp multi cluster mode disabled. 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp vpd mode disabled. 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectOdpMetrics, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshOcpConfig, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.461 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskLog, conf=ScheduledConf{initialDelay=0, cron='0 10 3 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTmpLogFiles, conf=ScheduledConf{initialDelay=0, cron='0 20 4 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanupDruidCache, conf=ScheduledConf{initialDelay=600, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=partitionDaemon, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskRecords, conf=ScheduledConf{initialDelay=0, cron='0 20 5 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkOdpClient, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=7 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkCpuBindCore, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostAgentStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAgentConfig, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=10, fixedRate=0, timeUnit=MINUTES, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=getHostAgentAlarms, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=1, fixedRate=0, timeUnit=MINUTES, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostVersion, conf=ScheduledConf{initialDelay=30, cron='null', fixedDelay=4, fixedRate=0, timeUnit=MINUTES, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostClockDiff, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkObproxyServers, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectObproxyMetrics, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkStandbyClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=120, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterSystemEvent, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllClusterCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllArbitrationService, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=300, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=5 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncAllTenantInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllStandbyTenant, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMajorCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMiniCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.462 INFO 21062 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllTenantServiceNameConflict, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2024-11-04 12:32:28.500 INFO 21062 --- [ocp-schedules-4,08941ac3e772b16e,dcb1d951893f929d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:32:28.500 INFO 21062 --- [ocp-schedules-10,8760ec76ca2efcf4,ce1329d1c0197786] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:32:28.501 INFO 21062 --- [ocp-schedules-1,5b2000ff29b44b0f,1bb9cb2cb54ff769] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:32:28.506 INFO 21062 --- [ocp-schedules-21,28b5fd29afb1647a,aafbb85630c2fd50] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:32:28.506 INFO 21062 --- [ocp-schedules-17,fd5c986dd4b52a9b,4415d936c543d7bc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:32:28.512 ERROR 21062 --- [ocp-schedules-17,fd5c986dd4b52a9b,4415d936c543d7bc] com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set 2024-11-04 12:32:28.513 INFO 21062 --- [ocp-schedules-25,f77ad389e302f069,3917539495ebeace] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:32:28.517 INFO 21062 --- [ocp-schedules-24,ebb23e1782bc7977,bbe7cb701e52588c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:32:28.517 INFO 21062 --- [ocp-schedules-11,b6246a47834d3525,02ee429c0d893f1e] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 12:32:28.520 INFO 21062 --- [ocp-schedules-17,fd5c986dd4b52a9b,4415d936c543d7bc] com.alibaba.druid.pool.DruidDataSource : {dataSource-2} inited 2024-11-04 12:32:28.519 INFO 21062 --- [ocp-schedules-24,ebb23e1782bc7977,bbe7cb701e52588c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:32:28.525 INFO 21062 --- [ocp-schedules-19,955070b0ccae4255,1103177a6c5a2fef] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:32:28.527 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:32:28.527 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:32:28.528 INFO 21062 --- [ocp-schedules-4,774a874c64f3c341,402a07e189fee717] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:32:28.528 INFO 21062 --- [ocp-schedules-16,161b1545e0799c89,960303463ee9b568] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 12:32:28.531 INFO 21062 --- [ocp-schedules-20,60f507e256148234,d2eb748699bd5545] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 12:32:28.531 ERROR 21062 --- [ocp-schedules-1,5b2000ff29b44b0f,1bb9cb2cb54ff769] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterInfo(ClusterSchedules.java:77) ... 15 common frames omitted 2024-11-04 12:32:28.531 INFO 21062 --- [ocp-schedules-2,51fcc35219fad28e,ef53e6586c0a4333] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:32:28.534 INFO 21062 --- [ocp-schedules-10,66e7ae6f413903a6,043c1e3ac8f878dc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:32:28.537 INFO 21062 --- [ocp-schedules-26,96fc6a1135b4ce27,836519416cd2182d] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2024-11-04 12:32:28.537 INFO 21062 --- [ocp-schedules-1,a80db0f25c50c203,35c903a1ad19884b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:32:28.539 INFO 21062 --- [ocp-schedules-15,ec3697225f074d46,134feaad908c1a21] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 12:32:28.540 INFO 21062 --- [ocp-schedules-13,f8875852cedb31b8,d0ca08a2f3fca21a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:32:28.540 ERROR 21062 --- [ocp-schedules-1,a80db0f25c50c203,35c903a1ad19884b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) ... 15 common frames omitted 2024-11-04 12:32:28.540 INFO 21062 --- [ocp-schedules-1,eea111e234a1483f,b32fc341a083ecc4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 12:32:28.541 INFO 21062 --- [ocp-schedules-1,eea111e234a1483f,b32fc341a083ecc4] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 12:32:28.541 INFO 21062 --- [ocp-schedules-13,f8875852cedb31b8,d0ca08a2f3fca21a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:32:28.541 INFO 21062 --- [ocp-schedules-23,f05b90d0d16815e6,d84c2a7c42b38cda] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:32:28.543 INFO 21062 --- [ocp-schedules-23,f05b90d0d16815e6,d84c2a7c42b38cda] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:32:28.551 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2024-11-04 12:32:28.551 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : Active profiles: [composite] 2024-11-04 12:32:28.551 INFO 21062 --- [ocp-schedules-29,511519028f53aec3,65f33afe9e260b45] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2024-11-04 12:32:28.551 INFO 21062 --- [ocp-schedules-29,511519028f53aec3,65f33afe9e260b45] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2024-11-04 12:32:28.552 WARN 21062 --- [ocp-async-5,3c245bd2f73e1a7d,aea8b03430a2b6fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:28.554 INFO 21062 --- [ocp-schedules-5,8fec4db99132a57d,4a6ea85a4f0f9341] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:32:28.553+08:00, clusters=[] 2024-11-04 12:32:28.554 INFO 21062 --- [ocp-schedules-28,8d057e55e8b61bd5,f1cc9d82dabed64a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:32:28.553+08:00, clusters=[] 2024-11-04 12:32:28.554 INFO 21062 --- [ocp-schedules-18,e8c21beffee63dd0,deef6ea70cb662df] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:32:28.554 INFO 21062 --- [ocp-schedules-25,37874c75b889bd94,70c9601813644bff] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:32:28.555 INFO 21062 --- [ocp-schedules-13,f8875852cedb31b8,d0ca08a2f3fca21a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:32:28.555 INFO 21062 --- [ocp-schedules-13,f8875852cedb31b8,d0ca08a2f3fca21a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:32:28.555 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : agent.log.query.http.timeout=60s 2024-11-04 12:32:28.555 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : backup.dbname.prefix=backup 2024-11-04 12:32:28.555 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : display-parse-failed-sql=true 2024-11-04 12:32:28.555 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : enable-sql.parse-service=true 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file=/home/dba/ocp-server/log/ocp.log 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.clean-history-on-start=true 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-history=100 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-size=100MB 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.name=/home/dba/obtest/logs/ocp-server.log 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.total-size-cap=10GB 2024-11-04 12:32:28.556 INFO 21062 --- [ocp-schedules-1,eea111e234a1483f,b32fc341a083ecc4] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.web=INFO 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(21062){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %.-2048m%n%wEx 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 21062 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %.-2048m%n%wEx 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : mgragent.log.level=info 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : monagent.log.level=info 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.hidden.parameter.whitelist=__min_full_resource_pool_memory,_rowsets_max_rows,_ob_max_thread_num,_enable_defensive_check,_bloom_filter_ratio,_io_read_batch_size,_io_read_redundant_limit_percentage,_enable_kv_feature 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.enabled=true 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.run.internal=5m 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.holder.capacity=2000 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.timeout.millis=10000 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.ob.connection.mode=direct 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.connect.timeout.ms=10000 2024-11-04 12:32:28.556 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.socket.timeout.ms=10000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.connect.timeout.ms=20000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.socket.timeout.ms=20000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.print.sql=true 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.query.global.timeout.millis=10000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.anomaly.event.analyzer-sql-cpu-percent-limit=15 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.config={"enabled":true, "analyzeRangeMin" : 5, "offsetMin": 1, "cpuPercentLimit" : 90} 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.enabled=true 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.collect.snapshot.status=active 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.snapshot.maxsize=1GB 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.version.full=4.3.2-20241012145836 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.auto.resolved.alarm-types=ob_tenant_task_timeout,ob_host_task_timeout,standby_tenant_sync_delay_too_long 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-expired-seconds=600 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2024-11-04 12:32:28.557 INFO 21062 --- [ocp-schedules-15,ec3697225f074d46,134feaad908c1a21] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 12:32:28.557 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2024-11-04 12:32:28.558 INFO 21062 --- [ocp-schedules-15,ec3697225f074d46,134feaad908c1a21] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2024-11-04 12:32:28.558 INFO 21062 --- [ocp-schedules-15,ec3697225f074d46,134feaad908c1a21] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.enabled=true 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event-readable-cluster-auth=true 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric.only-cache=true 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=6 2024-11-04 12:32:28.558 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=60 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.enabled=true 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.enabled=false 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.password=null 2024-11-04 12:32:28.559 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.username=null 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.application.uuid=0083958465584847af344ffec220e32d 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.audit.enabled=true 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.enabled=false 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-location= 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-name= 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2024-11-04 12:32:28.560 INFO 21062 --- [ocp-schedules-6,a2928a019036914e,58cd9d63796a7d44] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.obs.support.enabled=false 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2024-11-04 12:32:28.560 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2024-11-04 12:32:28.561 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2024-11-04 12:32:28.561 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.build.version.check=false 2024-11-04 12:32:28.561 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ce.supported.ob.regex=^oceanbase-actiondb-(\d+\.){2}(\d+)(\.\d+)?-([\d.]+)\.([a-zA-Z0-9]+)\.(x86_64|aarch64)(\.rpm)?$ 2024-11-04 12:32:28.561 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.check.host.http.read.timeout=5000 2024-11-04 12:32:28.561 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2024-11-04 12:32:28.561 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2024-11-04 12:32:28.561 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"ob.cluster.parameter.template.built-in.default-2-2-77.name"}] 2024-11-04 12:32:28.561 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:32:28.561 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:32:28.561 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:32:28.562 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.site.url= 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.use-request-address=true 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.cache.refresh.interval=60 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.connect-timeout=2000 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.socket-timeout=30000 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.initialSize=10 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.keepAlive=true 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxActive=100 2024-11-04 12:32:28.562 INFO 21062 --- [ocp-async-6,f54992b1e61c1654,2fbc504064da6ca1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:32:28.562 INFO 21062 --- [ocp-async-7,4726b84b0f544f5b,cf946c283897ae98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:28.562 INFO 21062 --- [ocp-schedules-3,91188c50db51b2c3,8a31c4f03c60872c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxWait=2000 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.minIdle=2 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.name=config-server-connect-pool 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.testWhileIdle=true 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.validationQuery=SELECT 1 FROM DUAL 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2024-11-04 12:32:28.562 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.event.snapshot.maxage-days=90 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.has.signature.keywords.list=PGP,signatures 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.built-in.dir=/home/dba/obtest/ocp/ocp-server/lib 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.dir=/home/dba/obtest/software 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.enabled=true 2024-11-04 12:32:28.563 INFO 21062 --- [ocp-schedules-17,fd5c986dd4b52a9b,4415d936c543d7bc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.invalid.keywords.list=NOT OK 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2024-11-04 12:32:28.563 INFO 21062 --- [ocp-schedules-17,fd5c986dd4b52a9b,4415d936c543d7bc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.enable=true 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.clockdiff.timeout.millis=20000 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2024-11-04 12:32:28.563 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.account.system.active=false 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth=composite 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/sensitiveDataKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate|ObRootServiceUnregister|GetObTenantInfoByServiceNames|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,performance/ash/reports/.*/data 2024-11-04 12:32:28.564 INFO 21062 --- [ocp-schedules-2,51fcc35219fad28e,ef53e6586c0a4333] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObRootServiceUnregister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,/api/v2/monitor/metric/scan.* 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=1 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.max-attempts=60 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obproxy,ocp_admin,monitor 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.sso-login-url= 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.cleanup.enabled=false 2024-11-04 12:32:28.564 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.days=15 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.tmp.dir=/home/dba/obtest/ocp/logs/ocp 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.max-download-speed=1024 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.query.http.read.timeout=70000 2024-11-04 12:32:28.565 INFO 21062 --- [ocp-schedules-24,ebb23e1782bc7977,bbe7cb701e52588c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:32:28.565 INFO 21062 --- [ocp-schedules-24,ebb23e1782bc7977,bbe7cb701e52588c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.cluster.name= 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.disconnect.alarm.minutes=2 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.async-compute.enabled=true 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.collect.interval.second=15 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async-compute.query.minute.enable=false 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.query.min.time.buffer.second=0 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.write.queue.capacity=5000000 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.enabled=true 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.request.timeout={"second_connect_timeout":1000,"second_read_timeout":1000,"second_request_timeout":2000,"minute_connect_timeout":10000,"minute_read_timeout":30000,"minute_request_timeout":40000} 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.custom.collectItem.sql.function.whiteList=count,time_to_usec,timestampdiff,coalesce,sum,max,avg,min,round,unix_timestamp,abs 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ob_cluster_server_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_data_async":1,"metric_hour_data":1098,"metric_daily_data":1098} 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.table-switch-time=1730694377 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2024-11-04 12:32:28.565 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.deletable.metricNames=process_count|app=HOST|name=observer,process_count|app=HOST|name=obproxy,process_cpu_percent|app=HOST|name=observer,process_cpu_percent|app=HOST|name=obproxy,process_exists|app=HOST|name=observer,process_exists|app=HOST|name=obproxy,process_fd_count|app=HOST|name=observer,process_fd_count|app=HOST|name=obproxy,process_memory_rss_bytes|app=HOST|name=observer,process_memory_rss_bytes|app=HOST|name=obproxy,process_thread_count|app=HOST|name=observer,process_thread_count|app=HOST|name=obproxy,process_uptime_seconds|app=HOST|name=observer,process_uptime_seconds|app=HOST|name=obproxy,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_latch_get_total,ob_latch_miss_total,ob_latch_sleep_total,ob_latch_immediate_get_total,ob_latch_immediate_miss_total,ob_latch_spin_get_total,ob_latch_wait_seconds_total,ob_tenant_context_memory_hold_bytes,ob_tenant_context_memory_used_bytes,ob_tenant_context_memory_count 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=30 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=30 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=180 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.logproxy.exporters=/metrics/logproxy 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.archive.downgrade.enabled=true 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.interval.factor=2 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.service.type=LOCAL 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.exporters=/metrics/ob/extra,/metrics/obcloud/extra 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.write.queue.capacity=5000000 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.prometheus.service.discovery.enabled=true 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy,/metrics/node/oblb,/metrics/node/obdns,/metrics/logproxy 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.write.queue.capacity=5000000 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.server.exporter.auth.enabled=true 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2024-11-04 12:32:28.566 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.ConnectionTimeout=2000 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.maximumPoolSize=200 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.minimumIdle=10 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.poolName=monitor-connect-pool 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.host=10.46.128.24 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.password=****** 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.port=2883 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.public-key=null 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.series.scan.batch-size=2000 2024-11-04 12:32:28.568 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.username=root@ocp_monitor 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.multicloud.monitordb.password-secret-key=****** 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2024-11-04 12:32:28.569 INFO 21062 --- [ocp-schedules-26,96fc6a1135b4ce27,836519416cd2182d] c.o.o.m.service.SeriesIdCleanService : HostIpSize=0, clusterSize=0, obproxyClusterSize=0 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist= 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.database.name.pattern=^[a-z][a-z_0-9]{1,127}$ 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.enable=false 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.standby.replication.role.privilege.objects=DBA_OB_TENANTS,DBA_OB_LS_HISTORY,GV$OB_UNITS,GV$OB_PARAMETERS,GV$OB_LOG_STAT,DBA_OB_LS,DBA_OB_ACCESS_POINT 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.switchover.check.to.primary.wait-period-seconds=3 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.clean-tmp-rpm=true 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obcloud.api.address= 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obproxy.query-with-ob-connections=true 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.cgroup.minimum-ob-version=4.0.0.0 2024-11-04 12:32:28.569 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.default.os.user=admin 2024-11-04 12:32:28.569 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2024-11-04 12:32:28.570 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:32:28.570 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2024-11-04 12:32:28.570 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.host.arm-lse-supported=true 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.prohibit.obsharding.operation=true 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.skip-set-proxyro-password=****** 2024-11-04 12:32:28.570 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.backup.status.refresh.interval=300000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.performance.status.refresh.interval=300000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.count-sql.slow-sql-explicitly=true 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.oas.stat.enabled=true 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.password=****** 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.public.key=****** 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=0 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable.in.147x-meta-cluster=false 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2024-11-04 12:32:28.571 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":1000} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=False 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2},"alarm":true} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2}, "alarm":true, "sysTenantEnabled":false} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2024-11-04 12:32:28.572 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=True 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2024-11-04 12:32:28.573 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.rows.limit=1000000 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table.limit=32 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table4.limit=20 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=5d 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=7d 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2024-11-04 12:32:28.575 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=10m 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled=false 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled.in.147x-meta-cluster=true 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql.query-parallel=0 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-multiply-of-plan-comparison={"0":10, "1":5, "10":2, "100":1.5, "1000":1.2} 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2024-11-04 12:32:28.576 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.from-header=false 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.header-candidates=X-Forwarded-For 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.private-key=****** 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.public-key=****** 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.event.included={sql: {}, daily_merge:{}, election:{}, physical_restore:{}, observice:{}, observer:{}, clog_restore:{}, warmup:{}, log_archive:{}, archive_log:{}, minor_merge:{}, ddl:{"create_partition", "batch_create_partition", "add_replica", "remove_member", "batch_remove_member", "batch_modify_quorum", "batch_change_replica", "migrate_replica", "remove_replica", "rebuild_replica"}, freeze:{"do minor freeze fail"}, storage:{"add_replica failed", "migrate_replica failed", "reubild_replica failed", "change_replica failed", "log is new enough, turn off rebuild flag", "remove_replica failed", "restore follower replica failed", "restore_replica failed", "batch change_replica failed", "copy global index failed", "restore_standby_replica failed", "rebuild_replica failed"}} 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.readonly.mode.enabled=false 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.site.url=http://10.46.128.24:8090 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.standby.tenant.delay.threshold=5000 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2024-11-04 12:32:28.577 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.minimum-timeout-secs=-1 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.prohibit.skip.names=[] 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.language=zh-CN 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.timezone= 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.event.excluded={balancer: {"start_sql_bkgd_task","finish_sql_bkgd_task"}, schema_recycler: {}} 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2024-11-04 12:32:28.578 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tag.usage.max.count=5000 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.prohibit.skip.names=["Upgrade OB cluster"] 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.fixedDelay=10 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.fixedDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.fixedDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.fixedDelay=10 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.fixedDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.fixedDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.fixedDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.fixedDelay=20 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.initialDelay=60 2024-11-04 12:32:28.579 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.fixedDelay=180 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.initialDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.fixedDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.initialDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.fixedDelay=10 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.initialDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.fixedDelay=600 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.initialDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.fixedDelay=600 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.initialDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.fixedDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.initialDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogClusters.fixedDelay=30 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogInstances.fixedDelay=60 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.fixedRate=30 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.initialDelay=120 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.fixedRate=30 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.initialDelay=120 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.fixedRate=30 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.initialDelay=120 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.fixedRate=30 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.initialDelay=120 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.fixedDelay=4 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.initialDelay=30 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.timeUnit=MINUTES 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.fixedDelay=1 2024-11-04 12:32:28.580 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.initialDelay=2 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.timeUnit=MINUTES 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.fixedDelay=10 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.initialDelay=2 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.timeUnit=MINUTES 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkOdpClient.fixedDelay=60 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkVpcConnect.fixedDelay=5 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskLog.cron=0 10 3 * * * 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskRecords.cron=0 20 5 * * * 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTmpLogFiles.cron=0 20 4 * * * 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.fixedDelay=300 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.initialDelay=600 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.collectOdpMetrics.fixedDelay=60 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.ocpReplicationDaemon.fixedDelay=5 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.fixedDelay=600 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.initialDelay=120 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.refreshOcpConfig.fixedDelay=180 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkAllArbitrationService.fixedRate=30 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterRootServer.fixedRate=300 2024-11-04 12:32:28.581 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterStatus.fixedRate=30 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkStandbyClusterInfo.fixedRate=30 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.collectAllClusterCompaction.fixedDelay=60 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterInfo.fixedRate=30 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterSystemEvent.fixedDelay=600 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncRootServer.fixedDelay=120 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllStandbyTenant.fixedRate=30 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllTenantServiceNameConflict.fixedDelay=60 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMajorCompaction.fixedDelay=60 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMiniCompaction.fixedDelay=300 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.syncAllTenantInfo.fixedDelay=60 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.cron=10/30 * * * * * 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.initialDelay=120 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.cron=10/30 * * * * * 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.initialDelay=120 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.fixedDelay=300 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.initialDelay=120 2024-11-04 12:32:28.582 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectSlowSqlForAlarm.fixedDelay=60 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectTransStatForAlarm.fixedDelay=60 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.oasSqlCuring.fixedDelay=60 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.pool.coreSize=30 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.mysql=^[a-z][a-z_0-9]{1,127}$ 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version=4.3.2 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version.full=4.3.2-20241012145836 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.context-path=/ 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.tracking.enabled=1 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.syslog-level=INFO 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.work-thread-num=8 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2024-11-04 12:32:28.583 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.address=0.0.0.0 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.port=8090 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.name=SESSION 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.same-site=null 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.secure=null 2024-11-04 12:32:28.584 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.timeout=30m 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.enabled=false 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-alias=null 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-password=null 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store= 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-password=****** 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.max-page-size=5000 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2024-11-04 12:32:28.585 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxActive=200 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.ConnectionTimeout=2000 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.maximumPoolSize=200 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.minimumIdle=10 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.poolName=metadb-connect-pool 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.enabled=false 2024-11-04 12:32:28.586 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-exclude=null 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2024-11-04 12:32:28.587 INFO 21062 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ****************************************************************************** 2024-11-04 12:32:28.591 INFO 21062 --- [ocp-schedules-3,91188c50db51b2c3,8a31c4f03c60872c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:32:28.594 INFO 21062 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : Started OcpCeServerApplication in 160.996 seconds (JVM running for 162.274) 2024-11-04 12:32:28.604 INFO 21062 --- [ocp-schedules-24,ebb23e1782bc7977,bbe7cb701e52588c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:32:28.604 INFO 21062 --- [ocp-schedules-24,ebb23e1782bc7977,bbe7cb701e52588c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:32:28.604 INFO 21062 --- [ocp-schedules-26,96fc6a1135b4ce27,836519416cd2182d] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2024-11-04 12:32:28.604 INFO 21062 --- [ocp-schedules-26,96fc6a1135b4ce27,836519416cd2182d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2024-11-04 12:32:28.604 INFO 21062 --- [ocp-schedules-26,96fc6a1135b4ce27,836519416cd2182d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2024-11-04 12:32:28.620 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:32:28.625 INFO 21062 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : Setup sensitive data utils, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYo+dfEA+ti3lp8yywLmeYA2IPgY0aLvYoP9drFzPTopFyNoOCSB6O+ieFk01zZw4wQLLUUF8WZV+FgS62Eyk2sfwbrRfiMS6z6SpCHJYTsrB99sefX8HeUFDXENz1L0GfutE734bJc06TI5vqzy8aUTtxcCW/mBZbRT4bp5Df5wIDAQAB 2024-11-04 12:32:28.627 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2024-11-04 12:32:28.627 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_dynamic_sql 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat_v2 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_server_event 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_async 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2024-11-04 12:32:28.628 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2024-11-04 12:32:28.629 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2024-11-04 12:32:28.629 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:32:28.629 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:32:28.630 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:32:28.630 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:32:28.630 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:32:28.633 INFO 21062 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2024-11-04 12:32:28.633 ERROR 21062 --- [ocp-async-8,b5ce112ab001f3c6,41b621781fffc657] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [task not exist., 999989] 2024-11-04 12:32:28.633 INFO 21062 --- [ocp-async-8,b5ce112ab001f3c6,41b621781fffc657] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Task not found, task id :999989 2024-11-04 12:32:28.634 INFO 21062 --- [ocp-async-8,b5ce112ab001f3c6,41b621781fffc657] c.o.o.s.c.l.ApplicationStartedListener : Begin to init proxyro password config in sys tenant 2024-11-04 12:32:28.634 INFO 21062 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP application uuid is 0083958465584847af344ffec220e32d. 2024-11-04 12:32:28.634 INFO 21062 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP boot ok! 2024-11-04 12:32:28.636 INFO 21062 --- [ocp-async-8,b5ce112ab001f3c6,41b621781fffc657] c.o.o.s.c.l.ApplicationStartedListener : Success to init proxyro password config in sys tenant 2024-11-04 12:32:28.636 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT2M41.039845283S 2024-11-04 12:32:28.636 INFO 21062 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2024-11-04 12:32:28.638 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:32:28.639 INFO 21062 --- [ocp-async-3,768419b418f25174,e4cf15ddd7dfb6b6] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [] 2024-11-04 12:32:28.639 INFO 21062 --- [ocp-async-3,768419b418f25174,e4cf15ddd7dfb6b6] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [] 2024-11-04 12:32:28.641 INFO 21062 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2024-11-04 12:32:28.641 INFO 21062 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : SetProxyIdAfterUpgrade created. 2024-11-04 12:32:28.653 WARN 21062 --- [main,,] c.o.o.o.i.ObproxyOperationServiceImpl : No obproxy needs to set proxy_id, input param is empty. 2024-11-04 12:32:28.653 INFO 21062 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : Submit task null success, assign proxy_id for obproxies []. 2024-11-04 12:32:28.653 INFO 21062 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2024-11-04 12:32:28.989 INFO 21062 --- [ocp-async-9,dd193f8f67a5a423,a6cd036f310baa01] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2024-11-04 12:32:29.049 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:32:29.210 INFO 21062 --- [ocp-async-9,dd193f8f67a5a423,a6cd036f310baa01] c.o.o.s.common.config.TaskEngineRunner : Startup subtask coordinator. subtask=592 2024-11-04 12:32:29.210 INFO 21062 --- [ocp-async-9,dd193f8f67a5a423,a6cd036f310baa01] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2024-11-04 12:32:29.262 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:29.264 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:29.539 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.969434513S 2024-11-04 12:32:29.539 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:32:29.621 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:32:29.621 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:32:29.651 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:32:29.725 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:32:29.725 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:32:29.725 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:32:29.725 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:32:29.725 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:32:29.726 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:32:29.735 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:32:29.861 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] o.s.boot.SpringApplication : Started application in 1.3 seconds (JVM running for 163.541) 2024-11-04 12:32:29.862 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.300774493S 2024-11-04 12:32:29.983 INFO 21062 --- [prometheus-minute-schedule-1,de144dd6784935e7,19ba774c47a123b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:32:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:32:30.358 INFO 21062 --- [ocp-schedules-8,822949aefe5fd186,2f3a482eedba2502] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:32:30.525 WARN 21062 --- [ocp-async-1,9ee0e4cc5acae1dd,f0c1a1c457f37330] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:30.526 INFO 21062 --- [ocp-async-5,6dd9c6f9cb16ea74,2a3234466ae090d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:32.264 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:32.266 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:32.526 WARN 21062 --- [ocp-async-10,f56fbc2bd434c841,bdbeb387cb8ee0c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:32.526 INFO 21062 --- [ocp-async-2,80f43ab4590e3f28,77c8e6f668a3a141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:32.983 INFO 21062 --- [prometheus-minute-schedule-1,64182510e18d2030,7c10ea3762e8d8e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:33.023 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2024-11-04 12:32:33.023 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,,] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' 2024-11-04 12:32:33.030 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,,] o.s.web.servlet.DispatcherServlet : Completed initialization in 7 ms 2024-11-04 12:32:33.556 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0b7d9e1358e4e677,34061babc2cea6e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.46.128.24, traceId=0b7d9e1358e4e677, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 12:32:33.562 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0b7d9e1358e4e677,34061babc2cea6e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.46.128.24, traceId=0b7d9e1358e4e677, duration=478 ms] 2024-11-04 12:32:33.691 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,0edbfcba88eda7da,3a413389e0122580] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.46.128.24, traceId=0edbfcba88eda7da, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 12:32:33.691 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,0edbfcba88eda7da,3a413389e0122580] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.46.128.24, traceId=0edbfcba88eda7da, duration=15 ms] 2024-11-04 12:32:34.038 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,963c50b93a519b33,374e02c76644b040] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.46.128.24, traceId=963c50b93a519b33, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 12:32:34.038 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,963c50b93a519b33,374e02c76644b040] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.46.128.24, traceId=963c50b93a519b33, duration=19 ms] 2024-11-04 12:32:34.054 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e9f6fe946dd9d8b7,b5435d66a95a25af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.46.128.24, traceId=e9f6fe946dd9d8b7, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 12:32:34.054 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e9f6fe946dd9d8b7,b5435d66a95a25af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.46.128.24, traceId=e9f6fe946dd9d8b7, duration=10 ms] 2024-11-04 12:32:34.069 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6ed37e6c6e67b8bd,c75fa45919a82f96] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.46.128.24, traceId=6ed37e6c6e67b8bd, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 12:32:34.069 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6ed37e6c6e67b8bd,c75fa45919a82f96] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.46.128.24, traceId=6ed37e6c6e67b8bd, duration=10 ms] 2024-11-04 12:32:34.086 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,17fb0915110a4360,7342dfd7b6ec4db5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.46.128.24, traceId=17fb0915110a4360, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 12:32:34.086 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,17fb0915110a4360,7342dfd7b6ec4db5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.46.128.24, traceId=17fb0915110a4360, duration=10 ms] 2024-11-04 12:32:34.202 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOverPreCheck?null, client=10.46.128.24, traceId=4a4111d71e2ba409, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(TakeOverClusterPreCheckParam), args=TakeOverClusterPreCheckParam(connectionMode=direct, clusterName=null, vpcName=null, vpcId=1, obClusterId=null, rootSyspassword=xxx proxyropassword=xxx saveToCredential=true, address=10.46.128.24, port=2881),] 2024-11-04 12:32:34.288 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=10.46.128.24, port=2881, database=oceanbase) 2024-11-04 12:32:34.288 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2024-11-04 12:32:34.353 INFO 21062 --- [obsdk-init-thread-1,4a4111d71e2ba409,1e67a43939e2e9bf] com.alibaba.druid.pool.DruidDataSource : {dataSource-3,ob-connector-datasource_direct_sys_root_10.46.128.24_2881} inited 2024-11-04 12:32:34.353 INFO 21062 --- [obsdk-init-thread-1,4a4111d71e2ba409,1e67a43939e2e9bf] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2024-11-04 12:32:34.353 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2024-11-04 12:32:34.365 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.367 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.380 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.381 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.386 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.387 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.390 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.391 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.400 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.400 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-11-04 12:32:34.419 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.420 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.423 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.423 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2024-11-04 12:32:34.435 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.435 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.438 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.439 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-11-04 12:32:34.450 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.450 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.453 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.454 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.462 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.463 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-11-04 12:32:34.472 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.472 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.475 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.476 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.478 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.479 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2024-11-04 12:32:34.490 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.491 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.501 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.501 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-11-04 12:32:34.526 WARN 21062 --- [ocp-async-7,4ef37071a694d43c,d2d2245c61e8d388] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:34.526 INFO 21062 --- [ocp-async-6,5cb4263ce04fb769,a3f319eecc40d569] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:34.556 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.556 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.566 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.568 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-11-04 12:32:34.613 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.614 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.625 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=ocp_meta, username=root, address=10.46.128.24, port=2883, database=meta_database) 2024-11-04 12:32:34.625 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2883, obsAddrList=null, username=root, tenantName=ocp_meta, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2024-11-04 12:32:34.631 INFO 21062 --- [obsdk-init-thread-2,4a4111d71e2ba409,0ae9dd2ad2af8704] com.alibaba.druid.pool.DruidDataSource : {dataSource-4,ob-connector-datasource_direct_ocp_meta_root_10.46.128.24_2883} inited 2024-11-04 12:32:34.631 INFO 21062 --- [obsdk-init-thread-2,4a4111d71e2ba409,0ae9dd2ad2af8704] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2883, obsAddrList=null, username=root, tenantName=ocp_meta, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2024-11-04 12:32:34.632 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2024-11-04 12:32:34.635 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.636 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:34.640 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.641 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:34.642 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.643 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-11-04 12:32:34.656 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.657 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.660 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.660 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.662 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.663 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:34.666 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:34.666 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-11-04 12:32:34.756 ERROR 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 10.46.128.24] 2024-11-04 12:32:34.788 ERROR 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 10.46.128.11] 2024-11-04 12:32:34.801 ERROR 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 10.46.128.12] 2024-11-04 12:32:34.809 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4a4111d71e2ba409,4a8e0281832717a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOverPreCheck client=10.46.128.24, traceId=4a4111d71e2ba409, duration=711 ms] 2024-11-04 12:32:35.266 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:35.267 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:35.983 INFO 21062 --- [prometheus-minute-schedule-1,f61b7b329f8ec893,21a1f9b20adff4ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:36.526 WARN 21062 --- [ocp-async-3,2aa5745063706fc1,bd0445395c182291] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:36.526 INFO 21062 --- [ocp-async-1,e5a8565ad62cf316,c38088f082c14218] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:37.749 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9c87857a5b20b9f3,47af0fbb4d2eeecd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.46.128.24, traceId=9c87857a5b20b9f3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-11-04 12:32:37.813 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9c87857a5b20b9f3,47af0fbb4d2eeecd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.46.128.24, traceId=9c87857a5b20b9f3, duration=99 ms] 2024-11-04 12:32:37.844 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1fff474f0e6ba9de,b0fe44086d3f967e] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hostTypes?null, client=10.46.128.24, traceId=1fff474f0e6ba9de, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.createHostType(CreateHostTypeRequest), args=CreateHostTypeRequest(name=, description=null),] 2024-11-04 12:32:37.863 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1fff474f0e6ba9de,b0fe44086d3f967e] c.o.o.c.h.service.HostTypeServiceImpl : Created hostType: 1 2024-11-04 12:32:37.864 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1fff474f0e6ba9de,b0fe44086d3f967e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hostTypes client=10.46.128.24, traceId=1fff474f0e6ba9de, duration=39 ms] 2024-11-04 12:32:37.897 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,3429fa9ffab8d936,ae025e20a88c36e1] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials?null, client=10.46.128.24, traceId=3429fa9ffab8d936, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.createCredential(CreateCredentialParam), args=CreateCredentialParam(targetType=HOST, validate=true, name=d742038c12cb, description=null, obJdbcCredentialProperty=null, sshCredentialProperty=SshCredentialProperty(type=PRIVATE_KEY, allowSudo=true, username=dba, passphrase=xxx RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAueiLC8Fp0L67urrdZq8mVKWoADu2asxQkuYCSRt/TQnlyhHu XqrLXq6+4GpMMecUV+YZw3Bz1flJgxt1+RVZZKD+uu66sHGfsBTtGpKds2oHAn9x c/zYsIExZB9R5LSjluay/p5uzwzO4O6p/IpShN16TMiBPXgBsddU4k1nQLRHQDXF eszzZgtCb0sCDQ2xV8UPPn9+zso/I/9QueIlOpRjhBgDzXL6eVjlAg5tF52jmzZl HY3Dk9Z8Jc8hdVXizfX0jqsGoXBgvrHupUdZO6r5p+gj6IOqjkk7EI9Xar4hA2r2 dgFo6LRXPdEaRMpQziL98WXnh5LYFe/UUktCfwIDAQABAoIBABwXfoNNO58WYymp OPFeQA2EZql6XGsw6m/4H5pKdGcAut9Y7EsTxd1NzHv+s/rKbzGFDJmSsX5AyBHu oVn9Ft/0z+kRT9Z/pOCnmNhuFTRaLir7dPpQ17F8JCl77hUAEsQnoh6FdDsHNW+Z wMfvYt6+1M8rcRQHwUHNRv9XMOX+1c9uYTDpeDAMN7zpURNBO8H8jq15TRpHL9rb Oopnd755cYp7KJG7rcKgkTMrVVJa3wOW7J0I9myCcMqz040guJwYYI+RuN0AJHjY Ubv4p6948sb+3WAqCPNhw8e+PpWbTmJjvz6mG+tF2PpziHGF0uSbbmV/Ybu3CvG9 RcFRrCECgYEA5PlGq11WPrmyaa2ASv0bs4IhV3uDxkz8li3H9jQlFL4waAp3MqsI ST6340Sn+hUfk2HxE7DJLB7HKPngS2uUFHvcafv2vRDsJJGAmr3IefjiDxK5wjCq j8zA5bYFhY2oMj9tL3VnSfRle0MWHCyjP6BJ3AfT9/NE97YXfa6oTw8CgYEAz9n+ d5V4vtG3feoDo4e6zMOdB/nJlMrMY89Vo/KzAS8KTkBMiswe2/pu3F3Q8RDpg/IU U11fMXAgOTa4+3893tgVpZAIyEQnPNlWlu2CPdhxvbuupL+XuZkb/RymqScMKUyv y4q2pT02lDXCpahQuk82jpc9u+ayQ4SFjTHh1ZECgYBYRWUiERR7j/kg1ls8DYGN XWBfKmWtn+ZMOw/4cTfOXFeq9EXpiZfjLMOVYNVlCpmmpr5aee/EC4V53YLDcJCg qPNbBGClmI2At3sdrIZJx1HWv8BPBZXrY+MFa3uLdgXegSclgZw1HXu13jH64JZ6 o991bKshzm3LFS7PK/MErwKBgQCVD6lG6ZMmhhAp5OGnLXfx0XeUeLbwVmAuVEr+ wNdPSgUCCWrbP3gCGp/WuhFWUOKnFb0t7BCQy/DQF0aiVqZwYldj3N/vf7EEGnnW 045L/CXWA/tDe4O8PumrpBiBxtcZ5z1Lo7dI2ZWCxOtLEmKSuqxNpnlHgDLn6Oo+ UD+9gQKBgQDbUqbbFtqqkpwGxd6qZx1kyTF67+7Q4MqlMYiPRCVMwjnYq1ljeTsK P4WOnjL08djvHxb0aEZuY/Q52WZwrrCTLRvZHFycUCin9Qs9bsgZoQlybOoi66mi VC9bqk99h 2024-11-04 12:32:37.932 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,3429fa9ffab8d936,ae025e20a88c36e1] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=HOST, validate=true, name=d742038c12cb, description=null, obJdbcCredentialProperty=null, sshCredentialProperty=SshCredentialProperty(type=PRIVATE_KEY, allowSudo=true, username=dba, passphrase=-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAueiLC8Fp0L67urrdZq8mVKWoADu2asxQkuYCSRt/TQnlyhHu XqrLXq6+4GpMMecUV+YZw3Bz1flJgxt1+RVZZKD+uu66sHGfsBTtGpKds2oHAn9x c/zYsIExZB9R5LSjluay/p5uzwzO4O6p/IpShN16TMiBPXgBsddU4k1nQLRHQDXF eszzZgtCb0sCDQ2xV8UPPn9+zso/I/9QueIlOpRjhBgDzXL6eVjlAg5tF52jmzZl HY3Dk9Z8Jc8hdVXizfX0jqsGoXBgvrHupUdZO6r5p+gj6IOqjkk7EI9Xar4hA2r2 dgFo6LRXPdEaRMpQziL98WXnh5LYFe/UUktCfwIDAQABAoIBABwXfoNNO58WYymp OPFeQA2EZql6XGsw6m/4H5pKdGcAut9Y7EsTxd1NzHv+s/rKbzGFDJmSsX5AyBHu oVn9Ft/0z+kRT9Z/pOCnmNhuFTRaLir7dPpQ17F8JCl77hUAEsQnoh6FdDsHNW+Z wMfvYt6+1M8rcRQHwUHNRv9XMOX+1c9uYTDpeDAMN7zpURNBO8H8jq15TRpHL9rb Oopnd755cYp7KJG7rcKgkTMrVVJa3wOW7J0I9myCcMqz040guJwYYI+RuN0AJHjY Ubv4p6948sb+3WAqCPNhw8e+PpWbTmJjvz6mG+tF2PpziHGF0uSbbmV/Ybu3CvG9 RcFRrCECgYEA5PlGq11WPrmyaa2ASv0bs4IhV3uDxkz8li3H9jQlFL4waAp3MqsI ST6340Sn+hUfk2HxE7DJLB7HKPngS2uUFHvcafv2vRDsJJGAmr3IefjiDxK5wjCq j8zA5bYFhY2oMj9tL3VnSfRle0MWHCyjP6BJ3AfT9/NE97YXfa6oTw8CgYEAz9n+ d5V4vtG3feoDo4e6zMOdB/nJlMrMY89Vo/KzAS8KTkBMiswe2/pu3F3Q8RDpg/IU U11fMXAgOTa4+3893tgVpZAIyEQnPNlWlu2CPdhxvbuupL+XuZkb/RymqScMKUyv y4q2pT02lDXCpahQuk82jpc9u+ayQ4SFjTHh1ZECgYBYRWUiERR7j/kg1ls8DYGN XWBfKmWtn+ZMOw/4cTfOXFeq9EXpiZfjLMOVYNVlCpmmpr5aee/EC4V53YLDcJCg qPNbBGClmI2At3sdrIZJx1HWv8BPBZXrY+MFa3uLdgXegSclgZw1HXu13jH64JZ6 o991bKshzm3LFS7PK/MErwKBgQCVD6lG6ZMmhhAp5OGnLXfx0XeUeLbwVmAuVEr+ wNdPSgUCCWrbP3gCGp/WuhFWUOKnFb0t7BCQy/DQF0aiVqZwYldj3N/vf7EEGnnW 045L/CXWA/tDe4O8PumrpBiBxtcZ5z1Lo7dI2ZWCxOtLEmKSuqxNpnlHgDLn6Oo+ UD+9gQKBgQDbUqbbFtqqkpwGxd6qZx1kyTF67+7Q4MqlMYiPRCVMwjnYq1ljeTsK P4WOnjL08djvHxb0aEZuY/Q52WZwrrCTLRvZHFycUCin9Qs9bsgZoQlybOoi66mi VC9bqk99hu+c8099gLmU645NBKYn4a9++/zETEaff+p4UOvigyIEHg== -----END RSA PRIVATE KEY----- , hostIdList=null, localPrivateKeyPath=null), agentCredentialProperty=null) 2024-11-04 12:32:37.933 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,3429fa9ffab8d936,ae025e20a88c36e1] c.o.o.s.c.CredentialServiceImpl : Validating credential:ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=SshCredentialProperty(type=PRIVATE_KEY, allowSudo=true, username=dba, passphrase=-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAueiLC8Fp0L67urrdZq8mVKWoADu2asxQkuYCSRt/TQnlyhHu XqrLXq6+4GpMMecUV+YZw3Bz1flJgxt1+RVZZKD+uu66sHGfsBTtGpKds2oHAn9x c/zYsIExZB9R5LSjluay/p5uzwzO4O6p/IpShN16TMiBPXgBsddU4k1nQLRHQDXF eszzZgtCb0sCDQ2xV8UPPn9+zso/I/9QueIlOpRjhBgDzXL6eVjlAg5tF52jmzZl HY3Dk9Z8Jc8hdVXizfX0jqsGoXBgvrHupUdZO6r5p+gj6IOqjkk7EI9Xar4hA2r2 dgFo6LRXPdEaRMpQziL98WXnh5LYFe/UUktCfwIDAQABAoIBABwXfoNNO58WYymp OPFeQA2EZql6XGsw6m/4H5pKdGcAut9Y7EsTxd1NzHv+s/rKbzGFDJmSsX5AyBHu oVn9Ft/0z+kRT9Z/pOCnmNhuFTRaLir7dPpQ17F8JCl77hUAEsQnoh6FdDsHNW+Z wMfvYt6+1M8rcRQHwUHNRv9XMOX+1c9uYTDpeDAMN7zpURNBO8H8jq15TRpHL9rb Oopnd755cYp7KJG7rcKgkTMrVVJa3wOW7J0I9myCcMqz040guJwYYI+RuN0AJHjY Ubv4p6948sb+3WAqCPNhw8e+PpWbTmJjvz6mG+tF2PpziHGF0uSbbmV/Ybu3CvG9 RcFRrCECgYEA5PlGq11WPrmyaa2ASv0bs4IhV3uDxkz8li3H9jQlFL4waAp3MqsI ST6340Sn+hUfk2HxE7DJLB7HKPngS2uUFHvcafv2vRDsJJGAmr3IefjiDxK5wjCq j8zA5bYFhY2oMj9tL3VnSfRle0MWHCyjP6BJ3AfT9/NE97YXfa6oTw8CgYEAz9n+ d5V4vtG3feoDo4e6zMOdB/nJlMrMY89Vo/KzAS8KTkBMiswe2/pu3F3Q8RDpg/IU U11fMXAgOTa4+3893tgVpZAIyEQnPNlWlu2CPdhxvbuupL+XuZkb/RymqScMKUyv y4q2pT02lDXCpahQuk82jpc9u+ayQ4SFjTHh1ZECgYBYRWUiERR7j/kg1ls8DYGN XWBfKmWtn+ZMOw/4cTfOXFeq9EXpiZfjLMOVYNVlCpmmpr5aee/EC4V53YLDcJCg qPNbBGClmI2At3sdrIZJx1HWv8BPBZXrY+MFa3uLdgXegSclgZw1HXu13jH64JZ6 o991bKshzm3LFS7PK/MErwKBgQCVD6lG6ZMmhhAp5OGnLXfx0XeUeLbwVmAuVEr+ wNdPSgUCCWrbP3gCGp/WuhFWUOKnFb0t7BCQy/DQF0aiVqZwYldj3N/vf7EEGnnW 045L/CXWA/tDe4O8PumrpBiBxtcZ5z1Lo7dI2ZWCxOtLEmKSuqxNpnlHgDLn6Oo+ UD+9gQKBgQDbUqbbFtqqkpwGxd6qZx1kyTF67+7Q4MqlMYiPRCVMwjnYq1ljeTsK P4WOnjL08djvHxb0aEZuY/Q52WZwrrCTLRvZHFycUCin9Qs9bsgZoQlybOoi66mi VC9bqk99hu+c8099gLmU645NBKYn4a9++/zETEaff+p4UOvigyIEHg== -----END RSA PRIVATE KEY----- , hostIdList=null, localPrivateKeyPath=null), agentCredentialProperty=null, batchSshValidateProperty=null) 2024-11-04 12:32:38.268 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:38.270 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:38.421 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,3429fa9ffab8d936,ae025e20a88c36e1] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=HOST, validate=true, name=d742038c12cb, description=null, obJdbcCredentialProperty=null, sshCredentialProperty=SshCredentialProperty(type=PRIVATE_KEY, allowSudo=true, username=dba, passphrase=-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAueiLC8Fp0L67urrdZq8mVKWoADu2asxQkuYCSRt/TQnlyhHu XqrLXq6+4GpMMecUV+YZw3Bz1flJgxt1+RVZZKD+uu66sHGfsBTtGpKds2oHAn9x c/zYsIExZB9R5LSjluay/p5uzwzO4O6p/IpShN16TMiBPXgBsddU4k1nQLRHQDXF eszzZgtCb0sCDQ2xV8UPPn9+zso/I/9QueIlOpRjhBgDzXL6eVjlAg5tF52jmzZl HY3Dk9Z8Jc8hdVXizfX0jqsGoXBgvrHupUdZO6r5p+gj6IOqjkk7EI9Xar4hA2r2 dgFo6LRXPdEaRMpQziL98WXnh5LYFe/UUktCfwIDAQABAoIBABwXfoNNO58WYymp OPFeQA2EZql6XGsw6m/4H5pKdGcAut9Y7EsTxd1NzHv+s/rKbzGFDJmSsX5AyBHu oVn9Ft/0z+kRT9Z/pOCnmNhuFTRaLir7dPpQ17F8JCl77hUAEsQnoh6FdDsHNW+Z wMfvYt6+1M8rcRQHwUHNRv9XMOX+1c9uYTDpeDAMN7zpURNBO8H8jq15TRpHL9rb Oopnd755cYp7KJG7rcKgkTMrVVJa3wOW7J0I9myCcMqz040guJwYYI+RuN0AJHjY Ubv4p6948sb+3WAqCPNhw8e+PpWbTmJjvz6mG+tF2PpziHGF0uSbbmV/Ybu3CvG9 RcFRrCECgYEA5PlGq11WPrmyaa2ASv0bs4IhV3uDxkz8li3H9jQlFL4waAp3MqsI ST6340Sn+hUfk2HxE7DJLB7HKPngS2uUFHvcafv2vRDsJJGAmr3IefjiDxK5wjCq j8zA5bYFhY2oMj9tL3VnSfRle0MWHCyjP6BJ3AfT9/NE97YXfa6oTw8CgYEAz9n+ d5V4vtG3feoDo4e6zMOdB/nJlMrMY89Vo/KzAS8KTkBMiswe2/pu3F3Q8RDpg/IU U11fMXAgOTa4+3893tgVpZAIyEQnPNlWlu2CPdhxvbuupL+XuZkb/RymqScMKUyv y4q2pT02lDXCpahQuk82jpc9u+ayQ4SFjTHh1ZECgYBYRWUiERR7j/kg1ls8DYGN XWBfKmWtn+ZMOw/4cTfOXFeq9EXpiZfjLMOVYNVlCpmmpr5aee/EC4V53YLDcJCg qPNbBGClmI2At3sdrIZJx1HWv8BPBZXrY+MFa3uLdgXegSclgZw1HXu13jH64JZ6 o991bKshzm3LFS7PK/MErwKBgQCVD6lG6ZMmhhAp5OGnLXfx0XeUeLbwVmAuVEr+ wNdPSgUCCWrbP3gCGp/WuhFWUOKnFb0t7BCQy/DQF0aiVqZwYldj3N/vf7EEGnnW 045L/CXWA/tDe4O8PumrpBiBxtcZ5z1Lo7dI2ZWCxOtLEmKSuqxNpnlHgDLn6Oo+ UD+9gQKBgQDbUqbbFtqqkpwGxd6qZx1kyTF67+7Q4MqlMYiPRCVMwjnYq1ljeTsK P4WOnjL08djvHxb0aEZuY/Q52WZwrrCTLRvZHFycUCin9Qs9bsgZoQlybOoi66mi VC9bqk99hu+c8099gLmU645NBKYn4a9++/zETEaff+p4UOvigyIEHg== -----END RSA PRIVATE KEY----- , hostIdList=null, localPrivateKeyPath=null), agentCredentialProperty=null) 2024-11-04 12:32:38.429 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,3429fa9ffab8d936,ae025e20a88c36e1] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=CREATE_CREDENTIAL, eventType=CREDENTIAL_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.CredentialController.createCredential, resourceType=CREDENTIAL, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=d742038c12cb, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=null, eventInfo={"param":{"name":"d742038c12cb","sshCredentialProperty":{"allowSudo":true,"passphrase":"******","type":"PRIVATE_KEY","username":"dba"},"targetType":"HOST","validate":true}}, executionType=NEW, requestUrl=http://10.46.128.24:8090/api/v2/profiles/me/credentials, clientIp=10.46.128.24, serverIp=10.46.128.24, startTime=2024-11-04T12:32:37.915+08:00, endTime=2024-11-04T12:32:38.427+08:00, role=null, aliUid=null) 2024-11-04 12:32:38.515 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,3429fa9ffab8d936,ae025e20a88c36e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials client=10.46.128.24, traceId=3429fa9ffab8d936, duration=641 ms] 2024-11-04 12:32:38.517 INFO 21062 --- [ocp-schedules-23,d62e844c359289e9,fc89d9baf4a2f6e2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:32:38.525 INFO 21062 --- [ocp-async-4,4b785c1cbe541240,fda338c108e3f9b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:32:38.527 WARN 21062 --- [ocp-async-2,dca7b46cbd2c23d3,6b5de1a77030d7b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:38.527 INFO 21062 --- [ocp-async-7,c45959c9b0ab3cc2,1a1f1c85a84e0fea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:38.549 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOver?null, client=10.46.128.24, traceId=e46f513af4706458, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster(TakeOverClusterParam), args=TakeOverClusterParam(switchConfigUrl=true, hostInfo=TakeOverClusterParam.HostInfo(kind=DEDICATED_PHYSICAL_MACHINE, publishPorts=null, hostTypeId=1, sshPort=1046, credentialId=1)),] 2024-11-04 12:32:38.567 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=ob-cluster-10.46.128.24#2881, method=takeOverCluster 2024-11-04 12:32:38.580 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.587 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.590 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.590 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.596 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.597 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.599 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.600 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-11-04 12:32:38.611 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.611 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.614 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.615 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2024-11-04 12:32:38.625 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.626 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.628 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.629 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-11-04 12:32:38.640 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.641 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.643 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.644 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.646 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.647 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-11-04 12:32:38.652 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.653 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.655 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.656 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.658 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.659 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2024-11-04 12:32:38.668 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.669 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.672 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.672 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-11-04 12:32:38.675 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.676 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.678 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.683 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-11-04 12:32:38.693 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.694 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.697 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.698 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:38.699 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.699 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:38.700 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.701 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-11-04 12:32:38.708 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.709 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.711 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.712 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:38.716 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:38.717 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-11-04 12:32:38.735 ERROR 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 10.46.128.24] 2024-11-04 12:32:38.744 ERROR 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 10.46.128.11] 2024-11-04 12:32:38.753 ERROR 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 10.46.128.12] 2024-11-04 12:32:38.783 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.c.host.service.RegionServiceImpl : Created region: 2 sys_region 2024-11-04 12:32:38.802 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.c.host.service.IdcServiceImpl : Created idc: 1 default_idc 2024-11-04 12:32:38.959 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.o.i.c.ClusterGroupServiceImpl : create cluster group, name=dba_test_ob, description=dba_test_ob, create by cluster service, affectedRows=1 2024-11-04 12:32:38.983 INFO 21062 --- [prometheus-minute-schedule-1,773cae72b120983d,324fe2e33191375a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:39.008 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2024-11-04 12:30:00.0, currentDate=Mon Nov 04 12:32:39 CST 2024 2024-11-04 12:32:39.008 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=null, nextRunTime=2024-11-04 12:30:00.0} 2024-11-04 12:32:39.014 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 12:32:39.023 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=dba_test_ob, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2024-11-04 12:32:39.081 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 12:32:39.082 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 12:32:39.089 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2024-11-04 12:30:04.0, currentDate=Mon Nov 04 12:32:39 CST 2024 2024-11-04 12:32:39.090 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=null, nextRunTime=2024-11-04 12:30:04.0} 2024-11-04 12:32:39.099 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 12:32:39.112 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=dba_test_ob, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2024-11-04 12:32:39.113 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=dba_test_ob, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2024-11-04 12:32:39.115 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.CredentialServiceImpl : Credential clusterName=dba_test_ob|tenantName=sys|username=root is already exists 2024-11-04 12:32:39.115 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.CredentialServiceImpl : Delete credential:clusterName=dba_test_ob|tenantName=sys|username=root 2024-11-04 12:32:39.101 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T12:32:39.099+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 12:32:39.125 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=dba_test_ob, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2024-11-04 12:32:39.125 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:39.126 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:39.127 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:39.128 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:39.128 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:39.129 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-11-04 12:32:39.138 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:39.139 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:39.142 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:39.143 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:39.145 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:39.146 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-11-04 12:32:39.148 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:39.148 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-11-04 12:32:39.160 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.o.i.c.ClusterOperationServiceImpl : meta cluster detected. 2024-11-04 12:32:39.270 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE APP GROUP, property:UpdateAppGroupProperty(serialNumbers=[null, null, null], appGroup=null) 2024-11-04 12:32:39.343 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by a1e196d99e, eventId=1, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["dba_test_ob"]} 2024-11-04 12:32:39.347 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Start cache subtask classes 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalStandby 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemotePrimarySenderTask 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemoteStandbySenderTask 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalStandby 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemotePrimarySenderTask 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePreScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.348 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemoteStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$RemoteStandby 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemoteStandbySenderTask 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemotePrimarySenderTask 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.349 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local 2024-11-04 12:32:39.350 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalStandby 2024-11-04 12:32:39.350 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalPrimary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$LocalPrimary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalPrimary 2024-11-04 12:32:39.350 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemotePrimarySenderTask 2024-11-04 12:32:39.350 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.350 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemoteStandbySenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemoteStandbySenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalPrimary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalPrimary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalPrimary 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalStandby 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Primary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$Primary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$Primary 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeClusterHealthCheckerScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.351 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask 2024-11-04 12:32:39.352 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=UpdateObClusterStatusTask, class=com.oceanbase.ocp.obops.internal.cluster.task.UpdateObClusterStatusTask, existClass=class com.oceanbase.ocp.service.task.business.host.UpdateObClusterStatusTask 2024-11-04 12:32:39.353 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Remove duplicated simple name cache, size=46 2024-11-04 12:32:39.353 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b04f0562bc713469] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask cache initialized, simpleNameSize=536, classNameSize=592 2024-11-04 12:32:39.355 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Take over primary OB cluster, nodeCount=41, taskCount=67 2024-11-04 12:32:39.355 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=null, startTime=2024-11-04T12:32:39.355+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Uninstall legacy ocp agent, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UninstallLegacyOcpAgentTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update observer status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerStatusTask, seriesId=22, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update zone status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.zone.UpdateObZoneStatusTask, seriesId=46, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=65, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=3, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Active agent modules, state=PENDING, operation=EX 2024-11-04 12:32:39.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=65} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.648 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=61} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null} 2024-11-04 12:32:39.652 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=50} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.655 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=54} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.673 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=45} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=2, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.682 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=64} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=68} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=2} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.690 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=13} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.693 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=10} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.696 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=55} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.698 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=42} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.700 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=12} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.703 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=9} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.705 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=21} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.708 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=25} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.711 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=41} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.720 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone3', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null} 2024-11-04 12:32:39.724 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=49} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.730 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=31} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null} 2024-11-04 12:32:39.733 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=23} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.735 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=15} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=2, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.738 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=63} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.741 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=62} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.743 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=26} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.746 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=51} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.749 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=11} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.752 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=66} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone3', tenantName='null', tenantId=null, obTenantId=null, serverId=3, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.755 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=28} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=53} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.760 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=56} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone3', tenantName='null', tenantId=null, obTenantId=null, serverId=3, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=37} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone3', tenantName='null', tenantId=null, obTenantId=null, serverId=3, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.764 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=43} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.767 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=52} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.769 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=17} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.772 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=35} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.774 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=7} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='10.46.128.11', deleted=false, createTime=null} 2024-11-04 12:32:39.776 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=3} BaseTagEntity{id=null, clusterName='dba_test_ob', clusterId=1, obClusterId=1730694047, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=2, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='10.46.128.24', deleted=false, createTime=null} 2024-11-04 12:32:39.779 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=44} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=3, svrIp='10.46.128.12', deleted=false, createTime=null} 2024-11-04 12:32:39.783 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=2, taskName=Take over primary OB cluster, argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}} 2024-11-04 12:32:39.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=NEW, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.46.128.24, serverIp=10.46.128.24, startTime=2024-11-04T12:32:38.556+08:00, endTime=null, role=null, aliUid=null) 2024-11-04 12:32:39.818 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e46f513af4706458,fe3b3f54c692c992] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOver client=10.46.128.24, traceId=e46f513af4706458, duration=1292 ms] 2024-11-04 12:32:40.004 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:32:40.004 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:32:40.263 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,99c0effb67995662] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:40.279 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,99c0effb67995662] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 29, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:40.335 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1, context=Context{parallelIdx=-1, stringMap={task_instance_id=1, task_operation=execute, latest_execution_start_time=2024-11-04T12:32:40.248+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=1}, listMap={}}, executor=10.46.128.24 2024-11-04 12:32:40.335 INFO 21062 --- [manual-subtask-executor1,e46f513af4706458,5babf156a11fd8f5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=29, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, service_name=dba_test_ob:1730694047, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, latest_execution_start_time=2024-11-04T12:32:40.275+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=29}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:40.343 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 12:32:40.344 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 12:32:40.344 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=ocp_monitor, username=root, address=10.46.128.24, port=2883, database=monitor_database) 2024-11-04 12:32:40.344 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2883, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2024-11-04 12:32:40.348 INFO 21062 --- [manual-subtask-executor1,e46f513af4706458,5babf156a11fd8f5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 29, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:40.353 INFO 21062 --- [obsdk-init-thread-3,dd193f8f67a5a423,6efccf87f8f6aead] com.alibaba.druid.pool.DruidDataSource : {dataSource-5,ob-connector-datasource_direct_ocp_monitor_root_10.46.128.24_2883} inited 2024-11-04 12:32:40.353 INFO 21062 --- [obsdk-init-thread-3,dd193f8f67a5a423,6efccf87f8f6aead] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=10.46.128.24, port=2883, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2024-11-04 12:32:40.353 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2024-11-04 12:32:40.354 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:40.355 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:40.372 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:40.372 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:32:40.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:40.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 12:32:40.415 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241102 2024-11-04 12:32:40.417 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:40.418 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:32:40.527 WARN 21062 --- [ocp-async-6,b684f341b84f1bf5,e4c40a81dea3d5eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:40.527 INFO 21062 --- [ocp-async-3,2102f1450b21a7e6,f4a8b1a6c70548e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:40.686 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:40.687 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:40.726 INFO 21062 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [dba_test_ob] cache 2024-11-04 12:32:40.727 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster dba_test_ob cache info from db 2024-11-04 12:32:40.731 INFO 21062 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-11-04 12:32:40.742 INFO 21062 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by a1e196d99e, eventId=1, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["dba_test_ob"]} 2024-11-04 12:32:40.783 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster dba_test_ob cache info from db success. 2024-11-04 12:32:40.842 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:40.844 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241103 2024-11-04 12:32:40.845 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:40.846 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:32:41.131 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.133 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:41.164 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:41.165 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241104 2024-11-04 12:32:41.166 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:32:41.270 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:41.271 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:41.431 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[65, 53, 26] 2024-11-04 12:32:41.436 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.438 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:41.472 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:41.473 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241105 2024-11-04 12:32:41.474 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.475 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:32:41.578 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1 2024-11-04 12:32:41.739 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.741 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:41.774 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:41.775 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241106 2024-11-04 12:32:41.775 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.776 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:32:41.797 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.805 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:41.808 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:41.808 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:32:41.809 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:32:41.919 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:32:41.919 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=110ms, period=120000ms 2024-11-04 12:32:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c6d6bd13c871fd58,ecaa0b50803e142a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:42.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:42.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:42.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:42.091 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241107 2024-11-04 12:32:42.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:42.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:32:42.301 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,6fe9a9b7341489e0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:42.311 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,6fe9a9b7341489e0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 53, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:42.321 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,6fe9a9b7341489e0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 65, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:42.324 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=26, context=Context{parallelIdx=1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:42.297+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=26}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:42.330 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=53, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:42.308+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=53}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:42.350 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=65, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:42.319+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=65}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:42.371 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:42.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:42.402 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:42.403 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241108 2024-11-04 12:32:42.404 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:42.404 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:32:42.510 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.e.internal.connector.Connectors : init global connector holder. 2024-11-04 12:32:42.514 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.e.i.connector.ConnectorHolder : Init connector holder, maxIdleSecs=3600 2024-11-04 12:32:42.527 WARN 21062 --- [ocp-async-5,4899340251ba04d3,3a67564ba23ec03e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:42.527 INFO 21062 --- [ocp-async-2,f4f9cf99851f961c,e838ddf3a1c08ae7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:42.781 INFO 21062 --- [heartbeat-alarm-detect-by-target-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-target, name=10.46.128.24:8090, consumerId=2, resources=[ShardingSlice(consumerId=2, type=OBCluster, resourceIds=[1]), ShardingSlice(consumerId=2, type=Host, resourceIds=[1, 2, 3])] 2024-11-04 12:32:42.792 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:42.794 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:42.822 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:42.823 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241109 2024-11-04 12:32:42.824 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:42.825 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:32:43.105 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:43.107 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:43.135 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:43.137 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241110 2024-11-04 12:32:43.137 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:43.138 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:32:43.142 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 12:32:43.142 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 12:32:43.142 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 12:32:43.406 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:43.408 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:43.414 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 12:32:43.414 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 12:32:43.414 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 12:32:43.423 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_7.4 2024-11-04 12:32:43.424 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3 2024-11-04 12:32:43.425 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3 2024-11-04 12:32:43.437 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:43.438 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241111 2024-11-04 12:32:43.439 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:43.440 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:32:43.714 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:43.715 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:43.744 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:43.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20241112 2024-11-04 12:32:43.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:43.747 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:32:43.756 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.24:1046 success 2024-11-04 12:32:43.757 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.12:1046 success 2024-11-04 12:32:43.757 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.11:1046 success 2024-11-04 12:32:43.770 INFO 21062 --- [heartbeat-metric-collect-10.46.128.24:80901,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=10.46.128.24:8090, consumerId=3, resources=[ShardingSlice(consumerId=3, type=OB_CLUSTER, resourceIds=[1]), ShardingSlice(consumerId=3, type=HOST_MONITOR, resourceIds=[1, 2, 3])] 2024-11-04 12:32:44.021 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.023 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2024-11-04 12:32:44.051 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2024-11-04 12:32:44.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:32:44.056 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.057 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 12:32:44.061 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.24 2024-11-04 12:32:44.065 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.24, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:1 2024-11-04 12:32:44.066 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.12 2024-11-04 12:32:44.068 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=10.46.128.24 2024-11-04 12:32:44.069 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.11 2024-11-04 12:32:44.071 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.12, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:2 2024-11-04 12:32:44.076 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.11, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:3 2024-11-04 12:32:44.078 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=10.46.128.12 2024-11-04 12:32:44.081 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=10.46.128.11 2024-11-04 12:32:44.110 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:44.110 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:44.110 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:44.121 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241027 2024-11-04 12:32:44.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.123 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:32:44.129 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.129 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.129 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.160 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 10.46.128.11 2024-11-04 12:32:44.164 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 10.46.128.24 2024-11-04 12:32:44.165 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 10.46.128.12 2024-11-04 12:32:44.167 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.173 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.174 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.174 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.176 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.179 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.181 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2024-11-04 12:32:44.183 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2024-11-04 12:32:44.186 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2024-11-04 12:32:44.188 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=10.46.128.11, port=1046, loopCount=1, minSuccessCount=1 2024-11-04 12:32:44.190 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=10.46.128.24, port=1046, loopCount=1, minSuccessCount=1 2024-11-04 12:32:44.192 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=10.46.128.12, port=1046, loopCount=1, minSuccessCount=1 2024-11-04 12:32:44.212 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:44.212 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:44.217 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:44.218 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.223 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.224 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.261 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 10.46.128.24 2024-11-04 12:32:44.264 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 10.46.128.11 2024-11-04 12:32:44.272 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:44.265 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 10.46.128.12 2024-11-04 12:32:44.274 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:44.274 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=whoami, out=dba, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.274 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=whoami, out=dba, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.276 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=whoami, out=dba, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.278 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=whoami, out=dba, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.279 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=whoami, out=dba, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.282 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[85] 2024-11-04 12:32:44.284 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=whoami, out=dba, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:44.287 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[91] 2024-11-04 12:32:44.291 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=85, maxAcceptableDelayMillis=3000 2024-11-04 12:32:44.292 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[93] 2024-11-04 12:32:44.294 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=91, maxAcceptableDelayMillis=3000 2024-11-04 12:32:44.298 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=93, maxAcceptableDelayMillis=3000 2024-11-04 12:32:44.299 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.46.128.24], timeoutMillis=20000 2024-11-04 12:32:44.301 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.46.128.11], timeoutMillis=20000 2024-11-04 12:32:44.303 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.46.128.12], timeoutMillis=20000 2024-11-04 12:32:44.322 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.323 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:44.357 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1730694764 0 0 , error= 2024-11-04 12:32:44.356 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1730694764 -1 -1 , error= 2024-11-04 12:32:44.357 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1730694764 -2 -2 , error= 2024-11-04 12:32:44.360 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2024-11-04 12:32:44.362 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:44.362 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=-1, maxDiff=50 2024-11-04 12:32:44.365 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=-2, maxDiff=50 2024-11-04 12:32:44.369 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241028 2024-11-04 12:32:44.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:32:44.374 INFO 21062 --- [manual-subtask-executor2,e46f513af4706458,8d7b02862500405e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:44.375 INFO 21062 --- [manual-subtask-executor4,e46f513af4706458,b248a20c2be4febd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 65, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:44.378 INFO 21062 --- [manual-subtask-executor3,e46f513af4706458,5eb4bafa77b1d93f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 53, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:44.527 WARN 21062 --- [ocp-async-10,a5f2727066f4d10c,a6ddcadde7d8cf59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:44.527 INFO 21062 --- [ocp-async-1,b15dbc12e4956274,3454c807ac5bc6bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:44.565 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.567 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:44.596 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:44.597 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241029 2024-11-04 12:32:44.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:32:44.797 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.799 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:44.827 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:44.828 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241030 2024-11-04 12:32:44.828 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:44.829 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:32:44.970 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[6] 2024-11-04 12:32:44.983 INFO 21062 --- [prometheus-minute-schedule-1,20c530791c8e75eb,0a08ed483f105e71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:45.038 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.039 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:45.067 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:45.068 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241031 2024-11-04 12:32:45.069 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.070 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:32:45.278 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.280 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:45.327 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:45.328 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241101 2024-11-04 12:32:45.329 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:32:45.341 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,1e1785dae9e5ef33] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 6, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:45.363 INFO 21062 --- [manual-subtask-executor5,e46f513af4706458,7bbd0e70af3213f0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=6, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:45.338+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=6}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:45.371 INFO 21062 --- [manual-subtask-executor5,e46f513af4706458,7bbd0e70af3213f0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 6, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:45.521 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.522 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:45.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:45.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241102 2024-11-04 12:32:45.552 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.553 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:32:45.765 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.766 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:45.793 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:45.794 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241103 2024-11-04 12:32:45.795 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:45.796 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:32:46.012 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.013 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:46.040 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:46.041 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241104 2024-11-04 12:32:46.042 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.042 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:32:46.150 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[17, 11, 43] 2024-11-04 12:32:46.295 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:46.326 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:46.327 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241105 2024-11-04 12:32:46.329 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:32:46.357 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,5fe2a5288b3540ab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 11, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:46.370 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,5fe2a5288b3540ab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 17, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:46.379 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,5fe2a5288b3540ab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 43, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:46.381 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=11, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:46.353+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=11}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:46.384 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2024-11-04 12:32:46.388 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=17, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:46.367+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=17}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:46.391 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2024-11-04 12:32:46.397 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=43, context=Context{parallelIdx=1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:46.376+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=43}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:46.400 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2024-11-04 12:32:46.412 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:46.412 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:46.418 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.422 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.423 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:46.432 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.461 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:46.468 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.468 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:46.471 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:46.476 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.481 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.513 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 10.46.128.12 2024-11-04 12:32:46.520 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 10.46.128.11 2024-11-04 12:32:46.520 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 10.46.128.24 2024-11-04 12:32:46.527 WARN 21062 --- [ocp-async-4,13500fdb879808e4,a446fefea64ab9f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:46.528 INFO 21062 --- [ocp-async-6,4d7f99aee74eb63a,63ffd9a7f13610c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:46.531 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "cm-test2.gy.ntes", "release": "5.10.0-30-amd64", "version": "#1 SMP Debian 5.10.218-1 (2024-06-01)", "machine": "x86_64", "processor": "unknown", "is_in_docker": false, "linux_distribution_name": "debian", "linux_distribution_release": "11"}, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.534 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "cm-test2.gy.ntes", "release": "5.10.0-30-amd64", "version": "#1 SMP Debian 5.10.218-1 (2024-06-01)", "machine": "x86_64", "processor": "unknown", "is_in_docker": false, "linux_distribution_name": "debian", "linux_distribution_release": "11"}, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.542 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "cm-test1.gy.ntes", "release": "5.10.0-30-amd64", "version": "#1 SMP Debian 5.10.218-1 (2024-06-01)", "machine": "x86_64", "processor": "unknown", "is_in_docker": false, "linux_distribution_name": "debian", "linux_distribution_release": "11"}, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.543 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.545 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "cm-test1.gy.ntes", "release": "5.10.0-30-amd64", "version": "#1 SMP Debian 5.10.218-1 (2024-06-01)", "machine": "x86_64", "processor": "unknown", "is_in_docker": false, "linux_distribution_name": "debian", "linux_distribution_release": "11"}, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.546 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:46.547 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "mongodb10-online-cmpri.gy.ntes", "release": "3.10.0-1160.el7.x86_64", "version": "#1 SMP Mon Oct 19 16:18:59 UTC 2020", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "centos", "linux_distribution_release": "7"}, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.551 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "mongodb10-online-cmpri.gy.ntes", "release": "3.10.0-1160.el7.x86_64", "version": "#1 SMP Mon Oct 19 16:18:59 UTC 2020", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "centos", "linux_distribution_release": "7"}, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:46.577 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:46.578 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241106 2024-11-04 12:32:46.580 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:32:46.672 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2024-11-04 12:32:46.672 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2024-11-04 12:32:46.672 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2024-11-04 12:32:46.680 INFO 21062 --- [manual-subtask-executor7,e46f513af4706458,a49fb17465a3862f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 17, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:46.683 INFO 21062 --- [manual-subtask-executor8,e46f513af4706458,89e6deed4ae41f95] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 43, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:46.684 INFO 21062 --- [manual-subtask-executor6,e46f513af4706458,7eb35f1c0486fc72] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 11, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:46.786 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.788 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:46.813 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:46.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241107 2024-11-04 12:32:46.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:46.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:32:47.032 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.034 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:47.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:47.062 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241108 2024-11-04 12:32:47.062 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.063 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:32:47.274 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:47.275 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:47.282 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.283 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:47.309 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:47.310 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241109 2024-11-04 12:32:47.311 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.312 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:32:47.319 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[33] 2024-11-04 12:32:47.394 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,4e50b802cd586180] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 33, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:47.414 INFO 21062 --- [manual-subtask-executor9,e46f513af4706458,5a470b476fc7aabe] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=33, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:47.391+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=33}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:47.423 INFO 21062 --- [manual-subtask-executor9,e46f513af4706458,5a470b476fc7aabe] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 33, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:47.533 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:47.561 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:47.562 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241110 2024-11-04 12:32:47.563 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.564 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:32:47.786 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.788 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:47.813 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:47.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241111 2024-11-04 12:32:47.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:47.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:32:47.983 INFO 21062 --- [prometheus-minute-schedule-1,553a8b9643d661ba,e34f09e4f0f65964] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:48.049 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.050 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:48.076 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:48.077 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20241112 2024-11-04 12:32:48.078 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.079 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:32:48.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.301 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2024-11-04 12:32:48.327 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2024-11-04 12:32:48.329 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:32:48.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 12:32:48.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241020 2024-11-04 12:32:48.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:32:48.495 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[49, 9, 41] 2024-11-04 12:32:48.517 INFO 21062 --- [ocp-schedules-4,fda5a5d53287112a,0b1ca3b27f7d7d3d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:32:48.525 INFO 21062 --- [ocp-async-8,c46ddbf66500edfa,65a97a4b3bc84270] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:32:48.528 WARN 21062 --- [ocp-async-5,c1432096e42dcbec,785b54a6e666609c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:48.530 INFO 21062 --- [ocp-async-3,16704441eab8b4e9,53f297e35eaee86e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:48.669 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.670 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:48.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:48.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241021 2024-11-04 12:32:48.704 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.704 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:32:48.995 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:48.997 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:49.024 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:49.026 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241022 2024-11-04 12:32:49.026 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:49.027 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:32:49.302 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:49.304 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:49.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:49.332 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241023 2024-11-04 12:32:49.333 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:49.334 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:32:49.413 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,0e959d6afc6ec719] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:49.423 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,0e959d6afc6ec719] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 41, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:49.431 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,0e959d6afc6ec719] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 49, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:49.434 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9, context=Context{parallelIdx=1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:49.410+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=9}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:49.437 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2024-11-04 12:32:49.441 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=41, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:49.420+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=41}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:49.444 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2024-11-04 12:32:49.450 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=49, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:49.429+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=49}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:49.452 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2024-11-04 12:32:49.459 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:49.466 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.468 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:49.475 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.476 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:49.484 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.505 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:49.513 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.520 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:49.526 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.529 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:49.535 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.552 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:49.568 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:49.577 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:49.598 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.601 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.603 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.604 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.607 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.609 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.611 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:49.615 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:49.637 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:49.641 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:49.643 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241024 2024-11-04 12:32:49.644 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:49.645 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.645 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:32:49.648 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:49.649 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:49.654 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.657 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.684 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.24 2024-11-04 12:32:49.700 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.11 2024-11-04 12:32:49.701 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.12 2024-11-04 12:32:49.737 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.740 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.742 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.742 WARN 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2024-11-04 12:32:49.744 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.747 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2024-11-04 12:32:49.749 WARN 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2024-11-04 12:32:49.754 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2024-11-04 12:32:49.784 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:49.789 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:49.791 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.798 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.832 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.11 2024-11-04 12:32:49.841 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.12 2024-11-04 12:32:49.869 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.872 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.874 WARN 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2024-11-04 12:32:49.876 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2024-11-04 12:32:49.878 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.879 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2024-11-04 12:32:49.881 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:49.886 WARN 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2024-11-04 12:32:49.888 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2024-11-04 12:32:49.890 INFO 21062 --- [manual-subtask-executor11,e46f513af4706458,bdc311fc62e32010] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 41, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:49.890 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2024-11-04 12:32:49.898 INFO 21062 --- [manual-subtask-executor12,e46f513af4706458,1de985e6a1e118aa] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 49, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:49.918 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:49.920 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:49.944 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:49.945 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241025 2024-11-04 12:32:49.946 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:49.946 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:32:50.233 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:50.235 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:50.263 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:50.265 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241026 2024-11-04 12:32:50.265 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:50.266 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:32:50.276 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:50.277 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:50.528 WARN 21062 --- [ocp-async-2,00bb01ad091868ac,16d1ee04a4f1c2b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:50.528 INFO 21062 --- [ocp-async-1,3c4eb56a2c331906,b7b5dde99efcb7b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:50.545 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:50.547 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:50.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:50.574 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241027 2024-11-04 12:32:50.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:50.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:32:50.818 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:50.822 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:50.825 WARN 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2024-11-04 12:32:50.827 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2024-11-04 12:32:50.854 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:50.856 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:50.858 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:50.861 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:50.885 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:50.886 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241028 2024-11-04 12:32:50.887 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:50.888 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:32:50.901 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.24 2024-11-04 12:32:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d9012c9d77c871de,b73b63737a5d7735] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:51.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:51.168 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:51.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:51.195 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241029 2024-11-04 12:32:51.196 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:51.197 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:32:51.482 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:51.483 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:51.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:51.509 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241030 2024-11-04 12:32:51.510 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:51.511 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:32:51.799 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:51.800 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:51.826 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:51.827 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241031 2024-11-04 12:32:51.827 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:51.828 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:32:52.032 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:52.035 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:52.038 WARN 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2024-11-04 12:32:52.040 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2024-11-04 12:32:52.043 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2024-11-04 12:32:52.052 INFO 21062 --- [manual-subtask-executor10,e46f513af4706458,75cb674fbe0d9f20] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:52.104 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:52.105 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:52.132 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:52.133 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241101 2024-11-04 12:32:52.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:52.135 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:32:52.417 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:52.419 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:52.446 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:52.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241102 2024-11-04 12:32:52.448 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:52.448 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:32:52.528 WARN 21062 --- [ocp-async-7,758f510f1d4eb091,d36b189f4bbb80f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:52.528 INFO 21062 --- [ocp-async-6,8660a80e5df03bdf,0e716b9962356759] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:52.729 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:52.730 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:52.757 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:52.758 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241103 2024-11-04 12:32:52.759 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:52.759 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:32:53.039 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:53.040 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:53.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:53.067 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241104 2024-11-04 12:32:53.068 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:53.068 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:32:53.179 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[40] 2024-11-04 12:32:53.277 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:53.278 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:53.355 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:53.356 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:53.382 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:53.383 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241105 2024-11-04 12:32:53.383 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:53.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:32:53.454 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,3c7f85c9fd0d7112] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 40, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:53.475 INFO 21062 --- [manual-subtask-executor13,e46f513af4706458,6ec30c843d6f0c7b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=40, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:53.451+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=40}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:53.483 INFO 21062 --- [manual-subtask-executor13,e46f513af4706458,6ec30c843d6f0c7b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 40, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:32:53.634 WARN 21062 --- [part-initial-worker,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : LockException while executing with jdbc lock part-worker Cannot acquire lock for key part-worker 2024-11-04 12:32:53.669 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:53.671 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:53.697 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:53.698 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241106 2024-11-04 12:32:53.698 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:53.699 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:32:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6a2a4119d7341d0c,470b55b077fcc4b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:53.984 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:53.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:54.012 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:54.013 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241107 2024-11-04 12:32:54.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:54.015 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:32:54.294 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:54.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:54.325 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:54.326 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241108 2024-11-04 12:32:54.327 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:54.327 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:32:54.341 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[2, 35, 21] 2024-11-04 12:32:54.472 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,6327010ae456ca22] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 2, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:54.482 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,6327010ae456ca22] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:54.490 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,6327010ae456ca22] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 35, operation:EXECUTE, state: RUNNING 2024-11-04 12:32:54.496 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=2, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:54.469+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=2}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:54.499 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2024-11-04 12:32:54.501 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=21, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:54.480+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=21}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:54.504 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2024-11-04 12:32:54.507 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=35, context=Context{parallelIdx=1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T12:32:54.488+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=35}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 12:32:54.510 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2024-11-04 12:32:54.521 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:54.522 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:54.528 WARN 21062 --- [ocp-async-9,8519d36646bca02b,cb022097169b6313] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:54.528 INFO 21062 --- [ocp-async-8,145b21e154087ca9,436596ddccb9cd50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:54.527 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.528 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:54.529 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.536 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.573 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:54.574 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:54.575 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:54.578 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.581 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.585 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.617 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:54.618 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:54.621 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:54.624 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:54.624 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:54.646 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:54.647 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241109 2024-11-04 12:32:54.647 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:54.648 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.648 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:32:54.651 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.658 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.660 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.670 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.673 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.693 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:54.699 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.704 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:54.709 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.710 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:54.718 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.745 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:54.756 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:54.758 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:54.771 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.774 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.789 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.792 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.804 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.807 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.813 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:54.818 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.832 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:54.838 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.843 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:54.850 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.865 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:54.880 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:54.889 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:54.891 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.895 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.914 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.917 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.937 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.937 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:54.939 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.946 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.960 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:54.965 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.976 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:54.982 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:54.989 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.012 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:55.014 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.017 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.023 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:55.047 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.049 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.057 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.062 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.066 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.068 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.088 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.094 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.103 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:55.104 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:55.105 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:55.109 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.112 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.131 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:55.132 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241110 2024-11-04 12:32:55.132 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:55.133 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:32:55.135 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.136 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:55.137 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.152 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:55.171 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.174 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.177 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.182 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.197 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.199 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.216 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.222 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.229 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.236 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:55.242 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.255 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.258 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.268 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:55.282 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:55.297 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.302 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.303 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.308 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.329 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.332 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.348 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.349 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.354 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.369 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:55.376 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.377 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.379 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.400 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:55.416 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:55.417 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.424 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.434 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.437 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.461 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.464 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.465 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.476 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.481 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.495 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.498 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.500 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:55.506 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.524 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:55.531 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:55.533 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:55.537 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.542 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.546 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:55.558 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.560 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:55.561 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.562 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241111 2024-11-04 12:32:55.564 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:55.565 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:32:55.589 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.591 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.594 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.600 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.605 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.615 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.617 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.630 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:55.637 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.652 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:55.661 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.666 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.676 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:55.685 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.687 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.713 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 12:32:55.720 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.722 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.732 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.737 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.749 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.752 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.754 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 12:32:55.756 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:55.759 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:55.765 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.780 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 12:32:55.793 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.799 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.805 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24 2024-11-04 12:32:55.816 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.819 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.821 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 12:32:55.824 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:55.841 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.860 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.865 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.867 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.869 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:55.871 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.872 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:55.899 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:55.900 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20241112 2024-11-04 12:32:55.901 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:55.901 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:32:55.908 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:55.913 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:55.919 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.943 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.946 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.965 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:55.988 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:55.991 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.993 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:55.994 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.036 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:56.037 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:56.044 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.070 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.073 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.085 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 12:32:56.111 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.112 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:56.114 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.120 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:56.120 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.157 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:56.163 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.168 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 12:32:56.202 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.205 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.205 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 12:32:56.207 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:56.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2024-11-04 12:32:56.239 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2024-11-04 12:32:56.241 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.241 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:32:56.242 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.242 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.244 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:56.245 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.245 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 12:32:56.250 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 12:32:56.251 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.253 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:56.258 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 12:32:56.278 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:56.279 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:56.283 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241005 2024-11-04 12:32:56.284 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.285 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241005 values less than (1728172800000000)) 2024-11-04 12:32:56.285 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:56.290 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.292 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 12:32:56.330 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.333 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.335 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 12:32:56.337 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 12:32:56.338 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:56.343 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 12:32:56.373 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.376 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:56.376 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.381 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 12:32:56.381 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.383 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 12:32:56.421 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 12:32:56.427 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.428 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 12:32:56.462 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.464 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:56.465 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.468 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.471 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 12:32:56.469 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.12 2024-11-04 12:32:56.473 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 12:32:56.484 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on cm-test2.gy.ntes. , extOut=null, exitStatus=1) 2024-11-04 12:32:56.487 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on cm-test2.gy.ntes. , extOut=null, exitStatus=1) 2024-11-04 12:32:56.489 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', return code:1, output:, err:Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on cm-test2.gy.ntes. 2024-11-04 12:32:56.492 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:56.492 ERROR 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1, cause:null 2024-11-04 12:32:56.493 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241006 2024-11-04 12:32:56.495 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.496 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241006 values less than (1728259200000000)) 2024-11-04 12:32:56.500 ERROR 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.removeFiles(SshExecutor.java:345) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.removeAgentFiles(AgentInstallationTaskService.java:151) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.removeAgentFiles() at com.oceanbase.ocp.service.task.business.host.UninstallLegacyOcpAgentTask.run(UninstallLegacyOcpAgentTask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:32:56.512 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 12:32:56.513 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,8d53dcfce45110c9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:EXECUTE, state: FAILED 2024-11-04 12:32:56.519 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.528 WARN 21062 --- [ocp-async-2,f9435bedab97e652,f7656dbe863abd1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:56.529 INFO 21062 --- [ocp-async-10,78517621ae47b110,9d8d13e7153c119d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:56.564 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.11 2024-11-04 12:32:56.576 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on cm-test1.gy.ntes. , extOut=null, exitStatus=1) 2024-11-04 12:32:56.579 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on cm-test1.gy.ntes. , extOut=null, exitStatus=1) 2024-11-04 12:32:56.582 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', return code:1, output:, err:Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on cm-test1.gy.ntes. 2024-11-04 12:32:56.585 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1, cause:null 2024-11-04 12:32:56.587 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.removeFiles(SshExecutor.java:345) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.removeAgentFiles(AgentInstallationTaskService.java:151) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.removeAgentFiles() at com.oceanbase.ocp.service.task.business.host.UninstallLegacyOcpAgentTask.run(UninstallLegacyOcpAgentTask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:32:56.601 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,2d04d6ce902f99c6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 2, operation:EXECUTE, state: FAILED 2024-11-04 12:32:56.751 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.752 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:56.778 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:56.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241007 2024-11-04 12:32:56.780 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:56.780 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241007 values less than (1728345600000000)) 2024-11-04 12:32:56.968 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.971 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:56.974 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 12:32:56.977 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:56.983 INFO 21062 --- [prometheus-minute-schedule-1,8ac1bf1bb7e3a22f,ba7ec524618e8e73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:32:57.007 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:57.015 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.024 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.026 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:57.051 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:57.052 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241008 2024-11-04 12:32:57.053 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241008 values less than (1728432000000000)) 2024-11-04 12:32:57.054 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:57.097 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.099 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.136 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:57.143 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.182 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:57.227 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.230 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.267 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:57.274 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.313 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.314 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 12:32:57.315 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:57.342 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:57.344 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241009 2024-11-04 12:32:57.344 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.345 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241009 values less than (1728518400000000)) 2024-11-04 12:32:57.358 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.361 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.364 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:57.398 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:57.405 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:57.444 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24 2024-11-04 12:32:57.586 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.587 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:57.613 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:57.614 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241010 2024-11-04 12:32:57.615 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.616 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241010 values less than (1728604800000000)) 2024-11-04 12:32:57.851 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.853 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:57.879 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:57.880 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241011 2024-11-04 12:32:57.881 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:57.882 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241011 values less than (1728691200000000)) 2024-11-04 12:32:58.124 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:58.126 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:58.153 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:58.154 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241012 2024-11-04 12:32:58.155 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:58.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241012 values less than (1728777600000000)) 2024-11-04 12:32:58.405 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:58.407 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:58.433 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:58.434 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241013 2024-11-04 12:32:58.435 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:58.435 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241013 values less than (1728864000000000)) 2024-11-04 12:32:58.500 INFO 21062 --- [ocp-schedules-7,efa488d17fde991f,d0f4b3638284643f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:32:58.500 INFO 21062 --- [ocp-schedules-19,3e68f608e76a48c6,044d5a1551610703] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:32:58.500 INFO 21062 --- [ocp-schedules-10,5ffac5fd8f0f37d8,272c04a916719785] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:32:58.502 INFO 21062 --- [ocp-schedules-24,73894951505a8e4f,6b4e690e5851fbbc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:32:58.505 INFO 21062 --- [ocp-schedules-7,efa488d17fde991f,d0f4b3638284643f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:32:58.505 INFO 21062 --- [ocp-schedules-10,5ffac5fd8f0f37d8,272c04a916719785] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:32:58.505 INFO 21062 --- [ocp-schedules-19,3e68f608e76a48c6,044d5a1551610703] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:32:58.506 INFO 21062 --- [ocp-schedules-1,d1183c8823e81694,f5826967d0d533fc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:32:58.513 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:32:58.521 INFO 21062 --- [Check status for all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_status_check', labels={app=OB, check_item=cluster connection, failed_reason=-, ob_cluster_id=1730694047, ob_cluster_name=dba_test_ob, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1730694778 : 2024-11-04 04:32:58 2024-11-04 12:32:58.522 INFO 21062 --- [ocp-schedules-14,b28ce20d7d280b73,e55acde377937644] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:32:58.526 INFO 21062 --- [ocp-async-1,71054def4fde596f,3f7eced2f96ea8b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:32:58.529 WARN 21062 --- [ocp-async-4,08790d50275ac632,0d2b23cad7ca0872] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:32:58.529 INFO 21062 --- [ocp-async-6,57c3daf9dfec4840,562d74d2ee6a7a36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:32:58.556 INFO 21062 --- [ocp-schedules-25,4a1e76c9e6a20e37,a96f77d5cd82e876] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:32:58.563 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:58.566 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:58.569 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 12:32:58.571 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 12:32:58.573 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 12:32:58.603 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:58.610 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:58.624 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:32:58.624 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:32:58.624 INFO 21062 --- [Sync all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_sync', labels={app=OB, failed_reason=-, ob_cluster_id=1730694047, ob_cluster_name=dba_test_ob, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1730694778 : 2024-11-04 04:32:58 2024-11-04 12:32:58.651 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.24 2024-11-04 12:32:58.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:58.679 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:58.706 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:58.707 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241014 2024-11-04 12:32:58.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:58.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241014 values less than (1728950400000000)) 2024-11-04 12:32:59.025 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.027 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:59.056 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:59.058 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241015 2024-11-04 12:32:59.058 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241015 values less than (1729036800000000)) 2024-11-04 12:32:59.275 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:59.279 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:32:59.281 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:32:59.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:59.304 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241016 2024-11-04 12:32:59.304 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.305 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241016 values less than (1729123200000000)) 2024-11-04 12:32:59.563 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.565 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:59.591 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:59.592 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241017 2024-11-04 12:32:59.593 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.594 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241017 values less than (1729209600000000)) 2024-11-04 12:32:59.789 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:59.793 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:59.795 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 12:32:59.798 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 12:32:59.829 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 12:32:59.836 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 12:32:59.845 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.847 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:32:59.872 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:32:59.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241018 2024-11-04 12:32:59.874 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:32:59.874 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241018 values less than (1729296000000000)) 2024-11-04 12:32:59.875 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.24 2024-11-04 12:32:59.893 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on mongodb10-online-cmpri.gy.ntes. , extOut=null, exitStatus=1) 2024-11-04 12:32:59.896 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on mongodb10-online-cmpri.gy.ntes. , extOut=null, exitStatus=1) 2024-11-04 12:32:59.899 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', return code:1, output:, err:Sorry, user dba is not allowed to execute '/usr/bin/bash -c rm -rf '/home/admin/ocp_agent/*'' as root on mongodb10-online-cmpri.gy.ntes. 2024-11-04 12:32:59.901 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1, cause:null 2024-11-04 12:32:59.904 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', exit code: 1 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.removeFiles(SshExecutor.java:345) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.removeAgentFiles(AgentInstallationTaskService.java:151) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.removeAgentFiles() at com.oceanbase.ocp.service.task.business.host.UninstallLegacyOcpAgentTask.run(UninstallLegacyOcpAgentTask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:32:59.916 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,66e1e5e772f8aa52] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 35, operation:EXECUTE, state: FAILED 2024-11-04 12:32:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7932c938b49f3fd0,db8299e93c5dd36d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694780, interval=60 2024-11-04 12:33:00.128 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.129 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:00.154 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:00.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241019 2024-11-04 12:33:00.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.157 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241019 values less than (1729382400000000)) 2024-11-04 12:33:00.217 INFO 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=18, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=6, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=30, startTime=2024-11-04T12:32:47.391+08:00, endTime=2024-11-04T12:32:47.418+08:00}, SubtaskInstanc 2024-11-04 12:33:00.242 WARN 21062 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:33:00.243 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 12:33:00.399 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.400 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:00.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:00.426 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241020 2024-11-04 12:33:00.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.428 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:33:00.529 WARN 21062 --- [ocp-async-8,18776f68d2517e17,69d5e2e768385317] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:00.529 INFO 21062 --- [ocp-async-5,05c36664c3d961c3,1e339807e2581740] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:00.673 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.675 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:00.701 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:00.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241021 2024-11-04 12:33:00.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.704 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:33:00.945 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.947 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:00.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:00.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241022 2024-11-04 12:33:00.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:00.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:33:01.224 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:01.226 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:01.253 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:01.255 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241023 2024-11-04 12:33:01.255 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:01.256 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:33:01.616 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:01.618 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:01.645 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:01.646 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241024 2024-11-04 12:33:01.647 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:01.648 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:33:01.874 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:01.876 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:01.901 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:01.902 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241025 2024-11-04 12:33:01.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:01.904 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:33:02.172 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:02.173 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:02.199 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:02.200 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241026 2024-11-04 12:33:02.201 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:02.202 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:33:02.281 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:02.282 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:02.439 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:02.440 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:02.465 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:02.466 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241027 2024-11-04 12:33:02.467 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:02.468 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:33:02.530 WARN 21062 --- [ocp-async-3,1bf3ee2e7fb38549,ead86215bd3f2088] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:02.530 INFO 21062 --- [ocp-async-2,43d2d889523f2654,07a083d9ec81f61c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:02.731 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:02.733 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:02.758 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:02.760 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241028 2024-11-04 12:33:02.760 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:02.761 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:33:02.983 INFO 21062 --- [prometheus-minute-schedule-1,6352e8aa0b82a2ff,7700fd7803ffb558] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:03.042 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.043 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:03.070 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:03.071 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241029 2024-11-04 12:33:03.072 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.073 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:33:03.323 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.325 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:03.350 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:03.351 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241030 2024-11-04 12:33:03.351 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.352 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:03.606 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.608 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:03.632 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:03.633 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241031 2024-11-04 12:33:03.634 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.634 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:03.874 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.875 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:03.901 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:03.902 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241101 2024-11-04 12:33:03.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:03.904 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:04.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:04.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:04.187 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:04.189 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241102 2024-11-04 12:33:04.192 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:04.193 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:04.512 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:04.513 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:04.530 WARN 21062 --- [ocp-async-7,62696306374db84a,390777875869dd02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:04.530 INFO 21062 --- [ocp-async-4,85e2b95b3b05cb68,c8be91e31e466ffb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:04.540 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:04.541 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241103 2024-11-04 12:33:04.542 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:04.543 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:04.780 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:04.781 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:04.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:04.809 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241104 2024-11-04 12:33:04.809 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:04.810 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:05.064 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.066 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:05.091 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:05.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241105 2024-11-04 12:33:05.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.094 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:05.282 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:05.283 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:05.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:05.357 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:05.358 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241106 2024-11-04 12:33:05.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:05.649 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.650 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:05.675 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:05.677 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241107 2024-11-04 12:33:05.677 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:05.914 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.915 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:05.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:05.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241108 2024-11-04 12:33:05.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:05.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:05.983 INFO 21062 --- [prometheus-minute-schedule-1,81598151bffd9a2f,bde28c0bccd6db86] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:06.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:06.215 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:06.242 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:06.243 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241109 2024-11-04 12:33:06.244 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:06.245 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:06.530 WARN 21062 --- [ocp-async-8,70300f6091d30df4,0e42a12868f99ab4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:06.530 INFO 21062 --- [ocp-async-9,dcdd0d5c1a6cb7d6,14e10b6395aa893f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:06.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:06.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:06.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:06.576 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241110 2024-11-04 12:33:06.576 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:06.577 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:06.902 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:06.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:06.928 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:06.929 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241111 2024-11-04 12:33:06.930 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:06.931 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:07.172 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.174 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:07.200 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:07.201 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20241112 2024-11-04 12:33:07.202 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.203 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:07.438 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.439 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2024-11-04 12:33:07.464 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2024-11-04 12:33:07.466 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.467 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:07.468 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.468 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 12:33:07.505 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241020 2024-11-04 12:33:07.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:33:07.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.809 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:07.834 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:07.835 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241021 2024-11-04 12:33:07.835 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:07.836 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:33:08.249 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:08.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:08.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:08.278 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241022 2024-11-04 12:33:08.278 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:08.279 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:33:08.283 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:08.284 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:08.516 INFO 21062 --- [ocp-schedules-19,3e68f608e76a48c6,044d5a1551610703] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:33:08.522 INFO 21062 --- [ocp-schedules-7,efa488d17fde991f,d0f4b3638284643f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:33:08.524 INFO 21062 --- [ocp-schedules-29,07197fc4bcaa328f,144f1adf585b105b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:33:08.526 INFO 21062 --- [ocp-async-9,fd16d21b7a6ca77a,54e18e5c1e8b5b24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:33:08.530 WARN 21062 --- [ocp-async-5,17410f65058f6686,b6cb74f4d57f312e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:08.530 INFO 21062 --- [ocp-async-10,4717cb4cad05a2f1,27e4e2c254a45ce8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:08.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:08.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:08.842 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:08.843 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241023 2024-11-04 12:33:08.844 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:08.844 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:33:08.983 INFO 21062 --- [prometheus-minute-schedule-1,871518cec8e027ae,aafbca41b6a11d59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:09.254 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:09.256 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:09.281 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:09.282 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241024 2024-11-04 12:33:09.283 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:09.284 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:33:09.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:09.780 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:09.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:09.809 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241025 2024-11-04 12:33:09.810 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:09.811 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:33:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:33:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:33:10.180 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:10.182 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:10.207 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:10.208 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241026 2024-11-04 12:33:10.209 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:10.209 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,tps_rt,transaction_commit_rt,transaction_rollback_rt,active_session,all_session,request_queue_time,trans_commit_log_count,clog_trans_log_total_size,clog_write_disk_rt,trans_commit_log_sync_rt,io_read_count,io_write_count,clog_io_read_count,clog_io_write_count,io_read_rt,io_write_rt,clog_io_read_rt,clog_io_write_rt,io_read_size,io_write_size,clog_io_read_size,clog_io_write_size,transaction_single_partition_count,transaction_multi_partition_count, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_cpu_assigned_percentage,ob_disk_used_percentage,ob_memory_assigned_percentage, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,tps_rt,transaction_commit_rt,transaction_rollback_rt,active_session,all_session,request_queue_time,trans_commit_log_count,clog_trans_log_total_size,clog_write_disk_rt,trans_commit_log_sync_rt,io_read_count,io_write_count,clog_io_read_count,clog_io_write_count,io_read_rt,io_write_rt,clog_io_read_rt,clog_io_write_rt,io_read_size,io_write_size,clog_io_read_size,clog_io_write_size,transaction_single_partition_count,transaction_multi_partition_count, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id 2024-11-04 12:33:10.372 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_cpu_assigned_percentage,ob_disk_used_percentage,ob_memory_assigned_percentage, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id 2024-11-04 12:33:10.373 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = load1,load5,load15,memory_used,memory_buffers,memory_free,memory_cached,io_read,io_write,io_read_time,io_write_time,io_read_byte,io_write_byte,net_recv,net_send, label = app=HOST, gbLabel = app, ob_cluster_name, ob_cluster_id 2024-11-04 12:33:10.373 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = load1,load5,load15,memory_used,memory_buffers,memory_free,memory_cached,io_read,io_write,io_read_time,io_write_time,io_read_byte,io_write_byte,net_recv,net_send, label = app=HOST, gbLabel = app, ob_cluster_name, ob_cluster_id 2024-11-04 12:33:10.373 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = load1,load5,load15,cpu_percent,cpu_system,cpu_user,cpu_iowait,io_util,io_qusize,io_read,io_write,io_read_time,io_write_time,io_read_byte,io_write_byte,net_recv,net_send, label = app=HOST, gbLabel = app, svr_ip 2024-11-04 12:33:10.373 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = load1,load5,load15,cpu_percent,cpu_system,cpu_user,cpu_iowait,io_util,io_qusize,io_read,io_write,io_read_time,io_write_time,io_read_byte,io_write_byte,net_recv,net_send, label = app=HOST, gbLabel = app, svr_ip 2024-11-04 12:33:10.374 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 0, ruleSize = 0. 2024-11-04 12:33:10.374 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 0, ruleSize = 0. 2024-11-04 12:33:10.530 WARN 21062 --- [ocp-async-1,3bf1c18683d0b29b,ce0d166e0745ae9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:10.530 INFO 21062 --- [ocp-async-2,398035ef8af24ad4,6c800037cedc6d15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:10.620 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:10.622 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:10.648 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:10.649 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241027 2024-11-04 12:33:10.650 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:10.651 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:33:11.045 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:11.047 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:11.072 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:11.073 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241028 2024-11-04 12:33:11.074 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:11.075 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:33:11.285 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:11.286 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:11.492 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:11.494 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:11.520 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:11.521 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241029 2024-11-04 12:33:11.522 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:11.522 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:33:11.912 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:11.913 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:11.939 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:11.940 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241030 2024-11-04 12:33:11.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:11.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e7ca32aad2d509d7,1fc4435f66e85fcd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:12.339 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:12.340 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:12.365 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:12.366 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241031 2024-11-04 12:33:12.367 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:12.367 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:12.531 WARN 21062 --- [ocp-async-6,009d72eb84542d23,a8fc1211e63066e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:12.531 INFO 21062 --- [ocp-async-4,d388940077c03892,005a8eade0825371] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:12.778 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:12.780 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:12.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:12.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241101 2024-11-04 12:33:12.809 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:12.809 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:13.209 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:13.210 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:13.238 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:13.239 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241102 2024-11-04 12:33:13.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:13.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:13.646 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:13.649 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:13.676 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:13.677 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241103 2024-11-04 12:33:13.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:13.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:13.829 ERROR 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing SQL_PERF_DIAGNOSER org.springframework.dao.TransientDataAccessResourceException: ### Error updating database. Cause: java.sql.SQLTransientConnectionException: (conn=1148743682) Table has no partition for value ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.save-Inline ### The error occurred while setting parameters ### SQL: REPLACE INTO ocp_perf_sql_diagnosis_task (task_name, begin_interval_time, end_interval_time, executor, status) VALUES (?, ?, ?, ?, ?) ### Cause: java.sql.SQLTransientConnectionException: (conn=1148743682) Table has no partition for value ; (conn=1148743682) Table has no partition for value; nested exception is java.sql.SQLTransientConnectionException: (conn=1148743682) Table has no partition for value at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy324.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy520.save(Unknown Source) at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.save(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:25) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:48) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$9ef387aa.getNextExecution() at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLTransientConnectionException: (conn=1148743682) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:256) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:170) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:247) ... 43 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2347) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2210) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2098) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 44 common frames omitted 2024-11-04 12:33:13.829 ERROR 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing PLAN_CHANGE_DETECTOR org.springframework.dao.TransientDataAccessResourceException: ### Error updating database. Cause: java.sql.SQLTransientConnectionException: (conn=1148747778) Table has no partition for value ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.save-Inline ### The error occurred while setting parameters ### SQL: REPLACE INTO ocp_perf_sql_diagnosis_task (task_name, begin_interval_time, end_interval_time, executor, status) VALUES (?, ?, ?, ?, ?) ### Cause: java.sql.SQLTransientConnectionException: (conn=1148747778) Table has no partition for value ; (conn=1148747778) Table has no partition for value; nested exception is java.sql.SQLTransientConnectionException: (conn=1148747778) Table has no partition for value at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy324.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy520.save(Unknown Source) at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.save(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:25) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:48) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$9ef387aa.getNextExecution() at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLTransientConnectionException: (conn=1148747778) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:256) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:170) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:247) ... 43 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2347) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2210) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2098) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 44 common frames omitted 2024-11-04 12:33:13.829 ERROR 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing SQL_INDEX_DIAGNOSER org.springframework.dao.TransientDataAccessResourceException: ### Error updating database. Cause: java.sql.SQLTransientConnectionException: (conn=1148745730) Table has no partition for value ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.save-Inline ### The error occurred while setting parameters ### SQL: REPLACE INTO ocp_perf_sql_diagnosis_task (task_name, begin_interval_time, end_interval_time, executor, status) VALUES (?, ?, ?, ?, ?) ### Cause: java.sql.SQLTransientConnectionException: (conn=1148745730) Table has no partition for value ; (conn=1148745730) Table has no partition for value; nested exception is java.sql.SQLTransientConnectionException: (conn=1148745730) Table has no partition for value at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy324.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy520.save(Unknown Source) at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.save(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:25) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:48) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$9ef387aa.getNextExecution() at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLTransientConnectionException: (conn=1148745730) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:256) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:170) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:247) ... 43 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2347) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2210) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2098) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 44 common frames omitted 2024-11-04 12:33:13.985 INFO 21062 --- [prometheus-cache-monitor1,50c340b657b15b51,a994f1f6be529ee9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2024-11-04 12:33:13.985 INFO 21062 --- [prometheus-cache-monitor1,50c340b657b15b51,a994f1f6be529ee9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2024-11-04 12:33:14.079 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:14.081 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:14.107 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:14.108 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241104 2024-11-04 12:33:14.108 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:14.109 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:14.286 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:14.287 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:14.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:14.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:14.531 WARN 21062 --- [ocp-async-9,499d59d4b95b7a99,7b91ac5832f00dc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:14.531 INFO 21062 --- [ocp-async-3,c7d161a43e6f28ab,26887e5d0769072c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:14.533 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:14.534 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241105 2024-11-04 12:33:14.534 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:14.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:14.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:14.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:14.979 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:14.981 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241106 2024-11-04 12:33:14.981 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:14.982 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:14.983 INFO 21062 --- [prometheus-minute-schedule-1,c8532e90aef39bdd,72b877822500af3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:15.372 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:15.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:15.399 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:15.401 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241107 2024-11-04 12:33:15.402 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:15.402 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:15.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:15.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:15.843 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:15.844 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241108 2024-11-04 12:33:15.845 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:15.845 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:16.275 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:16.276 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:16.301 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:16.302 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241109 2024-11-04 12:33:16.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:16.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:16.531 INFO 21062 --- [ocp-async-1,e7da8ca44a9892ee,a8fec6146c1f4ba7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:16.531 WARN 21062 --- [ocp-async-7,d1958b0892db8fb9,3cf4d4eabb928946] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:16.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:16.705 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:16.730 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:16.731 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241110 2024-11-04 12:33:16.731 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:16.732 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:17.138 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:17.139 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:17.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:17.168 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241111 2024-11-04 12:33:17.168 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:17.169 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:17.288 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:17.289 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:17.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:17.600 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:17.626 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:17.627 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20241112 2024-11-04 12:33:17.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:17.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:17.983 INFO 21062 --- [prometheus-minute-schedule-1,894e6c2bb718121d,3933c4d8655abd80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:18.053 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2024-11-04 12:33:18.078 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2024-11-04 12:33:18.080 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.081 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:18.081 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.082 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 12:33:18.101 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241020 2024-11-04 12:33:18.102 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.102 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:33:18.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.298 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:18.323 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:18.324 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241021 2024-11-04 12:33:18.324 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.325 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:33:18.526 INFO 21062 --- [ocp-async-2,8665559a1d2a09af,4e8d5925065ce3a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:33:18.527 INFO 21062 --- [ocp-schedules-19,5698ff2fc27cb86b,d05286d03e6ba1db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:33:18.531 INFO 21062 --- [ocp-async-8,f64678e298527ae2,8190c5ae29a46788] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:18.531 WARN 21062 --- [ocp-async-4,0064d40b6c070076,aa4a5ac279a45f89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:18.545 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.547 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:18.571 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:18.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241022 2024-11-04 12:33:18.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:33:18.777 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:18.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:18.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241023 2024-11-04 12:33:18.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:18.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:33:19.034 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.036 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:19.063 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:19.064 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241024 2024-11-04 12:33:19.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.066 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:33:19.266 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.268 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:19.295 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:19.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241025 2024-11-04 12:33:19.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.298 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:33:19.524 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.525 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:19.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:19.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241026 2024-11-04 12:33:19.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.552 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:33:19.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.704 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:19.729 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:19.730 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241027 2024-11-04 12:33:19.731 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.732 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:33:19.880 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.882 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:19.907 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:19.908 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241028 2024-11-04 12:33:19.908 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:19.909 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:33:20.063 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.064 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:20.088 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:20.089 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241029 2024-11-04 12:33:20.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:33:20.239 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:20.265 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:20.266 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241030 2024-11-04 12:33:20.267 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.267 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:20.289 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:20.290 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:20.416 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.418 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:20.442 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:20.443 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241031 2024-11-04 12:33:20.443 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.444 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:20.531 WARN 21062 --- [ocp-async-3,93c0feaab238987f,3ebb335ece07b619] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:20.531 INFO 21062 --- [ocp-async-5,1c9b47a16de915e6,565b93cb101d8093] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:20.592 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.594 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:20.618 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:20.620 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241101 2024-11-04 12:33:20.620 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.621 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:20.770 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.771 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:20.800 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:20.801 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241102 2024-11-04 12:33:20.801 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.802 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:20.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:20.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b8d6ec06063af792,6c1085cc8ee2c6c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:21.001 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:21.002 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241103 2024-11-04 12:33:21.003 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.003 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:21.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.161 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:21.187 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:21.188 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241104 2024-11-04 12:33:21.189 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:21.339 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:21.369 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:21.370 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241105 2024-11-04 12:33:21.371 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.372 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:21.522 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.523 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:21.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:21.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241106 2024-11-04 12:33:21.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:21.698 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.699 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:21.723 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:21.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241107 2024-11-04 12:33:21.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:21.875 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.876 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:21.901 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:21.902 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241108 2024-11-04 12:33:21.902 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:21.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:22.051 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.052 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:22.077 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:22.078 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241109 2024-11-04 12:33:22.079 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.079 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:22.235 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.237 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:22.264 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:22.265 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241110 2024-11-04 12:33:22.265 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.266 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:22.419 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.420 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:22.446 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:22.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241111 2024-11-04 12:33:22.448 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.448 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:22.532 WARN 21062 --- [ocp-async-1,f51c45b287e07a28,80e74d26e90b418d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:22.532 INFO 21062 --- [ocp-async-10,b66bb106ebafc94c,7b250140d844b209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:22.602 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.603 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:22.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:22.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20241112 2024-11-04 12:33:22.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:22.782 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.784 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2024-11-04 12:33:22.812 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2024-11-04 12:33:22.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:22.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 12:33:22.833 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241020 2024-11-04 12:33:22.833 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.834 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:33:22.978 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:22.979 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:23.005 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:23.006 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241021 2024-11-04 12:33:23.006 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.007 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:33:23.154 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.155 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:23.182 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:23.183 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241022 2024-11-04 12:33:23.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:33:23.291 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:23.292 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:23.333 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.334 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:23.362 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:23.363 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241023 2024-11-04 12:33:23.364 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.365 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:33:23.515 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.517 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:23.543 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:23.544 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241024 2024-11-04 12:33:23.545 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.545 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:33:23.696 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.697 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:23.723 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:23.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241025 2024-11-04 12:33:23.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:33:23.875 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.877 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:23.902 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:23.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241026 2024-11-04 12:33:23.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:23.904 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:33:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ed3f5345475943ad,6e7c46b426e59e4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:24.050 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.051 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:24.076 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:24.077 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241027 2024-11-04 12:33:24.078 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.078 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:33:24.227 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.228 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:24.253 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:24.254 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241028 2024-11-04 12:33:24.255 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.256 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:33:24.404 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.405 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:24.430 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:24.431 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241029 2024-11-04 12:33:24.432 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.432 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:33:24.532 WARN 21062 --- [ocp-async-2,05d697ae6b1118d2,442f974430169db1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:24.532 INFO 21062 --- [ocp-async-6,8d26c08067cb723e,ad56c45288f45e0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:24.580 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:24.605 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:24.606 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241030 2024-11-04 12:33:24.607 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.607 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:24.755 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.756 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:24.781 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:24.782 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241031 2024-11-04 12:33:24.783 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.784 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:24.934 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.936 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:24.961 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:24.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241101 2024-11-04 12:33:24.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:24.963 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:25.111 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.113 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:25.138 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:25.139 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241102 2024-11-04 12:33:25.139 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.140 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:25.292 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.293 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:25.319 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:25.320 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241103 2024-11-04 12:33:25.321 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.322 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:25.477 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.478 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:25.505 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:25.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241104 2024-11-04 12:33:25.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:25.662 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.663 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:25.689 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:25.690 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241105 2024-11-04 12:33:25.690 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.691 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:25.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.843 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:25.869 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:25.870 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241106 2024-11-04 12:33:25.870 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:25.871 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:26.021 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.023 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:26.049 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:26.050 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241107 2024-11-04 12:33:26.051 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.052 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:26.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.192 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:26.218 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:26.219 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241108 2024-11-04 12:33:26.219 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.220 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:26.292 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:26.293 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:26.371 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:26.398 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:26.399 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241109 2024-11-04 12:33:26.400 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.400 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:26.532 WARN 21062 --- [ocp-async-9,948554bfa11fd5e8,db731af1fbb5f32c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:26.532 INFO 21062 --- [ocp-async-3,d951f05733347be5,f2d8d90632955838] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:26.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.553 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:26.577 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:26.578 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241110 2024-11-04 12:33:26.578 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.579 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:26.734 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.735 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:26.761 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:26.762 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241111 2024-11-04 12:33:26.763 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.764 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:26.915 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.917 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:26.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:26.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20241112 2024-11-04 12:33:26.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:26.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:26.983 INFO 21062 --- [prometheus-minute-schedule-1,d1d6e2d7b6727992,b0780d1d935e924a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:26.984 INFO 21062 --- [prometheus-hour-schedule-1,023ece32057e8ebc,7382267c8d8e0fd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:33:27.094 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.095 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2024-11-04 12:33:27.119 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2024-11-04 12:33:27.121 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.121 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:27.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:27.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241030 2024-11-04 12:33:27.185 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.186 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:27.390 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.392 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:27.417 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:27.418 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241031 2024-11-04 12:33:27.418 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.419 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:27.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.679 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:27.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:27.705 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241101 2024-11-04 12:33:27.705 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:27.706 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:28.007 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.008 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:28.035 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:28.036 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241102 2024-11-04 12:33:28.037 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.037 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:28.298 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:28.323 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:28.324 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241103 2024-11-04 12:33:28.325 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.326 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:28.498 INFO 21062 --- [ocp-schedules-27,69236832e2c013f2,d03582d0449f0715] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:33:28.500 INFO 21062 --- [ocp-schedules-13,74f979d2c4d05768,1afc93a234a83bf3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:33:28.501 INFO 21062 --- [ocp-schedules-21,e9009e9457361667,f6025aea307b9fca] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:33:28.501 INFO 21062 --- [ocp-schedules-16,a7079ef2dad9b30b,e2fbea06543c52c9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:33:28.502 INFO 21062 --- [ocp-schedules-3,c6357e468a51af02,f9e14ccd21ed70fe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:33:28.503 INFO 21062 --- [ocp-schedules-27,69236832e2c013f2,d03582d0449f0715] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:33:28.505 INFO 21062 --- [ocp-schedules-16,a7079ef2dad9b30b,e2fbea06543c52c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:28.506 INFO 21062 --- [ocp-schedules-21,e9009e9457361667,f6025aea307b9fca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:28.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:33:28.507 INFO 21062 --- [ocp-schedules-1,df2a5d6e9144015a,eb2bdf3a384aa13f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:33:28.509 INFO 21062 --- [ocp-schedules-13,74f979d2c4d05768,1afc93a234a83bf3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:33:28.523 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.ocp.alarm.core.task.AlarmDetectTask : refresh host alarm wrapper 2024-11-04 12:33:28.526 INFO 21062 --- [ocp-async-7,8aae5d01a90b4cdc,7cd32a78288a24c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:33:28.526 INFO 21062 --- [ocp-schedules-19,85c6b88f5fa9283a,59add00df146c8b9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:33:28.538 INFO 21062 --- [ocp-schedules-25,1af7c134aa3e2628,35e40ff216dc3ace] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:33:28.538 WARN 21062 --- [ocp-async-4,7b80576406fb0658,6ba10973faefed7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:28.538 INFO 21062 --- [ocp-async-10,f0be48c3ece35c98,dbdf2386ec8c90fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:28.540 INFO 21062 --- [ocp-schedules-9,714390f10abb2752,f22280f5e0b3772e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:33:28.541 INFO 21062 --- [ocp-schedules-24,213a6f034ac3a998,41cec9a4813e3bf4] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 12:33:28.541 INFO 21062 --- [ocp-schedules-24,213a6f034ac3a998,41cec9a4813e3bf4] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 12:33:28.546 INFO 21062 --- [ocp-schedules-7,128a9e4980d8cd76,9c7df676825f7e91] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:33:28.547 INFO 21062 --- [ocp-schedules-9,714390f10abb2752,f22280f5e0b3772e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:28.550 INFO 21062 --- [ocp-schedules-7,128a9e4980d8cd76,9c7df676825f7e91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:28.552 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:33:28.554 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:33:28.555 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.ocp.alarm.core.task.AlarmDetectTask : refresh alarm target 2024-11-04 12:33:28.555 INFO 21062 --- [ocp-schedules-15,db04e68d52ad77df,031573787ea21d36] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:33:28.555 INFO 21062 --- [ocp-schedules-15,db04e68d52ad77df,031573787ea21d36] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:33:28.559 INFO 21062 --- [ocp-schedules-27,d1f8265eff89f20d,a01a47e84e5b3f7c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:33:28.561 INFO 21062 --- [ocp-schedules-19,85c6b88f5fa9283a,59add00df146c8b9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:33:28.562 INFO 21062 --- [ocp-schedules-8,018dcb8eeafb5c9f,e6d9cb09fa4ba4e5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:33:28.562 INFO 21062 --- [ocp-schedules-8,cec911c383b5c5db,ae4b61fbcfb1864d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:33:28.562 INFO 21062 --- [ocp-schedules-8,cec911c383b5c5db,ae4b61fbcfb1864d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:33:28.565 INFO 21062 --- [ocp-schedules-25,62cc0e4e38b5ac36,d4afa5e57ef0c918] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:33:28.582 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.a.service.TaskRepositoryService : fetchNeedExecute, tryAcquireTasksCount=1, fetchedCount=1 2024-11-04 12:33:28.582 INFO 21062 --- [ocp-schedules-25,62cc0e4e38b5ac36,d4afa5e57ef0c918] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:33:28.582 INFO 21062 --- [ocp-schedules-25,62cc0e4e38b5ac36,d4afa5e57ef0c918] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:33:28.586 INFO 21062 --- [ocp-schedules-2,02a64d0a1e3d1a8a,063146daa35d6dfd] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:33:28.586+08:00, clusters=[1] 2024-11-04 12:33:28.587 INFO 21062 --- [ocp-schedules-15,db04e68d52ad77df,031573787ea21d36] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:33:28.587 INFO 21062 --- [ocp-schedules-15,db04e68d52ad77df,031573787ea21d36] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:33:28.587 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.588 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:28.592 INFO 21062 --- [ocp-schedules-6,a90a28308e41bdd8,682aed63395790a5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:33:28.592+08:00, clusters=[1] 2024-11-04 12:33:28.595 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.alarm.core.process.AlarmProcessor : AlarmProcessor init done, lastProcessTime=2024-11-04T12:33:28.587958+08:00 2024-11-04 12:33:28.596 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.alarm.core.task.AlarmProcessTask : AlarmProcessTask created 2024-11-04 12:33:28.596 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.alarm.utils.task.AlarmTaskManager : start task, taskInfo=TaskInfo(id=1, uuid=354af545-dee8-4634-bc44-2ef0a637b63a, taskType=AlarmProcess, target=null, parameters=null, intervalSeconds=10, status=EXECUTING, executorName=10.46.128.24:21062, createAt=2024-11-04T04:28:22.561686Z, startAt=2024-11-04T04:33:28.558Z, endAt=null, heartbeatAt=2024-11-04T04:33:28.558Z, result=executing) 2024-11-04 12:33:28.596 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.alarm.utils.task.AlarmTaskManager : startNeedExecute, activeCount=0, tryStartCount=1, startedCount=1 2024-11-04 12:33:28.597 INFO 21062 --- [ocp-schedules-24,92a08789d73bec38,b51ac37edf301c29] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:33:28.603 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:33:28.603 INFO 21062 --- [ocp-schedules-29,4a9c660eabd3cfdf,578bd066f7aad6c5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:33:28.606 INFO 21062 --- [ocp-schedules-3,c00c4340288ae502,729fd3961b9bb1d8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:33:28.606 INFO 21062 --- [ocp-schedules-3,c00c4340288ae502,729fd3961b9bb1d8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:33:28.606 INFO 21062 --- [ocp-schedules-6,a90a28308e41bdd8,682aed63395790a5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:33:28.613 INFO 21062 --- [ocp-schedules-3,c00c4340288ae502,729fd3961b9bb1d8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:33:28.613 INFO 21062 --- [ocp-schedules-3,c00c4340288ae502,729fd3961b9bb1d8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:33:28.615 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:28.616 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241104 2024-11-04 12:33:28.616 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.617 INFO 21062 --- [ocp-schedules-3,c00c4340288ae502,729fd3961b9bb1d8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:33:28.617 INFO 21062 --- [ocp-schedules-3,c00c4340288ae502,729fd3961b9bb1d8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:33:28.617 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:28.625 INFO 21062 --- [ocp-schedules-10,5ffac5fd8f0f37d8,272c04a916719785] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:33:28.625 INFO 21062 --- [ocp-schedules-10,57902c2903176d62,c2e6eee3abab4129] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:33:28.629 INFO 21062 --- [ocp-schedules-2,02a64d0a1e3d1a8a,063146daa35d6dfd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:33:28.630 INFO 21062 --- [ocp-schedules-10,57902c2903176d62,c2e6eee3abab4129] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:28.665 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:33:28.665 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:33:28.676 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:33:28.673912+08:00, lastProcessTime=2024-11-04T12:33:28.587958+08:00 2024-11-04 12:33:28.695 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:33:28.697 ERROR 21062 --- [ocp-schedules-9,714390f10abb2752,f22280f5e0b3772e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:33:28.697 INFO 21062 --- [ocp-schedules-9,714390f10abb2752,f22280f5e0b3772e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:33:28.712 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2024-11-04 12:33:28.722 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets 2024-11-04 12:33:28.723 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2024-11-04 12:33:28.723 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2024-11-04 12:33:28.724 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2024-11-04 12:33:28.724 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2024-11-04 12:33:28.724 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2024-11-04 12:33:28.724 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2024-11-04 12:33:28.724 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2024-11-04 12:33:28.725 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets 2024-11-04 12:33:28.725 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2024-11-04 12:33:28.725 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2024-11-04 12:33:28.727 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2024-11-04 12:33:28.728 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2024-11-04 12:33:28.728 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.728 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2024-11-04 12:33:28.728 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2024-11-04 12:33:28.728 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2024-11-04 12:33:28.729 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-11-04 12:33:28.729 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2024-11-04 12:33:28.729 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2024-11-04 12:33:28.751 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2024-11-04 12:33:28.751 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2024-11-04 12:33:28.752 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.752 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2024-11-04 12:33:28.752 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2024-11-04 12:33:28.752 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2024-11-04 12:33:28.753 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.753 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2024-11-04 12:33:28.753 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2024-11-04 12:33:28.753 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:28.753 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2024-11-04 12:33:28.754 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2024-11-04 12:33:28.755 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2024-11-04 12:33:28.756 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2024-11-04 12:33:28.757 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2024-11-04 12:33:28.757 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2024-11-04 12:33:28.757 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2024-11-04 12:33:28.758 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2024-11-04 12:33:28.759 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets 2024-11-04 12:33:28.759 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2024-11-04 12:33:28.759 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2024-11-04 12:33:28.759 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2024-11-04 12:33:28.759 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2024-11-04 12:33:28.759 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2024-11-04 12:33:28.759 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2024-11-04 12:33:28.760 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2024-11-04 12:33:28.761 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2024-11-04 12:33:28.762 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2024-11-04 12:33:28.763 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2024-11-04 12:33:28.763 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2024-11-04 12:33:28.763 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2024-11-04 12:33:28.763 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2024-11-04 12:33:28.763 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2024-11-04 12:33:28.763 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2024-11-04 12:33:28.763 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2024-11-04 12:33:28.764 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2024-11-04 12:33:28.765 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2024-11-04 12:33:28.766 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2024-11-04 12:33:28.767 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2024-11-04 12:33:28.768 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2024-11-04 12:33:28.769 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2024-11-04 12:33:28.770 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2024-11-04 12:33:28.771 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2024-11-04 12:33:28.772 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2024-11-04 12:33:28.773 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2024-11-04 12:33:28.774 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2024-11-04 12:33:28.775 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2024-11-04 12:33:28.776 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2024-11-04 12:33:28.777 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2024-11-04 12:33:28.778 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2024-11-04 12:33:28.778 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2024-11-04 12:33:28.778 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2024-11-04 12:33:28.778 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2024-11-04 12:33:28.778 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2024-11-04 12:33:28.779 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2024-11-04 12:33:28.780 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2024-11-04 12:33:28.781 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2024-11-04 12:33:28.782 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2024-11-04 12:33:28.783 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2024-11-04 12:33:28.784 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2024-11-04 12:33:28.785 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2024-11-04 12:33:28.786 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2024-11-04 12:33:28.787 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2024-11-04 12:33:28.788 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2024-11-04 12:33:28.789 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2024-11-04 12:33:28.790 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2024-11-04 12:33:28.791 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2024-11-04 12:33:28.792 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2024-11-04 12:33:28.792 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2024-11-04 12:33:28.792 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.792 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2024-11-04 12:33:28.792 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2024-11-04 12:33:28.794 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2024-11-04 12:33:28.795 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.796 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets 2024-11-04 12:33:28.797 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2024-11-04 12:33:28.798 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2024-11-04 12:33:28.799 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2024-11-04 12:33:28.800 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2024-11-04 12:33:28.801 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2024-11-04 12:33:28.801 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2024-11-04 12:33:28.801 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2024-11-04 12:33:28.801 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2024-11-04 12:33:28.801 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2024-11-04 12:33:28.802 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2024-11-04 12:33:28.803 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2024-11-04 12:33:28.804 INFO 21062 --- [ocp-schedules-12,ac112a50f275c49f,15c264b730d8d266] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=129, updatedCount=129, failedToUpdateCount=0 2024-11-04 12:33:28.842 INFO 21062 --- [ocp-schedules-14,f101a05397930bbf,17696bf2f3b60188] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2024-11-04T12:32:00.344+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system-dba_test_ob admin-dba_test_ob, subscribedJudgerVersion subscription-0-2024-11-04T12:32:00.344+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system- admin- 2024-11-04 12:33:28.843 INFO 21062 --- [ocp-schedules-14,f101a05397930bbf,17696bf2f3b60188] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2024-11-04 12:33:28.843 INFO 21062 --- [ocp-schedules-14,f101a05397930bbf,17696bf2f3b60188] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2024-11-04T12:32:00.344+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system-dba_test_ob admin-dba_test_ob, subscribedJudgerVersion subscription-0-2024-11-04T12:32:00.344+08:00_groups-11-2024-11-04T12:25:53+08:00-2024-11-04T12:25:53+08:00_detectRules-536-2024-11-04T04:25:56.434604Z_readableClusters_2_system- admin- 2024-11-04 12:33:28.843 INFO 21062 --- [ocp-schedules-14,f101a05397930bbf,17696bf2f3b60188] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2024-11-04 12:33:28.843 INFO 21062 --- [ocp-schedules-14,f101a05397930bbf,17696bf2f3b60188] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2024-11-04 12:33:28.883 INFO 21062 --- [ocp-schedules-17,0bba0ae12634fa09,51db55552edb9350] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets , current targets ,,,,,C_dba_test_ob-1730694047-PRIMARY,C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24,C_dba_test_ob-1730694047/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24,H_10.46.128.11,H_10.46.128.12,H_10.46.128.24,H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.885 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.887 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:28.913 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:33:28.673912+08:00 2024-11-04 12:33:28.914 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:28.915 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241105 2024-11-04 12:33:28.916 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:28.917 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:28.980 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2024-11-04 12:33:28.990 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:28.990 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2024-11-04 12:33:28.990 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2024-11-04 12:33:28.990 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2024-11-04 12:33:28.992 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2024-11-04 12:33:28.993 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2024-11-04 12:33:28.994 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2024-11-04 12:33:28.995 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2024-11-04 12:33:28.996 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2024-11-04 12:33:28.997 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2024-11-04 12:33:28.998 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2024-11-04 12:33:28.999 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.999 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2024-11-04 12:33:28.999 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2024-11-04 12:33:28.999 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2024-11-04 12:33:28.999 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:28.999 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2024-11-04 12:33:28.999 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2024-11-04 12:33:29.000 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2024-11-04 12:33:29.001 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2024-11-04 12:33:29.002 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2024-11-04 12:33:29.003 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2024-11-04 12:33:29.004 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2024-11-04 12:33:29.005 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2024-11-04 12:33:29.006 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2024-11-04 12:33:29.007 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2024-11-04 12:33:29.008 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2024-11-04 12:33:29.009 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2024-11-04 12:33:29.010 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2024-11-04 12:33:29.011 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2024-11-04 12:33:29.012 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2024-11-04 12:33:29.013 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2024-11-04 12:33:29.014 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2024-11-04 12:33:29.015 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2024-11-04 12:33:29.016 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2024-11-04 12:33:29.017 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2024-11-04 12:33:29.018 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2024-11-04 12:33:29.019 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2024-11-04 12:33:29.021 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2024-11-04 12:33:29.022 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2024-11-04 12:33:29.022 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2024-11-04 12:33:29.022 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.022 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2024-11-04 12:33:29.022 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2024-11-04 12:33:29.023 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2024-11-04 12:33:29.024 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2024-11-04 12:33:29.025 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2024-11-04 12:33:29.026 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2024-11-04 12:33:29.027 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2024-11-04 12:33:29.027 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2024-11-04 12:33:29.027 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2024-11-04 12:33:29.027 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2024-11-04 12:33:29.027 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2024-11-04 12:33:29.029 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2024-11-04 12:33:29.030 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2024-11-04 12:33:29.031 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2024-11-04 12:33:29.032 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2024-11-04 12:33:29.033 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2024-11-04 12:33:29.034 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2024-11-04 12:33:29.035 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2024-11-04 12:33:29.036 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2024-11-04 12:33:29.036 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.036 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2024-11-04 12:33:29.036 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2024-11-04 12:33:29.036 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2024-11-04 12:33:29.037 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:29.037 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2024-11-04 12:33:29.037 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2024-11-04 12:33:29.037 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2024-11-04 12:33:29.037 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.037 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2024-11-04 12:33:29.037 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2024-11-04 12:33:29.038 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2024-11-04 12:33:29.039 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2024-11-04 12:33:29.040 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2024-11-04 12:33:29.040 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.040 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2024-11-04 12:33:29.040 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2024-11-04 12:33:29.040 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2024-11-04 12:33:29.041 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.041 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2024-11-04 12:33:29.041 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2024-11-04 12:33:29.041 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2024-11-04 12:33:29.041 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2024-11-04 12:33:29.041 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2024-11-04 12:33:29.041 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2024-11-04 12:33:29.042 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2024-11-04 12:33:29.043 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2024-11-04 12:33:29.044 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2024-11-04 12:33:29.045 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2024-11-04 12:33:29.046 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY/H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2024-11-04 12:33:29.047 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2024-11-04 12:33:29.048 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2024-11-04 12:33:29.048 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_10.46.128.11,H_10.46.128.12,H_10.46.128.24 2024-11-04 12:33:29.048 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2024-11-04 12:33:29.048 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2024-11-04 12:33:29.049 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2024-11-04 12:33:29.049 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2024-11-04 12:33:29.049 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2024-11-04 12:33:29.049 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2024-11-04 12:33:29.049 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2024-11-04 12:33:29.049 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2024-11-04 12:33:29.050 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_dba_test_ob-1730694047-PRIMARY 2024-11-04 12:33:29.050 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2024-11-04 12:33:29.050 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2024-11-04 12:33:29.050 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=129, updatedCount=129, failedToUpdateCount=0 2024-11-04 12:33:29.050 INFO 21062 --- [ocp-schedules-30,b56243d8841e4bd4,bd98730de4b0f75f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:33:29.177 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:29.179 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:29.204 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:29.205 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241106 2024-11-04 12:33:29.206 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:29.207 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:29.294 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:29.295 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:29.467 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:29.469 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:29.494 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:29.495 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241107 2024-11-04 12:33:29.496 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:29.497 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:29.752 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:29.754 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:29.829 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:29.830 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241108 2024-11-04 12:33:29.831 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:29.832 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6543b0bf7f62cdab,fc717b1744a61ac2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:30.095 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:30.096 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:30.120 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:30.121 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241109 2024-11-04 12:33:30.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:30.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:33:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:33:30.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:30.386 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:30.410 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:30.411 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241110 2024-11-04 12:33:30.412 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:30.413 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:30.538 WARN 21062 --- [ocp-async-9,9c7b737cb3d04e1f,f1915777288a0984] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:30.538 INFO 21062 --- [ocp-async-6,f2821d01cba31e79,d091fa80509fcb58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:30.675 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:30.677 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:30.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:30.704 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241111 2024-11-04 12:33:30.705 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:30.705 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:30.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:30.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:31.003 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:31.005 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20241112 2024-11-04 12:33:31.005 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.006 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:31.271 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.272 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2024-11-04 12:33:31.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2024-11-04 12:33:31.298 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.298 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:31.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 12:33:31.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241030 2024-11-04 12:33:31.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:31.556 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.558 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:31.583 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:31.584 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241031 2024-11-04 12:33:31.584 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.585 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:31.871 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:31.898 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:31.899 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241101 2024-11-04 12:33:31.899 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:31.900 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:32.174 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:32.176 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:32.201 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:32.202 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241102 2024-11-04 12:33:32.203 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:32.203 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:32.295 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:32.296 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:32.485 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:32.486 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:32.510 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:32.511 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241103 2024-11-04 12:33:32.512 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:32.513 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:32.539 WARN 21062 --- [ocp-async-3,98fdc3f714a1fb38,7f5dfadcdab1eb77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:32.539 INFO 21062 --- [ocp-async-5,b6899500032b96a4,a1313135fda44f15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:32.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:32.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:32.833 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:32.834 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241104 2024-11-04 12:33:32.835 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:32.836 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8a8108457f86f796,9684075451d92e20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:33.124 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:33.125 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:33.151 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:33.152 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241105 2024-11-04 12:33:33.152 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:33.153 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:33.450 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:33.451 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:33.476 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:33.477 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241106 2024-11-04 12:33:33.477 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:33.478 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:33.770 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:33.771 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:33.795 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:33.796 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241107 2024-11-04 12:33:33.796 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:33.797 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:34.086 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:34.087 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:34.113 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:34.114 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241108 2024-11-04 12:33:34.115 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:34.116 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:34.405 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:34.406 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:34.431 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:34.432 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241109 2024-11-04 12:33:34.433 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:34.434 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:34.539 WARN 21062 --- [ocp-async-7,90bc7a3fadb0e428,e43814b74234fe3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:34.539 INFO 21062 --- [ocp-async-2,367a2885c40c31d1,b507178b17c280b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:34.722 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:34.724 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:34.750 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:34.751 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241110 2024-11-04 12:33:34.752 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:34.752 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:35.043 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.044 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:35.069 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:35.070 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241111 2024-11-04 12:33:35.071 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.072 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:35.296 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:35.298 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:35.361 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.362 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:35.387 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:35.388 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20241112 2024-11-04 12:33:35.389 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.389 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:35.679 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.681 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2024-11-04 12:33:35.707 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2024-11-04 12:33:35.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:35.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.710 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 12:33:35.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241030 2024-11-04 12:33:35.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.727 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:35.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:35.980 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:35.981 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241031 2024-11-04 12:33:35.982 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:35.982 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b1f4fbe22d223c97,29b5b9b5b7da72d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:36.269 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:36.270 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:36.295 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:36.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241101 2024-11-04 12:33:36.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:36.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:36.539 WARN 21062 --- [ocp-async-8,007ca94c50c0e257,ded120b146ccb834] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:36.539 INFO 21062 --- [ocp-async-9,c56171926183802c,e9941700c1e28603] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:36.583 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:36.585 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:36.610 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:36.611 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241102 2024-11-04 12:33:36.612 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:36.612 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:36.899 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:36.901 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:36.925 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:36.926 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241103 2024-11-04 12:33:36.926 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:36.927 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:37.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:37.215 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:37.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:37.241 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241104 2024-11-04 12:33:37.242 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:37.243 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:37.528 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:37.530 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:37.556 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:37.557 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241105 2024-11-04 12:33:37.557 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:37.558 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:37.844 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:37.846 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:37.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:37.874 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241106 2024-11-04 12:33:37.875 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:37.876 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:38.161 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:38.163 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:38.188 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:38.189 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241107 2024-11-04 12:33:38.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:38.191 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:38.298 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:38.299 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:38.476 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:38.478 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:38.503 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:38.504 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241108 2024-11-04 12:33:38.505 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:38.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:38.507 INFO 21062 --- [ocp-schedules-21,e9009e9457361667,f6025aea307b9fca] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:33:38.507 INFO 21062 --- [ocp-schedules-16,a7079ef2dad9b30b,e2fbea06543c52c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:33:38.527 INFO 21062 --- [ocp-async-3,034448317e1eed5f,1bc2384665963e67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:33:38.539 WARN 21062 --- [ocp-async-5,600ef45704117031,24e8b2ba6e987830] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:38.539 INFO 21062 --- [ocp-async-7,0d58a154bd066bcd,9b92d005015b0328] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:38.542 INFO 21062 --- [ocp-schedules-9,c5cea52cb41743b8,b2bf41b70efe5d3a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:33:38.552 INFO 21062 --- [ocp-schedules-7,128a9e4980d8cd76,9c7df676825f7e91] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:33:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:33:38.588245+08:00, lastProcessTime=2024-11-04T12:33:28.673912+08:00 2024-11-04 12:33:38.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:33:38.588245+08:00 2024-11-04 12:33:38.790 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:38.792 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:38.818 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:38.819 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241109 2024-11-04 12:33:38.820 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:38.821 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:38.983 INFO 21062 --- [prometheus-minute-schedule-1,33559357785e1b0e,fe8c66c2e2020139] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:39.112 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:39.114 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:39.127 INFO 21062 --- [ocp-schedules-5,cd4fd95024cb4d6e,934e8a7df5ea650d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:33:39.155 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:39.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241110 2024-11-04 12:33:39.157 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:39.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:39.449 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:39.450 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:39.475 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:39.476 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241111 2024-11-04 12:33:39.476 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:39.477 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:39.767 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:39.768 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:39.795 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:39.796 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20241112 2024-11-04 12:33:39.797 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:39.798 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:33:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:33:40.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.094 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2024-11-04 12:33:40.119 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2024-11-04 12:33:40.120 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.121 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:40.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 12:33:40.183 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241028 2024-11-04 12:33:40.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:33:40.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.429 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:40.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:40.454 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241029 2024-11-04 12:33:40.454 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:33:40.540 WARN 21062 --- [ocp-async-8,bbea68779c2fb1e7,23b76e11abd16332] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:40.540 INFO 21062 --- [ocp-async-4,c331f350fc5e3c52,184897ef1aa69260] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:40.775 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.776 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:40.801 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:40.803 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241030 2024-11-04 12:33:40.803 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:40.804 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:41.132 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:41.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:41.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:41.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241031 2024-11-04 12:33:41.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:41.161 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:41.299 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:41.301 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:41.488 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:41.489 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:41.514 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:41.515 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241101 2024-11-04 12:33:41.515 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:41.516 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:41.837 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:41.839 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:41.864 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:41.865 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241102 2024-11-04 12:33:41.865 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:41.866 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a681682fced40f2a,4cd68fb23e5732bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:42.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:42.185 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:42.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:42.213 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241103 2024-11-04 12:33:42.213 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:42.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:42.540 WARN 21062 --- [ocp-async-9,14dbb1f0680935eb,da0609a9059e0663] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:42.540 INFO 21062 --- [ocp-async-10,1b249a78d8f1b2ac,6f0fc3d06f12bc2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:42.547 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:42.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:42.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:42.574 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241104 2024-11-04 12:33:42.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:42.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:42.916 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:42.918 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:42.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:42.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241105 2024-11-04 12:33:42.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:42.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:43.272 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:43.273 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:43.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:43.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241106 2024-11-04 12:33:43.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:43.300 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:43.625 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:43.626 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:43.652 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:43.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241107 2024-11-04 12:33:43.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:43.654 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:43.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:43.989 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:44.013 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:44.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241108 2024-11-04 12:33:44.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:44.015 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:44.301 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:44.302 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:44.347 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:44.348 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:44.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:44.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241109 2024-11-04 12:33:44.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:44.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:44.540 WARN 21062 --- [ocp-async-3,15ba0370cd29d0a5,96b2362478c0011f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:44.540 INFO 21062 --- [ocp-async-1,6a45d26a3fb8b09a,b24dbc5b5fc418c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:44.700 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:44.701 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:44.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:44.727 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241110 2024-11-04 12:33:44.727 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:44.728 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:44.983 INFO 21062 --- [prometheus-minute-schedule-1,09bd324ccafbada4,778e1c774ba2cfac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:45.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.060 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:45.086 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:45.087 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241111 2024-11-04 12:33:45.088 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.089 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:45.419 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.420 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:45.446 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:45.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20241112 2024-11-04 12:33:45.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.448 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:45.778 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2024-11-04 12:33:45.804 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2024-11-04 12:33:45.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:45.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:45.825 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241028 2024-11-04 12:33:45.825 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.826 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:33:45.994 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:45.996 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:46.020 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:46.021 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241029 2024-11-04 12:33:46.022 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.023 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:33:46.220 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.221 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:46.246 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:46.247 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241030 2024-11-04 12:33:46.248 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.248 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:46.445 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:46.475 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:46.476 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241031 2024-11-04 12:33:46.477 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.477 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:46.540 WARN 21062 --- [ocp-async-2,0cab1769e2f5e589,20076379087e8ee8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:46.540 INFO 21062 --- [ocp-async-8,234c534601f9e03b,378c906225042de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:46.674 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.675 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:46.700 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:46.701 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241101 2024-11-04 12:33:46.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:46.904 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.906 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:46.930 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:46.931 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241102 2024-11-04 12:33:46.931 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:46.932 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:47.129 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.131 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:47.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:47.157 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241103 2024-11-04 12:33:47.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:47.302 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:47.304 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:47.358 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:47.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:47.385 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241104 2024-11-04 12:33:47.386 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.387 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:47.589 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.590 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:47.615 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:47.616 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241105 2024-11-04 12:33:47.616 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.617 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:47.823 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.825 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:47.850 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:47.851 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241106 2024-11-04 12:33:47.852 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:47.852 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:47.983 INFO 21062 --- [prometheus-minute-schedule-1,745a4d7a16fe2407,5659ee0ab621e964] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:48.062 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.063 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:48.088 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:48.089 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241107 2024-11-04 12:33:48.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:48.294 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.295 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:48.319 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:48.320 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241108 2024-11-04 12:33:48.321 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.322 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:48.527 INFO 21062 --- [ocp-async-8,cd651660c25f0b97,0f032ecda679beb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:33:48.531 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.533 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:48.540 INFO 21062 --- [ocp-async-9,28895a9fcc06cdba,8c23da33e4181245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:48.540 WARN 21062 --- [ocp-async-6,866ce2a21c1a14c4,3f22218784d5f8be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:48.545 INFO 21062 --- [ocp-schedules-25,b53763d5812db517,e2cb940d504c4eef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:33:48.560 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:48.561 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241109 2024-11-04 12:33:48.562 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.563 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:33:48.588022+08:00, lastProcessTime=2024-11-04T12:33:38.588245+08:00 2024-11-04 12:33:48.605 INFO 21062 --- [ocp-schedules-11,bd0b66abe7b6913b,05a4499297584501] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:33:48.617 INFO 21062 --- [ocp-schedules-11,bd0b66abe7b6913b,05a4499297584501] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:33:48.617 INFO 21062 --- [ocp-schedules-11,bd0b66abe7b6913b,05a4499297584501] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:33:48.782 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:33:48.588022+08:00 2024-11-04 12:33:48.789 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.791 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:48.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:48.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241110 2024-11-04 12:33:48.817 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:48.818 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:49.032 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.033 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:49.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:49.060 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241111 2024-11-04 12:33:49.060 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:49.182 INFO 21062 --- [ocp-schedules-9,17793df6689f01f4,3b87944fffb3f944] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:33:49.274 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.275 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:49.300 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:49.302 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20241112 2024-11-04 12:33:49.302 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:49.516 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.517 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2024-11-04 12:33:49.545 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2024-11-04 12:33:49.547 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.547 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:49.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 12:33:49.555 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241005 2024-11-04 12:33:49.556 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.556 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241005 values less than (1728172800000000)) 2024-11-04 12:33:49.716 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.717 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:49.741 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:49.742 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241006 2024-11-04 12:33:49.743 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.744 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241006 values less than (1728259200000000)) 2024-11-04 12:33:49.927 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.928 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:49.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:49.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241007 2024-11-04 12:33:49.954 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:49.954 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241007 values less than (1728345600000000)) 2024-11-04 12:33:50.143 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.144 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:50.168 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:50.169 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241008 2024-11-04 12:33:50.170 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.170 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241008 values less than (1728432000000000)) 2024-11-04 12:33:50.304 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:50.306 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:50.354 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.356 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:50.382 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:50.383 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241009 2024-11-04 12:33:50.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241009 values less than (1728518400000000)) 2024-11-04 12:33:50.540 WARN 21062 --- [ocp-async-5,718c17947b61ec83,907d7e38632a2cf8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:50.541 INFO 21062 --- [ocp-async-3,15d8025545d4d673,8679da96e1f51490] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:50.570 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:50.596 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:50.597 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241010 2024-11-04 12:33:50.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241010 values less than (1728604800000000)) 2024-11-04 12:33:50.783 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.784 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:50.809 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:50.810 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241011 2024-11-04 12:33:50.810 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.811 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241011 values less than (1728691200000000)) 2024-11-04 12:33:50.983 INFO 21062 --- [prometheus-minute-schedule-1,5c7e999f0055ca40,3b99a9eb5c4fa814] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:50.997 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:50.998 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:51.022 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:51.024 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241012 2024-11-04 12:33:51.024 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.025 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241012 values less than (1728777600000000)) 2024-11-04 12:33:51.207 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.208 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:51.233 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:51.234 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241013 2024-11-04 12:33:51.235 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.236 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241013 values less than (1728864000000000)) 2024-11-04 12:33:51.426 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.428 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:51.452 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:51.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241014 2024-11-04 12:33:51.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.454 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241014 values less than (1728950400000000)) 2024-11-04 12:33:51.640 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.641 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:51.667 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:51.668 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241015 2024-11-04 12:33:51.668 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.669 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241015 values less than (1729036800000000)) 2024-11-04 12:33:51.854 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.856 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:51.881 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:51.882 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241016 2024-11-04 12:33:51.883 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:51.883 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241016 values less than (1729123200000000)) 2024-11-04 12:33:52.071 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.072 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:52.099 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:52.100 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241017 2024-11-04 12:33:52.101 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.101 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241017 values less than (1729209600000000)) 2024-11-04 12:33:52.291 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.293 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:52.319 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:52.320 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241018 2024-11-04 12:33:52.321 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.322 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241018 values less than (1729296000000000)) 2024-11-04 12:33:52.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.509 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:52.534 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:52.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241019 2024-11-04 12:33:52.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.536 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241019 values less than (1729382400000000)) 2024-11-04 12:33:52.541 INFO 21062 --- [ocp-async-3,b99f5bd3b00c3ecf,229a9daebb869a86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:52.541 WARN 21062 --- [ocp-async-7,a77df5befa8e10bd,1a67e28499c661ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:52.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.727 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:52.755 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:52.756 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241020 2024-11-04 12:33:52.757 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.758 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:33:52.948 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.949 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:52.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:52.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241021 2024-11-04 12:33:52.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:52.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:33:53.166 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:53.193 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:53.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241022 2024-11-04 12:33:53.195 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.195 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:33:53.306 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:53.308 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:53.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.386 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:53.412 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:53.413 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241023 2024-11-04 12:33:53.414 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.415 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:33:53.603 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.605 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:53.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:53.631 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241024 2024-11-04 12:33:53.631 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.632 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:33:53.820 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.822 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:53.847 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:53.848 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241025 2024-11-04 12:33:53.848 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:53.849 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:33:53.983 INFO 21062 --- [prometheus-minute-schedule-1,b19f329ff3aa9a12,75363d6f2f78e06e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:54.040 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.042 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:54.066 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:54.068 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241026 2024-11-04 12:33:54.068 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.069 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:33:54.257 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.259 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:54.285 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:54.286 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241027 2024-11-04 12:33:54.287 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.287 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:33:54.478 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.479 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:54.504 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:54.505 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241028 2024-11-04 12:33:54.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:33:54.541 WARN 21062 --- [ocp-async-10,78deb217da31682f,4ab1df36618424be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:54.541 INFO 21062 --- [ocp-async-8,a14863812b728649,fc8d72395ffe99c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:54.697 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.698 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:54.722 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:54.723 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241029 2024-11-04 12:33:54.723 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.724 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:33:54.923 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.924 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:54.949 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:54.950 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241030 2024-11-04 12:33:54.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:54.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:33:55.146 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.147 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:55.174 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:55.175 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241031 2024-11-04 12:33:55.176 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.176 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:33:55.364 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.366 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:55.391 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:55.392 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241101 2024-11-04 12:33:55.393 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.394 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:33:55.586 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.587 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:55.613 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:55.614 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241102 2024-11-04 12:33:55.614 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.615 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:33:55.810 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.811 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:55.838 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:55.839 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241103 2024-11-04 12:33:55.839 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:55.840 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:33:56.032 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.033 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:56.057 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:56.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241104 2024-11-04 12:33:56.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.060 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:33:56.253 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.255 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:56.281 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:56.282 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241105 2024-11-04 12:33:56.283 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.284 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:33:56.308 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:56.309 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:56.474 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.475 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:56.500 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:56.501 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241106 2024-11-04 12:33:56.502 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.502 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:33:56.541 WARN 21062 --- [ocp-async-9,10ac6b4f490cf665,b22032fd39b79760] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:56.541 INFO 21062 --- [ocp-async-4,72301d9904e605bf,516229286ba98658] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:56.690 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.692 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:56.716 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:56.717 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241107 2024-11-04 12:33:56.718 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.719 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:33:56.911 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.913 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:56.940 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:56.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241108 2024-11-04 12:33:56.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:56.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:33:56.983 INFO 21062 --- [prometheus-minute-schedule-1,99d2a9769d3756bb,4f571a15fa8b7b9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:33:57.133 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:57.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:57.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241109 2024-11-04 12:33:57.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.161 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:33:57.352 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.354 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:57.378 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:57.379 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241110 2024-11-04 12:33:57.379 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.380 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:33:57.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:57.597 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:57.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241111 2024-11-04 12:33:57.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:33:57.793 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.795 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:57.819 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:57.820 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20241112 2024-11-04 12:33:57.821 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:57.821 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:33:58.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.016 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2024-11-04 12:33:58.041 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2024-11-04 12:33:58.043 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.043 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:33:58.044 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.044 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 12:33:58.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241005 2024-11-04 12:33:58.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.066 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241005 values less than (1728172800000000)) 2024-11-04 12:33:58.271 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.272 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:33:58.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:33:58.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241006 2024-11-04 12:33:58.298 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241006 values less than (1728259200000000)) 2024-11-04 12:33:58.500 INFO 21062 --- [ocp-schedules-1,9befe84141b3ad1a,c752f8a83fd8ba22] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:33:58.500 INFO 21062 --- [ocp-schedules-5,0a5921a6687b0b64,74a6b9437318ba38] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:33:58.502 INFO 21062 --- [ocp-schedules-16,a277a65b38d108bf,5105c134087392cd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:33:58.507 INFO 21062 --- [ocp-schedules-1,9befe84141b3ad1a,c752f8a83fd8ba22] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:58.507 INFO 21062 --- [ocp-schedules-5,0a5921a6687b0b64,74a6b9437318ba38] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:58.508 INFO 21062 --- [ocp-schedules-2,719e3479b6d53e14,5f9cd1d2e0a5ce9a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:33:58.508 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:33:58.527 INFO 21062 --- [ocp-async-1,8cc90053779ec60b,75e38616f470f613] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:33:58.541 WARN 21062 --- [ocp-async-7,2bf4ad644a3fd2f9,978ff07160f98c1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:33:58.541 INFO 21062 --- [ocp-async-6,71cec7586d9247b7,185a8f5f832789bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:33:58.548 INFO 21062 --- [ocp-schedules-22,046a914c28be00cc,c2a2512ce4bcb69d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:33:58.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:33:58.562 INFO 21062 --- [ocp-schedules-26,ec10d8383060154c,c4aed297b2122c02] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:33:58.574 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:33:58.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241007 2024-11-04 12:33:58.576 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.576 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241007 values less than (1728345600000000)) 2024-11-04 12:33:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:33:58.587762+08:00, lastProcessTime=2024-11-04T12:33:48.588022+08:00 2024-11-04 12:33:58.666 INFO 21062 --- [ocp-schedules-10,57902c2903176d62,c2e6eee3abab4129] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:33:58.666 INFO 21062 --- [ocp-schedules-10,c2f9b10a2d8d394d,8566d052013de810] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:33:58.669 INFO 21062 --- [ocp-schedules-10,c2f9b10a2d8d394d,8566d052013de810] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:33:58.700 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:33:58.700 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:33:58.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:33:58.587762+08:00 2024-11-04 12:33:58.827 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.828 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:33:58.853 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:33:58.854 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241008 2024-11-04 12:33:58.854 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:58.855 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241008 values less than (1728432000000000)) 2024-11-04 12:33:59.107 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.109 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:33:59.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:33:59.135 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241009 2024-11-04 12:33:59.135 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.136 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241009 values less than (1728518400000000)) 2024-11-04 12:33:59.222 INFO 21062 --- [ocp-schedules-2,819b02a2835df239,1c4d3ef18343e7c0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:33:59.309 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:33:59.311 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:33:59.380 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.382 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:33:59.407 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:33:59.408 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241010 2024-11-04 12:33:59.409 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.410 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241010 values less than (1728604800000000)) 2024-11-04 12:33:59.658 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.660 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:33:59.684 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:33:59.685 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241011 2024-11-04 12:33:59.686 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.687 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241011 values less than (1728691200000000)) 2024-11-04 12:33:59.935 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.937 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:33:59.961 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:33:59.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241012 2024-11-04 12:33:59.963 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:33:59.963 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241012 values less than (1728777600000000)) 2024-11-04 12:33:59.983 INFO 21062 --- [prometheus-minute-schedule-1,035ff65f6df51659,ce91cd1e4bac796f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694840, interval=60 2024-11-04 12:34:00.216 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:00.218 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:00.243 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:00.244 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241013 2024-11-04 12:34:00.245 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:00.246 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241013 values less than (1728864000000000)) 2024-11-04 12:34:00.500 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:00.501 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:00.525 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:00.526 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241014 2024-11-04 12:34:00.526 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:00.527 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241014 values less than (1728950400000000)) 2024-11-04 12:34:00.541 WARN 21062 --- [ocp-async-10,76286abdaa28b236,49ef219dfe53b5ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:00.542 INFO 21062 --- [ocp-async-3,ab1fd7b265b459c0,236996ae0496d6fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:00.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:00.781 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:00.804 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:00.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241015 2024-11-04 12:34:00.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:00.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241015 values less than (1729036800000000)) 2024-11-04 12:34:01.062 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.063 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:01.089 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:01.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241016 2024-11-04 12:34:01.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.091 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241016 values less than (1729123200000000)) 2024-11-04 12:34:01.346 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.347 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:01.372 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:01.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241017 2024-11-04 12:34:01.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241017 values less than (1729209600000000)) 2024-11-04 12:34:01.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:01.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:01.654 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241018 2024-11-04 12:34:01.655 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.656 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241018 values less than (1729296000000000)) 2024-11-04 12:34:01.911 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.912 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:01.937 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:01.938 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241019 2024-11-04 12:34:01.938 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:01.939 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241019 values less than (1729382400000000)) 2024-11-04 12:34:02.185 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:02.186 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:02.210 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:02.211 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241020 2024-11-04 12:34:02.211 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:02.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:34:02.311 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:02.312 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:02.472 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:02.474 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:02.497 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:02.498 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241021 2024-11-04 12:34:02.499 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:02.499 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:34:02.542 WARN 21062 --- [ocp-async-5,7584a73f6d98f6d6,e9dedf4c460c1ac8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:02.542 INFO 21062 --- [ocp-async-5,e59a379a8c3d2939,f6c999ef77674bd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:02.757 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:02.758 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:02.783 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:02.784 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241022 2024-11-04 12:34:02.784 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:02.785 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:34:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7c018149d98454de,fd5f6b7065601814] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:03.056 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.057 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:03.082 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:03.083 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241023 2024-11-04 12:34:03.084 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.084 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:34:03.339 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:03.369 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:03.370 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241024 2024-11-04 12:34:03.370 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.371 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:34:03.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:03.652 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:03.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241025 2024-11-04 12:34:03.654 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.654 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:34:03.913 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.914 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:03.939 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:03.940 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241026 2024-11-04 12:34:03.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:03.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:34:04.192 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:04.193 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:04.217 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:04.218 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241027 2024-11-04 12:34:04.219 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:04.219 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:34:04.479 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:04.480 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:04.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:04.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241028 2024-11-04 12:34:04.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:04.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:34:04.542 WARN 21062 --- [ocp-async-1,e99dbe3a604ee303,54174f3d36767c84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:04.542 INFO 21062 --- [ocp-async-7,c88561887f602c63,ba711b54a94aaa07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:04.765 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:04.767 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:04.791 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:04.792 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241029 2024-11-04 12:34:04.792 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:04.793 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:34:05.058 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:05.084 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:05.085 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241030 2024-11-04 12:34:05.085 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.086 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:34:05.312 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:05.314 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:05.345 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.348 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:05.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:05.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241031 2024-11-04 12:34:05.376 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.377 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:34:05.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.639 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:05.664 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:05.665 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241101 2024-11-04 12:34:05.666 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.666 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:34:05.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:05.967 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:05.968 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241102 2024-11-04 12:34:05.969 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:05.969 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:34:05.983 INFO 21062 --- [prometheus-minute-schedule-1,536b3a22fe02d449,f43a9dad5ae5ece3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:06.229 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:06.230 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:06.256 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:06.257 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241103 2024-11-04 12:34:06.257 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:06.258 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:34:06.521 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:06.522 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:06.542 WARN 21062 --- [ocp-async-10,0a0a4cd30e938dd6,72a8c79ecafba9bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:06.542 INFO 21062 --- [ocp-async-6,f7ce30aea487a095,7b6b37f87aa69f18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:06.547 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:06.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241104 2024-11-04 12:34:06.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:06.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:34:06.811 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:06.813 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:06.837 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:06.838 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241105 2024-11-04 12:34:06.838 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:06.839 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:34:07.104 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:07.106 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:07.129 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:07.130 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241106 2024-11-04 12:34:07.131 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:07.131 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:34:07.393 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:07.395 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:07.418 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:07.419 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241107 2024-11-04 12:34:07.420 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:07.421 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:34:07.683 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:07.685 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:07.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:07.710 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241108 2024-11-04 12:34:07.711 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:07.711 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:34:07.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:07.977 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:08.003 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:08.005 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241109 2024-11-04 12:34:08.005 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:08.006 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:34:08.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:08.278 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:08.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:08.304 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241110 2024-11-04 12:34:08.305 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:08.306 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:34:08.314 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:08.315 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:08.508 INFO 21062 --- [ocp-schedules-5,0a5921a6687b0b64,74a6b9437318ba38] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:34:08.508 INFO 21062 --- [ocp-schedules-1,9befe84141b3ad1a,c752f8a83fd8ba22] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:34:08.527 INFO 21062 --- [ocp-async-3,2b4fc45532a1a556,43eb5e9b4c056e29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:34:08.542 WARN 21062 --- [ocp-async-5,c87685f869cb2da4,f6f944c65fe76202] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:08.542 INFO 21062 --- [ocp-async-9,d4b55e1dbe6599c7,2397fe8e4fdacabe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:08.552 INFO 21062 --- [ocp-schedules-27,6e38e918c554b2af,f508ccd178d31b4a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:34:08.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:08.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:34:08.589050+08:00, lastProcessTime=2024-11-04T12:33:58.587762+08:00 2024-11-04 12:34:08.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:08.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241111 2024-11-04 12:34:08.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:08.600 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:34:08.619 INFO 21062 --- [ocp-schedules-20,270b6e1b0df38feb,450ed0b956097fec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:34:08.625 INFO 21062 --- [ocp-schedules-20,270b6e1b0df38feb,450ed0b956097fec] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:34:08.625 INFO 21062 --- [ocp-schedules-20,270b6e1b0df38feb,450ed0b956097fec] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:34:08.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:34:08.589050+08:00 2024-11-04 12:34:08.860 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:08.861 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:08.885 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:08.886 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20241112 2024-11-04 12:34:08.886 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:08.887 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:34:08.983 INFO 21062 --- [prometheus-minute-schedule-1,1d10a3b16f266891,0674e5a94c82082b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:09.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2024-11-04 12:34:09.182 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2024-11-04 12:34:09.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:09.185 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.185 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 12:34:09.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241027 2024-11-04 12:34:09.213 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.213 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:34:09.260 INFO 21062 --- [ocp-schedules-23,33cfe448ebd80573,6cbd62fd6917a335] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:34:09.357 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:09.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:09.385 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241028 2024-11-04 12:34:09.386 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.387 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:34:09.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.537 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:09.563 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:09.564 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241029 2024-11-04 12:34:09.564 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.565 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:34:09.713 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.714 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:09.739 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:09.740 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241030 2024-11-04 12:34:09.741 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.742 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:34:09.890 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.891 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:09.915 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:09.916 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241031 2024-11-04 12:34:09.917 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:09.918 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:34:10.002 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:34:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:34:10.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.067 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:10.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:10.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241101 2024-11-04 12:34:10.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.094 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:34:10.244 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.245 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:10.270 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:10.271 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241102 2024-11-04 12:34:10.272 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.272 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:34:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:34:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:34:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:34:10.329 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:34:10.329 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:34:10.331 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 5. 2024-11-04 12:34:10.421 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.423 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:10.450 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:10.451 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241103 2024-11-04 12:34:10.452 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:34:10.542 WARN 21062 --- [ocp-async-1,3600ce8b17e12e89,b8168a2b146ecb1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:10.543 INFO 21062 --- [ocp-async-7,0bcac7599a88381a,c4d5de809f1508c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:10.601 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.602 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:10.626 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:10.627 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241104 2024-11-04 12:34:10.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:34:10.777 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:10.804 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:10.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241105 2024-11-04 12:34:10.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:34:10.959 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.960 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:10.984 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:10.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241106 2024-11-04 12:34:10.986 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:10.986 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:34:11.138 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.140 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:11.164 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:11.165 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241107 2024-11-04 12:34:11.166 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:34:11.315 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:11.316 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.317 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:11.318 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:11.342 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:11.343 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241108 2024-11-04 12:34:11.343 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.344 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:34:11.493 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.494 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:11.518 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:11.519 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241109 2024-11-04 12:34:11.519 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.520 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:34:11.668 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.670 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:11.693 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:11.694 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241110 2024-11-04 12:34:11.695 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.696 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:34:11.846 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.847 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:11.872 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:11.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241111 2024-11-04 12:34:11.874 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:11.875 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:34:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c1888b4a70bee55f,123699d789f14bc0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:12.026 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.027 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:12.052 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:12.053 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20241112 2024-11-04 12:34:12.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.055 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:34:12.195 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.196 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2024-11-04 12:34:12.220 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2024-11-04 12:34:12.222 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.223 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:12.223 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.224 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 12:34:12.253 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202408 2024-11-04 12:34:12.254 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.254 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:12.399 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.401 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2024-11-04 12:34:12.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2024-11-04 12:34:12.426 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202409 2024-11-04 12:34:12.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:12.543 WARN 21062 --- [ocp-async-8,512721dcfb1dda65,654dbd2bb46d5423] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:12.543 INFO 21062 --- [ocp-async-6,4c4e9a8494bbe840,f42ce2f8584e8b57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:12.576 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.577 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2024-11-04 12:34:12.600 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2024-11-04 12:34:12.601 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202410 2024-11-04 12:34:12.601 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.602 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:12.749 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.751 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2024-11-04 12:34:12.776 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2024-11-04 12:34:12.777 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202411 2024-11-04 12:34:12.777 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.778 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:12.926 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.927 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2024-11-04 12:34:12.950 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2024-11-04 12:34:12.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:12.954 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.954 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 12:34:12.978 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202408 2024-11-04 12:34:12.978 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:12.979 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:13.126 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.127 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2024-11-04 12:34:13.152 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2024-11-04 12:34:13.153 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202409 2024-11-04 12:34:13.154 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.155 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:13.301 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2024-11-04 12:34:13.327 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2024-11-04 12:34:13.328 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202410 2024-11-04 12:34:13.328 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.329 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:13.480 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.481 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2024-11-04 12:34:13.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2024-11-04 12:34:13.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202411 2024-11-04 12:34:13.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:13.659 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.661 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2024-11-04 12:34:13.687 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2024-11-04 12:34:13.689 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.689 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:13.690 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.691 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 12:34:13.705 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202408 2024-11-04 12:34:13.706 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.706 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:13.851 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.852 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2024-11-04 12:34:13.877 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2024-11-04 12:34:13.878 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202409 2024-11-04 12:34:13.879 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:13.880 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:13.985 INFO 21062 --- [prometheus-cache-monitor1,d851ad49d76c1db3,c7f3d58d0aff4c5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2024-11-04 12:34:13.985 INFO 21062 --- [prometheus-cache-monitor1,d851ad49d76c1db3,c7f3d58d0aff4c5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2024-11-04 12:34:14.028 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.029 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2024-11-04 12:34:14.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2024-11-04 12:34:14.055 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202410 2024-11-04 12:34:14.055 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.056 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:14.192 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2024-11-04 12:34:14.218 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2024-11-04 12:34:14.219 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202411 2024-11-04 12:34:14.220 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.220 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:14.317 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:14.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:14.366 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.367 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2024-11-04 12:34:14.391 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2024-11-04 12:34:14.392 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.393 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:14.393 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.394 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 12:34:14.424 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202408 2024-11-04 12:34:14.424 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:14.543 WARN 21062 --- [ocp-async-3,7910487ccee192cb,97398dce1c8879fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:14.543 INFO 21062 --- [ocp-async-5,2a7d54595dab728f,2b747754072d0140] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:14.565 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.567 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2024-11-04 12:34:14.590 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2024-11-04 12:34:14.591 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202409 2024-11-04 12:34:14.591 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.592 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:14.740 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.741 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2024-11-04 12:34:14.764 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2024-11-04 12:34:14.766 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202410 2024-11-04 12:34:14.766 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.767 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:14.916 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.918 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2024-11-04 12:34:14.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2024-11-04 12:34:14.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202411 2024-11-04 12:34:14.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:14.944 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a9c43106d2545b88,1283592ee97af924] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:15.094 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.096 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2024-11-04 12:34:15.119 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2024-11-04 12:34:15.121 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:15.123 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.123 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 12:34:15.140 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202408 2024-11-04 12:34:15.141 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.142 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:15.284 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.285 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2024-11-04 12:34:15.309 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2024-11-04 12:34:15.310 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202409 2024-11-04 12:34:15.311 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.312 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:15.463 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.464 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2024-11-04 12:34:15.489 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2024-11-04 12:34:15.490 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202410 2024-11-04 12:34:15.491 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.491 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:15.640 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.641 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2024-11-04 12:34:15.665 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2024-11-04 12:34:15.666 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202411 2024-11-04 12:34:15.667 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.668 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:15.818 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.820 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2024-11-04 12:34:15.846 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2024-11-04 12:34:15.848 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.849 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:15.849 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.850 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 12:34:15.875 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202408 2024-11-04 12:34:15.876 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:15.877 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:16.018 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.019 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2024-11-04 12:34:16.043 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2024-11-04 12:34:16.044 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202409 2024-11-04 12:34:16.045 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.046 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:16.185 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.186 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2024-11-04 12:34:16.211 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2024-11-04 12:34:16.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202410 2024-11-04 12:34:16.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.213 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:16.363 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.364 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2024-11-04 12:34:16.389 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2024-11-04 12:34:16.390 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202411 2024-11-04 12:34:16.390 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.391 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:16.540 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.542 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2024-11-04 12:34:16.543 WARN 21062 --- [ocp-async-9,98929e142df5e531,9c87f22558854326] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:16.545 INFO 21062 --- [ocp-async-2,528fd3e2825b0ca1,7e662487c0fccb9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:16.566 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2024-11-04 12:34:16.567 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:16.569 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.569 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 12:34:16.593 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202408 2024-11-04 12:34:16.594 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.594 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:16.735 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.737 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2024-11-04 12:34:16.761 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2024-11-04 12:34:16.762 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202409 2024-11-04 12:34:16.763 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.764 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:16.911 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.912 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2024-11-04 12:34:16.938 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2024-11-04 12:34:16.939 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202410 2024-11-04 12:34:16.939 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:16.940 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:17.091 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2024-11-04 12:34:17.116 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2024-11-04 12:34:17.117 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202411 2024-11-04 12:34:17.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:17.269 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.270 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2024-11-04 12:34:17.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2024-11-04 12:34:17.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.298 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:17.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 12:34:17.310 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202408 2024-11-04 12:34:17.310 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.311 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:17.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:17.319 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:17.454 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2024-11-04 12:34:17.479 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2024-11-04 12:34:17.480 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202409 2024-11-04 12:34:17.481 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.482 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:17.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2024-11-04 12:34:17.654 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2024-11-04 12:34:17.656 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202410 2024-11-04 12:34:17.656 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.657 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:17.804 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2024-11-04 12:34:17.831 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2024-11-04 12:34:17.832 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202411 2024-11-04 12:34:17.833 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.833 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:17.983 INFO 21062 --- [prometheus-minute-schedule-1,b698d2437b1efb55,d399302867dd9d0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:17.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:17.986 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2024-11-04 12:34:18.011 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2024-11-04 12:34:18.012 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.013 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:18.013 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 12:34:18.031 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202408 2024-11-04 12:34:18.032 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.032 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:18.164 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.165 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2024-11-04 12:34:18.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2024-11-04 12:34:18.191 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202409 2024-11-04 12:34:18.191 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.192 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:18.340 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2024-11-04 12:34:18.366 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2024-11-04 12:34:18.367 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202410 2024-11-04 12:34:18.367 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.368 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:18.521 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.523 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2024-11-04 12:34:18.527 INFO 21062 --- [ocp-async-7,fa98236a13b6c818,7f44cb94eeb1f9f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:34:18.543 WARN 21062 --- [ocp-async-10,a8ab6625f901a723,444484f2c2389eb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:18.545 INFO 21062 --- [ocp-async-6,dda5903a1f46b10b,e86f67313ede8c61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:18.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2024-11-04 12:34:18.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202411 2024-11-04 12:34:18.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:18.555 INFO 21062 --- [ocp-schedules-24,77bdb509c20008cd,2f323e3aa6202140] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:34:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:34:18.588122+08:00, lastProcessTime=2024-11-04T12:34:08.589050+08:00 2024-11-04 12:34:18.698 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.699 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2024-11-04 12:34:18.722 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2024-11-04 12:34:18.724 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:18.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 12:34:18.736 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202408 2024-11-04 12:34:18.736 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.737 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:34:18.588122+08:00 2024-11-04 12:34:18.881 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.882 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2024-11-04 12:34:18.908 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2024-11-04 12:34:18.909 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202409 2024-11-04 12:34:18.909 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:18.910 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:19.060 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2024-11-04 12:34:19.084 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2024-11-04 12:34:19.085 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202410 2024-11-04 12:34:19.086 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.086 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:19.235 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.237 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2024-11-04 12:34:19.261 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2024-11-04 12:34:19.262 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202411 2024-11-04 12:34:19.263 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.263 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:19.345 INFO 21062 --- [ocp-schedules-1,4c9557eab3a2172a,39ecec80d18bdc23] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:34:19.411 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.413 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2024-11-04 12:34:19.442 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2024-11-04 12:34:19.444 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.444 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:19.445 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.446 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 12:34:19.468 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202408 2024-11-04 12:34:19.469 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.469 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:19.609 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.610 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2024-11-04 12:34:19.634 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2024-11-04 12:34:19.634 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202409 2024-11-04 12:34:19.635 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.636 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:19.785 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.786 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2024-11-04 12:34:19.810 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2024-11-04 12:34:19.811 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202410 2024-11-04 12:34:19.812 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.812 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:19.960 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.961 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2024-11-04 12:34:19.987 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2024-11-04 12:34:19.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202411 2024-11-04 12:34:19.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:19.989 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:20.140 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.142 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2024-11-04 12:34:20.166 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2024-11-04 12:34:20.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:20.168 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.168 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 12:34:20.186 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202408 2024-11-04 12:34:20.186 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.187 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:20.320 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:20.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:20.332 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.334 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2024-11-04 12:34:20.358 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2024-11-04 12:34:20.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202409 2024-11-04 12:34:20.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.360 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:20.510 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.511 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2024-11-04 12:34:20.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2024-11-04 12:34:20.536 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202410 2024-11-04 12:34:20.537 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.537 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:20.543 WARN 21062 --- [ocp-async-3,7adbb0e66115ad2f,44d3465495459318] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:20.546 INFO 21062 --- [ocp-async-5,adba5eedf03a61a1,97bd4e677f65af13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:20.686 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.687 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2024-11-04 12:34:20.711 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2024-11-04 12:34:20.712 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202411 2024-11-04 12:34:20.713 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.713 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:20.861 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.863 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2024-11-04 12:34:20.888 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2024-11-04 12:34:20.889 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.889 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:20.890 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.891 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 12:34:20.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202408 2024-11-04 12:34:20.903 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:20.904 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a6cb27089f43b280,295b45e2f2b6ba01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:21.046 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.048 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2024-11-04 12:34:21.072 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2024-11-04 12:34:21.073 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202409 2024-11-04 12:34:21.074 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.075 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:21.224 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.225 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2024-11-04 12:34:21.249 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2024-11-04 12:34:21.250 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202410 2024-11-04 12:34:21.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.252 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:21.399 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.400 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2024-11-04 12:34:21.424 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2024-11-04 12:34:21.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202411 2024-11-04 12:34:21.426 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:21.571 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2024-11-04 12:34:21.597 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2024-11-04 12:34:21.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:21.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 12:34:21.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202408 2024-11-04 12:34:21.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:21.776 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.777 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2024-11-04 12:34:21.803 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2024-11-04 12:34:21.804 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202409 2024-11-04 12:34:21.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:21.958 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.959 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2024-11-04 12:34:21.983 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2024-11-04 12:34:21.984 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202410 2024-11-04 12:34:21.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:21.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:22.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.136 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2024-11-04 12:34:22.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2024-11-04 12:34:22.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202411 2024-11-04 12:34:22.161 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.161 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:22.314 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.315 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2024-11-04 12:34:22.339 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2024-11-04 12:34:22.340 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:22.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.342 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 12:34:22.357 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202408 2024-11-04 12:34:22.358 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.358 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:22.502 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.503 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2024-11-04 12:34:22.527 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2024-11-04 12:34:22.528 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202409 2024-11-04 12:34:22.529 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.529 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:22.544 WARN 21062 --- [ocp-async-9,b92bd7fd1b37f9b6,8815b3eaf984b551] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:22.546 INFO 21062 --- [ocp-async-2,ea7a6783af2038b7,af322865107235d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:22.677 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2024-11-04 12:34:22.701 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2024-11-04 12:34:22.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202410 2024-11-04 12:34:22.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:22.853 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.854 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2024-11-04 12:34:22.881 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2024-11-04 12:34:22.883 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202411 2024-11-04 12:34:22.883 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:22.884 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:23.033 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.034 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2024-11-04 12:34:23.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2024-11-04 12:34:23.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:23.062 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.062 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 12:34:23.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202408 2024-11-04 12:34:23.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:23.238 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:23.263 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2024-11-04 12:34:23.264 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202409 2024-11-04 12:34:23.265 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.266 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:23.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:23.322 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:23.413 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.415 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:23.439 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2024-11-04 12:34:23.440 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202410 2024-11-04 12:34:23.441 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.441 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:23.588 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.591 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:23.615 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2024-11-04 12:34:23.616 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202411 2024-11-04 12:34:23.617 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.617 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:23.765 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.767 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:23.791 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2024-11-04 12:34:23.793 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.794 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:23.794 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.795 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 12:34:23.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202408 2024-11-04 12:34:23.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:23.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.955 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2024-11-04 12:34:23.979 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2024-11-04 12:34:23.980 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202409 2024-11-04 12:34:23.981 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:23.981 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6ca87e9e07f34e56,4095ea96f1d71633] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:24.129 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.130 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2024-11-04 12:34:24.154 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2024-11-04 12:34:24.155 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202410 2024-11-04 12:34:24.155 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:24.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.304 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2024-11-04 12:34:24.328 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2024-11-04 12:34:24.329 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202411 2024-11-04 12:34:24.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:24.480 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.481 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2024-11-04 12:34:24.504 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2024-11-04 12:34:24.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:24.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 12:34:24.532 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202408 2024-11-04 12:34:24.532 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.533 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202408 values less than (1725148800000000)) 2024-11-04 12:34:24.544 WARN 21062 --- [ocp-async-10,1730a24f3f105862,970e6bb1982818f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:24.546 INFO 21062 --- [ocp-async-8,a196371402157e6e,5684f703fb9e0ee2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:24.675 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.676 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2024-11-04 12:34:24.700 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2024-11-04 12:34:24.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202409 2024-11-04 12:34:24.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202409 values less than (1727740800000000)) 2024-11-04 12:34:24.854 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.855 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2024-11-04 12:34:24.879 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2024-11-04 12:34:24.880 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202410 2024-11-04 12:34:24.881 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:24.881 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202410 values less than (1730419200000000)) 2024-11-04 12:34:25.034 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.036 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2024-11-04 12:34:25.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2024-11-04 12:34:25.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202411 2024-11-04 12:34:25.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.062 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202411 values less than (1733011200000000)) 2024-11-04 12:34:25.209 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.211 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2024-11-04 12:34:25.235 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2024-11-04 12:34:25.237 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.237 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:25.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.241 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 12:34:25.282 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241020 2024-11-04 12:34:25.283 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.284 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241020 values less than (1729468800000000)) 2024-11-04 12:34:25.446 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:25.476 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:25.477 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241021 2024-11-04 12:34:25.478 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.479 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241021 values less than (1729555200000000)) 2024-11-04 12:34:25.664 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.665 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:25.691 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:25.692 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241022 2024-11-04 12:34:25.692 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.693 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241022 values less than (1729641600000000)) 2024-11-04 12:34:25.881 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.883 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:25.906 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:25.907 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241023 2024-11-04 12:34:25.908 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:25.908 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241023 values less than (1729728000000000)) 2024-11-04 12:34:26.095 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.097 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:26.120 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:26.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241024 2024-11-04 12:34:26.122 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.123 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241024 values less than (1729814400000000)) 2024-11-04 12:34:26.309 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.311 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:26.322 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:26.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:26.335 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:26.336 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241025 2024-11-04 12:34:26.336 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.337 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241025 values less than (1729900800000000)) 2024-11-04 12:34:26.524 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.525 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:26.546 WARN 21062 --- [ocp-async-3,f0ed9a1249aaf6d7,4f8f2e40eac6e2e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:26.546 INFO 21062 --- [ocp-async-4,2885b1ef62b75cc7,44d9342b938842bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:26.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:26.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241026 2024-11-04 12:34:26.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.552 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241026 values less than (1729987200000000)) 2024-11-04 12:34:26.736 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.737 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:26.761 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:26.762 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241027 2024-11-04 12:34:26.763 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.763 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241027 values less than (1730073600000000)) 2024-11-04 12:34:26.949 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:26.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:26.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241028 2024-11-04 12:34:26.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:26.977 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:34:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3ff3fb0012ba4b3e,64611a1d294af77d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e47414515c544f31,20476b422b64ac23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:34:27.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.168 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:27.192 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:27.193 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241029 2024-11-04 12:34:27.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:34:27.383 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.385 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:27.409 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:27.411 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241030 2024-11-04 12:34:27.411 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.412 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:34:27.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.600 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:27.625 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:27.626 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241031 2024-11-04 12:34:27.626 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.627 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:34:27.812 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:27.840 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:27.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241101 2024-11-04 12:34:27.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:27.842 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:34:28.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.067 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:28.090 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:28.091 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241102 2024-11-04 12:34:28.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:34:28.283 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.284 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:28.309 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:28.310 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241103 2024-11-04 12:34:28.311 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.311 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:34:28.498 INFO 21062 --- [ocp-schedules-13,58c80c4fbc11798f,5362606cd98cb9da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:34:28.500 INFO 21062 --- [ocp-schedules-9,fb98ddbfb6798959,7b7961ab687fecd9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:34:28.500 INFO 21062 --- [ocp-schedules-15,494220cea07880f1,ef46fc5c9b3c2d34] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:34:28.500 INFO 21062 --- [ocp-schedules-22,10face1b0ab920c6,03bead24b350ad0a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:34:28.502 INFO 21062 --- [ocp-schedules-13,58c80c4fbc11798f,5362606cd98cb9da] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:34:28.502 INFO 21062 --- [ocp-schedules-12,b595111d1cc07a9b,6f3b72bf6150344e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:34:28.503 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.504 INFO 21062 --- [ocp-schedules-9,fb98ddbfb6798959,7b7961ab687fecd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:28.504 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:34:28.505 INFO 21062 --- [ocp-schedules-22,10face1b0ab920c6,03bead24b350ad0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:28.505 INFO 21062 --- [ocp-schedules-14,1b7ae1155084bb79,4e8b009d55c2833a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:34:28.508 INFO 21062 --- [ocp-schedules-15,494220cea07880f1,ef46fc5c9b3c2d34] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:34:28.516 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:34:28.516 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:34:28.526 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:34:28.526 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:34:28.526 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:34:28.526 INFO 21062 --- [ocp-schedules-7,a5de5761417084c6,abc49f3376267e26] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T12:34:28.526+08:00, tenantSize=0 2024-11-04 12:34:28.528 INFO 21062 --- [ocp-async-1,707bb41a604a107f,049f19a1b22b5742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:34:28.530 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:28.531 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241104 2024-11-04 12:34:28.532 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.533 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:34:28.537 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:34:28.537 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:34:28.537 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:34:28.541 INFO 21062 --- [ocp-schedules-2,f1b8d906d99ef025,acad590bb00551e1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:34:28.541 INFO 21062 --- [ocp-schedules-16,c620734539886225,166fed1c2f0822d4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:34:28.542 INFO 21062 --- [ocp-schedules-29,80189401dca00f33,2677c49b0a3b7cb3] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:34:28.544 INFO 21062 --- [ocp-schedules-2,f1b8d906d99ef025,acad590bb00551e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:28.545 INFO 21062 --- [ocp-schedules-17,ac1a75515a41db95,4659a33a9af834d6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:34:28.546 INFO 21062 --- [ocp-schedules-25,ca28cfeeca0c728a,5dc10cde76050dd4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:34:28.546 INFO 21062 --- [ocp-schedules-30,f831c7abe37cb609,342a715684749797] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:34:28.546 WARN 21062 --- [ocp-async-10,40ec5813ca0277d3,511a53ba7cf691f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:28.546 INFO 21062 --- [ocp-async-7,e1efd36a278475d2,5f8bef55b931419a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:28.546 INFO 21062 --- [ocp-schedules-25,ca28cfeeca0c728a,5dc10cde76050dd4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:34:28.546 INFO 21062 --- [ocp-schedules-17,ac1a75515a41db95,4659a33a9af834d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:34:28.546 INFO 21062 --- [ocp-schedules-30,f831c7abe37cb609,342a715684749797] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:34:28.552 INFO 21062 --- [ocp-schedules-15,81a1fd2e9dd9590f,bd27cc9ce3edacb5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":320,"seriesIdKeySize":320,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2024-11-04 12:34:28.552 INFO 21062 --- [ocp-schedules-15,81a1fd2e9dd9590f,bd27cc9ce3edacb5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:34:28.552 INFO 21062 --- [ocp-schedules-16,c620734539886225,166fed1c2f0822d4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:34:28.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:34:28.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:34:28.558 INFO 21062 --- [ocp-schedules-23,404fbe0115611a62,2d27f345f1ddac24] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:34:28.564 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:34:28.564 INFO 21062 --- [ocp-schedules-16,c620734539886225,166fed1c2f0822d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:34:28.564 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:34:28.564 INFO 21062 --- [ocp-schedules-13,e97ec6d0cf02cd8a,88df93f38cfaf511] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:34:28.564 INFO 21062 --- [ocp-schedules-7,a279fd0aa75f4ee9,72f739be1c36c94c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:34:28.567 INFO 21062 --- [ocp-schedules-24,f09329216ab8032b,91d0a25e7c005087] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 12:34:28.568 INFO 21062 --- [ocp-schedules-24,f09329216ab8032b,91d0a25e7c005087] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 12:34:28.568 INFO 21062 --- [ocp-schedules-3,dbc12e2c75c7aa91,882409e62d57b599] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:34:28.568 INFO 21062 --- [ocp-schedules-3,dbc12e2c75c7aa91,882409e62d57b599] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:34:28.578 WARN 21062 --- [ocp-schedules-16,c620734539886225,166fed1c2f0822d4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:34:28.578 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:34:28.578 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:34:28.581 INFO 21062 --- [ocp-schedules-16,c620734539886225,166fed1c2f0822d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:34:28.581 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:34:28.581 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:34:28.582 INFO 21062 --- [ocp-schedules-8,a1fe0dbf3857e8c5,cb180882cac044df] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:34:28.582 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:34:28.582 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:34:28.582 WARN 21062 --- [ocp-schedules-16,c620734539886225,166fed1c2f0822d4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:34:28.587 INFO 21062 --- [ocp-schedules-5,a9f55abbce62ee29,a8e1031465f35b6f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:34:28.587 INFO 21062 --- [ocp-schedules-5,a9f55abbce62ee29,a8e1031465f35b6f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:34:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:34:28.588200+08:00, lastProcessTime=2024-11-04T12:34:18.588122+08:00 2024-11-04 12:34:28.602 INFO 21062 --- [ocp-schedules-8,a1fe0dbf3857e8c5,cb180882cac044df] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:34:28.602 INFO 21062 --- [ocp-schedules-8,a1fe0dbf3857e8c5,cb180882cac044df] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:34:28.603 ERROR 21062 --- [ocp-schedules-16,c620734539886225,166fed1c2f0822d4] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2024-11-04 12:34:28.605 ERROR 21062 --- [ocp-schedules-2,f1b8d906d99ef025,acad590bb00551e1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:34:28.605 INFO 21062 --- [ocp-schedules-2,f1b8d906d99ef025,acad590bb00551e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:34:28.612 INFO 21062 --- [ocp-schedules-5,a9f55abbce62ee29,a8e1031465f35b6f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:34:28.612 INFO 21062 --- [ocp-schedules-5,a9f55abbce62ee29,a8e1031465f35b6f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:34:28.622 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:34:28.622 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:34:28.622 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:34:28.622 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:34:28.623 INFO 21062 --- [ocp-schedules-29,068717bce5e209a0,e6a148dd7627453c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:34:28.628 INFO 21062 --- [ocp-schedules-28,f3e7befb477cde97,b93e1c70df5af292] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:34:28.630 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=111, maxSyncedRecordId=709908318657990656 2024-11-04 12:34:28.630 INFO 21062 --- [ocp-schedules-1,555f00b00bb020c7,edf2fc3e7bce83b3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:34:28.630+08:00, clusters=[1] 2024-11-04 12:34:28.631 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:34:28.631 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:34:28.634 INFO 21062 --- [ocp-schedules-28,f3e7befb477cde97,b93e1c70df5af292] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:34:28.634 INFO 21062 --- [ocp-schedules-28,f3e7befb477cde97,b93e1c70df5af292] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:34:28.634 INFO 21062 --- [ocp-schedules-1,555f00b00bb020c7,edf2fc3e7bce83b3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:34:28.652 INFO 21062 --- [ocp-schedules-27,237f55c75ea218e3,721d836789b35400] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=320, maxSyncedRecordId=709908335011581952 2024-11-04 12:34:28.655 INFO 21062 --- [ocp-schedules-6,e41e4a8af2e5546e,12f5f868ddf437d7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:34:28.655+08:00, clusters=[1] 2024-11-04 12:34:28.698 INFO 21062 --- [ocp-schedules-19,f6f8266280b29e4d,46d1386a54fd2ba1] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:34:28.701 INFO 21062 --- [ocp-schedules-10,c2f9b10a2d8d394d,8566d052013de810] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:34:28.701 INFO 21062 --- [ocp-schedules-10,ca400df023a7dda8,933b6487496d564e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:34:28.701 INFO 21062 --- [ocp-schedules-19,f6f8266280b29e4d,46d1386a54fd2ba1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:28.704 INFO 21062 --- [ocp-schedules-10,ca400df023a7dda8,933b6487496d564e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:28.720 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.722 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:28.730 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:34:28.730 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:34:28.747 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:28.748 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241105 2024-11-04 12:34:28.749 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.749 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:34:28.759 INFO 21062 --- [ocp-schedules-6,e41e4a8af2e5546e,12f5f868ddf437d7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:34:28.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:34:28.588200+08:00 2024-11-04 12:34:28.787 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:34:28.788 ERROR 21062 --- [ocp-schedules-19,f6f8266280b29e4d,46d1386a54fd2ba1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:34:28.788 INFO 21062 --- [ocp-schedules-19,f6f8266280b29e4d,46d1386a54fd2ba1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:34:28.936 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.937 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:28.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:28.963 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241106 2024-11-04 12:34:28.964 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:28.964 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:34:29.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:29.181 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:29.183 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241107 2024-11-04 12:34:29.183 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:34:29.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:29.325 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:29.375 INFO 21062 --- [ocp-schedules-18,fbc023d089329071,40fa4a70e1cd6483] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:34:29.379 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:29.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:29.448 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241108 2024-11-04 12:34:29.449 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.449 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:34:29.640 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.642 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:29.666 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:29.667 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241109 2024-11-04 12:34:29.667 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.668 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:34:29.858 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.860 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:29.884 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:29.885 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241110 2024-11-04 12:34:29.885 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:29.886 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:34:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3c04f68040da5fee,46efa4eaedeeb4eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:30.076 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.077 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:30.101 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:30.102 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20241111 2024-11-04 12:34:30.103 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.103 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:34:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:34:30.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.299 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2024-11-04 12:34:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:34:30.323 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2024-11-04 12:34:30.325 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.325 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:30.326 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.327 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 12:34:30.336 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241028 2024-11-04 12:34:30.337 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.338 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241028 values less than (1730160000000000)) 2024-11-04 12:34:30.541 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.543 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:30.546 WARN 21062 --- [ocp-async-3,5784a32f299f47d7,645fed348a1b83e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:30.546 INFO 21062 --- [ocp-async-6,47e810ef1d91ce80,95ae8804677cffd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:30.567 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:30.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241029 2024-11-04 12:34:30.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.569 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241029 values less than (1730246400000000)) 2024-11-04 12:34:30.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:30.840 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:30.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241030 2024-11-04 12:34:30.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:30.842 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241030 values less than (1730332800000000)) 2024-11-04 12:34:31.091 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:31.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:31.117 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:31.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241031 2024-11-04 12:34:31.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:31.119 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241031 values less than (1730419200000000)) 2024-11-04 12:34:31.368 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:31.370 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:31.395 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:31.396 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241101 2024-11-04 12:34:31.396 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:31.397 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241101 values less than (1730505600000000)) 2024-11-04 12:34:31.652 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:31.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:31.723 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:31.724 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241102 2024-11-04 12:34:31.724 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:31.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241102 values less than (1730592000000000)) 2024-11-04 12:34:31.979 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:31.981 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:32.006 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:32.007 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241103 2024-11-04 12:34:32.007 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:32.008 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241103 values less than (1730678400000000)) 2024-11-04 12:34:32.263 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:32.264 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:32.289 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:32.290 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241104 2024-11-04 12:34:32.290 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:32.291 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241104 values less than (1730764800000000)) 2024-11-04 12:34:32.325 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:32.326 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:32.542 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:32.543 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:32.546 WARN 21062 --- [ocp-async-4,315bf0a2d3ba2b8e,b53eaf211be35a0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:32.547 INFO 21062 --- [ocp-async-9,b4b8784dccc4911f,2d14f2eee1e580b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:32.571 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:32.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241105 2024-11-04 12:34:32.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:32.574 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241105 values less than (1730851200000000)) 2024-11-04 12:34:32.829 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:32.830 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:32.855 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:32.856 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241106 2024-11-04 12:34:32.856 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:32.857 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241106 values less than (1730937600000000)) 2024-11-04 12:34:32.983 INFO 21062 --- [prometheus-minute-schedule-1,848243df3b1da2bf,4e15ae76474e184e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:33.108 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.109 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:33.132 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:33.133 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241107 2024-11-04 12:34:33.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241107 values less than (1731024000000000)) 2024-11-04 12:34:33.398 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.399 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:33.424 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:33.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241108 2024-11-04 12:34:33.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.426 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241108 values less than (1731110400000000)) 2024-11-04 12:34:33.681 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.683 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:33.707 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:33.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241109 2024-11-04 12:34:33.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241109 values less than (1731196800000000)) 2024-11-04 12:34:33.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.964 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:33.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:33.989 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241110 2024-11-04 12:34:33.989 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:33.990 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241110 values less than (1731283200000000)) 2024-11-04 12:34:34.246 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.248 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:34.273 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:34.274 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241111 2024-11-04 12:34:34.275 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.275 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241111 values less than (1731369600000000)) 2024-11-04 12:34:34.529 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.531 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:34.547 WARN 21062 --- [ocp-async-10,d7a3569547bd3b66,1dc23741b50649b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:34.547 INFO 21062 --- [ocp-async-2,313144a68ecc9382,9f7935a8be9b8863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:34.556 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:34.557 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20241112 2024-11-04 12:34:34.558 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.558 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20241112 values less than (1731456000000000)) 2024-11-04 12:34:34.812 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2024-11-04 12:34:34.838 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2024-11-04 12:34:34.839 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.840 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:34.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 12:34:34.864 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728057600000000 2024-11-04 12:34:34.865 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:34.865 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728057600000000 values less than (1728057600000000)) 2024-11-04 12:34:35.037 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.038 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:35.063 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:35.064 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728144000000000 2024-11-04 12:34:35.064 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728144000000000 values less than (1728144000000000)) 2024-11-04 12:34:35.257 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.258 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:35.284 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:35.285 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728230400000000 2024-11-04 12:34:35.286 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.287 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728230400000000 values less than (1728230400000000)) 2024-11-04 12:34:35.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:35.328 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:35.478 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.480 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:35.505 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:35.506 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728316800000000 2024-11-04 12:34:35.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.507 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728316800000000 values less than (1728316800000000)) 2024-11-04 12:34:35.699 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.700 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:35.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:35.727 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728403200000000 2024-11-04 12:34:35.728 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.728 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728403200000000 values less than (1728403200000000)) 2024-11-04 12:34:35.926 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.927 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:35.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:35.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728489600000000 2024-11-04 12:34:35.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:35.954 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728489600000000 values less than (1728489600000000)) 2024-11-04 12:34:35.983 INFO 21062 --- [prometheus-minute-schedule-1,d98c47e7a4f56abc,d9db95ce59de361f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:36.154 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.155 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:36.179 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:36.180 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728576000000000 2024-11-04 12:34:36.181 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.181 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728576000000000 values less than (1728576000000000)) 2024-11-04 12:34:36.377 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.378 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:36.403 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:36.404 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728662400000000 2024-11-04 12:34:36.404 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.405 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728662400000000 values less than (1728662400000000)) 2024-11-04 12:34:36.547 WARN 21062 --- [ocp-async-3,de7a734edf1f83ae,727a6150a7e3ed13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:36.547 INFO 21062 --- [ocp-async-8,07c3b9bbdeb90a4e,2cc077f0dc1449b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:36.603 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.604 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:36.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:36.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728748800000000 2024-11-04 12:34:36.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.631 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728748800000000 values less than (1728748800000000)) 2024-11-04 12:34:36.829 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.831 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:36.855 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:36.856 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728835200000000 2024-11-04 12:34:36.857 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:36.858 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728835200000000 values less than (1728835200000000)) 2024-11-04 12:34:37.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.061 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:37.085 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:37.086 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1728921600000000 2024-11-04 12:34:37.086 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.087 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1728921600000000 values less than (1728921600000000)) 2024-11-04 12:34:37.291 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.293 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:37.317 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:37.318 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729008000000000 2024-11-04 12:34:37.319 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.320 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729008000000000 values less than (1729008000000000)) 2024-11-04 12:34:37.523 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.524 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:37.548 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:37.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729094400000000 2024-11-04 12:34:37.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729094400000000 values less than (1729094400000000)) 2024-11-04 12:34:37.756 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.758 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:37.782 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:37.783 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729180800000000 2024-11-04 12:34:37.784 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.785 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729180800000000 values less than (1729180800000000)) 2024-11-04 12:34:37.992 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:37.993 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:38.018 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:38.019 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729267200000000 2024-11-04 12:34:38.020 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.021 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729267200000000 values less than (1729267200000000)) 2024-11-04 12:34:38.222 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.223 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:38.247 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:38.248 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729353600000000 2024-11-04 12:34:38.249 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.250 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729353600000000 values less than (1729353600000000)) 2024-11-04 12:34:38.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:38.330 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:38.462 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.464 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:38.489 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:38.490 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729440000000000 2024-11-04 12:34:38.491 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.491 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729440000000000 values less than (1729440000000000)) 2024-11-04 12:34:38.505 INFO 21062 --- [ocp-schedules-9,fb98ddbfb6798959,7b7961ab687fecd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:34:38.506 INFO 21062 --- [ocp-schedules-22,10face1b0ab920c6,03bead24b350ad0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:34:38.526 WARN 21062 --- [ocp-schedules-4,c4fbf3b9f5e22ba6,ccec96b5fe6b7576] c.o.o.c.d.lock.jdbc.DefaultLockManager : LockException while executing with jdbc lock part-worker Cannot acquire lock for key part-worker 2024-11-04 12:34:38.526 ERROR 21062 --- [ocp-schedules-4,c4fbf3b9f5e22ba6,ccec96b5fe6b7576] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.distributed.lock.LockException: Cannot acquire lock for key part-worker at com.oceanbase.ocp.core.distributed.lock.jdbc.DefaultLockManager.doExecuteWhileLocked(DefaultLockManager.java:97) at com.oceanbase.ocp.core.distributed.lock.jdbc.DefaultLockManager.executeWhileLocked(DefaultLockManager.java:81) at com.oceanbase.ocp.service.partitioning.PartitionRolloverService.watchAllWhileLocked(PartitionRolloverService.java:121) at com.oceanbase.ocp.server.common.scheduled.OcpSchedules.partitionDaemon(OcpSchedules.java:130) ... 15 common frames omitted 2024-11-04 12:34:38.528 INFO 21062 --- [ocp-async-6,d42c8257d7b79743,d208924a1a217283] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:34:38.547 WARN 21062 --- [ocp-async-9,5cf098ae391e23b1,d63644e246f85fd1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:38.547 INFO 21062 --- [ocp-async-10,0b1f15e78044e8d9,9eeda2405b1d58c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:38.553 INFO 21062 --- [ocp-schedules-14,5888e14feb99e427,e7acbb719a594a76] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:34:38.556 INFO 21062 --- [ocp-schedules-14,5888e14feb99e427,e7acbb719a594a76] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:38.556 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:34:38.560 INFO 21062 --- [ocp-schedules-11,27a66856b95c1df4,0a07dd8561be18b8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:34:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:34:38.588026+08:00, lastProcessTime=2024-11-04T12:34:28.588200+08:00 2024-11-04 12:34:38.704 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.705 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:38.731 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:38.732 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729526400000000 2024-11-04 12:34:38.733 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.733 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729526400000000 values less than (1729526400000000)) 2024-11-04 12:34:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:34:38.588026+08:00 2024-11-04 12:34:38.948 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.949 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:38.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:38.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729612800000000 2024-11-04 12:34:38.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:38.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729612800000000 values less than (1729612800000000)) 2024-11-04 12:34:38.983 INFO 21062 --- [prometheus-minute-schedule-1,8394b6744f684ca4,a6c7b6adbbb4e240] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:39.193 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:39.219 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:39.220 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729699200000000 2024-11-04 12:34:39.221 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.222 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729699200000000 values less than (1729699200000000)) 2024-11-04 12:34:39.435 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.436 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:39.460 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:39.461 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729785600000000 2024-11-04 12:34:39.462 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.463 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729785600000000 values less than (1729785600000000)) 2024-11-04 12:34:39.529 INFO 21062 --- [ocp-schedules-2,1ea5c325b7150ff8,9a64d86f2aa6ebff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:34:39.684 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.685 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:39.710 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:39.711 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729872000000000 2024-11-04 12:34:39.711 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.712 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729872000000000 values less than (1729872000000000)) 2024-11-04 12:34:39.930 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.931 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:39.955 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:39.956 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1729958400000000 2024-11-04 12:34:39.956 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:39.957 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1729958400000000 values less than (1729958400000000)) 2024-11-04 12:34:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:34:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:34:40.179 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.180 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:40.205 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:40.206 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730044800000000 2024-11-04 12:34:40.206 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.207 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730044800000000 values less than (1730044800000000)) 2024-11-04 12:34:40.430 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.431 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:40.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:40.456 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730131200000000 2024-11-04 12:34:40.457 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.458 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730131200000000 values less than (1730131200000000)) 2024-11-04 12:34:40.547 WARN 21062 --- [ocp-async-2,b62bb55ee4da7ac9,e82ba99dc668cfb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:40.547 INFO 21062 --- [ocp-async-3,9ab86721f4aea312,45ddc2661d94f38b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:40.681 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.682 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:40.707 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:40.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730217600000000 2024-11-04 12:34:40.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730217600000000 values less than (1730217600000000)) 2024-11-04 12:34:40.936 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.937 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:40.961 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:40.961 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730304000000000 2024-11-04 12:34:40.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:40.963 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730304000000000 values less than (1730304000000000)) 2024-11-04 12:34:41.186 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.188 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:41.213 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:41.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730390400000000 2024-11-04 12:34:41.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.215 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730390400000000 values less than (1730390400000000)) 2024-11-04 12:34:41.330 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:41.331 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:41.445 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.447 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:41.470 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:41.471 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730476800000000 2024-11-04 12:34:41.471 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.472 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730476800000000 values less than (1730476800000000)) 2024-11-04 12:34:41.701 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:41.728 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:41.729 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730563200000000 2024-11-04 12:34:41.729 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.730 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730563200000000 values less than (1730563200000000)) 2024-11-04 12:34:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:34:41.823 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:34:41.823 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 12:34:41.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.964 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:41.983 INFO 21062 --- [prometheus-minute-schedule-1,61f3422f3808b08f,b31662a203c2a37d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:41.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:41.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730649600000000 2024-11-04 12:34:41.989 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:41.990 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730649600000000 values less than (1730649600000000)) 2024-11-04 12:34:42.221 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:42.222 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:42.247 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:42.248 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730736000000000 2024-11-04 12:34:42.248 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:42.249 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730736000000000 values less than (1730736000000000)) 2024-11-04 12:34:42.485 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:42.486 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:42.510 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:42.511 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730822400000000 2024-11-04 12:34:42.512 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:42.512 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730822400000000 values less than (1730822400000000)) 2024-11-04 12:34:42.547 WARN 21062 --- [ocp-async-8,838344561c4155eb,c4dfc58120fef30e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:42.547 INFO 21062 --- [ocp-async-4,86b1c0c5a50daf3a,bed36c47e5e9aa30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:42.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:42.747 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:42.771 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:42.772 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1730908800000000 2024-11-04 12:34:42.772 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:42.773 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1730908800000000 values less than (1730908800000000)) 2024-11-04 12:34:43.008 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.009 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2024-11-04 12:34:43.034 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2024-11-04 12:34:43.035 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.036 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:43.037 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.037 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 12:34:43.056 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728057600000000 2024-11-04 12:34:43.057 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.057 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728057600000000 values less than (1728057600000000)) 2024-11-04 12:34:43.268 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.270 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:43.294 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:43.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728144000000000 2024-11-04 12:34:43.296 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.297 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728144000000000 values less than (1728144000000000)) 2024-11-04 12:34:43.552 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.554 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:43.578 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:43.579 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728230400000000 2024-11-04 12:34:43.580 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728230400000000 values less than (1728230400000000)) 2024-11-04 12:34:43.836 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.838 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:43.862 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:43.863 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728316800000000 2024-11-04 12:34:43.863 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:43.864 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728316800000000 values less than (1728316800000000)) 2024-11-04 12:34:44.121 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:44.123 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:44.150 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:44.151 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728403200000000 2024-11-04 12:34:44.152 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:44.153 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728403200000000 values less than (1728403200000000)) 2024-11-04 12:34:44.331 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:44.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:44.415 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:44.416 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:44.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:44.454 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728489600000000 2024-11-04 12:34:44.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:44.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728489600000000 values less than (1728489600000000)) 2024-11-04 12:34:44.547 WARN 21062 --- [ocp-async-6,568eb66d4095d463,ed1f034d607bf649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:44.548 INFO 21062 --- [ocp-async-9,c3a64c7a7f4031ee,6a8fab60f5a3fa53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:44.715 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:44.717 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:44.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:44.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728576000000000 2024-11-04 12:34:44.747 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:44.748 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728576000000000 values less than (1728576000000000)) 2024-11-04 12:34:44.983 INFO 21062 --- [prometheus-minute-schedule-1,1a258933f405b772,787d11ec5354121e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:45.010 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.012 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:45.035 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:45.036 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728662400000000 2024-11-04 12:34:45.037 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.037 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728662400000000 values less than (1728662400000000)) 2024-11-04 12:34:45.314 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.315 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:45.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:45.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728748800000000 2024-11-04 12:34:45.342 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.343 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728748800000000 values less than (1728748800000000)) 2024-11-04 12:34:45.611 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.613 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:45.637 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:45.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728835200000000 2024-11-04 12:34:45.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.639 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728835200000000 values less than (1728835200000000)) 2024-11-04 12:34:45.906 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.908 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:45.932 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:45.933 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1728921600000000 2024-11-04 12:34:45.934 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:45.935 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1728921600000000 values less than (1728921600000000)) 2024-11-04 12:34:46.205 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:46.207 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:46.231 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:46.232 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729008000000000 2024-11-04 12:34:46.232 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:46.233 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729008000000000 values less than (1729008000000000)) 2024-11-04 12:34:46.508 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:46.510 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:46.534 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:46.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729094400000000 2024-11-04 12:34:46.535 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:46.536 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729094400000000 values less than (1729094400000000)) 2024-11-04 12:34:46.548 WARN 21062 --- [ocp-async-10,a6e3b3cf4cf013c8,778ed14efd2ab24a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:46.548 INFO 21062 --- [ocp-async-2,ddb313e59b958114,60dc15e94222c2d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:46.811 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:46.812 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:46.837 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:46.838 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729180800000000 2024-11-04 12:34:46.839 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:46.840 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729180800000000 values less than (1729180800000000)) 2024-11-04 12:34:47.112 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:47.114 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:47.139 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:47.140 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729267200000000 2024-11-04 12:34:47.141 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:47.141 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729267200000000 values less than (1729267200000000)) 2024-11-04 12:34:47.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:47.333 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:47.416 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:47.417 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:47.442 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:47.443 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729353600000000 2024-11-04 12:34:47.443 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:47.444 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729353600000000 values less than (1729353600000000)) 2024-11-04 12:34:47.718 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:47.719 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:47.744 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:47.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729440000000000 2024-11-04 12:34:47.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:47.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729440000000000 values less than (1729440000000000)) 2024-11-04 12:34:47.983 INFO 21062 --- [prometheus-minute-schedule-1,46fc374921edd5c8,abd850127b3c33d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:48.031 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:48.033 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:48.057 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:48.058 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729526400000000 2024-11-04 12:34:48.058 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:48.059 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729526400000000 values less than (1729526400000000)) 2024-11-04 12:34:48.338 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:48.340 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:48.364 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:48.365 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729612800000000 2024-11-04 12:34:48.366 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:48.366 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729612800000000 values less than (1729612800000000)) 2024-11-04 12:34:48.528 INFO 21062 --- [ocp-async-2,af82dad09d08ccdc,56145cb7f8010453] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:34:48.548 WARN 21062 --- [ocp-async-8,9bb2425b98be2e62,1be57ab10fe0d21b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:48.548 INFO 21062 --- [ocp-async-3,6e6df04ac66d184e,e1843b2d8f9ebb72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:48.557 INFO 21062 --- [ocp-schedules-14,5888e14feb99e427,e7acbb719a594a76] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:34:48.563 INFO 21062 --- [ocp-schedules-17,26b676c371d483ee,59fe65e6be34f801] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:34:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:34:48.589085+08:00, lastProcessTime=2024-11-04T12:34:38.588026+08:00 2024-11-04 12:34:48.636 INFO 21062 --- [ocp-schedules-19,9ce25416aa7c6931,426f2dd79dc1c944] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:34:48.642 INFO 21062 --- [ocp-schedules-19,9ce25416aa7c6931,426f2dd79dc1c944] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:34:48.642 INFO 21062 --- [ocp-schedules-19,9ce25416aa7c6931,426f2dd79dc1c944] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:34:48.671 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:48.672 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:48.696 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:48.698 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729699200000000 2024-11-04 12:34:48.698 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:48.699 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729699200000000 values less than (1729699200000000)) 2024-11-04 12:34:48.775 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:34:48.589085+08:00 2024-11-04 12:34:48.986 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:48.987 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:49.012 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:49.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729785600000000 2024-11-04 12:34:49.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:49.015 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729785600000000 values less than (1729785600000000)) 2024-11-04 12:34:49.302 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:49.306 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:49.332 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:49.333 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729872000000000 2024-11-04 12:34:49.334 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:49.334 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729872000000000 values less than (1729872000000000)) 2024-11-04 12:34:49.556 INFO 21062 --- [ocp-schedules-4,6e1396193e0b12b7,cc1e779fa1956ac8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:34:49.620 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:49.621 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:49.646 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:49.647 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1729958400000000 2024-11-04 12:34:49.648 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:49.648 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1729958400000000 values less than (1729958400000000)) 2024-11-04 12:34:49.940 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:49.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:49.965 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:49.966 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730044800000000 2024-11-04 12:34:49.966 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:49.967 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730044800000000 values less than (1730044800000000)) 2024-11-04 12:34:50.262 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:50.263 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:50.290 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:50.291 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730131200000000 2024-11-04 12:34:50.292 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:50.293 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730131200000000 values less than (1730131200000000)) 2024-11-04 12:34:50.334 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:50.335 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:50.548 WARN 21062 --- [ocp-async-5,c72f2c2bdc894aab,bc26506b923e3104] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:50.548 INFO 21062 --- [ocp-async-5,754dcb3d1b7d80e0,e2a0064cbf47b1ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:50.591 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:50.592 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:50.618 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:50.619 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730217600000000 2024-11-04 12:34:50.620 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:50.620 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730217600000000 values less than (1730217600000000)) 2024-11-04 12:34:50.918 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:50.919 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:50.945 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:50.946 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730304000000000 2024-11-04 12:34:50.947 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:50.948 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730304000000000 values less than (1730304000000000)) 2024-11-04 12:34:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d0f08b4cc0622f0f,6da1b3dd198d1391] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:51.241 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:51.243 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:51.267 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:51.268 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730390400000000 2024-11-04 12:34:51.268 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:51.269 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730390400000000 values less than (1730390400000000)) 2024-11-04 12:34:51.563 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:51.564 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:51.587 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:51.588 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730476800000000 2024-11-04 12:34:51.589 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:51.590 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730476800000000 values less than (1730476800000000)) 2024-11-04 12:34:51.888 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:51.889 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:51.913 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:51.914 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730563200000000 2024-11-04 12:34:51.915 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:51.915 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730563200000000 values less than (1730563200000000)) 2024-11-04 12:34:52.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:52.213 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:52.238 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:52.239 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730649600000000 2024-11-04 12:34:52.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:52.241 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730649600000000 values less than (1730649600000000)) 2024-11-04 12:34:52.541 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:52.542 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:52.548 WARN 21062 --- [ocp-async-9,9e403285451027d9,e73bfc5510035db8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:52.548 INFO 21062 --- [ocp-async-1,58445873f9781a68,c6476e6c0bd71088] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:52.566 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:52.567 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730736000000000 2024-11-04 12:34:52.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:52.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730736000000000 values less than (1730736000000000)) 2024-11-04 12:34:52.871 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:52.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:52.896 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:52.897 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730822400000000 2024-11-04 12:34:52.898 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:52.899 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730822400000000 values less than (1730822400000000)) 2024-11-04 12:34:53.209 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.211 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:53.236 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:53.237 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1730908800000000 2024-11-04 12:34:53.238 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.239 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1730908800000000 values less than (1730908800000000)) 2024-11-04 12:34:53.336 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:53.336 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:53.552 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.554 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2024-11-04 12:34:53.578 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2024-11-04 12:34:53.580 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.580 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:53.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 12:34:53.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730044800 2024-11-04 12:34:53.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730044800 values less than (1730044800)) 2024-11-04 12:34:53.740 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.742 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:53.767 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:53.769 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730131200 2024-11-04 12:34:53.769 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.770 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730131200 values less than (1730131200)) 2024-11-04 12:34:53.916 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.917 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:53.941 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:53.942 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730217600 2024-11-04 12:34:53.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:53.943 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730217600 values less than (1730217600)) 2024-11-04 12:34:53.983 INFO 21062 --- [prometheus-minute-schedule-1,26211feee835d028,7252a52f5feb2a81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:54.091 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.092 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:54.117 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:54.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730304000 2024-11-04 12:34:54.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.119 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730304000 values less than (1730304000)) 2024-11-04 12:34:54.265 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.266 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:54.290 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:54.291 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730390400 2024-11-04 12:34:54.291 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.292 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730390400 values less than (1730390400)) 2024-11-04 12:34:54.440 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.442 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:54.465 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:54.466 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730476800 2024-11-04 12:34:54.467 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.468 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730476800 values less than (1730476800)) 2024-11-04 12:34:54.548 WARN 21062 --- [ocp-async-8,95290c998bfc435d,1b3493de5d73c5fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:54.548 INFO 21062 --- [ocp-async-7,eb2ab32329d171bb,86078f91af8c650e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:54.613 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.614 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:54.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:54.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730563200 2024-11-04 12:34:54.639 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.640 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730563200 values less than (1730563200)) 2024-11-04 12:34:54.787 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.789 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:54.818 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:54.819 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730649600 2024-11-04 12:34:54.820 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.821 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730649600 values less than (1730649600)) 2024-11-04 12:34:54.972 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:54.997 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:54.998 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730736000 2024-11-04 12:34:54.999 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:54.999 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730736000 values less than (1730736000)) 2024-11-04 12:34:55.149 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.151 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:55.175 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:55.176 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730822400 2024-11-04 12:34:55.176 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.177 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730822400 values less than (1730822400)) 2024-11-04 12:34:55.332 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.333 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:55.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:55.360 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1730908800 2024-11-04 12:34:55.360 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.361 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1730908800 values less than (1730908800)) 2024-11-04 12:34:55.518 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.519 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2024-11-04 12:34:55.542 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2024-11-04 12:34:55.544 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.545 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:34:55.546 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.546 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 12:34:55.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728057600 2024-11-04 12:34:55.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728057600 values less than (1728057600)) 2024-11-04 12:34:55.718 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.720 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:55.744 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:55.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728144000 2024-11-04 12:34:55.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.747 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728144000 values less than (1728144000)) 2024-11-04 12:34:55.900 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.901 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:55.925 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:55.926 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728230400 2024-11-04 12:34:55.926 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:55.927 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728230400 values less than (1728230400)) 2024-11-04 12:34:56.075 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.077 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:56.100 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:56.101 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728316800 2024-11-04 12:34:56.102 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.102 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728316800 values less than (1728316800)) 2024-11-04 12:34:56.250 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:56.276 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:56.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728403200 2024-11-04 12:34:56.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.278 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728403200 values less than (1728403200)) 2024-11-04 12:34:56.337 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:56.338 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:56.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.428 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:56.451 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:56.452 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728489600 2024-11-04 12:34:56.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.454 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728489600 values less than (1728489600)) 2024-11-04 12:34:56.548 WARN 21062 --- [ocp-async-4,5888d1f6cc51fc57,bbc4daac2f656a7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:56.549 INFO 21062 --- [ocp-async-3,facf80e1add43c16,2b0bb0b4e4e8b33d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:56.603 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.604 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:56.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:56.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728576000 2024-11-04 12:34:56.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.630 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728576000 values less than (1728576000)) 2024-11-04 12:34:56.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.780 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:56.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:56.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728662400 2024-11-04 12:34:56.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.808 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728662400 values less than (1728662400)) 2024-11-04 12:34:56.958 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.959 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fed41842853f060c,b1a380b09147576b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:34:56.983 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:56.984 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728748800 2024-11-04 12:34:56.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:56.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728748800 values less than (1728748800)) 2024-11-04 12:34:57.133 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:57.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:57.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728835200 2024-11-04 12:34:57.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728835200 values less than (1728835200)) 2024-11-04 12:34:57.310 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.311 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:57.335 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:57.336 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1728921600 2024-11-04 12:34:57.337 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.338 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1728921600 values less than (1728921600)) 2024-11-04 12:34:57.490 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.491 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:57.514 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:57.515 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729008000 2024-11-04 12:34:57.516 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.517 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729008000 values less than (1729008000)) 2024-11-04 12:34:57.669 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.670 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:57.692 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:57.693 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729094400 2024-11-04 12:34:57.693 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.694 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729094400 values less than (1729094400)) 2024-11-04 12:34:57.849 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.850 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:57.872 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:57.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729180800 2024-11-04 12:34:57.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:57.874 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729180800 values less than (1729180800)) 2024-11-04 12:34:58.027 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.028 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:58.052 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:58.053 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729267200 2024-11-04 12:34:58.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729267200 values less than (1729267200)) 2024-11-04 12:34:58.196 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.197 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:58.220 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:58.221 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729353600 2024-11-04 12:34:58.222 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.223 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729353600 values less than (1729353600)) 2024-11-04 12:34:58.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:58.397 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:58.398 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729440000 2024-11-04 12:34:58.399 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.400 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729440000 values less than (1729440000)) 2024-11-04 12:34:58.500 INFO 21062 --- [ocp-schedules-22,e2de44dea6ec7a47,ea04099957bdae05] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:34:58.500 INFO 21062 --- [ocp-schedules-16,ba76c02cb2bfd8e7,8e2a7bdae8cb2885] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:34:58.503 INFO 21062 --- [ocp-schedules-27,23de0d1e362f4088,fea102d507f6c887] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:34:58.503 INFO 21062 --- [ocp-schedules-22,e2de44dea6ec7a47,ea04099957bdae05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:58.504 INFO 21062 --- [ocp-schedules-16,ba76c02cb2bfd8e7,8e2a7bdae8cb2885] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:34:58.504 INFO 21062 --- [ocp-schedules-4,7e1dfc8a2c7545c5,334f45cbfa0b4411] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:34:58.513 INFO 21062 --- [ocp-schedules-15,41595928184c27d8,9575148c8e590f76] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:34:58.513 INFO 21062 --- [ocp-schedules-9,b4ab6aadd978182b,4eca886bbc001636] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:34:58.513 INFO 21062 --- [ocp-schedules-15,41595928184c27d8,9575148c8e590f76] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:34:58.513 INFO 21062 --- [ocp-schedules-9,b4ab6aadd978182b,4eca886bbc001636] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:34:58.513 INFO 21062 --- [ocp-schedules-20,40ee400384e2f43c,6151a95522a1009a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:34:58.513 INFO 21062 --- [ocp-schedules-20,40ee400384e2f43c,6151a95522a1009a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:34:58.531 INFO 21062 --- [ocp-async-5,27c95f7b73e6719e,41a05b35279e0bce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:34:58.549 WARN 21062 --- [ocp-async-10,b235f792b5f4aaa8,3b7c129e79c94d36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:34:58.549 INFO 21062 --- [ocp-async-10,b3f6dfbe0704d9de,18fa3e7ef7178f60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:34:58.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:58.566 INFO 21062 --- [ocp-schedules-12,c7570f5f9f95b413,488ab666a2b17d49] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:34:58.567 INFO 21062 --- [ocp-schedules-30,a8ab595a41acfbda,4dcb337541ffe019] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:34:58.572 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:58.573 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729526400 2024-11-04 12:34:58.574 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729526400 values less than (1729526400)) 2024-11-04 12:34:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:34:58.587249+08:00, lastProcessTime=2024-11-04T12:34:48.589085+08:00 2024-11-04 12:34:58.722 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.723 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:58.731 INFO 21062 --- [ocp-schedules-10,ca400df023a7dda8,933b6487496d564e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:34:58.731 INFO 21062 --- [ocp-schedules-10,190bfd6ba8dae906,d97ae0dc1f6c772b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:34:58.734 INFO 21062 --- [ocp-schedules-10,190bfd6ba8dae906,d97ae0dc1f6c772b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:34:58.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:58.747 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729612800 2024-11-04 12:34:58.748 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.748 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729612800 values less than (1729612800)) 2024-11-04 12:34:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:34:58.587249+08:00 2024-11-04 12:34:58.763 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:34:58.763 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:34:58.898 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.900 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:58.924 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:58.925 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729699200 2024-11-04 12:34:58.925 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:58.926 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729699200 values less than (1729699200)) 2024-11-04 12:34:59.079 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.080 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:59.103 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:59.104 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729785600 2024-11-04 12:34:59.105 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.105 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729785600 values less than (1729785600)) 2024-11-04 12:34:59.260 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.262 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:59.289 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:59.289 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729872000 2024-11-04 12:34:59.290 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.291 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729872000 values less than (1729872000)) 2024-11-04 12:34:59.338 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:34:59.339 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:34:59.442 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.444 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:59.467 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:59.468 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1729958400 2024-11-04 12:34:59.469 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.469 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1729958400 values less than (1729958400)) 2024-11-04 12:34:59.619 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.621 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:59.633 INFO 21062 --- [ocp-schedules-7,339279f47c119ab3,2aa5a62253a132bf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:34:59.648 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:59.649 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730044800 2024-11-04 12:34:59.650 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.651 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730044800 values less than (1730044800)) 2024-11-04 12:34:59.802 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.804 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:59.827 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:34:59.829 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730131200 2024-11-04 12:34:59.829 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.830 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730131200 values less than (1730131200)) 2024-11-04 12:34:59.979 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:34:59.980 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:34:59.983 INFO 21062 --- [prometheus-minute-schedule-1,cf026f69ddeb990f,a4d982c640f7831b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694900, interval=60 2024-11-04 12:35:00.004 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:00.005 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730217600 2024-11-04 12:35:00.006 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.006 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730217600 values less than (1730217600)) 2024-11-04 12:35:00.161 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.162 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:00.186 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:00.187 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730304000 2024-11-04 12:35:00.187 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.188 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730304000 values less than (1730304000)) 2024-11-04 12:35:00.340 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.342 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:00.367 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:00.368 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730390400 2024-11-04 12:35:00.368 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.369 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730390400 values less than (1730390400)) 2024-11-04 12:35:00.527 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.529 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:00.549 WARN 21062 --- [ocp-async-2,c52e701b3b4d6540,0c5c8cb5f03b334c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:00.549 INFO 21062 --- [ocp-async-7,d6bec1a442f7ecaf,26567abbfdcbfe42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:00.552 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:00.553 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730476800 2024-11-04 12:35:00.554 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.554 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730476800 values less than (1730476800)) 2024-11-04 12:35:00.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.709 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,be3bf9d513202db5] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:32:39.0, nextRun=2024-11-04 12:35:00.0, currentDate=Mon Nov 04 12:35:00 CST 2024 2024-11-04 12:35:00.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:00.709 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,be3bf9d513202db5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 12:32:39.0, nextRunTime=2024-11-04 12:35:00.0} 2024-11-04 12:35:00.732 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:00.733 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730563200 2024-11-04 12:35:00.734 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.735 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730563200 values less than (1730563200)) 2024-11-04 12:35:00.736 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,be3bf9d513202db5] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 12:35:00.741 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,be3bf9d513202db5] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 12:35:00.742 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,be3bf9d513202db5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 12:35:00.887 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.889 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:00.912 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:00.913 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730649600 2024-11-04 12:35:00.914 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:00.914 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730649600 values less than (1730649600)) 2024-11-04 12:35:01.066 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.067 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:01.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:01.094 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730736000 2024-11-04 12:35:01.094 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.095 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730736000 values less than (1730736000)) 2024-11-04 12:35:01.249 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:01.274 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:01.275 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730822400 2024-11-04 12:35:01.276 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730822400 values less than (1730822400)) 2024-11-04 12:35:01.430 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.431 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:01.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:01.456 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1730908800 2024-11-04 12:35:01.457 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.457 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1730908800 values less than (1730908800)) 2024-11-04 12:35:01.611 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.612 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2024-11-04 12:35:01.636 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2024-11-04 12:35:01.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:01.639 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.639 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 12:35:01.669 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2021 2024-11-04 12:35:01.669 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.670 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2021 values less than (1640995200)) 2024-11-04 12:35:01.815 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2024-11-04 12:35:01.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2024-11-04 12:35:01.842 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2022 2024-11-04 12:35:01.843 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.843 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2022 values less than (1672531200)) 2024-11-04 12:35:01.990 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:01.991 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2024-11-04 12:35:02.015 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2024-11-04 12:35:02.016 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2023 2024-11-04 12:35:02.017 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.017 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2023 values less than (1704067200)) 2024-11-04 12:35:02.164 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.165 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2024-11-04 12:35:02.189 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2024-11-04 12:35:02.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2024 2024-11-04 12:35:02.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.191 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2024 values less than (1735689600)) 2024-11-04 12:35:02.340 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:02.342 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:02.344 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.346 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2024-11-04 12:35:02.370 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2024-11-04 12:35:02.371 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2025 2024-11-04 12:35:02.372 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.372 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2025 values less than (1767225600)) 2024-11-04 12:35:02.521 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.523 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2024-11-04 12:35:02.547 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2024-11-04 12:35:02.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.549 WARN 21062 --- [ocp-async-6,b92fd7b49cd72396,e67b19f06c8a173a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:02.549 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:02.549 INFO 21062 --- [ocp-async-3,b9ce5e13bb1a33e2,6b10dab778c99481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:02.550 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 12:35:02.567 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2021 2024-11-04 12:35:02.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2021 values less than (1640995200)) 2024-11-04 12:35:02.712 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.714 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2024-11-04 12:35:02.742 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2024-11-04 12:35:02.743 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2022 2024-11-04 12:35:02.744 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2022 values less than (1672531200)) 2024-11-04 12:35:02.896 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.897 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2024-11-04 12:35:02.921 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2024-11-04 12:35:02.922 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2023 2024-11-04 12:35:02.922 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:02.923 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2023 values less than (1704067200)) 2024-11-04 12:35:02.983 INFO 21062 --- [prometheus-minute-schedule-1,37f3922351fac392,dc75cba4a8ca9cfc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:03.073 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.075 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2024-11-04 12:35:03.098 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2024-11-04 12:35:03.099 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2024 2024-11-04 12:35:03.100 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.100 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2024 values less than (1735689600)) 2024-11-04 12:35:03.250 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2024-11-04 12:35:03.275 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2024-11-04 12:35:03.276 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2025 2024-11-04 12:35:03.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2025 values less than (1767225600)) 2024-11-04 12:35:03.426 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2024-11-04 12:35:03.450 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2024-11-04 12:35:03.451 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.452 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:03.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.453 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 12:35:03.536 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241028 2024-11-04 12:35:03.536 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.537 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241028 values less than (1730044800)) 2024-11-04 12:35:03.722 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.723 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:03.748 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:03.749 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241029 2024-11-04 12:35:03.750 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.750 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241029 values less than (1730131200)) 2024-11-04 12:35:03.935 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.937 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:03.960 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:03.961 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241030 2024-11-04 12:35:03.961 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:03.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241030 values less than (1730217600)) 2024-11-04 12:35:04.147 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.149 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:04.172 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:04.173 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241031 2024-11-04 12:35:04.174 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.174 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241031 values less than (1730304000)) 2024-11-04 12:35:04.359 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.360 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:04.383 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:04.384 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241101 2024-11-04 12:35:04.385 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.385 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241101 values less than (1730390400)) 2024-11-04 12:35:04.549 WARN 21062 --- [ocp-async-1,011b60bb195cf385,2c9ac0f4770bcd71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:04.549 INFO 21062 --- [ocp-async-9,05bc544ab33b3300,5754c91bc690544b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:04.567 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.568 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:04.590 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:04.591 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241102 2024-11-04 12:35:04.592 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.593 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241102 values less than (1730476800)) 2024-11-04 12:35:04.779 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.781 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:04.805 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:04.806 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241103 2024-11-04 12:35:04.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.807 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241103 values less than (1730563200)) 2024-11-04 12:35:04.996 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:04.997 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:05.023 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:05.024 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241104 2024-11-04 12:35:05.025 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.025 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241104 values less than (1730649600)) 2024-11-04 12:35:05.212 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:05.237 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:05.239 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241105 2024-11-04 12:35:05.239 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.240 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241105 values less than (1730736000)) 2024-11-04 12:35:05.342 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:05.343 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:05.433 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.434 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:05.457 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:05.459 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241106 2024-11-04 12:35:05.459 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.460 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241106 values less than (1730822400)) 2024-11-04 12:35:05.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.654 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:05.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:05.679 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20241107 2024-11-04 12:35:05.680 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.680 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20241107 values less than (1730908800)) 2024-11-04 12:35:05.872 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.873 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2024-11-04 12:35:05.895 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2024-11-04 12:35:05.896 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.897 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:05.897 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.898 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 12:35:05.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241005 2024-11-04 12:35:05.954 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:05.954 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241005 values less than (1728057600)) 2024-11-04 12:35:05.983 INFO 21062 --- [prometheus-minute-schedule-1,b3ddf7470ad10e9e,02aef2ae603c2ce9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:06.132 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:06.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:06.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241006 2024-11-04 12:35:06.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241006 values less than (1728144000)) 2024-11-04 12:35:06.348 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.350 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:06.373 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:06.374 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241007 2024-11-04 12:35:06.375 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.376 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241007 values less than (1728230400)) 2024-11-04 12:35:06.549 WARN 21062 --- [ocp-async-2,0a2a9bf9280dde9c,825b999c2afab13c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:06.550 INFO 21062 --- [ocp-async-2,9891c91478b7bdd2,0f355bce1a92e097] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:06.562 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.564 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:06.587 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:06.588 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241008 2024-11-04 12:35:06.589 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.589 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241008 values less than (1728316800)) 2024-11-04 12:35:06.775 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.776 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:06.800 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:06.801 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241009 2024-11-04 12:35:06.802 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.802 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241009 values less than (1728403200)) 2024-11-04 12:35:06.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:06.990 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:07.014 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:07.015 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241010 2024-11-04 12:35:07.016 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.016 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241010 values less than (1728489600)) 2024-11-04 12:35:07.205 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.206 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:07.230 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:07.231 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241011 2024-11-04 12:35:07.231 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.232 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241011 values less than (1728576000)) 2024-11-04 12:35:07.423 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.424 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:07.448 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:07.449 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241012 2024-11-04 12:35:07.449 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.450 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241012 values less than (1728662400)) 2024-11-04 12:35:07.642 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.644 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:07.667 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:07.668 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241013 2024-11-04 12:35:07.669 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.670 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241013 values less than (1728748800)) 2024-11-04 12:35:07.869 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.871 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:07.895 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:07.896 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241014 2024-11-04 12:35:07.896 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:07.897 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241014 values less than (1728835200)) 2024-11-04 12:35:08.087 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.089 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:08.114 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:08.115 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241015 2024-11-04 12:35:08.115 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.116 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241015 values less than (1728921600)) 2024-11-04 12:35:08.306 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.307 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:08.330 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:08.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241016 2024-11-04 12:35:08.332 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.332 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241016 values less than (1729008000)) 2024-11-04 12:35:08.343 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:08.344 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:08.505 INFO 21062 --- [ocp-schedules-16,ba76c02cb2bfd8e7,8e2a7bdae8cb2885] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:35:08.505 INFO 21062 --- [ocp-schedules-22,e2de44dea6ec7a47,ea04099957bdae05] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:35:08.527 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.529 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:08.532 INFO 21062 --- [ocp-async-7,ef82a1a64381e094,37963fe5dca4e745] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:35:08.549 WARN 21062 --- [ocp-async-4,a6bd15a970600581,a79d969061701f3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:08.550 INFO 21062 --- [ocp-async-3,0a078eec9eacfe1d,1aa4b5de4a5dbc3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:08.551 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:08.553 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241017 2024-11-04 12:35:08.553 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.554 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241017 values less than (1729094400)) 2024-11-04 12:35:08.568 INFO 21062 --- [ocp-schedules-5,144c4955e08daedc,f1ab0df6f7984312] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:35:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:35:08.587406+08:00, lastProcessTime=2024-11-04T12:34:58.587249+08:00 2024-11-04 12:35:08.646 INFO 21062 --- [ocp-schedules-27,875c3f2c5e0fff85,0c38be7b54529822] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:35:08.652 INFO 21062 --- [ocp-schedules-27,875c3f2c5e0fff85,0c38be7b54529822] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:35:08.652 INFO 21062 --- [ocp-schedules-27,875c3f2c5e0fff85,0c38be7b54529822] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:35:08.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.747 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:08.771 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:08.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:35:08.587406+08:00 2024-11-04 12:35:08.772 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241018 2024-11-04 12:35:08.772 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.773 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241018 values less than (1729180800)) 2024-11-04 12:35:08.967 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.969 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:08.983 INFO 21062 --- [prometheus-minute-schedule-1,706bac174e1e7e70,f20b1f3c17de264d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:08.992 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:08.993 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241019 2024-11-04 12:35:08.993 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:08.994 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241019 values less than (1729267200)) 2024-11-04 12:35:09.189 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.191 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:09.214 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:09.215 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241020 2024-11-04 12:35:09.215 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.216 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241020 values less than (1729353600)) 2024-11-04 12:35:09.413 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.414 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:09.438 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:09.439 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241021 2024-11-04 12:35:09.439 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.440 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241021 values less than (1729440000)) 2024-11-04 12:35:09.636 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.638 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:09.662 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:09.663 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241022 2024-11-04 12:35:09.663 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.664 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241022 values less than (1729526400)) 2024-11-04 12:35:09.665 INFO 21062 --- [ocp-schedules-2,7475adf3811a7f0d,3e2b116f63cf31cc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:35:09.868 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.869 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:09.893 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:09.894 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241023 2024-11-04 12:35:09.894 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:09.895 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241023 values less than (1729612800)) 2024-11-04 12:35:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:35:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:35:10.093 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.095 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:10.117 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:10.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241024 2024-11-04 12:35:10.118 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.119 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241024 values less than (1729699200)) 2024-11-04 12:35:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:35:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:35:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:35:10.320 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.322 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:35:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:35:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:35:10.346 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:10.347 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241025 2024-11-04 12:35:10.347 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.348 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241025 values less than (1729785600)) 2024-11-04 12:35:10.550 WARN 21062 --- [ocp-async-5,348e83d80745b5a1,7b2272ab88d29b77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:10.550 INFO 21062 --- [ocp-async-9,0e94b70127098489,0305606afcbf90d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:10.554 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.555 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:10.578 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:10.580 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241026 2024-11-04 12:35:10.580 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241026 values less than (1729872000)) 2024-11-04 12:35:10.784 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.786 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:10.810 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:10.811 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241027 2024-11-04 12:35:10.812 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:10.813 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241027 values less than (1729958400)) 2024-11-04 12:35:11.012 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.013 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:11.043 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:11.044 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241028 2024-11-04 12:35:11.045 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.046 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241028 values less than (1730044800)) 2024-11-04 12:35:11.058 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=3, affectRows=3 2024-11-04 12:35:11.059 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=2, validCount=2 2024-11-04 12:35:11.061 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=8, affectRows=8 2024-11-04 12:35:11.061 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=7, validCount=7 2024-11-04 12:35:11.062 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=40, affectRows=40 2024-11-04 12:35:11.062 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=53, affectRows=53 2024-11-04 12:35:11.062 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=39, validCount=39 2024-11-04 12:35:11.062 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=25, affectRows=25 2024-11-04 12:35:11.062 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=29, affectRows=29 2024-11-04 12:35:11.062 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=25, validCount=25 2024-11-04 12:35:11.062 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=29, validCount=29 2024-11-04 12:35:11.063 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=53, validCount=53 2024-11-04 12:35:11.064 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=13, affectRows=13 2024-11-04 12:35:11.064 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=13, validCount=13 2024-11-04 12:35:11.066 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=16, affectRows=16 2024-11-04 12:35:11.066 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=16, validCount=16 2024-11-04 12:35:11.066 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=53, affectRows=53 2024-11-04 12:35:11.067 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=53, validCount=53 2024-11-04 12:35:11.067 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=1, affectRows=1 2024-11-04 12:35:11.067 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=1, validCount=1 2024-11-04 12:35:11.069 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=3, affectRows=3 2024-11-04 12:35:11.069 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=3, validCount=3 2024-11-04 12:35:11.072 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=6, affectRows=6 2024-11-04 12:35:11.072 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=6, validCount=6 2024-11-04 12:35:11.077 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=22, affectRows=22 2024-11-04 12:35:11.077 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=22, validCount=22 2024-11-04 12:35:11.077 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694840, nodeCount=50, affectRows=50 2024-11-04 12:35:11.077 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694840, nodeCount=50, validCount=50 2024-11-04 12:35:11.250 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:11.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:11.278 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241029 2024-11-04 12:35:11.279 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.279 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241029 values less than (1730131200)) 2024-11-04 12:35:11.344 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:11.346 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:11.486 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.487 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:11.511 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:11.512 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241030 2024-11-04 12:35:11.513 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.513 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241030 values less than (1730217600)) 2024-11-04 12:35:11.716 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.718 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:11.743 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:11.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241031 2024-11-04 12:35:11.745 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.746 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241031 values less than (1730304000)) 2024-11-04 12:35:11.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:11.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:11.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241101 2024-11-04 12:35:11.977 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:11.978 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241101 values less than (1730390400)) 2024-11-04 12:35:11.983 INFO 21062 --- [prometheus-minute-schedule-1,d4acc3205b8ba2e4,9e61c7b326900466] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:12.197 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.199 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:12.221 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:12.222 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241102 2024-11-04 12:35:12.223 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.223 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241102 values less than (1730476800)) 2024-11-04 12:35:12.438 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.439 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:12.463 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:12.464 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241103 2024-11-04 12:35:12.465 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.466 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241103 values less than (1730563200)) 2024-11-04 12:35:12.550 WARN 21062 --- [ocp-async-10,bd7a8afcf9bdd5cc,eac7ec08c25f038d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:12.550 INFO 21062 --- [ocp-async-2,ff3311bb6d16dbaf,1c748d4d05d34c87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:12.683 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.684 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:12.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:12.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241104 2024-11-04 12:35:12.709 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.710 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241104 values less than (1730649600)) 2024-11-04 12:35:12.928 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.929 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:12.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:12.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241105 2024-11-04 12:35:12.952 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:12.953 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241105 values less than (1730736000)) 2024-11-04 12:35:13.169 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.171 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:13.193 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:13.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241106 2024-11-04 12:35:13.195 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.196 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241106 values less than (1730822400)) 2024-11-04 12:35:13.416 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.418 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:13.441 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:13.442 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20241107 2024-11-04 12:35:13.443 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.444 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20241107 values less than (1730908800)) 2024-11-04 12:35:13.675 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.676 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2024-11-04 12:35:13.699 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2024-11-04 12:35:13.700 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.701 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:13.702 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.703 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 12:35:13.769 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2021 2024-11-04 12:35:13.770 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.771 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2021 values less than (1640995200)) 2024-11-04 12:35:13.947 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.948 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2024-11-04 12:35:13.972 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2024-11-04 12:35:13.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2022 2024-11-04 12:35:13.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:13.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2022 values less than (1672531200)) 2024-11-04 12:35:13.984 INFO 21062 --- [prometheus-cache-monitor1,15d7ce49161c6a1a,157df2d3431f0615] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":322,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:35:13.985 INFO 21062 --- [prometheus-cache-monitor1,15d7ce49161c6a1a,157df2d3431f0615] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:35:13.992 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:31:13.988, end=2024-11-04T12:33:13.988} 2024-11-04 12:35:14.010 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:35:14.012 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:35:14.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2024-11-04 12:35:14.182 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2024-11-04 12:35:14.183 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2023 2024-11-04 12:35:14.183 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.184 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2023 values less than (1704067200)) 2024-11-04 12:35:14.346 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:14.348 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:14.393 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.394 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2024-11-04 12:35:14.417 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2024-11-04 12:35:14.418 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2024 2024-11-04 12:35:14.419 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.420 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2024 values less than (1735689600)) 2024-11-04 12:35:14.550 WARN 21062 --- [ocp-async-4,399f8dab5d659a5d,84d0b45bd990cf51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:14.550 INFO 21062 --- [ocp-async-6,7101716e8a1a18bf,65b5a1b5418fec1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:14.603 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.604 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2024-11-04 12:35:14.627 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2024-11-04 12:35:14.628 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2025 2024-11-04 12:35:14.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.629 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2025 values less than (1767225600)) 2024-11-04 12:35:14.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.817 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2024-11-04 12:35:14.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2024-11-04 12:35:14.842 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.843 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:14.843 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.844 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 12:35:14.897 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2021 2024-11-04 12:35:14.898 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:14.898 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2021 values less than (1640995200)) 2024-11-04 12:35:14.983 INFO 21062 --- [prometheus-minute-schedule-1,da8c1259e1bd90d1,d39e1e4f8e91ef61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:15.072 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.074 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2024-11-04 12:35:15.098 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2024-11-04 12:35:15.098 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2022 2024-11-04 12:35:15.099 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.100 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2022 values less than (1672531200)) 2024-11-04 12:35:15.281 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.283 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2024-11-04 12:35:15.306 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2024-11-04 12:35:15.307 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2023 2024-11-04 12:35:15.308 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.308 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2023 values less than (1704067200)) 2024-11-04 12:35:15.494 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.496 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2024-11-04 12:35:15.519 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2024-11-04 12:35:15.521 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2024 2024-11-04 12:35:15.521 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.522 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2024 values less than (1735689600)) 2024-11-04 12:35:15.708 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.710 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2024-11-04 12:35:15.734 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2024-11-04 12:35:15.735 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2025 2024-11-04 12:35:15.735 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.736 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2025 values less than (1767225600)) 2024-11-04 12:35:15.924 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.925 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2024-11-04 12:35:15.949 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2024-11-04 12:35:15.950 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:15.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:15.951 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 12:35:16.023 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20241104 2024-11-04 12:35:16.023 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.024 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20241104 values less than (1730649600)) 2024-11-04 12:35:16.199 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.200 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2024-11-04 12:35:16.223 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2024-11-04 12:35:16.224 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20241105 2024-11-04 12:35:16.225 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.225 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20241105 values less than (1730736000)) 2024-11-04 12:35:16.407 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.408 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2024-11-04 12:35:16.432 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2024-11-04 12:35:16.433 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20241106 2024-11-04 12:35:16.433 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.434 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20241106 values less than (1730822400)) 2024-11-04 12:35:16.550 WARN 21062 --- [ocp-async-5,a30a2d6d36414f29,1128570a900aaeb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:16.550 INFO 21062 --- [ocp-async-1,68beffa3ad775729,9800d87e615f44b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:16.620 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.621 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2024-11-04 12:35:16.646 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2024-11-04 12:35:16.647 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20241107 2024-11-04 12:35:16.647 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.648 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20241107 values less than (1730908800)) 2024-11-04 12:35:16.832 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.833 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2024-11-04 12:35:16.857 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2024-11-04 12:35:16.858 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.858 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:16.859 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.860 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 12:35:16.889 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730044800 2024-11-04 12:35:16.889 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:16.890 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730044800 values less than (1730044800)) 2024-11-04 12:35:17.040 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.042 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:17.065 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:17.066 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730131200 2024-11-04 12:35:17.067 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.067 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730131200 values less than (1730131200)) 2024-11-04 12:35:17.218 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.219 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:17.242 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:17.243 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730217600 2024-11-04 12:35:17.243 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.244 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730217600 values less than (1730217600)) 2024-11-04 12:35:17.348 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:17.349 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:17.400 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.402 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:17.426 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:17.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730304000 2024-11-04 12:35:17.427 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.428 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730304000 values less than (1730304000)) 2024-11-04 12:35:17.585 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.587 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:17.610 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:17.611 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730390400 2024-11-04 12:35:17.611 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.612 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730390400 values less than (1730390400)) 2024-11-04 12:35:17.775 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.776 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:17.800 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:17.801 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730476800 2024-11-04 12:35:17.802 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.802 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730476800 values less than (1730476800)) 2024-11-04 12:35:17.960 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:17.983 INFO 21062 --- [prometheus-minute-schedule-1,0ce205ef592ca611,3b44fba2b63cb5b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:17.985 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:17.986 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730563200 2024-11-04 12:35:17.987 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:17.987 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730563200 values less than (1730563200)) 2024-11-04 12:35:18.149 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.150 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:18.173 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:18.174 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730649600 2024-11-04 12:35:18.175 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.176 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730649600 values less than (1730649600)) 2024-11-04 12:35:18.341 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.343 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:18.365 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:18.366 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730736000 2024-11-04 12:35:18.367 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.368 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730736000 values less than (1730736000)) 2024-11-04 12:35:18.531 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.532 INFO 21062 --- [ocp-async-9,c7871a79aab1f8b5,318606b2ccab7aee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:35:18.534 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:18.550 WARN 21062 --- [ocp-async-8,4fdc62f191c1aa19,c44678f0e34850c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:18.551 INFO 21062 --- [ocp-async-8,2a8178236a956a1e,9086ac3c6c3d8d93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:18.558 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:18.559 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730822400 2024-11-04 12:35:18.560 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.560 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730822400 values less than (1730822400)) 2024-11-04 12:35:18.571 INFO 21062 --- [ocp-schedules-3,161b7892fc903080,56b951efef2a57b3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:35:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:35:18.588910+08:00, lastProcessTime=2024-11-04T12:35:08.587406+08:00 2024-11-04 12:35:18.724 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:18.749 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:18.750 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1730908800 2024-11-04 12:35:18.751 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.752 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1730908800 values less than (1730908800)) 2024-11-04 12:35:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:35:18.588910+08:00 2024-11-04 12:35:18.921 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.922 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2024-11-04 12:35:18.945 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2024-11-04 12:35:18.947 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.948 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:18.948 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.949 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 12:35:18.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2021 2024-11-04 12:35:18.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:18.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2021 values less than (1640995200)) 2024-11-04 12:35:19.162 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.164 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2024-11-04 12:35:19.188 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2024-11-04 12:35:19.189 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2022 2024-11-04 12:35:19.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.190 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2022 values less than (1672531200)) 2024-11-04 12:35:19.408 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.410 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2024-11-04 12:35:19.434 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2024-11-04 12:35:19.435 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2023 2024-11-04 12:35:19.436 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.436 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2023 values less than (1704067200)) 2024-11-04 12:35:19.651 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2024-11-04 12:35:19.676 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2024-11-04 12:35:19.677 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2024 2024-11-04 12:35:19.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2024 values less than (1735689600)) 2024-11-04 12:35:19.768 INFO 21062 --- [ocp-schedules-11,cb3e73faf5af767e,3b40ab2b2cbd4c2d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:35:19.891 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.892 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2024-11-04 12:35:19.916 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2024-11-04 12:35:19.917 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2025 2024-11-04 12:35:19.918 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:19.918 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2025 values less than (1767225600)) 2024-11-04 12:35:20.133 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.134 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2024-11-04 12:35:20.156 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2024-11-04 12:35:20.158 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:20.159 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.160 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 12:35:20.203 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730131200 2024-11-04 12:35:20.203 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.204 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730131200 values less than (1730131200)) 2024-11-04 12:35:20.349 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:20.350 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:20.398 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.400 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:20.423 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:20.424 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730217600 2024-11-04 12:35:20.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.425 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730217600 values less than (1730217600)) 2024-11-04 12:35:20.551 WARN 21062 --- [ocp-async-7,aaca13ea448ebc49,00d3aa3c29b7867b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:20.551 INFO 21062 --- [ocp-async-6,6a8679fa04e97b58,f7870e1c03ed123a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:20.652 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.653 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:20.676 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:20.677 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730304000 2024-11-04 12:35:20.678 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.679 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730304000 values less than (1730304000)) 2024-11-04 12:35:20.908 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.910 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:20.934 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:20.935 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730390400 2024-11-04 12:35:20.935 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:20.936 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730390400 values less than (1730390400)) 2024-11-04 12:35:20.983 INFO 21062 --- [prometheus-minute-schedule-1,05893139340c51e3,bbcd008155c78ead] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:21.167 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.169 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:21.193 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:21.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730476800 2024-11-04 12:35:21.194 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.195 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730476800 values less than (1730476800)) 2024-11-04 12:35:21.429 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.430 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:21.454 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:21.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730563200 2024-11-04 12:35:21.455 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.456 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730563200 values less than (1730563200)) 2024-11-04 12:35:21.695 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.696 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:21.719 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:21.720 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730649600 2024-11-04 12:35:21.721 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.721 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730649600 values less than (1730649600)) 2024-11-04 12:35:21.962 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.963 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:21.986 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:21.987 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730736000 2024-11-04 12:35:21.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:21.988 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730736000 values less than (1730736000)) 2024-11-04 12:35:22.224 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.226 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:22.249 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:22.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730822400 2024-11-04 12:35:22.251 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.252 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730822400 values less than (1730822400)) 2024-11-04 12:35:22.501 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.503 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:22.530 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:22.531 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1730908800 2024-11-04 12:35:22.531 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.532 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1730908800 values less than (1730908800)) 2024-11-04 12:35:22.551 WARN 21062 --- [ocp-async-1,9ad8e44ac85a6bd9,b8363517433369bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:22.551 INFO 21062 --- [ocp-async-3,f75a72157011c4a6,cac938c3e3363da0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:22.787 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.788 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2024-11-04 12:35:22.813 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2024-11-04 12:35:22.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 12:35:22.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 12:35:22.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:35:22.818 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.818 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:22.819 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.820 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 12:35:22.851 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730131200 2024-11-04 12:35:22.851 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:22.852 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730131200 values less than (1730131200)) 2024-11-04 12:35:23.350 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:23.351 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:23.478 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:23.479 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:23.579 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:23.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730217600 2024-11-04 12:35:23.581 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:23.582 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730217600 values less than (1730217600)) 2024-11-04 12:35:23.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:23.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:23.859 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:23.860 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730304000 2024-11-04 12:35:23.860 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:23.861 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730304000 values less than (1730304000)) 2024-11-04 12:35:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8c4a1c0a81f7d440,464a821099f316d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:24.054 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.055 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:24.079 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:24.080 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730390400 2024-11-04 12:35:24.080 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.081 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730390400 values less than (1730390400)) 2024-11-04 12:35:24.276 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.277 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:24.301 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:24.302 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730476800 2024-11-04 12:35:24.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.303 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730476800 values less than (1730476800)) 2024-11-04 12:35:24.501 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.502 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:24.526 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:24.527 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730563200 2024-11-04 12:35:24.527 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.528 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730563200 values less than (1730563200)) 2024-11-04 12:35:24.551 WARN 21062 --- [ocp-async-2,0d7cd831b482ea28,92b6eeb49c56b218] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:24.551 INFO 21062 --- [ocp-async-10,c0105902312848c8,9b7d7fc46df06f4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:24.725 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.726 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:24.749 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:24.750 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730649600 2024-11-04 12:35:24.751 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.752 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730649600 values less than (1730649600)) 2024-11-04 12:35:24.949 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.950 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:24.973 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:24.974 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730736000 2024-11-04 12:35:24.975 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:24.976 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730736000 values less than (1730736000)) 2024-11-04 12:35:25.177 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.179 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:25.202 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:25.203 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730822400 2024-11-04 12:35:25.204 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.204 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730822400 values less than (1730822400)) 2024-11-04 12:35:25.404 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.406 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:25.430 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:25.431 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1730908800 2024-11-04 12:35:25.432 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.433 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1730908800 values less than (1730908800)) 2024-11-04 12:35:25.636 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.637 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2024-11-04 12:35:25.660 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2024-11-04 12:35:25.662 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.662 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:35:25.664 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.664 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 12:35:25.695 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730131200 2024-11-04 12:35:25.695 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.696 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730131200 values less than (1730131200)) 2024-11-04 12:35:25.886 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.887 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:25.911 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:25.912 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730217600 2024-11-04 12:35:25.912 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:25.913 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730217600 values less than (1730217600)) 2024-11-04 12:35:26.126 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.127 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:26.150 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:26.150 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730304000 2024-11-04 12:35:26.151 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.151 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730304000 values less than (1730304000)) 2024-11-04 12:35:26.352 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:26.353 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:26.363 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.365 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:26.390 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:26.391 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730390400 2024-11-04 12:35:26.391 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.392 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730390400 values less than (1730390400)) 2024-11-04 12:35:26.551 WARN 21062 --- [ocp-async-7,c63b19b36b512511,e9fc9ba640fec0f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:26.551 INFO 21062 --- [ocp-async-4,db242a4f354e2ad4,055b28d272920f9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:26.606 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.607 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:26.631 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:26.632 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730476800 2024-11-04 12:35:26.632 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.633 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730476800 values less than (1730476800)) 2024-11-04 12:35:26.844 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.845 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:26.869 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:26.870 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730563200 2024-11-04 12:35:26.871 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:26.872 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730563200 values less than (1730563200)) 2024-11-04 12:35:26.983 INFO 21062 --- [prometheus-minute-schedule-1,6793823059887085,8ce2c7483469a0db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ed9b22c23b51704a,57b000780f437744] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:35:27.088 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.089 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:27.114 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:27.115 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730649600 2024-11-04 12:35:27.115 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.116 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730649600 values less than (1730649600)) 2024-11-04 12:35:27.331 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.332 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:27.356 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:27.357 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730736000 2024-11-04 12:35:27.358 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.358 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730736000 values less than (1730736000)) 2024-11-04 12:35:27.575 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.576 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:27.598 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:27.599 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730822400 2024-11-04 12:35:27.600 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.601 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730822400 values less than (1730822400)) 2024-11-04 12:35:27.814 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.816 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:27.839 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:27.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1730908800 2024-11-04 12:35:27.841 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:27.842 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1730908800 values less than (1730908800)) 2024-11-04 12:35:28.099 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:35:28.101 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2024-11-04 12:35:28.126 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2024-11-04 12:35:28.127 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 12:35:28.136 INFO 21062 --- [subtask-executor1,dd193f8f67a5a423,b22f712462a5b02f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:35:28.498 INFO 21062 --- [ocp-schedules-29,77c0734794b293ba,98e1a6564fe45852] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:35:28.499 INFO 21062 --- [ocp-schedules-12,12f8d0127c5417da,4c94612bc4bbb830] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:35:28.500 INFO 21062 --- [ocp-schedules-21,6df3f2064ef3170e,a435fe59ca94dcfa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:35:28.500 INFO 21062 --- [ocp-schedules-28,d09dc1c653ed1b26,8e5d807240edb7ba] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:35:28.502 INFO 21062 --- [ocp-schedules-29,77c0734794b293ba,98e1a6564fe45852] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:35:28.502 INFO 21062 --- [ocp-schedules-2,f5e9d2e87efddc44,ab5b6831a0086393] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:35:28.503 INFO 21062 --- [ocp-schedules-21,6df3f2064ef3170e,a435fe59ca94dcfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:28.503 INFO 21062 --- [ocp-schedules-28,d09dc1c653ed1b26,8e5d807240edb7ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:35:28.504 INFO 21062 --- [ocp-schedules-12,12f8d0127c5417da,4c94612bc4bbb830] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:35:28.504 INFO 21062 --- [ocp-schedules-20,08b811d091ff64be,1a19bff9ac31418a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:35:28.510 INFO 21062 --- [ocp-schedules-6,a6c28320460fe615,b768dfe62acfac0e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:35:28.510 INFO 21062 --- [ocp-schedules-1,65f34f23b283a18b,4f1a5c354fa3280d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:35:28.510 INFO 21062 --- [ocp-schedules-1,65f34f23b283a18b,4f1a5c354fa3280d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:35:28.510 INFO 21062 --- [ocp-schedules-6,a6c28320460fe615,b768dfe62acfac0e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:35:28.511 INFO 21062 --- [ocp-schedules-26,3000c3a1c690fd16,0d8517b74a65b7d9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:35:28.511 INFO 21062 --- [ocp-schedules-26,3000c3a1c690fd16,0d8517b74a65b7d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:35:28.532 INFO 21062 --- [ocp-async-1,55533fb6d213cd82,0fa220cadbd49210] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:35:28.551 WARN 21062 --- [ocp-async-2,ba3baf61b015270d,28a6b15d0c4124a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:28.552 INFO 21062 --- [ocp-async-9,5ec2a3c6a8118d24,d8bb3eb77242969c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:28.566 INFO 21062 --- [ocp-schedules-5,8ec403ef60dba0c8,9f160a9d778298d0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:35:28.569 INFO 21062 --- [ocp-schedules-30,80aad7ded05728f4,656bc953b81c1f1c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:35:28.572 INFO 21062 --- [ocp-schedules-7,d74f7fc738d3b196,332420fe077a5b84] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:35:28.572 INFO 21062 --- [ocp-schedules-7,d74f7fc738d3b196,332420fe077a5b84] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:35:28.573 INFO 21062 --- [ocp-schedules-17,ff6ded651ce7a712,75a42f01cff253c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:35:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:35:28.588028+08:00, lastProcessTime=2024-11-04T12:35:18.588910+08:00 2024-11-04 12:35:28.602 INFO 21062 --- [ocp-schedules-20,b9228525891c7f5b,df8b8927041aae63] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:35:28.609 INFO 21062 --- [ocp-schedules-20,b9228525891c7f5b,df8b8927041aae63] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:35:28.609 INFO 21062 --- [ocp-schedules-20,b9228525891c7f5b,df8b8927041aae63] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:35:28.611 INFO 21062 --- [ocp-schedules-1,f400362c139561b8,cf537e06b79eecdf] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:35:28.611 INFO 21062 --- [ocp-schedules-1,f400362c139561b8,cf537e06b79eecdf] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:35:28.611 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:35:28.611 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:35:28.613 INFO 21062 --- [ocp-schedules-6,d457e48ec7afd7fe,b3d6469ccec8feba] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:35:28.613 INFO 21062 --- [ocp-schedules-6,d457e48ec7afd7fe,b3d6469ccec8feba] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:35:28.621 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:35:28.621 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:35:28.621 INFO 21062 --- [ocp-schedules-1,f400362c139561b8,cf537e06b79eecdf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:35:28.623 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:35:28.623 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:35:28.623 WARN 21062 --- [ocp-schedules-1,f400362c139561b8,cf537e06b79eecdf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:35:28.624 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:35:28.624 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:35:28.624 INFO 21062 --- [ocp-schedules-1,f400362c139561b8,cf537e06b79eecdf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:35:28.625 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:35:28.625 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:35:28.625 WARN 21062 --- [ocp-schedules-1,f400362c139561b8,cf537e06b79eecdf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:35:28.633 ERROR 21062 --- [ocp-schedules-1,f400362c139561b8,cf537e06b79eecdf] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:35:28.639 INFO 21062 --- [ocp-schedules-6,d457e48ec7afd7fe,b3d6469ccec8feba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:35:28.639 INFO 21062 --- [ocp-schedules-6,d457e48ec7afd7fe,b3d6469ccec8feba] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:35:28.653 INFO 21062 --- [ocp-schedules-11,72a14bb3016bfd2e,a95f074dadd56013] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:35:28.653 INFO 21062 --- [ocp-schedules-11,72a14bb3016bfd2e,a95f074dadd56013] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:35:28.653 INFO 21062 --- [ocp-schedules-29,abd7fa7cacbbd099,b6a0fd2bd5646d47] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:35:28.654 INFO 21062 --- [ocp-schedules-11,72a14bb3016bfd2e,a95f074dadd56013] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:35:28.654 INFO 21062 --- [ocp-schedules-11,72a14bb3016bfd2e,a95f074dadd56013] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:35:28.654 INFO 21062 --- [ocp-schedules-11,72a14bb3016bfd2e,a95f074dadd56013] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:35:28.654 INFO 21062 --- [ocp-schedules-11,72a14bb3016bfd2e,a95f074dadd56013] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908335011581952 2024-11-04 12:35:28.655 INFO 21062 --- [ocp-schedules-26,1f8f939552e1c601,4e885ff303cf606f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:35:28.656 INFO 21062 --- [ocp-schedules-11,72a14bb3016bfd2e,a95f074dadd56013] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=709908395925458944 2024-11-04 12:35:28.659 INFO 21062 --- [ocp-schedules-29,abd7fa7cacbbd099,b6a0fd2bd5646d47] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:35:28.659 INFO 21062 --- [ocp-schedules-29,abd7fa7cacbbd099,b6a0fd2bd5646d47] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:35:28.659 INFO 21062 --- [ocp-schedules-23,f03f557194e60a93,ddccd5c8fe964faa] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:35:28.659+08:00, clusters=[1] 2024-11-04 12:35:28.719 INFO 21062 --- [ocp-schedules-23,f03f557194e60a93,ddccd5c8fe964faa] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:35:28.764 INFO 21062 --- [ocp-schedules-10,190bfd6ba8dae906,d97ae0dc1f6c772b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:35:28.764 INFO 21062 --- [ocp-schedules-10,ddc5496cd03f744a,abd6cd1623961f11] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:35:28.767 INFO 21062 --- [ocp-schedules-10,ddc5496cd03f744a,abd6cd1623961f11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:28.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:35:28.588028+08:00 2024-11-04 12:35:28.784 INFO 21062 --- [ocp-schedules-27,bb58ee3631eb6370,2eba9c29bc5161c7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:35:28.784+08:00, clusters=[1] 2024-11-04 12:35:28.788 INFO 21062 --- [ocp-schedules-27,bb58ee3631eb6370,2eba9c29bc5161c7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:35:28.788 INFO 21062 --- [ocp-schedules-13,33673a3c320e0fce,cbcfdd20c937a444] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:35:28.791 INFO 21062 --- [ocp-schedules-13,33673a3c320e0fce,cbcfdd20c937a444] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:28.792 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:35:28.792 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:35:28.872 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:35:28.875 ERROR 21062 --- [ocp-schedules-13,33673a3c320e0fce,cbcfdd20c937a444] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:35:28.875 INFO 21062 --- [ocp-schedules-13,33673a3c320e0fce,cbcfdd20c937a444] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:35:28.993 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1, status:SUCCESS 2024-11-04 12:35:28.997 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1, status:SUCCESS 2024-11-04 12:35:29.353 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:29.354 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:29.795 INFO 21062 --- [ocp-schedules-5,921bcda554aba4c3,2ce4db2eb989abd6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:35:29.983 INFO 21062 --- [prometheus-minute-schedule-1,bb3d0869a1f72e1f,1700d784d40bccc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:35:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:35:30.358 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:35:30.358 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:35:30.367 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:35:30.367 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:35:30.367 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:35:30.367 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:35:30.374 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:35:30.374 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:35:30.374 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:35:30.375 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:35:30.414 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:35:30.425 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:35:30.425 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:35:30.426 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:35:30.426 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:35:30.426 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:35:30.434 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:35:30.552 WARN 21062 --- [ocp-async-7,1a1672e9959f5383,2adb652953d81693] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:30.552 INFO 21062 --- [ocp-async-7,bf621ea05869116e,4f3fc958be1a18f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:31.008 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:35:31.492 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.117780892S 2024-11-04 12:35:31.493 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:35:31.564 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:35:31.564 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:35:31.593 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:35:31.644 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:35:31.645 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:35:31.645 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:35:31.645 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:35:31.645 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:35:31.645 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:35:31.653 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:35:31.816 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] o.s.boot.SpringApplication : Started application in 1.449 seconds (JVM running for 345.495) 2024-11-04 12:35:31.816 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.449391846S 2024-11-04 12:35:32.310 INFO 21062 --- [ocp-schedules-15,8fc3bf029cb82ad5,4afd6850162c65ae] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:35:32.354 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:32.356 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:32.552 WARN 21062 --- [ocp-async-6,c10259717a198612,8351a36d9d45420f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:32.552 INFO 21062 --- [ocp-async-4,078cd62ff3ca2471,24d190e1c0b4bc01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:32.983 INFO 21062 --- [prometheus-minute-schedule-1,cd00062426773837,55d0c967f31abfac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:34.552 WARN 21062 --- [ocp-async-5,85cfe2fa5dee4b13,ab32228a1671f92a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:34.552 INFO 21062 --- [ocp-async-2,49cc03eaff80d222,b50a623ccbe1ccc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:35.356 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:35.357 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:35.983 INFO 21062 --- [prometheus-minute-schedule-1,51eb1de31a20cdba,764db192e9895f20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:36.552 WARN 21062 --- [ocp-async-8,38084237573f3b7a,e2dcdf3b6e09ef7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:36.552 INFO 21062 --- [ocp-async-9,f1a9ae1a6e39b34c,93acfc46284ffb53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:38.357 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:38.358 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:38.504 INFO 21062 --- [ocp-schedules-28,d09dc1c653ed1b26,8e5d807240edb7ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:35:38.505 INFO 21062 --- [ocp-schedules-21,6df3f2064ef3170e,a435fe59ca94dcfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:35:38.532 INFO 21062 --- [ocp-async-6,c7eb9045ed05dee2,3f4b55afb18975fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:35:38.552 WARN 21062 --- [ocp-async-4,aba457f8bde353f2,ea0b43ba5ba47482] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:38.552 INFO 21062 --- [ocp-async-5,646aedffdabc86bf,d3ee996952e79cf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:38.576 INFO 21062 --- [ocp-schedules-25,46875dbb1aeae175,d99d9e8028828b67] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:35:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:35:38.588216+08:00, lastProcessTime=2024-11-04T12:35:28.588028+08:00 2024-11-04 12:35:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:35:38.588216+08:00 2024-11-04 12:35:38.983 INFO 21062 --- [prometheus-minute-schedule-1,fb8be2f59e4b68f4,a6837c3baea2d38c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:39.904 INFO 21062 --- [ocp-schedules-9,7df0cb1d1a7faa72,d6d0049af0e1942d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:35:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:35:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:35:40.553 WARN 21062 --- [ocp-async-2,f72a488ecea08dc7,c2025f205f2bf731] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:40.553 INFO 21062 --- [ocp-async-8,bc2065bc47d54e4f,43b727a0df5fbcc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:41.358 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:41.359 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:41.983 INFO 21062 --- [prometheus-minute-schedule-1,315f5463da2919f9,afa0c70f86139292] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:42.553 WARN 21062 --- [ocp-async-10,06697c7fc1144422,979ba9e5f440cb8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:42.553 INFO 21062 --- [ocp-async-9,213500878f2bc236,c720d3f7102db5d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:44.360 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:44.361 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:44.553 WARN 21062 --- [ocp-async-4,bb2020ac9155aaec,9787aea83fea2412] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:44.553 INFO 21062 --- [ocp-async-7,f04e2c6586e3d8e3,9666603b77c10b7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:44.983 INFO 21062 --- [prometheus-minute-schedule-1,2159f952bccc5680,d7561fa5b0e48ac3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:46.553 WARN 21062 --- [ocp-async-2,31941b0a83a32f3f,67b3e664aa757a8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:46.553 INFO 21062 --- [ocp-async-1,8dfeee1f932fc951,f9da5d5fd3529b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:47.361 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:47.362 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:47.983 INFO 21062 --- [prometheus-minute-schedule-1,89512000f1ce6673,bc20d186e926f6d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:48.532 INFO 21062 --- [ocp-async-8,79a1eade69e6bbc1,7d786e595a52a212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:35:48.553 WARN 21062 --- [ocp-async-3,d2f3516be354df30,79d546323c9a6927] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:48.553 INFO 21062 --- [ocp-async-3,a691fead7ff62d1d,8c0f6e01ed406c1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:48.557 INFO 21062 --- [ocp-schedules-2,e08a3e34fca2e28a,2b6d454e443abae5] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:35:48.560 INFO 21062 --- [ocp-schedules-2,e08a3e34fca2e28a,2b6d454e443abae5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:48.561 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:35:48.579 INFO 21062 --- [ocp-schedules-9,c92dbeb97e810e19,b6444a4408aef66c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:35:48.595 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:35:48.593763+08:00, lastProcessTime=2024-11-04T12:35:38.588216+08:00 2024-11-04 12:35:48.661 INFO 21062 --- [ocp-schedules-15,4bece7a14779f711,2f0329f441efd89b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:35:48.667 INFO 21062 --- [ocp-schedules-15,4bece7a14779f711,2f0329f441efd89b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:35:48.667 INFO 21062 --- [ocp-schedules-15,4bece7a14779f711,2f0329f441efd89b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:35:48.779 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:35:48.593763+08:00 2024-11-04 12:35:49.934 INFO 21062 --- [ocp-schedules-7,b0ef1fe2d37b94f4,77ea4fe4f451a98a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:35:50.362 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:50.364 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:50.554 WARN 21062 --- [ocp-async-6,bb401527faa81b64,3420edaf74f30d5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:50.554 INFO 21062 --- [ocp-async-7,a0a99ca79148725a,ad600e460fbd55c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8379e329e1d488e1,a6c6b3db8a63308b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:52.554 WARN 21062 --- [ocp-async-5,e5638cce1435fdb3,3af295d063970576] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:52.554 INFO 21062 --- [ocp-async-1,7ca8996ee17ea4a7,ef92def392b8b366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:53.364 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:53.365 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:53.983 INFO 21062 --- [prometheus-minute-schedule-1,3f01ec4c9bd5fb44,e32d885ae9dc5334] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:54.554 WARN 21062 --- [ocp-async-9,6fcffd744f045b23,9d086022846aecca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:54.554 INFO 21062 --- [ocp-async-10,838279e871e7d7ba,b31a907c6fcecc4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:56.365 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:56.366 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:56.554 WARN 21062 --- [ocp-async-6,d6de54242b0cb152,5785ff1d2bb7f4cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:56.554 INFO 21062 --- [ocp-async-4,25d827a0b5140519,d82c757b4906b571] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:56.983 INFO 21062 --- [prometheus-minute-schedule-1,18fbe4da4180d214,7378444e92062da0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:58.500 INFO 21062 --- [ocp-schedules-14,11f17f7167321b18,2573c1a1cc88349c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:35:58.500 INFO 21062 --- [ocp-schedules-29,f1b20de347713d8c,936fef70fc9b9e27] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:35:58.503 INFO 21062 --- [ocp-schedules-16,b926478710de3f37,38beead100d44a12] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:35:58.504 INFO 21062 --- [ocp-schedules-14,11f17f7167321b18,2573c1a1cc88349c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:58.505 INFO 21062 --- [ocp-schedules-29,f1b20de347713d8c,936fef70fc9b9e27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:35:58.505 INFO 21062 --- [ocp-schedules-8,a555932f5d6c916e,8138ebcf663f4db3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:35:58.514 INFO 21062 --- [ocp-schedules-25,21209e6782e3f853,74c187f464ece2dc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:35:58.514 INFO 21062 --- [ocp-schedules-15,a90fba485c41dca2,8ba2b640dded0dce] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:35:58.514 INFO 21062 --- [ocp-schedules-11,8c5aa77e17b31013,8a8bf3b634a2d200] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:35:58.514 INFO 21062 --- [ocp-schedules-15,a90fba485c41dca2,8ba2b640dded0dce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:35:58.514 INFO 21062 --- [ocp-schedules-25,21209e6782e3f853,74c187f464ece2dc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:35:58.514 INFO 21062 --- [ocp-schedules-11,8c5aa77e17b31013,8a8bf3b634a2d200] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:35:58.533 INFO 21062 --- [ocp-async-4,5dd45564050469f6,98d7d10e65918305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:35:58.555 WARN 21062 --- [ocp-async-2,bb21974449378910,3b643472da134a5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:35:58.555 INFO 21062 --- [ocp-async-1,2e6d66ccc3ba26c9,b9bfae48d8eb7d60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:35:58.562 INFO 21062 --- [ocp-schedules-2,e08a3e34fca2e28a,2b6d454e443abae5] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:35:58.571 INFO 21062 --- [ocp-schedules-18,023481923e5bcebe,dc42b3b42af7917b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:35:58.581 INFO 21062 --- [ocp-schedules-17,e7c42d8a1502f656,059ecc6f018d897a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:35:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:35:58.588353+08:00, lastProcessTime=2024-11-04T12:35:48.593763+08:00 2024-11-04 12:35:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:35:58.588353+08:00 2024-11-04 12:35:58.792 INFO 21062 --- [ocp-schedules-10,ddc5496cd03f744a,abd6cd1623961f11] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:35:58.792 INFO 21062 --- [ocp-schedules-10,953a769dff9d04da,e90ef97d09d350db] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:35:58.796 INFO 21062 --- [ocp-schedules-10,953a769dff9d04da,e90ef97d09d350db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:35:58.824 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:35:58.824 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:35:59.366 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:35:59.367 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:35:59.983 INFO 21062 --- [prometheus-minute-schedule-1,88aa47bb7ac4717f,9a0e4908d347ab88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730694960, interval=60 2024-11-04 12:35:59.987 INFO 21062 --- [ocp-schedules-3,48a2c18679aa7c1d,6b23d78f4dcdb5f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:36:00.555 WARN 21062 --- [ocp-async-8,feac70cd4bfca74b,3db2cfc93879c2ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:00.555 INFO 21062 --- [ocp-async-8,460aa8ecfc9822cc,0ce8371bd5a25018] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:02.367 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:02.369 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:02.555 WARN 21062 --- [ocp-async-6,c187eaf4b6a1c04a,c2a32f2f86968370] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:02.555 INFO 21062 --- [ocp-async-3,9e590bb77cd73b22,aee4b0741387f2ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f325480766f2f4cd,ce22a33e81f6cde9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:04.555 WARN 21062 --- [ocp-async-7,ff9f1b0bc346dd7e,1b1245973c218102] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:04.555 INFO 21062 --- [ocp-async-2,16bdf3d6fa7329d6,2c1a6f54923dd91f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:05.369 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:05.370 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a775161142f0e732,bfed9c91ed19ebbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:06.555 WARN 21062 --- [ocp-async-9,33bcca3fe89be063,44957d5a053fe359] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:06.555 INFO 21062 --- [ocp-async-10,5cb94366240a98f8,59ab614905168874] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:08.370 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:08.371 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:08.506 INFO 21062 --- [ocp-schedules-29,f1b20de347713d8c,936fef70fc9b9e27] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:36:08.506 INFO 21062 --- [ocp-schedules-14,11f17f7167321b18,2573c1a1cc88349c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:36:08.533 INFO 21062 --- [ocp-async-10,5a538e43219be75a,89d18ccb627ccf0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:36:08.555 WARN 21062 --- [ocp-async-4,6b0b94d835af5e49,a4972697bc8c1697] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:08.555 INFO 21062 --- [ocp-async-8,deae256a26d52dfe,e75001d42119fd8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:08.584 INFO 21062 --- [ocp-schedules-9,e86248de5a188a37,b4a4472c0c3f32ed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:36:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:36:08.588676+08:00, lastProcessTime=2024-11-04T12:35:58.588353+08:00 2024-11-04 12:36:08.669 INFO 21062 --- [ocp-schedules-26,56a92ad6beac343c,2eb68090500390f5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:36:08.675 INFO 21062 --- [ocp-schedules-26,56a92ad6beac343c,2eb68090500390f5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:36:08.675 INFO 21062 --- [ocp-schedules-26,56a92ad6beac343c,2eb68090500390f5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:36:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:36:08.588676+08:00 2024-11-04 12:36:08.983 INFO 21062 --- [prometheus-minute-schedule-1,58de6d188b19b66c,182557fcbe15d8cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:10.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:36:10.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:36:10.018 INFO 21062 --- [ocp-schedules-3,a9728cc813f94729,9a4efb6953dc835d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:36:10.307 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:36:10.308 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:36:10.308 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:36:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:36:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:36:10.327 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:36:10.555 WARN 21062 --- [ocp-async-3,144831241da06734,2a7d134b0bab9ed5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:10.556 INFO 21062 --- [ocp-async-5,46da15a88e4d8c14,35ab9d42d9f49345] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:11.045 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=2, affectRows=2 2024-11-04 12:36:11.045 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=9, affectRows=9 2024-11-04 12:36:11.045 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=1, validCount=1 2024-11-04 12:36:11.045 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=8, validCount=8 2024-11-04 12:36:11.046 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=20, affectRows=20 2024-11-04 12:36:11.046 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=20, validCount=20 2024-11-04 12:36:11.046 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=1, affectRows=1 2024-11-04 12:36:11.046 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=1, validCount=1 2024-11-04 12:36:11.046 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=37, affectRows=37 2024-11-04 12:36:11.046 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=14, affectRows=14 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=14, validCount=14 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=37, validCount=37 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=40, affectRows=40 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=49, affectRows=49 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=39, validCount=39 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=44, affectRows=44 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=49, validCount=49 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=39, affectRows=39 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694900, nodeCount=67, affectRows=67 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=44, validCount=44 2024-11-04 12:36:11.047 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=39, validCount=39 2024-11-04 12:36:11.048 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694900, nodeCount=67, validCount=67 2024-11-04 12:36:11.371 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:11.372 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:11.983 INFO 21062 --- [prometheus-minute-schedule-1,19ff61b0111dccaf,4907d25c975b93b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:12.556 WARN 21062 --- [ocp-async-2,053f2f83565eee81,09106206871b76ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:12.556 INFO 21062 --- [ocp-async-1,ff566d324001548b,032020b4da9863a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:13.984 INFO 21062 --- [prometheus-cache-monitor1,8abea77f566c9675,60072768b0cbe639] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":644,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:36:13.984 INFO 21062 --- [prometheus-cache-monitor1,8abea77f566c9675,60072768b0cbe639] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:36:14.372 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:14.373 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:14.556 WARN 21062 --- [ocp-async-4,06452e437f9a38a6,9f8da740d3f55a4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:14.556 INFO 21062 --- [ocp-async-6,dc5adb0f53512ec8,9faf239ff5b6f954] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f47cc5e77e2c26f1,bbd92ead31afa4f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:16.556 WARN 21062 --- [ocp-async-3,884c8981cc3960c1,31763d731b19e647] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:16.556 INFO 21062 --- [ocp-async-7,37cdabda18bfb31a,2676dae78b633651] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:17.373 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:17.374 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:17.983 INFO 21062 --- [prometheus-minute-schedule-1,3e09867c6a1c0665,6fd747af59ecf86c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:18.533 INFO 21062 --- [ocp-async-5,db78106822664a71,b550c03151d8746f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:36:18.556 WARN 21062 --- [ocp-async-9,a28dc55cbbad57ec,0db6c8db60f0969a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:18.556 INFO 21062 --- [ocp-async-1,73ecd165f26ef909,fd17c893dfcd1f9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:18.587 INFO 21062 --- [ocp-schedules-6,291d04a2a97c850d,dc9bce0977f13e18] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:36:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:36:18.588033+08:00, lastProcessTime=2024-11-04T12:36:08.588676+08:00 2024-11-04 12:36:18.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:36:18.588033+08:00 2024-11-04 12:36:20.065 INFO 21062 --- [ocp-schedules-9,2b5da53a1dfb4d28,76bf6795e2274778] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:36:20.375 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:20.376 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:20.556 WARN 21062 --- [ocp-async-10,d4207484f330b550,9281fdbe571deba0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:20.557 INFO 21062 --- [ocp-async-6,f2b9e819f64d79de,5fc20759452fe8a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:20.983 INFO 21062 --- [prometheus-minute-schedule-1,09310427d4349ded,a564070d91c58881] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:22.557 WARN 21062 --- [ocp-async-7,253ce79ee78e3381,ec759faa0e23c695] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:22.557 INFO 21062 --- [ocp-async-8,ced7369a8c096c80,75db3585044ff4a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:23.376 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:23.377 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:23.983 INFO 21062 --- [prometheus-minute-schedule-1,185ef93611140ad7,6458b44fe93334bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:24.557 WARN 21062 --- [ocp-async-9,04c27c605eb93e41,9744371d371e3fdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:24.557 INFO 21062 --- [ocp-async-2,291b9b343a3032d5,15a329f32b36c6da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:26.378 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:26.379 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:26.557 WARN 21062 --- [ocp-async-10,040870443c812b26,2462bb15d1ba9b2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:26.557 INFO 21062 --- [ocp-async-4,4736e5c3030d8ff0,d68b1d95cfcd5724] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ca865ea816be022d,768030038409d3a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:26.984 INFO 21062 --- [prometheus-hour-schedule-1,55d29c6c2eeecc62,46d6697e2deebb46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:36:28.498 INFO 21062 --- [ocp-schedules-29,46170fa3fd023949,2a93b79f948b5c97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:36:28.499 INFO 21062 --- [ocp-schedules-3,65aaad437e430607,bfdaa1e60dc2569f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:36:28.500 INFO 21062 --- [ocp-schedules-9,c83a8671c3e4016f,63554ddbea66fa71] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:36:28.501 INFO 21062 --- [ocp-schedules-5,0f57c32c021d3dfb,e13777ed11899c01] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:36:28.502 INFO 21062 --- [ocp-schedules-29,46170fa3fd023949,2a93b79f948b5c97] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:36:28.502 INFO 21062 --- [ocp-schedules-13,a67c5f0a11f49ca7,d22d919430f59801] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:36:28.504 INFO 21062 --- [ocp-schedules-3,65aaad437e430607,bfdaa1e60dc2569f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:28.505 INFO 21062 --- [ocp-schedules-9,c83a8671c3e4016f,63554ddbea66fa71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:36:28.505 INFO 21062 --- [ocp-schedules-22,27fe69b1fe1813ff,9dc17967aa2b7cad] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:36:28.507 INFO 21062 --- [ocp-schedules-5,0f57c32c021d3dfb,e13777ed11899c01] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:36:28.511 INFO 21062 --- [ocp-schedules-18,d5865c28d7c2aacb,6893898765553fca] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:36:28.511 INFO 21062 --- [ocp-schedules-21,209b0d922aab4d8d,6c9168629b96a614] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:36:28.511 INFO 21062 --- [ocp-schedules-21,209b0d922aab4d8d,6c9168629b96a614] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:36:28.511 INFO 21062 --- [ocp-schedules-18,d5865c28d7c2aacb,6893898765553fca] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:36:28.511 INFO 21062 --- [ocp-schedules-26,6d7f46dbc6ec53f3,9c6cc15e345c3622] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:36:28.511 INFO 21062 --- [ocp-schedules-26,6d7f46dbc6ec53f3,9c6cc15e345c3622] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:36:28.533 INFO 21062 --- [ocp-async-3,92e4f188e7f59655,2a0e145c812a017d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:36:28.552 INFO 21062 --- [ocp-schedules-30,80c157f7caaeaaaa,70c6a9b1bb88c477] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":644,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":320,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:36:28.552 INFO 21062 --- [ocp-schedules-30,80c157f7caaeaaaa,70c6a9b1bb88c477] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:36:28.557 WARN 21062 --- [ocp-async-9,98229766c48a73c0,3419fa6dd2ced417] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:28.557 INFO 21062 --- [ocp-async-5,cd6bb0c23faed9e8,01062e84b1ffc918] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:28.562 INFO 21062 --- [ocp-schedules-20,0de90b2c1d757dec,1f1e899ae9121f9d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:36:28.564 INFO 21062 --- [ocp-schedules-20,0de90b2c1d757dec,1f1e899ae9121f9d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:36:28.569 INFO 21062 --- [ocp-schedules-15,3d842243b6bddaab,2b3dee34796f57a5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:36:28.573 INFO 21062 --- [ocp-schedules-17,1a48d882aec43b85,b97a8830f6e958b8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:36:28.576 INFO 21062 --- [ocp-schedules-8,0202285f4ec3b3cd,0a9a705d288b191c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:36:28.576 INFO 21062 --- [ocp-schedules-8,0202285f4ec3b3cd,0a9a705d288b191c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:36:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:36:28.588856+08:00, lastProcessTime=2024-11-04T12:36:18.588033+08:00 2024-11-04 12:36:28.590 INFO 21062 --- [ocp-schedules-6,0e47328060b0533c,ac9fc4822a82bb47] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:36:28.605 INFO 21062 --- [ocp-schedules-21,e3f89df2746c9157,deb8055633df37df] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:36:28.609 INFO 21062 --- [ocp-schedules-21,e3f89df2746c9157,deb8055633df37df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:28.609 INFO 21062 --- [ocp-schedules-18,eeae8de81d66e1c3,dc34290f6144735d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:36:28.615 INFO 21062 --- [ocp-schedules-18,eeae8de81d66e1c3,dc34290f6144735d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:36:28.615 INFO 21062 --- [ocp-schedules-18,eeae8de81d66e1c3,dc34290f6144735d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:36:28.639 INFO 21062 --- [ocp-schedules-14,435c397ea34cd8a0,9a9d392fb0daf1cc] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:36:28.639 INFO 21062 --- [ocp-schedules-14,435c397ea34cd8a0,9a9d392fb0daf1cc] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:36:28.641 INFO 21062 --- [ocp-schedules-26,c124b069e8b491ba,377b1d6c77d349b5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:36:28.641 INFO 21062 --- [ocp-schedules-26,c124b069e8b491ba,377b1d6c77d349b5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:36:28.641 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:36:28.641 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:36:28.651 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:36:28.651 INFO 21062 --- [ocp-schedules-26,c124b069e8b491ba,377b1d6c77d349b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:36:28.652 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:36:28.653 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:36:28.653 WARN 21062 --- [ocp-schedules-26,c124b069e8b491ba,377b1d6c77d349b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:36:28.653 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:36:28.654 INFO 21062 --- [ocp-schedules-26,c124b069e8b491ba,377b1d6c77d349b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:36:28.654 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:36:28.654 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:36:28.655 WARN 21062 --- [ocp-schedules-26,c124b069e8b491ba,377b1d6c77d349b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:36:28.656 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:36:28.656 INFO 21062 --- [ocp-schedules-23,1837513bb673c5c6,7729b122561ac0e8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:36:28.656 INFO 21062 --- [ocp-schedules-23,1837513bb673c5c6,7729b122561ac0e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:36:28.657 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:36:28.662 ERROR 21062 --- [ocp-schedules-26,c124b069e8b491ba,377b1d6c77d349b5] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2024-11-04 12:36:28.663 INFO 21062 --- [ocp-schedules-14,435c397ea34cd8a0,9a9d392fb0daf1cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:36:28.663 INFO 21062 --- [ocp-schedules-14,435c397ea34cd8a0,9a9d392fb0daf1cc] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:36:28.663 INFO 21062 --- [ocp-schedules-23,1837513bb673c5c6,7729b122561ac0e8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:36:28.663 INFO 21062 --- [ocp-schedules-23,1837513bb673c5c6,7729b122561ac0e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:36:28.668 ERROR 21062 --- [ocp-schedules-21,e3f89df2746c9157,deb8055633df37df] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:36:28.668 INFO 21062 --- [ocp-schedules-21,e3f89df2746c9157,deb8055633df37df] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:36:28.671 INFO 21062 --- [ocp-schedules-23,1837513bb673c5c6,7729b122561ac0e8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:36:28.671 INFO 21062 --- [ocp-schedules-23,1837513bb673c5c6,7729b122561ac0e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:36:28.677 INFO 21062 --- [ocp-schedules-16,fb0b71c18d0ebcfd,4eef23d385f52dd4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:36:28.683 INFO 21062 --- [ocp-schedules-16,fb0b71c18d0ebcfd,4eef23d385f52dd4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:36:28.683 INFO 21062 --- [ocp-schedules-16,fb0b71c18d0ebcfd,4eef23d385f52dd4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:36:28.687 INFO 21062 --- [ocp-schedules-29,781b7e854a8e29f9,964bfb469702e6ff] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:36:28.747 INFO 21062 --- [ocp-schedules-4,b5a85dacd94880c9,1ded096f2bdd4a88] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:36:28.747+08:00, clusters=[1] 2024-11-04 12:36:28.755 INFO 21062 --- [ocp-schedules-4,b5a85dacd94880c9,1ded096f2bdd4a88] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:36:28.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:36:28.588856+08:00 2024-11-04 12:36:28.815 INFO 21062 --- [ocp-schedules-1,7f975c266114db88,fc4cf07049216ea2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:36:28.815+08:00, clusters=[1] 2024-11-04 12:36:28.818 INFO 21062 --- [ocp-schedules-1,7f975c266114db88,fc4cf07049216ea2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:36:28.824 INFO 21062 --- [ocp-schedules-10,953a769dff9d04da,e90ef97d09d350db] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:36:28.825 INFO 21062 --- [ocp-schedules-10,37743cf50e81e39f,9b1b66360517129f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:36:28.828 INFO 21062 --- [ocp-schedules-10,37743cf50e81e39f,9b1b66360517129f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:28.854 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:36:28.854 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:36:28.876 INFO 21062 --- [ocp-schedules-12,904df5e6cb54524b,ddff174f4c2f8d68] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:36:28.879 INFO 21062 --- [ocp-schedules-12,904df5e6cb54524b,ddff174f4c2f8d68] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:28.965 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:36:28.967 ERROR 21062 --- [ocp-schedules-12,904df5e6cb54524b,ddff174f4c2f8d68] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:36:28.967 INFO 21062 --- [ocp-schedules-12,904df5e6cb54524b,ddff174f4c2f8d68] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:36:29.379 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:29.380 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:29.983 INFO 21062 --- [prometheus-minute-schedule-1,10e5a6e32118a8f7,bead095b1aada953] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:30.091 INFO 21062 --- [ocp-schedules-5,5412861a1dea42aa,5a02f08a11f313ab] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:36:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:36:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:36:30.558 WARN 21062 --- [ocp-async-10,de193018752989e6,610e6fa08940dfc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:30.558 INFO 21062 --- [ocp-async-1,b59dded4c4138a20,fad82fba01dfa3c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:32.380 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:32.381 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:32.558 WARN 21062 --- [ocp-async-7,69aa3d34f39e8a15,4fbfae3e7b976293] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:32.558 INFO 21062 --- [ocp-async-4,228a6e22330aef6a,085b86fb14f158b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:32.983 INFO 21062 --- [prometheus-minute-schedule-1,9edd6609330f425c,f62122f1b37d71aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:34.558 WARN 21062 --- [ocp-async-9,4d0edc74ae332bb5,b7dc46f5a558a111] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:34.558 INFO 21062 --- [ocp-async-8,09547993733e08de,d1e65dbc59191b8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:35.381 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:35.383 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:35.983 INFO 21062 --- [prometheus-minute-schedule-1,389a814089a3a2bb,b6644da171057a3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:36.558 WARN 21062 --- [ocp-async-10,ae44b00197fec6bd,c7cf9c9b607c82a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:36.558 INFO 21062 --- [ocp-async-2,9933734be823b178,3b2be64b4d5fdc34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:38.383 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:38.386 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:38.505 INFO 21062 --- [ocp-schedules-3,65aaad437e430607,bfdaa1e60dc2569f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:36:38.505 INFO 21062 --- [ocp-schedules-9,c83a8671c3e4016f,63554ddbea66fa71] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:36:38.533 INFO 21062 --- [ocp-async-7,863674adfe22879b,32ccc62aeb61f3ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:36:38.558 WARN 21062 --- [ocp-async-9,39646a744513acb2,b12ee5ffca659db0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:38.558 INFO 21062 --- [ocp-async-4,4e3c2c1346b8153c,bcbb4f272afe3610] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:36:38.587408+08:00, lastProcessTime=2024-11-04T12:36:28.588856+08:00 2024-11-04 12:36:38.593 INFO 21062 --- [ocp-schedules-5,454a37fca17334c5,3eddd99e2095a2aa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:36:38.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:36:38.587408+08:00 2024-11-04 12:36:38.983 INFO 21062 --- [prometheus-minute-schedule-1,2052355b06023c0d,9f841594800aee2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:40.004 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:36:40.006 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:36:40.136 INFO 21062 --- [ocp-schedules-5,95a4cc3cdd0af054,f3d469f18a0e5dd1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:36:40.559 WARN 21062 --- [ocp-async-8,651b1cbd60256a95,5c76a7c79cac2adf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:40.559 INFO 21062 --- [ocp-async-10,d4cffd0edea2cffa,9584a2d624f876d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:41.386 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:41.387 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:36:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:36:41.831 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:36:41.831 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:36:41.832 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:36:41.859 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:36:41.859 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 12:36:41.983 INFO 21062 --- [prometheus-minute-schedule-1,27d458bc46b81b31,33da297a7f10101f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:42.559 WARN 21062 --- [ocp-async-1,efb17a049b46bd58,624cac481ec8cbce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:42.559 INFO 21062 --- [ocp-async-2,7ea63daa83ae296b,b31a7f9221b06909] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:44.387 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:44.388 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:44.559 WARN 21062 --- [ocp-async-9,b26efb309e75853a,bba339ba02c57ea3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:44.559 INFO 21062 --- [ocp-async-6,24a74754d15b71bd,35f128f957c2eb49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3ca4691fc4a91049,f4536bdbc1458d41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:46.559 WARN 21062 --- [ocp-async-8,b65a91bdfa6c8a25,efc8a048d56fdd68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:46.559 INFO 21062 --- [ocp-async-3,18e67b1d0824e397,1c505745cd2337fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:47.389 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:47.390 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:47.983 INFO 21062 --- [prometheus-minute-schedule-1,ffc2215086ebc683,76ddb651105538bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:48.534 INFO 21062 --- [ocp-async-3,6674ff070d3e6737,9af2bef7099ed432] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:36:48.559 WARN 21062 --- [ocp-async-5,c5aefcad9fc3442e,55479be167fe7154] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:48.559 INFO 21062 --- [ocp-async-2,fcfb506056dee9cb,edf58f25017ee36d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:36:48.587186+08:00, lastProcessTime=2024-11-04T12:36:38.587408+08:00 2024-11-04 12:36:48.595 INFO 21062 --- [ocp-schedules-23,8566090511dcb69e,754642bd95db72ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:36:48.685 INFO 21062 --- [ocp-schedules-16,5799b6c87a650066,0f8aea6e084ffdaa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:36:48.691 INFO 21062 --- [ocp-schedules-16,5799b6c87a650066,0f8aea6e084ffdaa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:36:48.691 INFO 21062 --- [ocp-schedules-16,5799b6c87a650066,0f8aea6e084ffdaa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:36:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:36:48.587186+08:00 2024-11-04 12:36:50.165 INFO 21062 --- [ocp-schedules-9,8321747e5853a516,9b27e3d344277df2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:36:50.390 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:50.391 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:50.559 WARN 21062 --- [ocp-async-7,30d1d789187885d6,70bf9deb55042ec3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:50.560 INFO 21062 --- [ocp-async-6,fa5bd0b7f663f958,a77f60c0cc897d0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:50.983 INFO 21062 --- [prometheus-minute-schedule-1,e50df5f1c87b6d66,26280e86d85b31f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:52.560 WARN 21062 --- [ocp-async-4,a35768620d5f751b,28f49d5ffe9c3c0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:52.560 INFO 21062 --- [ocp-async-10,b05d430505a4d499,a7910b284706b548] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:53.391 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:53.393 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6d2cce1ac6889d98,2fe02c8684dec7e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:54.560 WARN 21062 --- [ocp-async-5,8326b0b033c6d150,2b3a61b2353e9183] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:54.560 INFO 21062 --- [ocp-async-1,b286d8c2047a91b6,82d208db94e2851b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:56.393 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:56.394 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:56.560 WARN 21062 --- [ocp-async-7,f77ed754ffa0c984,978d321bf30578e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:56.560 INFO 21062 --- [ocp-async-9,2d85a8a8fddf40d9,a4ece5acf628b7b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fd067cd0d696b96b,cad95ca283629b32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:36:58.500 INFO 21062 --- [ocp-schedules-21,f82482c888ca9a4f,543fece53ff8b693] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:36:58.500 INFO 21062 --- [ocp-schedules-22,fd3a9009fe3a346b,8643ee146347980d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:36:58.503 INFO 21062 --- [ocp-schedules-23,9e04e55013632cfe,4663bfd3070cdc03] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:36:58.504 INFO 21062 --- [ocp-schedules-21,f82482c888ca9a4f,543fece53ff8b693] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:36:58.505 INFO 21062 --- [ocp-schedules-22,fd3a9009fe3a346b,8643ee146347980d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:58.505 INFO 21062 --- [ocp-schedules-27,3509c4c897be1976,2db64782dbb33fe0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:36:58.512 INFO 21062 --- [ocp-schedules-8,e821e73a78f5efbc,fcba03c3d15901ba] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:36:58.512 INFO 21062 --- [ocp-schedules-3,83d83172ab1500d5,37ec894b14ff2ed8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:36:58.512 INFO 21062 --- [ocp-schedules-8,e821e73a78f5efbc,fcba03c3d15901ba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:36:58.512 INFO 21062 --- [ocp-schedules-3,83d83172ab1500d5,37ec894b14ff2ed8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:36:58.514 INFO 21062 --- [ocp-schedules-18,97cc1db7cf915f9f,53bb197fd3c59eda] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:36:58.514 INFO 21062 --- [ocp-schedules-18,97cc1db7cf915f9f,53bb197fd3c59eda] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:36:58.534 INFO 21062 --- [ocp-async-9,5670246c2772a102,db47b2959357ac9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:36:58.560 WARN 21062 --- [ocp-async-8,34f4939b0ff25e32,f3ab80fc0681e2c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:36:58.560 INFO 21062 --- [ocp-async-10,aedd3f4b0b782407,4624a4586d0c55d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:36:58.562 INFO 21062 --- [ocp-schedules-6,0999ced1995affec,e7411f391dcbe937] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:36:58.565 INFO 21062 --- [ocp-schedules-6,0999ced1995affec,e7411f391dcbe937] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:58.566 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:36:58.576 INFO 21062 --- [ocp-schedules-15,affeb108650875f0,8418b94bb94fb25b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:36:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:36:58.588243+08:00, lastProcessTime=2024-11-04T12:36:48.587186+08:00 2024-11-04 12:36:58.598 INFO 21062 --- [ocp-schedules-28,c4afadfeb69e17df,4e67fdb06bfb7e66] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:36:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:36:58.588243+08:00 2024-11-04 12:36:58.855 INFO 21062 --- [ocp-schedules-10,37743cf50e81e39f,9b1b66360517129f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:36:58.855 INFO 21062 --- [ocp-schedules-10,89e25a535531e35e,7147ef2e40eb555e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:36:58.858 INFO 21062 --- [ocp-schedules-10,89e25a535531e35e,7147ef2e40eb555e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:36:58.888 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:36:58.888 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:36:59.394 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:36:59.396 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:36:59.983 INFO 21062 --- [prometheus-minute-schedule-1,121a1cd763541a57,f8e81eed247d763c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695020, interval=60 2024-11-04 12:37:00.210 INFO 21062 --- [ocp-schedules-23,7816bcf726c95b1b,30c52172159a675f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:37:00.560 WARN 21062 --- [ocp-async-5,134008e1072076f3,85e411dc6213d6b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:00.561 INFO 21062 --- [ocp-async-3,379df2527f09888a,49f9863c25699f9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:02.396 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:02.397 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:02.560 WARN 21062 --- [ocp-async-7,376ea49902fdb2cc,9f0cb099d676b39b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:02.561 INFO 21062 --- [ocp-async-2,911423a2aadc3e01,7cdc7b61b12e1870] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:02.983 INFO 21062 --- [prometheus-minute-schedule-1,0bc4fd6e586596f7,f90cc962e68af44b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:04.561 WARN 21062 --- [ocp-async-6,a52b39acd250697e,28d4a44dbf705896] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:04.561 INFO 21062 --- [ocp-async-8,7048b42053777c5c,359bbc382ab4065c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:05.397 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:05.399 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:05.983 INFO 21062 --- [prometheus-minute-schedule-1,076ab0e979615b12,4b069aeef617c33b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:06.561 WARN 21062 --- [ocp-async-5,6c8b7bbf4ffb0ba1,efb653073a1fd4f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:06.562 INFO 21062 --- [ocp-async-1,d7863ddfc229559f,9d0db895b9875db6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:08.399 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:08.400 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:08.505 INFO 21062 --- [ocp-schedules-21,f82482c888ca9a4f,543fece53ff8b693] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:37:08.505 INFO 21062 --- [ocp-schedules-22,fd3a9009fe3a346b,8643ee146347980d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:37:08.534 INFO 21062 --- [ocp-async-1,2f417d2626548bf8,248f229ccce74837] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:37:08.561 WARN 21062 --- [ocp-async-3,d6e72cc6cf2b088e,3ff3c1a0e63cf778] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:08.562 INFO 21062 --- [ocp-async-9,4b09924f4f9a8029,08fe0ddfb8c2c608] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:08.566 INFO 21062 --- [ocp-schedules-6,0999ced1995affec,e7411f391dcbe937] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:37:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:37:08.587403+08:00, lastProcessTime=2024-11-04T12:36:58.588243+08:00 2024-11-04 12:37:08.600 INFO 21062 --- [ocp-schedules-16,8a00108d2222e48d,b52d997c562e375f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:37:08.693 INFO 21062 --- [ocp-schedules-12,6fe8f76762e819cf,6f87c4a53bfe471d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:37:08.700 INFO 21062 --- [ocp-schedules-12,6fe8f76762e819cf,6f87c4a53bfe471d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:37:08.700 INFO 21062 --- [ocp-schedules-12,6fe8f76762e819cf,6f87c4a53bfe471d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:37:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:37:08.587403+08:00 2024-11-04 12:37:08.983 INFO 21062 --- [prometheus-minute-schedule-1,bf4baed5aa0a640c,b01849662eee3463] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:37:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:37:10.240 INFO 21062 --- [ocp-schedules-29,ed56a7ee9fdb06e9,6e08cb64011c8b10] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:37:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:37:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:37:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:37:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:37:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:37:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:37:10.561 WARN 21062 --- [ocp-async-2,7bb289e59ee9cd2f,928e449e7ff8c3ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:10.562 INFO 21062 --- [ocp-async-4,06f969ead7a06f3b,31e1288611e28297] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=4, affectRows=4 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=4, validCount=4 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=12, affectRows=12 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=30, affectRows=30 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=17, affectRows=17 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=11, validCount=11 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=17, validCount=17 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=67, affectRows=67 2024-11-04 12:37:11.042 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=30, validCount=30 2024-11-04 12:37:11.043 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=49, affectRows=49 2024-11-04 12:37:11.043 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=66, validCount=66 2024-11-04 12:37:11.043 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=49, validCount=49 2024-11-04 12:37:11.044 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=63, affectRows=63 2024-11-04 12:37:11.044 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730694960, nodeCount=80, affectRows=80 2024-11-04 12:37:11.044 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=63, validCount=63 2024-11-04 12:37:11.044 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730694960, nodeCount=79, validCount=79 2024-11-04 12:37:11.400 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:11.404 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:11.983 INFO 21062 --- [prometheus-minute-schedule-1,f9e90176135b9384,bd099184bfa3da81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:12.562 WARN 21062 --- [ocp-async-8,d85e9c0f4cfc6b87,6618970e25e6caac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:12.562 INFO 21062 --- [ocp-async-10,248afaa0f97fa67d,08494f9b2cc2d4b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:13.984 INFO 21062 --- [prometheus-cache-monitor1,57d9127d8c2901b6,da0d3b5cac05570f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":966,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:37:13.984 INFO 21062 --- [prometheus-cache-monitor1,57d9127d8c2901b6,da0d3b5cac05570f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:37:14.037 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:33:13.988, end=2024-11-04T12:35:14.034} 2024-11-04 12:37:14.049 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:37:14.049 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:37:14.404 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:14.405 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:14.562 WARN 21062 --- [ocp-async-3,5b8e99b6091c4a89,78a45a738eb18fbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:14.562 INFO 21062 --- [ocp-async-7,f164baec216b3e73,64f89f15fcc7eef5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5d671aeb872fc1fb,873e37584f759a56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:16.562 WARN 21062 --- [ocp-async-2,036d3b59eacd0bc8,d65f4ef04787fa62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:16.562 INFO 21062 --- [ocp-async-6,d388a03f8ee04c48,57c8fe5cfc0f2a30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:17.405 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:17.406 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:17.983 INFO 21062 --- [prometheus-minute-schedule-1,49e8c92ae2a79c80,f1f4e80ccb85d1d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:18.534 INFO 21062 --- [ocp-async-4,0df2f93d9c76af7a,fca4e197d79a7898] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:37:18.562 WARN 21062 --- [ocp-async-5,80db70fee4221beb,0f2acc30c075b223] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:18.562 INFO 21062 --- [ocp-async-10,7e752b9d1fd0eb1a,213e4d77c4ef8a0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:37:18.588456+08:00, lastProcessTime=2024-11-04T12:37:08.587403+08:00 2024-11-04 12:37:18.603 INFO 21062 --- [ocp-schedules-15,f8e22f70503239b7,8469254f32772b8c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:37:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:37:18.588456+08:00 2024-11-04 12:37:20.286 INFO 21062 --- [ocp-schedules-1,0e5e8a238ae6cd0a,d9bbdba44e109a7c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:37:20.406 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:20.407 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:20.562 WARN 21062 --- [ocp-async-1,1b553a21e06f4b1e,f2832b19eee0dd82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:20.562 INFO 21062 --- [ocp-async-1,6557c3376e34a8ea,2e32a6676b253c43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:20.983 INFO 21062 --- [prometheus-minute-schedule-1,fedd1a0bf3f6f728,80e304e0f6361452] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:22.562 WARN 21062 --- [ocp-async-9,d539756bf80211ef,90b0b4cba88a903a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:22.563 INFO 21062 --- [ocp-async-6,169fe5fbe7c616be,0efd76dad1640ac1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:23.407 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:23.408 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b86b175ff5d57d67,239f318859d37151] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:24.563 WARN 21062 --- [ocp-async-5,db20a41bc67feffb,eccb74f6d73236c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:24.563 INFO 21062 --- [ocp-async-8,a1dd7ed044050704,f2af79d4d25f9c79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:26.408 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:26.409 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:26.563 WARN 21062 --- [ocp-async-7,f3d2bae89e966df2,74fc1140135a8f3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:26.563 INFO 21062 --- [ocp-async-3,125ef7491d77ba3d,934322ef3347f84d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:26.983 INFO 21062 --- [prometheus-minute-schedule-1,651afd823e2190c6,c0516b70f63b0f76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b5b490efa40821c5,c20326f4f24a04cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:37:28.498 INFO 21062 --- [ocp-schedules-4,91f1b69b7d472717,1feef05e4c5a8e77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:37:28.500 INFO 21062 --- [ocp-schedules-17,61689dbf792e87ac,21b8bb31b248a099] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:37:28.500 INFO 21062 --- [ocp-schedules-2,2d4c4500fd121fcb,a67ae55a2c546e83] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:37:28.500 INFO 21062 --- [ocp-schedules-23,fe1dd1530b3b23a3,1ce80a0027fb085e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:37:28.502 INFO 21062 --- [ocp-schedules-4,91f1b69b7d472717,1feef05e4c5a8e77] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:37:28.502 INFO 21062 --- [ocp-schedules-12,0820617eb416a2a8,17d47295ce47974d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:37:28.504 INFO 21062 --- [ocp-schedules-23,fe1dd1530b3b23a3,1ce80a0027fb085e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:28.504 INFO 21062 --- [ocp-schedules-17,61689dbf792e87ac,21b8bb31b248a099] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:28.504 INFO 21062 --- [ocp-schedules-18,5ffa622a4a4f1bd3,26f16fa38b630682] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:37:28.504 INFO 21062 --- [ocp-schedules-26,f824af73cb3893c9,ff91b0b02796adbc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 12:37:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:37:28.506 INFO 21062 --- [ocp-schedules-2,2d4c4500fd121fcb,a67ae55a2c546e83] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:37:28.507 INFO 21062 --- [ocp-schedules-26,f824af73cb3893c9,ff91b0b02796adbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:28.510 INFO 21062 --- [ocp-schedules-8,55020833cee069e5,8a1ed14be81a6c2e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:37:28.510 INFO 21062 --- [ocp-schedules-8,55020833cee069e5,8a1ed14be81a6c2e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:37:28.510 INFO 21062 --- [ocp-schedules-1,e43ea1b2b50ad353,27d1a3a30602c0be] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:37:28.510 INFO 21062 --- [ocp-schedules-1,e43ea1b2b50ad353,27d1a3a30602c0be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:37:28.510 INFO 21062 --- [ocp-schedules-24,286a3f8c4eda0a96,6c5f7f85b911dc9c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:37:28.510 INFO 21062 --- [ocp-schedules-24,286a3f8c4eda0a96,6c5f7f85b911dc9c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:37:28.534 INFO 21062 --- [ocp-async-2,4992e591ceb9fc9d,0a78001de8974922] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:37:28.542 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:37:28.542 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:37:28.549 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:37:28.549 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:37:28.549 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:37:28.556 INFO 21062 --- [ocp-schedules-15,0011589f75d77688,3edaf7b41e3b9e61] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 12:37:28.556 INFO 21062 --- [ocp-schedules-15,0011589f75d77688,3edaf7b41e3b9e61] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 12:37:28.560 INFO 21062 --- [ocp-schedules-11,ef64d2611172e887,aab071330ed4bf98] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 12:37:28.562 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:37:28.562 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:37:28.562 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:37:28.563 WARN 21062 --- [ocp-async-5,b8d482180707ff06,8b038dd29618b918] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:28.563 INFO 21062 --- [ocp-async-4,4e6f701940157195,a0940a54ca954ee4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:28.566 INFO 21062 --- [ocp-schedules-9,a174e97824db727e,ac04db31143394df] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:37:28.571 INFO 21062 --- [ocp-schedules-8,2994e667b4db7cb5,698d6aad144394ea] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:37:28.573 INFO 21062 --- [pool-12-thread-8,b076f19921ffeead,f50669404ebcc563] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 12:37:28.573 INFO 21062 --- [pool-12-thread-5,b076f19921ffeead,0337eb00f676a67a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 12:37:28.573 INFO 21062 --- [pool-12-thread-6,b076f19921ffeead,361810ddd93e4ff4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 12:37:28.573 INFO 21062 --- [pool-12-thread-5,b076f19921ffeead,0337eb00f676a67a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 12:37:28.573 INFO 21062 --- [pool-12-thread-7,b076f19921ffeead,3e2db3eb9d0f702f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 12:37:28.586 INFO 21062 --- [ocp-schedules-24,561b5ef06e4e0578,af425ca50d5c8196] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:37:28.586 INFO 21062 --- [ocp-schedules-24,561b5ef06e4e0578,af425ca50d5c8196] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:37:28.587 INFO 21062 --- [ocp-schedules-1,5776fdc326ae287f,bdefec9a7d82b8a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:37:28.587 INFO 21062 --- [ocp-schedules-11,ef64d2611172e887,aab071330ed4bf98] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 12:37:28.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:37:28.590359+08:00, lastProcessTime=2024-11-04T12:37:18.588456+08:00 2024-11-04 12:37:28.592 INFO 21062 --- [ForkJoinPool-11-worker-9,b076f19921ffeead,f452fb592315ba8c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 12:37:28.592 INFO 21062 --- [ForkJoinPool-11-worker-9,b076f19921ffeead,f452fb592315ba8c] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 12:37:28.592 INFO 21062 --- [ForkJoinPool-11-worker-9,b076f19921ffeead,f452fb592315ba8c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 12:37:28.592 INFO 21062 --- [ForkJoinPool-11-worker-2,b076f19921ffeead,0a213b14cc55a08c] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 12:37:28.593 INFO 21062 --- [pool-12-thread-8,b076f19921ffeead,f50669404ebcc563] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 12:37:28.594 INFO 21062 --- [pool-12-thread-7,b076f19921ffeead,3e2db3eb9d0f702f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 12:37:28.598 INFO 21062 --- [pool-20-thread-1,0011589f75d77688,de95416a0af164c0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 12:37:28.606 INFO 21062 --- [ocp-schedules-6,c929c3d7ac690210,43a2fe0fa475333c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:37:28.615 INFO 21062 --- [ocp-schedules-30,a0891df494681ce5,3ff66aa65e42ac2d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:37:28.619 INFO 21062 --- [ocp-schedules-30,a0891df494681ce5,3ff66aa65e42ac2d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:37:28.619 INFO 21062 --- [ocp-schedules-30,a0891df494681ce5,3ff66aa65e42ac2d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:37:28.628 WARN 21062 --- [ocp-schedules-15,0011589f75d77688,3edaf7b41e3b9e61] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:37:28.628 INFO 21062 --- [ocp-schedules-15,0011589f75d77688,3edaf7b41e3b9e61] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 12:37:28.650 INFO 21062 --- [pool-12-thread-6,b076f19921ffeead,361810ddd93e4ff4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 12:37:28.653 INFO 21062 --- [pool-12-thread-6,b076f19921ffeead,361810ddd93e4ff4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 12:37:28.663 INFO 21062 --- [ocp-schedules-20,e7a885e7d33b3afe,3b0c64cb95e1a34f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:37:28.663 INFO 21062 --- [ocp-schedules-20,e7a885e7d33b3afe,3b0c64cb95e1a34f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:37:28.671 INFO 21062 --- [ocp-schedules-5,0ab8b48108c073b2,3881874901b7fab9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:37:28.671 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:37:28.671 INFO 21062 --- [ocp-schedules-5,0ab8b48108c073b2,3881874901b7fab9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:37:28.671 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:37:28.676 INFO 21062 --- [ocp-schedules-7,967b10ad81a0b6f1,3f55937c2d643ae7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:37:28.676 INFO 21062 --- [ocp-schedules-7,967b10ad81a0b6f1,3f55937c2d643ae7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:37:28.676 INFO 21062 --- [ocp-schedules-7,967b10ad81a0b6f1,3f55937c2d643ae7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:37:28.676 INFO 21062 --- [ocp-schedules-7,967b10ad81a0b6f1,3f55937c2d643ae7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:37:28.677 INFO 21062 --- [ocp-schedules-7,967b10ad81a0b6f1,3f55937c2d643ae7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:37:28.677 INFO 21062 --- [ocp-schedules-7,967b10ad81a0b6f1,3f55937c2d643ae7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:37:28.680 INFO 21062 --- [ocp-schedules-5,0ab8b48108c073b2,3881874901b7fab9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:37:28.680 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:37:28.681 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:37:28.683 WARN 21062 --- [ocp-schedules-5,0ab8b48108c073b2,3881874901b7fab9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:37:28.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:37:28.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:37:28.683 INFO 21062 --- [ocp-schedules-5,0ab8b48108c073b2,3881874901b7fab9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:37:28.683 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:37:28.683 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:37:28.685 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:37:28.685 WARN 21062 --- [ocp-schedules-5,0ab8b48108c073b2,3881874901b7fab9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:37:28.685 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:37:28.688 INFO 21062 --- [ocp-schedules-20,e7a885e7d33b3afe,3b0c64cb95e1a34f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:37:28.688 INFO 21062 --- [ocp-schedules-20,e7a885e7d33b3afe,3b0c64cb95e1a34f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:37:28.694 ERROR 21062 --- [ocp-schedules-5,0ab8b48108c073b2,3881874901b7fab9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:37:28.700 INFO 21062 --- [pool-12-thread-5,b076f19921ffeead,0337eb00f676a67a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 12:37:28.702 INFO 21062 --- [ocp-schedules-27,a5a03d1906252d5e,16a340c7cb8b01df] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:37:28.706 INFO 21062 --- [ocp-schedules-27,a5a03d1906252d5e,16a340c7cb8b01df] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:37:28.706 INFO 21062 --- [ocp-schedules-27,a5a03d1906252d5e,16a340c7cb8b01df] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:37:28.713 INFO 21062 --- [ocp-schedules-21,3c0d3b3c2073d2bf,44de4e3a98d1bf7f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:37:28.734 INFO 21062 --- [pool-12-thread-5,b076f19921ffeead,0337eb00f676a67a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 12:37:28.781 INFO 21062 --- [ocp-schedules-14,d2a2f2a6c2ac752e,17cdd0af69044d11] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:37:28.781+08:00, clusters=[1] 2024-11-04 12:37:28.782 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:37:28.590359+08:00 2024-11-04 12:37:28.783 INFO 21062 --- [ocp-schedules-14,d2a2f2a6c2ac752e,17cdd0af69044d11] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:37:28.844 INFO 21062 --- [ocp-schedules-28,ce73d2359c06d93d,afd53a3e8bf71f60] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:37:28.844+08:00, clusters=[1] 2024-11-04 12:37:28.847 INFO 21062 --- [ocp-schedules-28,ce73d2359c06d93d,afd53a3e8bf71f60] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:37:28.888 INFO 21062 --- [ocp-schedules-10,89e25a535531e35e,7147ef2e40eb555e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:37:28.889 INFO 21062 --- [ocp-schedules-10,31d2410325b253f5,1b92a32b188295fe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:37:28.892 INFO 21062 --- [ocp-schedules-10,31d2410325b253f5,1b92a32b188295fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:28.918 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:37:28.918 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:37:28.968 INFO 21062 --- [ocp-schedules-19,f00aa7d574a9dbec,82dbaf17447853f1] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:37:28.971 INFO 21062 --- [ocp-schedules-19,f00aa7d574a9dbec,82dbaf17447853f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:29.054 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:37:29.056 ERROR 21062 --- [ocp-schedules-19,f00aa7d574a9dbec,82dbaf17447853f1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:37:29.057 INFO 21062 --- [ocp-schedules-19,f00aa7d574a9dbec,82dbaf17447853f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:37:29.410 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:29.411 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:29.983 INFO 21062 --- [prometheus-minute-schedule-1,aaa470a60dd6cb0f,89c557e2c950a654] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:37:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:37:30.311 INFO 21062 --- [ocp-schedules-2,13e224484be73d72,bd6a1b62346ab40d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:37:30.563 WARN 21062 --- [ocp-async-7,8b7ff9fa048bbeef,cf3cdace99385eaa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:30.563 INFO 21062 --- [ocp-async-10,ac989b70d1f78c2d,a452bbd22cf9d704] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:32.411 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:32.413 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:32.564 WARN 21062 --- [ocp-async-3,e234dc1cfd3ba790,422a31e3a1c1daa2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:32.564 INFO 21062 --- [ocp-async-9,4abf083883cb841a,157b9f392822794f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:32.983 INFO 21062 --- [prometheus-minute-schedule-1,63a2f7fd2c63bbe9,7f06969a2a0adf2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:34.564 WARN 21062 --- [ocp-async-5,adcc475ec69af0d0,f1883cc4b5d302e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:34.564 INFO 21062 --- [ocp-async-6,d343d728093bf7d3,4bce3141f923a8d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:35.413 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:35.414 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:35.983 INFO 21062 --- [prometheus-minute-schedule-1,a5a69ac5307a6ea6,051e739890ca7546] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:36.564 WARN 21062 --- [ocp-async-7,e13cfb8e2b11ae4a,b1282059f5b1482a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:36.564 INFO 21062 --- [ocp-async-8,15c54e27cf7d02dd,a71ae33408a1e413] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:38.414 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:38.415 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:38.507 INFO 21062 --- [ocp-schedules-23,fe1dd1530b3b23a3,1ce80a0027fb085e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:37:38.507 INFO 21062 --- [ocp-schedules-17,61689dbf792e87ac,21b8bb31b248a099] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:37:38.508 INFO 21062 --- [ocp-schedules-26,f824af73cb3893c9,ff91b0b02796adbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 12:37:38.535 INFO 21062 --- [ocp-async-3,c682716e40077b40,9272c6ec2d7affd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:37:38.564 WARN 21062 --- [ocp-async-9,4c9ba91a8038e237,1a395fd70cb6edd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:38.564 INFO 21062 --- [ocp-async-5,e577031d3661f6c4,291c8c419d3d6796] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:37:38.588462+08:00, lastProcessTime=2024-11-04T12:37:28.590359+08:00 2024-11-04 12:37:38.608 INFO 21062 --- [ocp-schedules-19,da1fa143a35ff395,e7a9ec8a56a5cbbd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:37:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:37:38.588462+08:00 2024-11-04 12:37:38.983 INFO 21062 --- [prometheus-minute-schedule-1,be7022dfb1efa075,b15c53d57c7e4ced] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:40.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:37:40.005 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:37:40.359 INFO 21062 --- [ocp-schedules-18,04d2681b20c251dc,62e5976997bd81e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:37:40.564 WARN 21062 --- [ocp-async-6,feed28a4d0629ca5,e9188e658d042f97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:40.564 INFO 21062 --- [ocp-async-7,6322fef7409a951e,16b76e6cd95a4955] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:41.416 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:41.417 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:41.983 INFO 21062 --- [prometheus-minute-schedule-1,9b45d244c8543374,1ec11966604dd5fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:42.564 WARN 21062 --- [ocp-async-8,bdbdbe93af66a743,d3ec183d88b26d79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:42.565 INFO 21062 --- [ocp-async-10,5f108ff1cf36a95e,392a07cb89d4b962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:44.417 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:44.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:44.565 WARN 21062 --- [ocp-async-9,524807a5b5ecab43,a611594d701272ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:44.565 INFO 21062 --- [ocp-async-1,8368d965c97fc88c,e92b9aacc5f164da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:44.983 INFO 21062 --- [prometheus-minute-schedule-1,24aeff0b051acecb,f0e547773a1eb2f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:46.565 WARN 21062 --- [ocp-async-6,d398deea7d56819a,f32a8b3f68bcc67a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:46.565 INFO 21062 --- [ocp-async-2,0ee5e15b54be0eea,737465c284d8ce58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:47.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:47.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:47.983 INFO 21062 --- [prometheus-minute-schedule-1,a8ea0c29bbfe3370,59100c6a03bca009] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:48.535 INFO 21062 --- [ocp-async-2,b8011685ca99c85e,8d169d7ee7cdbda9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:37:48.565 WARN 21062 --- [ocp-async-4,04254403e1d4c27b,40c0ddc61112d661] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:48.565 INFO 21062 --- [ocp-async-10,8f6182c6bac9c3dc,4865c58f220190a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:37:48.589789+08:00, lastProcessTime=2024-11-04T12:37:38.588462+08:00 2024-11-04 12:37:48.611 INFO 21062 --- [ocp-schedules-7,93b5352df99544f4,a739d86d8ad270d4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:37:48.709 INFO 21062 --- [ocp-schedules-15,822db377300b71ad,d355bf1195c475e1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:37:48.718 INFO 21062 --- [ocp-schedules-15,822db377300b71ad,d355bf1195c475e1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:37:48.718 INFO 21062 --- [ocp-schedules-15,822db377300b71ad,d355bf1195c475e1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:37:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:37:48.589789+08:00 2024-11-04 12:37:50.388 INFO 21062 --- [ocp-schedules-4,88c3758768fa7425,5c4179d3cc9f4fd9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:37:50.419 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:50.419 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:50.565 WARN 21062 --- [ocp-async-3,35031d76080009e7,853b60f74cfbffeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:50.566 INFO 21062 --- [ocp-async-1,744b1222d791fbcb,76cc2ff3489cb127] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4bf87e6d9954ecd6,912efcd828bcd4cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:52.565 WARN 21062 --- [ocp-async-7,c6b1ab1fd6a4c461,f43bd1c9cb3c57c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:52.565 INFO 21062 --- [ocp-async-5,96b4c4ed9a6f7186,c629fed0fcb5206a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:53.419 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:53.420 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1c220664fabc3e18,43e78c4e91d42c65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:54.566 INFO 21062 --- [ocp-async-4,359e1797c5b1bff2,c7a19829102482b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:54.566 WARN 21062 --- [ocp-async-8,854272c4568b51b8,30ece1169c4764a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:56.420 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:56.421 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:56.566 INFO 21062 --- [ocp-async-3,9dead8704c88b2d7,2c9ae73847137ba8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:56.566 WARN 21062 --- [ocp-async-9,6c748996788fd8d7,da4392097de7a898] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:56.983 INFO 21062 --- [prometheus-minute-schedule-1,662c5d1aa4e9d8da,ecf2c30ef7106dc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:37:58.500 INFO 21062 --- [ocp-schedules-22,599e3be4fff600a2,7cf13a63ffe67a64] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:37:58.500 INFO 21062 --- [ocp-schedules-8,79f041c0aac398a6,a17f0669da7524e3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:37:58.503 INFO 21062 --- [ocp-schedules-19,d20083a10aa57a3c,214f3fc2155b3efd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:37:58.504 INFO 21062 --- [ocp-schedules-8,79f041c0aac398a6,a17f0669da7524e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:58.504 INFO 21062 --- [ocp-schedules-22,599e3be4fff600a2,7cf13a63ffe67a64] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:58.505 INFO 21062 --- [ocp-schedules-15,1d0da480c00a06d8,e76f096bea00bf23] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:37:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:37:58.512 INFO 21062 --- [ocp-schedules-3,d9073def787c7de0,a2080bc883429579] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:37:58.512 INFO 21062 --- [ocp-schedules-20,9876f5d1f2d2ea24,c6122310ebcd9712] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:37:58.512 INFO 21062 --- [ocp-schedules-20,9876f5d1f2d2ea24,c6122310ebcd9712] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:37:58.512 INFO 21062 --- [ocp-schedules-3,d9073def787c7de0,a2080bc883429579] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:37:58.512 INFO 21062 --- [ocp-schedules-28,c23d10548228f07c,c4dffdca03156282] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:37:58.515 INFO 21062 --- [ocp-schedules-28,c23d10548228f07c,c4dffdca03156282] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:37:58.535 INFO 21062 --- [ocp-async-3,57072179bb75b0a6,9234906736fa8483] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:37:58.566 WARN 21062 --- [ocp-async-7,50c9089381e44493,a57d003f5c00d8bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:37:58.566 INFO 21062 --- [ocp-async-6,de0c1487258935bf,f957d32f83495c24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:37:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:37:58.587405+08:00, lastProcessTime=2024-11-04T12:37:48.589789+08:00 2024-11-04 12:37:58.589 INFO 21062 --- [ocp-schedules-5,6bc3fba087068cd7,3ac341c30f82f73f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:37:58.613 INFO 21062 --- [ocp-schedules-21,4c2913f5322898c8,1210f0b500c3ef8f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:37:58.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:37:58.587405+08:00 2024-11-04 12:37:58.918 INFO 21062 --- [ocp-schedules-10,31d2410325b253f5,1b92a32b188295fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:37:58.919 INFO 21062 --- [ocp-schedules-10,d1f55ac923089747,3251a0209f3d5577] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:37:58.922 INFO 21062 --- [ocp-schedules-10,d1f55ac923089747,3251a0209f3d5577] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:37:58.953 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:37:58.953 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:37:59.421 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:37:59.423 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:37:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d9348e4012b120dc,e7bec686b6f6610f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695080, interval=60 2024-11-04 12:38:00.432 INFO 21062 --- [ocp-schedules-20,65c352e1fbdf91b0,e6d461bc4d6e2eb8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:38:00.566 WARN 21062 --- [ocp-async-2,cbddb09b04bb4d92,bbd8b6e6dc6906e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:00.566 INFO 21062 --- [ocp-async-8,15d00f5e4eb2fcc3,f44e66ee5bc43491] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:02.423 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:02.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:02.566 INFO 21062 --- [ocp-async-9,0f4d987edc05d4e4,3e954df0aabde20a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:02.566 WARN 21062 --- [ocp-async-10,823bc6f73860c7dd,30ae548329e83a9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:02.983 INFO 21062 --- [prometheus-minute-schedule-1,8f004742dbf9c311,9a45632ac6791ba2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:04.567 INFO 21062 --- [ocp-async-7,ac79d81519157f20,4f529d23723fb160] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:04.567 WARN 21062 --- [ocp-async-5,80e9f771e7f4f648,5d53a8c5e8b53b8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:05.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:05.426 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:05.983 INFO 21062 --- [prometheus-minute-schedule-1,eeba185c0e249912,efae2cf57cb8b16a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:06.567 WARN 21062 --- [ocp-async-6,72fa342a033b3a09,0d87d92d661da59c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:06.567 INFO 21062 --- [ocp-async-2,5d33401bf1e17da2,3678523abc3060ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:08.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:08.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:08.505 INFO 21062 --- [ocp-schedules-8,79f041c0aac398a6,a17f0669da7524e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:38:08.505 INFO 21062 --- [ocp-schedules-22,599e3be4fff600a2,7cf13a63ffe67a64] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:38:08.535 INFO 21062 --- [ocp-async-4,3e990a4a8628a20e,f85bf37b46e4ec31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:38:08.567 INFO 21062 --- [ocp-schedules-17,d95500ba1e9c46ed,8584389bd4edb5bd] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:38:08.567 WARN 21062 --- [ocp-async-3,158d11391a19325f,95afe30dfb375252] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:08.567 INFO 21062 --- [ocp-async-8,7e70a53b603a0547,01260cc0f59a1408] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:08.570 INFO 21062 --- [ocp-schedules-17,d95500ba1e9c46ed,8584389bd4edb5bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:08.570 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:38:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:38:08.589052+08:00, lastProcessTime=2024-11-04T12:37:58.587405+08:00 2024-11-04 12:38:08.616 INFO 21062 --- [ocp-schedules-21,3133559315437b17,0f065d03bf681641] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:38:08.721 INFO 21062 --- [ocp-schedules-6,f1c4b7ba571495bd,5b8c2da374ffcda3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:38:08.727 INFO 21062 --- [ocp-schedules-6,f1c4b7ba571495bd,5b8c2da374ffcda3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:38:08.727 INFO 21062 --- [ocp-schedules-6,f1c4b7ba571495bd,5b8c2da374ffcda3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:38:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:38:08.589052+08:00 2024-11-04 12:38:08.983 INFO 21062 --- [prometheus-minute-schedule-1,14b16f463e6c91a9,45b335e68dc2f252] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:38:10.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:38:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:38:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:38:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:38:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:38:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:38:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:38:10.457 INFO 21062 --- [ocp-schedules-3,891f5f9b1c972687,5e0b73146033c596] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:38:10.567 WARN 21062 --- [ocp-async-1,68e0990439056233,32fc2b5eb29b0fdf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:10.567 INFO 21062 --- [ocp-async-9,2fafebc02c066e1a,5737446959b93c7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=3, affectRows=3 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=1, affectRows=1 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=3, validCount=3 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=1, validCount=1 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=23, affectRows=23 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=37, affectRows=37 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=10, affectRows=10 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=10, validCount=10 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=22, validCount=22 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=37, validCount=37 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=40, affectRows=40 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=56, affectRows=56 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=29, affectRows=29 2024-11-04 12:38:11.051 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=28, affectRows=28 2024-11-04 12:38:11.052 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=54, affectRows=54 2024-11-04 12:38:11.052 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=55, validCount=55 2024-11-04 12:38:11.052 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=40, validCount=40 2024-11-04 12:38:11.052 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=28, validCount=28 2024-11-04 12:38:11.052 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=29, validCount=29 2024-11-04 12:38:11.052 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=54, validCount=54 2024-11-04 12:38:11.052 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695020, nodeCount=41, affectRows=41 2024-11-04 12:38:11.053 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695020, nodeCount=40, validCount=40 2024-11-04 12:38:11.428 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:11.429 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:11.983 INFO 21062 --- [prometheus-minute-schedule-1,5a53dd6eca5ec7b6,d3155f09b3bc28b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:12.567 WARN 21062 --- [ocp-async-2,e0b1a0cd015f91fc,07aa5b9e3db3e5bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:12.567 INFO 21062 --- [ocp-async-7,324251e42e0abab1,5cd8d96cd51ccd12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:13.837 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T12:32:13.834, end=2024-11-04T12:37:13.834} 2024-11-04 12:38:13.837 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T12:32:13.834, end=2024-11-04T12:37:13.834} 2024-11-04 12:38:13.842 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 12:38:13.843 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 12:38:13.843 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:38:13.843 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:38:13.984 INFO 21062 --- [prometheus-cache-monitor1,0b93907ccd1f714d,d761ae57fca01d91] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:38:13.985 INFO 21062 --- [prometheus-cache-monitor1,0b93907ccd1f714d,d761ae57fca01d91] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:38:14.429 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:14.430 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:14.568 WARN 21062 --- [ocp-async-10,acc9f67fb5ab6234,5f7b183251e2ee69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:14.568 INFO 21062 --- [ocp-async-3,a1b4dc329ca1d3a9,7882d8e22746593c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a672467c58abef7b,4ba5609798af71c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:16.568 WARN 21062 --- [ocp-async-5,81a63207ef507ba4,6bb2e86f94f01391] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:16.568 INFO 21062 --- [ocp-async-1,60b95e20486feb2b,2381313face94ea8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:17.430 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:17.431 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:17.983 INFO 21062 --- [prometheus-minute-schedule-1,f20be36919d9a47d,b0a39e0b7d60544f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:18.535 INFO 21062 --- [ocp-async-9,ca0d838d5d97895c,b0b7d7034fb8312c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:38:18.568 WARN 21062 --- [ocp-async-6,350fc13433671fc8,d7d3ffa603a74f10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:18.568 INFO 21062 --- [ocp-async-7,4b3635f924329262,bb4f23a89ca32c13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:18.571 INFO 21062 --- [ocp-schedules-17,d95500ba1e9c46ed,8584389bd4edb5bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:38:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:38:18.589297+08:00, lastProcessTime=2024-11-04T12:38:08.589052+08:00 2024-11-04 12:38:18.618 INFO 21062 --- [ocp-schedules-2,baa7cc12a25e207d,fcf8aef89ec39b37] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:38:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:38:18.589297+08:00 2024-11-04 12:38:20.431 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:20.433 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:20.501 INFO 21062 --- [ocp-schedules-21,aa38b38c8c958864,bb647ff0d1563df2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:38:20.568 WARN 21062 --- [ocp-async-4,88ac2058f8c0f2a5,1b88cddeec791654] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:20.568 INFO 21062 --- [ocp-async-3,ff64454f4f7d4809,70afc95e7b1b93c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b4fb1d4618d21a3e,26713bd9bb112bd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:22.568 WARN 21062 --- [ocp-async-8,03bd5f7f2ee0f12d,6ef4e3c9b5eca427] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:22.568 INFO 21062 --- [ocp-async-1,8224fa5dc79b4d2a,1d52cbc0c246c74f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:23.433 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:23.434 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:23.983 INFO 21062 --- [prometheus-minute-schedule-1,df8acc8681c4165e,20d76c6d58ed407f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:24.569 WARN 21062 --- [ocp-async-6,6e0c1a03a32035b3,a06ef16ea4c1ae63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:24.569 INFO 21062 --- [ocp-async-2,c0a399b40fdf3cac,fedee3179134665d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:26.434 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:26.435 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:26.569 WARN 21062 --- [ocp-async-4,bf8acfa00f89a086,b734d599f7414cbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:26.569 INFO 21062 --- [ocp-async-10,e9032d8e9f1a81f9,3eda332025f4f5db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:26.983 INFO 21062 --- [prometheus-minute-schedule-1,4745970b68ac26d4,f0e4549193d3e154] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a02fa010470e5eea,10e120e22907e1f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:38:28.498 INFO 21062 --- [ocp-schedules-19,dcd2c166f88bba9b,513d01943dbf513b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:38:28.499 INFO 21062 --- [ocp-schedules-26,768c308b44df1bba,8643eeb030e7b629] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:38:28.500 INFO 21062 --- [ocp-schedules-2,2fdf812e7665f5dd,aed4c4d3be88cce0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:38:28.500 INFO 21062 --- [ocp-schedules-25,00b281038d212137,941ef5798c4ebf1a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:38:28.502 INFO 21062 --- [ocp-schedules-19,dcd2c166f88bba9b,513d01943dbf513b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:38:28.502 INFO 21062 --- [ocp-schedules-16,e642c8d7e59d1c1f,9e3af455d2051e81] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:38:28.504 INFO 21062 --- [ocp-schedules-2,2fdf812e7665f5dd,aed4c4d3be88cce0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:28.504 INFO 21062 --- [ocp-schedules-25,00b281038d212137,941ef5798c4ebf1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:28.504 INFO 21062 --- [ocp-schedules-14,bc0e258283d1ba7b,e99196332042a551] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:38:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:38:28.506 INFO 21062 --- [ocp-schedules-26,768c308b44df1bba,8643eeb030e7b629] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:38:28.511 INFO 21062 --- [ocp-schedules-11,4bb6626595a2c37e,6c413eaa024b7ace] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:38:28.511 INFO 21062 --- [ocp-schedules-11,4bb6626595a2c37e,6c413eaa024b7ace] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:38:28.511 INFO 21062 --- [ocp-schedules-29,2cea5447b1b7ad89,9ab7a4c7a6e90a94] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:38:28.511 INFO 21062 --- [ocp-schedules-29,2cea5447b1b7ad89,9ab7a4c7a6e90a94] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:38:28.511 INFO 21062 --- [ocp-schedules-23,3ef7ea3800faa8d9,6ea1590b4e2fc7a5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:38:28.511 INFO 21062 --- [ocp-schedules-23,3ef7ea3800faa8d9,6ea1590b4e2fc7a5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:38:28.536 INFO 21062 --- [ocp-async-5,406fe6459542911a,7e80d2b39f454f6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:38:28.553 INFO 21062 --- [ocp-schedules-6,9e9ac10102bc2248,0fca147f1685c3ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":320,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:38:28.553 INFO 21062 --- [ocp-schedules-6,9e9ac10102bc2248,0fca147f1685c3ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:38:28.569 WARN 21062 --- [ocp-async-6,51f7dde0453402b2,5c9c01486c3cb746] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:28.569 INFO 21062 --- [ocp-async-9,1afc13dd1d9cd050,87c5c544b9577965] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:28.574 INFO 21062 --- [ocp-schedules-3,7f008de48a7d5ce6,a45f0386ce7e47f4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:38:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:38:28.587408+08:00, lastProcessTime=2024-11-04T12:38:18.589297+08:00 2024-11-04 12:38:28.591 INFO 21062 --- [ocp-schedules-8,bcf14bc64645aa1f,f3ff6a292bb3e9bc] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:38:28.591 INFO 21062 --- [ocp-schedules-8,bcf14bc64645aa1f,f3ff6a292bb3e9bc] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:38:28.592 INFO 21062 --- [ocp-schedules-7,c77cd46079e90d25,40aa442ee3e2c023] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:38:28.619 INFO 21062 --- [ocp-schedules-16,33c0963427c8fbfd,c5e261ed0fa2dda7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:38:28.622 INFO 21062 --- [ocp-schedules-29,163f0487b5d3cbfe,c7b6dd63c1557853] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:38:28.624 INFO 21062 --- [ocp-schedules-16,33c0963427c8fbfd,c5e261ed0fa2dda7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:38:28.624 INFO 21062 --- [ocp-schedules-16,33c0963427c8fbfd,c5e261ed0fa2dda7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:38:28.669 INFO 21062 --- [ocp-schedules-23,038afe0dd4483a86,f72b27eeb448b76d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:38:28.672 INFO 21062 --- [ocp-schedules-23,038afe0dd4483a86,f72b27eeb448b76d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:28.677 INFO 21062 --- [ocp-schedules-19,cb42308ef70d0eff,dc6ed5481c6c2315] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:38:28.677 INFO 21062 --- [ocp-schedules-19,cb42308ef70d0eff,dc6ed5481c6c2315] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:38:28.678 INFO 21062 --- [ocp-schedules-19,cb42308ef70d0eff,dc6ed5481c6c2315] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:38:28.678 INFO 21062 --- [ocp-schedules-19,cb42308ef70d0eff,dc6ed5481c6c2315] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:38:28.679 INFO 21062 --- [ocp-schedules-19,cb42308ef70d0eff,dc6ed5481c6c2315] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:38:28.679 INFO 21062 --- [ocp-schedules-19,cb42308ef70d0eff,dc6ed5481c6c2315] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:38:28.688 INFO 21062 --- [ocp-schedules-1,1b6bf1659b8cd2ed,4ea166822639c47d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:38:28.688 INFO 21062 --- [ocp-schedules-1,1b6bf1659b8cd2ed,4ea166822639c47d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:38:28.704 INFO 21062 --- [ocp-schedules-30,b6301ca8c8c1ba54,674ac4997c2f8508] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:38:28.704 INFO 21062 --- [ocp-schedules-30,b6301ca8c8c1ba54,674ac4997c2f8508] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:38:28.704 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:38:28.704 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:38:28.713 INFO 21062 --- [ocp-schedules-1,1b6bf1659b8cd2ed,4ea166822639c47d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:38:28.713 INFO 21062 --- [ocp-schedules-1,1b6bf1659b8cd2ed,4ea166822639c47d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:38:28.714 INFO 21062 --- [ocp-schedules-30,b6301ca8c8c1ba54,674ac4997c2f8508] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:38:28.714 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:38:28.715 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:38:28.716 WARN 21062 --- [ocp-schedules-30,b6301ca8c8c1ba54,674ac4997c2f8508] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:38:28.716 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:38:28.716 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:38:28.717 INFO 21062 --- [ocp-schedules-30,b6301ca8c8c1ba54,674ac4997c2f8508] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:38:28.717 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:38:28.717 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:38:28.718 WARN 21062 --- [ocp-schedules-30,b6301ca8c8c1ba54,674ac4997c2f8508] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:38:28.718 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:38:28.718 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:38:28.725 ERROR 21062 --- [ocp-schedules-30,b6301ca8c8c1ba54,674ac4997c2f8508] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2024-11-04 12:38:28.730 ERROR 21062 --- [ocp-schedules-23,038afe0dd4483a86,f72b27eeb448b76d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:38:28.730 INFO 21062 --- [ocp-schedules-23,038afe0dd4483a86,f72b27eeb448b76d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:38:28.734 INFO 21062 --- [ocp-schedules-21,6cf0bb70c8f5d4a1,9cb6e2a13bc13624] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:38:28.739 INFO 21062 --- [ocp-schedules-21,6cf0bb70c8f5d4a1,9cb6e2a13bc13624] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:38:28.739 INFO 21062 --- [ocp-schedules-21,6cf0bb70c8f5d4a1,9cb6e2a13bc13624] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:38:28.740 INFO 21062 --- [ocp-schedules-15,7e9e66754dca1ed0,ab21b16a846f6d25] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:38:28.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:38:28.587408+08:00 2024-11-04 12:38:28.808 INFO 21062 --- [ocp-schedules-4,676d331b1d57ce03,4c17826509568752] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:38:28.808+08:00, clusters=[1] 2024-11-04 12:38:28.809 INFO 21062 --- [ocp-schedules-4,676d331b1d57ce03,4c17826509568752] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:38:28.872 INFO 21062 --- [ocp-schedules-28,8f297211f17e1e98,435b76c1a9ecd92e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:38:28.872+08:00, clusters=[1] 2024-11-04 12:38:28.875 INFO 21062 --- [ocp-schedules-28,8f297211f17e1e98,435b76c1a9ecd92e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:38:28.953 INFO 21062 --- [ocp-schedules-10,d1f55ac923089747,3251a0209f3d5577] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:38:28.953 INFO 21062 --- [ocp-schedules-10,4f77e99ef0f1627d,134f7a84b64c5df1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:38:28.957 INFO 21062 --- [ocp-schedules-10,4f77e99ef0f1627d,134f7a84b64c5df1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:28.984 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:38:28.984 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:38:29.057 INFO 21062 --- [ocp-schedules-12,a25dd228544b42d7,11ef6f2d18891eec] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:38:29.060 INFO 21062 --- [ocp-schedules-12,a25dd228544b42d7,11ef6f2d18891eec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:29.143 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:38:29.145 ERROR 21062 --- [ocp-schedules-12,a25dd228544b42d7,11ef6f2d18891eec] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:38:29.145 INFO 21062 --- [ocp-schedules-12,a25dd228544b42d7,11ef6f2d18891eec] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:38:29.435 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:29.436 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b6206d365f4eeffe,f84bcc1e8bd2361e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:38:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:38:30.529 INFO 21062 --- [ocp-schedules-6,3b4ea2cecdc2fbdc,c8ade1e3c154cc40] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:38:30.569 WARN 21062 --- [ocp-async-4,665eba559953e6cc,bf759c375eaa2946] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:30.569 INFO 21062 --- [ocp-async-7,582071ea232bc074,56bfb07305b628e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:32.310 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:38:32.310 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:38:32.316 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:38:32.316 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:38:32.316 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:38:32.316 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:38:32.322 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:38:32.322 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:38:32.322 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:38:32.323 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:38:32.363 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:38:32.371 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:38:32.371 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:38:32.371 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:38:32.371 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:38:32.371 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:38:32.379 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:38:32.436 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:32.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:32.569 WARN 21062 --- [ocp-async-8,8ca269ab94f3c017,340e835ba6e1b372] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:32.569 INFO 21062 --- [ocp-async-10,998ff0d2ea547816,9b62e1a85a109015] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:32.917 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:38:32.983 INFO 21062 --- [prometheus-minute-schedule-1,694a51347c1acd00,ef76174e5ee1f85c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:33.519 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.197008792S 2024-11-04 12:38:33.520 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:38:33.658 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:38:33.658 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:38:33.688 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:38:33.748 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:38:33.748 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:38:33.748 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:38:33.748 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:38:33.748 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:38:33.749 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:38:33.757 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:38:33.881 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] o.s.boot.SpringApplication : Started application in 1.564 seconds (JVM running for 527.56) 2024-11-04 12:38:33.881 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.564676076S 2024-11-04 12:38:34.262 INFO 21062 --- [ocp-schedules-11,202725d0719a93e4,adfb2073c86e5f2c] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:38:34.570 WARN 21062 --- [ocp-async-1,73f2c488db9a205d,1ef31e685875527f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:34.570 INFO 21062 --- [ocp-async-6,d5a52a92d5d40d39,609de99502706458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:35.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:35.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c32fa47b0d35c6eb,57ef8015a96a9338] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:36.570 WARN 21062 --- [ocp-async-9,8bc4e3e0b8835b13,a832bd9f8e11fb0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:36.570 INFO 21062 --- [ocp-async-4,5ad477f14b27e08a,3082a9c8beb79c54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:38.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:38.440 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:38.504 INFO 21062 --- [ocp-schedules-25,00b281038d212137,941ef5798c4ebf1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:38:38.505 INFO 21062 --- [ocp-schedules-2,2fdf812e7665f5dd,aed4c4d3be88cce0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:38:38.536 INFO 21062 --- [ocp-async-7,19ca6c6ac8aaf1dd,360cb4326d3d77a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:38:38.570 INFO 21062 --- [ocp-async-3,66496db16df78b9a,d149d67f36fa837e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:38.570 WARN 21062 --- [ocp-async-1,92eff37a70664f74,a332d4a6e35dc0b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:38:38.588208+08:00, lastProcessTime=2024-11-04T12:38:28.587408+08:00 2024-11-04 12:38:38.625 INFO 21062 --- [ocp-schedules-19,f4bb68cae0c3ac98,9b02a03db28c4146] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:38:38.750 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:38:38.588208+08:00 2024-11-04 12:38:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c9ab37aa4670ca02,79b56d34d9581cd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:38:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:38:40.570 WARN 21062 --- [ocp-async-9,af24c29959cab238,335071881bf4e13e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:40.570 INFO 21062 --- [ocp-async-6,cf2c5d4b6b9a3ea5,80defd375211c203] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:40.574 INFO 21062 --- [ocp-schedules-5,0153cf8b2981be8d,4436bd170554aa5a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:38:41.440 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:41.441 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:38:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:38:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:38:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:38:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:38:41.852 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:38:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 12:38:41.983 INFO 21062 --- [prometheus-minute-schedule-1,53d6c1df476467a4,5754b9203f89ec73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:42.570 INFO 21062 --- [ocp-async-2,4408969f1a2d6b62,610dadae30aaac55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:42.570 WARN 21062 --- [ocp-async-4,0281e3a1d522ab63,e75697ac5bbdb932] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:44.441 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:44.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:44.570 WARN 21062 --- [ocp-async-8,a3d0d0f25ca24ade,9b04dc53a5cbae04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:44.570 INFO 21062 --- [ocp-async-1,652193055e4ee6c7,9ebbca79e099af4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:44.983 INFO 21062 --- [prometheus-minute-schedule-1,516561371ef5d084,417565248b5691e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:46.570 WARN 21062 --- [ocp-async-3,ebf4563af698952d,a6494e2027042dfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:46.571 INFO 21062 --- [ocp-async-9,4d90a2d1388238e0,03ce3f6a125c02d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:47.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:47.443 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e72d560502364e4f,62e95334688703c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:48.536 INFO 21062 --- [ocp-async-9,f9909b8e56a33bdb,ecee64bec74fd38f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:38:48.571 WARN 21062 --- [ocp-async-6,9ceede102f08299f,27c91b0adbf9c33b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:48.571 INFO 21062 --- [ocp-async-4,93ea47c080370e99,e6f4333f3b069cf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:38:48.589611+08:00, lastProcessTime=2024-11-04T12:38:38.588208+08:00 2024-11-04 12:38:48.628 INFO 21062 --- [ocp-schedules-20,20246a9f67b99425,23c7012550d66c29] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:38:48.742 INFO 21062 --- [ocp-schedules-13,d9278958c4ec731b,e6a5388eb57d572d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:38:48.748 INFO 21062 --- [ocp-schedules-13,d9278958c4ec731b,e6a5388eb57d572d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:38:48.748 INFO 21062 --- [ocp-schedules-13,d9278958c4ec731b,e6a5388eb57d572d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:38:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:38:48.589611+08:00 2024-11-04 12:38:50.443 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:50.444 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:50.571 WARN 21062 --- [ocp-async-8,1ca15b06eeaa7443,9e4d998aec047867] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:50.571 INFO 21062 --- [ocp-async-4,b8b2cdee4d75f67b,4c02057250020572] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:50.597 INFO 21062 --- [ocp-schedules-14,789efc5ae1405aa4,6a6d4ccd4fcf43f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:38:50.983 INFO 21062 --- [prometheus-minute-schedule-1,c245bbaad12d98a0,6d850bd223ed1401] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:52.571 WARN 21062 --- [ocp-async-3,68f0a807090b7cb9,4952c333cbc5edda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:52.571 INFO 21062 --- [ocp-async-4,e6f74a2cf38cf23f,4587311140cf8e56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:53.444 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:53.445 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e42d9c82a9ef7b26,b4692b77c2eb013b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:54.571 WARN 21062 --- [ocp-async-6,64336701419b16ca,e4aaa7749022dbd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:54.571 INFO 21062 --- [ocp-async-2,f61572f8e80217b6,65e3f432e7627899] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:56.445 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:56.446 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:56.572 INFO 21062 --- [ocp-async-2,83ad072ee9502ec0,7fca01706389c2dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:56.572 WARN 21062 --- [ocp-async-1,b0eae6a34955b5c6,f45035c9faf156f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:56.983 INFO 21062 --- [prometheus-minute-schedule-1,bbec0da4a7ab7235,c8a2ec5ab037e252] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:38:58.500 INFO 21062 --- [ocp-schedules-8,e1756ecbcb544114,c50c885737620316] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:38:58.500 INFO 21062 --- [ocp-schedules-9,b8c5c31ef256f604,13e2e823ddbc3286] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:38:58.502 INFO 21062 --- [ocp-schedules-26,e2240034121b8f0b,98e2af197c0cdeb3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:38:58.504 INFO 21062 --- [ocp-schedules-8,e1756ecbcb544114,c50c885737620316] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:58.504 INFO 21062 --- [ocp-schedules-9,b8c5c31ef256f604,13e2e823ddbc3286] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:38:58.505 INFO 21062 --- [ocp-schedules-29,3d101e8716bceaa9,8039f26106654893] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:38:58.513 INFO 21062 --- [ocp-schedules-13,7855b7df4ad0e176,beb294cde051d46f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:38:58.513 INFO 21062 --- [ocp-schedules-27,f0ee600488cc5031,867f48c165071189] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:38:58.513 INFO 21062 --- [ocp-schedules-27,f0ee600488cc5031,867f48c165071189] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:38:58.513 INFO 21062 --- [ocp-schedules-13,7855b7df4ad0e176,beb294cde051d46f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:38:58.515 INFO 21062 --- [ocp-schedules-18,fef0912fa79768ff,8ce1dbad9887c382] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:38:58.515 INFO 21062 --- [ocp-schedules-18,fef0912fa79768ff,8ce1dbad9887c382] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:38:58.536 INFO 21062 --- [ocp-async-10,7c4e8ef25daf4e21,f62a7d6b9267a5b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:38:58.572 WARN 21062 --- [ocp-async-5,c0ed10762eb3fc70,b64d4f65820dcded] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:38:58.572 INFO 21062 --- [ocp-async-3,4dd05b7fffe16028,d844f4d119f698e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:38:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:38:58.587566+08:00, lastProcessTime=2024-11-04T12:38:48.589611+08:00 2024-11-04 12:38:58.594 INFO 21062 --- [ocp-schedules-2,168ec56ff7b534dd,caf4136d6c5f1526] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:38:58.630 INFO 21062 --- [ocp-schedules-30,38b65db83205e440,f17f3d9170f3df67] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:38:58.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:38:58.587566+08:00 2024-11-04 12:38:58.984 INFO 21062 --- [ocp-schedules-10,4f77e99ef0f1627d,134f7a84b64c5df1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:38:58.985 INFO 21062 --- [ocp-schedules-10,02348cf2fe342af7,a01d7de61c84fb14] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:38:58.988 INFO 21062 --- [ocp-schedules-10,02348cf2fe342af7,a01d7de61c84fb14] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:38:59.015 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:38:59.015 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:38:59.446 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:38:59.448 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:38:59.983 INFO 21062 --- [prometheus-minute-schedule-1,3382ff21023d1edb,366cb5c08fc8cf21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695140, interval=60 2024-11-04 12:39:00.572 WARN 21062 --- [ocp-async-6,d46e20263e4defe2,1e711d7feee04d36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:00.572 INFO 21062 --- [ocp-async-4,1322b58df679d3a8,2800357e357d552d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:00.649 INFO 21062 --- [ocp-schedules-22,f83961bb561258c1,9f621aa75eb44d01] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:39:02.448 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:02.449 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:02.572 WARN 21062 --- [ocp-async-1,cbf08f05dcd013cd,3cce967347aacefe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:02.572 INFO 21062 --- [ocp-async-8,dbeefc7aa513f5c3,2f7b538f35b5fb88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2c7ea9bbb7ef7f77,16f5e6b536ea1fcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:04.572 WARN 21062 --- [ocp-async-10,4190ed8347d066e4,c834af14711b17a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:04.572 INFO 21062 --- [ocp-async-5,1173655c1dc1d1f4,f56291be8a5805d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:05.449 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:05.450 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:05.983 INFO 21062 --- [prometheus-minute-schedule-1,65cff2bea9833c0c,f0667cf2f4eda514] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:06.572 WARN 21062 --- [ocp-async-7,dad205740baa68c1,06af717688a3f3e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:06.572 INFO 21062 --- [ocp-async-3,e37621f7e5cf715d,86425a92a6ff28aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:08.450 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:08.451 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:08.506 INFO 21062 --- [ocp-schedules-8,e1756ecbcb544114,c50c885737620316] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:39:08.506 INFO 21062 --- [ocp-schedules-9,b8c5c31ef256f604,13e2e823ddbc3286] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:39:08.536 INFO 21062 --- [ocp-async-4,630aee1a4ea345bb,82aac39297e51aed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:39:08.573 WARN 21062 --- [ocp-async-8,b06f882d045220ef,4cfb692f5f816524] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:08.573 INFO 21062 --- [ocp-async-2,cc537f9ed262fe38,aa61c3e938a1c0db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:39:08.588245+08:00, lastProcessTime=2024-11-04T12:38:58.587566+08:00 2024-11-04 12:39:08.632 INFO 21062 --- [ocp-schedules-24,81f406ea4eca6837,ef97806b4ec204f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:39:08.750 INFO 21062 --- [ocp-schedules-8,fee809d987ab9c5a,765aacb613b668fc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:39:08.756 INFO 21062 --- [ocp-schedules-8,fee809d987ab9c5a,765aacb613b668fc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:39:08.756 INFO 21062 --- [ocp-schedules-8,fee809d987ab9c5a,765aacb613b668fc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:39:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:39:08.588245+08:00 2024-11-04 12:39:08.983 INFO 21062 --- [prometheus-minute-schedule-1,408d2742c172bacf,6a889a7bd3ea4f45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:39:10.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:39:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:39:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:39:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:39:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:39:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:39:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:39:10.573 WARN 21062 --- [ocp-async-5,b9258ad6f735415b,58131dcf82663876] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:10.573 INFO 21062 --- [ocp-async-10,96bf7c8310623026,8b564548e9e6fb9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:10.676 INFO 21062 --- [ocp-schedules-7,0703c70462be3c7d,4742fa00a04d3220] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:39:11.053 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695080, nodeCount=39, affectRows=39 2024-11-04 12:39:11.053 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695080, nodeCount=75, affectRows=75 2024-11-04 12:39:11.053 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695080, nodeCount=39, validCount=39 2024-11-04 12:39:11.053 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695080, nodeCount=65, affectRows=65 2024-11-04 12:39:11.054 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695080, nodeCount=73, validCount=73 2024-11-04 12:39:11.054 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695080, nodeCount=60, affectRows=60 2024-11-04 12:39:11.054 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695080, nodeCount=65, validCount=65 2024-11-04 12:39:11.054 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695080, nodeCount=59, validCount=59 2024-11-04 12:39:11.054 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695080, nodeCount=83, affectRows=83 2024-11-04 12:39:11.055 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695080, nodeCount=83, validCount=83 2024-11-04 12:39:11.452 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:11.453 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c0f3bbf55abd8925,e3e02db92404bef8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:12.573 INFO 21062 --- [ocp-async-7,cf008ec717fc5540,4b35a252e34937cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:12.573 WARN 21062 --- [ocp-async-3,ce73870d12c510a3,661a4f7bcd266d93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,3202bbe4097d233f,acfc8c241d2f4110] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,3202bbe4097d233f,acfc8c241d2f4110] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:39:14.064 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:35:14.034, end=2024-11-04T12:37:14.059} 2024-11-04 12:39:14.073 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:39:14.073 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:39:14.453 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:14.454 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:14.573 INFO 21062 --- [ocp-async-4,dae9c19cf9836658,829149f0fb1954bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:14.573 WARN 21062 --- [ocp-async-1,37630813322793b3,4939931be3223777] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:14.983 INFO 21062 --- [prometheus-minute-schedule-1,83158eb3257212a5,16df04204b4aca54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:16.573 WARN 21062 --- [ocp-async-2,3cac6a022c3334c4,e9ae89da6179f4ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:16.574 INFO 21062 --- [ocp-async-5,48988605e638e142,dc15f5cb335d965b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:17.454 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:17.455 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:17.983 INFO 21062 --- [prometheus-minute-schedule-1,be78ce347c23301f,8bf1c4c3cb11d743] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:18.537 INFO 21062 --- [ocp-async-10,1f1ba9be47608d7d,adeced878d892500] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:39:18.571 INFO 21062 --- [ocp-schedules-3,923c62011018e9df,5a6e73f41bbf17da] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:39:18.574 WARN 21062 --- [ocp-async-3,513dd9fcaa1e17c2,a43584b75088530f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:18.574 INFO 21062 --- [ocp-async-3,d9cf51e9ba377a24,8d7a9cdf975dd846] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:18.574 INFO 21062 --- [ocp-schedules-3,923c62011018e9df,5a6e73f41bbf17da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:18.575 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:39:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:39:18.587847+08:00, lastProcessTime=2024-11-04T12:39:08.588245+08:00 2024-11-04 12:39:18.635 INFO 21062 --- [ocp-schedules-29,b330756d8e77f38e,06e94e6bc983791f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:39:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:39:18.587847+08:00 2024-11-04 12:39:20.455 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:20.456 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:20.574 WARN 21062 --- [ocp-async-8,9cec44e72261ba08,92016db7ba460d8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:20.574 INFO 21062 --- [ocp-async-1,0f038065639b126d,d3de4c551f59a926] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:20.727 INFO 21062 --- [ocp-schedules-7,47ba130b705d692a,472384dcdd1f5a4c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:39:20.983 INFO 21062 --- [prometheus-minute-schedule-1,12a5d6ad03608917,9b24ffc9e06d171e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:22.574 WARN 21062 --- [ocp-async-9,5256a01d0cad2a76,23fba459b895be4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:22.574 INFO 21062 --- [ocp-async-2,64fb18efe089e6e4,9973591d53512347] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:23.456 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:23.457 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:23.983 INFO 21062 --- [prometheus-minute-schedule-1,71fc043c826e69c6,ba24661fa6a11427] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:24.574 INFO 21062 --- [ocp-async-10,511530f47df28d3a,5c385c05c4cec375] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:24.574 WARN 21062 --- [ocp-async-7,d1e41e917d222319,6e1e2770fe40c3e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:26.457 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:26.458 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:26.574 WARN 21062 --- [ocp-async-8,5133be56fa03a4ce,dfebe5afe29b8178] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:26.574 INFO 21062 --- [ocp-async-3,1a0ae292c2b6a47c,e9871b7916f3f9b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:26.983 INFO 21062 --- [prometheus-minute-schedule-1,35f60e7cd75fa26c,d1b2693e57fafb67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:26.984 INFO 21062 --- [prometheus-hour-schedule-1,02e0dd8caeccc265,17731ab70c645edd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:39:28.498 INFO 21062 --- [ocp-schedules-7,ffa2f1e961257191,3e9a4560ac35dc0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:39:28.499 INFO 21062 --- [ocp-schedules-19,2f985f5064a6081e,9cb4b038e2002f22] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:39:28.499 INFO 21062 --- [ocp-schedules-17,5ff5281acd71fa7e,4def77686c4c8d8c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:39:28.500 INFO 21062 --- [ocp-schedules-13,b8ff9a832d36f2ef,892a0a11db301969] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:39:28.502 INFO 21062 --- [ocp-schedules-7,ffa2f1e961257191,3e9a4560ac35dc0a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:39:28.502 INFO 21062 --- [ocp-schedules-12,f425de30de1614e2,e5c88743fcbcd4f5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:39:28.504 INFO 21062 --- [ocp-schedules-17,5ff5281acd71fa7e,4def77686c4c8d8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:39:28.504 INFO 21062 --- [ocp-schedules-19,2f985f5064a6081e,9cb4b038e2002f22] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:39:28.505 INFO 21062 --- [ocp-schedules-18,12d40ec2079ecefe,b0b5095ffcc2abbe] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:39:28.507 INFO 21062 --- [ocp-schedules-13,b8ff9a832d36f2ef,892a0a11db301969] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:28.512 INFO 21062 --- [ocp-schedules-9,acee8968ed6e3580,77321e5a5f7ec785] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:39:28.512 INFO 21062 --- [ocp-schedules-9,acee8968ed6e3580,77321e5a5f7ec785] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:39:28.512 INFO 21062 --- [ocp-schedules-11,8aee94a7a4a88f33,84a5d297a640ca5f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:39:28.512 INFO 21062 --- [ocp-schedules-11,8aee94a7a4a88f33,84a5d297a640ca5f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:39:28.512 INFO 21062 --- [ocp-schedules-29,02e7eb0c4c84649b,33f67f3228d7b573] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:39:28.512 INFO 21062 --- [ocp-schedules-29,02e7eb0c4c84649b,33f67f3228d7b573] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:39:28.528 INFO 21062 --- [ocp-schedules-4,ade98a7c6b32b7cb,e67580a940a10380] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T12:39:28.528+08:00, tenantSize=0 2024-11-04 12:39:28.537 INFO 21062 --- [ocp-async-9,5767e4498fca1009,87621235a98fb2b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:39:28.565 INFO 21062 --- [ocp-schedules-23,a7e0ec1d7d955d6e,0674a88f2b4ae67e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:39:28.566 INFO 21062 --- [ocp-schedules-23,a7e0ec1d7d955d6e,0674a88f2b4ae67e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:39:28.575 INFO 21062 --- [ocp-async-2,ba386271f4413543,317bed0f9c8883bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:28.575 WARN 21062 --- [ocp-async-7,a741c8020c968c79,b72039a4f9393bc1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:28.575 INFO 21062 --- [ocp-schedules-3,923c62011018e9df,5a6e73f41bbf17da] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:39:28.576 INFO 21062 --- [ocp-schedules-2,78a5a13b0c6b33e7,f420fe88167529c4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:39:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:39:28.587742+08:00, lastProcessTime=2024-11-04T12:39:18.587847+08:00 2024-11-04 12:39:28.595 INFO 21062 --- [ocp-schedules-19,4c2244190379c831,5b6311c4fba128e3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:39:28.595 INFO 21062 --- [ocp-schedules-19,4c2244190379c831,5b6311c4fba128e3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:39:28.597 INFO 21062 --- [ocp-schedules-6,fc7545f07d7f84db,aa161e86997d6d50] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:39:28.624 INFO 21062 --- [ocp-schedules-18,2f2d0d770c689068,cf3a62fa8902641d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:39:28.629 INFO 21062 --- [ocp-schedules-18,2f2d0d770c689068,cf3a62fa8902641d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:39:28.629 INFO 21062 --- [ocp-schedules-18,2f2d0d770c689068,cf3a62fa8902641d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:39:28.639 INFO 21062 --- [ocp-schedules-11,ee0c7bf8e481094a,2175226005a1c6c9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:39:28.680 INFO 21062 --- [ocp-schedules-7,955861f2b3689a60,04c8f0a714432235] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:39:28.680 INFO 21062 --- [ocp-schedules-7,955861f2b3689a60,04c8f0a714432235] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:39:28.686 INFO 21062 --- [ocp-schedules-7,955861f2b3689a60,04c8f0a714432235] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:39:28.686 INFO 21062 --- [ocp-schedules-7,955861f2b3689a60,04c8f0a714432235] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:39:28.692 INFO 21062 --- [ocp-schedules-7,955861f2b3689a60,04c8f0a714432235] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:39:28.692 INFO 21062 --- [ocp-schedules-7,955861f2b3689a60,04c8f0a714432235] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:39:28.713 INFO 21062 --- [ocp-schedules-30,d438369e4a417491,dd7047b2ddb13526] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:39:28.713 INFO 21062 --- [ocp-schedules-30,d438369e4a417491,dd7047b2ddb13526] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:39:28.732 INFO 21062 --- [ocp-schedules-15,626a685b16a659bc,7a8c0a5f8d8cf902] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:39:28.733 INFO 21062 --- [ocp-schedules-15,626a685b16a659bc,7a8c0a5f8d8cf902] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:39:28.733 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:39:28.733 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:39:28.740 INFO 21062 --- [ocp-schedules-30,d438369e4a417491,dd7047b2ddb13526] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:39:28.740 INFO 21062 --- [ocp-schedules-30,d438369e4a417491,dd7047b2ddb13526] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:39:28.742 INFO 21062 --- [ocp-schedules-15,626a685b16a659bc,7a8c0a5f8d8cf902] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:39:28.743 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:39:28.743 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:39:28.744 WARN 21062 --- [ocp-schedules-15,626a685b16a659bc,7a8c0a5f8d8cf902] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:39:28.745 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:39:28.745 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:39:28.745 INFO 21062 --- [ocp-schedules-15,626a685b16a659bc,7a8c0a5f8d8cf902] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:39:28.746 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:39:28.746 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:39:28.746 WARN 21062 --- [ocp-schedules-15,626a685b16a659bc,7a8c0a5f8d8cf902] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:39:28.747 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:39:28.747 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:39:28.754 ERROR 21062 --- [ocp-schedules-15,626a685b16a659bc,7a8c0a5f8d8cf902] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2024-11-04 12:39:28.758 INFO 21062 --- [ocp-schedules-26,42f1876c006cd562,2ab0870e459b3449] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:39:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:39:28.587742+08:00 2024-11-04 12:39:28.763 INFO 21062 --- [ocp-schedules-26,42f1876c006cd562,2ab0870e459b3449] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:39:28.763 INFO 21062 --- [ocp-schedules-26,42f1876c006cd562,2ab0870e459b3449] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:39:28.767 INFO 21062 --- [ocp-schedules-20,16a0649b9148e50e,f9d74122a80df496] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:39:28.833 INFO 21062 --- [ocp-schedules-14,11114961ed8841a3,c52f174be9a1ca8b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:39:28.833+08:00, clusters=[1] 2024-11-04 12:39:28.835 INFO 21062 --- [ocp-schedules-14,11114961ed8841a3,c52f174be9a1ca8b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:39:28.900 INFO 21062 --- [ocp-schedules-27,1accda2f5786fd06,52769b752238e688] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:39:28.900+08:00, clusters=[1] 2024-11-04 12:39:28.904 INFO 21062 --- [ocp-schedules-27,1accda2f5786fd06,52769b752238e688] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:39:29.018 INFO 21062 --- [ocp-schedules-10,02348cf2fe342af7,a01d7de61c84fb14] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:39:29.018 INFO 21062 --- [ocp-schedules-10,6c09450a01b4f99a,a5a4e5f982352ff1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:39:29.021 INFO 21062 --- [ocp-schedules-10,6c09450a01b4f99a,a5a4e5f982352ff1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:29.050 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:39:29.050 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:39:29.145 INFO 21062 --- [ocp-schedules-22,2c6a96732b9c908c,b9b26618225eb63d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:39:29.149 INFO 21062 --- [ocp-schedules-22,2c6a96732b9c908c,b9b26618225eb63d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:29.234 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:39:29.235 ERROR 21062 --- [ocp-schedules-22,2c6a96732b9c908c,b9b26618225eb63d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:39:29.236 INFO 21062 --- [ocp-schedules-22,2c6a96732b9c908c,b9b26618225eb63d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:39:29.458 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:29.459 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:29.983 INFO 21062 --- [prometheus-minute-schedule-1,28a80c9d2f47b1c1,3092a12d0b380d0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:39:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:39:30.575 WARN 21062 --- [ocp-async-4,d293ded45e7ab2b2,9110960f94adc692] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:30.575 INFO 21062 --- [ocp-async-10,4a8e85c750fd649b,cf91f14a805a16fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:30.794 INFO 21062 --- [ocp-schedules-2,aa16beffd0abf3ec,6bb2df0af1366a20] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:39:32.459 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:32.460 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:32.575 INFO 21062 --- [ocp-async-1,c12b2b6aaa7af96f,d88cc82be04e47b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:32.575 WARN 21062 --- [ocp-async-5,66ecd723496f657c,fe348e937e186ce6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:32.983 INFO 21062 --- [prometheus-minute-schedule-1,70c3417794973f21,f09548d6221e8ceb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:34.575 WARN 21062 --- [ocp-async-7,09fb8fa0f49a3b54,ef0576624e69c2c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:34.575 INFO 21062 --- [ocp-async-6,058cc55e0684176b,f0b61d843f11b3c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:35.460 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:35.461 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:35.983 INFO 21062 --- [prometheus-minute-schedule-1,9bd6997fcf85dca8,a8c2b7a00c9f20ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:36.575 WARN 21062 --- [ocp-async-2,88b486d84f049063,129c815d8049bdb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:36.575 INFO 21062 --- [ocp-async-8,dc94ba4af461f739,67fd6d675397cd46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:38.461 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:38.462 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:38.505 INFO 21062 --- [ocp-schedules-17,5ff5281acd71fa7e,4def77686c4c8d8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:39:38.508 INFO 21062 --- [ocp-schedules-13,b8ff9a832d36f2ef,892a0a11db301969] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:39:38.537 INFO 21062 --- [ocp-async-10,47203789ed623300,21f5bd86286a6c02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:39:38.576 WARN 21062 --- [ocp-async-3,222b92b9ef859065,a67049e926d075ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:38.576 INFO 21062 --- [ocp-async-3,d06350ae041962ec,bd68eaf48b6dca9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:39:38.588244+08:00, lastProcessTime=2024-11-04T12:39:28.587742+08:00 2024-11-04 12:39:38.641 INFO 21062 --- [ocp-schedules-16,8beee4c970fbef48,a04beb778cc8c105] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:39:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:39:38.588244+08:00 2024-11-04 12:39:38.983 INFO 21062 --- [prometheus-minute-schedule-1,1339aac657e82f90,8c11441a90a526b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:39:40.004 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:39:40.576 WARN 21062 --- [ocp-async-6,eb03434d21110038,0aa8cdf1e6197fb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:40.576 INFO 21062 --- [ocp-async-9,679024650aae0b1d,66d9b7b6b5ba1824] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:40.839 INFO 21062 --- [ocp-schedules-2,f0fe357aeb27ff05,5d8526cb1462e3f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:39:41.462 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:41.464 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:41.983 INFO 21062 --- [prometheus-minute-schedule-1,423120ec7c2a91e9,7af2b1baccb301d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:42.576 WARN 21062 --- [ocp-async-4,715cc41d45bdc8bf,9c044dce4b215d4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:42.576 INFO 21062 --- [ocp-async-8,574a510a544aa4a4,d2846b91d3ba5378] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:44.464 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:44.465 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:44.576 WARN 21062 --- [ocp-async-5,4b8309471e12c29e,c0cf7033b3916183] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:44.576 INFO 21062 --- [ocp-async-5,96a8678074657eee,37fd067a2b845aa7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f7444daefb49aff3,ac698e5be83851b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:46.576 WARN 21062 --- [ocp-async-2,f766945ce4c23f37,6aa66531cc2c27dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:46.576 INFO 21062 --- [ocp-async-3,acbec9b2a1516688,b62165f62026a9db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:47.465 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:47.466 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:47.983 INFO 21062 --- [prometheus-minute-schedule-1,11a00c4a23df5900,1412ca4a625ad204] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:48.537 INFO 21062 --- [ocp-async-3,a726afcb8eaa2074,a05920e53c9055ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:39:48.577 WARN 21062 --- [ocp-async-4,424d20483856a2fc,370a6d956c5c2de6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:48.577 INFO 21062 --- [ocp-async-9,d836e49c1338666e,634c4a205eed17cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:39:48.589079+08:00, lastProcessTime=2024-11-04T12:39:38.588244+08:00 2024-11-04 12:39:48.643 INFO 21062 --- [ocp-schedules-28,5ad931c5b592a145,bed8363248059b14] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:39:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:39:48.589079+08:00 2024-11-04 12:39:48.765 INFO 21062 --- [ocp-schedules-3,69248ae135079093,aca0ae96cf8f1559] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:39:48.770 INFO 21062 --- [ocp-schedules-3,69248ae135079093,aca0ae96cf8f1559] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:39:48.770 INFO 21062 --- [ocp-schedules-3,69248ae135079093,aca0ae96cf8f1559] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:39:50.466 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:50.467 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:50.577 INFO 21062 --- [ocp-async-9,9e855a3cf9996318,f9f5acbc69d3b466] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:50.577 WARN 21062 --- [ocp-async-7,7280f33935d84590,edb231d4f94cbabe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:50.868 INFO 21062 --- [ocp-schedules-8,3c61cd5d56565fea,70b044167b711a7b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:39:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1783ae3c6663bc82,dfef0634c46d3fe8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:52.577 WARN 21062 --- [ocp-async-2,193c9e3f80d01729,ee49d833da69c4e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:52.577 INFO 21062 --- [ocp-async-6,e16b48e21f162c94,6d4401ea1b38d0fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:53.467 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:53.468 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:53.983 INFO 21062 --- [prometheus-minute-schedule-1,adc642dea033c477,c23162d924eb70f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:54.577 INFO 21062 --- [ocp-async-4,581182b6a7f309eb,ee2095a366b58462] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:54.577 WARN 21062 --- [ocp-async-8,ec214563cd778df6,dfd8716af86bb13b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:56.468 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:56.469 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:56.577 INFO 21062 --- [ocp-async-7,e3a7980ced75b78a,46c45af8fb968baf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:56.577 WARN 21062 --- [ocp-async-1,c3ace122d2b84956,4263ac7c2a6939fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d31bd12d9470b021,7327d2ac5976f68b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:39:58.503 INFO 21062 --- [ocp-schedules-25,fbe79eb3c81cd2fd,23eb52cc315b0534] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:39:58.503 INFO 21062 --- [ocp-schedules-6,99ea552f80298b87,e1a603d3331b61fa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:39:58.503 INFO 21062 --- [ocp-schedules-11,81abaae81336d552,049cbe78089afa93] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:39:58.507 INFO 21062 --- [ocp-schedules-29,50f722a9a35cecc7,ab7e84edf3665baf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:39:58.507 INFO 21062 --- [ocp-schedules-6,99ea552f80298b87,e1a603d3331b61fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:58.507 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:39:58.508 INFO 21062 --- [ocp-schedules-25,fbe79eb3c81cd2fd,23eb52cc315b0534] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:58.511 INFO 21062 --- [ocp-schedules-1,b3de81bff00fe6cb,23b2f23a5bc8f775] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:39:58.511 INFO 21062 --- [ocp-schedules-18,8d938417c3b5cd20,c3e9a641c5a6293b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:39:58.511 INFO 21062 --- [ocp-schedules-4,6c7a4d4afbf0321a,29c2f686c1892a97] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:39:58.511 INFO 21062 --- [ocp-schedules-18,8d938417c3b5cd20,c3e9a641c5a6293b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:39:58.511 INFO 21062 --- [ocp-schedules-4,6c7a4d4afbf0321a,29c2f686c1892a97] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:39:58.511 INFO 21062 --- [ocp-schedules-1,b3de81bff00fe6cb,23b2f23a5bc8f775] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:39:58.537 INFO 21062 --- [ocp-async-5,68f9d284a9d30c2e,fd4356d3e97f00e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:39:58.577 WARN 21062 --- [ocp-async-9,2a7890aa78da16f8,6ef0234f59921ed5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:39:58.577 INFO 21062 --- [ocp-async-2,3a580183525a0bce,dde6cb921a7bdec3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:39:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:39:58.588028+08:00, lastProcessTime=2024-11-04T12:39:48.589079+08:00 2024-11-04 12:39:58.600 INFO 21062 --- [ocp-schedules-3,4c5c76f70d5dffd6,e1267bf206f9faa4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:39:58.646 INFO 21062 --- [ocp-schedules-20,a24a589635b42f01,d722c361147cb0f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:39:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:39:58.588028+08:00 2024-11-04 12:39:59.050 INFO 21062 --- [ocp-schedules-10,6c09450a01b4f99a,a5a4e5f982352ff1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:39:59.051 INFO 21062 --- [ocp-schedules-10,f26033ea81b248a1,4bb6bcf9079562e0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:39:59.055 INFO 21062 --- [ocp-schedules-10,f26033ea81b248a1,4bb6bcf9079562e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:39:59.086 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:39:59.086 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:39:59.469 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:39:59.470 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:39:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2ec0991ab11220ee,20d3c29790efac7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695200, interval=60 2024-11-04 12:40:00.529 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e11240ffd30ee6c8] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:35:01.0, nextRun=2024-11-04 12:40:00.0, currentDate=Mon Nov 04 12:40:00 CST 2024 2024-11-04 12:40:00.529 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e11240ffd30ee6c8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 12:35:01.0, nextRunTime=2024-11-04 12:40:00.0} 2024-11-04 12:40:00.560 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e11240ffd30ee6c8] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 12:40:00.566 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e11240ffd30ee6c8] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 12:40:00.566 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e11240ffd30ee6c8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 12:40:00.578 WARN 21062 --- [ocp-async-6,a30f9593184db13e,e3321f21f6a1e02a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:00.578 INFO 21062 --- [ocp-async-8,2448ac03f5c1fff6,6c4ea7c6fcf567bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:00.914 INFO 21062 --- [ocp-schedules-14,42778f94c747fdab,e60e76d33db13831] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:40:02.470 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:02.472 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:02.578 WARN 21062 --- [ocp-async-1,32c240a7bf1c104f,510861fafc6e093b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:02.578 INFO 21062 --- [ocp-async-10,bda81fe976bc1c46,bdeb2a58c208ce1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:02.983 INFO 21062 --- [prometheus-minute-schedule-1,916f404f86d0eaa4,c7b9941c7baff346] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:04.578 WARN 21062 --- [ocp-async-5,9b1678e4754b500b,4449b5389f50c76d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:04.578 INFO 21062 --- [ocp-async-9,501f638430155f96,2d529938f6b0b89a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:04.635 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,82c14333988185b8] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:32:39.0, nextRun=2024-11-04 12:40:04.0, currentDate=Mon Nov 04 12:40:04 CST 2024 2024-11-04 12:40:04.635 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,82c14333988185b8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 12:32:39.0, nextRunTime=2024-11-04 12:40:04.0} 2024-11-04 12:40:04.642 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,82c14333988185b8] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 12:40:04.642 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,82c14333988185b8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T12:40:04.642+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 12:40:05.472 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:05.472 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:05.577 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,ba2b1403be3563a9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 69, operation:EXECUTE, state: RUNNING 2024-11-04 12:40:05.596 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=69, context=Context{parallelIdx=-1, stringMap={task_instance_id=3, task_operation=execute, latest_execution_start_time=2024-11-04T12:40:05.568+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=69}, listMap={}}, executor=10.46.128.24 2024-11-04 12:40:05.601 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 12:40:05.601 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 12:40:05.603 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.604 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:40:05.605 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.605 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.606 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.606 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 12:40:05.674 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.675 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.676 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.676 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 12:40:05.713 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.714 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.715 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.715 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 12:40:05.754 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.754 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.755 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.755 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 12:40:05.795 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.796 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.797 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.797 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 12:40:05.829 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.830 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.831 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.831 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 12:40:05.850 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.851 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.852 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.852 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 12:40:05.867 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.868 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.868 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.869 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 12:40:05.932 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.933 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.934 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.934 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 12:40:05.962 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.963 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.963 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.964 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 12:40:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e83d484d4290c735,d42b3b017cf544ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:05.993 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.994 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:05.994 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:05.995 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 12:40:06.004 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.005 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.005 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.006 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 12:40:06.019 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.019 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.020 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.020 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 12:40:06.025 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.026 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.027 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.027 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 12:40:06.033 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.034 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.035 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.035 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 12:40:06.055 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.055 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.056 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.056 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 12:40:06.061 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.061 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.062 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.062 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 12:40:06.080 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.080 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.081 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.081 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 12:40:06.091 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.091 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.092 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.092 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 12:40:06.102 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.102 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.103 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.103 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 12:40:06.112 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.113 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.113 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.113 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 12:40:06.118 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.119 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.120 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.120 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 12:40:06.124 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.125 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.125 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.125 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 12:40:06.130 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.130 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.131 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.131 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 12:40:06.140 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.140 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.141 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.141 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 12:40:06.145 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.146 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.146 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.147 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 12:40:06.164 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.164 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.165 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.165 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 12:40:06.176 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.176 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.177 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.178 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 12:40:06.182 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.183 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.183 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.184 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 12:40:06.201 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.201 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.202 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.202 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 12:40:06.212 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.212 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.213 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.213 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 12:40:06.230 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.230 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.231 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.231 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 12:40:06.236 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.236 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.237 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.237 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 12:40:06.253 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.253 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.254 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.255 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 12:40:06.286 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.287 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.287 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.288 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 12:40:06.294 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.295 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.295 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.296 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 12:40:06.313 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.314 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.315 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.316 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 12:40:06.336 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.337 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.338 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.338 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 12:40:06.348 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.349 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.349 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.350 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 12:40:06.354 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.355 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.355 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.356 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 12:40:06.371 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.372 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.372 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.373 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 12:40:06.382 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.382 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.383 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.383 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 12:40:06.393 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.393 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.394 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.394 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 12:40:06.425 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.426 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.427 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.427 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 12:40:06.439 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.440 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.441 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.441 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 12:40:06.453 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.454 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.455 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.455 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 12:40:06.467 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.467 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.468 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.469 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 12:40:06.485 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.485 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.486 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.486 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 12:40:06.501 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.501 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.502 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.502 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 12:40:06.540 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 12:40:06.540 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 12:40:06.542 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.542 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:40:06.546 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.547 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.547 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.548 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 12:40:06.566 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.567 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:40:06.568 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:40:06.568 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 12:40:06.578 WARN 21062 --- [ocp-async-2,9326edd0516df490,b2c781606890bbee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:06.578 INFO 21062 --- [ocp-async-4,a4f815cfd6c09718,3e09ec6e7a8fd096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:06.578 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 12:40:06.588 INFO 21062 --- [subtask-executor2,dd193f8f67a5a423,fa05052f350aa8ce] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 69, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:40:07.426 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3, status:SUCCESS 2024-11-04 12:40:07.428 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3, status:SUCCESS 2024-11-04 12:40:08.473 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:08.473 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:08.508 INFO 21062 --- [ocp-schedules-6,99ea552f80298b87,e1a603d3331b61fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:40:08.508 INFO 21062 --- [ocp-schedules-25,fbe79eb3c81cd2fd,23eb52cc315b0534] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:40:08.538 INFO 21062 --- [ocp-async-8,c3b0f41b31606af5,abfa8bfffbb29833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:40:08.578 WARN 21062 --- [ocp-async-1,2eaf15f18b403961,f36c700fa9f60dbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:08.578 INFO 21062 --- [ocp-async-7,c077c128743ad6b0,d39bd8adc6017710] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:40:08.587887+08:00, lastProcessTime=2024-11-04T12:39:58.588028+08:00 2024-11-04 12:40:08.648 INFO 21062 --- [ocp-schedules-15,8ad1cb53391c765f,597ab45cceb987f9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:40:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:40:08.587887+08:00 2024-11-04 12:40:08.772 INFO 21062 --- [ocp-schedules-4,93a10770da64ae86,69f3ec94ea1f721a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:40:08.777 INFO 21062 --- [ocp-schedules-4,93a10770da64ae86,69f3ec94ea1f721a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:40:08.777 INFO 21062 --- [ocp-schedules-4,93a10770da64ae86,69f3ec94ea1f721a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:40:08.983 INFO 21062 --- [prometheus-minute-schedule-1,cdccb0bcf1d017a9,ff423b4bc46ada51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:10.004 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:40:10.004 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:40:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:40:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:40:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:40:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:40:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:40:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:40:10.579 WARN 21062 --- [ocp-async-5,3d511414c9b66b6f,a3ca597c129db28f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:10.579 INFO 21062 --- [ocp-async-3,3fbf138313b9808e,0f87c87e7a9dc8c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:10.944 INFO 21062 --- [ocp-schedules-8,d83b6e426c82da4e,1c62b3d34813e40b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:40:11.059 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=2, affectRows=2 2024-11-04 12:40:11.059 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=2, validCount=2 2024-11-04 12:40:11.059 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=3, affectRows=3 2024-11-04 12:40:11.059 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=3, validCount=3 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=10, affectRows=10 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=39, affectRows=39 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=10, validCount=10 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=18, affectRows=18 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=38, affectRows=38 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=25, affectRows=25 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=50, affectRows=50 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=83, affectRows=83 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=18, validCount=18 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695140, nodeCount=54, affectRows=54 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=25, validCount=25 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=39, validCount=39 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=38, validCount=38 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=50, validCount=50 2024-11-04 12:40:11.061 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=53, validCount=53 2024-11-04 12:40:11.062 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695140, nodeCount=81, validCount=81 2024-11-04 12:40:11.474 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:11.475 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6e45ffbbfef3fda3,b6fecd8ffd2db847] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:12.579 WARN 21062 --- [ocp-async-2,96116992c0282918,f30d2307fd719a7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:12.579 INFO 21062 --- [ocp-async-6,16752568d9da0fb5,24d8e03e8e7c5a70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,28d9d7dadd4791cd,c6e7cdcfb0a3ae64] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,28d9d7dadd4791cd,c6e7cdcfb0a3ae64] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:40:14.475 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:14.476 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:14.579 WARN 21062 --- [ocp-async-10,f643322277d12cdd,3ab83209f8551880] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:14.579 INFO 21062 --- [ocp-async-8,980ab19ef02db770,828299091322f73b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9a76b815c9e9059a,7c1c7905dc3c04e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:16.579 WARN 21062 --- [ocp-async-7,205c9586bd355380,a424f6019139f8b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:16.579 INFO 21062 --- [ocp-async-9,42f643b03509d497,5e3eccc18cee3915] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:17.476 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:17.477 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ee706b140f1487e7,4daa6a8900b4f21c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:18.538 INFO 21062 --- [ocp-async-3,64779680f4183873,f11d2c6bfed37d22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:40:18.579 WARN 21062 --- [ocp-async-2,5c33ee60e29d3621,b16053c61b905550] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:18.579 INFO 21062 --- [ocp-async-2,38b60155797ac75e,6159fdb168865138] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:40:18.587840+08:00, lastProcessTime=2024-11-04T12:40:08.587887+08:00 2024-11-04 12:40:18.651 INFO 21062 --- [ocp-schedules-30,a95b8839096c08ce,c6ba6b4ac49b6194] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:40:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:40:18.587840+08:00 2024-11-04 12:40:20.477 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:20.478 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:20.580 INFO 21062 --- [ocp-async-10,72723b1982333ea6,546d75e61935b77b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:20.580 WARN 21062 --- [ocp-async-1,8756cd2eea4ba7ec,33b81cb72fa0735e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:20.983 INFO 21062 --- [prometheus-minute-schedule-1,1c8e7a8b0b1cac50,d3e2a024eb5517c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:20.989 INFO 21062 --- [ocp-schedules-14,fba61fce0d23ec73,1a7278128e7fc3cd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:40:22.580 WARN 21062 --- [ocp-async-5,057d428c5c4e2d77,2be5432b2a54c909] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:22.580 INFO 21062 --- [ocp-async-7,05f09807dcc12363,0506164eab124b47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:23.479 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:23.480 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:23.983 INFO 21062 --- [prometheus-minute-schedule-1,aecdcdd527d6ff3a,a51fe0c5a654a7d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:24.580 INFO 21062 --- [ocp-async-3,37454322fcc3d1f8,4927576fa089d79c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:24.580 WARN 21062 --- [ocp-async-4,bc56cfb12e069e4c,a4d7a76bc756afe5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:26.480 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:26.481 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:26.580 WARN 21062 --- [ocp-async-2,5489c7808273fe95,a4c7007180c667f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:26.580 INFO 21062 --- [ocp-async-1,eb977b1cc44f31f1,7afbf97ea60d16e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:26.983 INFO 21062 --- [prometheus-minute-schedule-1,7ea7431915e95af1,9deb5ddd52a262d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ad42804890caf446,24744f0a0cdc1b8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:40:28.498 INFO 21062 --- [ocp-schedules-11,54c0be0070ca1df2,02e6e36502a67a0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:40:28.499 INFO 21062 --- [ocp-schedules-17,c4bb5e99db6638ea,2a0419d89e353d1f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:40:28.500 INFO 21062 --- [ocp-schedules-20,3b8e3f60a8057c34,ba60ccd11db3421b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:40:28.500 INFO 21062 --- [ocp-schedules-12,b9981e7e9f313452,9674ed2c243be3aa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:40:28.502 INFO 21062 --- [ocp-schedules-11,54c0be0070ca1df2,02e6e36502a67a0e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:40:28.502 INFO 21062 --- [ocp-schedules-6,cf435ddc3aee94b6,5d7b7bafce9c6f45] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:40:28.503 INFO 21062 --- [ocp-schedules-12,b9981e7e9f313452,9674ed2c243be3aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:28.504 INFO 21062 --- [ocp-schedules-20,3b8e3f60a8057c34,ba60ccd11db3421b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:28.504 INFO 21062 --- [ocp-schedules-21,371db72f212363f1,cfebe104ad20e444] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:40:28.504 INFO 21062 --- [ocp-schedules-17,c4bb5e99db6638ea,2a0419d89e353d1f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:40:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:40:28.511 INFO 21062 --- [ocp-schedules-7,d161d5462dbb58f1,c4ea3d1536d84e42] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:40:28.511 INFO 21062 --- [ocp-schedules-7,d161d5462dbb58f1,c4ea3d1536d84e42] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:40:28.512 INFO 21062 --- [ocp-schedules-8,4d1b5bf91d9f6cae,0a28ab3293273181] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:40:28.512 INFO 21062 --- [ocp-schedules-8,4d1b5bf91d9f6cae,0a28ab3293273181] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:40:28.512 INFO 21062 --- [ocp-schedules-22,6c71350afdfcf2c3,e87917b353fff4ae] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:40:28.512 INFO 21062 --- [ocp-schedules-22,6c71350afdfcf2c3,e87917b353fff4ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:40:28.538 INFO 21062 --- [ocp-async-5,8261f85ca1a0b193,cd670cb22fbc4f78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:40:28.553 INFO 21062 --- [ocp-schedules-27,38b4e23fdf975398,d076ed38873eb64d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:40:28.553 INFO 21062 --- [ocp-schedules-27,38b4e23fdf975398,d076ed38873eb64d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:40:28.566 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:40:28.566 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:40:28.573 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:40:28.573 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:40:28.573 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:40:28.576 INFO 21062 --- [ocp-schedules-9,d8bd21a90731be55,ab0c935b0b5a9114] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:40:28.579 INFO 21062 --- [ocp-schedules-9,d8bd21a90731be55,ab0c935b0b5a9114] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:28.579 INFO 21062 --- [ocp-schedules-5,d1ebff6e590c0d55,e31e2f76bed60ac5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:40:28.579 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:40:28.580 INFO 21062 --- [ocp-async-7,386040060de7ef6c,e01018d64541ac68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:28.580 WARN 21062 --- [ocp-async-4,cf9affe08c957626,e34b6b28a729ac53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:28.583 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:40:28.583 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:40:28.583 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:40:28.586 INFO 21062 --- [ocp-schedules-19,13e3ddff0945544d,9db3ab30dc6f3d60] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:40:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:40:28.588193+08:00, lastProcessTime=2024-11-04T12:40:18.587840+08:00 2024-11-04 12:40:28.599 INFO 21062 --- [ocp-schedules-17,2ac1f4337bb1745e,d576af94096fae60] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:40:28.599 INFO 21062 --- [ocp-schedules-17,2ac1f4337bb1745e,d576af94096fae60] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:40:28.603 INFO 21062 --- [ocp-schedules-21,2fac57a22b055c56,c669aa0daf85be78] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:40:28.629 INFO 21062 --- [ocp-schedules-7,785141c906bd6253,c68108ecd0c0f81e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:40:28.635 INFO 21062 --- [ocp-schedules-7,785141c906bd6253,c68108ecd0c0f81e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:40:28.635 INFO 21062 --- [ocp-schedules-7,785141c906bd6253,c68108ecd0c0f81e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:40:28.654 INFO 21062 --- [ocp-schedules-22,6546ea04264ed3be,b1fccaf414c21052] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:40:28.696 INFO 21062 --- [ocp-schedules-29,5849ce674cf68499,b962cebe60039504] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:40:28.696 INFO 21062 --- [ocp-schedules-29,5849ce674cf68499,b962cebe60039504] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:40:28.697 INFO 21062 --- [ocp-schedules-29,5849ce674cf68499,b962cebe60039504] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:40:28.697 INFO 21062 --- [ocp-schedules-29,5849ce674cf68499,b962cebe60039504] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:40:28.698 INFO 21062 --- [ocp-schedules-29,5849ce674cf68499,b962cebe60039504] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:40:28.698 INFO 21062 --- [ocp-schedules-29,5849ce674cf68499,b962cebe60039504] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:40:28.730 INFO 21062 --- [ocp-schedules-24,bc51bedad66423a4,7cc755b5efa20f1c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:40:28.735 INFO 21062 --- [ocp-schedules-24,bc51bedad66423a4,7cc755b5efa20f1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:28.741 INFO 21062 --- [ocp-schedules-23,cfac46ddabc3058d,0f8c2784ebb4053f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:40:28.741 INFO 21062 --- [ocp-schedules-23,cfac46ddabc3058d,0f8c2784ebb4053f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:40:28.762 INFO 21062 --- [ocp-schedules-26,c4e5d92310aa8320,f279381aabd809be] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:40:28.762 INFO 21062 --- [ocp-schedules-26,c4e5d92310aa8320,f279381aabd809be] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:40:28.762 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:40:28.762 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:40:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:40:28.588193+08:00 2024-11-04 12:40:28.765 INFO 21062 --- [ocp-schedules-23,cfac46ddabc3058d,0f8c2784ebb4053f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:40:28.765 INFO 21062 --- [ocp-schedules-23,cfac46ddabc3058d,0f8c2784ebb4053f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:40:28.773 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:40:28.773 INFO 21062 --- [ocp-schedules-26,c4e5d92310aa8320,f279381aabd809be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:40:28.773 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:40:28.775 WARN 21062 --- [ocp-schedules-26,c4e5d92310aa8320,f279381aabd809be] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:40:28.775 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:40:28.775 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:40:28.776 INFO 21062 --- [ocp-schedules-26,c4e5d92310aa8320,f279381aabd809be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:40:28.776 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:40:28.776 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:40:28.777 WARN 21062 --- [ocp-schedules-26,c4e5d92310aa8320,f279381aabd809be] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:40:28.777 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:40:28.777 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:40:28.779 INFO 21062 --- [ocp-schedules-27,18a0b36f1d472138,def366c83fef40ce] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:40:28.786 ERROR 21062 --- [ocp-schedules-26,c4e5d92310aa8320,f279381aabd809be] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:40:28.787 INFO 21062 --- [ocp-schedules-27,18a0b36f1d472138,def366c83fef40ce] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:40:28.787 INFO 21062 --- [ocp-schedules-27,18a0b36f1d472138,def366c83fef40ce] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:40:28.793 INFO 21062 --- [ocp-schedules-18,adfda997afb2916b,991ae737479a6060] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:40:28.797 ERROR 21062 --- [ocp-schedules-24,bc51bedad66423a4,7cc755b5efa20f1c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:40:28.797 INFO 21062 --- [ocp-schedules-24,bc51bedad66423a4,7cc755b5efa20f1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:40:28.859 INFO 21062 --- [ocp-schedules-30,aa150746f19f1b5b,c03283f8abdbe677] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:40:28.859+08:00, clusters=[1] 2024-11-04 12:40:28.860 INFO 21062 --- [ocp-schedules-30,aa150746f19f1b5b,c03283f8abdbe677] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:40:28.931 INFO 21062 --- [ocp-schedules-28,8d5a490bf5652220,64e1ba29a2bee821] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:40:28.931+08:00, clusters=[1] 2024-11-04 12:40:28.938 INFO 21062 --- [ocp-schedules-28,8d5a490bf5652220,64e1ba29a2bee821] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:40:29.087 INFO 21062 --- [ocp-schedules-10,f26033ea81b248a1,4bb6bcf9079562e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:40:29.087 INFO 21062 --- [ocp-schedules-10,9370527f28efd281,e9560ad6807eb12e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:40:29.090 INFO 21062 --- [ocp-schedules-10,9370527f28efd281,e9560ad6807eb12e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:29.118 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:40:29.118 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:40:29.236 INFO 21062 --- [ocp-schedules-4,5d7ae43da8d3800f,804b45daa0dc7224] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:40:29.239 INFO 21062 --- [ocp-schedules-4,5d7ae43da8d3800f,804b45daa0dc7224] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:29.324 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:40:29.326 ERROR 21062 --- [ocp-schedules-4,5d7ae43da8d3800f,804b45daa0dc7224] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:40:29.326 INFO 21062 --- [ocp-schedules-4,5d7ae43da8d3800f,804b45daa0dc7224] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:40:29.481 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:29.482 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:29.983 INFO 21062 --- [prometheus-minute-schedule-1,cb06a9987a7be07a,fb635b46989aacc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:40:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:40:30.581 WARN 21062 --- [ocp-async-2,75405459d25325b9,5a4e349e2da9e783] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:30.581 INFO 21062 --- [ocp-async-3,d6c3b5826d8dc218,5cb288bcc351e63d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:31.059 INFO 21062 --- [ocp-schedules-6,65e678fcc5b6d582,5e279bd38cef524e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:40:32.482 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:32.483 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:32.581 WARN 21062 --- [ocp-async-1,b8a9fda5a52c6990,4dc178314c94a4d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:32.581 INFO 21062 --- [ocp-async-10,70d5edc4402881e6,b2d32d6b2bcd0028] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:32.983 INFO 21062 --- [prometheus-minute-schedule-1,e0ad01f9a20ab469,55b715df7072f0be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:34.581 INFO 21062 --- [ocp-async-9,ede445a1a414ec3b,b3e3709769b64e57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:34.581 WARN 21062 --- [ocp-async-4,3d1e2e85ace922ff,b398ff49349ab829] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:35.483 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:35.484 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:35.983 INFO 21062 --- [prometheus-minute-schedule-1,53b741af9641ee28,4189b0da39faf79f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:36.581 WARN 21062 --- [ocp-async-7,882f03e740770e0b,a872ee15375b995a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:36.581 INFO 21062 --- [ocp-async-2,26c2fc74ff40d268,ceaddc1d381a9bb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:38.484 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:38.485 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:38.504 INFO 21062 --- [ocp-schedules-12,b9981e7e9f313452,9674ed2c243be3aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:40:38.505 INFO 21062 --- [ocp-schedules-20,3b8e3f60a8057c34,ba60ccd11db3421b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:40:38.538 INFO 21062 --- [ocp-async-3,41f7714b00b2b082,08e6b3116197eaa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:40:38.579 INFO 21062 --- [ocp-schedules-9,d8bd21a90731be55,ab0c935b0b5a9114] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:40:38.581 INFO 21062 --- [ocp-async-5,f0751681e80fd40f,27b8c4e7971c7919] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:38.581 WARN 21062 --- [ocp-async-10,120e8827cf9b4c94,a71d4aacc4ae8b1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:40:38.587527+08:00, lastProcessTime=2024-11-04T12:40:28.588193+08:00 2024-11-04 12:40:38.656 INFO 21062 --- [ocp-schedules-21,0662136f2b7511ec,7ba8a13ab1776482] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:40:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:40:38.587527+08:00 2024-11-04 12:40:38.983 INFO 21062 --- [prometheus-minute-schedule-1,44709b4bba60e02c,ef85192eedc0b303] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:40:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:40:40.581 WARN 21062 --- [ocp-async-9,33e4047dde7483bd,08ac8efaf2d2409a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:40.581 INFO 21062 --- [ocp-async-6,9ef59696b5451749,bc3c31c940a43316] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:41.106 INFO 21062 --- [ocp-schedules-2,a87edbe8f58143e5,97810ebb0c6a968f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:40:41.485 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:41.486 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:40:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:40:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 12:40:41.983 INFO 21062 --- [prometheus-minute-schedule-1,b82e39a4219177c7,8c12092a13b30530] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:42.581 WARN 21062 --- [ocp-async-2,b6e7ef1a9446193f,bf3e0c92d1eb3f12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:42.581 INFO 21062 --- [ocp-async-8,899eedf95578d7da,00c0479e8828ec61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:44.486 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:44.487 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:44.582 INFO 21062 --- [ocp-async-1,424bec9fc1723efd,6a0991137121bd2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:44.582 WARN 21062 --- [ocp-async-10,270af850fb22d751,84f8d219e86020e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b73afa9a2c566209,0bea1e17835b3403] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:46.582 WARN 21062 --- [ocp-async-5,93ce322bb0be1ad1,da34fda9286ef2e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:46.582 INFO 21062 --- [ocp-async-1,3fbc42cdedf73cb2,5b77d96a8dd6ae5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:47.487 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:47.489 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8aca85ead6865ad9,be63569c252bff1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:48.539 INFO 21062 --- [ocp-async-1,f5c5668d6fc102fb,0e6dbc36803a321d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:40:48.582 WARN 21062 --- [ocp-async-6,ccf54afad7cbe07a,eb2da75fa0368668] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:48.582 INFO 21062 --- [ocp-async-2,c89ebfd7048ee671,fafab482b94bb7c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:40:48.588357+08:00, lastProcessTime=2024-11-04T12:40:38.587527+08:00 2024-11-04 12:40:48.659 INFO 21062 --- [ocp-schedules-24,05c89310a63ec215,b3f0a726cb0ee4ec] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:40:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:40:48.588357+08:00 2024-11-04 12:40:48.789 INFO 21062 --- [ocp-schedules-13,497ebbd8bb574368,ba25f7e111a584d8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:40:48.795 INFO 21062 --- [ocp-schedules-13,497ebbd8bb574368,ba25f7e111a584d8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:40:48.795 INFO 21062 --- [ocp-schedules-13,497ebbd8bb574368,ba25f7e111a584d8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:40:50.489 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:50.490 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:50.582 INFO 21062 --- [ocp-async-2,4f6803cb8353b6ae,f79006ba290e5445] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:50.582 WARN 21062 --- [ocp-async-10,e260eeaef560def9,99f425cae66d2e40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:50.983 INFO 21062 --- [prometheus-minute-schedule-1,42495087f7194721,01eba0d57b4c4404] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:51.134 INFO 21062 --- [ocp-schedules-15,a1dc359493730f0c,8d96062ac48fa894] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:40:52.582 WARN 21062 --- [ocp-async-2,52d149607aaee2ca,222f49da5ea73a87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:52.583 INFO 21062 --- [ocp-async-9,48e836932ed7cf7f,9fbe611361cc9ed2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:53.490 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:53.491 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1662d002141c1e43,1cf30a5405e9bc56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:54.583 WARN 21062 --- [ocp-async-1,b69e1b287474e833,ec0356cec9717ffd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:54.583 INFO 21062 --- [ocp-async-6,c783467b24d4195f,3fb20dc4a0623fc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:56.491 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:56.492 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:56.583 WARN 21062 --- [ocp-async-3,2769a49cc0f27789,be551eb80b91aca3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:56.583 INFO 21062 --- [ocp-async-10,907c32a50af659d8,9f061978c72bfde9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:56.983 INFO 21062 --- [prometheus-minute-schedule-1,adcb64a180f09c8a,574634fd410067c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:40:58.499 INFO 21062 --- [ocp-schedules-24,a10d094f9cf2f376,3abb54e97f62314d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:40:58.500 INFO 21062 --- [ocp-schedules-11,0d5916782c60053e,4f3e2936f6f8eb48] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:40:58.502 INFO 21062 --- [ocp-schedules-16,f0b85613163763e3,39fbd3a909939a49] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:40:58.503 INFO 21062 --- [ocp-schedules-24,a10d094f9cf2f376,3abb54e97f62314d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:40:58.504 INFO 21062 --- [ocp-schedules-11,0d5916782c60053e,4f3e2936f6f8eb48] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:58.505 INFO 21062 --- [ocp-schedules-5,7574db1cdd953323,d243f647015bfc64] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:40:58.511 INFO 21062 --- [ocp-schedules-3,e8dada4dc28fded7,96bdab0a76444dbd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:40:58.511 INFO 21062 --- [ocp-schedules-20,107f398c7d9dc10c,9691ee56a440d6c9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:40:58.511 INFO 21062 --- [ocp-schedules-3,e8dada4dc28fded7,96bdab0a76444dbd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:40:58.511 INFO 21062 --- [ocp-schedules-20,107f398c7d9dc10c,9691ee56a440d6c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:40:58.511 INFO 21062 --- [ocp-schedules-7,f94e97e8275069c0,797e8e68a63f7d19] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:40:58.511 INFO 21062 --- [ocp-schedules-7,f94e97e8275069c0,797e8e68a63f7d19] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:40:58.539 INFO 21062 --- [ocp-async-5,58fa047900300b4d,4a8516c724488c8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:40:58.583 WARN 21062 --- [ocp-async-2,f2fd4160072e3b9d,40d94bd46bd13e68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:40:58.583 INFO 21062 --- [ocp-async-7,53990f544ee06765,fd64749ff740c168] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:40:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:40:58.588074+08:00, lastProcessTime=2024-11-04T12:40:48.588357+08:00 2024-11-04 12:40:58.605 INFO 21062 --- [ocp-schedules-4,8a69062a997bac3e,ec1df9e2c26b0fae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:40:58.663 INFO 21062 --- [ocp-schedules-13,4ba29359d3094779,a53e66ae20fd4ba1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:40:58.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:40:58.588074+08:00 2024-11-04 12:40:59.118 INFO 21062 --- [ocp-schedules-10,9370527f28efd281,e9560ad6807eb12e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:40:59.119 INFO 21062 --- [ocp-schedules-10,e0e1d35853d99441,9feac0a5e8d41eae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:40:59.122 INFO 21062 --- [ocp-schedules-10,e0e1d35853d99441,9feac0a5e8d41eae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:40:59.151 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:40:59.151 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:40:59.492 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:40:59.493 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:40:59.983 INFO 21062 --- [prometheus-minute-schedule-1,63635aa079e727b3,da47aba4acce635a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695260, interval=60 2024-11-04 12:41:00.583 INFO 21062 --- [ocp-async-1,7e7b714c22aa2469,70f649167a40ace9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:00.583 WARN 21062 --- [ocp-async-8,da77399706119d92,11d3dc2a0d360488] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:01.182 INFO 21062 --- [ocp-schedules-21,471be199502b037d,be9e69044dfe6c4c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:41:02.493 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:02.495 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:02.583 INFO 21062 --- [ocp-async-3,46bb067614b4c81e,7aef9338fdf07e49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:02.583 WARN 21062 --- [ocp-async-4,8b6353adb46e60ab,2455ecb68f15c7f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:02.983 INFO 21062 --- [prometheus-minute-schedule-1,51a6878aba2334e9,228efc6d66cbed0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:04.583 WARN 21062 --- [ocp-async-5,459404ebca70b9fe,bf36423c8fc69b5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:04.583 INFO 21062 --- [ocp-async-2,3468e9f32d67e663,2a1fa339683b1a5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:05.495 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:05.496 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:05.983 INFO 21062 --- [prometheus-minute-schedule-1,bc7a56f95cff5be5,afaa61033ed33405] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:06.584 WARN 21062 --- [ocp-async-7,c15f68ad23ffbffe,62fef1e23a168608] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:06.584 INFO 21062 --- [ocp-async-8,70fba3c843d98c1d,2f349845f8fce67c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:08.497 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:08.498 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:08.504 INFO 21062 --- [ocp-schedules-24,a10d094f9cf2f376,3abb54e97f62314d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:41:08.505 INFO 21062 --- [ocp-schedules-11,0d5916782c60053e,4f3e2936f6f8eb48] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:41:08.539 INFO 21062 --- [ocp-async-1,2873ba4e523412bb,97bf380685264581] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:41:08.584 WARN 21062 --- [ocp-async-4,e1dcc79614c5818f,862599f2b4d385bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:08.584 INFO 21062 --- [ocp-async-10,3f8dde97e646df8b,931f71eb746c161e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:41:08.588237+08:00, lastProcessTime=2024-11-04T12:40:58.588074+08:00 2024-11-04 12:41:08.666 INFO 21062 --- [ocp-schedules-27,eca3a23658d617d1,7a5f33c253483975] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:41:08.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:41:08.588237+08:00 2024-11-04 12:41:08.798 INFO 21062 --- [ocp-schedules-28,ad4598458b67b334,c3c0f7059f69e5d1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:41:08.803 INFO 21062 --- [ocp-schedules-28,ad4598458b67b334,c3c0f7059f69e5d1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:41:08.803 INFO 21062 --- [ocp-schedules-28,ad4598458b67b334,c3c0f7059f69e5d1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:41:08.983 INFO 21062 --- [prometheus-minute-schedule-1,9bc3e452060f8613,25f55b2c728a0160] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:41:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:41:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:41:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:41:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:41:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:41:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:41:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:41:10.584 WARN 21062 --- [ocp-async-9,e6a73b1a291d46b5,95bd6d808b1ea424] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:10.584 INFO 21062 --- [ocp-async-5,7c6739393974c4d4,b649241b92f4b8bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:11.062 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695200, nodeCount=46, affectRows=46 2024-11-04 12:41:11.063 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695200, nodeCount=74, affectRows=74 2024-11-04 12:41:11.063 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695200, nodeCount=46, validCount=46 2024-11-04 12:41:11.063 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695200, nodeCount=70, affectRows=70 2024-11-04 12:41:11.063 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695200, nodeCount=64, affectRows=64 2024-11-04 12:41:11.063 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695200, nodeCount=72, validCount=72 2024-11-04 12:41:11.064 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695200, nodeCount=69, validCount=69 2024-11-04 12:41:11.064 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695200, nodeCount=64, validCount=64 2024-11-04 12:41:11.064 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695200, nodeCount=68, affectRows=68 2024-11-04 12:41:11.064 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695200, nodeCount=68, validCount=68 2024-11-04 12:41:11.208 INFO 21062 --- [ocp-schedules-29,ae0f5818e279c5ed,23cd78d4530db1d6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:41:11.498 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:11.500 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9570ac676df0c0ed,0332d0ad08459e15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:12.584 WARN 21062 --- [ocp-async-7,50137b2aef894c12,5ddcd51a52724e4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:12.584 INFO 21062 --- [ocp-async-6,443bbbc974fa5836,57de7f48c1960087] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:13.984 INFO 21062 --- [prometheus-cache-monitor1,1b52c5c2d4bb4c46,d44ce25ebe11b8f2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:41:13.984 INFO 21062 --- [prometheus-cache-monitor1,1b52c5c2d4bb4c46,d44ce25ebe11b8f2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:41:14.084 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:37:14.059, end=2024-11-04T12:39:14.082} 2024-11-04 12:41:14.089 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:41:14.089 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:41:14.500 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:14.501 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:14.584 WARN 21062 --- [ocp-async-1,1369093fcd9d12ca,2c65d23d5b7bb3b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:14.584 INFO 21062 --- [ocp-async-4,8d274d12babf6da4,39e8e6c7e506f576] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3208de43076d42f8,58cec1316be13ba6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:16.585 WARN 21062 --- [ocp-async-10,512106f7315a905b,9dea5e75a2c226c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:16.585 INFO 21062 --- [ocp-async-9,f45700f06cb415b9,715cbb19986b896e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:17.501 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:17.503 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d51c202c3ef0f3ec,261107c3eb772cc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:18.539 INFO 21062 --- [ocp-async-9,88f6a5591bc54574,a2066c5af84191d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:41:18.585 WARN 21062 --- [ocp-async-7,e14c22ac5ff03db6,c10f24476519f5e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:18.585 INFO 21062 --- [ocp-async-8,68f5dfbbb47b3245,863b24a7dc8b0f97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:41:18.590353+08:00, lastProcessTime=2024-11-04T12:41:08.588237+08:00 2024-11-04 12:41:18.669 INFO 21062 --- [ocp-schedules-16,b82f0f6ea470c8f9,7b90856900af359c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:41:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:41:18.590353+08:00 2024-11-04 12:41:20.503 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:20.505 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:20.585 WARN 21062 --- [ocp-async-1,82e8c1e4f5b6c610,3bb6c035cd5eacdf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:20.585 INFO 21062 --- [ocp-async-3,435cb5183fac4894,e54e4fbe3973b666] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:20.983 INFO 21062 --- [prometheus-minute-schedule-1,44c516138a8e9a4c,2a9b3b07dde86064] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:21.255 INFO 21062 --- [ocp-schedules-16,e1e3c1bfa1fe7714,e89b77af779e6772] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:41:22.585 WARN 21062 --- [ocp-async-10,873efaf1326f9a43,eff607fcda0da068] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:22.585 INFO 21062 --- [ocp-async-2,5de08695ab283cd4,c5050123eb32cec2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:23.506 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:23.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c060cdec4fa39921,05f71c9ed0b111c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:24.585 WARN 21062 --- [ocp-async-7,6e6eb1d037dea635,5a31ec572d7b4c49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:24.585 INFO 21062 --- [ocp-async-9,6ca26ef045adeafa,da98fa73604ebee4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:26.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:26.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:26.586 WARN 21062 --- [ocp-async-8,4038f8368ef316c8,2fa4e9c248a2b484] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:26.586 INFO 21062 --- [ocp-async-1,46a33eb1c0f0b494,25cc86ff68d337ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:26.983 INFO 21062 --- [prometheus-minute-schedule-1,be7f22d8e0a7d47a,4b05eba3910429e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:26.984 INFO 21062 --- [prometheus-hour-schedule-1,560aab55da594a8b,9c1c7d43f47318af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:41:28.498 INFO 21062 --- [ocp-schedules-4,3b7e9ca01c23ad44,918cf47a2404e67d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:41:28.499 INFO 21062 --- [ocp-schedules-20,8a7b3ce88aa15d80,1dc4f5f48c670776] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:41:28.499 INFO 21062 --- [ocp-schedules-29,621f37d8475b2ab5,96d11e3debb87061] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:41:28.500 INFO 21062 --- [ocp-schedules-6,a2244747e288ae64,6c156c6b83eb82e4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:41:28.502 INFO 21062 --- [ocp-schedules-3,0682beb2420e4b54,d7a19340ed0f9ef0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:41:28.503 INFO 21062 --- [ocp-schedules-4,3b7e9ca01c23ad44,918cf47a2404e67d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:41:28.503 INFO 21062 --- [ocp-schedules-6,a2244747e288ae64,6c156c6b83eb82e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:28.503 INFO 21062 --- [ocp-schedules-29,621f37d8475b2ab5,96d11e3debb87061] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:41:28.504 INFO 21062 --- [ocp-schedules-2,c293a77756795952,37fca9805760cea2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:41:28.505 INFO 21062 --- [ocp-schedules-20,8a7b3ce88aa15d80,1dc4f5f48c670776] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:41:28.512 INFO 21062 --- [ocp-schedules-9,239d0e322ebc06a4,7513f538e36975ee] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:41:28.512 INFO 21062 --- [ocp-schedules-22,b0b57cb6bb2b56c5,c85e1634779b603a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:41:28.512 INFO 21062 --- [ocp-schedules-9,239d0e322ebc06a4,7513f538e36975ee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:41:28.512 INFO 21062 --- [ocp-schedules-22,b0b57cb6bb2b56c5,c85e1634779b603a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:41:28.513 INFO 21062 --- [ocp-schedules-24,6f409698c8bed8b1,0b7ebdac4074c4ab] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:41:28.513 INFO 21062 --- [ocp-schedules-24,6f409698c8bed8b1,0b7ebdac4074c4ab] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:41:28.539 INFO 21062 --- [ocp-async-10,5309cdd6b56de891,1687c511b90ad162] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:41:28.581 INFO 21062 --- [ocp-schedules-28,3dd00eb4309936d0,f0fe80ef733d53b4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:41:28.586 WARN 21062 --- [ocp-async-2,3555298434b88b90,ad7e89f805072e25] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:28.586 INFO 21062 --- [ocp-async-7,18fbeba1f0814b55,bc00b0a2a4418e15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:41:28.587568+08:00, lastProcessTime=2024-11-04T12:41:18.590353+08:00 2024-11-04 12:41:28.603 INFO 21062 --- [ocp-schedules-15,8a97f1f18b19e0ba,59d861917ebcacd8] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:41:28.603 INFO 21062 --- [ocp-schedules-15,8a97f1f18b19e0ba,59d861917ebcacd8] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:41:28.607 INFO 21062 --- [ocp-schedules-12,5103da0603edea84,dcf787068e44d2e6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:41:28.635 INFO 21062 --- [ocp-schedules-20,cd012eddbd54c8ed,c662c3c96e6a3eda] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:41:28.640 INFO 21062 --- [ocp-schedules-20,cd012eddbd54c8ed,c662c3c96e6a3eda] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:41:28.640 INFO 21062 --- [ocp-schedules-20,cd012eddbd54c8ed,c662c3c96e6a3eda] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:41:28.671 INFO 21062 --- [ocp-schedules-9,8830028d7a531f58,e8a9daedadb2a6fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:41:28.699 INFO 21062 --- [ocp-schedules-22,dfef281f6d13250d,af3a250339a745e3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:41:28.699 INFO 21062 --- [ocp-schedules-22,dfef281f6d13250d,af3a250339a745e3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:41:28.700 INFO 21062 --- [ocp-schedules-22,dfef281f6d13250d,af3a250339a745e3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:41:28.700 INFO 21062 --- [ocp-schedules-22,dfef281f6d13250d,af3a250339a745e3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:41:28.700 INFO 21062 --- [ocp-schedules-22,dfef281f6d13250d,af3a250339a745e3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:41:28.700 INFO 21062 --- [ocp-schedules-22,dfef281f6d13250d,af3a250339a745e3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:41:28.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:41:28.587568+08:00 2024-11-04 12:41:28.765 INFO 21062 --- [ocp-schedules-4,0046ff27377d9fef,93e77035e5adabdf] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:41:28.765 INFO 21062 --- [ocp-schedules-4,0046ff27377d9fef,93e77035e5adabdf] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:41:28.792 INFO 21062 --- [ocp-schedules-4,0046ff27377d9fef,93e77035e5adabdf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:41:28.792 INFO 21062 --- [ocp-schedules-4,0046ff27377d9fef,93e77035e5adabdf] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:41:28.793 INFO 21062 --- [ocp-schedules-13,a278582f47325328,7f82e168d89d24e4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:41:28.793 INFO 21062 --- [ocp-schedules-13,a278582f47325328,7f82e168d89d24e4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:41:28.794 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:41:28.794 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:41:28.803 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:41:28.804 INFO 21062 --- [ocp-schedules-13,a278582f47325328,7f82e168d89d24e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:41:28.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:41:28.805 INFO 21062 --- [ocp-schedules-17,4a532e9dda199826,688cced6875e4e27] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:41:28.805 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:41:28.805 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:41:28.806 WARN 21062 --- [ocp-schedules-13,a278582f47325328,7f82e168d89d24e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:41:28.806 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:41:28.806 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:41:28.806 INFO 21062 --- [ocp-schedules-13,a278582f47325328,7f82e168d89d24e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:41:28.807 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:41:28.808 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:41:28.808 WARN 21062 --- [ocp-schedules-13,a278582f47325328,7f82e168d89d24e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:41:28.810 INFO 21062 --- [ocp-schedules-17,4a532e9dda199826,688cced6875e4e27] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:41:28.810 INFO 21062 --- [ocp-schedules-17,4a532e9dda199826,688cced6875e4e27] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:41:28.816 ERROR 21062 --- [ocp-schedules-13,a278582f47325328,7f82e168d89d24e4] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:41:28.821 INFO 21062 --- [ocp-schedules-26,3d75ef5273f9b7dd,ee6cbb493e4cb955] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:41:28.886 INFO 21062 --- [ocp-schedules-18,357abb3845dcbb16,b32e2778f5711628] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:41:28.886+08:00, clusters=[1] 2024-11-04 12:41:28.887 INFO 21062 --- [ocp-schedules-18,357abb3845dcbb16,b32e2778f5711628] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:41:28.964 INFO 21062 --- [ocp-schedules-25,c22327a8a6b79fa3,81804eb0bdc703a4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:41:28.964+08:00, clusters=[1] 2024-11-04 12:41:28.967 INFO 21062 --- [ocp-schedules-25,c22327a8a6b79fa3,81804eb0bdc703a4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:41:29.152 INFO 21062 --- [ocp-schedules-10,e0e1d35853d99441,9feac0a5e8d41eae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:41:29.152 INFO 21062 --- [ocp-schedules-10,a51e6ebde34fc84d,4d09a9b084991615] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:41:29.155 INFO 21062 --- [ocp-schedules-10,a51e6ebde34fc84d,4d09a9b084991615] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:29.185 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:41:29.185 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:41:29.326 INFO 21062 --- [ocp-schedules-27,6266d65fe1751e0e,83fe7bf543219697] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:41:29.329 INFO 21062 --- [ocp-schedules-27,6266d65fe1751e0e,83fe7bf543219697] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:29.413 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:41:29.415 ERROR 21062 --- [ocp-schedules-27,6266d65fe1751e0e,83fe7bf543219697] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:41:29.415 INFO 21062 --- [ocp-schedules-27,6266d65fe1751e0e,83fe7bf543219697] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:41:29.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:29.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:29.983 INFO 21062 --- [prometheus-minute-schedule-1,465c7e37a5a837d7,90198f67ca7beef9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:41:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:41:30.586 WARN 21062 --- [ocp-async-8,bd5796d521cd16ea,bd5bb7e9299c224b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:30.586 INFO 21062 --- [ocp-async-9,89e14f785a9a41a8,3c23e9eb5aa69780] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:31.280 INFO 21062 --- [ocp-schedules-21,dbdfdc7b3bc785eb,5651b213b763de0a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:41:32.510 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:32.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:32.586 WARN 21062 --- [ocp-async-1,7656a1016ca0d0ae,f951d37816d45126] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:32.587 INFO 21062 --- [ocp-async-3,a0f0f089e4d35ebd,9b5a113c4f4073f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:32.983 INFO 21062 --- [prometheus-minute-schedule-1,dd5fd6e091703f65,6960e58bef831282] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:34.262 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:41:34.262 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:41:34.269 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:41:34.269 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:41:34.269 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:41:34.270 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:41:34.275 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:41:34.276 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:41:34.276 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:41:34.276 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:41:34.313 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:41:34.322 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:41:34.322 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:41:34.322 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:41:34.322 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:41:34.322 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:41:34.330 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:41:34.587 WARN 21062 --- [ocp-async-2,7c33238299f5392d,9386d084f13e9627] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:34.587 INFO 21062 --- [ocp-async-5,c6dfcbd9a7171cea,04f62c16453ae75d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:34.847 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:41:35.448 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.172615054S 2024-11-04 12:41:35.448 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:41:35.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:35.513 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:35.538 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:41:35.538 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:41:35.568 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:41:35.629 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:41:35.629 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:41:35.629 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:41:35.629 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:41:35.629 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:41:35.630 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:41:35.641 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:41:35.822 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] o.s.boot.SpringApplication : Started application in 1.552 seconds (JVM running for 709.501) 2024-11-04 12:41:35.822 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.552767632S 2024-11-04 12:41:35.983 INFO 21062 --- [prometheus-minute-schedule-1,6403871016983695,1c9f39a7c255c948] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:36.313 INFO 21062 --- [ocp-schedules-1,1533c08135e17770,d89040d5180879ac] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:41:36.587 WARN 21062 --- [ocp-async-7,a0163d3bb7b8e573,f6c201ad3a14b522] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:36.587 INFO 21062 --- [ocp-async-8,a418269569462c4c,e11e6f2c8b739bb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:38.504 INFO 21062 --- [ocp-schedules-6,a2244747e288ae64,6c156c6b83eb82e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:41:38.504 INFO 21062 --- [ocp-schedules-29,621f37d8475b2ab5,96d11e3debb87061] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:41:38.514 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:38.515 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:38.540 INFO 21062 --- [ocp-async-9,644c976b1ffc8144,65a8dec67d618b61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:41:38.580 INFO 21062 --- [ocp-schedules-21,07e86e0277464538,faa5209c465fc92b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:41:38.583 INFO 21062 --- [ocp-schedules-21,07e86e0277464538,faa5209c465fc92b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:38.584 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:41:38.587 WARN 21062 --- [ocp-async-10,2c6332c89a7f4a82,7cd5fe4f656837dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:38.587 INFO 21062 --- [ocp-async-3,a9c74c3a3a549afc,a2dfd9a2212eba4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:41:38.589078+08:00, lastProcessTime=2024-11-04T12:41:28.587568+08:00 2024-11-04 12:41:38.674 INFO 21062 --- [ocp-schedules-13,f18a91f5d2660d7b,14df411be577dc0c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:41:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:41:38.589078+08:00 2024-11-04 12:41:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ed44f958c76ea799,af525146c61d62e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:40.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:41:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:41:40.587 WARN 21062 --- [ocp-async-6,5b58267211b86ec0,eea3af031c9680db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:40.587 INFO 21062 --- [ocp-async-6,6c62c49c83612920,3b8a787c6a074168] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:41.323 INFO 21062 --- [ocp-schedules-27,a13a0b8ed29754d7,cc31a2b9a9b80e5a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:41:41.515 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:41.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bbae751d11a309c9,6fb28b259f8ebdea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:42.587 WARN 21062 --- [ocp-async-4,d71f0dbd5f216fc5,c5682c7c7c302811] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:42.587 INFO 21062 --- [ocp-async-8,443c4b089ae98002,cec9776837a5785d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:44.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:44.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:44.587 WARN 21062 --- [ocp-async-8,d7e8d2949a88cfae,56e721ff9762aba4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:44.588 INFO 21062 --- [ocp-async-1,80dc4656f34dbaeb,c3bd8ec22200c61e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:44.983 INFO 21062 --- [prometheus-minute-schedule-1,0c4fa687798a5aed,86ea3dc9fb84fae6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:46.588 WARN 21062 --- [ocp-async-3,1c3cc48e2b8ffe46,4b58da4e94f5f24d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:46.588 INFO 21062 --- [ocp-async-1,fb1d23a959dd3e55,10159f8da91b9053] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:47.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:47.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:47.983 INFO 21062 --- [prometheus-minute-schedule-1,75995eb608197ed0,15a129d9c17bfae0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:48.540 INFO 21062 --- [ocp-async-1,f337d06cc91c159d,a5486d153a2523c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:41:48.584 INFO 21062 --- [ocp-schedules-21,07e86e0277464538,faa5209c465fc92b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:41:48.588 INFO 21062 --- [ocp-async-6,17acfa4bd1cf287b,121c79a4e4f0770d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:48.588 WARN 21062 --- [ocp-async-4,387f379212435f75,950ec6152d354089] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:41:48.589694+08:00, lastProcessTime=2024-11-04T12:41:38.589078+08:00 2024-11-04 12:41:48.676 INFO 21062 --- [ocp-schedules-17,d2951098974784a9,79f1a9df35984b2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:41:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:41:48.589694+08:00 2024-11-04 12:41:48.812 INFO 21062 --- [ocp-schedules-12,4d899cbc60e92ea4,bbebb4bbca08af5f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:41:48.818 INFO 21062 --- [ocp-schedules-12,4d899cbc60e92ea4,bbebb4bbca08af5f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:41:48.818 INFO 21062 --- [ocp-schedules-12,4d899cbc60e92ea4,bbebb4bbca08af5f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:41:50.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:50.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:50.588 INFO 21062 --- [ocp-async-6,e0deb5f20edc265c,2c7760d853c5bbec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:50.588 WARN 21062 --- [ocp-async-10,81c505b9c7e6135f,5733cf32bd8200b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:50.983 INFO 21062 --- [prometheus-minute-schedule-1,997dc991f4cb3c0b,bdc5e708e9257ca1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:51.350 INFO 21062 --- [ocp-schedules-30,80f8235cfa71e102,5d24db8852070e26] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:41:52.588 WARN 21062 --- [ocp-async-7,b2ec3f1b9467e56b,4a488c906f7a17d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:52.588 INFO 21062 --- [ocp-async-6,e055df65b546cbfe,c2c9c46524e9bd1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:53.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:53.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e40b6dc83d3265c3,7c66daf981375dc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:54.588 INFO 21062 --- [ocp-async-1,7d0b23b503d612d4,1fe6e325822daa4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:54.588 WARN 21062 --- [ocp-async-4,0b1d7765b696167c,416f385f6bdfb338] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:56.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:56.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:56.589 WARN 21062 --- [ocp-async-8,3bac725f2bcda3c7,89f7e3d1bf3cd172] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:56.589 INFO 21062 --- [ocp-async-10,91922398a8902dc4,c58bd9e6b31070d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:56.983 INFO 21062 --- [prometheus-minute-schedule-1,37ed3dbffd8247ea,9c895ecd601bf886] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:41:58.500 INFO 21062 --- [ocp-schedules-12,f23611f18f21c7fc,1ef73f2a83de54de] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:41:58.500 INFO 21062 --- [ocp-schedules-18,588a5496d1d5d0a3,bee0d884bd56f49b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:41:58.503 INFO 21062 --- [ocp-schedules-17,5cccb648f7d54791,c21abbb047f0a0f6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:41:58.504 INFO 21062 --- [ocp-schedules-12,f23611f18f21c7fc,1ef73f2a83de54de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:58.505 INFO 21062 --- [ocp-schedules-18,588a5496d1d5d0a3,bee0d884bd56f49b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:41:58.505 INFO 21062 --- [ocp-schedules-22,9e8e351ff0dd5bae,be32237d43496886] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:41:58.514 INFO 21062 --- [ocp-schedules-19,22f5213a08f2e1e6,991f29244fc6f904] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:41:58.514 INFO 21062 --- [ocp-schedules-16,b77aa86e0c9ee91b,cc1043bc321129df] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:41:58.514 INFO 21062 --- [ocp-schedules-8,7d98e7f8e4af6d0e,d7b1e9bbafae4c32] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:41:58.514 INFO 21062 --- [ocp-schedules-16,b77aa86e0c9ee91b,cc1043bc321129df] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:41:58.514 INFO 21062 --- [ocp-schedules-19,22f5213a08f2e1e6,991f29244fc6f904] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:41:58.515 INFO 21062 --- [ocp-schedules-8,7d98e7f8e4af6d0e,d7b1e9bbafae4c32] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:41:58.540 INFO 21062 --- [ocp-async-3,6cb9dd95e2fb4e1b,d5ab712a89e087f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:41:58.589 INFO 21062 --- [ocp-async-7,d1828aa93e87b4fd,de8c5fe40ec07066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:41:58.589 WARN 21062 --- [ocp-async-5,bfd48b4f5ef2d671,6ce9f90950d59f85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:41:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:41:58.588592+08:00, lastProcessTime=2024-11-04T12:41:48.589694+08:00 2024-11-04 12:41:58.609 INFO 21062 --- [ocp-schedules-26,8384db19006ee4b9,df244d43af3c467f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:41:58.679 INFO 21062 --- [ocp-schedules-5,3bf7166142a4cee6,55260f306abf167d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:41:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:41:58.588592+08:00 2024-11-04 12:41:59.186 INFO 21062 --- [ocp-schedules-10,a51e6ebde34fc84d,4d09a9b084991615] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:41:59.186 INFO 21062 --- [ocp-schedules-10,cb0f67edbbc8742f,5c4d618f3951feae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:41:59.189 INFO 21062 --- [ocp-schedules-10,cb0f67edbbc8742f,5c4d618f3951feae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:41:59.219 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:41:59.220 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:41:59.523 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:41:59.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:41:59.983 INFO 21062 --- [prometheus-minute-schedule-1,b91ca2bd27124d92,c9b7a10b6c9b09bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695320, interval=60 2024-11-04 12:42:00.589 INFO 21062 --- [ocp-async-4,5b8e62c7530d708b,fb9a4d87f09b17db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:00.589 WARN 21062 --- [ocp-async-9,c783a60f47305679,a0c1ee2dc205dac7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:01.398 INFO 21062 --- [ocp-schedules-28,277d0dac2fcd359c,891a1e57ae7cde1a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:42:02.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:02.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:02.589 INFO 21062 --- [ocp-async-8,d7de194b40eae1ba,a49da810b1fe4385] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:02.589 WARN 21062 --- [ocp-async-2,c1bb655b666a9fde,0b23dc26ce906f62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:02.983 INFO 21062 --- [prometheus-minute-schedule-1,cc86138b38262115,f3a0c32ec1b8db74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:04.589 WARN 21062 --- [ocp-async-3,08abb1a9069f134e,3ca2cdf63bdbe477] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:04.589 INFO 21062 --- [ocp-async-5,8726ee0cb8c02d5d,7f433b7c04af2b65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:05.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:05.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:05.983 INFO 21062 --- [prometheus-minute-schedule-1,567def7d5b4eb212,b694299e7bed113a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:06.589 WARN 21062 --- [ocp-async-9,c3e4b9d7813cb4d9,e2a1b6d01a3e2199] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:06.589 INFO 21062 --- [ocp-async-7,c8b7ee7c911232ec,1fde45cda520f605] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:08.505 INFO 21062 --- [ocp-schedules-12,f23611f18f21c7fc,1ef73f2a83de54de] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:42:08.508 INFO 21062 --- [ocp-schedules-18,588a5496d1d5d0a3,bee0d884bd56f49b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:42:08.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:08.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:08.541 INFO 21062 --- [ocp-async-4,6832cbd2d882150e,f09490a425f60a9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:42:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:42:08.587791+08:00, lastProcessTime=2024-11-04T12:41:58.588592+08:00 2024-11-04 12:42:08.590 WARN 21062 --- [ocp-async-10,e68c85c1801ca5c4,d8a7398bc451c39d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:08.590 INFO 21062 --- [ocp-async-2,f392774f5eb59cf7,17b02e5afdc6699f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:08.682 INFO 21062 --- [ocp-schedules-1,e77e1e87bdb0b18d,0a07607581d06c83] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:42:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:42:08.587791+08:00 2024-11-04 12:42:08.821 INFO 21062 --- [ocp-schedules-4,83a389f8215ecb51,469c24d3db565bce] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:42:08.827 INFO 21062 --- [ocp-schedules-4,83a389f8215ecb51,469c24d3db565bce] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:42:08.827 INFO 21062 --- [ocp-schedules-4,83a389f8215ecb51,469c24d3db565bce] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:42:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a444a0da91f4975b,c0ed8898cff5bd85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:42:10.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:42:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:42:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:42:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:42:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:42:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:42:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:42:10.590 WARN 21062 --- [ocp-async-3,f86893ab26e2b316,d70e74b3737c83e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:10.590 INFO 21062 --- [ocp-async-6,831f6f08a19fff6a,9d5ffb92c1422966] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:11.066 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=9, affectRows=9 2024-11-04 12:42:11.066 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=1, affectRows=1 2024-11-04 12:42:11.066 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=15, affectRows=15 2024-11-04 12:42:11.066 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=31, affectRows=31 2024-11-04 12:42:11.066 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=1, validCount=1 2024-11-04 12:42:11.066 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=24, affectRows=24 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=15, affectRows=15 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=9, validCount=9 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=15, validCount=15 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=14, validCount=14 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=24, validCount=24 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=31, validCount=31 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=31, affectRows=31 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=41, affectRows=41 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=41, affectRows=41 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=50, affectRows=50 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=31, validCount=31 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695260, nodeCount=64, affectRows=64 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=40, validCount=40 2024-11-04 12:42:11.067 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=41, validCount=41 2024-11-04 12:42:11.068 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=50, validCount=50 2024-11-04 12:42:11.068 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695260, nodeCount=63, validCount=63 2024-11-04 12:42:11.424 INFO 21062 --- [ocp-schedules-21,8a2f4793039cda98,1a869cc3b1be69d2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:42:11.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:11.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e9b26123c02080f1,ac2475d087b6b72c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:12.590 WARN 21062 --- [ocp-async-9,5c83b1c335873bdf,715d5f1d902b8472] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:12.590 INFO 21062 --- [ocp-async-1,bb782c4d4574cd83,ade2339df448beaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:13.984 INFO 21062 --- [prometheus-cache-monitor1,df2c66961457e477,b30fe531b07effa0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:42:13.985 INFO 21062 --- [prometheus-cache-monitor1,df2c66961457e477,b30fe531b07effa0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:42:14.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:14.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:14.590 WARN 21062 --- [ocp-async-4,b7d7d28d7f0bde97,06e3901a9c3f016c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:14.590 INFO 21062 --- [ocp-async-4,5f74a0a39326aa24,57cd3b46d1e17308] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f4d4d403cbf19f75,bb80881b581e583e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:16.590 WARN 21062 --- [ocp-async-2,dc72db0db8d1503f,5e3f78e510b51efd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:16.590 INFO 21062 --- [ocp-async-3,230fde7248337858,5dc70fbbdb009b31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:17.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:17.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4f9ead0300877a0a,6a95e202aab8c6f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:18.541 INFO 21062 --- [ocp-async-3,9719f7d0321a4f17,f2c18991e0005540] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:42:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:42:18.589080+08:00, lastProcessTime=2024-11-04T12:42:08.587791+08:00 2024-11-04 12:42:18.591 WARN 21062 --- [ocp-async-9,507273ff0f43433d,20b4b315ac7fbcbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:18.591 INFO 21062 --- [ocp-async-9,bf003ac3f427c4bc,d2712c64af353dfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:18.684 INFO 21062 --- [ocp-schedules-23,085e5d7d4729627a,45bbe1e5f6dceecd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:42:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:42:18.589080+08:00 2024-11-04 12:42:20.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:20.535 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:20.591 WARN 21062 --- [ocp-async-10,175203e7a8f441d8,c84a5d4a4c0a9ebd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:20.591 INFO 21062 --- [ocp-async-8,29ec3e093e334f14,5860bd8d6e195be0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:20.983 INFO 21062 --- [prometheus-minute-schedule-1,492f222d33fa2a39,1dbaf7cf37098164] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:21.467 INFO 21062 --- [ocp-schedules-25,bb28cb35ecf456d6,76e6aaa1d83381a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:42:22.591 WARN 21062 --- [ocp-async-2,a1012db5577de608,30e1d54b7a97bbd1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:22.591 INFO 21062 --- [ocp-async-5,5f27805a2a0d7373,3285f4c029b1a0ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:23.535 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:23.537 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:23.983 INFO 21062 --- [prometheus-minute-schedule-1,5195aa42bd630b7e,184dcc7b89adce63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:24.591 WARN 21062 --- [ocp-async-6,7eb8f10d3e286af1,4f6c7dc3af256690] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:24.591 INFO 21062 --- [ocp-async-7,4819cd76600ef629,141947d2acb7281a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:26.537 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:26.538 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:26.591 WARN 21062 --- [ocp-async-9,fe4f1a6d68c46533,8b124fac1fe4ce49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:26.591 INFO 21062 --- [ocp-async-10,e24cc8776ded2775,ab037182731954dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:26.983 INFO 21062 --- [prometheus-minute-schedule-1,a99ded7aa560ba9d,55719b7d85127928] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ccf7175e222db837,fe4853e7e1c0bc61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:42:28.498 INFO 21062 --- [ocp-schedules-17,c35b8e9274575f8d,ca5184ed437d8e94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:42:28.499 INFO 21062 --- [ocp-schedules-8,22304f4b671e8058,543bbc4c66b6cac7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:42:28.500 INFO 21062 --- [ocp-schedules-22,c406a11932a244a1,1e2ea00fd716c906] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:42:28.500 INFO 21062 --- [ocp-schedules-4,57950e202d402009,9948f62e8fab535a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:42:28.502 INFO 21062 --- [ocp-schedules-27,6567e2467d2e6bf2,2d3325bbd6967ebe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:42:28.503 INFO 21062 --- [ocp-schedules-17,c35b8e9274575f8d,ca5184ed437d8e94] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:42:28.504 INFO 21062 --- [ocp-schedules-8,22304f4b671e8058,543bbc4c66b6cac7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:42:28.505 INFO 21062 --- [ocp-schedules-28,0c1f6abc3f710be0,ab3fbfaeab420149] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:42:28.505 INFO 21062 --- [ocp-schedules-4,57950e202d402009,9948f62e8fab535a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:28.505 INFO 21062 --- [ocp-schedules-24,730a64a414f1a73d,2817dc3bcb127582] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 12:42:28.506 INFO 21062 --- [ocp-schedules-22,c406a11932a244a1,1e2ea00fd716c906] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:42:28.508 INFO 21062 --- [ocp-schedules-24,730a64a414f1a73d,2817dc3bcb127582] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:28.513 INFO 21062 --- [ocp-schedules-26,8ada651929e44cb7,92b7b13a7a591461] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:42:28.513 INFO 21062 --- [ocp-schedules-1,bcf2998937425bbc,acb73693fee6d6ea] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:42:28.513 INFO 21062 --- [ocp-schedules-12,598ccd14ecbf2576,d9ec117c44d0993b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:42:28.513 INFO 21062 --- [ocp-schedules-1,bcf2998937425bbc,acb73693fee6d6ea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:42:28.513 INFO 21062 --- [ocp-schedules-26,8ada651929e44cb7,92b7b13a7a591461] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:42:28.513 INFO 21062 --- [ocp-schedules-12,598ccd14ecbf2576,d9ec117c44d0993b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:42:28.525 INFO 21062 --- [ocp-schedules-3,8061ac1902f4bafc,6c3073e9a4644ef5] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 12:42:28.525 INFO 21062 --- [ocp-schedules-3,8061ac1902f4bafc,6c3073e9a4644ef5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 12:42:28.541 INFO 21062 --- [ocp-async-2,124afe29c6bc8464,054450cf4b4b2d99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:42:28.541 INFO 21062 --- [ocp-schedules-13,a7bb8a7b38d03cac,339eef1eea0c3d3a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 12:42:28.545 INFO 21062 --- [ocp-schedules-13,a7bb8a7b38d03cac,339eef1eea0c3d3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:28.553 INFO 21062 --- [ocp-schedules-5,da0646c08fa09317,693f84e7bde7c5d0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:42:28.554 INFO 21062 --- [ocp-schedules-5,da0646c08fa09317,693f84e7bde7c5d0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:42:28.561 INFO 21062 --- [ocp-schedules-18,d5768fcebe11ff65,94936564fb4bc947] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 12:42:28.561 INFO 21062 --- [ocp-schedules-18,d5768fcebe11ff65,94936564fb4bc947] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 12:42:28.561 INFO 21062 --- [ocp-schedules-18,d5768fcebe11ff65,94936564fb4bc947] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=322 2024-11-04 12:42:28.566 INFO 21062 --- [ocp-schedules-6,e61ece2be52dea61,0bf230f9cbc04dd6] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:42:28.567 INFO 21062 --- [ocp-schedules-6,e61ece2be52dea61,0bf230f9cbc04dd6] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:42:28.573 INFO 21062 --- [pool-12-thread-12,f7eccff47933a8a9,d4198100097ad900] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 12:42:28.573 INFO 21062 --- [pool-12-thread-11,f7eccff47933a8a9,51e1b487c1883ff0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 12:42:28.573 INFO 21062 --- [pool-12-thread-9,f7eccff47933a8a9,17409104a4957c03] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 12:42:28.574 INFO 21062 --- [pool-12-thread-10,f7eccff47933a8a9,e185d32a66c40192] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 12:42:28.580 INFO 21062 --- [ForkJoinPool-11-worker-11,f7eccff47933a8a9,840cf515bbe6288f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 12:42:28.580 INFO 21062 --- [ForkJoinPool-11-worker-11,f7eccff47933a8a9,840cf515bbe6288f] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 12:42:28.580 INFO 21062 --- [ForkJoinPool-11-worker-11,f7eccff47933a8a9,840cf515bbe6288f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 12:42:28.580 INFO 21062 --- [pool-12-thread-11,f7eccff47933a8a9,51e1b487c1883ff0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 12:42:28.581 INFO 21062 --- [pool-12-thread-10,f7eccff47933a8a9,e185d32a66c40192] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 12:42:28.581 INFO 21062 --- [pool-12-thread-10,f7eccff47933a8a9,e185d32a66c40192] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 12:42:28.583 INFO 21062 --- [ocp-schedules-28,ab6fb4e0cf039a69,182e0436fdc54ee4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:42:28.588 INFO 21062 --- [ocp-schedules-1,7ddce340e1272e59,75c7f2f1bcc68eb3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 12:42:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:42:28.588345+08:00, lastProcessTime=2024-11-04T12:42:18.589080+08:00 2024-11-04 12:42:28.591 WARN 21062 --- [ocp-async-5,f4dae9af3072ef1f,67abe5aaac6d9f4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:28.591 INFO 21062 --- [ocp-async-6,fa1ba961fcc8f71d,af127b9c7144d9e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:28.600 INFO 21062 --- [ocp-schedules-1,7ddce340e1272e59,75c7f2f1bcc68eb3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 12:42:28.602 INFO 21062 --- [ocp-schedules-18,d5768fcebe11ff65,94936564fb4bc947] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=322 2024-11-04 12:42:28.604 INFO 21062 --- [pool-12-thread-9,f7eccff47933a8a9,17409104a4957c03] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 12:42:28.607 INFO 21062 --- [ocp-schedules-7,7f0ef47d4fbc9f18,fb16516ef829ed76] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:42:28.607 INFO 21062 --- [ocp-schedules-7,7f0ef47d4fbc9f18,fb16516ef829ed76] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:42:28.609 INFO 21062 --- [ForkJoinPool-11-worker-11,f7eccff47933a8a9,d335e53dd3cdcbf7] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 12:42:28.609 INFO 21062 --- [pool-12-thread-12,f7eccff47933a8a9,d4198100097ad900] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 12:42:28.611 INFO 21062 --- [ocp-schedules-23,4375f7d0ede8f85e,b737e798dd24de7f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:42:28.612 INFO 21062 --- [pool-12-thread-9,f7eccff47933a8a9,17409104a4957c03] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 12:42:28.617 INFO 21062 --- [pool-12-thread-9,f7eccff47933a8a9,17409104a4957c03] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 12:42:28.628 INFO 21062 --- [ocp-schedules-14,be384aa78634c3d9,92c2723e423b1ca6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 12:42:28.628 INFO 21062 --- [ocp-schedules-14,be384aa78634c3d9,92c2723e423b1ca6] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 12:42:28.640 INFO 21062 --- [ocp-schedules-29,ee8c363f46f871bc,6ab67d0b7a621a29] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:42:28.645 INFO 21062 --- [ocp-schedules-29,ee8c363f46f871bc,6ab67d0b7a621a29] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:42:28.645 INFO 21062 --- [ocp-schedules-29,ee8c363f46f871bc,6ab67d0b7a621a29] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:42:28.654 INFO 21062 --- [pool-21-thread-1,be384aa78634c3d9,26c5587cf647257f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 12:42:28.656 ERROR 21062 --- [ocp-schedules-13,a7bb8a7b38d03cac,339eef1eea0c3d3a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:42:28.656 INFO 21062 --- [ocp-schedules-13,a7bb8a7b38d03cac,339eef1eea0c3d3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 12:42:28.680 WARN 21062 --- [ocp-schedules-14,be384aa78634c3d9,92c2723e423b1ca6] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:42:28.680 INFO 21062 --- [ocp-schedules-14,be384aa78634c3d9,92c2723e423b1ca6] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 12:42:28.687 INFO 21062 --- [ocp-schedules-19,c62a6b0bcbfd3cb2,d1e46c05e490f9df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:42:28.701 INFO 21062 --- [ocp-schedules-20,06972d70bc78e06f,99d7c53e75387c96] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:42:28.701 INFO 21062 --- [ocp-schedules-20,06972d70bc78e06f,99d7c53e75387c96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:42:28.706 INFO 21062 --- [ocp-schedules-20,06972d70bc78e06f,99d7c53e75387c96] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:42:28.706 INFO 21062 --- [ocp-schedules-20,06972d70bc78e06f,99d7c53e75387c96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:42:28.712 INFO 21062 --- [ocp-schedules-20,06972d70bc78e06f,99d7c53e75387c96] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:42:28.712 INFO 21062 --- [ocp-schedules-20,06972d70bc78e06f,99d7c53e75387c96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:42:28.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:42:28.588345+08:00 2024-11-04 12:42:28.792 INFO 21062 --- [ocp-schedules-15,bfadff20474a8985,f3da86902e0d41ff] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:42:28.792 INFO 21062 --- [ocp-schedules-15,bfadff20474a8985,f3da86902e0d41ff] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:42:28.797 INFO 21062 --- [ocp-schedules-9,f570eb9eae1dc54a,7bbb7bac09e3ad4e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:42:28.800 INFO 21062 --- [ocp-schedules-9,f570eb9eae1dc54a,7bbb7bac09e3ad4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:28.818 INFO 21062 --- [ocp-schedules-15,bfadff20474a8985,f3da86902e0d41ff] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:42:28.818 INFO 21062 --- [ocp-schedules-15,bfadff20474a8985,f3da86902e0d41ff] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:42:28.823 INFO 21062 --- [ocp-schedules-25,13d0743f6415aee6,0ec0c9b82755a1c6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:42:28.823 INFO 21062 --- [ocp-schedules-25,13d0743f6415aee6,0ec0c9b82755a1c6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:42:28.824 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:42:28.824 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:42:28.829 INFO 21062 --- [ocp-schedules-21,b104bc47f6e8832d,1249bcb8bed6530e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:42:28.833 INFO 21062 --- [ocp-schedules-25,13d0743f6415aee6,0ec0c9b82755a1c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:42:28.833 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:42:28.834 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:42:28.835 WARN 21062 --- [ocp-schedules-25,13d0743f6415aee6,0ec0c9b82755a1c6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:42:28.835 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:42:28.835 INFO 21062 --- [ocp-schedules-21,b104bc47f6e8832d,1249bcb8bed6530e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:42:28.835 INFO 21062 --- [ocp-schedules-21,b104bc47f6e8832d,1249bcb8bed6530e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:42:28.835 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:42:28.835 INFO 21062 --- [ocp-schedules-25,13d0743f6415aee6,0ec0c9b82755a1c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:42:28.835 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:42:28.836 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:42:28.836 WARN 21062 --- [ocp-schedules-25,13d0743f6415aee6,0ec0c9b82755a1c6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:42:28.837 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:42:28.837 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:42:28.842 ERROR 21062 --- [ocp-schedules-25,13d0743f6415aee6,0ec0c9b82755a1c6] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2024-11-04 12:42:28.844 INFO 21062 --- [ocp-schedules-30,eadd04a07069aa01,4ebf3267e5b28128] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:42:28.854 ERROR 21062 --- [ocp-schedules-9,f570eb9eae1dc54a,7bbb7bac09e3ad4e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:42:28.854 INFO 21062 --- [ocp-schedules-9,f570eb9eae1dc54a,7bbb7bac09e3ad4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:42:28.911 INFO 21062 --- [ocp-schedules-22,e66dbdca60d47406,cc8f5f76a24a570b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:42:28.911+08:00, clusters=[1] 2024-11-04 12:42:28.912 INFO 21062 --- [ocp-schedules-22,e66dbdca60d47406,cc8f5f76a24a570b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:42:28.992 INFO 21062 --- [ocp-schedules-6,23abc452931ecede,5d3a5746335da648] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:42:28.992+08:00, clusters=[1] 2024-11-04 12:42:28.995 INFO 21062 --- [ocp-schedules-6,23abc452931ecede,5d3a5746335da648] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:42:29.220 INFO 21062 --- [ocp-schedules-10,cb0f67edbbc8742f,5c4d618f3951feae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:42:29.220 INFO 21062 --- [ocp-schedules-10,4d4ee04f447a30e8,badfcd07affceae4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:42:29.224 INFO 21062 --- [ocp-schedules-10,4d4ee04f447a30e8,badfcd07affceae4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:29.253 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:42:29.253 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:42:29.416 INFO 21062 --- [ocp-schedules-12,28fde51fd4674c53,ce6c4a454bc87b1b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:42:29.419 INFO 21062 --- [ocp-schedules-12,28fde51fd4674c53,ce6c4a454bc87b1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:29.501 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:42:29.502 ERROR 21062 --- [ocp-schedules-12,28fde51fd4674c53,ce6c4a454bc87b1b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:42:29.502 INFO 21062 --- [ocp-schedules-12,28fde51fd4674c53,ce6c4a454bc87b1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:42:29.538 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:29.538 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:29.983 INFO 21062 --- [prometheus-minute-schedule-1,a8779b0bc7151e69,9985e12d1076be91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:42:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:42:30.592 WARN 21062 --- [ocp-async-9,2e99495313c86d0a,1f9d0f16e3f37353] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:30.592 INFO 21062 --- [ocp-async-7,73fd730ea95c2513,48fe3bde227e46a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:31.495 INFO 21062 --- [ocp-schedules-17,9d443e6c1349d2aa,ace2c0bf3f1bf5b9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:42:32.539 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:32.541 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:32.592 WARN 21062 --- [ocp-async-8,65a1194d4186b413,8149775b9e28152f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:32.592 INFO 21062 --- [ocp-async-7,78e5434dd2490492,e1611b37dd007d29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:32.983 INFO 21062 --- [prometheus-minute-schedule-1,185bd76582b61179,0f9be0e6f2b19d19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:34.592 WARN 21062 --- [ocp-async-3,f8ff5c6efa8fc57f,102f6bff79034077] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:34.592 INFO 21062 --- [ocp-async-7,1fa48e07104cd02b,17fb57e998fbf1d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:35.541 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:35.542 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:35.983 INFO 21062 --- [prometheus-minute-schedule-1,88dd5c0c8a580b3e,7409d70520f8d953] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:36.592 WARN 21062 --- [ocp-async-6,f60d3a64ab5cb6f6,9284bd76315d4108] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:36.592 INFO 21062 --- [ocp-async-7,e66faa547ebb95e6,79689f20722d9e14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:38.505 INFO 21062 --- [ocp-schedules-8,22304f4b671e8058,543bbc4c66b6cac7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:42:38.505 INFO 21062 --- [ocp-schedules-4,57950e202d402009,9948f62e8fab535a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:42:38.509 INFO 21062 --- [ocp-schedules-24,730a64a414f1a73d,2817dc3bcb127582] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 12:42:38.541 INFO 21062 --- [ocp-async-8,0b640e13b8271252,8c213cf439a2e870] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:42:38.543 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:38.543 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:42:38.587655+08:00, lastProcessTime=2024-11-04T12:42:28.588345+08:00 2024-11-04 12:42:38.592 WARN 21062 --- [ocp-async-10,74d393010d339986,fae7c70187ec75e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:38.592 INFO 21062 --- [ocp-async-2,d9060e05d5ed2a22,b35c41cb3e31f52a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:38.689 INFO 21062 --- [ocp-schedules-27,c414964861679d9b,a14e4e1eeac202cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:42:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:42:38.587655+08:00 2024-11-04 12:42:38.983 INFO 21062 --- [prometheus-minute-schedule-1,9f30a203bd45f0ce,2a38491ab62c6869] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:42:40.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:42:40.592 INFO 21062 --- [ocp-async-1,0fec1e483d9538e4,6677b5524eff4c71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:40.593 WARN 21062 --- [ocp-async-5,09f5e0e8b15e8c50,05ef6eaf6d257910] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:41.538 INFO 21062 --- [ocp-schedules-11,d06b6d2f9782f707,7a977f6d743c4ac8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:42:41.544 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:41.545 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:42:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:42:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:42:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:42:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:42:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:42:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 12:42:41.983 INFO 21062 --- [prometheus-minute-schedule-1,50f67078eabd5be5,2b4b039fd6c8d117] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:42.593 WARN 21062 --- [ocp-async-4,5fb94af48ed0f205,d1ca58291b505f84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:42.593 INFO 21062 --- [ocp-async-9,cd328dc634f61564,7bd9c60f1162c781] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:44.545 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:44.546 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:44.593 WARN 21062 --- [ocp-async-8,a9a58c386600ff42,127ab5e203aa4f3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:44.593 INFO 21062 --- [ocp-async-8,c583659bf2dfee53,5dc511da0badd804] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5a8d34884e336f0a,62e1aeda3cf41629] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:46.593 WARN 21062 --- [ocp-async-2,dbd88aaf07ef046f,94a7b6f608fe19f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:46.593 INFO 21062 --- [ocp-async-2,7f997d4488b86c08,06d7b5fc70ae08dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:47.546 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:47.547 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:47.983 INFO 21062 --- [prometheus-minute-schedule-1,1e5001bd49861e89,1aeb843a277a964e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:48.542 INFO 21062 --- [ocp-async-1,fdac12f650142909,bc93ec5b4a39e9db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:42:48.585 INFO 21062 --- [ocp-schedules-18,a88ed7ec0150909d,dd1dcc700f8e3c58] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:42:48.588 INFO 21062 --- [ocp-schedules-18,a88ed7ec0150909d,dd1dcc700f8e3c58] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:48.589 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:42:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:42:48.589947+08:00, lastProcessTime=2024-11-04T12:42:38.587655+08:00 2024-11-04 12:42:48.593 WARN 21062 --- [ocp-async-4,7bbd782563a98f6f,64ab96a521004fed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:48.593 INFO 21062 --- [ocp-async-7,7d670ee88da32558,243d373e67bf8034] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:48.691 INFO 21062 --- [ocp-schedules-12,5868eca82ad0f35e,f817d6cd17853120] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:42:48.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:42:48.589947+08:00 2024-11-04 12:42:48.837 INFO 21062 --- [ocp-schedules-25,2f8a30337a7e62f7,7c9871c0518534ef] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:42:48.842 INFO 21062 --- [ocp-schedules-25,2f8a30337a7e62f7,7c9871c0518534ef] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:42:48.842 INFO 21062 --- [ocp-schedules-25,2f8a30337a7e62f7,7c9871c0518534ef] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:42:50.547 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:50.549 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:50.593 WARN 21062 --- [ocp-async-3,27606bfdc89e8681,eb793b5a54159472] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:50.594 INFO 21062 --- [ocp-async-10,e5cddc04fc5c091a,3d49b58b0c8d360c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:50.983 INFO 21062 --- [prometheus-minute-schedule-1,07aac516ba20eeaf,92beb1cba6c3845b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:51.563 INFO 21062 --- [ocp-schedules-28,065b8b8b4253e52b,3a0845dff0c6649f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:42:52.594 WARN 21062 --- [ocp-async-5,4a1d84860462ec4a,000f30fc8b12aca8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:52.594 INFO 21062 --- [ocp-async-6,d4230d0936500951,f575e632a7360c96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:53.549 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:53.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:53.983 INFO 21062 --- [prometheus-minute-schedule-1,7eb627f92c5e8486,ddb03383d3253581] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:54.594 WARN 21062 --- [ocp-async-4,82f220e149c06027,202547ec568aea4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:54.594 INFO 21062 --- [ocp-async-1,ae11bc37b14a624a,07222d476fb22688] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:56.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:56.551 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:56.594 WARN 21062 --- [ocp-async-3,29883f2bcd848d68,2773588cca3d125a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:56.594 INFO 21062 --- [ocp-async-7,3a69f45051233d27,8da2f288e038bb6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:56.983 INFO 21062 --- [prometheus-minute-schedule-1,a0244b41f57e818f,6cd80bf8d0e4976f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:42:58.499 INFO 21062 --- [ocp-schedules-5,de3af0f3bcd706e5,6221d78cae5246fd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:42:58.500 INFO 21062 --- [ocp-schedules-26,cb3ee57e271f76ea,989a48e17280dd60] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:42:58.502 INFO 21062 --- [ocp-schedules-16,fa802e059f90ef03,27367a1886a45d49] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:42:58.505 INFO 21062 --- [ocp-schedules-5,de3af0f3bcd706e5,6221d78cae5246fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:58.505 INFO 21062 --- [ocp-schedules-26,cb3ee57e271f76ea,989a48e17280dd60] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:42:58.506 INFO 21062 --- [ocp-schedules-27,ba8ace93636475ad,4a91522e4eb4a320] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:42:58.512 INFO 21062 --- [ocp-schedules-22,2cc38e04aa787fbd,735b4668d042cb80] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:42:58.512 INFO 21062 --- [ocp-schedules-22,2cc38e04aa787fbd,735b4668d042cb80] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:42:58.512 INFO 21062 --- [ocp-schedules-4,3d8bb0d3de177e42,51fa5c0bc3dc5bc5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:42:58.512 INFO 21062 --- [ocp-schedules-4,3d8bb0d3de177e42,51fa5c0bc3dc5bc5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:42:58.512 INFO 21062 --- [ocp-schedules-30,9744632d510f192e,222dfe68f92e3c32] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:42:58.512 INFO 21062 --- [ocp-schedules-30,9744632d510f192e,222dfe68f92e3c32] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:42:58.542 INFO 21062 --- [ocp-async-10,00258a887abb0390,6191d665c472182a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:42:58.589 INFO 21062 --- [ocp-schedules-18,a88ed7ec0150909d,dd1dcc700f8e3c58] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:42:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:42:58.589020+08:00, lastProcessTime=2024-11-04T12:42:48.589947+08:00 2024-11-04 12:42:58.594 WARN 21062 --- [ocp-async-5,c92bb2a973abfce4,1d08d27489b8838f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:42:58.594 INFO 21062 --- [ocp-async-2,f7a057be7c8b7aaf,22fbbfc97bed1a31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:42:58.613 INFO 21062 --- [ocp-schedules-7,35768da4baf3143a,a2f94aeae69c362f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:42:58.694 INFO 21062 --- [ocp-schedules-9,f906202b96853838,88071edd038c147b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:42:58.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:42:58.589020+08:00 2024-11-04 12:42:59.254 INFO 21062 --- [ocp-schedules-10,4d4ee04f447a30e8,badfcd07affceae4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:42:59.254 INFO 21062 --- [ocp-schedules-10,eed2319c5c87200f,b5be3805b2693193] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:42:59.257 INFO 21062 --- [ocp-schedules-10,eed2319c5c87200f,b5be3805b2693193] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:42:59.289 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:42:59.289 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:42:59.551 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:42:59.552 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:42:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e85465390a1b8d56,9aaa05470ae933aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695380, interval=60 2024-11-04 12:43:00.594 WARN 21062 --- [ocp-async-4,645661db4f0add8d,56ada258588da510] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:00.594 INFO 21062 --- [ocp-async-9,fdb61b9440aaf4e3,3782d9e43e625a82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:01.607 INFO 21062 --- [ocp-schedules-18,a608e80ed26811cd,53e4c9ac695b0801] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:43:02.553 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:02.554 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:02.594 WARN 21062 --- [ocp-async-8,932260d49b56aff5,b0816f38bb9cc889] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:02.594 INFO 21062 --- [ocp-async-3,fd9de564f2b1cf3d,9203f81df57b362a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:02.983 INFO 21062 --- [prometheus-minute-schedule-1,51b9ff2f0c797421,293474a5921640b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:04.594 WARN 21062 --- [ocp-async-5,99d04125743eba0b,f91599cc9ce2ed50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:04.594 INFO 21062 --- [ocp-async-10,38a2b0f7c84b1b92,b7c58d90748ec21a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:05.554 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:05.555 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:05.983 INFO 21062 --- [prometheus-minute-schedule-1,87bbf2d91770fe38,5d52dba7ea391515] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:06.595 INFO 21062 --- [ocp-async-2,330d4d299dba0f45,25823cd4336444bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:06.595 WARN 21062 --- [ocp-async-4,e148675800242024,b6c4603e9394e032] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:08.506 INFO 21062 --- [ocp-schedules-26,cb3ee57e271f76ea,989a48e17280dd60] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:43:08.506 INFO 21062 --- [ocp-schedules-5,de3af0f3bcd706e5,6221d78cae5246fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:43:08.542 INFO 21062 --- [ocp-async-2,9397a8bcd1b93931,1c9d58a1042cec0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:43:08.556 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:08.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:43:08.588735+08:00, lastProcessTime=2024-11-04T12:42:58.589020+08:00 2024-11-04 12:43:08.595 INFO 21062 --- [ocp-async-8,a8ff2925c82d29f3,baab23f7e190bc83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:08.595 WARN 21062 --- [ocp-async-7,da78bb4792faea92,aefc0b7378a0ce3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:08.696 INFO 21062 --- [ocp-schedules-15,a762ed3608656b5b,46b1c60c603f509b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:43:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:43:08.588735+08:00 2024-11-04 12:43:08.845 INFO 21062 --- [ocp-schedules-21,d92a340668c3b3f0,3a3151fb961d0ef4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:43:08.851 INFO 21062 --- [ocp-schedules-21,d92a340668c3b3f0,3a3151fb961d0ef4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:43:08.851 INFO 21062 --- [ocp-schedules-21,d92a340668c3b3f0,3a3151fb961d0ef4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:43:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d038c65c7a10c60d,84632863f1aec91a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:43:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:43:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:43:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:43:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:43:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:43:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:43:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:43:10.595 WARN 21062 --- [ocp-async-6,4a1a26c1fd223493,eccd80e9e755b6d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:10.595 INFO 21062 --- [ocp-async-5,c89123c5c6898a0c,100836d86c48a613] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:11.072 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=3, affectRows=3 2024-11-04 12:43:11.072 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=9, affectRows=9 2024-11-04 12:43:11.072 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=11, affectRows=11 2024-11-04 12:43:11.072 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=3, validCount=3 2024-11-04 12:43:11.073 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=9, validCount=9 2024-11-04 12:43:11.073 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=11, validCount=11 2024-11-04 12:43:11.073 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=46, affectRows=46 2024-11-04 12:43:11.073 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=37, affectRows=37 2024-11-04 12:43:11.074 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=37, validCount=37 2024-11-04 12:43:11.074 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=45, validCount=45 2024-11-04 12:43:11.074 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=27, affectRows=27 2024-11-04 12:43:11.074 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=39, affectRows=39 2024-11-04 12:43:11.074 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=26, affectRows=26 2024-11-04 12:43:11.074 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=27, validCount=27 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=39, validCount=39 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=26, validCount=26 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=55, affectRows=55 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=6, affectRows=6 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=6, validCount=6 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=44, affectRows=44 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=53, validCount=53 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=44, validCount=44 2024-11-04 12:43:11.075 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695320, nodeCount=19, affectRows=19 2024-11-04 12:43:11.076 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695320, nodeCount=19, validCount=19 2024-11-04 12:43:11.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:11.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:11.633 INFO 21062 --- [ocp-schedules-12,c10b04f773b6ee22,25f826dcdefb15c9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:43:11.983 INFO 21062 --- [prometheus-minute-schedule-1,493870bf49f3a7c3,df5a5a15bd226b9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:12.595 WARN 21062 --- [ocp-async-1,af81547c79653a08,281733a0763d2b9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:12.595 INFO 21062 --- [ocp-async-4,6ed035bd3aa8cae9,1206f5a3bdc069a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:13.859 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T12:37:13.834, end=2024-11-04T12:42:13.852} 2024-11-04 12:43:13.859 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T12:37:13.834, end=2024-11-04T12:42:13.851} 2024-11-04 12:43:13.869 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 12:43:13.869 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:43:13.869 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 12:43:13.869 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:43:13.984 INFO 21062 --- [prometheus-cache-monitor1,7deba57dc12c3457,92d1b63ff6b990c5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:43:13.984 INFO 21062 --- [prometheus-cache-monitor1,7deba57dc12c3457,92d1b63ff6b990c5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:43:14.100 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:39:14.082, end=2024-11-04T12:41:14.097} 2024-11-04 12:43:14.103 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:43:14.103 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:43:14.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:14.560 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:14.595 WARN 21062 --- [ocp-async-7,45fbff41b10f48a3,bc631ef068630176] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:14.595 INFO 21062 --- [ocp-async-2,fec1909a044ba495,adad51ab1789818b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:14.983 INFO 21062 --- [prometheus-minute-schedule-1,26893f7550fa7bca,07d70a11b73c71e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:16.595 WARN 21062 --- [ocp-async-6,b5c69081e9bf49c8,7802d69d88984e27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:16.595 INFO 21062 --- [ocp-async-8,d4610db742151511,75637d3a735906ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:17.560 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:17.561 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:17.983 INFO 21062 --- [prometheus-minute-schedule-1,fbf13ad060507541,40fbe08e04aa8d75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:18.542 INFO 21062 --- [ocp-async-8,784dc0a1f711c7a1,b9a5de60c37acc21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:43:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:43:18.587987+08:00, lastProcessTime=2024-11-04T12:43:08.588735+08:00 2024-11-04 12:43:18.596 WARN 21062 --- [ocp-async-9,ca55e3d056d9de7a,c4697198a9362dd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:18.596 INFO 21062 --- [ocp-async-1,27ae3ee087a1e3ce,c2efc729176d2fce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:18.699 INFO 21062 --- [ocp-schedules-19,9f93112550cfe3ad,0df930943646a1f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:43:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:43:18.587987+08:00 2024-11-04 12:43:20.561 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:20.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:20.596 WARN 21062 --- [ocp-async-7,8b64332e4a3589a1,3e32967bcc626b58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:20.596 INFO 21062 --- [ocp-async-3,a116e2f01d7d51fe,a5ff1781259854d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5601a8aa70b7fe76,9cdfa833ed0ef942] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:21.677 INFO 21062 --- [ocp-schedules-8,860df077c106acc2,9ba8df01e043d528] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:43:22.596 WARN 21062 --- [ocp-async-6,c0c35708fe23cc84,50fde7649ab7de78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:22.596 INFO 21062 --- [ocp-async-10,73e0175ca0edcf74,e77aeba7d1d88746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:23.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:23.565 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:23.983 INFO 21062 --- [prometheus-minute-schedule-1,e5c533043ee845d7,437639638670cf1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:24.596 WARN 21062 --- [ocp-async-5,4d2a13c042617966,9ee2d47f165b30fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:24.596 INFO 21062 --- [ocp-async-9,6fab82aa16b27cea,f26dc0d51fe6f727] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:26.565 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:26.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:26.596 WARN 21062 --- [ocp-async-7,bf470b5daeffbe83,c41ff8feeb20a90c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:26.596 INFO 21062 --- [ocp-async-1,e11262cbbb46b1a6,439f012785cb21e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:26.983 INFO 21062 --- [prometheus-minute-schedule-1,797837d447d76e9f,5b20df970ce4283f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:26.984 INFO 21062 --- [prometheus-hour-schedule-1,bee4aa459b00ef02,2063159f14ec046b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:43:28.498 INFO 21062 --- [ocp-schedules-13,9ebf033feb5a272a,18254bbbafe4eb2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:43:28.500 INFO 21062 --- [ocp-schedules-19,2fd01d7012c4ae89,b266926488707a41] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:43:28.500 INFO 21062 --- [ocp-schedules-9,25d899a2c07ce622,09d6c9912169aa73] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:43:28.500 INFO 21062 --- [ocp-schedules-14,10a3db06728591dd,90877f6ce14cdef7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:43:28.502 INFO 21062 --- [ocp-schedules-13,9ebf033feb5a272a,18254bbbafe4eb2d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:43:28.502 INFO 21062 --- [ocp-schedules-22,c4851c7d3e750807,09b6a7fa0d1cd8ea] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:43:28.503 INFO 21062 --- [ocp-schedules-19,2fd01d7012c4ae89,b266926488707a41] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:43:28.504 INFO 21062 --- [ocp-schedules-14,10a3db06728591dd,90877f6ce14cdef7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:28.505 INFO 21062 --- [ocp-schedules-27,3aeb5762c2087248,ed16f83b295a67cc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:43:28.505 INFO 21062 --- [ocp-schedules-9,25d899a2c07ce622,09d6c9912169aa73] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:43:28.510 INFO 21062 --- [ocp-schedules-30,45ef6e0c6813f712,f18a6283af23a5be] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:43:28.510 INFO 21062 --- [ocp-schedules-24,df257eaecbef882f,5a2a473a184eb884] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:43:28.510 INFO 21062 --- [ocp-schedules-24,df257eaecbef882f,5a2a473a184eb884] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:43:28.510 INFO 21062 --- [ocp-schedules-30,45ef6e0c6813f712,f18a6283af23a5be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:43:28.511 INFO 21062 --- [ocp-schedules-4,d52113424ca6e344,84fd9271e4f3cc3a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:43:28.511 INFO 21062 --- [ocp-schedules-4,d52113424ca6e344,84fd9271e4f3cc3a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:43:28.542 INFO 21062 --- [ocp-async-6,ede5706c11994a1e,6fd52943072e6a7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:43:28.544 INFO 21062 --- [ocp-schedules-12,ff0a12e7470b3d61,d928dc346706cbdd] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 12:43:28.544 INFO 21062 --- [ocp-schedules-12,ff0a12e7470b3d61,d928dc346706cbdd] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 12:43:28.586 INFO 21062 --- [ocp-schedules-16,36f3075260bf1b1c,051225c9f50a54f8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:43:28.586 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:43:28.586 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:43:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:43:28.588679+08:00, lastProcessTime=2024-11-04T12:43:18.587987+08:00 2024-11-04 12:43:28.593 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:43:28.593 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:43:28.593 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:43:28.597 INFO 21062 --- [ocp-async-10,872bdd926a4bc581,f44c8af322489e51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:28.597 WARN 21062 --- [ocp-async-5,e90f7900ee0a74c8,97ffdbc34c5dd466] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:28.604 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:43:28.604 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:43:28.604 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:43:28.608 INFO 21062 --- [ocp-schedules-3,b1429e3ec8ce2150,fffee1b2a0b15195] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:43:28.611 INFO 21062 --- [ocp-schedules-9,79fcc1f5c9ceca68,624c69462cb2063f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:43:28.611 INFO 21062 --- [ocp-schedules-9,79fcc1f5c9ceca68,624c69462cb2063f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:43:28.615 INFO 21062 --- [ocp-schedules-22,e33c76f0d8dfac07,9aed87ff8246d5d9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:43:28.645 INFO 21062 --- [ocp-schedules-30,f58939c72489d701,154f63898f916253] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:43:28.651 INFO 21062 --- [ocp-schedules-30,f58939c72489d701,154f63898f916253] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:43:28.651 INFO 21062 --- [ocp-schedules-30,f58939c72489d701,154f63898f916253] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:43:28.702 INFO 21062 --- [ocp-schedules-4,708f2395d88c1875,fed859617bc59c85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:43:28.712 INFO 21062 --- [ocp-schedules-23,4c0d5418142a904b,cd4941b520d33f6a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:43:28.712 INFO 21062 --- [ocp-schedules-23,4c0d5418142a904b,cd4941b520d33f6a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:43:28.713 INFO 21062 --- [ocp-schedules-23,4c0d5418142a904b,cd4941b520d33f6a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:43:28.713 INFO 21062 --- [ocp-schedules-23,4c0d5418142a904b,cd4941b520d33f6a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:43:28.714 INFO 21062 --- [ocp-schedules-23,4c0d5418142a904b,cd4941b520d33f6a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:43:28.714 INFO 21062 --- [ocp-schedules-23,4c0d5418142a904b,cd4941b520d33f6a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:43:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:43:28.588679+08:00 2024-11-04 12:43:28.818 INFO 21062 --- [ocp-schedules-18,046d922c21971adb,4808c2c9423e33a5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:43:28.818 INFO 21062 --- [ocp-schedules-18,046d922c21971adb,4808c2c9423e33a5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:43:28.845 INFO 21062 --- [ocp-schedules-18,046d922c21971adb,4808c2c9423e33a5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:43:28.845 INFO 21062 --- [ocp-schedules-18,046d922c21971adb,4808c2c9423e33a5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:43:28.852 INFO 21062 --- [ocp-schedules-8,d6e999513f4c5004,4889275d616a9261] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:43:28.852 INFO 21062 --- [ocp-schedules-8,d6e999513f4c5004,4889275d616a9261] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:43:28.852 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:43:28.852 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:43:28.853 INFO 21062 --- [ocp-schedules-12,7524854098312910,1862afd2e3e1907f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:43:28.858 INFO 21062 --- [ocp-schedules-12,7524854098312910,1862afd2e3e1907f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:43:28.859 INFO 21062 --- [ocp-schedules-12,7524854098312910,1862afd2e3e1907f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:43:28.862 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:43:28.863 INFO 21062 --- [ocp-schedules-8,d6e999513f4c5004,4889275d616a9261] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:43:28.863 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:43:28.864 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:43:28.865 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:43:28.865 WARN 21062 --- [ocp-schedules-8,d6e999513f4c5004,4889275d616a9261] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:43:28.865 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:43:28.865 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:43:28.865 INFO 21062 --- [ocp-schedules-8,d6e999513f4c5004,4889275d616a9261] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:43:28.867 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:43:28.867 WARN 21062 --- [ocp-schedules-8,d6e999513f4c5004,4889275d616a9261] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:43:28.867 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:43:28.869 INFO 21062 --- [ocp-schedules-25,65b667fc6e40e617,4d6a09440a0b8472] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:43:28.877 ERROR 21062 --- [ocp-schedules-8,d6e999513f4c5004,4889275d616a9261] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:43:28.938 INFO 21062 --- [ocp-schedules-28,abf9a43721537c45,80d184733c6b18de] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:43:28.938+08:00, clusters=[1] 2024-11-04 12:43:28.939 INFO 21062 --- [ocp-schedules-28,abf9a43721537c45,80d184733c6b18de] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:43:29.021 INFO 21062 --- [ocp-schedules-2,edf90011091a2140,4e274945373c0ff5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:43:29.021+08:00, clusters=[1] 2024-11-04 12:43:29.025 INFO 21062 --- [ocp-schedules-2,edf90011091a2140,4e274945373c0ff5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:43:29.289 INFO 21062 --- [ocp-schedules-10,eed2319c5c87200f,b5be3805b2693193] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:43:29.290 INFO 21062 --- [ocp-schedules-10,ab0ea7d25ffed03a,4607ab42c3455989] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:43:29.293 INFO 21062 --- [ocp-schedules-10,ab0ea7d25ffed03a,4607ab42c3455989] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:29.321 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:43:29.321 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:43:29.503 INFO 21062 --- [ocp-schedules-15,539b5fc54c6fa6d8,73610593a66702bb] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:43:29.506 INFO 21062 --- [ocp-schedules-15,539b5fc54c6fa6d8,73610593a66702bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:29.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:29.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:29.594 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:43:29.599 ERROR 21062 --- [ocp-schedules-15,539b5fc54c6fa6d8,73610593a66702bb] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:43:29.599 INFO 21062 --- [ocp-schedules-15,539b5fc54c6fa6d8,73610593a66702bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:43:29.983 INFO 21062 --- [prometheus-minute-schedule-1,37ed673179e6d573,02493542ae0ff192] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:43:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:43:30.597 INFO 21062 --- [ocp-async-9,7504915fed7f0a93,84fea8e24d10c6bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:30.597 WARN 21062 --- [ocp-async-7,773b22c1bc895bea,c41ee5a9ef211c99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:31.701 INFO 21062 --- [ocp-schedules-22,d65652614810ced6,ec18e568387638fe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:43:32.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:32.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:32.597 INFO 21062 --- [ocp-async-9,fd404a698dfc7c74,755bd8506ef53bb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:32.597 WARN 21062 --- [ocp-async-1,8942247040f66ea3,218621b77d52037a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:32.983 INFO 21062 --- [prometheus-minute-schedule-1,827fca854c8778ab,9d4f9b48a6b68184] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:34.597 WARN 21062 --- [ocp-async-8,d9f35deb3e2c0984,50f78a1d004294ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:34.597 INFO 21062 --- [ocp-async-9,2f7a8200203ffd07,e362dfa3f941d6f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:35.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:35.569 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:35.983 INFO 21062 --- [prometheus-minute-schedule-1,8b51d3ab62939ad1,693b49d8e55152e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:36.597 WARN 21062 --- [ocp-async-10,272dc12367e90928,f6ec2dd3962e124a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:36.597 INFO 21062 --- [ocp-async-9,3b07c805c11d4076,47741a2ad790d5c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:38.504 INFO 21062 --- [ocp-schedules-19,2fd01d7012c4ae89,b266926488707a41] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:43:38.504 INFO 21062 --- [ocp-schedules-14,10a3db06728591dd,90877f6ce14cdef7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:43:38.542 INFO 21062 --- [ocp-async-3,0df8db1e0944444d,3b4097fd57d14569] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:43:38.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:38.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:43:38.588226+08:00, lastProcessTime=2024-11-04T12:43:28.588679+08:00 2024-11-04 12:43:38.598 WARN 21062 --- [ocp-async-6,1eddcdcaadc5ee51,4d3bee556b45b3bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:38.598 INFO 21062 --- [ocp-async-1,0dc478fc1d1a6f61,1b092b918df1c5bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:38.704 INFO 21062 --- [ocp-schedules-9,abb49ee74bb257c4,d1ec5a6607f222a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:43:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:43:38.588226+08:00 2024-11-04 12:43:38.983 INFO 21062 --- [prometheus-minute-schedule-1,f2b954bd83135350,e866e3d6fb7f19a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:40.002 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:43:40.002 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:43:40.598 WARN 21062 --- [ocp-async-5,b1f3dac565b9d19a,f8815c7408ce3f90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:40.598 INFO 21062 --- [ocp-async-4,182f79412644ef64,0f61bc3b2831ee4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:41.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:41.572 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:41.751 INFO 21062 --- [ocp-schedules-2,2c13021b1fb7df2d,d8267205ab1c092a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:43:41.983 INFO 21062 --- [prometheus-minute-schedule-1,829408b779de7542,1058fb292cb13877] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:42.598 INFO 21062 --- [ocp-async-2,ad9c2dafbdec803b,eb9c03bfc1c95fb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:42.598 WARN 21062 --- [ocp-async-9,28eb8688289e3212,a6d93c827769fbeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:44.572 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:44.573 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:44.598 WARN 21062 --- [ocp-async-3,8bddb1c40652b9e6,3de9a70d521d098b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:44.598 INFO 21062 --- [ocp-async-6,86680cd01b8eda5d,877679fe20904117] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6377f42fddd60fd9,c4bde6fb6cacd4a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:46.598 WARN 21062 --- [ocp-async-5,8daa1c051bf764a0,c4730981766c1781] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:46.598 INFO 21062 --- [ocp-async-1,82de73457e6b26e4,cb0da92bc63e759c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:47.573 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:47.574 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b0f16aa4439cbcb5,c9fc3eb0136c985a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:48.543 INFO 21062 --- [ocp-async-4,bbfe158d4e7bce38,63fb860e3f77871f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:43:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:43:48.589937+08:00, lastProcessTime=2024-11-04T12:43:38.588226+08:00 2024-11-04 12:43:48.598 INFO 21062 --- [ocp-async-9,eb4e5a2ee3f0ae26,2a5eb16c540ef649] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:48.599 WARN 21062 --- [ocp-async-7,8e352fd659b7ab4f,be6b3c3d0fba84c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:48.707 INFO 21062 --- [ocp-schedules-25,2365aa7cce3c1a4e,2fe19e04e4dc8a23] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:43:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:43:48.589937+08:00 2024-11-04 12:43:48.860 INFO 21062 --- [ocp-schedules-29,a5076a58675005be,6a73f359602f14b9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:43:48.866 INFO 21062 --- [ocp-schedules-29,a5076a58675005be,6a73f359602f14b9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:43:48.866 INFO 21062 --- [ocp-schedules-29,a5076a58675005be,6a73f359602f14b9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:43:50.574 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:50.575 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:50.599 INFO 21062 --- [ocp-async-3,1c4a81bb7322097f,421f1d5eccb08528] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:50.599 WARN 21062 --- [ocp-async-8,434385ffb5d9ffe1,f317e83a02f58f61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ebd420118008c643,8044840dcc3f6bbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:51.777 INFO 21062 --- [ocp-schedules-13,edd0c79f284b075d,172ac9850761ec99] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:43:52.599 WARN 21062 --- [ocp-async-5,291abfcd9c6a64ac,23f05d715917f494] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:52.599 INFO 21062 --- [ocp-async-10,e2f44fa23583452b,ec04a801c7bb986f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:53.575 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:53.576 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:53.983 INFO 21062 --- [prometheus-minute-schedule-1,7f34e14c3ef68c2b,e1bc9dabf19b2e0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:54.599 WARN 21062 --- [ocp-async-4,181e5bd524ff7936,54f31176da72a186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:54.599 INFO 21062 --- [ocp-async-7,62ebb43e47055ea5,a7b274d6cf19587b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:56.576 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:56.577 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:56.599 INFO 21062 --- [ocp-async-9,3bc2551867029d95,de6391d612d8ac35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:56.599 WARN 21062 --- [ocp-async-8,c220092b30be8c84,8d11414b96f1d60a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f6874b29ea93f485,673040b6401db204] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:43:58.500 INFO 21062 --- [ocp-schedules-6,bc44680cc27b8de4,cb44b26ca6d6a48d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:43:58.500 INFO 21062 --- [ocp-schedules-27,2413e84188655de6,1e2347c98f67bc88] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:43:58.502 INFO 21062 --- [ocp-schedules-8,cf7f857f4d8dae7a,189cc1a498cde4c7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:43:58.504 INFO 21062 --- [ocp-schedules-6,bc44680cc27b8de4,cb44b26ca6d6a48d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:58.504 INFO 21062 --- [ocp-schedules-11,1f35208f8314db9b,bbdda87cd56fe316] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:43:58.504 INFO 21062 --- [ocp-schedules-27,2413e84188655de6,1e2347c98f67bc88] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:43:58.511 INFO 21062 --- [ocp-schedules-2,bc7144601999c2d3,e8b96e044cbba9a0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:43:58.511 INFO 21062 --- [ocp-schedules-5,2c3cc8dbf1aab36d,e0ec01d54c4de232] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:43:58.511 INFO 21062 --- [ocp-schedules-2,bc7144601999c2d3,e8b96e044cbba9a0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:43:58.511 INFO 21062 --- [ocp-schedules-5,2c3cc8dbf1aab36d,e0ec01d54c4de232] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:43:58.511 INFO 21062 --- [ocp-schedules-30,a042289ad8890915,6370764ca13a38cb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:43:58.511 INFO 21062 --- [ocp-schedules-30,a042289ad8890915,6370764ca13a38cb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:43:58.543 INFO 21062 --- [ocp-async-9,ddbe2178366fcadc,18ec3d696a6180bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:43:58.589 INFO 21062 --- [ocp-schedules-25,424cfa3c1c5ead38,041f7bf00dc7f70c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:43:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:43:58.588672+08:00, lastProcessTime=2024-11-04T12:43:48.589937+08:00 2024-11-04 12:43:58.593 INFO 21062 --- [ocp-schedules-25,424cfa3c1c5ead38,041f7bf00dc7f70c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:58.593 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:43:58.599 INFO 21062 --- [ocp-async-5,32ea62b226829f2c,e6d50ae74466cd67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:43:58.599 WARN 21062 --- [ocp-async-1,ebc091d8e90f5a67,2abdc9476125528b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:43:58.618 INFO 21062 --- [ocp-schedules-22,601eae7b00809816,52a41ba42807df16] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:43:58.709 INFO 21062 --- [ocp-schedules-29,599431aa80d1a6f0,8ab25cc53fd301e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:43:58.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:43:58.588672+08:00 2024-11-04 12:43:59.322 INFO 21062 --- [ocp-schedules-10,ab0ea7d25ffed03a,4607ab42c3455989] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:43:59.322 INFO 21062 --- [ocp-schedules-10,8379870a1d10aad1,e3752ae0ddacd1d2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:43:59.326 INFO 21062 --- [ocp-schedules-10,8379870a1d10aad1,e3752ae0ddacd1d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:43:59.354 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:43:59.354 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:43:59.577 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:43:59.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:43:59.983 INFO 21062 --- [prometheus-minute-schedule-1,eb20968458dc99f5,4187e2f0b00e5bc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695440, interval=60 2024-11-04 12:44:00.600 WARN 21062 --- [ocp-async-2,321821e3d3ea2b06,e9e4184d077b4de6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:00.600 INFO 21062 --- [ocp-async-4,d1722d8d10751450,944440e7dd5edda5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:01.824 INFO 21062 --- [ocp-schedules-11,2f253e37251fc6e8,85cdde7f2d942045] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:44:02.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:02.580 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:02.600 WARN 21062 --- [ocp-async-6,1122becdfc881206,64042c9fbb821b93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:02.600 INFO 21062 --- [ocp-async-8,eafdc21081c701a6,6a4bde4f454061ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:02.983 INFO 21062 --- [prometheus-minute-schedule-1,cc7aa796c6850ff0,20728f16add34d14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:04.600 INFO 21062 --- [ocp-async-9,813011568e56b206,6a4359dcf92c4edf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:04.600 WARN 21062 --- [ocp-async-1,a30ec979dba1b9a5,5eee7790a6bd0c5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:05.580 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:05.581 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c9a4f039b5f97399,2a8ff092d4f62be3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:06.600 INFO 21062 --- [ocp-async-5,6af8d3a04f8e71d0,5078549251596e57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:06.600 WARN 21062 --- [ocp-async-2,f22262622704c8e2,aefdbfc666b0f8a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:08.505 INFO 21062 --- [ocp-schedules-6,bc44680cc27b8de4,cb44b26ca6d6a48d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:44:08.505 INFO 21062 --- [ocp-schedules-27,2413e84188655de6,1e2347c98f67bc88] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:44:08.543 INFO 21062 --- [ocp-async-5,642cb94995688e9f,34088c60cd43c830] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:44:08.581 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:08.582 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:44:08.588461+08:00, lastProcessTime=2024-11-04T12:43:58.588672+08:00 2024-11-04 12:44:08.593 INFO 21062 --- [ocp-schedules-25,424cfa3c1c5ead38,041f7bf00dc7f70c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:44:08.600 INFO 21062 --- [ocp-async-6,211f01fdf595250d,7d736b9db0e602b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:08.600 WARN 21062 --- [ocp-async-3,5c94296c4175c830,70e9a1328436e960] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:08.712 INFO 21062 --- [ocp-schedules-15,a7c1203b429b1f12,e256c81cee13e4c4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:44:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:44:08.588461+08:00 2024-11-04 12:44:08.869 INFO 21062 --- [ocp-schedules-4,6b94232400c59ce9,26994f20498888ad] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:44:08.875 INFO 21062 --- [ocp-schedules-4,6b94232400c59ce9,26994f20498888ad] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:44:08.875 INFO 21062 --- [ocp-schedules-4,6b94232400c59ce9,26994f20498888ad] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:44:08.983 INFO 21062 --- [prometheus-minute-schedule-1,6d8be075f9a6750f,7df7797a36cb0885] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:44:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:44:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:44:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:44:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:44:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:44:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:44:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:44:10.600 INFO 21062 --- [ocp-async-1,4c0293563a65764d,66f10a0db06050d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:10.600 WARN 21062 --- [ocp-async-10,ca82fa38962b0c5d,126ccfabd92cadbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:11.076 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=6, affectRows=6 2024-11-04 12:44:11.076 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=6, validCount=6 2024-11-04 12:44:11.076 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=14, affectRows=14 2024-11-04 12:44:11.076 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=13, affectRows=13 2024-11-04 12:44:11.076 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=14, validCount=14 2024-11-04 12:44:11.076 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=12, validCount=12 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=40, affectRows=40 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=43, affectRows=43 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=25, affectRows=25 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=37, affectRows=37 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=40, validCount=40 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=25, validCount=25 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=43, validCount=43 2024-11-04 12:44:11.077 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=37, validCount=37 2024-11-04 12:44:11.078 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=67, affectRows=67 2024-11-04 12:44:11.078 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695380, nodeCount=77, affectRows=77 2024-11-04 12:44:11.078 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=67, validCount=67 2024-11-04 12:44:11.079 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695380, nodeCount=75, validCount=75 2024-11-04 12:44:11.582 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:11.583 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:11.849 INFO 21062 --- [ocp-schedules-21,345b5ddfae19597f,b2f4c72c435a6749] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:44:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e173114bb1fe54ab,88e0f2ba171b1e52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:12.600 INFO 21062 --- [ocp-async-2,2ad95fd5481b7b87,fcb3dcde0b3d361d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:12.600 WARN 21062 --- [ocp-async-7,57952f6790fef7d2,36dd6b50706a76ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:13.984 INFO 21062 --- [prometheus-cache-monitor1,0b519cba62e53a15,67b1c3e13c258a97] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:44:13.984 INFO 21062 --- [prometheus-cache-monitor1,0b519cba62e53a15,67b1c3e13c258a97] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:44:14.583 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:14.584 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:14.601 WARN 21062 --- [ocp-async-3,cc213c8dcf2be060,47c21dc213237d35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:14.601 INFO 21062 --- [ocp-async-4,49c21bbcd93799ce,03cc71c7657722cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:14.983 INFO 21062 --- [prometheus-minute-schedule-1,fdc1025346062201,ddcd81aef4574d0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:16.601 INFO 21062 --- [ocp-async-6,37aed08242db90d9,3a10c7bb6998bfad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:16.601 WARN 21062 --- [ocp-async-10,d4ea880ffe315310,a9fc913baae40687] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:17.584 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:17.585 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:17.983 INFO 21062 --- [prometheus-minute-schedule-1,db3cdcf92dcfe679,09c3bf52c7cb8794] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:18.543 INFO 21062 --- [ocp-async-6,9e0f08d140379fde,b20ff77967e93d4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:44:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:44:18.587174+08:00, lastProcessTime=2024-11-04T12:44:08.588461+08:00 2024-11-04 12:44:18.601 INFO 21062 --- [ocp-async-7,32613096a6791eb9,6adcd461b1a04093] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:18.601 WARN 21062 --- [ocp-async-5,1f7080bda827bd8e,bae95b8359c4291c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:18.715 INFO 21062 --- [ocp-schedules-9,caf75d6164c08edc,1ca5b66b79f7f533] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:44:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:44:18.587174+08:00 2024-11-04 12:44:20.586 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:20.587 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:20.601 WARN 21062 --- [ocp-async-8,fae551d5d9dc5a59,9f00a342bd0831c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:20.601 INFO 21062 --- [ocp-async-3,44ccd8a324716c96,f8f6dce1745bc349] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:20.983 INFO 21062 --- [prometheus-minute-schedule-1,625acfc17cf0daac,3dbc4719add56ff5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:21.891 INFO 21062 --- [ocp-schedules-16,cf5925c9f376f5f5,4b0e5ceb63444012] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:44:22.601 WARN 21062 --- [ocp-async-10,938fa2c5757f0090,75bd5e37d4725a4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:22.601 INFO 21062 --- [ocp-async-9,155b2819679b8864,561b8c0748c91475] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:23.587 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:23.588 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:23.983 INFO 21062 --- [prometheus-minute-schedule-1,3ca098c9f7ec3a53,08117b538b0626ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:24.601 WARN 21062 --- [ocp-async-5,e7ee6b7d8363d5d8,577872d916c75703] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:24.601 INFO 21062 --- [ocp-async-1,d364559c8de023c8,27b702c948323e36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:26.588 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:26.589 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:26.602 WARN 21062 --- [ocp-async-8,a4d5c8341bf6033e,99a5a993d6027aa3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:26.602 INFO 21062 --- [ocp-async-7,29fa3e0f3424617a,84610ac686a3cfe3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:26.983 INFO 21062 --- [prometheus-minute-schedule-1,7107b62b7a8d70df,e3a92b61792b947a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2e5ae39c1ca83f59,dc3230f68e47acb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:44:28.498 INFO 21062 --- [ocp-schedules-9,b467a8ad8bca5508,c586e9d56414b387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:44:28.499 INFO 21062 --- [ocp-schedules-24,50ed6fba9f8d1f38,b1552d7c972ebf4f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:44:28.500 INFO 21062 --- [ocp-schedules-29,76b419a58fd975db,ab9c80dd61c95645] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:44:28.500 INFO 21062 --- [ocp-schedules-21,c24e2e6b8f3ad913,bfb03245e1834dc6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:44:28.502 INFO 21062 --- [ocp-schedules-18,476d79ac878a90fa,5d9df43690cc2049] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:44:28.503 INFO 21062 --- [ocp-schedules-9,b467a8ad8bca5508,c586e9d56414b387] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:44:28.503 INFO 21062 --- [ocp-schedules-29,76b419a58fd975db,ab9c80dd61c95645] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:28.503 INFO 21062 --- [ocp-schedules-21,c24e2e6b8f3ad913,bfb03245e1834dc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:44:28.504 INFO 21062 --- [ocp-schedules-24,50ed6fba9f8d1f38,b1552d7c972ebf4f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:44:28.505 INFO 21062 --- [ocp-schedules-8,5bf0c27f997b33d0,e23dc43763f05435] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:44:28.511 INFO 21062 --- [ocp-schedules-12,a0f72ebeabf92058,f0ec7a5d630439b8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:44:28.511 INFO 21062 --- [ocp-schedules-12,a0f72ebeabf92058,f0ec7a5d630439b8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:44:28.511 INFO 21062 --- [ocp-schedules-17,e1fef648d2b8f6f3,07e51717c21cad49] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:44:28.511 INFO 21062 --- [ocp-schedules-17,e1fef648d2b8f6f3,07e51717c21cad49] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:44:28.511 INFO 21062 --- [ocp-schedules-1,b023952466266177,dfb0be96f09646b9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:44:28.511 INFO 21062 --- [ocp-schedules-1,b023952466266177,dfb0be96f09646b9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:44:28.531 INFO 21062 --- [ocp-schedules-13,de5cca179f49f7e1,11b1027a440f577c] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T12:44:28.531+08:00, tenantSize=0 2024-11-04 12:44:28.543 INFO 21062 --- [ocp-async-10,7f88987ba13455fe,642716687b4f5ead] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:44:28.554 INFO 21062 --- [ocp-schedules-20,c0735c3a95e67204,7f54538f74b09b7d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:44:28.554 INFO 21062 --- [ocp-schedules-20,c0735c3a95e67204,7f54538f74b09b7d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:44:28.577 INFO 21062 --- [ocp-schedules-22,274b171165349df9,178a28d78ad2d7d4] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 12:44:28.577 INFO 21062 --- [ocp-schedules-22,274b171165349df9,178a28d78ad2d7d4] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 12:44:28.588 INFO 21062 --- [ocp-schedules-11,92946b8b8521e0de,e05f989d91635826] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:44:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:44:28.588025+08:00, lastProcessTime=2024-11-04T12:44:18.587174+08:00 2024-11-04 12:44:28.602 WARN 21062 --- [ocp-async-5,20a7439138b5a387,9228ad805d04f4c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:28.602 INFO 21062 --- [ocp-async-9,5c52e82f650e452f,9b54e824ec247512] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:28.614 INFO 21062 --- [ocp-schedules-24,8c497f7ab26192f3,68b54fe13098d256] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:44:28.614 INFO 21062 --- [ocp-schedules-24,8c497f7ab26192f3,68b54fe13098d256] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:44:28.620 INFO 21062 --- [ocp-schedules-18,569472ff98146a54,6f7f477b4c0191b1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:44:28.651 INFO 21062 --- [ocp-schedules-12,856c49dd7b9a5147,c196dce61670d3ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:44:28.656 INFO 21062 --- [ocp-schedules-12,856c49dd7b9a5147,c196dce61670d3ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:44:28.656 INFO 21062 --- [ocp-schedules-12,856c49dd7b9a5147,c196dce61670d3ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:44:28.715 INFO 21062 --- [ocp-schedules-1,8b2e6770d5a96c58,c8d7f219e5a2f53f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:44:28.715 INFO 21062 --- [ocp-schedules-1,8b2e6770d5a96c58,c8d7f219e5a2f53f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:44:28.716 INFO 21062 --- [ocp-schedules-1,8b2e6770d5a96c58,c8d7f219e5a2f53f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:44:28.716 INFO 21062 --- [ocp-schedules-1,8b2e6770d5a96c58,c8d7f219e5a2f53f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:44:28.716 INFO 21062 --- [ocp-schedules-1,8b2e6770d5a96c58,c8d7f219e5a2f53f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:44:28.716 INFO 21062 --- [ocp-schedules-1,8b2e6770d5a96c58,c8d7f219e5a2f53f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:44:28.718 INFO 21062 --- [ocp-schedules-9,5e65e0420053bfde,9f5e5dd7e2269e21] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:44:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:44:28.588025+08:00 2024-11-04 12:44:28.845 INFO 21062 --- [ocp-schedules-13,04a92246d1b07d75,15fd1d8d74cce369] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:44:28.845 INFO 21062 --- [ocp-schedules-13,04a92246d1b07d75,15fd1d8d74cce369] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:44:28.854 INFO 21062 --- [ocp-schedules-25,79d7434da2acb235,3d79e64677835434] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:44:28.857 INFO 21062 --- [ocp-schedules-25,79d7434da2acb235,3d79e64677835434] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:28.874 INFO 21062 --- [ocp-schedules-13,04a92246d1b07d75,15fd1d8d74cce369] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:44:28.874 INFO 21062 --- [ocp-schedules-13,04a92246d1b07d75,15fd1d8d74cce369] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:44:28.876 INFO 21062 --- [ocp-schedules-20,d08744a42f9d8d79,82bc7330c960d96b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:44:28.882 INFO 21062 --- [ocp-schedules-20,d08744a42f9d8d79,82bc7330c960d96b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:44:28.882 INFO 21062 --- [ocp-schedules-20,d08744a42f9d8d79,82bc7330c960d96b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:44:28.885 INFO 21062 --- [ocp-schedules-19,64957ece4713f602,a555e25ccffd29bf] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:44:28.885 INFO 21062 --- [ocp-schedules-19,64957ece4713f602,a555e25ccffd29bf] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:44:28.885 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:44:28.885 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:44:28.894 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:44:28.894 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:44:28.895 INFO 21062 --- [ocp-schedules-19,64957ece4713f602,a555e25ccffd29bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:44:28.896 INFO 21062 --- [ocp-schedules-14,427a80894585ab5a,5cf34aa1564b951b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:44:28.896 WARN 21062 --- [ocp-schedules-19,64957ece4713f602,a555e25ccffd29bf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:44:28.896 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:44:28.896 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:44:28.897 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:44:28.897 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:44:28.897 INFO 21062 --- [ocp-schedules-19,64957ece4713f602,a555e25ccffd29bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:44:28.898 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:44:28.898 WARN 21062 --- [ocp-schedules-19,64957ece4713f602,a555e25ccffd29bf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:44:28.898 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:44:28.906 ERROR 21062 --- [ocp-schedules-19,64957ece4713f602,a555e25ccffd29bf] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:44:28.913 ERROR 21062 --- [ocp-schedules-25,79d7434da2acb235,3d79e64677835434] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:44:28.913 INFO 21062 --- [ocp-schedules-25,79d7434da2acb235,3d79e64677835434] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:44:28.964 INFO 21062 --- [ocp-schedules-23,c77fdd37e01df240,fc11808ad7e828a7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:44:28.964+08:00, clusters=[1] 2024-11-04 12:44:28.965 INFO 21062 --- [ocp-schedules-23,c77fdd37e01df240,fc11808ad7e828a7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:44:29.050 INFO 21062 --- [ocp-schedules-15,23ca23f54aee803b,de1893085d82c1cc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:44:29.050+08:00, clusters=[1] 2024-11-04 12:44:29.054 INFO 21062 --- [ocp-schedules-15,23ca23f54aee803b,de1893085d82c1cc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:44:29.354 INFO 21062 --- [ocp-schedules-10,8379870a1d10aad1,e3752ae0ddacd1d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:44:29.355 INFO 21062 --- [ocp-schedules-10,e377e33446265066,926b6030d87fde9c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:44:29.358 INFO 21062 --- [ocp-schedules-10,e377e33446265066,926b6030d87fde9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:29.388 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:44:29.388 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:44:29.589 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:29.590 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:29.599 INFO 21062 --- [ocp-schedules-6,a264e19635fdabd9,97f1672b609dd475] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:44:29.603 INFO 21062 --- [ocp-schedules-6,a264e19635fdabd9,97f1672b609dd475] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:29.689 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:44:29.691 ERROR 21062 --- [ocp-schedules-6,a264e19635fdabd9,97f1672b609dd475] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:44:29.691 INFO 21062 --- [ocp-schedules-6,a264e19635fdabd9,97f1672b609dd475] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:44:29.983 INFO 21062 --- [prometheus-minute-schedule-1,9366a6371dfc7c88,ca784e4ea1c316d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:44:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:44:30.602 WARN 21062 --- [ocp-async-8,1a3f41bebcfef9ab,931a34aac1accded] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:30.602 INFO 21062 --- [ocp-async-1,c7b4a954fec0dffe,9bb3284af8fc9140] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:31.949 INFO 21062 --- [ocp-schedules-2,45bc863183931586,19790abf08824e21] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:44:32.590 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:32.591 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:32.602 WARN 21062 --- [ocp-async-7,7a28589791c50ea3,69a71abc3021734a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:32.602 INFO 21062 --- [ocp-async-3,22e5235d3e94e94f,cac340d2d544c0a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7016b6e0841d61bc,ef2bfa7c8b09c631] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:34.602 WARN 21062 --- [ocp-async-3,4ae15ddbe4b34d8f,297e9fe8b0a089ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:34.602 INFO 21062 --- [ocp-async-6,fea391c1ef4588bd,f381e77b271aa2be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:35.591 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:35.592 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:35.983 INFO 21062 --- [prometheus-minute-schedule-1,659c9b4d6fcd11d5,9e3ed781e97a7559] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:36.313 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:44:36.313 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:44:36.320 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:44:36.320 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:44:36.320 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:44:36.320 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:44:36.326 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:44:36.326 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:44:36.326 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:44:36.326 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:44:36.364 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:44:36.373 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:44:36.373 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:44:36.373 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:44:36.373 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:44:36.373 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:44:36.381 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:44:36.602 WARN 21062 --- [ocp-async-9,268c4eadcb470e0e,341a2047cc025b49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:36.602 INFO 21062 --- [ocp-async-6,edf8aa3786d44e55,f0bbb63644cf5988] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:36.918 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:44:37.518 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.192036178S 2024-11-04 12:44:37.518 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:44:37.616 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:44:37.616 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:44:37.642 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:44:37.717 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:44:37.717 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:44:37.718 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:44:37.718 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:44:37.718 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:44:37.718 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:44:37.729 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:44:37.903 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] o.s.boot.SpringApplication : Started application in 1.583 seconds (JVM running for 891.582) 2024-11-04 12:44:37.903 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.583546168S 2024-11-04 12:44:38.393 INFO 21062 --- [ocp-schedules-19,2ad054911159fc51,5edcff9605c9ede0] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:44:38.504 INFO 21062 --- [ocp-schedules-21,c24e2e6b8f3ad913,bfb03245e1834dc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:44:38.504 INFO 21062 --- [ocp-schedules-29,76b419a58fd975db,ab9c80dd61c95645] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:44:38.531 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 12:44:38.531 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 12:44:38.533 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.534 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:44:38.535 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.536 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.536 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.536 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 12:44:38.544 INFO 21062 --- [ocp-async-1,9bcdff121ec969c3,4ffb774e448caa86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:44:38.553 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.553 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.554 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.554 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 12:44:38.568 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.568 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.569 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.569 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 12:44:38.583 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.584 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.584 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.585 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 12:44:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:44:38.588425+08:00, lastProcessTime=2024-11-04T12:44:28.588025+08:00 2024-11-04 12:44:38.592 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:38.593 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:38.598 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.599 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.599 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.599 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 12:44:38.602 WARN 21062 --- [ocp-async-7,6f6637e52ee5b7a2,753380a07c30ff74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:38.603 INFO 21062 --- [ocp-async-10,6366838855401e3c,768eb5dc9607712b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:38.605 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.605 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.606 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.606 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 12:44:38.611 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.611 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.612 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.612 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 12:44:38.618 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.618 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.619 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.619 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 12:44:38.625 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.626 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.626 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.627 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 12:44:38.632 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.633 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.633 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.634 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 12:44:38.640 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.640 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.641 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.641 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 12:44:38.650 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.650 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.651 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.651 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 12:44:38.666 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.667 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.667 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.668 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 12:44:38.673 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.673 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.674 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.674 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 12:44:38.680 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.680 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.681 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.682 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 12:44:38.687 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.687 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.688 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.688 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 12:44:38.693 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.693 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.694 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.694 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 12:44:38.699 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.699 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.699 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.703 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 12:44:38.708 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.708 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.708 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.709 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 12:44:38.713 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.713 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.714 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.714 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 12:44:38.718 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.718 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.719 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.719 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 12:44:38.721 INFO 21062 --- [ocp-schedules-20,2b7f00132d8dd058,4cdb1a99b7fca9e2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:44:38.724 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.724 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.725 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.725 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 12:44:38.730 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.730 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.731 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.731 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 12:44:38.736 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.736 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.737 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.737 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 12:44:38.741 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.741 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.742 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.742 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 12:44:38.746 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.747 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.747 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.748 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 12:44:38.752 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.753 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.753 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.754 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 12:44:38.758 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.759 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.759 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.760 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 12:44:38.764 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.765 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.765 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.765 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 12:44:38.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:44:38.588425+08:00 2024-11-04 12:44:38.770 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.770 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.771 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.771 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 12:44:38.775 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.776 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.776 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.777 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 12:44:38.781 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.781 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.782 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.782 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 12:44:38.787 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.787 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.788 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.788 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 12:44:38.792 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.793 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.793 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.794 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 12:44:38.801 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.801 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.802 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.802 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 12:44:38.808 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.809 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.809 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.810 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 12:44:38.815 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.815 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.816 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.816 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 12:44:38.821 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.822 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.822 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.822 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 12:44:38.826 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.826 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.827 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.827 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 12:44:38.831 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.831 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.832 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.832 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 12:44:38.836 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.836 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.837 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.837 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 12:44:38.841 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.841 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.842 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.842 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 12:44:38.851 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.852 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.852 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.853 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 12:44:38.875 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.876 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.876 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.877 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 12:44:38.883 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.884 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.884 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.885 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 12:44:38.891 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.891 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.892 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.892 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 12:44:38.898 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.898 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.899 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.899 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 12:44:38.903 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.903 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.904 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.906 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 12:44:38.911 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.911 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.912 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.912 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 12:44:38.917 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 12:44:38.917 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 12:44:38.918 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.918 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:44:38.919 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.920 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.920 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.921 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 12:44:38.928 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.928 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:44:38.929 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:38.929 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 12:44:38.933 INFO 21062 --- [ocp-schedules-9,41effe70df58585c,ca49388e0929f719] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 12:44:38.983 INFO 21062 --- [prometheus-minute-schedule-1,f96b302806244ddd,1ede86100494eb98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:44:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:44:40.603 WARN 21062 --- [ocp-async-2,9c1452299affa6fb,c12c1b9a45c39333] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:40.603 INFO 21062 --- [ocp-async-5,7cbbb485ae960ac1,05f4141df5a13cb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:41.594 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:41.595 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:44:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:44:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:44:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:44:41.854 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:44:41.854 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 12:44:41.983 INFO 21062 --- [prometheus-minute-schedule-1,014cb70f81bd95d1,295f53beead6cdac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:41.999 INFO 21062 --- [ocp-schedules-9,e2539f0044d067be,2ca08ee5f070d38b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:44:42.603 INFO 21062 --- [ocp-async-4,18c4411ee8725caa,3b97da0588793f60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:42.603 WARN 21062 --- [ocp-async-8,af3a4b55c92ad444,7e5ee087277d11df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:44.595 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:44.596 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:44.603 WARN 21062 --- [ocp-async-1,a07793b17ea3c300,233b8033848044d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:44.603 INFO 21062 --- [ocp-async-1,a8c796e64fe7f0b6,cc65b9804ee9e309] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:44.983 INFO 21062 --- [prometheus-minute-schedule-1,49f99bed92eb8594,e31539b2b43cef2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:46.603 WARN 21062 --- [ocp-async-10,1094ae4777b85094,ead75cc6d3198c38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:46.603 INFO 21062 --- [ocp-async-2,b613f682995940d0,e50be758e57a77c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:47.596 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:47.597 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:47.983 INFO 21062 --- [prometheus-minute-schedule-1,7e57e708c7049e97,ed8f8630fd4951a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:48.544 INFO 21062 --- [ocp-async-2,cda1d452800c9f85,f621707ba7475285] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:44:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:44:48.588015+08:00, lastProcessTime=2024-11-04T12:44:38.588425+08:00 2024-11-04 12:44:48.603 WARN 21062 --- [ocp-async-8,6d12ddb533329bf8,67eda540a6b925fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:48.603 INFO 21062 --- [ocp-async-6,7b2ad1799a3fa6ad,6bdb2d90396380ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:48.723 INFO 21062 --- [ocp-schedules-25,f1e9963620143ad5,a6e45d3f514204b3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:44:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:44:48.588015+08:00 2024-11-04 12:44:48.885 INFO 21062 --- [ocp-schedules-27,dd9d1c4b7d99864b,52fcfb3628d3916a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:44:48.891 INFO 21062 --- [ocp-schedules-27,dd9d1c4b7d99864b,52fcfb3628d3916a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:44:48.891 INFO 21062 --- [ocp-schedules-27,dd9d1c4b7d99864b,52fcfb3628d3916a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:44:50.597 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:50.598 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:50.604 WARN 21062 --- [ocp-async-7,f6eed1a9647d2855,69333b107df441d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:50.604 INFO 21062 --- [ocp-async-3,6cc0e1118472267f,1103c571db5c6c7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:50.983 INFO 21062 --- [prometheus-minute-schedule-1,0423cb6e1a841cd5,0781906f3d955cad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:52.025 INFO 21062 --- [ocp-schedules-28,f277418151b44e1d,422cfad4687580d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:44:52.604 WARN 21062 --- [ocp-async-10,e5ec9c275f499709,e7359c3eccc1482b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:52.604 INFO 21062 --- [ocp-async-9,f274dcd46bf994aa,515c86c8b1d7a41e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:53.599 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:53.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:53.983 INFO 21062 --- [prometheus-minute-schedule-1,3dea252738546cbb,2889e0c022726c36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:54.604 WARN 21062 --- [ocp-async-2,cb3714853fad4dd5,8c332cc9edf66e42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:54.604 INFO 21062 --- [ocp-async-8,ee536e48df8be0dc,68a1b91b14dcd945] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:56.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:56.601 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:56.604 WARN 21062 --- [ocp-async-6,0dbf39f957ae2b5a,2890da6e0daf196c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:56.604 INFO 21062 --- [ocp-async-7,8a5bdd7e7a447559,cb8aeb8d8c182e8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e23981b36a4765a7,c094f06cc6736a59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:44:58.500 INFO 21062 --- [ocp-schedules-11,e4bbcc62fe0da03a,1684b6c92d17e52f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:44:58.500 INFO 21062 --- [ocp-schedules-3,45b3ae214d20c2c2,3fd3fa00e9b8ae7f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:44:58.502 INFO 21062 --- [ocp-schedules-14,e4de21417de4fde1,9a25b1f4ed7ea71d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:44:58.503 INFO 21062 --- [ocp-schedules-11,e4bbcc62fe0da03a,1684b6c92d17e52f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:58.504 INFO 21062 --- [ocp-schedules-3,45b3ae214d20c2c2,3fd3fa00e9b8ae7f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:44:58.504 INFO 21062 --- [ocp-schedules-7,8f427ac872549091,1624290cb520ee86] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:44:58.511 INFO 21062 --- [ocp-schedules-24,baf5f790c51d4ef1,79f76d554930c899] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:44:58.511 INFO 21062 --- [ocp-schedules-23,e309ac398de7492a,55774dd49add46c8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:44:58.511 INFO 21062 --- [ocp-schedules-23,e309ac398de7492a,55774dd49add46c8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:44:58.511 INFO 21062 --- [ocp-schedules-24,baf5f790c51d4ef1,79f76d554930c899] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:44:58.511 INFO 21062 --- [ocp-schedules-8,db34de1433cf8782,1bf068da3bb418e6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:44:58.511 INFO 21062 --- [ocp-schedules-8,db34de1433cf8782,1bf068da3bb418e6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:44:58.544 INFO 21062 --- [ocp-async-7,ea46cc89d9fb868a,182c424aa9c6431c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:44:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:44:58.589080+08:00, lastProcessTime=2024-11-04T12:44:48.588015+08:00 2024-11-04 12:44:58.604 INFO 21062 --- [ocp-async-10,b1a02aa49537d344,981ffe190cccc801] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:44:58.604 WARN 21062 --- [ocp-async-5,a36ad3cef665e262,6ed57f4ad46ad717] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:44:58.622 INFO 21062 --- [ocp-schedules-27,4df343059a8df4b5,a8a80d5268348064] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:44:58.726 INFO 21062 --- [ocp-schedules-28,a315fbee6f6e21c6,9dbee9746b5829fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:44:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:44:58.589080+08:00 2024-11-04 12:44:59.389 INFO 21062 --- [ocp-schedules-10,e377e33446265066,926b6030d87fde9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:44:59.389 INFO 21062 --- [ocp-schedules-10,38756a3ed1bc6e85,dc40e2d6ca527241] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:44:59.393 INFO 21062 --- [ocp-schedules-10,38756a3ed1bc6e85,dc40e2d6ca527241] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:44:59.422 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:44:59.422 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:44:59.601 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:44:59.603 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:44:59.983 INFO 21062 --- [prometheus-minute-schedule-1,b884461058df805e,d401548147cbda98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695500, interval=60 2024-11-04 12:45:00.326 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4f973280e7dc21b1] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:40:01.0, nextRun=2024-11-04 12:45:00.0, currentDate=Mon Nov 04 12:45:00 CST 2024 2024-11-04 12:45:00.326 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4f973280e7dc21b1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 12:40:01.0, nextRunTime=2024-11-04 12:45:00.0} 2024-11-04 12:45:00.353 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4f973280e7dc21b1] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 12:45:00.358 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4f973280e7dc21b1] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 12:45:00.358 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4f973280e7dc21b1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 12:45:00.604 WARN 21062 --- [ocp-async-2,2a99e2b90bac02b6,0eb6a4e6f456c6f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:00.604 INFO 21062 --- [ocp-async-4,b5aefc8d9ef12076,79ac23c0a3e378e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:02.064 INFO 21062 --- [ocp-schedules-1,0cd76b1ab1d0123c,76c97a6dd7004084] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:45:02.603 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:02.604 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:02.604 WARN 21062 --- [ocp-async-6,8b8fe8185441a698,1b6bf5dec504c550] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:02.605 INFO 21062 --- [ocp-async-1,080e98524678ca7e,e2950a0e2e14b3fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:02.983 INFO 21062 --- [prometheus-minute-schedule-1,848cc3af02c0a1bd,c8da9fde6e3a63db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:04.605 WARN 21062 --- [ocp-async-3,12b2b949b31b72a5,41d2c55d6308312d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:04.605 INFO 21062 --- [ocp-async-5,85e3fa04e4107acb,879871bdc1250525] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:05.604 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:05.605 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f8f5138e7c8740a7,cc9c85ebf0345ce6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:06.605 WARN 21062 --- [ocp-async-10,da7034737544c504,a2cb3190261e80a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:06.605 INFO 21062 --- [ocp-async-2,a037201a81a0b0e8,56679d94b69cba66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:08.504 INFO 21062 --- [ocp-schedules-3,45b3ae214d20c2c2,3fd3fa00e9b8ae7f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:45:08.504 INFO 21062 --- [ocp-schedules-11,e4bbcc62fe0da03a,1684b6c92d17e52f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:45:08.544 INFO 21062 --- [ocp-async-2,99d4f5aad5f52d41,0498ab361caadd58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:45:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:45:08.588363+08:00, lastProcessTime=2024-11-04T12:44:58.589080+08:00 2024-11-04 12:45:08.593 INFO 21062 --- [ocp-schedules-22,df041a2ae825fc7c,4fe7672469fac0c8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:45:08.597 INFO 21062 --- [ocp-schedules-22,df041a2ae825fc7c,4fe7672469fac0c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:08.598 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:45:08.605 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:08.605 INFO 21062 --- [ocp-async-6,f711e0298a5693df,a6cc273b2a1301e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:08.605 WARN 21062 --- [ocp-async-7,dc4729fe9cd97bf5,0d22ac954d71ad0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:08.605 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:08.728 INFO 21062 --- [ocp-schedules-16,b17ff195024bf89a,a1e857999f1f35cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:45:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:45:08.588363+08:00 2024-11-04 12:45:08.893 INFO 21062 --- [ocp-schedules-3,51c1c47880a3967d,053776e7b88feafa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:45:08.899 INFO 21062 --- [ocp-schedules-3,51c1c47880a3967d,053776e7b88feafa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:45:08.899 INFO 21062 --- [ocp-schedules-3,51c1c47880a3967d,053776e7b88feafa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:45:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f5fbdf3ca95ecbda,516654e4ed290359] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:45:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:45:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:45:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:45:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:45:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:45:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:45:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:45:10.605 INFO 21062 --- [ocp-async-3,0668a841799dc0a6,a58001618ad9dceb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:10.605 WARN 21062 --- [ocp-async-9,305673e8e4660333,951acbf8dc1dd389] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:11.079 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=3, affectRows=3 2024-11-04 12:45:11.079 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=3, validCount=3 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=5, affectRows=5 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=5, validCount=5 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=10, affectRows=10 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=14, affectRows=14 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=10, validCount=10 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=14, validCount=14 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=19, affectRows=19 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=19, validCount=19 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=21, affectRows=21 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=34, affectRows=34 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=21, validCount=21 2024-11-04 12:45:11.080 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=34, validCount=34 2024-11-04 12:45:11.081 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=43, affectRows=43 2024-11-04 12:45:11.081 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=47, affectRows=47 2024-11-04 12:45:11.081 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=1, affectRows=1 2024-11-04 12:45:11.081 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=1, validCount=1 2024-11-04 12:45:11.081 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=43, validCount=43 2024-11-04 12:45:11.081 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=47, validCount=47 2024-11-04 12:45:11.081 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=50, affectRows=50 2024-11-04 12:45:11.082 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695440, nodeCount=75, affectRows=75 2024-11-04 12:45:11.082 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=49, validCount=49 2024-11-04 12:45:11.082 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695440, nodeCount=73, validCount=73 2024-11-04 12:45:11.606 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:11.606 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a5284202c58ea012,88e18188e49433aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:12.090 INFO 21062 --- [ocp-schedules-4,e7bde45a4022b13a,d1bfdccaae83027c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:45:12.605 WARN 21062 --- [ocp-async-8,85cd3b65942019a6,0e92656692454e69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:12.605 INFO 21062 --- [ocp-async-10,8bdbb971138227bb,368e0f6662a26f20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:13.984 INFO 21062 --- [prometheus-cache-monitor1,3fc6715da14e19b4,a05870e75025ee27] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:45:13.984 INFO 21062 --- [prometheus-cache-monitor1,3fc6715da14e19b4,a05870e75025ee27] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:45:14.115 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:41:14.097, end=2024-11-04T12:43:14.113} 2024-11-04 12:45:14.125 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:45:14.125 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:45:14.605 WARN 21062 --- [ocp-async-7,537c7dde9bc4a16f,97829a9981b2981b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:14.605 INFO 21062 --- [ocp-async-4,a913e26dc5388f01,a028297486fffc41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:14.606 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:14.607 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:14.983 INFO 21062 --- [prometheus-minute-schedule-1,cd1aa1ef47276582,6c0c2098ca5bf420] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:16.606 WARN 21062 --- [ocp-async-6,f53980753bdc95f4,8b64803ac32896ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:16.606 INFO 21062 --- [ocp-async-9,3899237bb387caf7,0909e811b5ae64da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:17.607 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:17.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5cb574460321c24b,b0674473c5d46246] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:18.544 INFO 21062 --- [ocp-async-9,0ac4f01aa31d709f,9f7235f3175b9408] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:45:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:45:18.588803+08:00, lastProcessTime=2024-11-04T12:45:08.588363+08:00 2024-11-04 12:45:18.598 INFO 21062 --- [ocp-schedules-22,df041a2ae825fc7c,4fe7672469fac0c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:45:18.606 WARN 21062 --- [ocp-async-8,0f69b562fba0e4d9,f816bf476483231f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:18.606 INFO 21062 --- [ocp-async-2,94a70d11ba280104,d93d0fe839da1615] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:18.731 INFO 21062 --- [ocp-schedules-25,4f5e3acb632ffddf,c4250c0582c52ca4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:45:18.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:45:18.588803+08:00 2024-11-04 12:45:20.606 WARN 21062 --- [ocp-async-2,1d2f3f0977bae541,f7bcd5342e739328] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:20.606 INFO 21062 --- [ocp-async-7,b85627f331fa9ff2,b5c25995c0e03d75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:20.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:20.609 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:20.983 INFO 21062 --- [prometheus-minute-schedule-1,2670a68a87060860,59081b4fa8eb61de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:22.132 INFO 21062 --- [ocp-schedules-16,71aa55cb04a93633,8673f8dcafc504a6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:45:22.606 WARN 21062 --- [ocp-async-7,f26a6915a78923cf,7183052f2fb3bc8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:22.606 INFO 21062 --- [ocp-async-6,88fe219e9ee0bf4a,9ba65580e309f9b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:23.609 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:23.610 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:23.983 INFO 21062 --- [prometheus-minute-schedule-1,5493868fffcb80b7,108f06e02eda5d43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:24.606 WARN 21062 --- [ocp-async-3,951ef987b579ebba,4900f43448a44aa2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:24.606 INFO 21062 --- [ocp-async-6,7e130220c860fec5,29d118eb24eaa9aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:26.607 WARN 21062 --- [ocp-async-2,f13b9d7194005dec,ce640425faef5d18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:26.607 INFO 21062 --- [ocp-async-6,e591aaf9b63a7665,dc57e043d9b94fe6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:26.610 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:26.611 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:26.983 INFO 21062 --- [prometheus-minute-schedule-1,5902374c9ef47083,70aed117e68309f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:26.984 INFO 21062 --- [prometheus-hour-schedule-1,58170e4c24a92ca3,5e86aba1a26056c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:45:28.498 INFO 21062 --- [ocp-schedules-20,1fb3ab7f61b8a877,423d809e3a5b122d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:45:28.499 INFO 21062 --- [ocp-schedules-21,6f3be8a53925116c,77d7d42c48eb4640] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:45:28.500 INFO 21062 --- [ocp-schedules-25,2896696c707f25d8,65345aa24cc2541f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:45:28.500 INFO 21062 --- [ocp-schedules-23,551f77eea1916bf1,55c28aa0015b3778] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:45:28.502 INFO 21062 --- [ocp-schedules-29,55a09f16925bd1b6,bd06c38bd043d80f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:45:28.503 INFO 21062 --- [ocp-schedules-20,1fb3ab7f61b8a877,423d809e3a5b122d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:45:28.504 INFO 21062 --- [ocp-schedules-23,551f77eea1916bf1,55c28aa0015b3778] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:28.504 INFO 21062 --- [ocp-schedules-25,2896696c707f25d8,65345aa24cc2541f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:28.506 INFO 21062 --- [ocp-schedules-21,6f3be8a53925116c,77d7d42c48eb4640] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:45:28.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:45:28.507 INFO 21062 --- [ocp-schedules-30,faa3bfc2cc056da9,6ee794abe1556fbd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:45:28.511 INFO 21062 --- [ocp-schedules-17,0de8298188689274,0494bd79da2affe5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:45:28.511 INFO 21062 --- [ocp-schedules-1,3b68a55b193d94db,27f0d9080d4de614] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:45:28.511 INFO 21062 --- [ocp-schedules-1,3b68a55b193d94db,27f0d9080d4de614] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:45:28.511 INFO 21062 --- [ocp-schedules-17,0de8298188689274,0494bd79da2affe5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:45:28.511 INFO 21062 --- [ocp-schedules-16,5dd4eb24372c56d0,bd2194006259b633] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:45:28.511 INFO 21062 --- [ocp-schedules-16,5dd4eb24372c56d0,bd2194006259b633] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:45:28.545 INFO 21062 --- [ocp-async-9,28a8f4827ad82a65,573930af823c722d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:45:28.568 INFO 21062 --- [ocp-schedules-2,c0f99a1c7b903e30,3d0d94287b24293a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:45:28.569 INFO 21062 --- [ocp-schedules-2,c0f99a1c7b903e30,3d0d94287b24293a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:45:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:45:28.587643+08:00, lastProcessTime=2024-11-04T12:45:18.588803+08:00 2024-11-04 12:45:28.590 INFO 21062 --- [ocp-schedules-13,797c8d00190403ec,27f78747d3c5f386] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:45:28.607 WARN 21062 --- [ocp-async-5,9a02a73901643503,5054a76ff8752ea4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:28.607 INFO 21062 --- [ocp-async-3,9cfb6671c4a2ccf3,2c7c9efaf1a3c584] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:28.618 INFO 21062 --- [ocp-schedules-8,915b4fb5605b7ed6,4f4e4765534f946e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:45:28.618 INFO 21062 --- [ocp-schedules-8,915b4fb5605b7ed6,4f4e4765534f946e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:45:28.624 INFO 21062 --- [ocp-schedules-4,1f5fc04d987f6d86,02407a602636d1a1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:45:28.656 INFO 21062 --- [ocp-schedules-29,e71e0ee70ee01d85,9014b3493128f05c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:45:28.660 INFO 21062 --- [ocp-schedules-29,e71e0ee70ee01d85,9014b3493128f05c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:45:28.660 INFO 21062 --- [ocp-schedules-29,e71e0ee70ee01d85,9014b3493128f05c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:45:28.717 INFO 21062 --- [ocp-schedules-30,c047567298ed6777,e2d5642bbc36c11a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:45:28.717 INFO 21062 --- [ocp-schedules-30,c047567298ed6777,e2d5642bbc36c11a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:45:28.723 INFO 21062 --- [ocp-schedules-30,c047567298ed6777,e2d5642bbc36c11a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:45:28.723 INFO 21062 --- [ocp-schedules-30,c047567298ed6777,e2d5642bbc36c11a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:45:28.728 INFO 21062 --- [ocp-schedules-30,c047567298ed6777,e2d5642bbc36c11a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:45:28.728 INFO 21062 --- [ocp-schedules-30,c047567298ed6777,e2d5642bbc36c11a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:45:28.733 INFO 21062 --- [ocp-schedules-1,67ca6a473bc4157c,6bc4b093685d21e4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:45:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:45:28.587643+08:00 2024-11-04 12:45:28.874 INFO 21062 --- [ocp-schedules-14,99f308e094124e54,e1c8260d3855239d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:45:28.874 INFO 21062 --- [ocp-schedules-14,99f308e094124e54,e1c8260d3855239d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:45:28.901 INFO 21062 --- [ocp-schedules-14,99f308e094124e54,e1c8260d3855239d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:45:28.901 INFO 21062 --- [ocp-schedules-14,99f308e094124e54,e1c8260d3855239d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:45:28.901 INFO 21062 --- [ocp-schedules-28,68da8c1d628969c7,6dfc0870c01c9bed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:45:28.908 INFO 21062 --- [ocp-schedules-28,68da8c1d628969c7,6dfc0870c01c9bed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:45:28.908 INFO 21062 --- [ocp-schedules-28,68da8c1d628969c7,6dfc0870c01c9bed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:45:28.914 INFO 21062 --- [ocp-schedules-3,4a951ee61391c1ad,66f8de5c39935294] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:45:28.914 INFO 21062 --- [ocp-schedules-3,4a951ee61391c1ad,66f8de5c39935294] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:45:28.914 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:45:28.914 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:45:28.921 INFO 21062 --- [ocp-schedules-15,9aec0c94ae4fd264,e11e7b2bdc6ba4b0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:45:28.923 INFO 21062 --- [ocp-schedules-3,4a951ee61391c1ad,66f8de5c39935294] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:45:28.923 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:45:28.924 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:45:28.925 WARN 21062 --- [ocp-schedules-3,4a951ee61391c1ad,66f8de5c39935294] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:45:28.925 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:45:28.925 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:45:28.926 INFO 21062 --- [ocp-schedules-3,4a951ee61391c1ad,66f8de5c39935294] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:45:28.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:45:28.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:45:28.927 WARN 21062 --- [ocp-schedules-3,4a951ee61391c1ad,66f8de5c39935294] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:45:28.927 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:45:28.927 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:45:28.933 ERROR 21062 --- [ocp-schedules-3,4a951ee61391c1ad,66f8de5c39935294] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2024-11-04 12:45:28.991 INFO 21062 --- [ocp-schedules-5,e320ed5183297a26,3792160aa0a52f64] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:45:28.991+08:00, clusters=[1] 2024-11-04 12:45:28.992 INFO 21062 --- [ocp-schedules-5,e320ed5183297a26,3792160aa0a52f64] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:45:29.081 INFO 21062 --- [ocp-schedules-9,2e32f9e4ef4709bb,f9523de0a0fc8956] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:45:29.081+08:00, clusters=[1] 2024-11-04 12:45:29.084 INFO 21062 --- [ocp-schedules-9,2e32f9e4ef4709bb,f9523de0a0fc8956] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:45:29.423 INFO 21062 --- [ocp-schedules-10,38756a3ed1bc6e85,dc40e2d6ca527241] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:45:29.423 INFO 21062 --- [ocp-schedules-10,5228274569a392e9,103c186edde0a8c7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:45:29.427 INFO 21062 --- [ocp-schedules-10,5228274569a392e9,103c186edde0a8c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:29.456 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:45:29.456 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:45:29.611 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:29.612 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:29.691 INFO 21062 --- [ocp-schedules-29,e7ce8f6299223234,9527efef8a39aa38] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:45:29.695 INFO 21062 --- [ocp-schedules-29,e7ce8f6299223234,9527efef8a39aa38] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:29.783 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:45:29.789 ERROR 21062 --- [ocp-schedules-29,e7ce8f6299223234,9527efef8a39aa38] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:45:29.789 INFO 21062 --- [ocp-schedules-29,e7ce8f6299223234,9527efef8a39aa38] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:45:29.983 INFO 21062 --- [prometheus-minute-schedule-1,aa0b6f03b9fb5d72,573f02bf24ccf9e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:45:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:45:30.607 WARN 21062 --- [ocp-async-10,6b6735a9e864b7b8,5849fef31d9f449a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:30.607 INFO 21062 --- [ocp-async-2,602319a8f09f1ce1,94f3347da44215d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:32.158 INFO 21062 --- [ocp-schedules-28,e6df1849e8d0fc1c,be9cf95774e419db] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:45:32.607 WARN 21062 --- [ocp-async-1,2dd24b04f80f0b7c,1bdd74b73f94e609] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:32.607 INFO 21062 --- [ocp-async-6,7cc71d8125e210d7,fbe913232544ea7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:32.612 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:32.613 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:32.983 INFO 21062 --- [prometheus-minute-schedule-1,116e4db75522bc66,0a3fafcc930a0ec6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:34.607 WARN 21062 --- [ocp-async-9,bc1970b5384c66a3,b582524e235d4296] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:34.607 INFO 21062 --- [ocp-async-5,1e806f23d77e1627,6823c64c93ad3cd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:35.613 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:35.614 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c554d532dc521f3b,c0b58300ff5b95f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:36.608 WARN 21062 --- [ocp-async-3,bf23908bcec968a0,a1200db904b932dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:36.608 INFO 21062 --- [ocp-async-10,f8abe2c05e9e1da8,de2c4c75c0d1b4ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:38.506 INFO 21062 --- [ocp-schedules-23,551f77eea1916bf1,55c28aa0015b3778] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:45:38.507 INFO 21062 --- [ocp-schedules-25,2896696c707f25d8,65345aa24cc2541f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:45:38.545 INFO 21062 --- [ocp-async-1,1b1c4543ca57ed11,f28f6ca53eb0e5c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:45:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:45:38.588678+08:00, lastProcessTime=2024-11-04T12:45:28.587643+08:00 2024-11-04 12:45:38.608 WARN 21062 --- [ocp-async-7,1669a159615a0216,c3ba23874ce41439] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:38.608 INFO 21062 --- [ocp-async-7,4c3721215074dfc8,c413f652b0825341] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:38.614 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:38.615 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:38.736 INFO 21062 --- [ocp-schedules-24,e733bfbcb489ce70,4d75a36e73165bd9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:45:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:45:38.588678+08:00 2024-11-04 12:45:38.983 INFO 21062 --- [prometheus-minute-schedule-1,294736b457c89e0e,b41175508cd38156] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:40.002 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:45:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:45:40.608 WARN 21062 --- [ocp-async-8,2b252c7bb539389e,3defac3cd9f69b65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:40.608 INFO 21062 --- [ocp-async-5,73cb41d292451ead,9a610ad571f1cbb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:41.615 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:41.616 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:41.985 INFO 21062 --- [prometheus-minute-schedule-1,d88fbc0ac06c0ef7,4de6ad9776833c7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:42.201 INFO 21062 --- [ocp-schedules-29,fe4864a4f9898710,d636d9fa6db9274c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:45:42.608 WARN 21062 --- [ocp-async-4,caf149896d57f7ab,00c9c09d3b9e11ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:42.608 INFO 21062 --- [ocp-async-10,502a9c473b1450d3,46b18b2ba470741c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:44.608 WARN 21062 --- [ocp-async-1,293267e5b21b14cd,c7045cc6285a2747] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:44.608 INFO 21062 --- [ocp-async-6,d62aa0726fb07cb3,2a319843bac3c9b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:44.616 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:44.617 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:44.983 INFO 21062 --- [prometheus-minute-schedule-1,72e6f3e45b894e48,55f96f02c15dae30] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:46.608 WARN 21062 --- [ocp-async-7,7d7073067e814d20,dd617d8f612a2597] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:46.608 INFO 21062 --- [ocp-async-8,f429cedaa5fcec1e,d89a8843247fdd20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:47.617 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:47.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:47.983 INFO 21062 --- [prometheus-minute-schedule-1,ae0d00a8c48c78dc,98a1f4fa2678457f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:48.545 INFO 21062 --- [ocp-async-8,f0043e81f7b9760b,4ecba37e9788bfc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:45:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:45:48.587671+08:00, lastProcessTime=2024-11-04T12:45:38.588678+08:00 2024-11-04 12:45:48.609 WARN 21062 --- [ocp-async-4,5a4db6db316beffa,8b46adb6d5464923] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:48.609 INFO 21062 --- [ocp-async-2,edf96d5165bc28eb,e35643ff4350bb2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:48.738 INFO 21062 --- [ocp-schedules-28,bdc7da9c02235696,7498111caad0e8dd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:45:48.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:45:48.587671+08:00 2024-11-04 12:45:48.910 INFO 21062 --- [ocp-schedules-7,e1055d8bb802f2cd,59fcaf7662a26ce4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:45:48.917 INFO 21062 --- [ocp-schedules-7,e1055d8bb802f2cd,59fcaf7662a26ce4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:45:48.917 INFO 21062 --- [ocp-schedules-7,e1055d8bb802f2cd,59fcaf7662a26ce4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:45:50.609 WARN 21062 --- [ocp-async-1,c9dbc9f33facc26f,2fbe83f079359d58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:50.609 INFO 21062 --- [ocp-async-1,a0b07d9d725d4582,c45894cedac2ec57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:50.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:50.619 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:50.983 INFO 21062 --- [prometheus-minute-schedule-1,cfe1357b5b620387,7a84c6869b9382de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:52.228 INFO 21062 --- [ocp-schedules-12,e868b409a08bd3a1,9f13f17bd6f28d77] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:45:52.609 WARN 21062 --- [ocp-async-7,175db35a958033b0,5584bd5e3ad92f78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:52.609 INFO 21062 --- [ocp-async-3,1f95111a7af3bc60,1da73411b3b63349] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:53.619 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:53.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e7b7d867ae03338f,86487541bb8690aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:54.609 WARN 21062 --- [ocp-async-5,115e89f528390861,eb8b13c05ff33da2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:54.609 INFO 21062 --- [ocp-async-4,b1641b2b7ac8a769,3accac7637deb691] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:56.609 WARN 21062 --- [ocp-async-2,055cf5bbd9c47c7d,a97c047b33f74654] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:56.609 INFO 21062 --- [ocp-async-9,fa791991908e8177,ff9e37d69a2f304c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:56.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:56.621 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f6e43f4511bb9324,3fa34108cb3751a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:45:58.500 INFO 21062 --- [ocp-schedules-24,56ab280474584a5e,a4dd200d95e4804e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:45:58.500 INFO 21062 --- [ocp-schedules-27,9f994f05d6df8159,3f15aea31fff5793] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:45:58.502 INFO 21062 --- [ocp-schedules-15,90ee621d40acca83,0ac0936b6762ca9c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:45:58.505 INFO 21062 --- [ocp-schedules-27,9f994f05d6df8159,3f15aea31fff5793] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:58.505 INFO 21062 --- [ocp-schedules-7,0efa1f56d7844c77,cbe98099867c761f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:45:58.506 INFO 21062 --- [ocp-schedules-24,56ab280474584a5e,a4dd200d95e4804e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:45:58.513 INFO 21062 --- [ocp-schedules-9,1ed255d02a600128,3139b2bf8e22e180] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:45:58.513 INFO 21062 --- [ocp-schedules-8,9aedcfe1b52d4090,8c5f39480f9054d9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:45:58.513 INFO 21062 --- [ocp-schedules-22,4fa4e520a0da47ea,ba8306321163eb27] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:45:58.513 INFO 21062 --- [ocp-schedules-8,9aedcfe1b52d4090,8c5f39480f9054d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:45:58.513 INFO 21062 --- [ocp-schedules-22,4fa4e520a0da47ea,ba8306321163eb27] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:45:58.513 INFO 21062 --- [ocp-schedules-9,1ed255d02a600128,3139b2bf8e22e180] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:45:58.545 INFO 21062 --- [ocp-async-9,d07d3706c7da9ba7,9d9c15ef114abe05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:45:58.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:45:58.592151+08:00, lastProcessTime=2024-11-04T12:45:48.587671+08:00 2024-11-04 12:45:58.609 INFO 21062 --- [ocp-async-7,5afd6f7e7861bce9,01876c0bb9dddb6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:45:58.609 WARN 21062 --- [ocp-async-8,c0da86eabb6fb527,576c43aa1eb0f5a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:45:58.627 INFO 21062 --- [ocp-schedules-6,c41bef2edb2ae372,3e4b44738d3e16bd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:45:58.741 INFO 21062 --- [ocp-schedules-2,96d7a68815aff25a,75498cfec1619294] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:45:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:45:58.592151+08:00 2024-11-04 12:45:59.457 INFO 21062 --- [ocp-schedules-10,5228274569a392e9,103c186edde0a8c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:45:59.457 INFO 21062 --- [ocp-schedules-10,24f3d63719aa57a1,91f9f2e7cb890ae6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:45:59.460 INFO 21062 --- [ocp-schedules-10,24f3d63719aa57a1,91f9f2e7cb890ae6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:45:59.488 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:45:59.488 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:45:59.621 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:45:59.622 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:45:59.983 INFO 21062 --- [prometheus-minute-schedule-1,4546b3cd4a9f3aca,d8e309a181929298] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695560, interval=60 2024-11-04 12:46:00.610 INFO 21062 --- [ocp-async-5,ad3eb920fb81ad07,cb436d8bf2c3434c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:00.610 WARN 21062 --- [ocp-async-10,9ca2ced5db4010cf,5038b994b8e18641] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:02.268 INFO 21062 --- [ocp-schedules-16,92655b9548991055,d6911c3d19c68c56] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:46:02.610 WARN 21062 --- [ocp-async-6,119413ee9a52d1b9,d2092732a08f829f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:02.610 INFO 21062 --- [ocp-async-2,6f7e3b4a3632d48e,bb01acfea0e5c5d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:02.622 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:02.623 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5a991afc9c68c0c0,ce8122cb1c4fb1b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:04.610 WARN 21062 --- [ocp-async-8,bc6db1e879e3d6a2,18c8b15d8755dbe0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:04.610 INFO 21062 --- [ocp-async-1,2aa45388675424a5,5c710a7590a2db14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:05.623 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:05.624 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:05.983 INFO 21062 --- [prometheus-minute-schedule-1,bf983872c1090ed2,c8e801886ed730fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:06.610 WARN 21062 --- [ocp-async-7,15ce6048af0e0d25,fa54bd625b31ffe0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:06.610 INFO 21062 --- [ocp-async-10,95be88f6fe701314,d99dc888c6a36884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:08.506 INFO 21062 --- [ocp-schedules-27,9f994f05d6df8159,3f15aea31fff5793] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:46:08.507 INFO 21062 --- [ocp-schedules-24,56ab280474584a5e,a4dd200d95e4804e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:46:08.545 INFO 21062 --- [ocp-async-10,d9431d8ca8fca75b,e9b6e61f9c67bd68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:46:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:46:08.588597+08:00, lastProcessTime=2024-11-04T12:45:58.592151+08:00 2024-11-04 12:46:08.610 WARN 21062 --- [ocp-async-6,5c1ec0512faa8457,fc82f1748f638c91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:08.610 INFO 21062 --- [ocp-async-9,53ea7a94e928eea3,347c4d5b0ef9105d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:08.625 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:08.625 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:08.744 INFO 21062 --- [ocp-schedules-17,bfbf94c76eca8207,e37cdde1d3c7be4b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:46:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:46:08.588597+08:00 2024-11-04 12:46:08.919 INFO 21062 --- [ocp-schedules-20,9e53cbf963521e73,0d4d85fcbedbc584] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:46:08.927 INFO 21062 --- [ocp-schedules-20,9e53cbf963521e73,0d4d85fcbedbc584] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:46:08.927 INFO 21062 --- [ocp-schedules-20,9e53cbf963521e73,0d4d85fcbedbc584] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:46:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a4c884fc2f128017,cb0fb6a200ec3337] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:46:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:46:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:46:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:46:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:46:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:46:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:46:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:46:10.611 WARN 21062 --- [ocp-async-8,b4ab214ad459a730,96368577aa5ecbca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:10.611 INFO 21062 --- [ocp-async-3,863b65ebb5861cd9,8bdea338883fbe85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:11.084 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=3, affectRows=3 2024-11-04 12:46:11.084 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=2, validCount=2 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=4, affectRows=4 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=4, validCount=4 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=21, affectRows=21 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=44, affectRows=44 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=28, affectRows=28 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=14, affectRows=14 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=29, affectRows=29 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=51, affectRows=51 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=51, affectRows=51 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=44, affectRows=44 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=14, validCount=14 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=21, validCount=21 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=28, validCount=28 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695500, nodeCount=33, affectRows=33 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=44, validCount=44 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=29, validCount=29 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=44, validCount=44 2024-11-04 12:46:11.203 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=51, validCount=51 2024-11-04 12:46:11.204 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=49, validCount=49 2024-11-04 12:46:11.204 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695500, nodeCount=33, validCount=33 2024-11-04 12:46:11.625 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:11.626 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:11.983 INFO 21062 --- [prometheus-minute-schedule-1,230ea42cd7d6c9f4,39f9b7938ff46962] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:12.293 INFO 21062 --- [ocp-schedules-28,4b7f5a29daf25552,c1a7684b2e6315a5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:46:12.611 WARN 21062 --- [ocp-async-4,125b06f35a437265,25ab7f69c266827c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:12.611 INFO 21062 --- [ocp-async-7,8682de4e213d5387,879721360da80a5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,867b1c2edce8401f,d265eb359c1de6d7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,867b1c2edce8401f,d265eb359c1de6d7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:46:14.611 WARN 21062 --- [ocp-async-5,75967f0eebd8f196,3ce04b6165896df6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:14.611 INFO 21062 --- [ocp-async-6,f93f822be9da5ed8,70705bce73b28725] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:14.626 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:14.627 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3d44513e41d5d838,79c7e90c184f7aff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:16.611 INFO 21062 --- [ocp-async-9,1a08e79cea3818ad,af8cd7c4ec93479d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:16.611 WARN 21062 --- [ocp-async-8,a6179637f6f47f57,7959f560a13dc6ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:17.627 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:17.628 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ad416d381126d3a4,f576d89a9e70b28d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:18.546 INFO 21062 --- [ocp-async-9,27502ac899db7491,0bdcc0f94428c404] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:46:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:46:18.589511+08:00, lastProcessTime=2024-11-04T12:46:08.588597+08:00 2024-11-04 12:46:18.598 INFO 21062 --- [ocp-schedules-5,a6193b7c1ec29c40,975586f4027158b9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:46:18.601 INFO 21062 --- [ocp-schedules-5,a6193b7c1ec29c40,975586f4027158b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:18.602 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:46:18.611 WARN 21062 --- [ocp-async-4,0f059b163933ca8b,2ccecfcb35c91ca3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:18.611 INFO 21062 --- [ocp-async-3,d231a56e0ecd0b6c,a135b709fcdc11e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:18.746 INFO 21062 --- [ocp-schedules-27,9e281d4d03f1cc4a,d174679cf259c1dd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:46:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:46:18.589511+08:00 2024-11-04 12:46:20.611 WARN 21062 --- [ocp-async-5,7cdbeda891e64a14,fc7a733d9209a408] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:20.611 INFO 21062 --- [ocp-async-3,44417bffdb8e6626,7573061c67a62b86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:20.628 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:20.629 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:20.983 INFO 21062 --- [prometheus-minute-schedule-1,d7cdf233871d529d,36f3c96f720315d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:22.340 INFO 21062 --- [ocp-schedules-1,e7c0a9c23ab25150,8790b078f691e067] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:46:22.612 WARN 21062 --- [ocp-async-8,ab4c2f88278b02cb,35df893d2cca94cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:22.612 INFO 21062 --- [ocp-async-3,1fa2c817a280faca,9299dda737e2c399] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:23.629 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:23.630 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6dbc89640832f87a,070907e341bdd1d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:24.612 INFO 21062 --- [ocp-async-3,70d1bdb3cad172a8,211c4a08ab86b17a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:24.612 WARN 21062 --- [ocp-async-4,67da8327ab472554,ac0f7f93713eb65b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:26.612 WARN 21062 --- [ocp-async-5,2b117267b88c2fe9,1fc1ab16920cbabc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:26.612 INFO 21062 --- [ocp-async-3,10364cb790311dba,583c196279197bca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:26.630 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:26.631 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e5d36e1008fbe8cf,46983a73c668a2ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9ed040d4c7b7c459,d34a976aca651c89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:46:28.498 INFO 21062 --- [ocp-schedules-16,1d24e914bc906ab2,de0417613db9120c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:46:28.499 INFO 21062 --- [ocp-schedules-19,738f021908a40c08,31192d81582a1f8b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:46:28.500 INFO 21062 --- [ocp-schedules-12,40203af8920996c7,ed1177ebf8b5c30a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:46:28.500 INFO 21062 --- [ocp-schedules-1,1ce30ce8283e6502,8e60c9af76613f49] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:46:28.502 INFO 21062 --- [ocp-schedules-16,1d24e914bc906ab2,de0417613db9120c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:46:28.502 INFO 21062 --- [ocp-schedules-22,c09035c4c60d2e3f,4993adda2e1494f3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:46:28.503 INFO 21062 --- [ocp-schedules-12,40203af8920996c7,ed1177ebf8b5c30a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:28.503 INFO 21062 --- [ocp-schedules-1,1ce30ce8283e6502,8e60c9af76613f49] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:46:28.504 INFO 21062 --- [ocp-schedules-24,9913c1d5ef1a7811,decd8dc8e1afa768] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:46:28.505 INFO 21062 --- [ocp-schedules-19,738f021908a40c08,31192d81582a1f8b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:46:28.509 INFO 21062 --- [ocp-schedules-4,19a9fc1eb603dc7b,fbe7c0f6c075e3de] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:46:28.509 INFO 21062 --- [ocp-schedules-4,19a9fc1eb603dc7b,fbe7c0f6c075e3de] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:46:28.510 INFO 21062 --- [ocp-schedules-15,7fe4e18489dcdc10,57a664966f8f597d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:46:28.510 INFO 21062 --- [ocp-schedules-15,7fe4e18489dcdc10,57a664966f8f597d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:46:28.510 INFO 21062 --- [ocp-schedules-23,6652f699c29cfc2b,0ab16c8095030f33] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:46:28.510 INFO 21062 --- [ocp-schedules-23,6652f699c29cfc2b,0ab16c8095030f33] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:46:28.546 INFO 21062 --- [ocp-async-1,818159b0e02a31c3,fe969182204f6c6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:46:28.554 INFO 21062 --- [ocp-schedules-8,aa00b92843c8b98d,ff3296dffb83fa4a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:46:28.554 INFO 21062 --- [ocp-schedules-8,aa00b92843c8b98d,ff3296dffb83fa4a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:46:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:46:28.587510+08:00, lastProcessTime=2024-11-04T12:46:18.589511+08:00 2024-11-04 12:46:28.593 INFO 21062 --- [ocp-schedules-9,90ee687abd80b0a2,2889ceaf44885eb9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:46:28.602 INFO 21062 --- [ocp-schedules-5,a6193b7c1ec29c40,975586f4027158b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:46:28.608 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:46:28.608 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:46:28.612 WARN 21062 --- [ocp-async-9,779a2ba6e63f1636,1cf09bfa3ea63051] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:28.612 INFO 21062 --- [ocp-async-4,e2815ed66fd9780c,ca3e672ceed94efa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:28.616 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:46:28.616 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:46:28.616 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:46:28.622 INFO 21062 --- [ocp-schedules-19,f82e9c6b239985c3,8d0b43692f6b49dd] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:46:28.622 INFO 21062 --- [ocp-schedules-19,f82e9c6b239985c3,8d0b43692f6b49dd] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:46:28.626 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:46:28.626 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:46:28.626 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:46:28.629 INFO 21062 --- [ocp-schedules-22,22efec0c4cfea9b8,828310f493adb02d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:46:28.629 INFO 21062 --- [ocp-schedules-14,548afba507c2b14c,d3d9af67f8f6f9e8] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:46:28.661 INFO 21062 --- [ocp-schedules-15,1ded7c7fcd39ae97,6e3160552e3fee7d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:46:28.665 INFO 21062 --- [ocp-schedules-15,1ded7c7fcd39ae97,6e3160552e3fee7d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:46:28.665 INFO 21062 --- [ocp-schedules-15,1ded7c7fcd39ae97,6e3160552e3fee7d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:46:28.733 INFO 21062 --- [ocp-schedules-16,8c8b21a952c2658e,18f59a4b89f01d20] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:46:28.733 INFO 21062 --- [ocp-schedules-16,8c8b21a952c2658e,18f59a4b89f01d20] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:46:28.734 INFO 21062 --- [ocp-schedules-16,8c8b21a952c2658e,18f59a4b89f01d20] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:46:28.734 INFO 21062 --- [ocp-schedules-16,8c8b21a952c2658e,18f59a4b89f01d20] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:46:28.735 INFO 21062 --- [ocp-schedules-16,8c8b21a952c2658e,18f59a4b89f01d20] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:46:28.735 INFO 21062 --- [ocp-schedules-16,8c8b21a952c2658e,18f59a4b89f01d20] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:46:28.748 INFO 21062 --- [ocp-schedules-6,1cafa94964a14385,552b5a28d8c633bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:46:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:46:28.587510+08:00 2024-11-04 12:46:28.901 INFO 21062 --- [ocp-schedules-21,fda545b0165af682,63f4d18e3d03ff9d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:46:28.901 INFO 21062 --- [ocp-schedules-21,fda545b0165af682,63f4d18e3d03ff9d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:46:28.914 INFO 21062 --- [ocp-schedules-25,b5d9a25847c52b9a,02d835e64b8e3778] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:46:28.917 INFO 21062 --- [ocp-schedules-25,b5d9a25847c52b9a,02d835e64b8e3778] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:28.928 INFO 21062 --- [ocp-schedules-21,fda545b0165af682,63f4d18e3d03ff9d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:46:28.928 INFO 21062 --- [ocp-schedules-21,fda545b0165af682,63f4d18e3d03ff9d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:46:28.930 INFO 21062 --- [ocp-schedules-8,906a004fcbfe7f28,1c9ea84f0f8ea066] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:46:28.936 INFO 21062 --- [ocp-schedules-8,906a004fcbfe7f28,1c9ea84f0f8ea066] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:46:28.936 INFO 21062 --- [ocp-schedules-8,906a004fcbfe7f28,1c9ea84f0f8ea066] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:46:28.940 INFO 21062 --- [ocp-schedules-11,fbd20c24f401f0ad,1a761aaadea50a95] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:46:28.940 INFO 21062 --- [ocp-schedules-11,fbd20c24f401f0ad,1a761aaadea50a95] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:46:28.940 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:46:28.940 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:46:28.946 INFO 21062 --- [ocp-schedules-29,c3f3672c3ccfa724,1862666255872651] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:46:28.949 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:46:28.950 INFO 21062 --- [ocp-schedules-11,fbd20c24f401f0ad,1a761aaadea50a95] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:46:28.950 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:46:28.952 WARN 21062 --- [ocp-schedules-11,fbd20c24f401f0ad,1a761aaadea50a95] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:46:28.952 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:46:28.952 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:46:28.953 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:46:28.953 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:46:28.953 INFO 21062 --- [ocp-schedules-11,fbd20c24f401f0ad,1a761aaadea50a95] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:46:28.954 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:46:28.954 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:46:28.954 WARN 21062 --- [ocp-schedules-11,fbd20c24f401f0ad,1a761aaadea50a95] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:46:28.965 ERROR 21062 --- [ocp-schedules-11,fbd20c24f401f0ad,1a761aaadea50a95] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:46:28.975 ERROR 21062 --- [ocp-schedules-25,b5d9a25847c52b9a,02d835e64b8e3778] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:46:28.975 INFO 21062 --- [ocp-schedules-25,b5d9a25847c52b9a,02d835e64b8e3778] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:46:29.017 INFO 21062 --- [ocp-schedules-28,7c5b1bf1c04328f2,187b1f547e8ac54a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:46:29.017+08:00, clusters=[1] 2024-11-04 12:46:29.018 INFO 21062 --- [ocp-schedules-28,7c5b1bf1c04328f2,187b1f547e8ac54a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:46:29.109 INFO 21062 --- [ocp-schedules-7,263066dbd40f00a5,07f4b82a9f3fe6d3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:46:29.109+08:00, clusters=[1] 2024-11-04 12:46:29.113 INFO 21062 --- [ocp-schedules-7,263066dbd40f00a5,07f4b82a9f3fe6d3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:46:29.489 INFO 21062 --- [ocp-schedules-10,24f3d63719aa57a1,91f9f2e7cb890ae6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:46:29.489 INFO 21062 --- [ocp-schedules-10,647cbf37e7aec449,c06b2087daaf4007] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:46:29.493 INFO 21062 --- [ocp-schedules-10,647cbf37e7aec449,c06b2087daaf4007] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:29.520 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:46:29.521 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:46:29.631 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:29.632 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:29.789 INFO 21062 --- [ocp-schedules-15,0488f37bf39405d7,651a7971778c296b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:46:29.794 INFO 21062 --- [ocp-schedules-15,0488f37bf39405d7,651a7971778c296b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:29.877 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:46:29.879 ERROR 21062 --- [ocp-schedules-15,0488f37bf39405d7,651a7971778c296b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:46:29.879 INFO 21062 --- [ocp-schedules-15,0488f37bf39405d7,651a7971778c296b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:46:29.983 INFO 21062 --- [prometheus-minute-schedule-1,24069afcad4f1a0e,61e80f58f023fbe5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:46:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:46:30.612 WARN 21062 --- [ocp-async-10,9fea92f1f4fc0270,d9c338efbd2bb858] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:30.612 INFO 21062 --- [ocp-async-5,dbcdc282647eb208,4f4f636b0e556904] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:32.399 INFO 21062 --- [ocp-schedules-29,6ba5380dfae54553,a81081c03e346559] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:46:32.613 WARN 21062 --- [ocp-async-2,48a5f49529579d42,2fc2cda20dabc349] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:32.613 INFO 21062 --- [ocp-async-2,df1a529356c06581,5d7e998e83949924] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:32.632 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:32.633 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:32.983 INFO 21062 --- [prometheus-minute-schedule-1,06aa137e2f1ee1d5,ca3b9b2fd3e7b2a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:34.613 WARN 21062 --- [ocp-async-9,2914af76c857e030,79c12b61d18a2fd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:34.613 INFO 21062 --- [ocp-async-1,f40ca6a2b460906f,a5973453bb66432d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:35.633 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:35.634 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:35.983 INFO 21062 --- [prometheus-minute-schedule-1,8ca70b17abca6a21,22bca808dac47930] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:36.613 WARN 21062 --- [ocp-async-10,e8a5ac280cd1d879,83fde1c7d45bbde4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:36.613 INFO 21062 --- [ocp-async-4,8a3f303aa65ddff1,c10487b029c95aa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:38.504 INFO 21062 --- [ocp-schedules-12,40203af8920996c7,ed1177ebf8b5c30a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:46:38.504 INFO 21062 --- [ocp-schedules-1,1ce30ce8283e6502,8e60c9af76613f49] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:46:38.546 INFO 21062 --- [ocp-async-5,1c23a77ec48f77f1,309027c6a129af85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:46:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:46:38.588679+08:00, lastProcessTime=2024-11-04T12:46:28.587510+08:00 2024-11-04 12:46:38.613 WARN 21062 --- [ocp-async-8,9bc8ff659c86f253,bae9bbda28feff94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:38.613 INFO 21062 --- [ocp-async-2,a8d95b9c3e5a2a92,2b1e267a2fe0941e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:38.634 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:38.635 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:38.751 INFO 21062 --- [ocp-schedules-18,998245f2ec548257,edfe90616f4bc43d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:46:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:46:38.588679+08:00 2024-11-04 12:46:38.983 INFO 21062 --- [prometheus-minute-schedule-1,4d27bf9d35dab90d,3a6a0831f55d8de9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:40.002 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:46:40.002 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:46:40.613 WARN 21062 --- [ocp-async-7,436d97ec3672ecdf,54a3b2b11f8bc7c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:40.614 INFO 21062 --- [ocp-async-7,12349c1b61c50ee3,66cdaaa6b8370807] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:41.635 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:41.637 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:46:41.829 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:46:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=32ms, period=120000ms 2024-11-04 12:46:41.983 INFO 21062 --- [prometheus-minute-schedule-1,96781013a13e0ea1,8a39f0f292a88e16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:42.440 INFO 21062 --- [ocp-schedules-6,9b839bb90a21e2bc,40c880c015049fca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:46:42.614 WARN 21062 --- [ocp-async-6,165b6835ba69e22d,dc3520dfc6a25505] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:42.614 INFO 21062 --- [ocp-async-4,883be4f0e0002987,811802e8ce41c751] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:44.614 WARN 21062 --- [ocp-async-5,a2c422c953917c97,94e165e10f9fe2ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:44.614 INFO 21062 --- [ocp-async-8,48684d3e71752a5f,f57bf9b7523fa69f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:44.637 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:44.638 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:44.983 INFO 21062 --- [prometheus-minute-schedule-1,86397428b3a5a5f5,78a5bdd1420faac2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:46.614 WARN 21062 --- [ocp-async-1,5e27f99f11f3ad71,e23f718a264e08b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:46.614 INFO 21062 --- [ocp-async-2,72efd2ad27c7c776,4716b34612c80ab5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:47.638 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:47.639 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:47.983 INFO 21062 --- [prometheus-minute-schedule-1,348a98469d69efba,6ec0d2de40bc3229] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:48.546 INFO 21062 --- [ocp-async-2,b5865c35d32f3350,80db6432a2603097] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:46:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:46:48.589661+08:00, lastProcessTime=2024-11-04T12:46:38.588679+08:00 2024-11-04 12:46:48.614 WARN 21062 --- [ocp-async-6,111cf987e0252706,759e2d1dddbc075a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:48.614 INFO 21062 --- [ocp-async-6,45cc92f2e5eddc01,2c8109bfe97f0996] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:48.753 INFO 21062 --- [ocp-schedules-3,f522e8a7b121ad17,d3d124ae37c8c492] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:46:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:46:48.589661+08:00 2024-11-04 12:46:48.938 INFO 21062 --- [ocp-schedules-29,b64291cf1c7cd8d0,0dd023cf1396239c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:46:48.945 INFO 21062 --- [ocp-schedules-29,b64291cf1c7cd8d0,0dd023cf1396239c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:46:48.945 INFO 21062 --- [ocp-schedules-29,b64291cf1c7cd8d0,0dd023cf1396239c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:46:50.614 WARN 21062 --- [ocp-async-5,7af17b3adb65ff5a,2a0e4c88b0f8aa2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:50.614 INFO 21062 --- [ocp-async-9,767c4c060775bc32,265365979462078d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:50.639 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:50.640 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ee0a675fc35c5748,eac6386cc6da58cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:52.464 INFO 21062 --- [ocp-schedules-1,3ada96838e4eb1c2,a6da99cd96c5f44d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:46:52.614 WARN 21062 --- [ocp-async-1,10029595c29b1127,51644b13e52c8f63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:52.614 INFO 21062 --- [ocp-async-10,a4f76975eda36da5,5a9e38796b875620] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:53.640 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:53.641 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:53.983 INFO 21062 --- [prometheus-minute-schedule-1,16a03359e15e1575,2ef4c5826d0e915b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:54.615 WARN 21062 --- [ocp-async-7,1eb2fd4fcdf3252c,94be4d7639604d4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:54.615 INFO 21062 --- [ocp-async-3,4403c35a893bb47c,a90ae90c39bb10d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:56.615 WARN 21062 --- [ocp-async-5,6730ccffb5c5a914,5a9881d2b39df82f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:56.615 INFO 21062 --- [ocp-async-6,eccf948f7a697119,ec079a96ee1b4b7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:56.641 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:56.642 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:56.983 INFO 21062 --- [prometheus-minute-schedule-1,514466710083a756,f46e8b6b1c8d7025] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:46:58.500 INFO 21062 --- [ocp-schedules-5,3f7431d22c2bc8fb,24b09f20f2fa0af4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:46:58.500 INFO 21062 --- [ocp-schedules-7,105a21900484d4b5,f706f7bd7d87f4be] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:46:58.502 INFO 21062 --- [ocp-schedules-26,5913908ac52d00e6,a655abe695120ce5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:46:58.504 INFO 21062 --- [ocp-schedules-7,105a21900484d4b5,f706f7bd7d87f4be] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:58.504 INFO 21062 --- [ocp-schedules-5,3f7431d22c2bc8fb,24b09f20f2fa0af4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:58.504 INFO 21062 --- [ocp-schedules-12,98f5cbeeb8481150,b1848a6630b5537c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:46:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:46:58.512 INFO 21062 --- [ocp-schedules-22,4b702d91ec199671,d8224984f323d968] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:46:58.512 INFO 21062 --- [ocp-schedules-8,f0b7e7858fd18529,3dc2fc27e14e1e8c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:46:58.512 INFO 21062 --- [ocp-schedules-8,f0b7e7858fd18529,3dc2fc27e14e1e8c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:46:58.512 INFO 21062 --- [ocp-schedules-22,4b702d91ec199671,d8224984f323d968] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:46:58.512 INFO 21062 --- [ocp-schedules-11,86c53ab873bc9bda,05e717791c15dbcf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:46:58.512 INFO 21062 --- [ocp-schedules-11,86c53ab873bc9bda,05e717791c15dbcf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:46:58.547 INFO 21062 --- [ocp-async-6,516cfbf99ae953aa,9378e11f462e0dba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:46:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:46:58.588674+08:00, lastProcessTime=2024-11-04T12:46:48.589661+08:00 2024-11-04 12:46:58.615 INFO 21062 --- [ocp-async-1,81cf16dd289e78d9,37e26b0757e8eb52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:46:58.615 WARN 21062 --- [ocp-async-2,90a771c925da038d,4b9d9aba3882b2fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:46:58.631 INFO 21062 --- [ocp-schedules-21,94dcff28cb1823c5,4134038dfbc73b69] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:46:58.756 INFO 21062 --- [ocp-schedules-16,8f060d9797f5f865,f57993c1f1a8f8c3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:46:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:46:58.588674+08:00 2024-11-04 12:46:59.521 INFO 21062 --- [ocp-schedules-10,647cbf37e7aec449,c06b2087daaf4007] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:46:59.521 INFO 21062 --- [ocp-schedules-10,4e6a9854818245fe,d43822423a94aec3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:46:59.525 INFO 21062 --- [ocp-schedules-10,4e6a9854818245fe,d43822423a94aec3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:46:59.553 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:46:59.553 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:46:59.642 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:46:59.643 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:46:59.983 INFO 21062 --- [prometheus-minute-schedule-1,8700086ea4aa7277,277c926a32bade1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695620, interval=60 2024-11-04 12:47:00.615 INFO 21062 --- [ocp-async-7,5e72642bb3b75927,817bfab4af3b7f01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:00.615 WARN 21062 --- [ocp-async-4,06f3c9922d0442ac,a60a47f26942923c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:02.506 INFO 21062 --- [ocp-schedules-6,e31251dc2d2a5d09,363dc3373c9e3436] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:47:02.615 WARN 21062 --- [ocp-async-8,35651518d8c258af,5b52c7ed484ec3e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:02.615 INFO 21062 --- [ocp-async-5,bcff3842cf3a7f25,63c463a70a24b83d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:02.643 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:02.644 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:02.983 INFO 21062 --- [prometheus-minute-schedule-1,bcfaaf5364738e1b,3ac4c8e7806f39e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:04.615 WARN 21062 --- [ocp-async-2,6a9c9026067889c4,e61fb9827da944d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:04.615 INFO 21062 --- [ocp-async-9,d133c07e41946a88,3385371604cc0180] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:05.644 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:05.645 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:05.983 INFO 21062 --- [prometheus-minute-schedule-1,41c2987cbfbbf8b9,4f92a18e2f688c61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:06.616 WARN 21062 --- [ocp-async-4,6da5d4c2318a1f31,f413cc7e8661b0e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:06.616 INFO 21062 --- [ocp-async-1,501884b801fdd229,26edb7bc91a1b2c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:08.504 INFO 21062 --- [ocp-schedules-7,105a21900484d4b5,f706f7bd7d87f4be] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:47:08.505 INFO 21062 --- [ocp-schedules-5,3f7431d22c2bc8fb,24b09f20f2fa0af4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:47:08.547 INFO 21062 --- [ocp-async-1,917b3e99c006b175,795f2f7ab8329bad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:47:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:47:08.588677+08:00, lastProcessTime=2024-11-04T12:46:58.588674+08:00 2024-11-04 12:47:08.616 WARN 21062 --- [ocp-async-8,2fe24ecfe7f543f4,8d7b57ed4bf10039] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:08.616 INFO 21062 --- [ocp-async-8,4a8c0058d09b359c,4877ec20004701c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:08.645 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:08.646 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:08.759 INFO 21062 --- [ocp-schedules-12,88b0f77556fdbe29,eeec1858991337bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:47:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:47:08.588677+08:00 2024-11-04 12:47:08.949 INFO 21062 --- [ocp-schedules-7,cfe86bd79a76c6bb,24ae61010e03617f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:47:08.954 INFO 21062 --- [ocp-schedules-7,cfe86bd79a76c6bb,24ae61010e03617f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:47:08.954 INFO 21062 --- [ocp-schedules-7,cfe86bd79a76c6bb,24ae61010e03617f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:47:08.983 INFO 21062 --- [prometheus-minute-schedule-1,e56ebf21257ec853,01df9bf1c64b356a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:10.002 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:47:10.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:47:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:47:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:47:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:47:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:47:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:47:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:47:10.616 WARN 21062 --- [ocp-async-10,9ab365545ecae07d,e8678ef29bf49d3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:10.616 INFO 21062 --- [ocp-async-2,955c5475ddfa4aa1,5f2d62ae01239f20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:11.088 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695560, nodeCount=27, affectRows=27 2024-11-04 12:47:11.088 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695560, nodeCount=40, affectRows=40 2024-11-04 12:47:11.088 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695560, nodeCount=43, affectRows=43 2024-11-04 12:47:11.088 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695560, nodeCount=27, validCount=27 2024-11-04 12:47:11.088 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695560, nodeCount=40, validCount=40 2024-11-04 12:47:11.088 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695560, nodeCount=42, validCount=42 2024-11-04 12:47:11.088 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695560, nodeCount=64, affectRows=64 2024-11-04 12:47:11.089 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695560, nodeCount=57, affectRows=57 2024-11-04 12:47:11.089 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695560, nodeCount=64, validCount=64 2024-11-04 12:47:11.089 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695560, nodeCount=91, affectRows=91 2024-11-04 12:47:11.089 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695560, nodeCount=56, validCount=56 2024-11-04 12:47:11.089 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695560, nodeCount=90, validCount=90 2024-11-04 12:47:11.646 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:11.647 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:11.983 INFO 21062 --- [prometheus-minute-schedule-1,685c0327e2d7e108,fc376a6d5200b390] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:12.530 INFO 21062 --- [ocp-schedules-24,1609414959e23365,64ad38eff72b6ac7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:47:12.616 WARN 21062 --- [ocp-async-4,c77eec4ea1c7d3da,887f71e4340c667c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:12.616 INFO 21062 --- [ocp-async-3,c1cf5ad7d7c720b9,56010096a97b076e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:13.984 INFO 21062 --- [prometheus-cache-monitor1,1311ffe8819ac219,89b1dceaff069c98] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:47:13.985 INFO 21062 --- [prometheus-cache-monitor1,1311ffe8819ac219,89b1dceaff069c98] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:47:14.137 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:43:14.113, end=2024-11-04T12:45:14.134} 2024-11-04 12:47:14.141 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:47:14.142 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:47:14.616 WARN 21062 --- [ocp-async-7,ed978bbc814e1ec8,a0af9fbe96bc0eea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:14.616 INFO 21062 --- [ocp-async-6,33408b15dd6de162,39b812aa45db9c5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:14.647 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:14.648 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5f28285b2e7eca23,4fb42739485466e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:16.617 WARN 21062 --- [ocp-async-6,956cd0b82aa9cbe8,63e253d9d1754ae5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:16.617 INFO 21062 --- [ocp-async-8,10d6f36f65150875,85903b3ed12aa289] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:17.648 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:17.649 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8645a23e9a663b15,1d5a9c6d5666dbf8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:18.547 INFO 21062 --- [ocp-async-8,f52c9572e514a836,6a1b43917840a017] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:47:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:47:18.588167+08:00, lastProcessTime=2024-11-04T12:47:08.588677+08:00 2024-11-04 12:47:18.617 INFO 21062 --- [ocp-async-2,40224f8ffb3c7a11,938b20a59cf42ab9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:18.617 WARN 21062 --- [ocp-async-4,b2b9a3d1008b218c,947b15f9ddfd2007] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:18.762 INFO 21062 --- [ocp-schedules-11,d54feafff7f4bdd2,6d8fe888938ee4c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:47:18.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:47:18.588167+08:00 2024-11-04 12:47:20.617 INFO 21062 --- [ocp-async-2,9627149589479e41,dd12dc02c01703a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:20.617 WARN 21062 --- [ocp-async-7,0a3b872cce9921fc,cbbdf211fffa07f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:20.649 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:20.650 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e3255516f1e515e0,ae43f62374aa8c34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:22.573 INFO 21062 --- [ocp-schedules-2,3ed410e3a802ddc9,f1b3b8abe0fd3a02] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:47:22.617 INFO 21062 --- [ocp-async-2,ace6cf9e2e5bf45b,eb4143e48988d40c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:22.617 WARN 21062 --- [ocp-async-10,8084e4fed059923e,45e48d1d3d1f8ea0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:23.650 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:23.651 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4b7b317907b6c7fa,5218d87123e256b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:24.617 WARN 21062 --- [ocp-async-4,cdae57adbb6a1184,e4983d21f3197c5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:24.617 INFO 21062 --- [ocp-async-2,07d9fd52d81a54f9,3a78eb7b034c3f91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:26.617 WARN 21062 --- [ocp-async-7,0be5cdd072d0cf40,fadeaa23b900a91c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:26.617 INFO 21062 --- [ocp-async-2,f348bf49c104a970,f3c66b19fdad7fb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:26.651 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:26.652 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:26.983 INFO 21062 --- [prometheus-minute-schedule-1,0bd4c5dbf1e4e5f1,5ea4a697ab7ef204] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b34e7056117e3efa,cc96f2f5cb33adbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:47:28.500 INFO 21062 --- [ocp-schedules-13,f2f13b6b3e9846bd,fe9745a0ae86cdf9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:47:28.500 INFO 21062 --- [ocp-schedules-27,b4cc360cbcfc79fb,365bff364ebeef17] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:47:28.502 INFO 21062 --- [ocp-schedules-9,464d24796930cdb2,48a9ef68e10696bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:47:28.502 INFO 21062 --- [ocp-schedules-12,94c8e4ba537cc4e3,46dfa8f99c2659e6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:47:28.503 INFO 21062 --- [ocp-schedules-13,f2f13b6b3e9846bd,fe9745a0ae86cdf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:28.503 INFO 21062 --- [ocp-schedules-27,b4cc360cbcfc79fb,365bff364ebeef17] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:28.504 INFO 21062 --- [ocp-schedules-29,0c2577591426ad2e,c09ecd3f1c9ba135] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:47:28.504 INFO 21062 --- [ocp-schedules-23,639cb3eb7310b4cf,d1e58efbcf0a4bdf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:47:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:47:28.504 INFO 21062 --- [ocp-schedules-5,9a25cb3020b0dab8,139d5056949d80f7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 12:47:28.506 INFO 21062 --- [ocp-schedules-9,464d24796930cdb2,48a9ef68e10696bd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:47:28.508 INFO 21062 --- [ocp-schedules-5,9a25cb3020b0dab8,139d5056949d80f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:28.509 INFO 21062 --- [ocp-schedules-29,0c2577591426ad2e,c09ecd3f1c9ba135] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:47:28.513 INFO 21062 --- [ocp-schedules-20,7314184aed0053a8,34cc4830ea8398a3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:47:28.513 INFO 21062 --- [ocp-schedules-20,7314184aed0053a8,34cc4830ea8398a3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:47:28.513 INFO 21062 --- [ocp-schedules-3,cf8ddf6441564afa,5ca0f00284b4fe64] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:47:28.513 INFO 21062 --- [ocp-schedules-3,cf8ddf6441564afa,5ca0f00284b4fe64] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:47:28.513 INFO 21062 --- [ocp-schedules-26,62b36c2023653405,eb8055addcb107b4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:47:28.513 INFO 21062 --- [ocp-schedules-26,62b36c2023653405,eb8055addcb107b4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:47:28.528 INFO 21062 --- [ocp-schedules-18,842518ec3e9d1aa3,85e65642607e159f] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 12:47:28.528 INFO 21062 --- [ocp-schedules-18,842518ec3e9d1aa3,85e65642607e159f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 12:47:28.547 INFO 21062 --- [ocp-async-9,910680fd1de104ab,c9ba6b289dced21f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:47:28.576 INFO 21062 --- [pool-12-thread-13,152a29267f48a7ac,f87dc28edaf7de51] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 12:47:28.576 INFO 21062 --- [pool-12-thread-13,152a29267f48a7ac,f87dc28edaf7de51] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 12:47:28.577 INFO 21062 --- [pool-12-thread-16,152a29267f48a7ac,d1985f33fdd901b5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 12:47:28.578 INFO 21062 --- [pool-12-thread-14,152a29267f48a7ac,b1929ff251deee13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 12:47:28.578 INFO 21062 --- [pool-12-thread-15,152a29267f48a7ac,d5bda22dcfb98e91] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 12:47:28.581 INFO 21062 --- [pool-12-thread-13,152a29267f48a7ac,f87dc28edaf7de51] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 12:47:28.582 INFO 21062 --- [pool-12-thread-14,152a29267f48a7ac,b1929ff251deee13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 12:47:28.582 INFO 21062 --- [pool-12-thread-14,152a29267f48a7ac,b1929ff251deee13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 12:47:28.583 INFO 21062 --- [ForkJoinPool-11-worker-4,152a29267f48a7ac,567606268bb57f8a] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 12:47:28.583 INFO 21062 --- [pool-12-thread-16,152a29267f48a7ac,d1985f33fdd901b5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 12:47:28.584 INFO 21062 --- [ForkJoinPool-11-worker-13,152a29267f48a7ac,b514d8b0782ede66] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 12:47:28.584 INFO 21062 --- [ForkJoinPool-11-worker-13,152a29267f48a7ac,b514d8b0782ede66] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 12:47:28.584 INFO 21062 --- [ForkJoinPool-11-worker-13,152a29267f48a7ac,b514d8b0782ede66] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 12:47:28.584 INFO 21062 --- [pool-12-thread-15,152a29267f48a7ac,d5bda22dcfb98e91] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 12:47:28.587 INFO 21062 --- [pool-12-thread-13,152a29267f48a7ac,f87dc28edaf7de51] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 12:47:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:47:28.587845+08:00, lastProcessTime=2024-11-04T12:47:18.588167+08:00 2024-11-04 12:47:28.596 INFO 21062 --- [ocp-schedules-1,60b6d449904b94ae,89b1bb7186ea2214] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:47:28.601 INFO 21062 --- [ocp-schedules-21,ead922d90db6354e,49da1b631110b9b4] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 12:47:28.602 INFO 21062 --- [ocp-schedules-23,32e7a7cdf49f0fab,f53008067b523747] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:47:28.605 INFO 21062 --- [ocp-schedules-23,32e7a7cdf49f0fab,f53008067b523747] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:28.606 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:47:28.612 INFO 21062 --- [ocp-schedules-21,ead922d90db6354e,49da1b631110b9b4] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 12:47:28.617 INFO 21062 --- [ocp-async-8,f3c4101db06fa96c,8ace092dfac3a98b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:28.617 WARN 21062 --- [ocp-async-4,33b1223ec9731c15,28a805c578429590] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:28.627 INFO 21062 --- [ocp-schedules-3,40f80eecf9b89519,ae5656507e49db1f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:47:28.627 INFO 21062 --- [ocp-schedules-3,40f80eecf9b89519,ae5656507e49db1f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:47:28.633 INFO 21062 --- [ocp-schedules-26,ebc0e7149b4bae3e,1e400b2f14af847b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:47:28.667 INFO 21062 --- [ocp-schedules-18,d4397dce1f83e31d,ed1a6179c910b4bc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:47:28.676 INFO 21062 --- [ocp-schedules-18,d4397dce1f83e31d,ed1a6179c910b4bc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:47:28.676 INFO 21062 --- [ocp-schedules-18,d4397dce1f83e31d,ed1a6179c910b4bc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:47:28.680 INFO 21062 --- [ocp-schedules-14,ee3ecaf181732b4a,84f09797bc61697b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 12:47:28.680 INFO 21062 --- [ocp-schedules-14,ee3ecaf181732b4a,84f09797bc61697b] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 12:47:28.706 INFO 21062 --- [pool-22-thread-1,ee3ecaf181732b4a,0c9472061048ae00] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 12:47:28.734 WARN 21062 --- [ocp-schedules-14,ee3ecaf181732b4a,84f09797bc61697b] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:47:28.734 INFO 21062 --- [ocp-schedules-14,ee3ecaf181732b4a,84f09797bc61697b] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 12:47:28.737 INFO 21062 --- [ocp-schedules-28,48e43d095cc6be56,8301c21b8178c9ec] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:47:28.737 INFO 21062 --- [ocp-schedules-28,48e43d095cc6be56,8301c21b8178c9ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:47:28.738 INFO 21062 --- [ocp-schedules-28,48e43d095cc6be56,8301c21b8178c9ec] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:47:28.738 INFO 21062 --- [ocp-schedules-28,48e43d095cc6be56,8301c21b8178c9ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:47:28.739 INFO 21062 --- [ocp-schedules-28,48e43d095cc6be56,8301c21b8178c9ec] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:47:28.739 INFO 21062 --- [ocp-schedules-28,48e43d095cc6be56,8301c21b8178c9ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:47:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:47:28.587845+08:00 2024-11-04 12:47:28.765 INFO 21062 --- [ocp-schedules-2,e92bb3f91a7d4320,502efcc02d1ba19d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:47:28.929 INFO 21062 --- [ocp-schedules-16,812b5da76db13b85,88035e021b03c10a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:47:28.929 INFO 21062 --- [ocp-schedules-16,812b5da76db13b85,88035e021b03c10a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:47:28.954 INFO 21062 --- [ocp-schedules-16,812b5da76db13b85,88035e021b03c10a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:47:28.954 INFO 21062 --- [ocp-schedules-16,812b5da76db13b85,88035e021b03c10a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:47:28.957 INFO 21062 --- [ocp-schedules-11,ed1d17eef2a8dc9d,398370c46ed6c327] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:47:28.963 INFO 21062 --- [ocp-schedules-11,ed1d17eef2a8dc9d,398370c46ed6c327] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:47:28.963 INFO 21062 --- [ocp-schedules-11,ed1d17eef2a8dc9d,398370c46ed6c327] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:47:28.973 INFO 21062 --- [ocp-schedules-22,df3db05fc1ec4d49,2eb8b9040b7c4d5b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:47:28.973 INFO 21062 --- [ocp-schedules-8,573af108bdf91a5e,e40ccf3cbf79ffa5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:47:28.973 INFO 21062 --- [ocp-schedules-8,573af108bdf91a5e,e40ccf3cbf79ffa5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:47:28.973 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:47:28.973 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:47:28.983 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:47:28.983 INFO 21062 --- [ocp-schedules-8,573af108bdf91a5e,e40ccf3cbf79ffa5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:47:28.984 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:47:28.985 WARN 21062 --- [ocp-schedules-8,573af108bdf91a5e,e40ccf3cbf79ffa5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:47:28.985 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:47:28.986 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:47:28.986 INFO 21062 --- [ocp-schedules-8,573af108bdf91a5e,e40ccf3cbf79ffa5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:47:28.986 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:47:28.986 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:47:28.987 WARN 21062 --- [ocp-schedules-8,573af108bdf91a5e,e40ccf3cbf79ffa5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:47:28.987 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:47:28.987 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:47:28.994 ERROR 21062 --- [ocp-schedules-8,573af108bdf91a5e,e40ccf3cbf79ffa5] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:47:29.078 INFO 21062 --- [ocp-schedules-6,4778d036d477a230,f7d806ecd32a5ec4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:47:29.078+08:00, clusters=[1] 2024-11-04 12:47:29.079 INFO 21062 --- [ocp-schedules-6,4778d036d477a230,f7d806ecd32a5ec4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:47:29.138 INFO 21062 --- [ocp-schedules-7,17201d1ac53180c7,9536a8f440cef66e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:47:29.138+08:00, clusters=[1] 2024-11-04 12:47:29.142 INFO 21062 --- [ocp-schedules-7,17201d1ac53180c7,9536a8f440cef66e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:47:29.553 INFO 21062 --- [ocp-schedules-10,4e6a9854818245fe,d43822423a94aec3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:47:29.554 INFO 21062 --- [ocp-schedules-10,7386a05e0d120e73,6ab739e0396033c7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:47:29.557 INFO 21062 --- [ocp-schedules-10,7386a05e0d120e73,6ab739e0396033c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:29.585 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:47:29.585 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:47:29.652 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:29.653 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:29.881 INFO 21062 --- [ocp-schedules-17,3a385009e08a2022,6bf37d0db0e41fec] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:47:29.884 INFO 21062 --- [ocp-schedules-17,3a385009e08a2022,6bf37d0db0e41fec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:29.967 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:47:29.969 ERROR 21062 --- [ocp-schedules-17,3a385009e08a2022,6bf37d0db0e41fec] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:47:29.969 INFO 21062 --- [ocp-schedules-17,3a385009e08a2022,6bf37d0db0e41fec] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:47:29.983 INFO 21062 --- [prometheus-minute-schedule-1,88a1fe6d0cc7929e,deb9f5be330288a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:47:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:47:30.618 WARN 21062 --- [ocp-async-7,e1e36b9cd3c71eb0,9d968d9d09787b0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:30.618 INFO 21062 --- [ocp-async-1,e96b8fba0ba844dd,08a1b6ad88e62bec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:32.599 INFO 21062 --- [ocp-schedules-16,1be80f54e4d2b817,ea20950a9a5fc076] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:47:32.618 WARN 21062 --- [ocp-async-5,7942fc2c8c590de4,bd3af72830c2657c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:32.618 INFO 21062 --- [ocp-async-2,b0b88950c34719ea,60449ce245df3815] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:32.653 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:32.654 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3b40530304d1df40,878df708140ef550] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:34.618 WARN 21062 --- [ocp-async-9,2579eab1c529e11d,02f7088e65476aac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:34.618 INFO 21062 --- [ocp-async-4,93a6ff0c86ff5d12,fd1889b2efc1af41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:35.654 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:35.655 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5b74e6b5160dad73,e55ddf44d7b4d509] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:36.618 WARN 21062 --- [ocp-async-8,67baca40b2fb10f7,29c844d621d22d32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:36.618 INFO 21062 --- [ocp-async-7,0d1c098c3177d134,fe599ab35a97b0ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:38.393 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:47:38.393 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:47:38.399 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:47:38.400 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:47:38.400 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:47:38.400 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:47:38.406 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:47:38.406 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:47:38.406 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:47:38.406 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:47:38.447 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:47:38.456 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:47:38.456 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:47:38.456 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:47:38.456 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:47:38.456 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:47:38.464 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:47:38.504 INFO 21062 --- [ocp-schedules-27,b4cc360cbcfc79fb,365bff364ebeef17] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:47:38.504 INFO 21062 --- [ocp-schedules-13,f2f13b6b3e9846bd,fe9745a0ae86cdf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:47:38.508 INFO 21062 --- [ocp-schedules-5,9a25cb3020b0dab8,139d5056949d80f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 12:47:38.547 INFO 21062 --- [ocp-async-1,08af165c98de2e7a,b895e9f2bf28b08f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:47:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:47:38.588413+08:00, lastProcessTime=2024-11-04T12:47:28.587845+08:00 2024-11-04 12:47:38.606 INFO 21062 --- [ocp-schedules-23,32e7a7cdf49f0fab,f53008067b523747] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:47:38.618 WARN 21062 --- [ocp-async-10,2730f81c4e03170c,ce37fbe40e8b144d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:38.618 INFO 21062 --- [ocp-async-2,5a73e2c9d7365bd7,afdd18109f7bc5e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:38.655 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:38.656 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:47:38.588413+08:00 2024-11-04 12:47:38.767 INFO 21062 --- [ocp-schedules-25,766998b28d403ec9,e13770d3aa3a23cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:47:38.983 INFO 21062 --- [prometheus-minute-schedule-1,408d54d8a1efcc43,a7553fb15592e7a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:39.011 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:47:39.542 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.135607494S 2024-11-04 12:47:39.542 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:47:39.642 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:47:39.642 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:47:39.671 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:47:39.723 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:47:39.723 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:47:39.723 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:47:39.723 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:47:39.723 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:47:39.723 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:47:39.732 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:47:39.853 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] o.s.boot.SpringApplication : Started application in 1.453 seconds (JVM running for 1073.532) 2024-11-04 12:47:39.853 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.453640497S 2024-11-04 12:47:40.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:47:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:47:40.332 INFO 21062 --- [ocp-schedules-20,877f7f9103c13556,a5203a5e9383b172] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:47:40.619 WARN 21062 --- [ocp-async-4,426aacf8142bd466,6e0ddebc0299e5fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:40.619 INFO 21062 --- [ocp-async-3,791a06bd21f6c7c0,e233081d633592ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:41.656 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:41.657 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:41.983 INFO 21062 --- [prometheus-minute-schedule-1,fa1632ecfd5b4fdd,c68d85123513db5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:42.619 WARN 21062 --- [ocp-async-6,18a182d7bec86b7e,51fd8f7880ede320] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:42.619 INFO 21062 --- [ocp-async-7,6acee49022240655,9d8be691b3b1acf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:42.637 INFO 21062 --- [ocp-schedules-24,146b29d13a18d48c,c6828fac28a0e413] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:47:44.619 WARN 21062 --- [ocp-async-10,2475c6d92a47b349,de8b45e2efc4629d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:44.619 INFO 21062 --- [ocp-async-5,3c1c1f96ab93c481,dd5f58479407782c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:44.658 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:44.658 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:44.983 INFO 21062 --- [prometheus-minute-schedule-1,7b925f35e929c799,3296085c05a93012] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:46.619 WARN 21062 --- [ocp-async-2,c833d4a9b43689d1,a90481a2b4c524bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:46.619 INFO 21062 --- [ocp-async-4,3cedb35263278413,99ed2f7959d1cb83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:47.659 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:47.660 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:47.983 INFO 21062 --- [prometheus-minute-schedule-1,eabb52e5d1810df1,3e2f1767b6a7a7b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:48.548 INFO 21062 --- [ocp-async-4,4a0c962191039db9,08c18ee620b5417a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:47:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:47:48.589563+08:00, lastProcessTime=2024-11-04T12:47:38.588413+08:00 2024-11-04 12:47:48.619 WARN 21062 --- [ocp-async-6,8d3ac3a6a18a990e,3a382da61c898ae0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:48.619 INFO 21062 --- [ocp-async-1,b0f9d2c292fe6664,feaed7e9a8cac7f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:48.770 INFO 21062 --- [ocp-schedules-27,3cf38f96495a3f5a,06f2197e69ddf392] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:47:48.787 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:47:48.589563+08:00 2024-11-04 12:47:48.966 INFO 21062 --- [ocp-schedules-29,864e47bfe93c7637,e0550237312fcb07] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:47:48.974 INFO 21062 --- [ocp-schedules-29,864e47bfe93c7637,e0550237312fcb07] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:47:48.974 INFO 21062 --- [ocp-schedules-29,864e47bfe93c7637,e0550237312fcb07] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:47:50.619 WARN 21062 --- [ocp-async-9,6a49970261f23965,a5752f2d0752fd1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:50.619 INFO 21062 --- [ocp-async-10,1713871b92c5e519,7782afbc78565aa6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:50.660 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:50.661 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1b78d74d6e2eeccd,e3eb642919160b81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:52.620 WARN 21062 --- [ocp-async-2,5d5541665dd4ee35,f5dd6567445c879e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:52.620 INFO 21062 --- [ocp-async-8,94752fd6462c83df,aa97bcc7303bdc60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:52.661 INFO 21062 --- [ocp-schedules-27,3d0e24131f79b7a5,66188d59f10ba0ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:47:53.661 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:53.662 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a7be9bdf695bcb70,dfc1444de607e43b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:54.620 WARN 21062 --- [ocp-async-6,051ebc0025fb9c74,1bc84f952b4a5af3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:54.620 INFO 21062 --- [ocp-async-3,9ce12d3f9fe97feb,ce798a093b06ea94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:56.620 WARN 21062 --- [ocp-async-9,6c13488bb753545b,040314287bc667c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:56.620 INFO 21062 --- [ocp-async-1,25ec80f763124ffd,68c64e0657739cbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:56.662 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:56.662 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:56.983 INFO 21062 --- [prometheus-minute-schedule-1,9638a9c425d2b89c,3e2f56b5aed4f645] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:47:58.500 INFO 21062 --- [ocp-schedules-1,a32b077b1e3fe44e,4121e5c4621c434d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:47:58.500 INFO 21062 --- [ocp-schedules-8,ca3c0b9d67b22c11,f0a9f0f28ac6421b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:47:58.502 INFO 21062 --- [ocp-schedules-13,d858c0e3d85ccea3,577243e8c7e84468] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:47:58.504 INFO 21062 --- [ocp-schedules-1,a32b077b1e3fe44e,4121e5c4621c434d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:58.504 INFO 21062 --- [ocp-schedules-8,ca3c0b9d67b22c11,f0a9f0f28ac6421b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:58.504 INFO 21062 --- [ocp-schedules-23,d7d83198665ac4c2,db01f749fbbb53d8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:47:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:47:58.512 INFO 21062 --- [ocp-schedules-30,4b9203514c31cc2d,3fe9c05822524ecc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:47:58.512 INFO 21062 --- [ocp-schedules-30,4b9203514c31cc2d,3fe9c05822524ecc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:47:58.512 INFO 21062 --- [ocp-schedules-27,2f06aefe65a4d776,a6a7c8b44ad9e6fd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:47:58.512 INFO 21062 --- [ocp-schedules-27,2f06aefe65a4d776,a6a7c8b44ad9e6fd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:47:58.512 INFO 21062 --- [ocp-schedules-9,811a9bd5e10db4c9,a0a20471ff8d6f3e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:47:58.512 INFO 21062 --- [ocp-schedules-9,811a9bd5e10db4c9,a0a20471ff8d6f3e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:47:58.548 INFO 21062 --- [ocp-async-1,856dcc06a958e711,3ced19645afdf5a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:47:58.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:47:58.590783+08:00, lastProcessTime=2024-11-04T12:47:48.589563+08:00 2024-11-04 12:47:58.620 WARN 21062 --- [ocp-async-2,77cfae324e218667,5d7cfd8be7b288ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:47:58.620 INFO 21062 --- [ocp-async-10,1c8a702c0d85df6f,81477aeb1b03e4a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:47:58.635 INFO 21062 --- [ocp-schedules-19,9f6edf16805bc739,20f01f459d876a0e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:47:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:47:58.590783+08:00 2024-11-04 12:47:58.773 INFO 21062 --- [ocp-schedules-25,d2b0891caf47172a,51a3661e250db7a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:47:59.586 INFO 21062 --- [ocp-schedules-10,7386a05e0d120e73,6ab739e0396033c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:47:59.586 INFO 21062 --- [ocp-schedules-10,6c9600c536419e59,bd05a75952cff827] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:47:59.589 INFO 21062 --- [ocp-schedules-10,6c9600c536419e59,bd05a75952cff827] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:47:59.618 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:47:59.618 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:47:59.663 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:47:59.663 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:47:59.983 INFO 21062 --- [prometheus-minute-schedule-1,451dbec7eb58c6e7,cf83f6044fb3ce00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695680, interval=60 2024-11-04 12:48:00.620 INFO 21062 --- [ocp-async-10,824c9bc952291a3f,b66be4ce80f8e40b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:00.620 WARN 21062 --- [ocp-async-6,cff6321db8740132,b1ed895790fa7a36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:02.620 WARN 21062 --- [ocp-async-9,26f5ddd9eb20946b,437968b15a08f221] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:02.620 INFO 21062 --- [ocp-async-10,ce86301dcf015e20,dfd10cf3bed57c66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:02.664 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:02.664 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:02.704 INFO 21062 --- [ocp-schedules-5,9adc5face0839df3,f2cb5e0708a590bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:48:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f5c94df1972b7886,cfb3486f84ce6c53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:04.620 WARN 21062 --- [ocp-async-2,a5a1c07ed2255cb5,2feab417436e528f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:04.620 INFO 21062 --- [ocp-async-10,4df0b318b6f2a1e9,b5c471318515e9df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:05.664 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:05.665 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:05.983 INFO 21062 --- [prometheus-minute-schedule-1,fe26db38b93bcf37,bcf2219401da6098] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:06.621 INFO 21062 --- [ocp-async-10,c694ccc255451d3e,e63528d8b442aa24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:06.621 WARN 21062 --- [ocp-async-6,f8429fff3663a760,a74f63c9f45b9b2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:08.505 INFO 21062 --- [ocp-schedules-8,ca3c0b9d67b22c11,f0a9f0f28ac6421b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:48:08.505 INFO 21062 --- [ocp-schedules-1,a32b077b1e3fe44e,4121e5c4621c434d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:48:08.548 INFO 21062 --- [ocp-async-3,f5a5e459b507cec8,6373747580c16d47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:48:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:48:08.589424+08:00, lastProcessTime=2024-11-04T12:47:58.590783+08:00 2024-11-04 12:48:08.621 INFO 21062 --- [ocp-async-9,d6e6bc597c57f7ca,7b4f9753d2a794f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:08.621 WARN 21062 --- [ocp-async-1,ffad63fe7d228901,f61edffe6fc45439] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:08.665 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:08.666 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:48:08.589424+08:00 2024-11-04 12:48:08.775 INFO 21062 --- [ocp-schedules-26,ea5904ce6957e19b,dc5026f99f40cdf1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:48:08.977 INFO 21062 --- [ocp-schedules-8,b95f614c122e9b90,d76395ed9dbe12d2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:48:08.983 INFO 21062 --- [prometheus-minute-schedule-1,db4487d96c279d07,f53953b38d57c722] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:08.983 INFO 21062 --- [ocp-schedules-8,b95f614c122e9b90,d76395ed9dbe12d2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:48:08.983 INFO 21062 --- [ocp-schedules-8,b95f614c122e9b90,d76395ed9dbe12d2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:48:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:48:10.004 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:48:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:48:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:48:10.307 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:48:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:48:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:48:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:48:10.621 WARN 21062 --- [ocp-async-4,3ffb207cbca9d737,ee56fc75e7170373] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:10.621 INFO 21062 --- [ocp-async-2,8dd6f4405a6d1352,bc476ff13d7dfe83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:11.095 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=6, affectRows=6 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=6, affectRows=6 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=9, affectRows=9 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=3, affectRows=3 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=4, affectRows=4 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=4, affectRows=4 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=13, affectRows=13 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=10, affectRows=10 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=5, affectRows=5 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=14, affectRows=14 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=5, affectRows=5 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=11, affectRows=11 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=6, validCount=6 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=3, validCount=3 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=4, validCount=4 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=4, affectRows=4 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=4, validCount=4 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=6, validCount=6 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=5, validCount=5 2024-11-04 12:48:11.096 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=4, affectRows=4 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=9, validCount=9 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=5, validCount=5 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=4, validCount=4 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=10, validCount=10 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=14, validCount=14 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=4, validCount=4 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=13, validCount=13 2024-11-04 12:48:11.097 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=10, validCount=10 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=10, affectRows=10 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=13, affectRows=13 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=10, validCount=10 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=7, affectRows=7 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=7, validCount=7 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=13, validCount=13 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=30, affectRows=30 2024-11-04 12:48:11.099 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=23, affectRows=23 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=19, affectRows=19 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=30, validCount=30 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=36, affectRows=36 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=23, validCount=23 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=19, validCount=19 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=44, affectRows=44 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=36, validCount=36 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=44, validCount=44 2024-11-04 12:48:11.100 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695620, nodeCount=42, affectRows=42 2024-11-04 12:48:11.101 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695620, nodeCount=40, validCount=40 2024-11-04 12:48:11.666 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:11.668 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:11.983 INFO 21062 --- [prometheus-minute-schedule-1,013c588c31b5b834,6ebb35b9420f2bbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:12.621 WARN 21062 --- [ocp-async-7,defc4a18f7e1550f,e5a3a9f25367cb87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:12.621 INFO 21062 --- [ocp-async-6,18e9e309a283ac65,277d9af9674ed2c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:12.729 INFO 21062 --- [ocp-schedules-27,04fe780b807fd13c,0d3d6d690afecc38] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:48:13.882 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T12:42:13.851, end=2024-11-04T12:47:13.88} 2024-11-04 12:48:13.886 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T12:42:13.852, end=2024-11-04T12:47:13.88} 2024-11-04 12:48:13.894 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 12:48:13.894 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 12:48:13.894 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:48:13.894 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:48:13.984 INFO 21062 --- [prometheus-cache-monitor1,fb3386b0fd010194,f419f5e91b82b164] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:48:13.984 INFO 21062 --- [prometheus-cache-monitor1,fb3386b0fd010194,f419f5e91b82b164] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:48:14.621 WARN 21062 --- [ocp-async-3,272d4f88d8f41c3d,6ac08a1f030fa86b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:14.621 INFO 21062 --- [ocp-async-1,3a894b6bf67d9ad1,c36cc67d18f3c892] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:14.668 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:14.669 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:14.983 INFO 21062 --- [prometheus-minute-schedule-1,87fd7845cc91872f,bdcb05b9e7f74586] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:16.621 WARN 21062 --- [ocp-async-9,5228c0df1a81e8c1,dcd95238c68b07ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:16.621 INFO 21062 --- [ocp-async-4,81a13bd0344cd4db,e6ad2f589fa2cdcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:17.669 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:17.669 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ee98a6747ea63195,05fde4b8f732cb0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:18.548 INFO 21062 --- [ocp-async-4,0cc0193c1d83870e,50c1383b2ac3a65e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:48:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:48:18.588627+08:00, lastProcessTime=2024-11-04T12:48:08.589424+08:00 2024-11-04 12:48:18.622 WARN 21062 --- [ocp-async-7,060773faa53f9c83,fcbc142358a9fa2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:18.622 INFO 21062 --- [ocp-async-10,87675fe2778e9f2a,706c20840a733de3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:48:18.588627+08:00 2024-11-04 12:48:18.778 INFO 21062 --- [ocp-schedules-27,bbd49e7832bda3dd,a4fa78aefd463488] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:48:20.622 WARN 21062 --- [ocp-async-3,1280f30d363524fe,1614b9934e6cce38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:20.622 INFO 21062 --- [ocp-async-5,1733e83a9f044763,b8a33b3e2112154d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:20.670 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:20.670 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:20.983 INFO 21062 --- [prometheus-minute-schedule-1,39e23a61ccdcfd0a,4f2d0947c69eae0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:22.622 WARN 21062 --- [ocp-async-9,00ad21429c301024,837b0baf14d5b7cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:22.622 INFO 21062 --- [ocp-async-8,d126df440465b017,b48562ab40b3af0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:22.773 INFO 21062 --- [ocp-schedules-18,659c7701d259b03f,59617d1c94669d6c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:48:23.671 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:23.672 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8e45bbc7d5c7a974,a10a4c0c565f7a42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:24.622 WARN 21062 --- [ocp-async-4,c1292e964211e68c,01bd41ed24568629] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:24.622 INFO 21062 --- [ocp-async-7,d1d85245401cfa85,2a48d9ab4b98a154] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:26.622 WARN 21062 --- [ocp-async-10,c959aa12713d5373,de7ea67b837ce74d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:26.622 INFO 21062 --- [ocp-async-3,4aa2c5d47f0d3a3c,2f78ff683076ae2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:26.672 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:26.672 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:26.983 INFO 21062 --- [prometheus-minute-schedule-1,95703500ec549f2b,70c6a5869d536b00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:26.984 INFO 21062 --- [prometheus-hour-schedule-1,575abab7d738de3d,80d31bb5b521aece] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:48:28.498 INFO 21062 --- [ocp-schedules-30,589dd92b810fa6e2,7883e906597a09f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:48:28.499 INFO 21062 --- [ocp-schedules-16,b079905a0167312a,11f4a5615bfbeff5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:48:28.500 INFO 21062 --- [ocp-schedules-20,8c8f4cb1e3b226f6,dde2d00c2952d401] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:48:28.501 INFO 21062 --- [ocp-schedules-25,d5fae3c551698bfc,3948504ffb87d3a4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:48:28.502 INFO 21062 --- [ocp-schedules-30,589dd92b810fa6e2,7883e906597a09f3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:48:28.502 INFO 21062 --- [ocp-schedules-28,8d6916934c39008d,58aa926e87fc8b4e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:48:28.503 INFO 21062 --- [ocp-schedules-16,b079905a0167312a,11f4a5615bfbeff5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:28.503 INFO 21062 --- [ocp-schedules-20,8c8f4cb1e3b226f6,dde2d00c2952d401] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:28.504 INFO 21062 --- [ocp-schedules-29,16b7a27d3bd7b123,2bf7a196cce0b650] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:48:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:48:28.506 INFO 21062 --- [ocp-schedules-25,d5fae3c551698bfc,3948504ffb87d3a4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:48:28.511 INFO 21062 --- [ocp-schedules-1,a2105a47e1b0314c,0b24b90fa99f58bd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:48:28.511 INFO 21062 --- [ocp-schedules-13,6279aa9108f4d71b,c786b870397e0f4b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:48:28.511 INFO 21062 --- [ocp-schedules-1,a2105a47e1b0314c,0b24b90fa99f58bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:48:28.511 INFO 21062 --- [ocp-schedules-13,6279aa9108f4d71b,c786b870397e0f4b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:48:28.512 INFO 21062 --- [ocp-schedules-4,6d5b1b27a3ee9ed4,f072d02854bc389a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:48:28.512 INFO 21062 --- [ocp-schedules-4,6d5b1b27a3ee9ed4,f072d02854bc389a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:48:28.549 INFO 21062 --- [ocp-async-9,7e03a15b24a8a4fd,a35f9975d3170d83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:48:28.555 INFO 21062 --- [ocp-schedules-12,c1bc52b31a2e386f,9ac401d789bdd563] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:48:28.555 INFO 21062 --- [ocp-schedules-12,c1bc52b31a2e386f,9ac401d789bdd563] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:48:28.570 INFO 21062 --- [ocp-schedules-14,4cbbe998e3945e9d,54fcd10f9b3e74dd] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:48:28.570 INFO 21062 --- [ocp-schedules-14,4cbbe998e3945e9d,54fcd10f9b3e74dd] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:48:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:48:28.587621+08:00, lastProcessTime=2024-11-04T12:48:18.588627+08:00 2024-11-04 12:48:28.598 INFO 21062 --- [ocp-schedules-15,95959105e9cf9596,fe1629da3e27024b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:48:28.622 WARN 21062 --- [ocp-async-2,bbb2d5506017e63f,b7c84c5ce446d8e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:28.622 INFO 21062 --- [ocp-async-4,6f8e56e8f05ff80a,f30daa26502592d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:28.630 INFO 21062 --- [ocp-schedules-18,77443550878a2310,9afef027c0d8b501] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:48:28.630 INFO 21062 --- [ocp-schedules-18,77443550878a2310,9afef027c0d8b501] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:48:28.638 INFO 21062 --- [ocp-schedules-29,66086fbd2e063714,fc5d4c5bdd8c4f63] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:48:28.676 INFO 21062 --- [ocp-schedules-1,22a5275898ba2f14,ec6c2c0aad78b915] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:48:28.681 INFO 21062 --- [ocp-schedules-1,22a5275898ba2f14,ec6c2c0aad78b915] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:48:28.681 INFO 21062 --- [ocp-schedules-1,22a5275898ba2f14,ec6c2c0aad78b915] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:48:28.740 INFO 21062 --- [ocp-schedules-13,4312c4b0da93883a,a72faf6520be260f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:48:28.740 INFO 21062 --- [ocp-schedules-13,4312c4b0da93883a,a72faf6520be260f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:48:28.747 INFO 21062 --- [ocp-schedules-13,4312c4b0da93883a,a72faf6520be260f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:48:28.747 INFO 21062 --- [ocp-schedules-13,4312c4b0da93883a,a72faf6520be260f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:48:28.783 INFO 21062 --- [ocp-schedules-30,e73fe60bc99b208f,6761d7b5a23da5a0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:48:28.787 INFO 21062 --- [ocp-schedules-13,4312c4b0da93883a,a72faf6520be260f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:48:28.787 INFO 21062 --- [ocp-schedules-13,4312c4b0da93883a,a72faf6520be260f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:48:28.797 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:48:28.587621+08:00 2024-11-04 12:48:28.955 INFO 21062 --- [ocp-schedules-8,ba1d5eba5695e387,aa3629738ff91d31] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:48:28.955 INFO 21062 --- [ocp-schedules-8,ba1d5eba5695e387,aa3629738ff91d31] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:48:28.975 INFO 21062 --- [ocp-schedules-12,d092c86b5bf40d5e,b9a25a8386a9bfa2] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:48:28.978 INFO 21062 --- [ocp-schedules-12,d092c86b5bf40d5e,b9a25a8386a9bfa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:28.985 INFO 21062 --- [ocp-schedules-8,ba1d5eba5695e387,aa3629738ff91d31] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:48:28.985 INFO 21062 --- [ocp-schedules-8,ba1d5eba5695e387,aa3629738ff91d31] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:48:28.986 INFO 21062 --- [ocp-schedules-6,eaac685362803683,1b7962bb697492d7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:48:28.992 INFO 21062 --- [ocp-schedules-6,eaac685362803683,1b7962bb697492d7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:48:28.992 INFO 21062 --- [ocp-schedules-6,eaac685362803683,1b7962bb697492d7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:48:29.000 INFO 21062 --- [ocp-schedules-17,14bb5d2a3ddd1486,9c3a861e4b2c6553] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:48:29.002 INFO 21062 --- [ocp-schedules-3,3af9717ed1e7d31f,7ff1bd6572b51423] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:48:29.002 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:48:29.002 INFO 21062 --- [ocp-schedules-3,3af9717ed1e7d31f,7ff1bd6572b51423] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:48:29.002 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:48:29.012 INFO 21062 --- [ocp-schedules-3,3af9717ed1e7d31f,7ff1bd6572b51423] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:48:29.012 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:48:29.012 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:48:29.014 WARN 21062 --- [ocp-schedules-3,3af9717ed1e7d31f,7ff1bd6572b51423] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:48:29.014 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:48:29.014 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:48:29.015 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:48:29.015 INFO 21062 --- [ocp-schedules-3,3af9717ed1e7d31f,7ff1bd6572b51423] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:48:29.015 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:48:29.016 WARN 21062 --- [ocp-schedules-3,3af9717ed1e7d31f,7ff1bd6572b51423] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:48:29.016 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:48:29.016 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:48:29.028 ERROR 21062 --- [ocp-schedules-3,3af9717ed1e7d31f,7ff1bd6572b51423] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:48:29.038 ERROR 21062 --- [ocp-schedules-12,d092c86b5bf40d5e,b9a25a8386a9bfa2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:48:29.038 INFO 21062 --- [ocp-schedules-12,d092c86b5bf40d5e,b9a25a8386a9bfa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:48:29.106 INFO 21062 --- [ocp-schedules-26,61eacad4e16c46d6,bbb55b574d5940e6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:48:29.106+08:00, clusters=[1] 2024-11-04 12:48:29.109 INFO 21062 --- [ocp-schedules-26,61eacad4e16c46d6,bbb55b574d5940e6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:48:29.168 INFO 21062 --- [ocp-schedules-22,89a286e6f693c6ca,eaddb6fdf7339b45] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:48:29.168+08:00, clusters=[1] 2024-11-04 12:48:29.175 INFO 21062 --- [ocp-schedules-22,89a286e6f693c6ca,eaddb6fdf7339b45] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:48:29.619 INFO 21062 --- [ocp-schedules-10,6c9600c536419e59,bd05a75952cff827] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:48:29.619 INFO 21062 --- [ocp-schedules-10,acdd8cf89adf9786,c0ebbff368911f3b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:48:29.623 INFO 21062 --- [ocp-schedules-10,acdd8cf89adf9786,c0ebbff368911f3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:29.651 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:48:29.651 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:48:29.673 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:29.674 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:29.969 INFO 21062 --- [ocp-schedules-24,1e0ff98b1b5285f6,1145f9eb550d1b0a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:48:29.973 INFO 21062 --- [ocp-schedules-24,1e0ff98b1b5285f6,1145f9eb550d1b0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7c56e04395a83e74,99f1eb0546b2c627] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:30.060 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:48:30.062 ERROR 21062 --- [ocp-schedules-24,1e0ff98b1b5285f6,1145f9eb550d1b0a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:48:30.062 INFO 21062 --- [ocp-schedules-24,1e0ff98b1b5285f6,1145f9eb550d1b0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:48:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:48:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:48:30.623 INFO 21062 --- [ocp-async-7,9e05996a9150ba2c,819cb1bc5ce58cfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:30.623 WARN 21062 --- [ocp-async-10,6cbab6e33c5ab37c,570970542fe824ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:32.623 WARN 21062 --- [ocp-async-3,b1f2c0c878ac1043,b0080ac48a9b09ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:32.623 INFO 21062 --- [ocp-async-5,e69fc2b7352dd87d,6450945bdce5b2b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:32.674 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:32.675 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:32.797 INFO 21062 --- [ocp-schedules-1,f6cd580970123dce,12ce6e4441234291] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:48:32.983 INFO 21062 --- [prometheus-minute-schedule-1,839f24fba09a8fcd,348f6c47810fc6ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:34.623 WARN 21062 --- [ocp-async-8,384fd7b16479f583,22922a52db2ec5f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:34.623 INFO 21062 --- [ocp-async-2,c643c99e6e8a9788,3695b38b95f48be3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:35.675 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:35.676 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:35.983 INFO 21062 --- [prometheus-minute-schedule-1,8f85eff8ca8fca96,4ba5bc8ba5f4a23b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:36.623 WARN 21062 --- [ocp-async-4,77cc416c0a2e920d,95126788329c4cf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:36.624 INFO 21062 --- [ocp-async-4,0b8a55dbc46e8ddc,e968bac4274a7906] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:38.504 INFO 21062 --- [ocp-schedules-20,8c8f4cb1e3b226f6,dde2d00c2952d401] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:48:38.505 INFO 21062 --- [ocp-schedules-16,b079905a0167312a,11f4a5615bfbeff5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:48:38.549 INFO 21062 --- [ocp-async-7,70563491775a1aab,e8dc1c55832e856e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:48:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:48:38.588246+08:00, lastProcessTime=2024-11-04T12:48:28.587621+08:00 2024-11-04 12:48:38.606 INFO 21062 --- [ocp-schedules-22,4e5f487d8e466684,292f3f9347ec443e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:48:38.609 INFO 21062 --- [ocp-schedules-22,4e5f487d8e466684,292f3f9347ec443e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:38.610 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:48:38.624 WARN 21062 --- [ocp-async-9,6db346299a4b4b89,92917896ad8a381c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:38.624 INFO 21062 --- [ocp-async-9,86fd094d4bdf1be0,e8fe1f519e0437d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:38.677 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:38.677 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:48:38.588246+08:00 2024-11-04 12:48:38.785 INFO 21062 --- [ocp-schedules-21,0a952dd7213fdfa6,223cb599f7711055] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:48:38.983 INFO 21062 --- [prometheus-minute-schedule-1,bfc980903453c9fc,6e4ee514069ddcdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:48:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:48:40.624 WARN 21062 --- [ocp-async-6,48f7cfb5bb01b0c2,3f146f6764c883fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:40.624 INFO 21062 --- [ocp-async-2,21b10d800c66caa0,48c3eae4c1cd3246] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:41.677 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:41.678 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:48:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:48:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:48:41.830 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:48:41.830 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:48:41.856 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:48:41.856 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 12:48:41.983 INFO 21062 --- [prometheus-minute-schedule-1,fe2ef8ef083e4f9d,81f0f3b809b689fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:42.624 WARN 21062 --- [ocp-async-1,9471561f9ac24649,f2be0b6d2d8c8f3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:42.624 INFO 21062 --- [ocp-async-4,926aacdd3da71390,0ceeffdc4aa777c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:42.839 INFO 21062 --- [ocp-schedules-17,2711498ca2b685c0,3272b91c3fe10c9a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:48:44.624 WARN 21062 --- [ocp-async-3,b3c356962fc349ba,979968ce61035671] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:44.624 INFO 21062 --- [ocp-async-3,060cd928895bed2e,983d2f907987eabf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:44.678 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:44.679 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:44.983 INFO 21062 --- [prometheus-minute-schedule-1,0e8ef63fd051b311,93aa25cb5989fc2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:46.624 WARN 21062 --- [ocp-async-9,0c1ef2a25f45816e,0bbe79ffbc413cec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:46.624 INFO 21062 --- [ocp-async-6,31be7afe6ba6cced,7c47e87158add13e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:47.679 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:47.680 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:47.983 INFO 21062 --- [prometheus-minute-schedule-1,14214b99e4386f18,9af6ec26a465cc4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:48.549 INFO 21062 --- [ocp-async-6,86ad8f871836fd1d,56ddd2c8abf92e94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:48:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:48:48.587853+08:00, lastProcessTime=2024-11-04T12:48:38.588246+08:00 2024-11-04 12:48:48.610 INFO 21062 --- [ocp-schedules-22,4e5f487d8e466684,292f3f9347ec443e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:48:48.625 WARN 21062 --- [ocp-async-1,ef3c181c85daa288,3127f4a180b4a408] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:48.625 INFO 21062 --- [ocp-async-7,117c8cb735d361a8,ec8cc2f7e091c7fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:48:48.587853+08:00 2024-11-04 12:48:48.788 INFO 21062 --- [ocp-schedules-25,3f07ab1bbf37af97,7977f6105539238e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:48:48.994 INFO 21062 --- [ocp-schedules-21,baf4f3312d06806e,11d2d2caf4835518] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:48:49.000 INFO 21062 --- [ocp-schedules-21,baf4f3312d06806e,11d2d2caf4835518] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:48:49.001 INFO 21062 --- [ocp-schedules-21,baf4f3312d06806e,11d2d2caf4835518] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:48:50.625 WARN 21062 --- [ocp-async-5,25ce8dfecc2af82f,0d292750222cbff2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:50.625 INFO 21062 --- [ocp-async-5,35e9ae04515abbef,0a456b6b639bb22d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:50.681 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:50.681 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:50.983 INFO 21062 --- [prometheus-minute-schedule-1,0156513411d3e8d9,bf6263577486e645] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:52.625 WARN 21062 --- [ocp-async-5,c72a563ccefa1655,46de692452091e1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:52.625 INFO 21062 --- [ocp-async-5,3c12c5d0de15f3f6,9deb41d7b522a567] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:52.865 INFO 21062 --- [ocp-schedules-13,eaafbf031da00a09,8d10b83285bbc391] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:48:53.682 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:53.682 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:53.983 INFO 21062 --- [prometheus-minute-schedule-1,de9e5566c4c9bc4e,e61155a33e0d4ad2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:54.625 WARN 21062 --- [ocp-async-2,d9dc5a7e1eaf3a9e,5795dddf259f5d95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:54.625 INFO 21062 --- [ocp-async-5,885f1d5e95f4fe4b,00f17686dc37a417] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:56.625 WARN 21062 --- [ocp-async-5,75cc65d9f3cb4017,30ef695faafd5d19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:56.625 INFO 21062 --- [ocp-async-7,cd9d32df89f37ad8,0647ed1ba2d14931] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:56.683 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:56.683 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ed0b895ea61b3406,65e4c90cfcc7c44b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:48:58.500 INFO 21062 --- [ocp-schedules-7,3b22b247c0eb8469,3279707f31136952] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:48:58.500 INFO 21062 --- [ocp-schedules-28,900fc85df96fe76d,b638dc6f3d00e188] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:48:58.502 INFO 21062 --- [ocp-schedules-9,0a4a72aedbd0ff8d,4117e2b1cc3a450e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:48:58.504 INFO 21062 --- [ocp-schedules-7,3b22b247c0eb8469,3279707f31136952] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:58.504 INFO 21062 --- [ocp-schedules-17,2b9e5fbd69112869,5d2765eb1b3fa448] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:48:58.504 INFO 21062 --- [ocp-schedules-28,900fc85df96fe76d,b638dc6f3d00e188] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:48:58.513 INFO 21062 --- [ocp-schedules-11,15b04b813d50ad27,6c8cd0234ce7c290] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:48:58.513 INFO 21062 --- [ocp-schedules-1,1135f6826942b945,9726fbb56692bfb2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:48:58.514 INFO 21062 --- [ocp-schedules-1,1135f6826942b945,9726fbb56692bfb2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:48:58.514 INFO 21062 --- [ocp-schedules-11,15b04b813d50ad27,6c8cd0234ce7c290] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:48:58.514 INFO 21062 --- [ocp-schedules-5,425f62739969bcc3,09e0244283c07e53] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:48:58.514 INFO 21062 --- [ocp-schedules-5,425f62739969bcc3,09e0244283c07e53] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:48:58.549 INFO 21062 --- [ocp-async-7,19b5ba7c9d31049f,d1628763a9f80771] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:48:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:48:58.588247+08:00, lastProcessTime=2024-11-04T12:48:48.587853+08:00 2024-11-04 12:48:58.625 WARN 21062 --- [ocp-async-10,f1f2bfa11d83a23d,1458e0b49fc00b93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:48:58.625 INFO 21062 --- [ocp-async-9,891b9a1031a8baf8,fec2d92bfdb7b7af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:48:58.640 INFO 21062 --- [ocp-schedules-23,79e69e89b5715830,512591190f573c91] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:48:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:48:58.588247+08:00 2024-11-04 12:48:58.791 INFO 21062 --- [ocp-schedules-30,6d3a162ea426beb6,8774536486a0ce16] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:48:59.651 INFO 21062 --- [ocp-schedules-10,acdd8cf89adf9786,c0ebbff368911f3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:48:59.652 INFO 21062 --- [ocp-schedules-10,5823a36cfcc75fca,98960f1cce60ba0f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:48:59.655 INFO 21062 --- [ocp-schedules-10,5823a36cfcc75fca,98960f1cce60ba0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:48:59.683 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:48:59.684 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:48:59.685 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:48:59.685 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:48:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2ef2b09b8fedb0f9,4042ee492c75307a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695740, interval=60 2024-11-04 12:49:00.626 WARN 21062 --- [ocp-async-1,6f53e7c20156bda0,d245ffc5f9d3b142] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:00.626 INFO 21062 --- [ocp-async-9,95845e9c895ffdea,8c367cf1bc977a46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:02.626 WARN 21062 --- [ocp-async-5,86b64c6dc9539693,f5e0b74a5335d171] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:02.626 INFO 21062 --- [ocp-async-3,a0941e5fb8ccb5de,1d1d18d112d9eb5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:02.685 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:02.685 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:02.905 INFO 21062 --- [ocp-schedules-12,c521ae9c8e8959e0,b7640d39c66a0201] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:49:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7b41d98763f5cb59,7e4f7de2adf08f8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:04.626 WARN 21062 --- [ocp-async-7,bee562a1b1b4e9cb,7db15cba28a1660c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:04.629 INFO 21062 --- [ocp-async-10,a5267817287533aa,cef69f434c39d78b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:05.685 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:05.686 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ba85231649c9c142,4794a84b58e27c3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:06.626 WARN 21062 --- [ocp-async-2,48b7f5535a3fa032,8e130fbf82e14dd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:06.629 INFO 21062 --- [ocp-async-1,dd2b73d0442d8cb9,22d5b9049cc9a528] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:08.505 INFO 21062 --- [ocp-schedules-7,3b22b247c0eb8469,3279707f31136952] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:49:08.505 INFO 21062 --- [ocp-schedules-28,900fc85df96fe76d,b638dc6f3d00e188] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:49:08.549 INFO 21062 --- [ocp-async-1,5d67351f4ee82679,126eed8a086d0ca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:49:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:49:08.588452+08:00, lastProcessTime=2024-11-04T12:48:58.588247+08:00 2024-11-04 12:49:08.626 WARN 21062 --- [ocp-async-5,435d2ea88f196bc5,ed20361b221ba344] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:08.629 INFO 21062 --- [ocp-async-8,b908083050413d74,46bd588f2ce4b385] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:08.687 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:08.687 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:49:08.588452+08:00 2024-11-04 12:49:08.794 INFO 21062 --- [ocp-schedules-27,206f35cf864e4ad3,d3cd35183678b532] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:49:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b645d7fd13327348,f12b29a68f8c6cda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:09.003 INFO 21062 --- [ocp-schedules-18,883b5df1cf584ced,02dd8c6118c59cd0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:49:09.009 INFO 21062 --- [ocp-schedules-18,883b5df1cf584ced,02dd8c6118c59cd0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:49:09.009 INFO 21062 --- [ocp-schedules-18,883b5df1cf584ced,02dd8c6118c59cd0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:49:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:49:10.004 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:49:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:49:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:49:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:49:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:49:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:49:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:49:10.627 WARN 21062 --- [ocp-async-7,43fc6a6ef526df7f,bdece9ce83f3cd9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:10.629 INFO 21062 --- [ocp-async-6,d4a2fb1b851b0038,fcbd348be3c789b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:11.099 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695680, nodeCount=62, affectRows=62 2024-11-04 12:49:11.099 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695680, nodeCount=71, affectRows=71 2024-11-04 12:49:11.099 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695680, nodeCount=89, affectRows=89 2024-11-04 12:49:11.099 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695680, nodeCount=61, validCount=61 2024-11-04 12:49:11.100 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695680, nodeCount=71, validCount=71 2024-11-04 12:49:11.100 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695680, nodeCount=89, validCount=89 2024-11-04 12:49:11.100 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695680, nodeCount=100, affectRows=100 2024-11-04 12:49:11.100 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695680, nodeCount=98, validCount=98 2024-11-04 12:49:11.687 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:11.688 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:11.983 INFO 21062 --- [prometheus-minute-schedule-1,bf9c2a181bfa89ac,1f0d396992878e4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:12.627 WARN 21062 --- [ocp-async-2,7263937fb704f5ba,86f73cc3fc66c26d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:12.630 INFO 21062 --- [ocp-async-4,5d8032357be6c898,b2bfb0994c06cf33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:12.932 INFO 21062 --- [ocp-schedules-3,ac5e032485c091c1,abd86f56b7641dc0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:49:13.984 INFO 21062 --- [prometheus-cache-monitor1,92f446282625671b,e6be6fe6e011a9b9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:49:13.984 INFO 21062 --- [prometheus-cache-monitor1,92f446282625671b,e6be6fe6e011a9b9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:49:14.155 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:45:14.134, end=2024-11-04T12:47:14.152} 2024-11-04 12:49:14.159 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:49:14.159 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:49:14.627 WARN 21062 --- [ocp-async-1,4280d45bae921219,06952bb05491d4b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:14.630 INFO 21062 --- [ocp-async-5,67f28fa6ed10b21d,de2c916d14b672c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:14.688 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:14.689 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:14.983 INFO 21062 --- [prometheus-minute-schedule-1,22a68a1f0a77ea1a,88b7a2fbbeca87b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:16.627 WARN 21062 --- [ocp-async-8,0d73e4edbf69caa0,d1b965a9d09a2f58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:16.630 INFO 21062 --- [ocp-async-7,c19e6ae8e24eee43,84834553d4e79ec6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:17.689 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:17.691 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5a8ce7c26176bac2,b0407d5d045bd53e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:18.550 INFO 21062 --- [ocp-async-7,66ab4f1b4db4b55e,61abee5b5ba6ab89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:49:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:49:18.589951+08:00, lastProcessTime=2024-11-04T12:49:08.588452+08:00 2024-11-04 12:49:18.627 WARN 21062 --- [ocp-async-2,25881c6f4781c452,f504188d4b2c872f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:18.630 INFO 21062 --- [ocp-async-9,a4c14a02aace5919,d04518f7fe679064] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:49:18.589951+08:00 2024-11-04 12:49:18.796 INFO 21062 --- [ocp-schedules-6,b6ebee89a73e7820,6c67b6364db44692] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:49:20.627 WARN 21062 --- [ocp-async-1,de70d0bac6117a2a,20b4a669fc552ae1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:20.630 INFO 21062 --- [ocp-async-3,61d917c6967ccdb1,721a833bcf2dd740] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:20.691 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:20.692 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b412df7eb5bb992c,964a1bbad06127de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:22.628 WARN 21062 --- [ocp-async-8,5b2fbcc66931aca4,c949111e22f202a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:22.630 INFO 21062 --- [ocp-async-10,49d34966e59be029,38d1d15c5211d67c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:22.972 INFO 21062 --- [ocp-schedules-8,b20ac3d8c6f87a39,a772bdc80ba3d8bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:49:23.692 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:23.693 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8ac59c7cfbcbf86a,15010776aea8b13d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:24.628 WARN 21062 --- [ocp-async-6,5a9a5772fb7a1cab,793d7c07f42fe3c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:24.631 INFO 21062 --- [ocp-async-2,ca78f37390648650,dfe24f2839af7cd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:26.628 WARN 21062 --- [ocp-async-9,fe5650f770957d18,6887f11474a33386] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:26.631 INFO 21062 --- [ocp-async-1,bb5804c59f415e90,200536fd7d5d41b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:26.693 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:26.694 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:26.983 INFO 21062 --- [prometheus-minute-schedule-1,c300b737e1fbd0dd,112ac69cb9131e9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:26.984 INFO 21062 --- [prometheus-hour-schedule-1,dc7d704e76071b3f,192fac66c118b3fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:49:28.498 INFO 21062 --- [ocp-schedules-5,60154b93892f1870,98abc5c40e73c5a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:49:28.499 INFO 21062 --- [ocp-schedules-1,2c4528eb7c11730c,0e1b092f022253df] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:49:28.500 INFO 21062 --- [ocp-schedules-29,bc3ef1d4fc5ac9f9,1c8cdce4f8555a13] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:49:28.500 INFO 21062 --- [ocp-schedules-8,e0ad775d167c6016,17717aab8333f961] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:49:28.502 INFO 21062 --- [ocp-schedules-19,9ba9da348192d51e,2e0bbf3c38be1b4c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:49:28.503 INFO 21062 --- [ocp-schedules-5,60154b93892f1870,98abc5c40e73c5a3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:49:28.503 INFO 21062 --- [ocp-schedules-1,2c4528eb7c11730c,0e1b092f022253df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:28.504 INFO 21062 --- [ocp-schedules-29,bc3ef1d4fc5ac9f9,1c8cdce4f8555a13] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:49:28.505 INFO 21062 --- [ocp-schedules-11,dd27470e8447e386,d7b5280e43b33368] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:49:28.506 INFO 21062 --- [ocp-schedules-8,e0ad775d167c6016,17717aab8333f961] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:49:28.510 INFO 21062 --- [ocp-schedules-6,944ff6d2c915e587,e901c2a4474f3245] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:49:28.510 INFO 21062 --- [ocp-schedules-28,788143cd4be7408e,ad51b7b92bc50a4c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:49:28.511 INFO 21062 --- [ocp-schedules-6,944ff6d2c915e587,e901c2a4474f3245] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:49:28.511 INFO 21062 --- [ocp-schedules-28,788143cd4be7408e,ad51b7b92bc50a4c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:49:28.511 INFO 21062 --- [ocp-schedules-16,b2e2888ed282aedb,0f1e17de58a0e52b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:49:28.511 INFO 21062 --- [ocp-schedules-16,b2e2888ed282aedb,0f1e17de58a0e52b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:49:28.535 INFO 21062 --- [ocp-schedules-13,6f1361fb7335104a,1566134f0ee7f55e] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T12:49:28.535+08:00, tenantSize=0 2024-11-04 12:49:28.550 INFO 21062 --- [ocp-async-8,fb8ac087a5aa2e8a,55eef282d6b441cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:49:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:49:28.588523+08:00, lastProcessTime=2024-11-04T12:49:18.589951+08:00 2024-11-04 12:49:28.600 INFO 21062 --- [ocp-schedules-25,64b3bfec7ffb2d7d,13a62776aff7e3bd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:49:28.628 WARN 21062 --- [ocp-async-10,d3fd21fe7bff8576,64a6f1f8a56b2a0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:28.630 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:49:28.630 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:49:28.631 INFO 21062 --- [ocp-async-6,e0caff3fe3f86ccf,2673fe6dd9a13d28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:28.634 INFO 21062 --- [ocp-schedules-27,76eef44c02c32ad5,b7762f541142871f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:49:28.634 INFO 21062 --- [ocp-schedules-27,76eef44c02c32ad5,b7762f541142871f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:49:28.637 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:49:28.637 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:49:28.637 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:49:28.642 INFO 21062 --- [ocp-schedules-8,7ff33eb97ca9f426,fdca6635d83d2754] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:49:28.647 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:49:28.647 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:49:28.647 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:49:28.651 INFO 21062 --- [ocp-schedules-12,c56bb16b981608ca,a19404d264c18fd8] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:49:28.681 INFO 21062 --- [ocp-schedules-11,913bc7c2ee34cf5d,0a6ac45cd5e64ef9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:49:28.688 INFO 21062 --- [ocp-schedules-11,913bc7c2ee34cf5d,0a6ac45cd5e64ef9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:49:28.688 INFO 21062 --- [ocp-schedules-11,913bc7c2ee34cf5d,0a6ac45cd5e64ef9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:49:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:49:28.588523+08:00 2024-11-04 12:49:28.792 INFO 21062 --- [ocp-schedules-28,556475b88c58899f,3b4394a227f76f78] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:49:28.792 INFO 21062 --- [ocp-schedules-28,556475b88c58899f,3b4394a227f76f78] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:49:28.793 INFO 21062 --- [ocp-schedules-28,556475b88c58899f,3b4394a227f76f78] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:49:28.793 INFO 21062 --- [ocp-schedules-28,556475b88c58899f,3b4394a227f76f78] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:49:28.793 INFO 21062 --- [ocp-schedules-28,556475b88c58899f,3b4394a227f76f78] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:49:28.793 INFO 21062 --- [ocp-schedules-28,556475b88c58899f,3b4394a227f76f78] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:49:28.799 INFO 21062 --- [ocp-schedules-7,ecc9d983bfa19612,82af69498efbc7d1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:49:28.985 INFO 21062 --- [ocp-schedules-13,be00c397d8cb004c,85a33ae6fc3910ab] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:49:28.986 INFO 21062 --- [ocp-schedules-13,be00c397d8cb004c,85a33ae6fc3910ab] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:49:29.011 INFO 21062 --- [ocp-schedules-24,a689d98ca083d191,a2b7d3b497fce0c9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:49:29.014 INFO 21062 --- [ocp-schedules-13,be00c397d8cb004c,85a33ae6fc3910ab] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:49:29.014 INFO 21062 --- [ocp-schedules-13,be00c397d8cb004c,85a33ae6fc3910ab] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:49:29.017 INFO 21062 --- [ocp-schedules-24,a689d98ca083d191,a2b7d3b497fce0c9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:49:29.017 INFO 21062 --- [ocp-schedules-24,a689d98ca083d191,a2b7d3b497fce0c9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:49:29.026 INFO 21062 --- [ocp-schedules-30,519eb7c7f012c720,53911c888727cfb5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:49:29.039 INFO 21062 --- [ocp-schedules-23,b989c0d2a7d49f01,80a3c8c671761a23] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:49:29.039 INFO 21062 --- [ocp-schedules-23,b989c0d2a7d49f01,80a3c8c671761a23] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:49:29.039 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:49:29.039 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:49:29.049 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:49:29.050 INFO 21062 --- [ocp-schedules-23,b989c0d2a7d49f01,80a3c8c671761a23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:49:29.050 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:49:29.051 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:49:29.051 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:49:29.051 WARN 21062 --- [ocp-schedules-23,b989c0d2a7d49f01,80a3c8c671761a23] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:49:29.052 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:49:29.052 INFO 21062 --- [ocp-schedules-23,b989c0d2a7d49f01,80a3c8c671761a23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:49:29.052 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:49:29.053 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:49:29.053 WARN 21062 --- [ocp-schedules-23,b989c0d2a7d49f01,80a3c8c671761a23] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:49:29.053 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:49:29.059 ERROR 21062 --- [ocp-schedules-23,b989c0d2a7d49f01,80a3c8c671761a23] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2024-11-04 12:49:29.136 INFO 21062 --- [ocp-schedules-4,f3a7b37b3cc8fd8f,74f02d9d3b24faff] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:49:29.136+08:00, clusters=[1] 2024-11-04 12:49:29.137 INFO 21062 --- [ocp-schedules-4,f3a7b37b3cc8fd8f,74f02d9d3b24faff] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:49:29.200 INFO 21062 --- [ocp-schedules-20,19c166959bed9718,f08b30aa2ce28e80] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:49:29.200+08:00, clusters=[1] 2024-11-04 12:49:29.204 INFO 21062 --- [ocp-schedules-20,19c166959bed9718,f08b30aa2ce28e80] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:49:29.685 INFO 21062 --- [ocp-schedules-10,5823a36cfcc75fca,98960f1cce60ba0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:49:29.685 INFO 21062 --- [ocp-schedules-10,e71c4194573408fd,c1a113280516838a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:49:29.689 INFO 21062 --- [ocp-schedules-10,e71c4194573408fd,c1a113280516838a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:29.694 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:29.695 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:29.717 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:49:29.717 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:49:29.983 INFO 21062 --- [prometheus-minute-schedule-1,65ee0d7a5ede3a08,b295ed95064cc7d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:30.062 INFO 21062 --- [ocp-schedules-19,0e2ba307bb2e08da,9eae070da4ef13a7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:49:30.066 INFO 21062 --- [ocp-schedules-19,0e2ba307bb2e08da,9eae070da4ef13a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:30.151 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:49:30.152 ERROR 21062 --- [ocp-schedules-19,0e2ba307bb2e08da,9eae070da4ef13a7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:49:30.152 INFO 21062 --- [ocp-schedules-19,0e2ba307bb2e08da,9eae070da4ef13a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:49:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:49:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:49:30.629 WARN 21062 --- [ocp-async-2,fcd52649efa1b071,76f3ca6019292658] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:30.631 INFO 21062 --- [ocp-async-9,51ed377a7a58049e,287887a2fd0fdc70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:32.629 WARN 21062 --- [ocp-async-3,f63a53aa3c0f4182,cbfa621feacc03e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:32.631 INFO 21062 --- [ocp-async-1,e8540dbde1810af0,85aaa47d88f0d311] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:32.695 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:32.696 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d64883f0d8c5fd1b,d99a230d84d531d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:32.999 INFO 21062 --- [ocp-schedules-6,13d327f6bf0a5196,f58d5c64d4c5d3e1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:49:34.629 WARN 21062 --- [ocp-async-7,33575d3734783e13,ee35a050001aaae4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:34.632 INFO 21062 --- [ocp-async-10,c3d8b307b5c60853,b468d0b718efc95e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:35.696 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:35.697 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b6c5214526c37321,bd36482fcc153ba7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:36.629 WARN 21062 --- [ocp-async-6,80cbc6d9ec8bf3d9,4b04919bd7bc9bc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:36.632 INFO 21062 --- [ocp-async-2,ca7e962b1941712f,0c193434e057afb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:38.504 INFO 21062 --- [ocp-schedules-29,bc3ef1d4fc5ac9f9,1c8cdce4f8555a13] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:49:38.504 INFO 21062 --- [ocp-schedules-1,2c4528eb7c11730c,0e1b092f022253df] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:49:38.550 INFO 21062 --- [ocp-async-9,625a995763722507,906361b9e46ef92d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:49:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:49:38.588794+08:00, lastProcessTime=2024-11-04T12:49:28.588523+08:00 2024-11-04 12:49:38.629 WARN 21062 --- [ocp-async-8,aed220b425045dfc,e4785447d299bb63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:38.632 INFO 21062 --- [ocp-async-1,76cc35ab7923ba9c,d9d5cc225789059f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:38.697 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:38.698 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:49:38.588794+08:00 2024-11-04 12:49:38.801 INFO 21062 --- [ocp-schedules-24,247b5c66ca3d9063,0489aaaed848dfbe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:49:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ce38f9edb6cc899d,368cfe06c97d35f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:40.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:49:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:49:40.629 WARN 21062 --- [ocp-async-4,eb7b4dc8a9f6fa75,67877044efee54ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:40.632 INFO 21062 --- [ocp-async-10,9bcca828569083a2,3e853dbb04478afc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:41.699 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:41.700 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:41.983 INFO 21062 --- [prometheus-minute-schedule-1,23339c639b6c2b1b,9693c7be5a866e07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:42.629 WARN 21062 --- [ocp-async-5,5a8ace649720d315,8cf087eaa126639b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:42.632 INFO 21062 --- [ocp-async-2,c28dd0ef67185446,488c30c0db220247] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:43.040 INFO 21062 --- [ocp-schedules-15,cb6c22814071fc66,f1361d031a8d2d98] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:49:44.630 WARN 21062 --- [ocp-async-3,a83ae770848c2cb2,0677b368623d2c12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:44.632 INFO 21062 --- [ocp-async-8,199921db75366c35,e124d369b7db2bf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:44.700 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:44.702 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6ba6e2afd36c90ce,0a8c495c56f753d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:46.630 WARN 21062 --- [ocp-async-1,78d0be17e83e1d09,9d966034edf3465a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:46.633 INFO 21062 --- [ocp-async-4,22471cce1019f4e9,3a51839bd03789a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:47.702 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:47.703 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:47.983 INFO 21062 --- [prometheus-minute-schedule-1,405e71e64432a1aa,e8a32a7ebb224fe6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:48.550 INFO 21062 --- [ocp-async-4,491468330bad960d,bb7b384eefbc9372] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:49:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:49:48.590562+08:00, lastProcessTime=2024-11-04T12:49:38.588794+08:00 2024-11-04 12:49:48.614 INFO 21062 --- [ocp-schedules-30,49e48e07994e881e,2120576ab863c90b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:49:48.619 INFO 21062 --- [ocp-schedules-30,49e48e07994e881e,2120576ab863c90b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:48.619 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:49:48.630 WARN 21062 --- [ocp-async-5,6ea95c9d6c2de6f0,9e6aa21f84a182f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:48.633 INFO 21062 --- [ocp-async-9,2a3976af7a68d711,3f6a1e85612a1f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:49:48.590562+08:00 2024-11-04 12:49:48.804 INFO 21062 --- [ocp-schedules-24,6dfbef8fd4addaf4,39a7fdf27d145179] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:49:49.019 INFO 21062 --- [ocp-schedules-13,a1fb8a0206e0fa70,778458f7de15174b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:49:49.027 INFO 21062 --- [ocp-schedules-13,a1fb8a0206e0fa70,778458f7de15174b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:49:49.027 INFO 21062 --- [ocp-schedules-13,a1fb8a0206e0fa70,778458f7de15174b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:49:50.630 WARN 21062 --- [ocp-async-9,d73dbfd018a05582,bac237d8ab1fc274] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:50.633 INFO 21062 --- [ocp-async-7,3b0c3383f42168a3,bc7b1fe6557ea020] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:50.703 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:50.704 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:50.983 INFO 21062 --- [prometheus-minute-schedule-1,fe42ce0da7fadee9,50b5c0c9d1bd7d46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:52.630 WARN 21062 --- [ocp-async-7,5105c8904a63633f,497181b8787714c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:52.633 INFO 21062 --- [ocp-async-1,a9edd6e157f0b444,6287b1b43436be6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:53.067 INFO 21062 --- [ocp-schedules-6,c6ba97ae68592d14,d3d49b7a02c09554] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:49:53.704 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:53.705 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d8a41ae332622194,8a67f528fddadaed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:54.630 WARN 21062 --- [ocp-async-1,f8ff6eaa7b004109,23356d9115cd4fc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:54.633 INFO 21062 --- [ocp-async-10,b0f50dc5236ac6e3,2ab57eed33bddb84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:56.631 WARN 21062 --- [ocp-async-10,9bb05eda04cee4d7,d821ddac05a6a311] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:56.633 INFO 21062 --- [ocp-async-9,69fe6a06ba4cc2da,d14a20235d469167] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:56.705 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:56.706 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:56.983 INFO 21062 --- [prometheus-minute-schedule-1,9252f9374bca08c1,bc1a4d253bcebafd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:49:58.500 INFO 21062 --- [ocp-schedules-2,c26be1d45a49f784,8371b96f458c7e56] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:49:58.500 INFO 21062 --- [ocp-schedules-18,1e2155ff57374870,0ff80f4555350ef3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:49:58.502 INFO 21062 --- [ocp-schedules-4,88f2ab309d60eb43,bd2b268c9980b487] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:49:58.504 INFO 21062 --- [ocp-schedules-18,1e2155ff57374870,0ff80f4555350ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:58.504 INFO 21062 --- [ocp-schedules-2,c26be1d45a49f784,8371b96f458c7e56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:49:58.505 INFO 21062 --- [ocp-schedules-26,d6d6d87d48389938,d42ec3863d76aeb5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:49:58.511 INFO 21062 --- [ocp-schedules-28,0efe16e8af1d44a0,bae08743310f9eb3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:49:58.511 INFO 21062 --- [ocp-schedules-9,f2574422e3126195,ec801ce171109ab3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:49:58.511 INFO 21062 --- [ocp-schedules-3,133713b32046e6dd,ac5706be94d6661c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:49:58.511 INFO 21062 --- [ocp-schedules-3,133713b32046e6dd,ac5706be94d6661c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:49:58.511 INFO 21062 --- [ocp-schedules-28,0efe16e8af1d44a0,bae08743310f9eb3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:49:58.511 INFO 21062 --- [ocp-schedules-9,f2574422e3126195,ec801ce171109ab3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:49:58.550 INFO 21062 --- [ocp-async-9,b7d5b0ec23ce1e77,1e39ff02a43748c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:49:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:49:58.588444+08:00, lastProcessTime=2024-11-04T12:49:48.590562+08:00 2024-11-04 12:49:58.620 INFO 21062 --- [ocp-schedules-30,49e48e07994e881e,2120576ab863c90b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:49:58.631 WARN 21062 --- [ocp-async-7,9fa9b9b40a049b54,598a68cbfa16cced] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:49:58.634 INFO 21062 --- [ocp-async-4,fea324f028b1796d,e5c23d7d3027687b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:49:58.645 INFO 21062 --- [ocp-schedules-21,6727b15abdbb5649,c7cff6789257e3fe] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:49:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:49:58.588444+08:00 2024-11-04 12:49:58.807 INFO 21062 --- [ocp-schedules-16,044b7f435e3e1b5d,b13cbd9f4bdaa480] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:49:59.706 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:49:59.707 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:49:59.717 INFO 21062 --- [ocp-schedules-10,e71c4194573408fd,c1a113280516838a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:49:59.717 INFO 21062 --- [ocp-schedules-10,87af88d9aa1e742a,908470627b31e500] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:49:59.721 INFO 21062 --- [ocp-schedules-10,87af88d9aa1e742a,908470627b31e500] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:49:59.750 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:49:59.750 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:49:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e0bad2f880a9bee2,68c251996265613c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695800, interval=60 2024-11-04 12:50:00.106 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,12eef08bcd832c0d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:45:00.0, nextRun=2024-11-04 12:50:00.0, currentDate=Mon Nov 04 12:50:00 CST 2024 2024-11-04 12:50:00.107 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,12eef08bcd832c0d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 12:45:00.0, nextRunTime=2024-11-04 12:50:00.0} 2024-11-04 12:50:00.134 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,12eef08bcd832c0d] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 12:50:00.138 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,12eef08bcd832c0d] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 12:50:00.138 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,12eef08bcd832c0d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 12:50:00.631 WARN 21062 --- [ocp-async-1,6774929c34f604fd,0858e271316b014f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:00.634 INFO 21062 --- [ocp-async-2,6c811754cf2953c9,0fa07f98218e1222] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:02.631 WARN 21062 --- [ocp-async-10,8a8fe1db12250f93,771209f1c58bb0e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:02.634 INFO 21062 --- [ocp-async-3,488cc2d8212f99fe,d781b4575a8f7bd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:02.707 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:02.708 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:02.983 INFO 21062 --- [prometheus-minute-schedule-1,6c562c526610d038,32505bf4f547f18a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:03.107 INFO 21062 --- [ocp-schedules-1,0d0e635c47410734,77fbf163d4a3463f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:50:04.209 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2057db3087a20eac] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:40:05.0, nextRun=2024-11-04 12:50:04.0, currentDate=Mon Nov 04 12:50:04 CST 2024 2024-11-04 12:50:04.209 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2057db3087a20eac] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 12:40:05.0, nextRunTime=2024-11-04 12:50:04.0} 2024-11-04 12:50:04.215 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2057db3087a20eac] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 12:50:04.215 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2057db3087a20eac] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T12:50:04.215+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 12:50:04.631 WARN 21062 --- [ocp-async-9,b1eb8dc972251e84,ecb6f837e338ae19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:04.634 INFO 21062 --- [ocp-async-7,2e98ec574ef8b9e5,c7610511890a866d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:04.922 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,fc4ed6e78082c7a1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 70, operation:EXECUTE, state: RUNNING 2024-11-04 12:50:04.942 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=70, context=Context{parallelIdx=-1, stringMap={task_instance_id=4, task_operation=execute, latest_execution_start_time=2024-11-04T12:50:04.913+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=70}, listMap={}}, executor=10.46.128.24 2024-11-04 12:50:04.947 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 12:50:04.947 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 12:50:04.948 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.949 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:50:04.950 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.950 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:04.951 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.951 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 12:50:04.968 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.968 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:04.969 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.969 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 12:50:04.982 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.983 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:04.983 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.984 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 12:50:04.998 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:04.999 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.000 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.000 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 12:50:05.014 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.014 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.015 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.015 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 12:50:05.021 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.021 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.022 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.022 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 12:50:05.027 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.027 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.028 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.028 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 12:50:05.033 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.033 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.034 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.034 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 12:50:05.041 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.041 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.042 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.042 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 12:50:05.048 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.048 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.049 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.050 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 12:50:05.056 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.056 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.057 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.057 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 12:50:05.067 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.067 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.068 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.068 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 12:50:05.081 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.081 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.082 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.082 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 12:50:05.087 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.087 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.088 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.088 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 12:50:05.094 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.095 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.096 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.096 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 12:50:05.103 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.103 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.104 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.105 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 12:50:05.109 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.109 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.110 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.110 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 12:50:05.114 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.115 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.115 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.116 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 12:50:05.120 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.120 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.121 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.121 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 12:50:05.125 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.126 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.126 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.127 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 12:50:05.130 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.131 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.131 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.132 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 12:50:05.136 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.137 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.138 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.138 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 12:50:05.143 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.144 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.144 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.145 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 12:50:05.149 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.150 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.150 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.151 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 12:50:05.156 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.156 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.157 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.157 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 12:50:05.161 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.161 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.162 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.162 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 12:50:05.167 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.167 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.167 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.168 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 12:50:05.172 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.173 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.174 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.174 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 12:50:05.179 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.179 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.180 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.180 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 12:50:05.184 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.185 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.186 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.186 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 12:50:05.190 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.190 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.191 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.191 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 12:50:05.196 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.196 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.197 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.197 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 12:50:05.201 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.202 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.202 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.203 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 12:50:05.207 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.208 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.208 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.209 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 12:50:05.216 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.217 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.217 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.218 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 12:50:05.225 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.225 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.226 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.226 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 12:50:05.231 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.231 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.232 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.232 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 12:50:05.238 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.239 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.239 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.239 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 12:50:05.243 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.243 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.244 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.244 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 12:50:05.249 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.249 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.250 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.250 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 12:50:05.254 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.254 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.255 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.255 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 12:50:05.258 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.259 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.259 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.260 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 12:50:05.268 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.269 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.269 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.270 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 12:50:05.290 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.290 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.291 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.291 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 12:50:05.297 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.298 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.298 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.299 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 12:50:05.305 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.305 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.306 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.306 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 12:50:05.312 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.313 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.313 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.314 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 12:50:05.318 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.318 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.319 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.319 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 12:50:05.323 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.324 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.324 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.325 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 12:50:05.329 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 12:50:05.329 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 12:50:05.330 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.331 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:50:05.332 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.333 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.334 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.334 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 12:50:05.340 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.340 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:50:05.341 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:50:05.342 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 12:50:05.346 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 12:50:05.356 INFO 21062 --- [subtask-executor3,dd193f8f67a5a423,a2f1c1656caaca18] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 70, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 12:50:05.708 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:05.709 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:05.983 INFO 21062 --- [prometheus-minute-schedule-1,4f30f45d29f2dd2e,b53b0ce3af7f81ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:06.228 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4, status:SUCCESS 2024-11-04 12:50:06.230 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4, status:SUCCESS 2024-11-04 12:50:06.631 WARN 21062 --- [ocp-async-4,21caafb30d0a856e,2f40b4accf9f8b5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:06.634 INFO 21062 --- [ocp-async-1,7bb2eca581e6d482,202280ca0e3fe17d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:08.505 INFO 21062 --- [ocp-schedules-2,c26be1d45a49f784,8371b96f458c7e56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:50:08.505 INFO 21062 --- [ocp-schedules-18,1e2155ff57374870,0ff80f4555350ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:50:08.551 INFO 21062 --- [ocp-async-1,70540032518bf747,00e9cdeb8f58b05e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:50:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:50:08.589295+08:00, lastProcessTime=2024-11-04T12:49:58.588444+08:00 2024-11-04 12:50:08.632 WARN 21062 --- [ocp-async-10,3cdecbbc25d94c28,06eccc11f4299506] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:08.635 INFO 21062 --- [ocp-async-8,2d1939b6610d55c6,9ae43822503d4580] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:08.709 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:08.710 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:08.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:50:08.589295+08:00 2024-11-04 12:50:08.809 INFO 21062 --- [ocp-schedules-1,362a677dfc33a968,ac382b16f4803291] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:50:08.983 INFO 21062 --- [prometheus-minute-schedule-1,90fcd5ac8e85eaaf,f9f4ad22379a9e14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:09.029 INFO 21062 --- [ocp-schedules-2,8d0e6674f17454ef,00df2c941228a42b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:50:09.035 INFO 21062 --- [ocp-schedules-2,8d0e6674f17454ef,00df2c941228a42b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:50:09.035 INFO 21062 --- [ocp-schedules-2,8d0e6674f17454ef,00df2c941228a42b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:50:10.002 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:50:10.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:50:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:50:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:50:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:50:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:50:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:50:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:50:10.632 WARN 21062 --- [ocp-async-9,09d0004c9f922aa3,ad19e2543a9bfbb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:10.635 INFO 21062 --- [ocp-async-6,91a5bc372d3e965b,ff9be95c019b1328] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:11.102 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695740, nodeCount=56, affectRows=56 2024-11-04 12:50:11.102 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695740, nodeCount=35, affectRows=35 2024-11-04 12:50:11.103 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695740, nodeCount=61, affectRows=61 2024-11-04 12:50:11.103 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695740, nodeCount=88, affectRows=88 2024-11-04 12:50:11.103 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695740, nodeCount=35, validCount=35 2024-11-04 12:50:11.103 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695740, nodeCount=56, validCount=56 2024-11-04 12:50:11.103 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695740, nodeCount=60, validCount=60 2024-11-04 12:50:11.103 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695740, nodeCount=82, affectRows=82 2024-11-04 12:50:11.103 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695740, nodeCount=87, validCount=87 2024-11-04 12:50:11.104 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695740, nodeCount=81, validCount=81 2024-11-04 12:50:11.710 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:11.711 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:11.983 INFO 21062 --- [prometheus-minute-schedule-1,fca43cc113b57d53,e9484204058fb796] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:12.632 WARN 21062 --- [ocp-async-4,ceb6572d674e9282,5233597177b76750] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:12.635 INFO 21062 --- [ocp-async-5,a7cf08f55e887a3c,cbea58de2fddd91c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:13.133 INFO 21062 --- [ocp-schedules-8,91ed136cd1946b3b,f7273ce167ab3a99] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:50:13.984 INFO 21062 --- [prometheus-cache-monitor1,37e921990bb53bfc,2d7768da466376d5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:50:13.984 INFO 21062 --- [prometheus-cache-monitor1,37e921990bb53bfc,2d7768da466376d5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:50:14.632 WARN 21062 --- [ocp-async-1,d61941a7e5140173,d373821c505b8346] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:14.635 INFO 21062 --- [ocp-async-10,499947e5ae77b2c0,ea88ef72d92933a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:14.711 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:14.712 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a47a9c330e6603ca,f0d5baf85a5af42f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:16.632 WARN 21062 --- [ocp-async-8,2e22be71f25d6050,73981d6c7536e676] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:16.635 INFO 21062 --- [ocp-async-9,c3cdd2bedd61bd67,bfc8c2594b4d3663] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:17.712 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:17.713 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d5167c1e84cef835,7ef4d2f8e63a6945] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:18.551 INFO 21062 --- [ocp-async-9,d201a0b7ac8e2e00,2b5a1595082449ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:50:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:50:18.588664+08:00, lastProcessTime=2024-11-04T12:50:08.589295+08:00 2024-11-04 12:50:18.632 WARN 21062 --- [ocp-async-4,153f13b39ea1945c,c06842a3f6b1f502] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:18.635 INFO 21062 --- [ocp-async-2,6eee75b024b2da91,31e8ffcb940cac51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:50:18.588664+08:00 2024-11-04 12:50:18.812 INFO 21062 --- [ocp-schedules-12,5c88dd91d2513100,d8bc79f24efd91db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:50:20.633 WARN 21062 --- [ocp-async-1,90a8b1366463c3ec,35959ac91cb6ec1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:20.636 INFO 21062 --- [ocp-async-3,dcbc3c616d254b07,000c214507f0242f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:20.713 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:20.714 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a70737d1e2862c31,f1d931d6924aff52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:22.633 WARN 21062 --- [ocp-async-8,611a2b507361b4aa,0ad04515c9504171] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:22.636 INFO 21062 --- [ocp-async-7,1db900ab3287f400,9a2f1541f97f5913] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:23.174 INFO 21062 --- [ocp-schedules-16,cc15d5ff24e847f1,d7f0966ff8e1886b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:50:23.714 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:23.715 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:23.983 INFO 21062 --- [prometheus-minute-schedule-1,1658b75ce47956c2,d31d728c29d4edec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:24.633 WARN 21062 --- [ocp-async-6,1f3f250016ca67c3,9481cdbb8b22484a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:24.636 INFO 21062 --- [ocp-async-4,b5233b9e410a2432,02c2d80face07852] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:26.633 WARN 21062 --- [ocp-async-2,ade834aaab78d358,c0e20557152078c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:26.636 INFO 21062 --- [ocp-async-1,ce1fdc2db05c8baa,2e095664e9cc33a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:26.715 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:26.715 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:26.983 INFO 21062 --- [prometheus-minute-schedule-1,4bef214492f97c96,9377150e8a4b7363] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c4e7605bafa84a09,c01bdf996004bbe0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:50:28.498 INFO 21062 --- [ocp-schedules-14,c95cba1d24db028c,0ece43ad5dd08e49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:50:28.500 INFO 21062 --- [ocp-schedules-27,add15ce29818efd1,b1eceb5a5361f43a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:50:28.500 INFO 21062 --- [ocp-schedules-5,355d648255ace0c5,08fac8dec26c0f3c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:50:28.500 INFO 21062 --- [ocp-schedules-13,cd9a8969294e57bd,b80bfab827d16882] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:50:28.502 INFO 21062 --- [ocp-schedules-14,c95cba1d24db028c,0ece43ad5dd08e49] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:50:28.502 INFO 21062 --- [ocp-schedules-15,3c8f98de557cd5bd,38afb410776b7bf4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:50:28.504 INFO 21062 --- [ocp-schedules-27,add15ce29818efd1,b1eceb5a5361f43a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:28.504 INFO 21062 --- [ocp-schedules-13,cd9a8969294e57bd,b80bfab827d16882] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:50:28.505 INFO 21062 --- [ocp-schedules-20,a259c885d474c635,bc37d546433f9063] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:50:28.506 INFO 21062 --- [ocp-schedules-5,355d648255ace0c5,08fac8dec26c0f3c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:50:28.510 INFO 21062 --- [ocp-schedules-3,70c6379cc1c7d3b3,cde1b9984fbf3aa1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:50:28.510 INFO 21062 --- [ocp-schedules-23,a464bb1f7b78cefe,5d1ef3db3d1c9701] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:50:28.510 INFO 21062 --- [ocp-schedules-3,70c6379cc1c7d3b3,cde1b9984fbf3aa1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:50:28.510 INFO 21062 --- [ocp-schedules-23,a464bb1f7b78cefe,5d1ef3db3d1c9701] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:50:28.511 INFO 21062 --- [ocp-schedules-19,ed1e85bdff75f8c1,84ea77be5d86ac00] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:50:28.511 INFO 21062 --- [ocp-schedules-19,ed1e85bdff75f8c1,84ea77be5d86ac00] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:50:28.551 INFO 21062 --- [ocp-async-8,8cde450efac7c13a,2d9979af87bb5a89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:50:28.555 INFO 21062 --- [ocp-schedules-30,1683b709668d97ae,43cc2e08fefea37e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:50:28.555 INFO 21062 --- [ocp-schedules-30,1683b709668d97ae,43cc2e08fefea37e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:50:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:50:28.588673+08:00, lastProcessTime=2024-11-04T12:50:18.588664+08:00 2024-11-04 12:50:28.603 INFO 21062 --- [ocp-schedules-28,c17a190ecd6b66e9,fcb70cb11b0ce831] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:50:28.633 WARN 21062 --- [ocp-async-7,37afa902cfd74690,fd970f154c75c64b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:28.636 INFO 21062 --- [ocp-async-6,d3c8b9dd0e4f0fa5,73affdcd57aee74b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:28.638 INFO 21062 --- [ocp-schedules-9,2aa48ee9e867a3c2,d7c678805eb19da1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:50:28.638 INFO 21062 --- [ocp-schedules-9,2aa48ee9e867a3c2,d7c678805eb19da1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:50:28.646 INFO 21062 --- [ocp-schedules-11,fb74d8bd111d39ec,ed907e4fe2558138] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:50:28.688 INFO 21062 --- [ocp-schedules-15,42b90efa778dd942,c9653b32a0ba618d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:50:28.693 INFO 21062 --- [ocp-schedules-15,42b90efa778dd942,c9653b32a0ba618d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:50:28.693 INFO 21062 --- [ocp-schedules-15,42b90efa778dd942,c9653b32a0ba618d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:50:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:50:28.588673+08:00 2024-11-04 12:50:28.794 INFO 21062 --- [ocp-schedules-23,930432b8094f15b1,49a1b4949bdfef81] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:50:28.794 INFO 21062 --- [ocp-schedules-23,930432b8094f15b1,49a1b4949bdfef81] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:50:28.795 INFO 21062 --- [ocp-schedules-23,930432b8094f15b1,49a1b4949bdfef81] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:50:28.795 INFO 21062 --- [ocp-schedules-23,930432b8094f15b1,49a1b4949bdfef81] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:50:28.796 INFO 21062 --- [ocp-schedules-23,930432b8094f15b1,49a1b4949bdfef81] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:50:28.796 INFO 21062 --- [ocp-schedules-23,930432b8094f15b1,49a1b4949bdfef81] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:50:28.815 INFO 21062 --- [ocp-schedules-14,ec770324e8e1c536,1dff44c09ee08374] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:50:29.015 INFO 21062 --- [ocp-schedules-1,53cce50b5eead259,36c868ba9340ff24] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:50:29.015 INFO 21062 --- [ocp-schedules-1,53cce50b5eead259,36c868ba9340ff24] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:50:29.037 INFO 21062 --- [ocp-schedules-18,30d2a2106f8adec9,1172ff68920be572] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:50:29.039 INFO 21062 --- [ocp-schedules-30,bf98cae7473b9da8,8725710532fc56e7] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:50:29.042 INFO 21062 --- [ocp-schedules-30,bf98cae7473b9da8,8725710532fc56e7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:29.042 INFO 21062 --- [ocp-schedules-18,30d2a2106f8adec9,1172ff68920be572] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:50:29.043 INFO 21062 --- [ocp-schedules-18,30d2a2106f8adec9,1172ff68920be572] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:50:29.043 INFO 21062 --- [ocp-schedules-1,53cce50b5eead259,36c868ba9340ff24] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:50:29.043 INFO 21062 --- [ocp-schedules-1,53cce50b5eead259,36c868ba9340ff24] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:50:29.053 INFO 21062 --- [ocp-schedules-22,58dffa65db388b5c,4c3a09df1979c778] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:50:29.067 INFO 21062 --- [ocp-schedules-24,d2088a3267fd420b,753c6f2294f484e9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:50:29.067 INFO 21062 --- [ocp-schedules-24,d2088a3267fd420b,753c6f2294f484e9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:50:29.067 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:50:29.067 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:50:29.077 INFO 21062 --- [ocp-schedules-24,d2088a3267fd420b,753c6f2294f484e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:50:29.077 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:50:29.077 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:50:29.079 WARN 21062 --- [ocp-schedules-24,d2088a3267fd420b,753c6f2294f484e9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:50:29.079 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:50:29.079 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:50:29.079 INFO 21062 --- [ocp-schedules-24,d2088a3267fd420b,753c6f2294f484e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:50:29.080 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:50:29.080 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:50:29.081 WARN 21062 --- [ocp-schedules-24,d2088a3267fd420b,753c6f2294f484e9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:50:29.081 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:50:29.082 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:50:29.093 ERROR 21062 --- [ocp-schedules-24,d2088a3267fd420b,753c6f2294f484e9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 12:50:29.104 ERROR 21062 --- [ocp-schedules-30,bf98cae7473b9da8,8725710532fc56e7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:50:29.105 INFO 21062 --- [ocp-schedules-30,bf98cae7473b9da8,8725710532fc56e7] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:50:29.164 INFO 21062 --- [ocp-schedules-16,021490491a32a35c,f372cacbea376da1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:50:29.164+08:00, clusters=[1] 2024-11-04 12:50:29.166 INFO 21062 --- [ocp-schedules-16,021490491a32a35c,f372cacbea376da1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:50:29.230 INFO 21062 --- [ocp-schedules-25,245b88f276f515eb,41deb347ddc09283] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:50:29.230+08:00, clusters=[1] 2024-11-04 12:50:29.233 INFO 21062 --- [ocp-schedules-25,245b88f276f515eb,41deb347ddc09283] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:50:29.716 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:29.717 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:29.751 INFO 21062 --- [ocp-schedules-10,87af88d9aa1e742a,908470627b31e500] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:50:29.751 INFO 21062 --- [ocp-schedules-10,6bd8b3230c0481bc,a23ea374c591e100] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:50:29.754 INFO 21062 --- [ocp-schedules-10,6bd8b3230c0481bc,a23ea374c591e100] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:29.783 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:50:29.783 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:50:29.983 INFO 21062 --- [prometheus-minute-schedule-1,2c6d1e11e2f4ca4e,468be8462f82a813] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:30.153 INFO 21062 --- [ocp-schedules-1,f948b4d2e0891bf8,22c78254d754786b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:50:30.156 INFO 21062 --- [ocp-schedules-1,f948b4d2e0891bf8,22c78254d754786b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:30.241 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:50:30.244 ERROR 21062 --- [ocp-schedules-1,f948b4d2e0891bf8,22c78254d754786b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:50:30.244 INFO 21062 --- [ocp-schedules-1,f948b4d2e0891bf8,22c78254d754786b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:50:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:50:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:50:30.633 WARN 21062 --- [ocp-async-4,7e9ba16e4e8cab01,ffc826c06dafdc47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:30.637 INFO 21062 --- [ocp-async-2,7a3ab289e894bf6a,d41551ce291eeefe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:32.634 WARN 21062 --- [ocp-async-3,8e3614c0fdd746fd,3284ab6c8609a3cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:32.637 INFO 21062 --- [ocp-async-1,c6e7eeda5b38b2c2,136f046edda5cbb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:32.717 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:32.718 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a36e868d5c8b4089,7bc20d001decca4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:33.200 INFO 21062 --- [ocp-schedules-26,01bca364a7649239,59e10f47b97aee64] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:50:34.634 WARN 21062 --- [ocp-async-9,7ce64e5ddfa7b24b,ef7da6390c333844] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:34.637 INFO 21062 --- [ocp-async-7,caed7b6513aa25fa,9c1e812d1b34ce06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:35.718 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:35.719 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:35.983 INFO 21062 --- [prometheus-minute-schedule-1,28d61fe1aaf488bc,ce7a2068a98e6447] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:36.634 WARN 21062 --- [ocp-async-6,2ad72c46b4859db6,cfdafd5e5693fc95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:36.637 INFO 21062 --- [ocp-async-4,d2b0d5bd2c3178cb,8236089570dbe53d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:38.505 INFO 21062 --- [ocp-schedules-27,add15ce29818efd1,b1eceb5a5361f43a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:50:38.505 INFO 21062 --- [ocp-schedules-13,cd9a8969294e57bd,b80bfab827d16882] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:50:38.551 INFO 21062 --- [ocp-async-2,a53b1e8cecb2158b,9cf85cd49e151a63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:50:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:50:38.588987+08:00, lastProcessTime=2024-11-04T12:50:28.588673+08:00 2024-11-04 12:50:38.634 WARN 21062 --- [ocp-async-8,99c28489fee21a10,dbf423ad169179d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:38.637 INFO 21062 --- [ocp-async-1,9d7590ba1be27158,7626f04f08aedc26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:38.719 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:38.720 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:50:38.588987+08:00 2024-11-04 12:50:38.817 INFO 21062 --- [ocp-schedules-27,dfa5c68b6d926f2c,e31e4d81c2b1ca69] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:50:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ec959380366971e2,547ab29b85c9d916] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:50:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:50:40.332 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:50:40.332 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:50:40.339 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:50:40.339 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:50:40.339 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:50:40.339 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:50:40.360 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:50:40.360 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:50:40.360 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:50:40.360 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:50:40.400 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:50:40.409 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:50:40.409 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:50:40.409 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:50:40.409 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:50:40.409 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:50:40.417 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:50:40.634 WARN 21062 --- [ocp-async-5,7b649e18d6515dea,90c9c06b7bdd3030] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:40.637 INFO 21062 --- [ocp-async-7,b1be53e549a1b312,07233de8cd5f2215] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:40.957 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:50:41.543 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.184606255S 2024-11-04 12:50:41.544 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:50:41.648 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:50:41.648 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:50:41.679 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:50:41.744 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:50:41.744 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:50:41.744 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:50:41.744 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:50:41.744 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:50:41.744 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:50:41.757 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:50:41.759 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:50:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:50:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 12:50:41.946 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] o.s.boot.SpringApplication : Started application in 1.606 seconds (JVM running for 1255.625) 2024-11-04 12:50:41.946 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.607090655S 2024-11-04 12:50:41.983 INFO 21062 --- [prometheus-minute-schedule-1,216ac884b4212ad6,368a6419dc7c948d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:42.440 INFO 21062 --- [ocp-schedules-17,483dad401ca0a76d,26ddeb51d29b19b7] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:50:42.634 WARN 21062 --- [ocp-async-4,7b3ad180c5651580,095b5b3a0a63a73d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:42.638 INFO 21062 --- [ocp-async-10,74401b57034db92b,4c285d84d1efc1cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:43.278 INFO 21062 --- [ocp-schedules-30,2f9e76595d89bfd1,68832a792dd1c45c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:50:44.635 WARN 21062 --- [ocp-async-10,c009050bab8cc76b,11c2a4540c2a4a56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:44.638 INFO 21062 --- [ocp-async-3,5299e473233c0f67,92644353c4f33955] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:44.760 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:44.760 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a6e711bbf43b8d8e,5d187eb4e010944e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:46.635 WARN 21062 --- [ocp-async-3,0dc3e06e01514a7e,0cdf745f1468fc20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:46.638 INFO 21062 --- [ocp-async-1,ce19d2dd1f7047ef,4d093d88503cc35b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:47.761 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:47.762 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:47.983 INFO 21062 --- [prometheus-minute-schedule-1,313807182ae93b1b,d526bf3fef848c7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:48.551 INFO 21062 --- [ocp-async-1,8a3500d5c779f593,3ef6e87b0b1d2615] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:50:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:50:48.587721+08:00, lastProcessTime=2024-11-04T12:50:38.588987+08:00 2024-11-04 12:50:48.635 WARN 21062 --- [ocp-async-7,253cb1aa60f573c2,0096b24dbd27e3be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:48.638 INFO 21062 --- [ocp-async-4,f384845a6d52c771,4f1832f0495f7385] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:50:48.587721+08:00 2024-11-04 12:50:48.820 INFO 21062 --- [ocp-schedules-20,d644c4ed6bf8e813,329b3a41c3bd3088] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:50:49.045 INFO 21062 --- [ocp-schedules-5,f84554dfcda0586a,627f31d9ee1683ea] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:50:49.052 INFO 21062 --- [ocp-schedules-5,f84554dfcda0586a,627f31d9ee1683ea] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:50:49.052 INFO 21062 --- [ocp-schedules-5,f84554dfcda0586a,627f31d9ee1683ea] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:50:50.635 WARN 21062 --- [ocp-async-4,fe61e5be4a355324,7f55b66573af4b76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:50.638 INFO 21062 --- [ocp-async-8,8338461bfc80dc0e,713b6df6fb8929a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:50.762 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:50.763 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:50.983 INFO 21062 --- [prometheus-minute-schedule-1,68697d7b4cee843f,68b175472882e110] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:52.635 WARN 21062 --- [ocp-async-8,254c1165ee1dbb39,e949408e099a5173] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:52.638 INFO 21062 --- [ocp-async-6,6dc0c73e4836bacb,b8134515de05e072] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:53.308 INFO 21062 --- [ocp-schedules-15,c18276f2d2c80561,e236882d947f4919] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:50:53.763 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:53.763 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9e63cb29eac27089,6137f52324bf29f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:54.635 WARN 21062 --- [ocp-async-1,1b709babd579a523,a8bb3f500014fb8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:54.639 INFO 21062 --- [ocp-async-7,672c3e417622ebad,b76b4e0a7435717c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:56.636 WARN 21062 --- [ocp-async-2,2c8f7b11949bd6f1,f696736363892175] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:56.639 INFO 21062 --- [ocp-async-4,ed945aff10829ea1,c1819b56eeeebc96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:56.764 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:56.764 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:56.983 INFO 21062 --- [prometheus-minute-schedule-1,72912f60019ece02,51b91d4b7787ff13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:50:58.500 INFO 21062 --- [ocp-schedules-2,335899b78b7c66ae,f376c602e2949197] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:50:58.500 INFO 21062 --- [ocp-schedules-8,4a9a49b7cd13b67f,bde954fb3f257d82] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:50:58.502 INFO 21062 --- [ocp-schedules-9,99909965e2b67c78,85e38dd711a6ca37] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:50:58.505 INFO 21062 --- [ocp-schedules-2,335899b78b7c66ae,f376c602e2949197] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:58.505 INFO 21062 --- [ocp-schedules-8,4a9a49b7cd13b67f,bde954fb3f257d82] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:50:58.506 INFO 21062 --- [ocp-schedules-30,de0fab9f046a360c,2afeeab580a9b125] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:50:58.512 INFO 21062 --- [ocp-schedules-29,d3f7b3e27b3c80db,dde8acbc350c4312] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:50:58.513 INFO 21062 --- [ocp-schedules-29,d3f7b3e27b3c80db,dde8acbc350c4312] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:50:58.513 INFO 21062 --- [ocp-schedules-11,405c99d2c33544a9,b7480c6e6a1a72f4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:50:58.513 INFO 21062 --- [ocp-schedules-11,405c99d2c33544a9,b7480c6e6a1a72f4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:50:58.513 INFO 21062 --- [ocp-schedules-18,15b198d346e05022,36ab7e4e9c826131] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:50:58.513 INFO 21062 --- [ocp-schedules-18,15b198d346e05022,36ab7e4e9c826131] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:50:58.552 INFO 21062 --- [ocp-async-4,7a65af968b781339,97bbd998bd883da5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:50:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:50:58.589858+08:00, lastProcessTime=2024-11-04T12:50:48.587721+08:00 2024-11-04 12:50:58.620 INFO 21062 --- [ocp-schedules-19,f9536be9d5270d97,f59df33d662a9f30] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:50:58.623 INFO 21062 --- [ocp-schedules-19,f9536be9d5270d97,f59df33d662a9f30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:58.623 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:50:58.636 WARN 21062 --- [ocp-async-8,7bea22218c2bd0b2,8e10878d474c7f69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:50:58.639 INFO 21062 --- [ocp-async-5,c267260e38d489d8,728dea4a6449cd37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:50:58.649 INFO 21062 --- [ocp-schedules-25,779cfa5904e05c13,1331f7a844f6f521] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:50:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:50:58.589858+08:00 2024-11-04 12:50:58.823 INFO 21062 --- [ocp-schedules-23,a13d026f55dea626,6dbe1a775d5d7aa3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:50:59.765 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:50:59.766 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:50:59.784 INFO 21062 --- [ocp-schedules-10,6bd8b3230c0481bc,a23ea374c591e100] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:50:59.784 INFO 21062 --- [ocp-schedules-10,a127e23de6d014d8,5f2dc47e1d6cd5a1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:50:59.788 INFO 21062 --- [ocp-schedules-10,a127e23de6d014d8,5f2dc47e1d6cd5a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:50:59.817 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:50:59.817 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:50:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ae0706d025be8368,588f4049dacbe798] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695860, interval=60 2024-11-04 12:51:00.636 WARN 21062 --- [ocp-async-1,4aefdcf732def201,a43b3a5fb8e969d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:00.639 INFO 21062 --- [ocp-async-10,49b9d325bb1ddbf2,533f4fa264e529c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:02.636 WARN 21062 --- [ocp-async-2,382830f87238d8cb,04caaa6c22a63106] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:02.639 INFO 21062 --- [ocp-async-9,4cb82adca9548f3b,856be164af2dda8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:02.766 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:02.767 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e94dab7ab6307c5f,32bd919876d49246] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:03.348 INFO 21062 --- [ocp-schedules-4,af81b1c93c73c9d0,112988d253f8d674] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:51:04.636 WARN 21062 --- [ocp-async-4,28cf247704492c14,8fb3eb5202afe3f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:04.639 INFO 21062 --- [ocp-async-8,37d63450334e9027,05c1985a3573dfd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:05.767 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:05.768 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:05.983 INFO 21062 --- [prometheus-minute-schedule-1,362382f7d3764ac3,fac254bf1296620a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:06.636 WARN 21062 --- [ocp-async-5,5fdba472334e8541,aeb1fb63351a05a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:06.639 INFO 21062 --- [ocp-async-1,e8d4077b2657cdc1,eab932c761c77955] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:08.506 INFO 21062 --- [ocp-schedules-2,335899b78b7c66ae,f376c602e2949197] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:51:08.506 INFO 21062 --- [ocp-schedules-8,4a9a49b7cd13b67f,bde954fb3f257d82] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:51:08.552 INFO 21062 --- [ocp-async-1,ce15febe8377fb70,35479fcbc9fcf543] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:51:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:51:08.588895+08:00, lastProcessTime=2024-11-04T12:50:58.589858+08:00 2024-11-04 12:51:08.624 INFO 21062 --- [ocp-schedules-19,f9536be9d5270d97,f59df33d662a9f30] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:51:08.637 WARN 21062 --- [ocp-async-2,62315959f96be768,6c665e4cf84918de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:08.640 INFO 21062 --- [ocp-async-3,27bcc46c3c5a3927,79b9321b61748cdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:51:08.588895+08:00 2024-11-04 12:51:08.768 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:08.769 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:08.826 INFO 21062 --- [ocp-schedules-4,12b8afef825b6304,148f93da6afdfed1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:51:08.983 INFO 21062 --- [prometheus-minute-schedule-1,785320d228383c6c,98e58ca012035280] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:09.054 INFO 21062 --- [ocp-schedules-2,a84f54c87a35433f,d9109834c7d849b7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:51:09.060 INFO 21062 --- [ocp-schedules-2,a84f54c87a35433f,d9109834c7d849b7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:51:09.060 INFO 21062 --- [ocp-schedules-2,a84f54c87a35433f,d9109834c7d849b7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:51:10.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:51:10.004 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:51:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:51:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:51:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:51:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:51:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:51:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:51:10.637 WARN 21062 --- [ocp-async-4,7e0a37413873bdbc,c491fd845d0b4341] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:10.640 INFO 21062 --- [ocp-async-6,a2c80d5d9dd89364,743365024dad5d8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:11.106 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695800, nodeCount=16, affectRows=16 2024-11-04 12:51:11.106 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695800, nodeCount=16, validCount=16 2024-11-04 12:51:11.106 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695800, nodeCount=28, affectRows=28 2024-11-04 12:51:11.106 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695800, nodeCount=23, affectRows=23 2024-11-04 12:51:11.107 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695800, nodeCount=28, validCount=28 2024-11-04 12:51:11.107 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695800, nodeCount=23, validCount=23 2024-11-04 12:51:11.107 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695800, nodeCount=47, affectRows=47 2024-11-04 12:51:11.107 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695800, nodeCount=47, validCount=47 2024-11-04 12:51:11.107 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695800, nodeCount=67, affectRows=67 2024-11-04 12:51:11.107 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695800, nodeCount=66, affectRows=66 2024-11-04 12:51:11.108 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695800, nodeCount=65, validCount=65 2024-11-04 12:51:11.108 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695800, nodeCount=65, validCount=65 2024-11-04 12:51:11.108 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695800, nodeCount=75, affectRows=75 2024-11-04 12:51:11.108 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695800, nodeCount=75, validCount=75 2024-11-04 12:51:11.769 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:11.774 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:11.983 INFO 21062 --- [prometheus-minute-schedule-1,99d803b6e12c9246,4f36045ccee9fa40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:12.637 WARN 21062 --- [ocp-async-5,fb3693c0b8ad7940,c959e5cd766ad3c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:12.640 INFO 21062 --- [ocp-async-7,18c26aa25edd40d3,ad8aba1a368a2964] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:13.374 INFO 21062 --- [ocp-schedules-1,57ea17a7befa4302,221227d30fc17d70] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:51:13.984 INFO 21062 --- [prometheus-cache-monitor1,33d3cb18e15defac,3f1ed50b452e8b01] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:51:13.984 INFO 21062 --- [prometheus-cache-monitor1,33d3cb18e15defac,3f1ed50b452e8b01] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:51:14.170 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:47:14.152, end=2024-11-04T12:49:14.167} 2024-11-04 12:51:14.181 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:51:14.181 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:51:14.637 WARN 21062 --- [ocp-async-10,ca39d49a09c4c0c5,b3b7ef95d14180f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:14.640 INFO 21062 --- [ocp-async-2,ae8d0a59cdfee65d,1e992914f0c6a5d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:14.774 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:14.777 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:14.983 INFO 21062 --- [prometheus-minute-schedule-1,34875b0ac14163d2,1a172020c9b0d43c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:16.637 WARN 21062 --- [ocp-async-3,15d692c5fd3f6c1b,103eb679801d222d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:16.641 INFO 21062 --- [ocp-async-4,80e8739322717cf3,ce7c6d70c109a6bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:17.777 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:17.779 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9a0df13b55dfb4b6,53fcff7fcb3b9866] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:18.552 INFO 21062 --- [ocp-async-4,2935a14a1ad425cc,88fc98fcdacce809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:51:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:51:18.588678+08:00, lastProcessTime=2024-11-04T12:51:08.588895+08:00 2024-11-04 12:51:18.638 WARN 21062 --- [ocp-async-5,2b6698299bc9deb2,437ae389da665d19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:18.641 INFO 21062 --- [ocp-async-1,90e379a70d19f8fd,454a4f3a83a57fe2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:51:18.588678+08:00 2024-11-04 12:51:18.829 INFO 21062 --- [ocp-schedules-28,48c325ce8fac6420,88e25c988f939f36] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:51:20.638 WARN 21062 --- [ocp-async-10,a87010af4d83e48f,39d8d9c77464d651] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:20.641 INFO 21062 --- [ocp-async-9,6077cfcafcb6d6b6,77d1789691f99771] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:20.779 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:20.780 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ee5420a4d8079f09,379e0f7256af0a5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:22.638 WARN 21062 --- [ocp-async-3,e8eb1bfce08736d0,0bbd52e0112135c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:22.641 INFO 21062 --- [ocp-async-8,6902d98114a3ec6f,1a21d7a9f88566b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:23.416 INFO 21062 --- [ocp-schedules-25,86c96f62944828c2,ae7d3293f2bcb388] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:51:23.780 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:23.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a0267bcb7849bc8a,d92b7c02664ce458] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:24.638 WARN 21062 --- [ocp-async-6,77f033c04d70d06c,809794aa6d3753b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:24.641 INFO 21062 --- [ocp-async-5,8c97cb6769ddf51c,4960f59d60a115e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:26.638 WARN 21062 --- [ocp-async-1,a0a856147203d3a1,102c74f9857be5e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:26.641 INFO 21062 --- [ocp-async-10,cea9365e25d59007,ac60124ab6e479af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:26.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:26.782 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:26.983 INFO 21062 --- [prometheus-minute-schedule-1,a35bda2c8592a7bd,6ac022a4466475c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c2fd5e8858be5643,fe1ba829ec063d98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:51:28.498 INFO 21062 --- [ocp-schedules-28,147dcf59f90cc564,0d5d3aae72c60631] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:51:28.499 INFO 21062 --- [ocp-schedules-4,897897b20814dcb8,8f80dc5e26fba84c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:51:28.500 INFO 21062 --- [ocp-schedules-12,4434626ba8f0db25,efb4a2daeff54ffb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:51:28.500 INFO 21062 --- [ocp-schedules-24,3a09177f6cc5d4ee,2de013018b03fb60] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:51:28.502 INFO 21062 --- [ocp-schedules-28,147dcf59f90cc564,0d5d3aae72c60631] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:51:28.502 INFO 21062 --- [ocp-schedules-8,c8a1db2ffa3e2ca8,c489e4bfff5ef3cc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:51:28.503 INFO 21062 --- [ocp-schedules-12,4434626ba8f0db25,efb4a2daeff54ffb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:51:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:51:28.504 INFO 21062 --- [ocp-schedules-24,3a09177f6cc5d4ee,2de013018b03fb60] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:51:28.504 INFO 21062 --- [ocp-schedules-27,83290f3a137cd360,2d3cc84e4150e620] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:51:28.506 INFO 21062 --- [ocp-schedules-4,897897b20814dcb8,8f80dc5e26fba84c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:51:28.511 INFO 21062 --- [ocp-schedules-1,ce7a98330dadeb59,e34fe0148cd78052] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:51:28.511 INFO 21062 --- [ocp-schedules-1,ce7a98330dadeb59,e34fe0148cd78052] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:51:28.511 INFO 21062 --- [ocp-schedules-19,994b65a874749f65,e94350470759be12] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:51:28.511 INFO 21062 --- [ocp-schedules-22,2858d511dae75311,9e5a18d89c27601e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:51:28.511 INFO 21062 --- [ocp-schedules-19,994b65a874749f65,e94350470759be12] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:51:28.511 INFO 21062 --- [ocp-schedules-22,2858d511dae75311,9e5a18d89c27601e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:51:28.552 INFO 21062 --- [ocp-async-3,cec5e52c4ebcc46f,d78d1f9ba7d759b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:51:28.571 INFO 21062 --- [ocp-schedules-11,d62fb1b0ba801b70,a8371a7fd746b18e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:51:28.572 INFO 21062 --- [ocp-schedules-11,d62fb1b0ba801b70,a8371a7fd746b18e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:51:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:51:28.588026+08:00, lastProcessTime=2024-11-04T12:51:18.588678+08:00 2024-11-04 12:51:28.605 INFO 21062 --- [ocp-schedules-29,ddc0d9f2878f9832,d58e8493d7bcfd55] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:51:28.638 WARN 21062 --- [ocp-async-8,cf78e89994cf68a5,72c5263c922b7e17] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:28.642 INFO 21062 --- [ocp-async-6,148a845a12adee72,c881cb1b36994f45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:28.642 INFO 21062 --- [ocp-schedules-18,43b8382399f0b00e,e139a3991074816d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:51:28.642 INFO 21062 --- [ocp-schedules-18,43b8382399f0b00e,e139a3991074816d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:51:28.651 INFO 21062 --- [ocp-schedules-4,275b1e996dd54302,ffe5892c03dad3b5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:51:28.693 INFO 21062 --- [ocp-schedules-27,1c27498e141140da,b054ff4bf1abecd9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:51:28.698 INFO 21062 --- [ocp-schedules-27,1c27498e141140da,b054ff4bf1abecd9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:51:28.698 INFO 21062 --- [ocp-schedules-27,1c27498e141140da,b054ff4bf1abecd9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:51:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:51:28.588026+08:00 2024-11-04 12:51:28.797 INFO 21062 --- [ocp-schedules-19,55fdc35b78786960,1956b593d5c571eb] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:51:28.797 INFO 21062 --- [ocp-schedules-19,55fdc35b78786960,1956b593d5c571eb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:51:28.803 INFO 21062 --- [ocp-schedules-19,55fdc35b78786960,1956b593d5c571eb] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:51:28.803 INFO 21062 --- [ocp-schedules-19,55fdc35b78786960,1956b593d5c571eb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:51:28.807 INFO 21062 --- [ocp-schedules-19,55fdc35b78786960,1956b593d5c571eb] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:51:28.807 INFO 21062 --- [ocp-schedules-19,55fdc35b78786960,1956b593d5c571eb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:51:28.831 INFO 21062 --- [ocp-schedules-21,6ad49b411a62c44b,287516c5d4d489d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:51:29.043 INFO 21062 --- [ocp-schedules-20,c3ecebada400ad2e,052b0e6f01e8a301] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:51:29.043 INFO 21062 --- [ocp-schedules-20,c3ecebada400ad2e,052b0e6f01e8a301] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:51:29.062 INFO 21062 --- [ocp-schedules-13,96acdbfab8d0eeb4,9f6fa5ddc42f7426] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:51:29.068 INFO 21062 --- [ocp-schedules-13,96acdbfab8d0eeb4,9f6fa5ddc42f7426] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:51:29.068 INFO 21062 --- [ocp-schedules-13,96acdbfab8d0eeb4,9f6fa5ddc42f7426] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:51:29.070 INFO 21062 --- [ocp-schedules-20,c3ecebada400ad2e,052b0e6f01e8a301] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:51:29.070 INFO 21062 --- [ocp-schedules-20,c3ecebada400ad2e,052b0e6f01e8a301] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:51:29.079 INFO 21062 --- [ocp-schedules-23,93b849bc449118b1,746c10a74c79e25f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:51:29.101 INFO 21062 --- [ocp-schedules-5,0a86d352d972f234,5569d513804a47e9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:51:29.101 INFO 21062 --- [ocp-schedules-5,0a86d352d972f234,5569d513804a47e9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:51:29.101 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:51:29.102 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:51:29.112 INFO 21062 --- [ocp-schedules-5,0a86d352d972f234,5569d513804a47e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:51:29.112 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:51:29.112 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:51:29.114 WARN 21062 --- [ocp-schedules-5,0a86d352d972f234,5569d513804a47e9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:51:29.114 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:51:29.114 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:51:29.115 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:51:29.115 INFO 21062 --- [ocp-schedules-5,0a86d352d972f234,5569d513804a47e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:51:29.115 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:51:29.116 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:51:29.116 WARN 21062 --- [ocp-schedules-5,0a86d352d972f234,5569d513804a47e9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:51:29.117 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:51:29.123 ERROR 21062 --- [ocp-schedules-5,0a86d352d972f234,5569d513804a47e9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 12:51:29.191 INFO 21062 --- [ocp-schedules-16,4a65ad3854d7a445,0598fb8a92711cec] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:51:29.191+08:00, clusters=[1] 2024-11-04 12:51:29.192 INFO 21062 --- [ocp-schedules-16,4a65ad3854d7a445,0598fb8a92711cec] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:51:29.260 INFO 21062 --- [ocp-schedules-9,a0f84f2a12b22808,630ab5b54033260d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:51:29.260+08:00, clusters=[1] 2024-11-04 12:51:29.264 INFO 21062 --- [ocp-schedules-9,a0f84f2a12b22808,630ab5b54033260d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:51:29.782 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:29.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:29.818 INFO 21062 --- [ocp-schedules-10,a127e23de6d014d8,5f2dc47e1d6cd5a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:51:29.818 INFO 21062 --- [ocp-schedules-10,ded25472d8b162ef,82fe070a7776424a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:51:29.822 INFO 21062 --- [ocp-schedules-10,ded25472d8b162ef,82fe070a7776424a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:51:29.850 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:51:29.850 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:51:29.983 INFO 21062 --- [prometheus-minute-schedule-1,1cf4b77238ad3705,d6a80b83480db1f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:30.244 INFO 21062 --- [ocp-schedules-23,20f956a4309cfb6e,229b452c2948beef] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:51:30.248 INFO 21062 --- [ocp-schedules-23,20f956a4309cfb6e,229b452c2948beef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:51:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:51:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:51:30.332 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:51:30.334 ERROR 21062 --- [ocp-schedules-23,20f956a4309cfb6e,229b452c2948beef] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:51:30.334 INFO 21062 --- [ocp-schedules-23,20f956a4309cfb6e,229b452c2948beef] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:51:30.639 WARN 21062 --- [ocp-async-5,11660b0618535eb6,eeb4e1a9bf2afe0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:30.642 INFO 21062 --- [ocp-async-1,adb302fd011670ca,114b9a3f7165720b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:32.639 WARN 21062 --- [ocp-async-2,72dab50b4bd49d8b,65f95df76b22eb03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:32.642 INFO 21062 --- [ocp-async-9,2ac27d26592540f6,acc987975986b606] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:32.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:32.784 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:32.983 INFO 21062 --- [prometheus-minute-schedule-1,61c2cef20dee5c06,c148cf3ab87bedb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:33.442 INFO 21062 --- [ocp-schedules-9,8db2c8f46d54af44,83d7bfd80fcb8dc6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:51:34.639 WARN 21062 --- [ocp-async-4,494ae208c194af95,e30527ae7263a45f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:34.642 INFO 21062 --- [ocp-async-8,fe591c5e87749613,3a459bdbc5f38f16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:35.784 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:35.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:35.990 INFO 21062 --- [prometheus-minute-schedule-1,046bc7a9bec45497,7c74b3ad82301c93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:36.639 WARN 21062 --- [ocp-async-6,26eb86e6f50428e4,861a9623241cde78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:36.642 INFO 21062 --- [ocp-async-5,af5c0b309ff5fcfc,457c86176c9bc543] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:38.504 INFO 21062 --- [ocp-schedules-12,4434626ba8f0db25,efb4a2daeff54ffb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:51:38.505 INFO 21062 --- [ocp-schedules-24,3a09177f6cc5d4ee,2de013018b03fb60] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:51:38.553 INFO 21062 --- [ocp-async-1,eb787dfcdcc2ab88,52ee2eeaefca4d5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:51:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:51:38.587842+08:00, lastProcessTime=2024-11-04T12:51:28.588026+08:00 2024-11-04 12:51:38.639 WARN 21062 --- [ocp-async-3,95d50ba41b86bf00,eab509603aea4563] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:38.642 INFO 21062 --- [ocp-async-9,4e560b74a373de77,d99558c2bcf2269a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:51:38.587842+08:00 2024-11-04 12:51:38.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:38.787 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:38.834 INFO 21062 --- [ocp-schedules-14,38160e67277a7239,7e29c7eb1a9c7a49] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:51:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3af353b97a27d899,ac9f90a9c4a7bd79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:51:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:51:40.639 WARN 21062 --- [ocp-async-9,1203c4acdf8dec0e,3fd87a915e562d58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:40.643 INFO 21062 --- [ocp-async-8,86bb9b30c634ad47,cdc492f2b9f3d528] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:41.787 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:41.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bbdfebeb2f507442,73ebc16771dc3317] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:42.640 WARN 21062 --- [ocp-async-8,8ce50a238d2d5b9e,756945454a11a183] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:42.643 INFO 21062 --- [ocp-async-10,3d3be0ec4e3a6e56,535c56bc3e5766de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:43.484 INFO 21062 --- [ocp-schedules-18,043ec1c6f1182103,27ec798b37df0d9e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:51:44.640 WARN 21062 --- [ocp-async-10,d2ee8097ce5d3882,277ed1fb10cf0cbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:44.643 INFO 21062 --- [ocp-async-2,25d82aa47695c829,f02da33c254065b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:44.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:44.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f9ac51a22e516f61,507c828200fec226] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:46.640 WARN 21062 --- [ocp-async-2,6c00aa0360452b47,83b83561a7321557] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:46.643 INFO 21062 --- [ocp-async-9,7ad0a8d075b20501,55773ebd707a2561] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:47.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:47.791 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:47.983 INFO 21062 --- [prometheus-minute-schedule-1,aa4667081b7dbb4e,237453b3aa16a957] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:48.553 INFO 21062 --- [ocp-async-6,772ad0d4e3756d6b,8c26b447d467f93a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:51:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:51:48.588722+08:00, lastProcessTime=2024-11-04T12:51:38.587842+08:00 2024-11-04 12:51:48.640 WARN 21062 --- [ocp-async-8,bc9e0df23c33d96f,0c829e96522d78fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:48.643 INFO 21062 --- [ocp-async-1,14dcaa5c5e425ac0,eb3bd333af519a32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:51:48.588722+08:00 2024-11-04 12:51:48.837 INFO 21062 --- [ocp-schedules-26,1f5ca9cc9f7c506b,9e5d69155d3cbb2a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:51:49.070 INFO 21062 --- [ocp-schedules-5,c465f5125f92d2cb,052b2d3437bfb821] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:51:49.076 INFO 21062 --- [ocp-schedules-5,c465f5125f92d2cb,052b2d3437bfb821] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:51:49.076 INFO 21062 --- [ocp-schedules-5,c465f5125f92d2cb,052b2d3437bfb821] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:51:50.640 WARN 21062 --- [ocp-async-10,417be3a4b6f4bbb3,84b3fca4825cc3ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:50.643 INFO 21062 --- [ocp-async-4,9c28f726232c26b0,221dce96e7632f6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:50.791 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:50.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:50.983 INFO 21062 --- [prometheus-minute-schedule-1,fbb5a15b973669b5,dea578deb0cd031b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:52.641 WARN 21062 --- [ocp-async-2,13c38f28ea5a6787,8a028e52c3a1dcc3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:52.644 INFO 21062 --- [ocp-async-7,229f6646117166c0,3b332e3d70f955d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:53.508 INFO 21062 --- [ocp-schedules-23,f3f8a5ec8c63efe7,e7dbbc1e2e8349a6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:51:53.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:53.793 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c0f6b660b2e450c3,eadc31d282c13bd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:54.641 WARN 21062 --- [ocp-async-6,312eedee1cc64fad,a0ff19912f5cedac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:54.645 INFO 21062 --- [ocp-async-5,90b3d7a69a7f4077,2feecd80eb79fdca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:56.641 WARN 21062 --- [ocp-async-1,cb237ddc3291a8e6,0d7bb3106c5f5e34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:56.646 INFO 21062 --- [ocp-async-3,c6b07fb8a50ff228,734e2c82b874dda5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:56.793 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:56.794 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:56.983 INFO 21062 --- [prometheus-minute-schedule-1,da61a530f7312377,8661b5ce1a56aa0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:51:58.500 INFO 21062 --- [ocp-schedules-24,6f68f21d45d2a893,1b6ab98bfa9d9592] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:51:58.500 INFO 21062 --- [ocp-schedules-26,f8518ede565915b9,903bcc2e71f1a7f2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:51:58.502 INFO 21062 --- [ocp-schedules-11,4d80c66ccbd15154,32b969350b13d391] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:51:58.504 INFO 21062 --- [ocp-schedules-24,6f68f21d45d2a893,1b6ab98bfa9d9592] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:51:58.504 INFO 21062 --- [ocp-schedules-26,f8518ede565915b9,903bcc2e71f1a7f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:51:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:51:58.505 INFO 21062 --- [ocp-schedules-17,e6de223d40ac6271,d01af07163f2c1c2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:51:58.511 INFO 21062 --- [ocp-schedules-4,b5fdabf3627e1144,839073baf7eec9bd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:51:58.511 INFO 21062 --- [ocp-schedules-12,c0d7db2d23b0bf92,f9443400a247163c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:51:58.511 INFO 21062 --- [ocp-schedules-12,c0d7db2d23b0bf92,f9443400a247163c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:51:58.511 INFO 21062 --- [ocp-schedules-4,b5fdabf3627e1144,839073baf7eec9bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:51:58.511 INFO 21062 --- [ocp-schedules-16,39da383d168c5bc6,ebdb277cf22cd20e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:51:58.511 INFO 21062 --- [ocp-schedules-16,39da383d168c5bc6,ebdb277cf22cd20e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:51:58.553 INFO 21062 --- [ocp-async-3,27fb8918e3196170,a3c730924498efa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:51:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:51:58.587629+08:00, lastProcessTime=2024-11-04T12:51:48.588722+08:00 2024-11-04 12:51:58.641 WARN 21062 --- [ocp-async-2,6f35513ae1348e1b,98170f5a0a2aa80a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:51:58.646 INFO 21062 --- [ocp-async-7,e8969e6bef4010e7,b55df625a25d7446] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:51:58.653 INFO 21062 --- [ocp-schedules-3,7fcdd1853faea8f5,51033ea61bc5c91d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:51:58.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:51:58.587629+08:00 2024-11-04 12:51:58.841 INFO 21062 --- [ocp-schedules-29,80d21ea2faba404f,978278ff8ff9b787] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:51:59.794 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:51:59.796 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:51:59.850 INFO 21062 --- [ocp-schedules-10,ded25472d8b162ef,82fe070a7776424a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:51:59.850 INFO 21062 --- [ocp-schedules-10,332ee43a76b89886,5dab4bda6e014500] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:51:59.855 INFO 21062 --- [ocp-schedules-10,332ee43a76b89886,5dab4bda6e014500] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:51:59.883 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:51:59.883 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:51:59.983 INFO 21062 --- [prometheus-minute-schedule-1,bc21f716a034977c,5b2bbdf95737e330] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695920, interval=60 2024-11-04 12:52:00.641 WARN 21062 --- [ocp-async-6,2d6240441ccf2c16,66214a15b54e7ad9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:00.646 INFO 21062 --- [ocp-async-5,95ebda6c9b60b756,37102d964863f4d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:02.641 WARN 21062 --- [ocp-async-1,6729bdf66d34ebec,180014f3160a5154] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:02.646 INFO 21062 --- [ocp-async-3,226476cb907dd38a,83a0e47904cef9fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:02.796 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:02.797 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7ee0888c603dc044,6ca6b8062642b59f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:03.549 INFO 21062 --- [ocp-schedules-22,eae316eccbd693b8,e68da91bf1bcc877] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:52:04.641 WARN 21062 --- [ocp-async-4,bffb3881b2b51071,0700d1ebdad83947] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:04.646 INFO 21062 --- [ocp-async-9,a6874aca592923c3,167536c6def2c772] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:05.797 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:05.799 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:05.983 INFO 21062 --- [prometheus-minute-schedule-1,55fad02a3ffcdbd5,88a4762837413c86] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:06.642 WARN 21062 --- [ocp-async-7,e0d4769e29a1671f,0f8a3678bd6f559b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:06.646 INFO 21062 --- [ocp-async-8,451d8ee0e5275b3d,bb5838124856794c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:08.505 INFO 21062 --- [ocp-schedules-26,f8518ede565915b9,903bcc2e71f1a7f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:52:08.505 INFO 21062 --- [ocp-schedules-24,6f68f21d45d2a893,1b6ab98bfa9d9592] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:52:08.553 INFO 21062 --- [ocp-async-8,46af8fab32b44567,871bc24a8fb05f12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:52:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:52:08.587773+08:00, lastProcessTime=2024-11-04T12:51:58.587629+08:00 2024-11-04 12:52:08.624 INFO 21062 --- [ocp-schedules-8,3c99d81420a03dd8,c50829586432b498] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:52:08.627 INFO 21062 --- [ocp-schedules-8,3c99d81420a03dd8,c50829586432b498] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:08.628 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:52:08.642 WARN 21062 --- [ocp-async-1,8470278f9e7bf600,41b43030e2d729f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:08.647 INFO 21062 --- [ocp-async-3,dd6e02019879325b,78fb06a7a61199bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:52:08.587773+08:00 2024-11-04 12:52:08.799 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:08.800 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:08.843 INFO 21062 --- [ocp-schedules-20,91ac9b5547a69455,e7cb153e3df4f46e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:52:08.983 INFO 21062 --- [prometheus-minute-schedule-1,4d459bf900337374,6e614846e01732df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:09.077 INFO 21062 --- [ocp-schedules-7,0dddabc0db258ced,1f6ed2447758040f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:52:09.083 INFO 21062 --- [ocp-schedules-7,0dddabc0db258ced,1f6ed2447758040f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:52:09.083 INFO 21062 --- [ocp-schedules-7,0dddabc0db258ced,1f6ed2447758040f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:52:10.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:52:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:52:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:52:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:52:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:52:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:52:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:52:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:52:10.642 WARN 21062 --- [ocp-async-4,3e8e6d1c4a51f5f5,7d2afc10dd93281e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:10.647 INFO 21062 --- [ocp-async-9,3ad75df10f174a67,5f9f04b298c76830] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:11.110 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=2, affectRows=2 2024-11-04 12:52:11.110 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=2, validCount=2 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=6, affectRows=6 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=6, affectRows=6 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=6, validCount=6 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=6, validCount=6 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=18, affectRows=18 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=18, validCount=18 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=18, affectRows=18 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=18, validCount=18 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=28, affectRows=28 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=39, affectRows=39 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=32, affectRows=32 2024-11-04 12:52:11.111 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=24, affectRows=24 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=55, affectRows=55 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=28, validCount=28 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=43, affectRows=43 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=24, validCount=24 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=38, validCount=38 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=32, validCount=32 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=54, validCount=54 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695860, nodeCount=51, affectRows=51 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=42, validCount=42 2024-11-04 12:52:11.112 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695860, nodeCount=51, validCount=51 2024-11-04 12:52:11.800 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:11.802 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:11.983 INFO 21062 --- [prometheus-minute-schedule-1,2bb6fe5311f37b7e,bc458ca38486b4cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:12.642 WARN 21062 --- [ocp-async-7,91f9e931c65343fe,804636c9b7827c4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:12.647 INFO 21062 --- [ocp-async-8,25bd06e2e2d00109,66a7dad3b31716f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:13.575 INFO 21062 --- [ocp-schedules-29,b115104b80bcf592,d8ef936babd02c13] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:52:13.984 INFO 21062 --- [prometheus-cache-monitor1,2887eb952c9d5b33,8bbcc9ccbd0d78cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:52:13.985 INFO 21062 --- [prometheus-cache-monitor1,2887eb952c9d5b33,8bbcc9ccbd0d78cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:52:14.642 WARN 21062 --- [ocp-async-5,3f556a477483570b,c114fc4538c9d7fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:14.647 INFO 21062 --- [ocp-async-10,8eec4db6e9c1ce7b,6f40f92062fd7607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:14.802 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:14.803 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5156caa35f768e20,75385c2b2bcdae4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:16.643 WARN 21062 --- [ocp-async-3,2933cad7c2fd5d1c,10765e112727a4a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:16.647 INFO 21062 --- [ocp-async-2,9fd4fb48938276ef,2f8ac55c97b71fc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:17.803 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:17.804 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:17.983 INFO 21062 --- [prometheus-minute-schedule-1,e32de23e36d269c2,60c90f7b3863df94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:18.553 INFO 21062 --- [ocp-async-2,f38cf7362aa7ae34,53bd21978f136ef1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:52:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:52:18.588415+08:00, lastProcessTime=2024-11-04T12:52:08.587773+08:00 2024-11-04 12:52:18.629 INFO 21062 --- [ocp-schedules-8,3c99d81420a03dd8,c50829586432b498] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:52:18.643 WARN 21062 --- [ocp-async-7,0e5b8647e144a381,bb378b7c1d269ffd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:18.648 INFO 21062 --- [ocp-async-8,155d562e1f6988c2,c812b2b87f75eb63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:52:18.588415+08:00 2024-11-04 12:52:18.846 INFO 21062 --- [ocp-schedules-18,6512f564a9dd81ad,ef8c5119fe2c8dd2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:52:20.643 WARN 21062 --- [ocp-async-5,36cad0683dd6e2cc,de315bfcd3f54064] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:20.648 INFO 21062 --- [ocp-async-10,143d7a1a97e94d1e,41acb51c0789965e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:20.804 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:20.805 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:20.983 INFO 21062 --- [prometheus-minute-schedule-1,2b35e3fc87394d26,761985aad659fda5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:22.643 WARN 21062 --- [ocp-async-3,b7a3d4849cf6778c,d366a71f54f783d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:22.648 INFO 21062 --- [ocp-async-2,2e8852e9f6985ec6,199ed50fa93f36df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:23.618 INFO 21062 --- [ocp-schedules-20,dc92b617b5d9e1bc,4055dd7f88faeff9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:52:23.805 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:23.806 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:23.983 INFO 21062 --- [prometheus-minute-schedule-1,52fa73fd062abd57,bda3bc618e5bb87b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:24.643 WARN 21062 --- [ocp-async-9,7ff6212befc507a3,1b038d7390b75815] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:24.648 INFO 21062 --- [ocp-async-6,6b7e60a7e5c7f863,125e9ad04a9dfee8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:26.643 WARN 21062 --- [ocp-async-8,fdea8f5a04f351a0,15b9c06b4d56c875] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:26.648 INFO 21062 --- [ocp-async-1,1a9e465956e93b3d,2aafaed877abfe9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:26.806 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:26.807 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:26.983 INFO 21062 --- [prometheus-minute-schedule-1,dc55b828ea2c83cb,5f4497952a0e628e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2e0b06509ac1911d,486b155cad3b3d8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:52:28.498 INFO 21062 --- [ocp-schedules-4,24107b3ce4cf94da,2fd46281e7d4fe86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:52:28.499 INFO 21062 --- [ocp-schedules-21,37fbc0ccf0159a36,6afac283a5602b4a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:52:28.499 INFO 21062 --- [ocp-schedules-27,7bdddc94cbb51791,7dfc6c89baa47058] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:52:28.500 INFO 21062 --- [ocp-schedules-11,541146f00d2b28e9,de3ae4034dc798a2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:52:28.502 INFO 21062 --- [ocp-schedules-4,24107b3ce4cf94da,2fd46281e7d4fe86] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:52:28.502 INFO 21062 --- [ocp-schedules-5,fa20cf97ceac9074,d82cbae3424f5867] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:52:28.504 INFO 21062 --- [ocp-schedules-9,bddd318b9193df70,b76a267d3f497d89] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:52:28.504 INFO 21062 --- [ocp-schedules-21,37fbc0ccf0159a36,6afac283a5602b4a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:52:28.504 INFO 21062 --- [ocp-schedules-12,74c248e3be037bbb,7e634b855a9032de] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 12:52:28.504 INFO 21062 --- [ocp-schedules-27,7bdddc94cbb51791,7dfc6c89baa47058] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:28.504 INFO 21062 --- [ocp-schedules-11,541146f00d2b28e9,de3ae4034dc798a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:52:28.507 INFO 21062 --- [ocp-schedules-12,74c248e3be037bbb,7e634b855a9032de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:28.512 INFO 21062 --- [ocp-schedules-6,8d8a57ba45c4ea89,80be20476824c34f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:52:28.512 INFO 21062 --- [ocp-schedules-1,ee7c3a430500e0d7,87ffa27ce48e2476] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:52:28.512 INFO 21062 --- [ocp-schedules-6,8d8a57ba45c4ea89,80be20476824c34f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:52:28.512 INFO 21062 --- [ocp-schedules-1,ee7c3a430500e0d7,87ffa27ce48e2476] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:52:28.512 INFO 21062 --- [ocp-schedules-8,ffa16bbd7e32187d,f9daf0e5894eace2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:52:28.512 INFO 21062 --- [ocp-schedules-8,ffa16bbd7e32187d,f9daf0e5894eace2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:52:28.528 INFO 21062 --- [ocp-schedules-14,7aba9ebaa9e96461,9f2a10c554b6215a] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 12:52:28.528 INFO 21062 --- [ocp-schedules-14,7aba9ebaa9e96461,9f2a10c554b6215a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 12:52:28.553 INFO 21062 --- [ocp-async-3,2d4e93d559af8cce,d73deadf35e905ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:52:28.555 INFO 21062 --- [ocp-schedules-30,8da798e9a32be8e6,2ad6af6b5d400783] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:52:28.555 INFO 21062 --- [ocp-schedules-30,8da798e9a32be8e6,2ad6af6b5d400783] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:52:28.572 INFO 21062 --- [pool-12-thread-3,f5244aab9e068643,b0bb18a9716b4290] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 12:52:28.573 INFO 21062 --- [pool-12-thread-1,f5244aab9e068643,50bf0bb190a61197] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 12:52:28.573 INFO 21062 --- [pool-12-thread-4,f5244aab9e068643,f1e95ab74fd89532] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 12:52:28.574 INFO 21062 --- [pool-12-thread-2,f5244aab9e068643,8dae1500b6f646ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 12:52:28.577 INFO 21062 --- [ForkJoinPool-11-worker-6,f5244aab9e068643,49c47ad3e3bee0bb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 12:52:28.577 INFO 21062 --- [ForkJoinPool-11-worker-6,f5244aab9e068643,49c47ad3e3bee0bb] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 12:52:28.577 INFO 21062 --- [ForkJoinPool-11-worker-6,f5244aab9e068643,49c47ad3e3bee0bb] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 12:52:28.577 INFO 21062 --- [pool-12-thread-3,f5244aab9e068643,b0bb18a9716b4290] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 12:52:28.579 INFO 21062 --- [pool-12-thread-2,f5244aab9e068643,8dae1500b6f646ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 12:52:28.579 INFO 21062 --- [pool-12-thread-2,f5244aab9e068643,8dae1500b6f646ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 12:52:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:52:28.588676+08:00, lastProcessTime=2024-11-04T12:52:18.588415+08:00 2024-11-04 12:52:28.600 INFO 21062 --- [pool-12-thread-1,f5244aab9e068643,50bf0bb190a61197] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 12:52:28.604 INFO 21062 --- [pool-12-thread-1,f5244aab9e068643,50bf0bb190a61197] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 12:52:28.606 INFO 21062 --- [ocp-schedules-18,e201c523bf4ef446,da9799826ea11b1d] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 12:52:28.606 INFO 21062 --- [ForkJoinPool-11-worker-6,f5244aab9e068643,4c770c23fbb907d5] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 12:52:28.606 INFO 21062 --- [pool-12-thread-4,f5244aab9e068643,f1e95ab74fd89532] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 12:52:28.606 INFO 21062 --- [ocp-schedules-18,e201c523bf4ef446,da9799826ea11b1d] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 12:52:28.606 INFO 21062 --- [ocp-schedules-18,e201c523bf4ef446,da9799826ea11b1d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=322 2024-11-04 12:52:28.607 INFO 21062 --- [ocp-schedules-21,62f8a2cc5bafa0a9,5780bcbda7624b06] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:52:28.607 INFO 21062 --- [pool-12-thread-1,f5244aab9e068643,50bf0bb190a61197] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 12:52:28.613 INFO 21062 --- [ocp-schedules-9,ba0ffe71b7752551,8134e2b1259d3208] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 12:52:28.619 INFO 21062 --- [ocp-schedules-18,e201c523bf4ef446,da9799826ea11b1d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=322 2024-11-04 12:52:28.625 INFO 21062 --- [ocp-schedules-9,ba0ffe71b7752551,8134e2b1259d3208] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 12:52:28.644 WARN 21062 --- [ocp-async-2,6d97299a31a25a37,f01b4c89c64686c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:28.646 INFO 21062 --- [ocp-schedules-1,086b5925c788f6dc,7ce9cebc83f9ba48] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:52:28.646 INFO 21062 --- [ocp-schedules-1,086b5925c788f6dc,7ce9cebc83f9ba48] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:52:28.649 INFO 21062 --- [ocp-async-7,58d829eaa400696b,47715237ad588e18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:28.651 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:52:28.651 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:52:28.655 INFO 21062 --- [ocp-schedules-14,b573e673ccc58357,6e8ec945eefd0aeb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:52:28.656 INFO 21062 --- [ocp-schedules-23,6bd7312df3cf06bf,7777b0d0da7d1f4c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 12:52:28.658 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:52:28.658 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:52:28.658 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:52:28.659 INFO 21062 --- [ocp-schedules-23,6bd7312df3cf06bf,7777b0d0da7d1f4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:28.670 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:52:28.670 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:52:28.670 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:52:28.674 INFO 21062 --- [ocp-schedules-3,44ae2b57feaf6ae7,814274f16afabd74] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:52:28.698 INFO 21062 --- [ocp-schedules-29,84216b7e914e31eb,8ad1d3bfb42a52ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:52:28.703 INFO 21062 --- [ocp-schedules-29,84216b7e914e31eb,8ad1d3bfb42a52ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:52:28.703 INFO 21062 --- [ocp-schedules-29,84216b7e914e31eb,8ad1d3bfb42a52ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:52:28.735 INFO 21062 --- [ocp-schedules-20,34c34b40fca79665,2b61478453541a4c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 12:52:28.735 INFO 21062 --- [ocp-schedules-20,34c34b40fca79665,2b61478453541a4c] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 12:52:28.740 ERROR 21062 --- [ocp-schedules-23,6bd7312df3cf06bf,7777b0d0da7d1f4c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:52:28.740 INFO 21062 --- [ocp-schedules-23,6bd7312df3cf06bf,7777b0d0da7d1f4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 12:52:28.760 INFO 21062 --- [pool-23-thread-1,34c34b40fca79665,28ba3e0e1aba675b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 12:52:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:52:28.588676+08:00 2024-11-04 12:52:28.787 WARN 21062 --- [ocp-schedules-20,34c34b40fca79665,2b61478453541a4c] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:52:28.787 INFO 21062 --- [ocp-schedules-20,34c34b40fca79665,2b61478453541a4c] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 12:52:28.814 INFO 21062 --- [ocp-schedules-16,78b651d5f68b2fd8,47dce0f5190a4e74] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:52:28.814 INFO 21062 --- [ocp-schedules-16,78b651d5f68b2fd8,47dce0f5190a4e74] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:52:28.814 INFO 21062 --- [ocp-schedules-16,78b651d5f68b2fd8,47dce0f5190a4e74] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:52:28.814 INFO 21062 --- [ocp-schedules-16,78b651d5f68b2fd8,47dce0f5190a4e74] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:52:28.815 INFO 21062 --- [ocp-schedules-16,78b651d5f68b2fd8,47dce0f5190a4e74] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:52:28.815 INFO 21062 --- [ocp-schedules-16,78b651d5f68b2fd8,47dce0f5190a4e74] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:52:28.849 INFO 21062 --- [ocp-schedules-30,3cd64d8aac3d576a,4026ce9c81ead43f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:52:29.070 INFO 21062 --- [ocp-schedules-7,8f339f6977d8ee50,6e20b4edf1e1575f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:52:29.070 INFO 21062 --- [ocp-schedules-7,8f339f6977d8ee50,6e20b4edf1e1575f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:52:29.085 INFO 21062 --- [ocp-schedules-24,3d7d5c01cb568348,67722eb6b11ad982] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:52:29.090 INFO 21062 --- [ocp-schedules-24,3d7d5c01cb568348,67722eb6b11ad982] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:52:29.090 INFO 21062 --- [ocp-schedules-24,3d7d5c01cb568348,67722eb6b11ad982] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:52:29.095 INFO 21062 --- [ocp-schedules-7,8f339f6977d8ee50,6e20b4edf1e1575f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:52:29.095 INFO 21062 --- [ocp-schedules-7,8f339f6977d8ee50,6e20b4edf1e1575f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:52:29.103 INFO 21062 --- [ocp-schedules-2,1d3736952e2c233a,e0af7e4866b344cf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:52:29.105 INFO 21062 --- [ocp-schedules-15,4eb24d3ff11c5147,a6f06492dd11fd65] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:52:29.108 INFO 21062 --- [ocp-schedules-15,4eb24d3ff11c5147,a6f06492dd11fd65] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:29.130 INFO 21062 --- [ocp-schedules-13,f2a1e9fba8c0d705,b8eab61ed02f4705] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:52:29.130 INFO 21062 --- [ocp-schedules-13,f2a1e9fba8c0d705,b8eab61ed02f4705] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:52:29.130 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:52:29.130 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:52:29.140 INFO 21062 --- [ocp-schedules-13,f2a1e9fba8c0d705,b8eab61ed02f4705] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:52:29.140 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:52:29.141 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:52:29.142 WARN 21062 --- [ocp-schedules-13,f2a1e9fba8c0d705,b8eab61ed02f4705] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:52:29.142 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:52:29.142 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:52:29.143 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:52:29.143 INFO 21062 --- [ocp-schedules-13,f2a1e9fba8c0d705,b8eab61ed02f4705] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:52:29.143 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:52:29.144 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:52:29.144 WARN 21062 --- [ocp-schedules-13,f2a1e9fba8c0d705,b8eab61ed02f4705] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:52:29.144 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:52:29.150 ERROR 21062 --- [ocp-schedules-13,f2a1e9fba8c0d705,b8eab61ed02f4705] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 12:52:29.161 ERROR 21062 --- [ocp-schedules-15,4eb24d3ff11c5147,a6f06492dd11fd65] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:52:29.161 INFO 21062 --- [ocp-schedules-15,4eb24d3ff11c5147,a6f06492dd11fd65] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:52:29.217 INFO 21062 --- [ocp-schedules-18,4d07b1e95dd8383b,9b1e9322fab0e5c3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:52:29.216+08:00, clusters=[1] 2024-11-04 12:52:29.219 INFO 21062 --- [ocp-schedules-18,4d07b1e95dd8383b,9b1e9322fab0e5c3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:52:29.288 INFO 21062 --- [ocp-schedules-9,9189c7c441dc7d96,9bc35ba5b97c710b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:52:29.288+08:00, clusters=[1] 2024-11-04 12:52:29.291 INFO 21062 --- [ocp-schedules-9,9189c7c441dc7d96,9bc35ba5b97c710b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:52:29.807 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:29.808 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:29.884 INFO 21062 --- [ocp-schedules-10,332ee43a76b89886,5dab4bda6e014500] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:52:29.884 INFO 21062 --- [ocp-schedules-10,be9e4b2d84c8ca1d,5ca33cd6918a5fbf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:52:29.888 INFO 21062 --- [ocp-schedules-10,be9e4b2d84c8ca1d,5ca33cd6918a5fbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:29.915 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:52:29.915 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:52:29.983 INFO 21062 --- [prometheus-minute-schedule-1,81e6aea8f8045798,a940b2ca4da14651] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:52:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:52:30.334 INFO 21062 --- [ocp-schedules-13,a77ceb66ccc3f529,87b69686e25d1691] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:52:30.337 INFO 21062 --- [ocp-schedules-13,a77ceb66ccc3f529,87b69686e25d1691] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:30.419 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:52:30.420 ERROR 21062 --- [ocp-schedules-13,a77ceb66ccc3f529,87b69686e25d1691] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:52:30.420 INFO 21062 --- [ocp-schedules-13,a77ceb66ccc3f529,87b69686e25d1691] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:52:30.644 WARN 21062 --- [ocp-async-6,c56e151b49ddadd6,e424d84ca2c8421e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:30.649 INFO 21062 --- [ocp-async-1,a0378dca77b0a247,13ffb6f8a12b7f97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:32.644 WARN 21062 --- [ocp-async-5,61575fd98b72928d,d4752554dea74ca5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:32.649 INFO 21062 --- [ocp-async-3,6e027f2bf70c04ef,d8783bd8ccf11185] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:32.808 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:32.809 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a4e7aac20207c811,aca6e74cf7d4ddc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:33.643 INFO 21062 --- [ocp-schedules-17,10551e0a7e425f82,ab8280893c646d31] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:52:34.644 WARN 21062 --- [ocp-async-4,77e69aa25673a1c2,483dd0a2130319cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:34.649 INFO 21062 --- [ocp-async-9,2b9705a177255ec5,451624b0b37eb3f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:35.809 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:35.810 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:35.983 INFO 21062 --- [prometheus-minute-schedule-1,3243e161406452ef,92c8f7c8123aed85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:36.644 WARN 21062 --- [ocp-async-7,b61bf41dc4e7ee7c,419b29e5861eee4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:36.649 INFO 21062 --- [ocp-async-8,e44666d39df13fb8,93f4accadd550bb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:38.505 INFO 21062 --- [ocp-schedules-11,541146f00d2b28e9,de3ae4034dc798a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:52:38.505 INFO 21062 --- [ocp-schedules-27,7bdddc94cbb51791,7dfc6c89baa47058] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:52:38.508 INFO 21062 --- [ocp-schedules-12,74c248e3be037bbb,7e634b855a9032de] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 12:52:38.554 INFO 21062 --- [ocp-async-1,6e93b672866b8020,1d30fdcc5ce23757] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:52:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:52:38.587900+08:00, lastProcessTime=2024-11-04T12:52:28.588676+08:00 2024-11-04 12:52:38.644 WARN 21062 --- [ocp-async-10,a61e83802fb7df38,9e3b8e077626ca6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:38.649 INFO 21062 --- [ocp-async-4,e18d6eab6e852449,f2cfe90ea1890421] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:52:38.587900+08:00 2024-11-04 12:52:38.810 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:38.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:38.852 INFO 21062 --- [ocp-schedules-11,d0755265333ff177,10b8b9b9a36cda24] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:52:38.983 INFO 21062 --- [prometheus-minute-schedule-1,08b927459239d3d8,0b606e440a9c5fd4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:52:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:52:40.645 WARN 21062 --- [ocp-async-2,7fa39e58a0c55fcd,7d68311c80660fdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:40.650 INFO 21062 --- [ocp-async-7,96e1cc85a297d320,726e3d95c3ebd6fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:52:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:52:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:52:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:52:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:52:41.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:41.812 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:41.829 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:52:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 12:52:41.983 INFO 21062 --- [prometheus-minute-schedule-1,374a32752a932355,a7b72f39370ce239] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:42.645 WARN 21062 --- [ocp-async-7,1ae0935d9a542ace,0a20eb73cbe448af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:42.650 INFO 21062 --- [ocp-async-1,9e7a4246095721ad,cc2214f437cdbf01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:43.685 INFO 21062 --- [ocp-schedules-12,6bef7a7fabf0e18f,7f8b6895e7802b49] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:52:44.645 WARN 21062 --- [ocp-async-1,b03911d28a9d390e,7c923a74ae770a4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:44.650 INFO 21062 --- [ocp-async-10,9649e31d4fac592f,6979bb54dc77eda1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:44.812 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:44.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d8fb9f7fe78197e0,b6f17f1adcd3516e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:46.645 WARN 21062 --- [ocp-async-10,112fd4ad5d4d6374,6dfd1eb3fc48058d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:46.650 INFO 21062 --- [ocp-async-2,53612bfe437e6e17,bc15ee353f90103c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:47.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:47.815 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:47.983 INFO 21062 --- [prometheus-minute-schedule-1,58523129627f7d21,eca5c995436c0b8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:48.554 INFO 21062 --- [ocp-async-9,ca45e8211cb03803,8231ee0fe6e31677] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:52:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:52:48.589717+08:00, lastProcessTime=2024-11-04T12:52:38.587900+08:00 2024-11-04 12:52:48.679 WARN 21062 --- [ocp-async-7,87c54f49cbd3096c,a9caa76a312e1c05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:48.679 INFO 21062 --- [ocp-async-6,5b438eaec7305fad,a9d048dd72d8fbd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:48.799 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:52:48.589717+08:00 2024-11-04 12:52:48.854 INFO 21062 --- [ocp-schedules-5,7be4c46bc6ddc2fa,a7eb32751907155a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:52:49.092 INFO 21062 --- [ocp-schedules-3,fb58d9b900a5f205,3d9c0ac9d005f9c9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:52:49.098 INFO 21062 --- [ocp-schedules-3,fb58d9b900a5f205,3d9c0ac9d005f9c9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:52:49.098 INFO 21062 --- [ocp-schedules-3,fb58d9b900a5f205,3d9c0ac9d005f9c9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:52:50.645 WARN 21062 --- [ocp-async-6,c484c7c7838c4747,c05469a617de4241] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:50.679 INFO 21062 --- [ocp-async-5,d3c38234e76f5094,bcd60515bca10d8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:50.815 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:50.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:50.983 INFO 21062 --- [prometheus-minute-schedule-1,9eb17cddad1555fa,6db569bcf8f4e4de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:52.645 WARN 21062 --- [ocp-async-5,b01c8911d49e3573,5c1c77dd75f62ea0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:52.680 INFO 21062 --- [ocp-async-2,9128db322becf44d,4e8d395536dfc1fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:53.711 INFO 21062 --- [ocp-schedules-11,34fddf50c4348e60,1cdbabd39c863377] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:52:53.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:53.817 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:53.983 INFO 21062 --- [prometheus-minute-schedule-1,86eefa43cd4d0894,4d75d5bbd9fd6b62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:54.646 WARN 21062 --- [ocp-async-2,99855e138b0d2e76,8a2fe12ec9b481b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:54.680 INFO 21062 --- [ocp-async-7,2aea4939149f285a,f14ee73adbf15cee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:56.646 WARN 21062 --- [ocp-async-7,10a8671f1a5db57c,af9ce3901acc07cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:56.680 INFO 21062 --- [ocp-async-6,95bc411e06e62088,2e4f118cab0a3bd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:56.817 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:56.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:56.983 INFO 21062 --- [prometheus-minute-schedule-1,de57274ea362b2b4,eb6aa7972f8d394f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:52:58.500 INFO 21062 --- [ocp-schedules-24,5615ea6e3431e169,1f866ac36556e5ec] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:52:58.500 INFO 21062 --- [ocp-schedules-6,d71fc95e1ec8bb93,339d56341ad242b4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:52:58.502 INFO 21062 --- [ocp-schedules-9,0caf205b422203de,0d9018b900ddd7e8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:52:58.503 INFO 21062 --- [ocp-schedules-24,5615ea6e3431e169,1f866ac36556e5ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:52:58.505 INFO 21062 --- [ocp-schedules-21,3deb37e4fd74c8d2,c90a7a42b2ec5a6d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:52:58.505 INFO 21062 --- [ocp-schedules-6,d71fc95e1ec8bb93,339d56341ad242b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:58.510 INFO 21062 --- [ocp-schedules-5,71a3dce4449c50f4,498600a7c9d90b2f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:52:58.510 INFO 21062 --- [ocp-schedules-5,71a3dce4449c50f4,498600a7c9d90b2f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:52:58.511 INFO 21062 --- [ocp-schedules-29,102cc76fc1a9d602,9b64ef1458aa5ab6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:52:58.511 INFO 21062 --- [ocp-schedules-29,102cc76fc1a9d602,9b64ef1458aa5ab6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:52:58.511 INFO 21062 --- [ocp-schedules-8,5246f9dee8549f45,fe4e29b8d2c688b7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:52:58.511 INFO 21062 --- [ocp-schedules-8,5246f9dee8549f45,fe4e29b8d2c688b7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:52:58.554 INFO 21062 --- [ocp-async-3,eda96b65d31c01af,865d3ce759e3246d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:52:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:52:58.588025+08:00, lastProcessTime=2024-11-04T12:52:48.589717+08:00 2024-11-04 12:52:58.646 WARN 21062 --- [ocp-async-5,28af54a4423e5a22,5670f22b36397c9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:52:58.658 INFO 21062 --- [ocp-schedules-7,b3ab7ed8f4257178,30cef947ddb603ef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:52:58.680 INFO 21062 --- [ocp-async-9,bf0765d7d8d5ec0a,4a77514d2aafbd70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:52:58.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:52:58.588025+08:00 2024-11-04 12:52:58.857 INFO 21062 --- [ocp-schedules-23,79cabaf1e76b43a6,560f86316119bc69] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:52:59.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:52:59.820 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:52:59.915 INFO 21062 --- [ocp-schedules-10,be9e4b2d84c8ca1d,5ca33cd6918a5fbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:52:59.915 INFO 21062 --- [ocp-schedules-10,7a6d35e4ed25b710,55943b0a05510306] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:52:59.919 INFO 21062 --- [ocp-schedules-10,7a6d35e4ed25b710,55943b0a05510306] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:52:59.949 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:52:59.949 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:52:59.983 INFO 21062 --- [prometheus-minute-schedule-1,22120487c39d4f47,b7339ff257d13267] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730695980, interval=60 2024-11-04 12:53:00.646 WARN 21062 --- [ocp-async-2,6cfb65e2c17b93af,cc853b95d530d090] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:00.680 INFO 21062 --- [ocp-async-8,12572ba2dd2e1d0a,6e2f57d0664264a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:02.646 WARN 21062 --- [ocp-async-7,2c52f44380b45714,b8a33635bc08a1f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:02.681 INFO 21062 --- [ocp-async-3,215a882fb480375d,62e433ff4dd37706] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:02.820 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:02.821 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e41dcbbfc193a712,717c94290903bcef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:03.749 INFO 21062 --- [ocp-schedules-1,614339b9c6677876,152e7d419199c7d6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:53:04.647 WARN 21062 --- [ocp-async-6,5886e5b38a61b86c,d81d9b6b08d71e02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:04.681 INFO 21062 --- [ocp-async-4,0abf59ef5807612b,76fea46a9329e6f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:05.821 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:05.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:05.983 INFO 21062 --- [prometheus-minute-schedule-1,29bb01660c222a94,d7be75633faf468b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:06.647 WARN 21062 --- [ocp-async-9,ccc5d6a399128708,f6ca46cb92562cef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:06.681 INFO 21062 --- [ocp-async-1,c871d35f3d1fa25c,982c4ff71ad18ff2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:08.504 INFO 21062 --- [ocp-schedules-24,5615ea6e3431e169,1f866ac36556e5ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:53:08.506 INFO 21062 --- [ocp-schedules-6,d71fc95e1ec8bb93,339d56341ad242b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:53:08.554 INFO 21062 --- [ocp-async-1,bea541a508237eab,9dc02935c4978d9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:53:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:53:08.588449+08:00, lastProcessTime=2024-11-04T12:52:58.588025+08:00 2024-11-04 12:53:08.647 WARN 21062 --- [ocp-async-7,7e0b968526c73dd6,ca11fd9de52bcfba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:08.681 INFO 21062 --- [ocp-async-3,ae1f21b4567d7ab4,9c1a069344c144a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:08.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:53:08.588449+08:00 2024-11-04 12:53:08.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:08.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:08.860 INFO 21062 --- [ocp-schedules-24,77368f799da41630,2596b993006a42ac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:53:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8eec0dddb41b1d32,46a0bd25acd2cd3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:09.100 INFO 21062 --- [ocp-schedules-20,c7317aa7a7542e9a,1c05a1530e6f0bf9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:53:09.106 INFO 21062 --- [ocp-schedules-20,c7317aa7a7542e9a,1c05a1530e6f0bf9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:53:09.106 INFO 21062 --- [ocp-schedules-20,c7317aa7a7542e9a,1c05a1530e6f0bf9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:53:10.004 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:53:10.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:53:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:53:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:53:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:53:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:53:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:53:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:53:10.647 WARN 21062 --- [ocp-async-6,ef41bc8bea83cb1a,43c5aa2cad506ec7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:10.681 INFO 21062 --- [ocp-async-4,4577dc30df1cd146,0d49bc60b1ece10b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:11.114 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=1, affectRows=1 2024-11-04 12:53:11.114 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=1, validCount=1 2024-11-04 12:53:11.114 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=1, affectRows=1 2024-11-04 12:53:11.114 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=0, validCount=0 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=12, affectRows=12 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=7, affectRows=7 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=12, validCount=12 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=17, affectRows=17 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=7, validCount=7 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=17, validCount=17 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=33, affectRows=33 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=46, affectRows=46 2024-11-04 12:53:11.115 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=28, affectRows=28 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=28, validCount=28 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=20, affectRows=20 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=63, affectRows=63 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=33, validCount=33 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=20, validCount=20 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=46, validCount=46 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=26, affectRows=26 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=61, validCount=61 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=26, validCount=26 2024-11-04 12:53:11.116 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695920, nodeCount=68, affectRows=68 2024-11-04 12:53:11.117 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695920, nodeCount=68, validCount=68 2024-11-04 12:53:11.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:11.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:11.983 INFO 21062 --- [prometheus-minute-schedule-1,16034840c0ef7c39,0f83d94cb276d65e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:12.648 WARN 21062 --- [ocp-async-9,8231f9ff520e9213,8c5c8fdfc72a9d12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:12.682 INFO 21062 --- [ocp-async-2,7c649c3592cbd19f,5ff877da06e69c3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:13.774 INFO 21062 --- [ocp-schedules-5,8fa01cd4a7ddca20,e58cd691537dd14b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:53:13.907 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T12:47:13.88, end=2024-11-04T12:52:13.904} 2024-11-04 12:53:13.907 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T12:47:13.88, end=2024-11-04T12:52:13.904} 2024-11-04 12:53:13.919 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 12:53:13.919 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 12:53:13.919 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:53:13.919 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:53:13.984 INFO 21062 --- [prometheus-cache-monitor1,3d68631d9dd6ce26,7d62976297113e9d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:53:13.984 INFO 21062 --- [prometheus-cache-monitor1,3d68631d9dd6ce26,7d62976297113e9d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:53:14.192 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:49:14.167, end=2024-11-04T12:51:14.189} 2024-11-04 12:53:14.196 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:53:14.196 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:53:14.648 WARN 21062 --- [ocp-async-8,a398127f2f3c2acb,c4fc4d9e65ae71c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:14.682 INFO 21062 --- [ocp-async-7,1fc3fbdd7d5f8906,52d301a2449467c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:14.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:14.826 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9e04741dc3daf474,40f580907b4038c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:16.648 WARN 21062 --- [ocp-async-3,6e4a5d78c21f3586,a607a1f797cae374] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:16.682 INFO 21062 --- [ocp-async-6,d3dc6b25e37ee87c,1c54d1d70b753b85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:17.826 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:17.829 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9dd0dc6b9e74273a,d28584d540274c8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:18.554 INFO 21062 --- [ocp-async-6,0116fe4c8a9d0139,3c8bc783372351e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:53:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:53:18.588573+08:00, lastProcessTime=2024-11-04T12:53:08.588449+08:00 2024-11-04 12:53:18.629 INFO 21062 --- [ocp-schedules-13,9dd67a378aa77246,2bcffc815b5860f8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:53:18.632 INFO 21062 --- [ocp-schedules-13,9dd67a378aa77246,2bcffc815b5860f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:53:18.633 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:53:18.648 WARN 21062 --- [ocp-async-9,e5f3a13655a09c7a,6e381a1a348e8670] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:18.682 INFO 21062 --- [ocp-async-2,1a8b161693482fe3,eab3253e67524ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:53:18.588573+08:00 2024-11-04 12:53:18.863 INFO 21062 --- [ocp-schedules-25,71913ed65d38f79c,1a6291a11b9dc306] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:53:20.648 WARN 21062 --- [ocp-async-8,29a01ddfd5b0fc7c,745df17b703973fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:20.683 INFO 21062 --- [ocp-async-7,8e74b66698d9015e,aeebf29b6ade265f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:20.829 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:20.830 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5a59e28efab0c727,8d465ca9c2d20788] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:22.648 WARN 21062 --- [ocp-async-3,94fe225d80195daa,20ee3e9f54a469e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:22.683 INFO 21062 --- [ocp-async-6,ce5b7f7dd2cefe71,78fb72d31a827077] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:23.830 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:23.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:23.849 INFO 21062 --- [ocp-schedules-24,2feae16668125132,ab78f67106055c1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:53:23.983 INFO 21062 --- [prometheus-minute-schedule-1,cbe05d7f81c6f2e3,22458635a669e33b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:24.649 WARN 21062 --- [ocp-async-4,908c40366693276b,aca5fe80077b9c44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:24.683 INFO 21062 --- [ocp-async-1,54fab82809309bb8,b27239144cf738d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:26.649 WARN 21062 --- [ocp-async-2,fd2ac51768e94433,32baa4774bd285ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:26.683 INFO 21062 --- [ocp-async-10,a2f9368518f78a6a,ea1fd93f27ff74de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:26.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:26.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f8296ce9640e0f2a,c8ec7211e2b0d733] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:26.984 INFO 21062 --- [prometheus-hour-schedule-1,bf013a276a86616e,c4551167cdaa4b27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:53:28.498 INFO 21062 --- [ocp-schedules-20,2973734c49491e71,455558fdc1fcb609] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:53:28.499 INFO 21062 --- [ocp-schedules-15,393b550f359aec16,7b533c3605ba6b54] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:53:28.500 INFO 21062 --- [ocp-schedules-6,44e1779bd7d49b0f,4d2307d66e3ba407] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:53:28.500 INFO 21062 --- [ocp-schedules-23,a619d9bf39fe657d,c0edf06821f7dac2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:53:28.502 INFO 21062 --- [ocp-schedules-20,2973734c49491e71,455558fdc1fcb609] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:53:28.502 INFO 21062 --- [ocp-schedules-18,6dd1b795880ac81b,33a4ddbc96001da5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:53:28.504 INFO 21062 --- [ocp-schedules-6,44e1779bd7d49b0f,4d2307d66e3ba407] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:53:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:53:28.505 INFO 21062 --- [ocp-schedules-23,a619d9bf39fe657d,c0edf06821f7dac2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:53:28.506 INFO 21062 --- [ocp-schedules-2,25d5dcd2257ce245,54c114ef545cc21d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:53:28.507 INFO 21062 --- [ocp-schedules-15,393b550f359aec16,7b533c3605ba6b54] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:53:28.511 INFO 21062 --- [ocp-schedules-30,9c2d224178e5c282,f66992c66338f29b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:53:28.511 INFO 21062 --- [ocp-schedules-7,9925a99c905756c0,113d8b35f4ffaad5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:53:28.511 INFO 21062 --- [ocp-schedules-30,9c2d224178e5c282,f66992c66338f29b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:53:28.511 INFO 21062 --- [ocp-schedules-16,1437e098a3530e67,b34d7482b5c654e4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:53:28.511 INFO 21062 --- [ocp-schedules-7,9925a99c905756c0,113d8b35f4ffaad5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:53:28.511 INFO 21062 --- [ocp-schedules-16,1437e098a3530e67,b34d7482b5c654e4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:53:28.547 INFO 21062 --- [ocp-schedules-21,8f4e1060bf4331cf,e5864986cf1c5467] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 12:53:28.547 INFO 21062 --- [ocp-schedules-21,8f4e1060bf4331cf,e5864986cf1c5467] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 12:53:28.554 INFO 21062 --- [ocp-async-3,1ec1a1c15028fb3d,7eb596996d910d79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:53:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:53:28.587738+08:00, lastProcessTime=2024-11-04T12:53:18.588573+08:00 2024-11-04 12:53:28.610 INFO 21062 --- [ocp-schedules-8,b219d8262ffaead0,67517fc44b61db0b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:53:28.633 INFO 21062 --- [ocp-schedules-13,9dd67a378aa77246,2bcffc815b5860f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:53:28.649 WARN 21062 --- [ocp-async-6,a547dcf53ebe16e5,113eaa7632e6e333] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:28.650 INFO 21062 --- [ocp-schedules-17,e54dc5f6ae5a8c3f,3414251c678c7443] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:53:28.650 INFO 21062 --- [ocp-schedules-17,e54dc5f6ae5a8c3f,3414251c678c7443] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:53:28.659 INFO 21062 --- [ocp-schedules-18,1156725edc57cbbc,78b46e1846a3d3ab] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:53:28.683 INFO 21062 --- [ocp-async-9,d1769352ae5e6751,13b249090d80a0ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:28.703 INFO 21062 --- [ocp-schedules-7,b8e46cd3f1b9dca3,a0e04ccbe2cd98fd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:53:28.708 INFO 21062 --- [ocp-schedules-7,b8e46cd3f1b9dca3,a0e04ccbe2cd98fd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:53:28.708 INFO 21062 --- [ocp-schedules-7,b8e46cd3f1b9dca3,a0e04ccbe2cd98fd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:53:28.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:53:28.587738+08:00 2024-11-04 12:53:28.816 INFO 21062 --- [ocp-schedules-20,c01b3b099901c2ae,cfeca1cb68682c6f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:53:28.816 INFO 21062 --- [ocp-schedules-20,c01b3b099901c2ae,cfeca1cb68682c6f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:53:28.817 INFO 21062 --- [ocp-schedules-20,c01b3b099901c2ae,cfeca1cb68682c6f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:53:28.817 INFO 21062 --- [ocp-schedules-20,c01b3b099901c2ae,cfeca1cb68682c6f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:53:28.817 INFO 21062 --- [ocp-schedules-20,c01b3b099901c2ae,cfeca1cb68682c6f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:53:28.817 INFO 21062 --- [ocp-schedules-20,c01b3b099901c2ae,cfeca1cb68682c6f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:53:28.865 INFO 21062 --- [ocp-schedules-25,60549391933241ca,3b8d49e708408a8e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:53:29.095 INFO 21062 --- [ocp-schedules-24,7b83e0bae09e2ccf,bbfa45459c252190] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:53:29.095 INFO 21062 --- [ocp-schedules-24,7b83e0bae09e2ccf,bbfa45459c252190] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:53:29.109 INFO 21062 --- [ocp-schedules-3,7d26dd2763a436c0,98cf754b22200bb4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:53:29.114 INFO 21062 --- [ocp-schedules-3,7d26dd2763a436c0,98cf754b22200bb4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:53:29.114 INFO 21062 --- [ocp-schedules-3,7d26dd2763a436c0,98cf754b22200bb4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:53:29.121 INFO 21062 --- [ocp-schedules-24,7b83e0bae09e2ccf,bbfa45459c252190] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:53:29.121 INFO 21062 --- [ocp-schedules-24,7b83e0bae09e2ccf,bbfa45459c252190] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:53:29.129 INFO 21062 --- [ocp-schedules-27,162b5f45f0953715,3c8e6e4f71546429] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:53:29.158 INFO 21062 --- [ocp-schedules-14,6dc81ff903d17f32,7767bf24ee9da8b5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:53:29.158 INFO 21062 --- [ocp-schedules-14,6dc81ff903d17f32,7767bf24ee9da8b5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:53:29.158 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:53:29.158 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:53:29.168 INFO 21062 --- [ocp-schedules-14,6dc81ff903d17f32,7767bf24ee9da8b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:53:29.168 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:53:29.168 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:53:29.170 WARN 21062 --- [ocp-schedules-14,6dc81ff903d17f32,7767bf24ee9da8b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:53:29.170 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:53:29.170 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:53:29.171 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:53:29.171 INFO 21062 --- [ocp-schedules-14,6dc81ff903d17f32,7767bf24ee9da8b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:53:29.171 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:53:29.172 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:53:29.172 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:53:29.172 WARN 21062 --- [ocp-schedules-14,6dc81ff903d17f32,7767bf24ee9da8b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:53:29.179 ERROR 21062 --- [ocp-schedules-14,6dc81ff903d17f32,7767bf24ee9da8b5] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:53:29.244 INFO 21062 --- [ocp-schedules-28,910c49626867ad6c,09d078d94b95cfc4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:53:29.244+08:00, clusters=[1] 2024-11-04 12:53:29.245 INFO 21062 --- [ocp-schedules-28,910c49626867ad6c,09d078d94b95cfc4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:53:29.315 INFO 21062 --- [ocp-schedules-19,3db4c817c16e1432,70c4c3d15ffb22e7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:53:29.315+08:00, clusters=[1] 2024-11-04 12:53:29.319 INFO 21062 --- [ocp-schedules-19,3db4c817c16e1432,70c4c3d15ffb22e7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:53:29.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:29.833 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:29.949 INFO 21062 --- [ocp-schedules-10,7a6d35e4ed25b710,55943b0a05510306] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:53:29.949 INFO 21062 --- [ocp-schedules-10,27f210b96c5e9f82,336fb9cab68f25a0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:53:29.953 INFO 21062 --- [ocp-schedules-10,27f210b96c5e9f82,336fb9cab68f25a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:53:29.980 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:53:29.980 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:53:29.983 INFO 21062 --- [prometheus-minute-schedule-1,a860827edaa31bec,7fa04cfa1bbe22a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:53:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:53:30.421 INFO 21062 --- [ocp-schedules-22,0198586cf9c407f5,5751d0cec631cb3c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:53:30.424 INFO 21062 --- [ocp-schedules-22,0198586cf9c407f5,5751d0cec631cb3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:53:30.512 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:53:30.516 ERROR 21062 --- [ocp-schedules-22,0198586cf9c407f5,5751d0cec631cb3c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:53:30.516 INFO 21062 --- [ocp-schedules-22,0198586cf9c407f5,5751d0cec631cb3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:53:30.649 WARN 21062 --- [ocp-async-9,ed73988a5cecb41f,089cf769e7692fe3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:30.683 INFO 21062 --- [ocp-async-10,c2d7517dd1ef9d29,8348ba96cea0ec7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:32.649 WARN 21062 --- [ocp-async-10,446263c1fc662f13,6256e230e6d1bb58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:32.684 INFO 21062 --- [ocp-async-3,0dda6fd5bb778987,3fdde5940091da4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:32.833 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:32.834 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:32.983 INFO 21062 --- [prometheus-minute-schedule-1,48a25712656d0c0b,7c0dfefb603f9888] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:33.874 INFO 21062 --- [ocp-schedules-17,c5bfb655f90d041b,fc980946780bdb3b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:53:34.649 WARN 21062 --- [ocp-async-3,98cfd1e397aef2da,b0cc253249c647c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:34.684 INFO 21062 --- [ocp-async-4,e33cad658da9a9a7,785bfa595d36317f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:35.834 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:35.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5a747adbcc66f4f4,8b3fc596ded1329e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:36.650 WARN 21062 --- [ocp-async-4,e22daa9705574fa4,ac3c6bf278d1c305] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:36.684 INFO 21062 --- [ocp-async-2,d47afd58ecdf5f62,531ce755d6aa4644] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:38.504 INFO 21062 --- [ocp-schedules-6,44e1779bd7d49b0f,4d2307d66e3ba407] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:53:38.505 INFO 21062 --- [ocp-schedules-23,a619d9bf39fe657d,c0edf06821f7dac2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:53:38.555 INFO 21062 --- [ocp-async-2,a8cf2fde43148c0d,563a41af5d02b262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:53:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:53:38.587839+08:00, lastProcessTime=2024-11-04T12:53:28.587738+08:00 2024-11-04 12:53:38.650 WARN 21062 --- [ocp-async-8,929f0320f08bb095,70db02965b2b5d59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:38.684 INFO 21062 --- [ocp-async-5,176875a5311d7bfe,5661397ea92e2f14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:53:38.587839+08:00 2024-11-04 12:53:38.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:38.836 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:38.867 INFO 21062 --- [ocp-schedules-6,2e880cc7cd9f8cd2,e79e1713b14bfc58] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:53:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5311f0fbf2714c34,524e4d7db33849d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:40.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:53:40.004 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:53:40.650 WARN 21062 --- [ocp-async-5,902cfc5e77111b63,591d34d7e5cd4085] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:40.684 INFO 21062 --- [ocp-async-9,3e03c6410fd6713f,83a9742a45b80cb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:41.836 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:41.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:41.983 INFO 21062 --- [prometheus-minute-schedule-1,dcbde105a4e578c9,b247e9de4deb0bfa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:42.440 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:53:42.440 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:53:42.450 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:53:42.450 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:53:42.450 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:53:42.450 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:53:42.456 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:53:42.456 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:53:42.456 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:53:42.456 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:53:42.492 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:53:42.502 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:53:42.502 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:53:42.503 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:53:42.503 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:53:42.503 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:53:42.511 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:53:42.650 WARN 21062 --- [ocp-async-9,d763b9be89a5712e,81851d8325710031] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:42.684 INFO 21062 --- [ocp-async-10,f49ed9af7d8ad8bc,1512ff7f2453172c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:43.063 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:53:43.621 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.16438617S 2024-11-04 12:53:43.621 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:53:43.727 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:53:43.727 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:53:43.758 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:53:43.816 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:53:43.816 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:53:43.816 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:53:43.816 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:53:43.816 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:53:43.816 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:53:43.827 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:53:43.915 INFO 21062 --- [ocp-schedules-9,bd60ef62e516c907,3fd0963c986d94f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:53:44.019 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] o.s.boot.SpringApplication : Started application in 1.568 seconds (JVM running for 1437.698) 2024-11-04 12:53:44.019 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.569143877S 2024-11-04 12:53:44.487 INFO 21062 --- [ocp-schedules-8,7ab04c145006fe39,375d88841ad2f3ef] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:53:44.650 WARN 21062 --- [ocp-async-10,dbac6307aa6a6be1,5a1779bb0368b1e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:44.685 INFO 21062 --- [ocp-async-7,d5799fd18d84d51d,a1821d95ef57d31b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:44.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:44.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:44.983 INFO 21062 --- [prometheus-minute-schedule-1,10ef07dfa079980b,1e53033bbcac94e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:46.651 WARN 21062 --- [ocp-async-7,a30dec519c33de21,5219c6f043900d73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:46.685 INFO 21062 --- [ocp-async-6,4f352e5c601fb491,69f262ce3d84860c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:47.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:47.839 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:47.983 INFO 21062 --- [prometheus-minute-schedule-1,aa6018596da2692e,006b2fcd3c602fcb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:48.555 INFO 21062 --- [ocp-async-6,a4de4a7256e23ea3,9d7ef61723edd994] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:53:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:53:48.587787+08:00, lastProcessTime=2024-11-04T12:53:38.587839+08:00 2024-11-04 12:53:48.651 WARN 21062 --- [ocp-async-9,5caaacde4dbcc0b3,1c53289acdf10f8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:48.685 INFO 21062 --- [ocp-async-2,75102f537355d01b,9ce9b35e6c795ac4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:48.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:53:48.587787+08:00 2024-11-04 12:53:48.871 INFO 21062 --- [ocp-schedules-12,0036eee578b53b6d,7731dbd52aac270a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:53:49.117 INFO 21062 --- [ocp-schedules-6,ec946ff7112599b7,a6ea6109ee8489d2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:53:49.124 INFO 21062 --- [ocp-schedules-6,ec946ff7112599b7,a6ea6109ee8489d2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:53:49.124 INFO 21062 --- [ocp-schedules-6,ec946ff7112599b7,a6ea6109ee8489d2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:53:50.651 WARN 21062 --- [ocp-async-10,bed8e6a9f4f729de,cca6e4afca83d60a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:50.685 INFO 21062 --- [ocp-async-3,eb57db28ba94c5b5,43591bd94f76f2cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:50.839 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:50.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a0bc7f9407b109fe,fe20af86888c3fc0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:52.651 WARN 21062 --- [ocp-async-7,2563938a4049d842,be75bd485e68969c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:52.685 INFO 21062 --- [ocp-async-6,32a900b9c051caaf,078df7fde81312ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:53.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:53.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:53.942 INFO 21062 --- [ocp-schedules-8,3f9d66681a821dd5,5fa8c36165c5b2c1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:53:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ad9b9ff81849d012,eb5f5ea13ea190a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:54.651 WARN 21062 --- [ocp-async-4,5944b0d2e03bc417,8e467d933681f157] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:54.686 INFO 21062 --- [ocp-async-1,1b81832954b30eae,18c71f087c5e26bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:56.651 WARN 21062 --- [ocp-async-2,bced35d7d094b48e,e7248492d45084ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:56.686 INFO 21062 --- [ocp-async-8,a595cbe2aed94662,1cf106d4345fad5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:56.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:56.842 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:56.983 INFO 21062 --- [prometheus-minute-schedule-1,518d8defc3192998,520a743800d3b639] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:58.500 INFO 21062 --- [ocp-schedules-13,9194a42508808810,4ea42386f4919845] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:53:58.500 INFO 21062 --- [ocp-schedules-4,a18b4df9c93801a8,5d2c278196566b00] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:53:58.502 INFO 21062 --- [ocp-schedules-26,ef06218c03eaf42b,1ef61e2088090879] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:53:58.503 INFO 21062 --- [ocp-schedules-13,9194a42508808810,4ea42386f4919845] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:53:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:53:58.504 INFO 21062 --- [ocp-schedules-4,a18b4df9c93801a8,5d2c278196566b00] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:53:58.505 INFO 21062 --- [ocp-schedules-11,5889d2244e7de9d5,59517f5fd54d8fc7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:53:58.513 INFO 21062 --- [ocp-schedules-9,9436a9b43080c3bb,45b20a23fcacc5da] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:53:58.513 INFO 21062 --- [ocp-schedules-27,dadce607ee6b4cd7,5ff2d662fd3c7a2f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:53:58.513 INFO 21062 --- [ocp-schedules-9,9436a9b43080c3bb,45b20a23fcacc5da] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:53:58.513 INFO 21062 --- [ocp-schedules-27,dadce607ee6b4cd7,5ff2d662fd3c7a2f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:53:58.514 INFO 21062 --- [ocp-schedules-22,a39795893e5b46bc,7b506d8c4f17c799] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:53:58.514 INFO 21062 --- [ocp-schedules-22,a39795893e5b46bc,7b506d8c4f17c799] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:53:58.555 INFO 21062 --- [ocp-async-8,0f36b6b83de906f1,40491a4c03964271] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:53:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:53:58.588582+08:00, lastProcessTime=2024-11-04T12:53:48.587787+08:00 2024-11-04 12:53:58.652 WARN 21062 --- [ocp-async-3,8fe08476d4d57db9,a242fbae80ea8425] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:53:58.662 INFO 21062 --- [ocp-schedules-17,5c1752cacca8b3f2,b2377143d6da49ce] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:53:58.686 INFO 21062 --- [ocp-async-6,1ec132eeea9d4f30,96072ecaa07faf70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:53:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:53:58.588582+08:00 2024-11-04 12:53:58.874 INFO 21062 --- [ocp-schedules-2,624556b0bf55dc5b,5c678218d6b09745] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:53:59.842 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:53:59.843 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:53:59.981 INFO 21062 --- [ocp-schedules-10,27f210b96c5e9f82,336fb9cab68f25a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:53:59.981 INFO 21062 --- [ocp-schedules-10,f5071de94556e349,513891acf3bc212f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:53:59.983 INFO 21062 --- [prometheus-minute-schedule-1,30fd010bd57fe398,986769cc91c2755e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696040, interval=60 2024-11-04 12:53:59.985 INFO 21062 --- [ocp-schedules-10,f5071de94556e349,513891acf3bc212f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:00.016 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:54:00.016 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:54:00.652 WARN 21062 --- [ocp-async-6,616a0be52101a41d,2b9eb4000499860a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:00.686 INFO 21062 --- [ocp-async-1,1166fcd893d8d92c,69916de7cf6fc343] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:02.652 WARN 21062 --- [ocp-async-1,be03650e19546b66,4fe1fa211dfe7259] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:02.686 INFO 21062 --- [ocp-async-8,4769952fed25403c,4f12b05783d7daba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:02.844 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:02.844 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5671604980f9caf5,bd920634964c637a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:03.986 INFO 21062 --- [ocp-schedules-30,809b46a469802301,8762fa22cd4b5a29] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:54:04.652 WARN 21062 --- [ocp-async-8,256cfb51962be2e7,a74bc117874b75d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:04.687 INFO 21062 --- [ocp-async-5,3fc880b92bd15d53,cd7befd6d5ab95af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:05.845 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:05.845 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:05.984 INFO 21062 --- [prometheus-minute-schedule-1,a26a515fa1d2b431,5f3731f6f5fd9da1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:06.652 WARN 21062 --- [ocp-async-5,3c2be5e26ee202e0,d2e45ec627367336] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:06.687 INFO 21062 --- [ocp-async-9,5cc7e2e577667f43,1ca3556284842ff0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:08.504 INFO 21062 --- [ocp-schedules-13,9194a42508808810,4ea42386f4919845] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:54:08.505 INFO 21062 --- [ocp-schedules-4,a18b4df9c93801a8,5d2c278196566b00] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:54:08.555 INFO 21062 --- [ocp-async-4,b929c844c5134b47,9a8b6a0f20398571] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:54:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:54:08.588460+08:00, lastProcessTime=2024-11-04T12:53:58.588582+08:00 2024-11-04 12:54:08.653 WARN 21062 --- [ocp-async-1,720458cf2dd653b9,1918cda77ae4d9d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:08.687 INFO 21062 --- [ocp-async-8,f4a59264051e63d9,36c411d2b107791f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:54:08.588460+08:00 2024-11-04 12:54:08.846 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:08.847 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:08.876 INFO 21062 --- [ocp-schedules-1,6ba6e9293ee05344,0255321d5cba79f5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:54:08.983 INFO 21062 --- [prometheus-minute-schedule-1,10b68206cccfdeb4,d814bf5d69e0ad08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:09.126 INFO 21062 --- [ocp-schedules-13,a483ee7dd7234bd5,4da13a7cffb3a95b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:54:09.132 INFO 21062 --- [ocp-schedules-13,a483ee7dd7234bd5,4da13a7cffb3a95b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:54:09.132 INFO 21062 --- [ocp-schedules-13,a483ee7dd7234bd5,4da13a7cffb3a95b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:54:10.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:54:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:54:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:54:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:54:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:54:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:54:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:54:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:54:10.653 WARN 21062 --- [ocp-async-8,b7796ac6f95bf6b5,a31ed5d727bbbf0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:10.687 INFO 21062 --- [ocp-async-5,385e7578804015ca,2925a9c75906af06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695980, nodeCount=26, affectRows=26 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695980, nodeCount=48, affectRows=48 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695980, nodeCount=18, affectRows=18 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695980, nodeCount=29, affectRows=29 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695980, nodeCount=26, validCount=26 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695980, nodeCount=18, validCount=18 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695980, nodeCount=29, validCount=29 2024-11-04 12:54:11.119 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695980, nodeCount=47, validCount=47 2024-11-04 12:54:11.120 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695980, nodeCount=63, affectRows=63 2024-11-04 12:54:11.120 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695980, nodeCount=59, affectRows=59 2024-11-04 12:54:11.120 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730695980, nodeCount=79, affectRows=79 2024-11-04 12:54:11.120 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695980, nodeCount=62, validCount=62 2024-11-04 12:54:11.120 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695980, nodeCount=59, validCount=59 2024-11-04 12:54:11.120 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730695980, nodeCount=78, validCount=78 2024-11-04 12:54:11.847 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:11.849 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9e968a27022e0a02,c2f21e339b6cbdaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:12.653 WARN 21062 --- [ocp-async-5,810c0dbe0b230d9a,e1b886021595c948] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:12.687 INFO 21062 --- [ocp-async-9,683e0a9fdc1c0047,386000f7952706d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:13.984 INFO 21062 --- [prometheus-cache-monitor1,d7dd03374dee7bfa,a28f4afd77cb44f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:54:13.985 INFO 21062 --- [prometheus-cache-monitor1,d7dd03374dee7bfa,a28f4afd77cb44f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:54:14.011 INFO 21062 --- [ocp-schedules-28,c82b0e1a9c164d3d,1b896134308a73f7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:54:14.653 WARN 21062 --- [ocp-async-9,5d5d2cbc168b5e20,5e0882b5241ece09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:14.687 INFO 21062 --- [ocp-async-10,68831d3cadada83e,2ac8518fee182a42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:14.849 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:14.850 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:14.983 INFO 21062 --- [prometheus-minute-schedule-1,da587da39f17bac9,e5ecbdcb0e14319f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:16.653 WARN 21062 --- [ocp-async-10,488acbffa332dd47,052c268ef927e204] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:16.688 INFO 21062 --- [ocp-async-7,c54ec5ac6560f022,b31f605fdfdfad7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:17.850 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:17.851 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:17.983 INFO 21062 --- [prometheus-minute-schedule-1,64d75bdf879e0f32,76d908de0c4be30b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:18.556 INFO 21062 --- [ocp-async-3,c37e544f628602f4,914905627b91ea6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:54:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:54:18.588243+08:00, lastProcessTime=2024-11-04T12:54:08.588460+08:00 2024-11-04 12:54:18.653 WARN 21062 --- [ocp-async-5,5fbdbe941827ac3d,ce4370742b839ba2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:18.688 INFO 21062 --- [ocp-async-9,f30ad6d366906421,a0a852a691db638b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:54:18.588243+08:00 2024-11-04 12:54:18.878 INFO 21062 --- [ocp-schedules-15,c2387352b438680c,898363e407ce1055] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:54:20.654 WARN 21062 --- [ocp-async-9,2d0e5189dfe3d82c,bc91eb768d8f426e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:20.688 INFO 21062 --- [ocp-async-2,2d665194bef5a3a9,66136a59a9781967] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:20.852 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:20.853 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:20.983 INFO 21062 --- [prometheus-minute-schedule-1,857542f353ef315b,44ff3373f32cbbc4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:22.654 WARN 21062 --- [ocp-async-2,980c1c0ff6d08666,4466c04ede8327d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:22.688 INFO 21062 --- [ocp-async-3,3c8c39b835f0e0da,417cd49b33b138d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:23.853 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:23.854 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:23.983 INFO 21062 --- [prometheus-minute-schedule-1,20ed8d6e50d6bed4,e89dde734343fe04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:24.059 INFO 21062 --- [ocp-schedules-3,81b715af8e8a9632,50c3409bca7e188c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:54:24.654 WARN 21062 --- [ocp-async-3,6c7e647f4feb79f6,7c7c0c9a33efb69e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:24.689 INFO 21062 --- [ocp-async-6,e996d8a76f45aca1,63177a1f253c18b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:26.654 WARN 21062 --- [ocp-async-6,e0bebe465e4c4198,2a3cc005d6b40636] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:26.689 INFO 21062 --- [ocp-async-10,c7e4cf9dfc9da57a,2a865fb4d31ff823] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:26.854 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:26.855 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:26.983 INFO 21062 --- [prometheus-minute-schedule-1,91125b1f66f43587,29bea9f8bba8a6af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:26.984 INFO 21062 --- [prometheus-hour-schedule-1,058ff9c28ac8e9cb,37655d6b18c2a6c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:54:28.498 INFO 21062 --- [ocp-schedules-13,f4ae166ad357c45d,7570c48e6a659928] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:54:28.500 INFO 21062 --- [ocp-schedules-9,c26086ea66dce16f,e5baa585ac21302b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:54:28.500 INFO 21062 --- [ocp-schedules-4,5d7ed0232a423158,b67ae2f6fea6797d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:54:28.500 INFO 21062 --- [ocp-schedules-26,3dab74216352ec5a,556deb854a98e680] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:54:28.502 INFO 21062 --- [ocp-schedules-13,f4ae166ad357c45d,7570c48e6a659928] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:54:28.502 INFO 21062 --- [ocp-schedules-14,5af74023c4c46939,bac62b96c5f34a1b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:54:28.503 INFO 21062 --- [ocp-schedules-4,5d7ed0232a423158,b67ae2f6fea6797d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:28.504 INFO 21062 --- [ocp-schedules-26,3dab74216352ec5a,556deb854a98e680] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:54:28.504 INFO 21062 --- [ocp-schedules-11,d7316271e91fb22a,afcdc9a2718cd896] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:54:28.505 INFO 21062 --- [ocp-schedules-9,c26086ea66dce16f,e5baa585ac21302b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:54:28.510 INFO 21062 --- [ocp-schedules-8,a51de8ee6341dbe0,67529a580f4d351e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:54:28.510 INFO 21062 --- [ocp-schedules-28,eb4972f2e17bc9f7,b8a80b5cc43eff70] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:54:28.510 INFO 21062 --- [ocp-schedules-28,eb4972f2e17bc9f7,b8a80b5cc43eff70] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:54:28.510 INFO 21062 --- [ocp-schedules-8,a51de8ee6341dbe0,67529a580f4d351e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:54:28.511 INFO 21062 --- [ocp-schedules-5,572c53e9215bea94,4b19b94b29c8f4a0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:54:28.511 INFO 21062 --- [ocp-schedules-5,572c53e9215bea94,4b19b94b29c8f4a0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:54:28.537 INFO 21062 --- [ocp-schedules-29,d2f0541791eac996,8bc690a205741a28] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T12:54:28.537+08:00, tenantSize=0 2024-11-04 12:54:28.556 INFO 21062 --- [ocp-schedules-19,77e091c1d8df74e6,b3a3dee83a80d000] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:54:28.556 INFO 21062 --- [ocp-async-2,f34c11a31445f632,cb7f0aee4e09619e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:54:28.556 INFO 21062 --- [ocp-schedules-19,77e091c1d8df74e6,b3a3dee83a80d000] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:54:28.573 INFO 21062 --- [ocp-schedules-12,328ec38c85f0d3f0,96c61f7a8e008912] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:54:28.574 INFO 21062 --- [ocp-schedules-12,328ec38c85f0d3f0,96c61f7a8e008912] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:54:28.585 INFO 21062 --- [ocp-schedules-22,3c9ecc3310021ef4,240a2e41e0a1fe6e] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 12:54:28.585 INFO 21062 --- [ocp-schedules-22,3c9ecc3310021ef4,240a2e41e0a1fe6e] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 12:54:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:54:28.587407+08:00, lastProcessTime=2024-11-04T12:54:18.588243+08:00 2024-11-04 12:54:28.612 INFO 21062 --- [ocp-schedules-2,19dc8dbd4e4b99ce,0c77a1822c4b765c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:54:28.633 INFO 21062 --- [ocp-schedules-1,3d2086a9b3fa9a13,5f7c65e0b7c056db] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:54:28.636 INFO 21062 --- [ocp-schedules-1,3d2086a9b3fa9a13,5f7c65e0b7c056db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:28.637 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:54:28.654 INFO 21062 --- [ocp-schedules-17,74fdaa21821ed2b9,26b240c91033834e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:54:28.654 INFO 21062 --- [ocp-schedules-17,74fdaa21821ed2b9,26b240c91033834e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:54:28.654 WARN 21062 --- [ocp-async-7,7dce218eb0241ac1,2e6fcb11f89f5776] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:28.664 INFO 21062 --- [ocp-schedules-14,e9e36f5cbc7f18d0,3eee0b3007e79634] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:54:28.689 INFO 21062 --- [ocp-async-5,76d4d0dc8efc16a0,612620676464676a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:28.708 INFO 21062 --- [ocp-schedules-13,cbcc84eb9fae9f09,747dc3700418e916] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:54:28.714 INFO 21062 --- [ocp-schedules-13,cbcc84eb9fae9f09,747dc3700418e916] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:54:28.714 INFO 21062 --- [ocp-schedules-13,cbcc84eb9fae9f09,747dc3700418e916] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:54:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:54:28.587407+08:00 2024-11-04 12:54:28.818 INFO 21062 --- [ocp-schedules-25,5ef1abc28daa5d23,22cf5013161989f8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:54:28.818 INFO 21062 --- [ocp-schedules-25,5ef1abc28daa5d23,22cf5013161989f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:54:28.825 INFO 21062 --- [ocp-schedules-25,5ef1abc28daa5d23,22cf5013161989f8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:54:28.825 INFO 21062 --- [ocp-schedules-25,5ef1abc28daa5d23,22cf5013161989f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:54:28.830 INFO 21062 --- [ocp-schedules-25,5ef1abc28daa5d23,22cf5013161989f8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:54:28.830 INFO 21062 --- [ocp-schedules-25,5ef1abc28daa5d23,22cf5013161989f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:54:28.881 INFO 21062 --- [ocp-schedules-3,a8b5007c5efae6e4,f1932d7a774c632e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:54:29.121 INFO 21062 --- [ocp-schedules-24,ef9c3ca3a77b92e1,82a7f25776a441cb] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:54:29.121 INFO 21062 --- [ocp-schedules-24,ef9c3ca3a77b92e1,82a7f25776a441cb] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:54:29.134 INFO 21062 --- [ocp-schedules-20,1f29638f95893808,fa40fbf63514c744] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:54:29.140 INFO 21062 --- [ocp-schedules-20,1f29638f95893808,fa40fbf63514c744] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:54:29.140 INFO 21062 --- [ocp-schedules-20,1f29638f95893808,fa40fbf63514c744] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:54:29.148 INFO 21062 --- [ocp-schedules-24,ef9c3ca3a77b92e1,82a7f25776a441cb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:54:29.148 INFO 21062 --- [ocp-schedules-24,ef9c3ca3a77b92e1,82a7f25776a441cb] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:54:29.155 INFO 21062 --- [ocp-schedules-21,c30b29b6cacd5f4b,00eda4c04e2da011] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:54:29.161 INFO 21062 --- [ocp-schedules-27,31a0ab6109c8843a,cc3fb1b54090f249] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:54:29.165 INFO 21062 --- [ocp-schedules-27,31a0ab6109c8843a,cc3fb1b54090f249] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:29.188 INFO 21062 --- [ocp-schedules-7,abc3a181fadc0066,71098bf8b2d98423] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:54:29.188 INFO 21062 --- [ocp-schedules-7,abc3a181fadc0066,71098bf8b2d98423] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:54:29.188 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:54:29.188 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:54:29.198 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:54:29.198 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:54:29.198 INFO 21062 --- [ocp-schedules-7,abc3a181fadc0066,71098bf8b2d98423] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:54:29.200 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:54:29.200 WARN 21062 --- [ocp-schedules-7,abc3a181fadc0066,71098bf8b2d98423] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:54:29.200 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:54:29.201 INFO 21062 --- [ocp-schedules-7,abc3a181fadc0066,71098bf8b2d98423] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:54:29.201 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:54:29.201 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:54:29.202 WARN 21062 --- [ocp-schedules-7,abc3a181fadc0066,71098bf8b2d98423] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:54:29.202 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:54:29.203 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:54:29.209 ERROR 21062 --- [ocp-schedules-7,abc3a181fadc0066,71098bf8b2d98423] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 12:54:29.222 ERROR 21062 --- [ocp-schedules-27,31a0ab6109c8843a,cc3fb1b54090f249] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:54:29.222 INFO 21062 --- [ocp-schedules-27,31a0ab6109c8843a,cc3fb1b54090f249] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:54:29.270 INFO 21062 --- [ocp-schedules-18,1fd211a3a9d4a06b,6f48b395cbdd0f9c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:54:29.270+08:00, clusters=[1] 2024-11-04 12:54:29.271 INFO 21062 --- [ocp-schedules-18,1fd211a3a9d4a06b,6f48b395cbdd0f9c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:54:29.344 INFO 21062 --- [ocp-schedules-6,649963995f6aafa2,d89b777bd5c72354] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:54:29.344+08:00, clusters=[1] 2024-11-04 12:54:29.348 INFO 21062 --- [ocp-schedules-6,649963995f6aafa2,d89b777bd5c72354] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:54:29.855 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:29.856 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:29.983 INFO 21062 --- [prometheus-minute-schedule-1,c849b0a1ce0a335e,0b9ddfb4f1dec066] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:30.016 INFO 21062 --- [ocp-schedules-10,f5071de94556e349,513891acf3bc212f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:54:30.016 INFO 21062 --- [ocp-schedules-10,882b1ce2044e5a7e,5796076990f906d9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:54:30.020 INFO 21062 --- [ocp-schedules-10,882b1ce2044e5a7e,5796076990f906d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:30.048 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:54:30.048 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:54:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:54:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:54:30.516 INFO 21062 --- [ocp-schedules-18,48132dab5ac51907,2a25655de2d49e02] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:54:30.520 INFO 21062 --- [ocp-schedules-18,48132dab5ac51907,2a25655de2d49e02] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:30.606 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:54:30.608 ERROR 21062 --- [ocp-schedules-18,48132dab5ac51907,2a25655de2d49e02] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:54:30.608 INFO 21062 --- [ocp-schedules-18,48132dab5ac51907,2a25655de2d49e02] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:54:30.655 WARN 21062 --- [ocp-async-4,bd02eea90f9d9ee9,39add54d819f5a7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:30.689 INFO 21062 --- [ocp-async-6,99f081af4591dd81,685edba0b245df20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:32.655 WARN 21062 --- [ocp-async-9,3bcac41ebbf799ad,b3bb3f8356b0fb2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:32.689 INFO 21062 --- [ocp-async-1,e946d9e043a7bd93,ba26529f5037e362] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:32.856 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:32.857 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:32.983 INFO 21062 --- [prometheus-minute-schedule-1,cdbefc2a51e66776,417fd569ff85d558] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:34.086 INFO 21062 --- [ocp-schedules-6,868d8037a416c192,1db9e983269a79c6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:54:34.655 WARN 21062 --- [ocp-async-8,ae6aff95d1c52407,6a8d47443066af69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:34.689 INFO 21062 --- [ocp-async-7,20f1135a3bbc50cf,6dd2cb6d689032e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:35.857 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:35.858 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:35.983 INFO 21062 --- [prometheus-minute-schedule-1,191ca51afbffd3d9,6e7398fccb000d3b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:36.655 WARN 21062 --- [ocp-async-5,d3325022c5e865ff,4729817b36e0e771] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:36.690 INFO 21062 --- [ocp-async-4,39f95b7571b563de,07ef96c0ace16f9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:38.504 INFO 21062 --- [ocp-schedules-4,5d7ed0232a423158,b67ae2f6fea6797d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:54:38.504 INFO 21062 --- [ocp-schedules-26,3dab74216352ec5a,556deb854a98e680] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:54:38.556 INFO 21062 --- [ocp-async-6,677d5bd9442c99ad,9cf5de7a2e420746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:54:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:54:38.588196+08:00, lastProcessTime=2024-11-04T12:54:28.587407+08:00 2024-11-04 12:54:38.637 INFO 21062 --- [ocp-schedules-1,3d2086a9b3fa9a13,5f7c65e0b7c056db] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:54:38.655 WARN 21062 --- [ocp-async-2,7d914af2769d8483,63f26f2cc570df71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:38.690 INFO 21062 --- [ocp-async-1,038f1b24a3571ddc,6e8d7d8c9e3f6366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:54:38.588196+08:00 2024-11-04 12:54:38.858 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:38.859 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:38.884 INFO 21062 --- [ocp-schedules-25,2c1b4478d92d7f90,35c43a40f8e3368f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:54:38.940 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 12:54:38.940 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 12:54:38.942 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.942 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:54:38.944 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.944 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:38.945 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.945 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 12:54:38.961 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.961 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:38.962 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.962 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 12:54:38.975 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.975 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:38.976 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.976 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 12:54:38.983 INFO 21062 --- [prometheus-minute-schedule-1,33e4125380f1d0a4,199bf164cdfacfdf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:38.991 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.991 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:38.992 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:38.992 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 12:54:39.005 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.005 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.005 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.006 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 12:54:39.011 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.012 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.012 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.013 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 12:54:39.017 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.017 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.018 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.018 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 12:54:39.022 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.023 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.023 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.024 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 12:54:39.030 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.030 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.031 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.031 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 12:54:39.036 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.037 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.037 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.038 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 12:54:39.043 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.044 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.044 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.045 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 12:54:39.053 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.053 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.054 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.054 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 12:54:39.070 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.070 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.071 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.071 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 12:54:39.076 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.076 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.077 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.077 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 12:54:39.083 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.083 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.084 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.084 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 12:54:39.091 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.091 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.091 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.092 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 12:54:39.096 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.097 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.097 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.097 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 12:54:39.101 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.102 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.102 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.103 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 12:54:39.107 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.107 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.108 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.108 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 12:54:39.113 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.113 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.114 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.114 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 12:54:39.118 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.118 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.119 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.119 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 12:54:39.124 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.124 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.125 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.125 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 12:54:39.129 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.130 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.130 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.131 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 12:54:39.135 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.135 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.136 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.136 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 12:54:39.140 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.141 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.141 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.142 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 12:54:39.146 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.146 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.147 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.147 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 12:54:39.152 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.152 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.152 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.153 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 12:54:39.159 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.159 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.160 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.160 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 12:54:39.165 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.165 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.166 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.166 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 12:54:39.170 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.171 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.171 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.171 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 12:54:39.176 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.177 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.177 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.178 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 12:54:39.182 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.182 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.183 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.183 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 12:54:39.188 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.188 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.188 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.189 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 12:54:39.193 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.193 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.194 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.194 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 12:54:39.201 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.201 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.202 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.202 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 12:54:39.209 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.209 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.210 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.210 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 12:54:39.216 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.216 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.217 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.217 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 12:54:39.222 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.223 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.223 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.223 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 12:54:39.227 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.228 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.228 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.228 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 12:54:39.232 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.232 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.233 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.233 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 12:54:39.237 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.237 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.238 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.238 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 12:54:39.242 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.242 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.243 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.243 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 12:54:39.252 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.252 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.253 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.254 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 12:54:39.275 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.275 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.276 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.276 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 12:54:39.283 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.283 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.284 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.284 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 12:54:39.290 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.290 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.291 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.292 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 12:54:39.297 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.298 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.298 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.298 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 12:54:39.302 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.303 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.303 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.304 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 12:54:39.307 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.308 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.308 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.309 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 12:54:39.313 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 12:54:39.313 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 12:54:39.315 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.315 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:54:39.316 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.317 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.317 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.317 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 12:54:39.323 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.324 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 12:54:39.324 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:39.325 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 12:54:39.329 INFO 21062 --- [ocp-schedules-4,b17c0d5eea8c80b4,b717b6cd91bd9853] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 12:54:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:54:40.004 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:54:40.655 WARN 21062 --- [ocp-async-3,4b6d29e84abcab07,05e3d47695872d60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:40.690 INFO 21062 --- [ocp-async-7,cb6e194713be3032,15638bf898c8de68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:54:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:54:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:54:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:54:41.856 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:54:41.856 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-11-04 12:54:41.859 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:41.860 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:41.983 INFO 21062 --- [prometheus-minute-schedule-1,b5ad064b89e75a16,6fa3407bb58851fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:42.656 WARN 21062 --- [ocp-async-4,d887a14b8702da5a,f657675832916360] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:42.690 INFO 21062 --- [ocp-async-6,cd35c2edd4a2790b,55ac09b6f918a414] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:44.129 INFO 21062 --- [ocp-schedules-16,684a3eedd12fbd30,f8b64b585da262b1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:54:44.656 WARN 21062 --- [ocp-async-9,3bb5406c09673257,fbe7f9bfcacc0ac7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:44.690 INFO 21062 --- [ocp-async-8,4a6cbe7f4525b01f,08c05ffaa75dc2dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:44.860 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:44.861 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b18873412dba8862,7a4206e5a838068d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:46.656 WARN 21062 --- [ocp-async-1,5c56bf9c8fa3695c,6ad78089262b3c33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:46.691 INFO 21062 --- [ocp-async-5,d32056c60a2e844f,dfbb8f0c13d64022] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:47.861 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:47.862 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8f9ea4c48b48c2d7,d132698b53ea5b04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:48.556 INFO 21062 --- [ocp-async-5,86e02f496232e8b1,4f3bc169e624dae5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:54:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:54:48.588933+08:00, lastProcessTime=2024-11-04T12:54:38.588196+08:00 2024-11-04 12:54:48.656 WARN 21062 --- [ocp-async-4,f730536634ffd4e9,5394fb9559d9a9a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:48.691 INFO 21062 --- [ocp-async-6,89f342b7c0f29155,3510b9ebee48a54f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:54:48.588933+08:00 2024-11-04 12:54:48.887 INFO 21062 --- [ocp-schedules-12,809b6e89164bc392,8e4bbadb5923aa95] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:54:49.143 INFO 21062 --- [ocp-schedules-19,7d0f6c9bde47ad37,fb99c75785e3a359] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:54:49.150 INFO 21062 --- [ocp-schedules-19,7d0f6c9bde47ad37,fb99c75785e3a359] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:54:49.150 INFO 21062 --- [ocp-schedules-19,7d0f6c9bde47ad37,fb99c75785e3a359] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:54:50.656 WARN 21062 --- [ocp-async-9,470bd4f2b845feb4,67877a749e5cbccb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:50.691 INFO 21062 --- [ocp-async-8,00496b20d76fd005,27f29a1ad84fbcda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:50.863 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:50.863 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:50.983 INFO 21062 --- [prometheus-minute-schedule-1,19f55338cd7857fa,fc069dd2f8401648] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:52.656 WARN 21062 --- [ocp-async-1,6595d9e525b6144b,cccc21bad97a4a0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:52.691 INFO 21062 --- [ocp-async-5,f3dd1ce84c23bbe6,a5ef95d3167bad5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:53.864 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:53.864 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:53.983 INFO 21062 --- [prometheus-minute-schedule-1,cbc9d74dce778a06,f960120f33bfa5b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:54.154 INFO 21062 --- [ocp-schedules-25,11a9febeea1af9bc,99d3a0ab2e3b1c3e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:54:54.657 WARN 21062 --- [ocp-async-7,e16045125d146db0,401faa3d51dd090f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:54.691 INFO 21062 --- [ocp-async-10,0dde1c925e40bba0,a778885ad4af5718] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:56.657 WARN 21062 --- [ocp-async-6,d5061fd81cf26cca,06dba32f49e59d19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:56.691 INFO 21062 --- [ocp-async-2,9cf4562d01c6d5f5,b566515e2674d37e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:56.865 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:56.865 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:56.983 INFO 21062 --- [prometheus-minute-schedule-1,2969ed00c8506c9b,4e12aeed1a060d15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:54:58.500 INFO 21062 --- [ocp-schedules-18,5aa7efadcaa578fd,55066d3b19548900] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:54:58.500 INFO 21062 --- [ocp-schedules-21,dd6e43fd123a5995,c5603b32f06d3528] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:54:58.502 INFO 21062 --- [ocp-schedules-5,359aceffb66b45d3,9906b53c674b1bab] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:54:58.503 INFO 21062 --- [ocp-schedules-18,5aa7efadcaa578fd,55066d3b19548900] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:54:58.504 INFO 21062 --- [ocp-schedules-21,dd6e43fd123a5995,c5603b32f06d3528] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:54:58.504 INFO 21062 --- [ocp-schedules-9,2b1d037fcb972f33,bc61be34a75425d5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:54:58.511 INFO 21062 --- [ocp-schedules-19,8810475b832675bf,ed86175f3936d597] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:54:58.511 INFO 21062 --- [ocp-schedules-4,eb07ac6e70850735,50e209740e435974] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:54:58.511 INFO 21062 --- [ocp-schedules-19,8810475b832675bf,ed86175f3936d597] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:54:58.511 INFO 21062 --- [ocp-schedules-4,eb07ac6e70850735,50e209740e435974] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:54:58.511 INFO 21062 --- [ocp-schedules-11,621a37edeac5fc96,496da7f42d462f6a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:54:58.511 INFO 21062 --- [ocp-schedules-11,621a37edeac5fc96,496da7f42d462f6a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:54:58.556 INFO 21062 --- [ocp-async-2,64819ee95465a0a8,cde1b02a957c5ec5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:54:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:54:58.589506+08:00, lastProcessTime=2024-11-04T12:54:48.588933+08:00 2024-11-04 12:54:58.657 WARN 21062 --- [ocp-async-1,2584e50d24ea2b91,12f4d6f366ac93b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:54:58.667 INFO 21062 --- [ocp-schedules-13,edc6d01d976fcd34,b6ec5e907a6cef10] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:54:58.691 INFO 21062 --- [ocp-async-5,cfacca7495636967,79d62c869909fc31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:54:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:54:58.589506+08:00 2024-11-04 12:54:58.890 INFO 21062 --- [ocp-schedules-27,6e924801e4ac6d5c,5332703ea4926907] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:54:59.865 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:54:59.867 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:54:59.983 INFO 21062 --- [prometheus-minute-schedule-1,34d20267f84a59ff,b2229b3a90020e26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696100, interval=60 2024-11-04 12:55:00.049 INFO 21062 --- [ocp-schedules-10,882b1ce2044e5a7e,5796076990f906d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:55:00.049 INFO 21062 --- [ocp-schedules-10,a67ff6946768cf1c,077e2e379d2e9e4b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:55:00.053 INFO 21062 --- [ocp-schedules-10,a67ff6946768cf1c,077e2e379d2e9e4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:00.081 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:55:00.081 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:55:00.657 WARN 21062 --- [ocp-async-5,8e1bd34d6d1d27ca,f1acdc6cd502d09d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:00.692 INFO 21062 --- [ocp-async-10,7e3370bd677f948c,f41575708967acf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:00.953 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d1bbe23344082699] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:50:00.0, nextRun=2024-11-04 12:55:00.0, currentDate=Mon Nov 04 12:55:00 CST 2024 2024-11-04 12:55:00.954 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d1bbe23344082699] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 12:50:00.0, nextRunTime=2024-11-04 12:55:00.0} 2024-11-04 12:55:00.981 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d1bbe23344082699] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 12:55:00.986 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d1bbe23344082699] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 12:55:00.986 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d1bbe23344082699] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 12:55:02.657 WARN 21062 --- [ocp-async-10,56d30b432259a189,c90d1bd5ffbe8682] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:02.692 INFO 21062 --- [ocp-async-2,c208a245e9faf9c5,49188559f437a62b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:02.867 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:02.868 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a5814e0e1e3ec680,83883355688f39ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:04.196 INFO 21062 --- [ocp-schedules-23,6613ad934f5d4557,0c24fa34963f5a79] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:55:04.658 WARN 21062 --- [ocp-async-2,0ff398dfe4c359d4,db2dbc3156b16c91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:04.692 INFO 21062 --- [ocp-async-3,87f48f5a97c336d7,db7ce1622813baac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:05.868 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:05.869 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d9f2c3de8a5afede,ad146c04885aa71c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:06.658 WARN 21062 --- [ocp-async-3,98ea01ebddf7a875,88fc303110045328] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:06.692 INFO 21062 --- [ocp-async-4,710df8db8e4ee78a,21b498969329f231] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:08.504 INFO 21062 --- [ocp-schedules-18,5aa7efadcaa578fd,55066d3b19548900] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:55:08.505 INFO 21062 --- [ocp-schedules-21,dd6e43fd123a5995,c5603b32f06d3528] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:55:08.557 INFO 21062 --- [ocp-async-7,beb6d6f38bda5496,b020abe47aaa7613] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:55:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:55:08.589079+08:00, lastProcessTime=2024-11-04T12:54:58.589506+08:00 2024-11-04 12:55:08.658 WARN 21062 --- [ocp-async-10,d95dab0957709eff,0fa6fa96c17fc57d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:08.692 INFO 21062 --- [ocp-async-2,571c05f15b15f92b,ff368fb5684e94db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:55:08.589079+08:00 2024-11-04 12:55:08.869 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:08.870 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:08.893 INFO 21062 --- [ocp-schedules-11,7ced7bb2d04a7dd0,882eda1b770ad216] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:55:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8993289a7860238a,e1bda4ec6e597d2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:09.153 INFO 21062 --- [ocp-schedules-27,5a0f63c75616eb62,0b2afb4539550df0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:55:09.159 INFO 21062 --- [ocp-schedules-27,5a0f63c75616eb62,0b2afb4539550df0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:55:09.159 INFO 21062 --- [ocp-schedules-27,5a0f63c75616eb62,0b2afb4539550df0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:55:10.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:55:10.004 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:55:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:55:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:55:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:55:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:55:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:55:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:55:10.658 WARN 21062 --- [ocp-async-2,a7a339bbcbe49b3a,2f3fd4fe6031dab4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:10.693 INFO 21062 --- [ocp-async-3,5209719e50a29735,a5d3e8bdccf61ad7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:11.122 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=2, affectRows=2 2024-11-04 12:55:11.122 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=2, validCount=2 2024-11-04 12:55:11.124 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=9, affectRows=9 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=61, affectRows=61 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=9, validCount=9 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=21, affectRows=21 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=46, affectRows=46 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=43, affectRows=43 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=47, affectRows=47 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=29, affectRows=29 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=14, affectRows=14 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=29, affectRows=29 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696040, nodeCount=21, affectRows=21 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=21, validCount=21 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=59, validCount=59 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=14, validCount=14 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=21, validCount=21 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=29, validCount=29 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=29, validCount=29 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=43, validCount=43 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=45, validCount=45 2024-11-04 12:55:11.125 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696040, nodeCount=47, validCount=47 2024-11-04 12:55:11.870 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:11.871 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:11.983 INFO 21062 --- [prometheus-minute-schedule-1,7f3684a8cc0e6e00,f5ff137a05845957] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:12.658 WARN 21062 --- [ocp-async-3,206217263feda0e3,cbe5d28a3884a99b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:12.693 INFO 21062 --- [ocp-async-4,8f48b8d5f006e9db,200991e6aefbdda4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:13.984 INFO 21062 --- [prometheus-cache-monitor1,3e6292d5641bc0c0,88c807924948e721] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:55:13.985 INFO 21062 --- [prometheus-cache-monitor1,3e6292d5641bc0c0,88c807924948e721] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:55:14.218 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:51:14.189, end=2024-11-04T12:53:14.215} 2024-11-04 12:55:14.222 INFO 21062 --- [ocp-schedules-12,cd14ba349b3e77df,8c3a57f2fcc3c203] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:55:14.239 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:55:14.239 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:55:14.658 WARN 21062 --- [ocp-async-4,6b59475a39d331a8,c17fc584ed1a3ab0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:14.693 INFO 21062 --- [ocp-async-9,b2127eea88aef687,0367e2bb9cd11f40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:14.871 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:14.872 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:14.983 INFO 21062 --- [prometheus-minute-schedule-1,73771a484bea08ed,b4a17bfa22b8cd68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:16.659 WARN 21062 --- [ocp-async-9,d32f5950898e19f0,1d11aaec7088d6df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:16.693 INFO 21062 --- [ocp-async-1,5e9f88bb623e2032,0550d9057f575dba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:17.872 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:17.873 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5d80724f706cf8f2,3f9ceda261881c0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:18.557 INFO 21062 --- [ocp-async-8,e9dd3b518a29d4a1,fc0ca65714915d42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:55:18.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:55:18.591061+08:00, lastProcessTime=2024-11-04T12:55:08.589079+08:00 2024-11-04 12:55:18.659 WARN 21062 --- [ocp-async-3,f1a5a990a96ac7ee,2f828c457ee574b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:18.693 INFO 21062 --- [ocp-async-4,105230ce8e647faa,e41a91124c1903ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:55:18.591061+08:00 2024-11-04 12:55:18.896 INFO 21062 --- [ocp-schedules-11,656faf7e16722a8f,c45648baf49cf90c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:55:20.659 WARN 21062 --- [ocp-async-4,e64ccd1c26d95c80,d06900b36af9899a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:20.694 INFO 21062 --- [ocp-async-6,6f053714b384f0f8,10f64fff41bfb902] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:20.873 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:20.874 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:20.983 INFO 21062 --- [prometheus-minute-schedule-1,142407f8bc1722e5,a440f41d1db6c724] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:22.659 WARN 21062 --- [ocp-async-6,804ed9bb3001a140,f674bf8948e2b7f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:22.694 INFO 21062 --- [ocp-async-8,d0f2209c5034c6f3,bad2f6da5ad93a11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:23.874 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:23.875 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:23.983 INFO 21062 --- [prometheus-minute-schedule-1,951f6d428570ea69,905c9b053cd8eecd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:24.265 INFO 21062 --- [ocp-schedules-30,0b42e6834afc740a,ffbffc6b8be8ffc4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:55:24.659 WARN 21062 --- [ocp-async-8,e446fb71609f3ae2,aa9bd3e5956e322e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:24.694 INFO 21062 --- [ocp-async-5,2ef0d2213ae97ddb,a96674a689ecc42d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:26.660 WARN 21062 --- [ocp-async-5,a03d883ef5164c3a,41cbf86a6ff981ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:26.694 INFO 21062 --- [ocp-async-9,86e24046233e926e,cca8fc98bb107500] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:26.875 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:26.876 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9a892eb7c14f0859,833b4b001067baee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:26.984 INFO 21062 --- [prometheus-hour-schedule-1,31bb5fac4f3e035b,4e0ddd8eda4adb0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:55:28.498 INFO 21062 --- [ocp-schedules-24,bd809dd3814487ce,65437cae048d60bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:55:28.499 INFO 21062 --- [ocp-schedules-28,6231119607c89d30,0fc7a9c98aa2932b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:55:28.500 INFO 21062 --- [ocp-schedules-30,ded546c08e4bda6f,1910b9c89ff4e6ea] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:55:28.500 INFO 21062 --- [ocp-schedules-29,eab11df9ae896270,be72056a0ee17338] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:55:28.502 INFO 21062 --- [ocp-schedules-24,bd809dd3814487ce,65437cae048d60bd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:55:28.503 INFO 21062 --- [ocp-schedules-15,fe53456f9e7a28c5,3e778585cb4776f0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:55:28.504 INFO 21062 --- [ocp-schedules-12,8715c14d0f01af85,d2596686c03f6c58] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:55:28.504 INFO 21062 --- [ocp-schedules-29,eab11df9ae896270,be72056a0ee17338] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:28.504 INFO 21062 --- [ocp-schedules-30,ded546c08e4bda6f,1910b9c89ff4e6ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:28.504 INFO 21062 --- [ocp-schedules-28,6231119607c89d30,0fc7a9c98aa2932b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:55:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:55:28.510 INFO 21062 --- [ocp-schedules-27,666a83806dcec8a7,cc16994825b9119a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:55:28.510 INFO 21062 --- [ocp-schedules-4,3379ba8722b1eea0,41c1656ca513a086] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:55:28.510 INFO 21062 --- [ocp-schedules-3,fd1c11c34223f679,0cdcdb4824c275b2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:55:28.510 INFO 21062 --- [ocp-schedules-4,3379ba8722b1eea0,41c1656ca513a086] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:55:28.510 INFO 21062 --- [ocp-schedules-3,fd1c11c34223f679,0cdcdb4824c275b2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:55:28.510 INFO 21062 --- [ocp-schedules-27,666a83806dcec8a7,cc16994825b9119a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:55:28.557 INFO 21062 --- [ocp-async-6,d05db902e909bde3,3c5362ddb258d573] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:55:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:55:28.587313+08:00, lastProcessTime=2024-11-04T12:55:18.591061+08:00 2024-11-04 12:55:28.614 INFO 21062 --- [ocp-schedules-11,5c521d7bbfeda8eb,e3268bfc4359a576] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:55:28.658 INFO 21062 --- [ocp-schedules-8,c84ddc9d31a066b3,243daa2ede95f3bd] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:55:28.658 INFO 21062 --- [ocp-schedules-8,c84ddc9d31a066b3,243daa2ede95f3bd] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:55:28.660 WARN 21062 --- [ocp-async-2,3c62417ab301f482,ec9bc284a02efaed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:28.669 INFO 21062 --- [ocp-schedules-13,5df7a50c7e1ff3d4,44f1e631a7b0cd58] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:55:28.674 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:55:28.674 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:55:28.682 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:55:28.682 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:55:28.682 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:55:28.692 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:55:28.692 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:55:28.692 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:55:28.694 INFO 21062 --- [ocp-async-3,e937e2ecfdcee427,15efa808e3a3fcd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:28.696 INFO 21062 --- [ocp-schedules-20,358e2a35bf249ca7,00eb196e737d4b5d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:55:28.714 INFO 21062 --- [ocp-schedules-26,7689a959b2eb0963,141f53f423cda7ca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:55:28.719 INFO 21062 --- [ocp-schedules-26,7689a959b2eb0963,141f53f423cda7ca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:55:28.719 INFO 21062 --- [ocp-schedules-26,7689a959b2eb0963,141f53f423cda7ca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:55:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:55:28.587313+08:00 2024-11-04 12:55:28.836 INFO 21062 --- [ocp-schedules-3,25e65a739c49f495,9e71ab9ee6e09a9e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:55:28.836 INFO 21062 --- [ocp-schedules-3,25e65a739c49f495,9e71ab9ee6e09a9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:55:28.836 INFO 21062 --- [ocp-schedules-3,25e65a739c49f495,9e71ab9ee6e09a9e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:55:28.836 INFO 21062 --- [ocp-schedules-3,25e65a739c49f495,9e71ab9ee6e09a9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:55:28.837 INFO 21062 --- [ocp-schedules-3,25e65a739c49f495,9e71ab9ee6e09a9e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:55:28.837 INFO 21062 --- [ocp-schedules-3,25e65a739c49f495,9e71ab9ee6e09a9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:55:28.899 INFO 21062 --- [ocp-schedules-24,bca867628b5dde15,864cf22de9edc650] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:55:29.148 INFO 21062 --- [ocp-schedules-1,cf091d28ac57ca52,8374d7022009fb87] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:55:29.148 INFO 21062 --- [ocp-schedules-1,cf091d28ac57ca52,8374d7022009fb87] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:55:29.162 INFO 21062 --- [ocp-schedules-25,f1c7ff4ae470e087,0f7fad833da8e39c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:55:29.168 INFO 21062 --- [ocp-schedules-25,f1c7ff4ae470e087,0f7fad833da8e39c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:55:29.168 INFO 21062 --- [ocp-schedules-25,f1c7ff4ae470e087,0f7fad833da8e39c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:55:29.176 INFO 21062 --- [ocp-schedules-1,cf091d28ac57ca52,8374d7022009fb87] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:55:29.176 INFO 21062 --- [ocp-schedules-1,cf091d28ac57ca52,8374d7022009fb87] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:55:29.181 INFO 21062 --- [ocp-schedules-22,a2524993c85d780c,6e112abd45e10acf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:55:29.218 INFO 21062 --- [ocp-schedules-5,6200052053b71f27,1e8e8d759df9b3d8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:55:29.218 INFO 21062 --- [ocp-schedules-5,6200052053b71f27,1e8e8d759df9b3d8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:55:29.218 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:55:29.218 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:55:29.228 INFO 21062 --- [ocp-schedules-5,6200052053b71f27,1e8e8d759df9b3d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:55:29.228 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:55:29.229 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:55:29.230 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:55:29.230 WARN 21062 --- [ocp-schedules-5,6200052053b71f27,1e8e8d759df9b3d8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:55:29.231 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:55:29.231 INFO 21062 --- [ocp-schedules-5,6200052053b71f27,1e8e8d759df9b3d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:55:29.231 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:55:29.232 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:55:29.233 WARN 21062 --- [ocp-schedules-5,6200052053b71f27,1e8e8d759df9b3d8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:55:29.233 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:55:29.233 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:55:29.239 ERROR 21062 --- [ocp-schedules-5,6200052053b71f27,1e8e8d759df9b3d8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 12:55:29.297 INFO 21062 --- [ocp-schedules-6,82851dfbfcdbd64c,127119bba4c72202] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:55:29.297+08:00, clusters=[1] 2024-11-04 12:55:29.298 INFO 21062 --- [ocp-schedules-6,82851dfbfcdbd64c,127119bba4c72202] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:55:29.376 INFO 21062 --- [ocp-schedules-7,501a2bc0935ed24c,a69ff72620cb8930] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:55:29.376+08:00, clusters=[1] 2024-11-04 12:55:29.380 INFO 21062 --- [ocp-schedules-7,501a2bc0935ed24c,a69ff72620cb8930] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:55:29.876 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:29.877 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:29.983 INFO 21062 --- [prometheus-minute-schedule-1,358d961b7609ddd2,1792136ea362609d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:30.082 INFO 21062 --- [ocp-schedules-10,a67ff6946768cf1c,077e2e379d2e9e4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:55:30.082 INFO 21062 --- [ocp-schedules-10,dee46fb1e69acab4,70785bbec70fd157] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:55:30.086 INFO 21062 --- [ocp-schedules-10,dee46fb1e69acab4,70785bbec70fd157] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:30.116 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:55:30.116 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:55:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:55:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:55:30.608 INFO 21062 --- [ocp-schedules-16,d63ad9798a965f1d,86996903e0a715ae] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:55:30.612 INFO 21062 --- [ocp-schedules-16,d63ad9798a965f1d,86996903e0a715ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:30.660 WARN 21062 --- [ocp-async-7,1336ffb838990d84,ff23aa7e8ab3d57c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:30.694 INFO 21062 --- [ocp-async-10,996895e35a2b187f,ea71bdc20b96576b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:30.697 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:55:30.699 ERROR 21062 --- [ocp-schedules-16,d63ad9798a965f1d,86996903e0a715ae] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:55:30.700 INFO 21062 --- [ocp-schedules-16,d63ad9798a965f1d,86996903e0a715ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:55:32.660 WARN 21062 --- [ocp-async-4,9ffc51393a37dac1,b697eabedea67d99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:32.695 INFO 21062 --- [ocp-async-6,cc14d6afcc7f40be,7b40aa20da8dd752] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:32.877 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:32.878 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d2d7d49cf9f15fe1,669cc805da4cc2ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:34.292 INFO 21062 --- [ocp-schedules-13,247dff8533c78a58,6c64e87558d86b56] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:55:34.660 WARN 21062 --- [ocp-async-1,5e8a88bf986203fd,adb5ce63fc7991e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:34.695 INFO 21062 --- [ocp-async-8,9626d61d8c0b8857,3b348b1bcd4aba32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:35.878 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:35.879 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:35.983 INFO 21062 --- [prometheus-minute-schedule-1,2c8cc2e8e4624fc0,d1018db816a72d18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:36.661 WARN 21062 --- [ocp-async-3,2c66a303d484763a,d0c48f656b448ba9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:36.695 INFO 21062 --- [ocp-async-5,cca7cf16eb1bf7cb,4a5cb5c9b5bc0d51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:38.505 INFO 21062 --- [ocp-schedules-29,eab11df9ae896270,be72056a0ee17338] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:55:38.505 INFO 21062 --- [ocp-schedules-30,ded546c08e4bda6f,1910b9c89ff4e6ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:55:38.557 INFO 21062 --- [ocp-async-10,a98e7e6ea60c40c8,f1d2ccfddedd2870] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:55:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:55:38.589872+08:00, lastProcessTime=2024-11-04T12:55:28.587313+08:00 2024-11-04 12:55:38.637 INFO 21062 --- [ocp-schedules-9,3ea48a6d913b0d1e,450517244fb8d866] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:55:38.640 INFO 21062 --- [ocp-schedules-9,3ea48a6d913b0d1e,450517244fb8d866] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:38.641 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:55:38.661 WARN 21062 --- [ocp-async-9,48053dd46a4f40ee,3b09eb1881157b6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:38.695 INFO 21062 --- [ocp-async-1,7886e5849de7e99a,b835155841ec5581] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:55:38.589872+08:00 2024-11-04 12:55:38.879 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:38.880 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:38.901 INFO 21062 --- [ocp-schedules-3,81d6f788b44aecd4,db04c4aeb9b3ef57] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:55:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0504de26198d801c,100fb9891b97cf9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:55:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:55:40.661 WARN 21062 --- [ocp-async-2,4d07854fa6557488,29860e62adccfe2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:40.695 INFO 21062 --- [ocp-async-3,3474f98dacbeabe6,034b4ba8c0cb37ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:41.880 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:41.881 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:41.983 INFO 21062 --- [prometheus-minute-schedule-1,3f758035bb66f255,29928b0e249cc763] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:42.661 WARN 21062 --- [ocp-async-5,c08ec6ead5f770ca,88a176e2f0bdf9d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:42.695 INFO 21062 --- [ocp-async-10,534144c008c5ec05,97d40da0e01768f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:44.335 INFO 21062 --- [ocp-schedules-27,5e7a75cb5be6761d,ef7880fd2df961d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:55:44.661 WARN 21062 --- [ocp-async-4,82e0ffcfbeea2745,6fa12184bcda73b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:44.696 INFO 21062 --- [ocp-async-6,0f3bcceca6bb4f76,7d5fbf0914f0a7ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:44.881 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:44.882 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5abdca381d2f1ddb,3c6041935dfb7653] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:46.661 WARN 21062 --- [ocp-async-1,01ca4144707356fc,33a54aab16dd17b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:46.696 INFO 21062 --- [ocp-async-8,6a69725af854f0cc,73aabbd9a6770580] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:47.882 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:47.883 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fd518b117d15e89f,f90f25987ddbaf0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:48.557 INFO 21062 --- [ocp-async-8,b8a0bafdddb9d953,6d018f706752e405] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:55:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:55:48.589020+08:00, lastProcessTime=2024-11-04T12:55:38.589872+08:00 2024-11-04 12:55:48.641 INFO 21062 --- [ocp-schedules-9,3ea48a6d913b0d1e,450517244fb8d866] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:55:48.662 WARN 21062 --- [ocp-async-5,49211e6c1dec8307,49a9cb49c27e8881] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:48.696 INFO 21062 --- [ocp-async-10,013e3e855ec4b95f,7d6c84c8b65bdd7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:55:48.589020+08:00 2024-11-04 12:55:48.903 INFO 21062 --- [ocp-schedules-21,5f51e70a81e34cd5,91f3b1c7995110aa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:55:49.171 INFO 21062 --- [ocp-schedules-8,58d53bbcd37dd20e,813b6647198e73f1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:55:49.177 INFO 21062 --- [ocp-schedules-8,58d53bbcd37dd20e,813b6647198e73f1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:55:49.177 INFO 21062 --- [ocp-schedules-8,58d53bbcd37dd20e,813b6647198e73f1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:55:50.662 WARN 21062 --- [ocp-async-4,067499ef8576d972,392fe1223edbd30c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:50.696 INFO 21062 --- [ocp-async-9,db05343a8073b48a,2a988bbf4f529a58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:50.883 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:50.884 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:50.983 INFO 21062 --- [prometheus-minute-schedule-1,364ce8284e18d43a,cef64b8d5d3869c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:52.662 WARN 21062 --- [ocp-async-1,f857fb6416291a02,fa1e1b58714026d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:52.696 INFO 21062 --- [ocp-async-2,50d432f54d5cd598,0a7fe06daa138ace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:53.884 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:53.885 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e1a1f7c0423bfd74,24115e2756702fe8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:54.362 INFO 21062 --- [ocp-schedules-8,126c13a3dedde562,4732714e513145bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:55:54.662 WARN 21062 --- [ocp-async-3,5563669e646f88b0,b531d12a527d237f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:54.697 INFO 21062 --- [ocp-async-5,24f756606a10c1c3,4c5dcc8839778ffc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:56.662 WARN 21062 --- [ocp-async-10,29d7da6b6fda49b6,2cc79425b5b6f44b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:56.697 INFO 21062 --- [ocp-async-4,aa496484e21f8299,c76ba84b6cae3588] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:56.885 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:56.886 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:56.983 INFO 21062 --- [prometheus-minute-schedule-1,5e9433e569e148a2,c22749e57d7bf7a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:55:58.500 INFO 21062 --- [ocp-schedules-14,c8183effe7386266,ff74bb40bfcfbdaf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:55:58.500 INFO 21062 --- [ocp-schedules-23,daac28ac88dd801c,628aba2123b4769d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:55:58.502 INFO 21062 --- [ocp-schedules-20,812f4734fdb8a1b4,9cb8a2afbe0fc858] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:55:58.504 INFO 21062 --- [ocp-schedules-14,c8183effe7386266,ff74bb40bfcfbdaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:58.504 INFO 21062 --- [ocp-schedules-23,daac28ac88dd801c,628aba2123b4769d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:55:58.504 INFO 21062 --- [ocp-schedules-22,a5ffb0051be60c7d,22b5ea11fb11f849] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:55:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:55:58.512 INFO 21062 --- [ocp-schedules-13,2bb75f74dde075fb,9975ed6f5db23b8a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:55:58.512 INFO 21062 --- [ocp-schedules-9,66f993c94e9b0a5f,e9781d3ed0a8f651] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:55:58.512 INFO 21062 --- [ocp-schedules-9,66f993c94e9b0a5f,e9781d3ed0a8f651] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:55:58.512 INFO 21062 --- [ocp-schedules-13,2bb75f74dde075fb,9975ed6f5db23b8a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:55:58.512 INFO 21062 --- [ocp-schedules-18,816dc4bdd36bbac0,cf1bdbfe711bec63] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:55:58.512 INFO 21062 --- [ocp-schedules-18,816dc4bdd36bbac0,cf1bdbfe711bec63] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:55:58.558 INFO 21062 --- [ocp-async-6,6055cbd393ce05aa,2e324d9031f88b7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:55:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:55:58.588023+08:00, lastProcessTime=2024-11-04T12:55:48.589020+08:00 2024-11-04 12:55:58.663 WARN 21062 --- [ocp-async-1,9db6fa3c48a8febb,9e3247e09fb0bb88] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:55:58.672 INFO 21062 --- [ocp-schedules-8,37bc68a0f38cfc6d,1aa31d5aab51523b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:55:58.697 INFO 21062 --- [ocp-async-8,cf50454bfe81c181,c904df8fcdc681dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:55:58.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:55:58.588023+08:00 2024-11-04 12:55:58.906 INFO 21062 --- [ocp-schedules-3,0377f2fc2372cf9a,b5875b58bb0b542a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:55:59.886 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:55:59.887 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:55:59.983 INFO 21062 --- [prometheus-minute-schedule-1,60e474f660c91576,8ddf08b276f0f01e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696160, interval=60 2024-11-04 12:56:00.117 INFO 21062 --- [ocp-schedules-10,dee46fb1e69acab4,70785bbec70fd157] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:56:00.117 INFO 21062 --- [ocp-schedules-10,ce337d0e88fa770d,1ddaf4f402e572e8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:56:00.121 INFO 21062 --- [ocp-schedules-10,ce337d0e88fa770d,1ddaf4f402e572e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:00.152 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:56:00.152 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:56:00.663 WARN 21062 --- [ocp-async-8,c8a5cbdd0b19d5a0,10afc57e3381ede2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:00.697 INFO 21062 --- [ocp-async-5,a8f2f09c70324942,44385fb5c73c05a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:02.663 WARN 21062 --- [ocp-async-5,eb51e248baa78ce2,7916bfde49614568] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:02.697 INFO 21062 --- [ocp-async-6,184d7f57d0c819a7,172be74830177496] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:02.887 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:02.888 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:02.983 INFO 21062 --- [prometheus-minute-schedule-1,20275121e047e8d4,502d92508e907b54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:04.402 INFO 21062 --- [ocp-schedules-4,cda236fdff14da30,8d9522d8a977e2b6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:56:04.663 WARN 21062 --- [ocp-async-6,9a51684eaf6b6308,71e50c1815dba19c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:04.697 INFO 21062 --- [ocp-async-2,223a5cba0551618c,0ed2880863b16ae8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:05.888 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:05.889 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:05.983 INFO 21062 --- [prometheus-minute-schedule-1,3fc6b6210c46d784,0348f6ae8bf38c7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:06.664 WARN 21062 --- [ocp-async-2,64c94afc3fd5e646,4e36f014880a2475] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:06.698 INFO 21062 --- [ocp-async-7,81bed10f78192b20,8eb7aff1b496c36d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:08.505 INFO 21062 --- [ocp-schedules-23,daac28ac88dd801c,628aba2123b4769d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:56:08.505 INFO 21062 --- [ocp-schedules-14,c8183effe7386266,ff74bb40bfcfbdaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:56:08.558 INFO 21062 --- [ocp-async-8,2f026218a9f22b23,a4015758c82ff5ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:56:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:56:08.587831+08:00, lastProcessTime=2024-11-04T12:55:58.588023+08:00 2024-11-04 12:56:08.664 WARN 21062 --- [ocp-async-5,d8334f43835d59e0,f01e03c81adf857b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:08.698 INFO 21062 --- [ocp-async-4,74075c119c72cfcb,ead52a58c87dc458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:56:08.587831+08:00 2024-11-04 12:56:08.889 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:08.890 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:08.909 INFO 21062 --- [ocp-schedules-15,1993985f6cc91b93,d474f85683f3076e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:56:08.983 INFO 21062 --- [prometheus-minute-schedule-1,cf27d648981a1e8a,1ebdcc2d6592a782] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:09.179 INFO 21062 --- [ocp-schedules-21,f732ccf2df1d8c66,a0a8b60436341cc1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:56:09.186 INFO 21062 --- [ocp-schedules-21,f732ccf2df1d8c66,a0a8b60436341cc1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:56:09.186 INFO 21062 --- [ocp-schedules-21,f732ccf2df1d8c66,a0a8b60436341cc1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:56:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:56:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:56:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:56:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:56:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:56:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:56:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:56:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:56:10.664 WARN 21062 --- [ocp-async-4,10552b95b481a35e,68b4d165581fd9fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:10.698 INFO 21062 --- [ocp-async-1,3bca1735b1b4e9b1,f84d4d4284ddbd22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=4, affectRows=4 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=4, validCount=4 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=13, affectRows=13 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=10, affectRows=10 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=13, validCount=13 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=13, affectRows=13 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=24, affectRows=24 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=10, validCount=10 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=32, affectRows=32 2024-11-04 12:56:11.125 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=20, affectRows=20 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=13, validCount=13 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=24, validCount=24 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=1, affectRows=1 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=19, validCount=19 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=1, validCount=1 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=32, validCount=32 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=33, affectRows=33 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=64, affectRows=64 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=47, affectRows=47 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696100, nodeCount=61, affectRows=61 2024-11-04 12:56:11.126 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=33, validCount=33 2024-11-04 12:56:11.127 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=47, validCount=47 2024-11-04 12:56:11.127 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=63, validCount=63 2024-11-04 12:56:11.127 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696100, nodeCount=60, validCount=60 2024-11-04 12:56:11.890 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:11.891 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:11.983 INFO 21062 --- [prometheus-minute-schedule-1,79b0d84603798168,3d4e6027ad064b47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:12.664 WARN 21062 --- [ocp-async-1,f1608c1979fbd608,e80b6e1f2fe40bc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:12.698 INFO 21062 --- [ocp-async-8,abff229dc302ea2b,283bc0f7c66b40cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:13.984 INFO 21062 --- [prometheus-cache-monitor1,c07d76a2ded3ce91,a4fe9e21085dcda0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:56:13.985 INFO 21062 --- [prometheus-cache-monitor1,c07d76a2ded3ce91,a4fe9e21085dcda0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:56:14.428 INFO 21062 --- [ocp-schedules-22,14a782f2f38d07c1,4a2d08895f03587d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:56:14.664 WARN 21062 --- [ocp-async-8,f7643d12a8d02c9e,d4b10f121c4ab3aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:14.698 INFO 21062 --- [ocp-async-10,e5cc2483dd68b7da,850ecc23777aa775] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:14.891 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:14.892 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:14.983 INFO 21062 --- [prometheus-minute-schedule-1,0ecf7869babb5ed2,58d6e89f4d7e7007] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:16.665 WARN 21062 --- [ocp-async-10,b550a1b7b64240e0,6f9a71e81e190039] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:16.699 INFO 21062 --- [ocp-async-9,2fc52157693d24e6,7ca674fcff1d5a8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:17.892 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:17.893 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:17.983 INFO 21062 --- [prometheus-minute-schedule-1,c83f8c7906154bae,8623a9afadd613c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:18.558 INFO 21062 --- [ocp-async-2,088155015c37201c,23a72fc48bd78602] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:56:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:56:18.588228+08:00, lastProcessTime=2024-11-04T12:56:08.587831+08:00 2024-11-04 12:56:18.665 WARN 21062 --- [ocp-async-1,aadf739d49a9e2cc,4b86f8233dbb09d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:18.699 INFO 21062 --- [ocp-async-8,b2d7a83cfa8f5b8f,1ce7f02249c943c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:56:18.588228+08:00 2024-11-04 12:56:18.912 INFO 21062 --- [ocp-schedules-12,eff147e439264517,487315a850f11af1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:56:20.665 WARN 21062 --- [ocp-async-8,2d9f3befbaceba88,d77731c38ab55d2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:20.699 INFO 21062 --- [ocp-async-10,ea3743ae461a4573,44782360ba549761] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:20.893 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:20.894 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:20.983 INFO 21062 --- [prometheus-minute-schedule-1,0b0be22ed5053f58,a291d7cf1dfdfabe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:22.665 WARN 21062 --- [ocp-async-10,0eb6d2cd875e6c4c,67a5f6a34a1d3573] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:22.699 INFO 21062 --- [ocp-async-2,1424f5bc5b5a2e7f,d6f4ff62de3f37b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:23.894 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:23.896 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ce3f1d890e5f74b0,6b04bd76264c4825] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:24.468 INFO 21062 --- [ocp-schedules-15,b5de1af06e57eb75,100288e1d63dd07c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:56:24.665 WARN 21062 --- [ocp-async-2,2a1a066f63028b53,9ab6ecee1ee4a8e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:24.699 INFO 21062 --- [ocp-async-3,c7ef1d3da53f66cd,ab3ca02f79a91ad1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:26.666 WARN 21062 --- [ocp-async-3,07c112dcf206c3fd,903629e333fc39b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:26.700 INFO 21062 --- [ocp-async-5,dd1e940bc90fcb42,6ff56fb14cb7d55d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:26.896 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:26.897 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:26.983 INFO 21062 --- [prometheus-minute-schedule-1,16c06d2f0fcd5be4,270aa69630e5a53d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ac420d31f5f9e2eb,6e32a2f51171b109] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:56:28.498 INFO 21062 --- [ocp-schedules-20,5bfdda04b2dc8438,6cc78f58a1d707a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:56:28.499 INFO 21062 --- [ocp-schedules-11,17677a9b2f777f85,0e1e6a611dac3670] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:56:28.500 INFO 21062 --- [ocp-schedules-6,45d86b19c770ced4,31f1554c7348239e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:56:28.500 INFO 21062 --- [ocp-schedules-3,f189fe120b7555fc,2eec7e1a57f29928] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:56:28.542 INFO 21062 --- [ocp-schedules-12,b8eaa0327508d322,8074648560411df6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:56:28.542 INFO 21062 --- [ocp-schedules-6,45d86b19c770ced4,31f1554c7348239e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:28.542 INFO 21062 --- [ocp-schedules-3,f189fe120b7555fc,2eec7e1a57f29928] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:28.543 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:56:28.544 INFO 21062 --- [ocp-schedules-30,a56b9505a08810f1,01a29dc59390025b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:56:28.544 INFO 21062 --- [ocp-schedules-20,5bfdda04b2dc8438,6cc78f58a1d707a6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:56:28.545 INFO 21062 --- [ocp-schedules-11,17677a9b2f777f85,0e1e6a611dac3670] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:56:28.553 INFO 21062 --- [ocp-schedules-22,f3339696225cce36,8e929040cd9edebb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:56:28.553 INFO 21062 --- [ocp-schedules-22,f3339696225cce36,8e929040cd9edebb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:56:28.553 INFO 21062 --- [ocp-schedules-23,05a5cc6fe607aa61,8501d02ac013cfa1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:56:28.553 INFO 21062 --- [ocp-schedules-23,05a5cc6fe607aa61,8501d02ac013cfa1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:56:28.554 INFO 21062 --- [ocp-schedules-18,d0b9f25c1c01ce01,a26f432deeee2708] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:56:28.554 INFO 21062 --- [ocp-schedules-18,d0b9f25c1c01ce01,a26f432deeee2708] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:56:28.556 INFO 21062 --- [ocp-schedules-7,80e15193dbdd0a85,e6fc4ca1debec94f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:56:28.556 INFO 21062 --- [ocp-schedules-7,80e15193dbdd0a85,e6fc4ca1debec94f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:56:28.558 INFO 21062 --- [ocp-async-10,871a6955aa8269b9,0726966c459ea586] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:56:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:56:28.588460+08:00, lastProcessTime=2024-11-04T12:56:18.588228+08:00 2024-11-04 12:56:28.616 INFO 21062 --- [ocp-schedules-29,2a6ff5e7df3a1ad3,2f73bcbb431bb169] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:56:28.663 INFO 21062 --- [ocp-schedules-27,19c95617b21b7985,886d9a80dcfbb52d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:56:28.663 INFO 21062 --- [ocp-schedules-27,19c95617b21b7985,886d9a80dcfbb52d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:56:28.666 WARN 21062 --- [ocp-async-4,2839b3794835d605,dce4b22231ba451f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:28.674 INFO 21062 --- [ocp-schedules-24,d3a257e3ea08aac7,cfee2416507400f1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:56:28.700 INFO 21062 --- [ocp-async-7,c439cf5ff7283738,203a49ba700f220e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:28.719 INFO 21062 --- [ocp-schedules-2,7c32f5bd5ac6605c,a8fad1a8ea75688b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:56:28.724 INFO 21062 --- [ocp-schedules-2,7c32f5bd5ac6605c,a8fad1a8ea75688b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:56:28.724 INFO 21062 --- [ocp-schedules-2,7c32f5bd5ac6605c,a8fad1a8ea75688b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:56:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:56:28.588460+08:00 2024-11-04 12:56:28.838 INFO 21062 --- [ocp-schedules-26,f734bc2c77df4db2,7eacb54edaf57338] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:56:28.838 INFO 21062 --- [ocp-schedules-26,f734bc2c77df4db2,7eacb54edaf57338] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:56:28.839 INFO 21062 --- [ocp-schedules-26,f734bc2c77df4db2,7eacb54edaf57338] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:56:28.839 INFO 21062 --- [ocp-schedules-26,f734bc2c77df4db2,7eacb54edaf57338] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:56:28.840 INFO 21062 --- [ocp-schedules-26,f734bc2c77df4db2,7eacb54edaf57338] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:56:28.840 INFO 21062 --- [ocp-schedules-26,f734bc2c77df4db2,7eacb54edaf57338] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:56:28.914 INFO 21062 --- [ocp-schedules-30,61c7d0ed3617cdf3,cdd669a107a049dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:56:29.177 INFO 21062 --- [ocp-schedules-21,1ac5f4418ef34450,41ba7e91dc62677f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:56:29.177 INFO 21062 --- [ocp-schedules-21,1ac5f4418ef34450,41ba7e91dc62677f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:56:29.188 INFO 21062 --- [ocp-schedules-28,18d8d581e54c98da,f22ddfeef46e95d1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:56:29.195 INFO 21062 --- [ocp-schedules-28,18d8d581e54c98da,f22ddfeef46e95d1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:56:29.195 INFO 21062 --- [ocp-schedules-28,18d8d581e54c98da,f22ddfeef46e95d1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:56:29.203 INFO 21062 --- [ocp-schedules-21,1ac5f4418ef34450,41ba7e91dc62677f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:56:29.203 INFO 21062 --- [ocp-schedules-21,1ac5f4418ef34450,41ba7e91dc62677f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:56:29.208 INFO 21062 --- [ocp-schedules-4,5c737bdfb862704a,2b9bf81824942dd9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:56:29.222 INFO 21062 --- [ocp-schedules-5,ffcb3eee946782e5,6b073acbbdd256c0] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:56:29.226 INFO 21062 --- [ocp-schedules-5,ffcb3eee946782e5,6b073acbbdd256c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:29.248 INFO 21062 --- [ocp-schedules-8,e6a5f4ecfe542457,730fb9e67c63ea4a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:56:29.248 INFO 21062 --- [ocp-schedules-8,e6a5f4ecfe542457,730fb9e67c63ea4a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:56:29.248 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:56:29.248 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:56:29.257 INFO 21062 --- [ocp-schedules-8,e6a5f4ecfe542457,730fb9e67c63ea4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:56:29.258 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:56:29.259 WARN 21062 --- [ocp-schedules-8,e6a5f4ecfe542457,730fb9e67c63ea4a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:56:29.260 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:56:29.260 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:56:29.260 INFO 21062 --- [ocp-schedules-8,e6a5f4ecfe542457,730fb9e67c63ea4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:56:29.260 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:56:29.261 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:56:29.261 WARN 21062 --- [ocp-schedules-8,e6a5f4ecfe542457,730fb9e67c63ea4a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:56:29.262 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:56:29.262 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:56:29.263 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:56:29.267 ERROR 21062 --- [ocp-schedules-8,e6a5f4ecfe542457,730fb9e67c63ea4a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 12:56:29.282 ERROR 21062 --- [ocp-schedules-5,ffcb3eee946782e5,6b073acbbdd256c0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:56:29.283 INFO 21062 --- [ocp-schedules-5,ffcb3eee946782e5,6b073acbbdd256c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:56:29.325 INFO 21062 --- [ocp-schedules-15,9f6f6f86e6d1ba30,99061585246bfd06] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:56:29.325+08:00, clusters=[1] 2024-11-04 12:56:29.326 INFO 21062 --- [ocp-schedules-15,9f6f6f86e6d1ba30,99061585246bfd06] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:56:29.405 INFO 21062 --- [ocp-schedules-14,7b2b82a341cd7949,15ac653401edaec2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:56:29.405+08:00, clusters=[1] 2024-11-04 12:56:29.409 INFO 21062 --- [ocp-schedules-14,7b2b82a341cd7949,15ac653401edaec2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:56:29.897 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:29.898 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:29.983 INFO 21062 --- [prometheus-minute-schedule-1,912ee37b8dd5c72e,d90466dc3d32842f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:30.153 INFO 21062 --- [ocp-schedules-10,ce337d0e88fa770d,1ddaf4f402e572e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:56:30.153 INFO 21062 --- [ocp-schedules-10,7e76477ca4735aee,c9e3a79a9e639455] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:56:30.156 INFO 21062 --- [ocp-schedules-10,7e76477ca4735aee,c9e3a79a9e639455] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:30.185 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:56:30.185 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:56:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:56:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:56:30.666 WARN 21062 --- [ocp-async-1,aca3cc97f27d940d,2f6a29cc596a5b24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:30.700 INFO 21062 --- [ocp-schedules-14,91439b082977a619,cfb8783691206d1c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:56:30.700 INFO 21062 --- [ocp-async-6,87bde9e97a2992f0,49892e985760a0db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:30.704 INFO 21062 --- [ocp-schedules-14,91439b082977a619,cfb8783691206d1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:30.786 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:56:30.788 ERROR 21062 --- [ocp-schedules-14,91439b082977a619,cfb8783691206d1c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:56:30.788 INFO 21062 --- [ocp-schedules-14,91439b082977a619,cfb8783691206d1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:56:32.666 WARN 21062 --- [ocp-async-5,90bc3334178779e0,a812a71d42db9990] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:32.700 INFO 21062 --- [ocp-async-10,f5b01d89bc3880c0,de54aff67afed783] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:32.898 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:32.899 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:32.983 INFO 21062 --- [prometheus-minute-schedule-1,884d4da9a22f66b2,c956c4e5be42425b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:34.493 INFO 21062 --- [ocp-schedules-8,3d5e97d6175cc87d,a712ec00c6f2d281] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:56:34.667 WARN 21062 --- [ocp-async-9,1172d3788a2e9a51,79c85c4a2a934cc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:34.700 INFO 21062 --- [ocp-async-2,8ee8adbf71f2f504,988375f2d9d74978] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:35.899 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:35.900 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:35.983 INFO 21062 --- [prometheus-minute-schedule-1,96830de73ccd0109,6bc79c2b7ea106fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:36.667 WARN 21062 --- [ocp-async-7,0c8b86d632e082a9,6a0135859fe2cdae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:36.700 INFO 21062 --- [ocp-async-3,0a054e7ae50a2525,4eaeb411f64fba25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:38.543 INFO 21062 --- [ocp-schedules-3,f189fe120b7555fc,2eec7e1a57f29928] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:56:38.544 INFO 21062 --- [ocp-schedules-6,45d86b19c770ced4,31f1554c7348239e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:56:38.558 INFO 21062 --- [ocp-async-3,2bd8af286e4798be,1074df6e6b8f3876] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:56:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:56:38.587846+08:00, lastProcessTime=2024-11-04T12:56:28.588460+08:00 2024-11-04 12:56:38.667 WARN 21062 --- [ocp-async-8,630080cc68107afb,469da660bcc4a520] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:38.701 INFO 21062 --- [ocp-async-9,8fb14649c2b5f691,af72a6da25349e4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:56:38.587846+08:00 2024-11-04 12:56:38.900 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:38.901 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:38.917 INFO 21062 --- [ocp-schedules-2,2c2b2cae45888f9f,f4045501b14def45] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:56:38.983 INFO 21062 --- [prometheus-minute-schedule-1,2cf3cb1d1e23a208,156cb6a15b925aaa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:56:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:56:40.667 WARN 21062 --- [ocp-async-4,9a347c75dc87af3b,cec2da6fbe1ad17c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:40.701 INFO 21062 --- [ocp-async-7,55c49f4b8ab6698d,459e7ec7eb79e5d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:56:41.823 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:56:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 12:56:41.901 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:41.902 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d168723773615bcb,b6d16ac04fd82688] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:42.667 WARN 21062 --- [ocp-async-1,a27a420cfc1aa298,6e9a9d5895197814] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:42.701 INFO 21062 --- [ocp-async-3,bc4800150391838a,4df2bba06488e989] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:44.487 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:56:44.487 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:56:44.494 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:56:44.494 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:56:44.494 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:56:44.495 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:56:44.500 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:56:44.500 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:56:44.500 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:56:44.500 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:56:44.534 INFO 21062 --- [ocp-schedules-27,50f8e5cfa48f165e,856a44603123c5dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:56:44.538 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:56:44.551 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:56:44.551 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:56:44.552 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:56:44.552 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:56:44.552 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:56:44.561 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:56:44.668 WARN 21062 --- [ocp-async-3,55b1cc5918fcd761,fdebd2bbe5d88a70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:44.701 INFO 21062 --- [ocp-async-10,09a5d79a0ce4f931,ec98d61d2531f70a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:44.902 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:44.903 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a775122351a5d57e,d533516533ccb12f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:45.107 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:56:45.722 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.221944198S 2024-11-04 12:56:45.723 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:56:45.824 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:56:45.824 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:56:45.848 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:56:45.908 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:56:45.908 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:56:45.908 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:56:45.908 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:56:45.908 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:56:45.908 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:56:45.919 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:56:46.101 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] o.s.boot.SpringApplication : Started application in 1.606 seconds (JVM running for 1619.78) 2024-11-04 12:56:46.101 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.607008716S 2024-11-04 12:56:46.523 INFO 21062 --- [ocp-schedules-7,ade3d8f7736ab1df,bb6845f6ac73a00c] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:56:46.668 WARN 21062 --- [ocp-async-10,5bbafd6ed9be8811,d7223a8bb984d37f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:46.701 INFO 21062 --- [ocp-async-2,c224260fe7d4b12c,a00a201712c4f4d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:47.903 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:47.904 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:47.983 INFO 21062 --- [prometheus-minute-schedule-1,44ff0150dc260107,f8ee39825969f3ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:48.559 INFO 21062 --- [ocp-async-4,9d1a62e9fc29b206,13ea7a2b721ffcaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:56:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:56:48.589669+08:00, lastProcessTime=2024-11-04T12:56:38.587846+08:00 2024-11-04 12:56:48.642 INFO 21062 --- [ocp-schedules-4,b0a0cf1a665716ec,1bbe7e3f941ab9bb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:56:48.647 INFO 21062 --- [ocp-schedules-4,b0a0cf1a665716ec,1bbe7e3f941ab9bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:48.647 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:56:48.668 WARN 21062 --- [ocp-async-7,bb3ebc2531c13085,47f89f5e547450c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:48.701 INFO 21062 --- [ocp-async-6,ece7bb72957649f7,b29a3625bbcef718] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:56:48.589669+08:00 2024-11-04 12:56:48.920 INFO 21062 --- [ocp-schedules-30,e95784f035f0d43b,c6bcd98f4fc933b6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:56:49.197 INFO 21062 --- [ocp-schedules-1,02374b8a28412e76,f07b2ed59eb26e9a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:56:49.204 INFO 21062 --- [ocp-schedules-1,02374b8a28412e76,f07b2ed59eb26e9a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:56:49.204 INFO 21062 --- [ocp-schedules-1,02374b8a28412e76,f07b2ed59eb26e9a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:56:50.668 WARN 21062 --- [ocp-async-6,b53291be437a0442,6f17f532fbd5755b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:50.702 INFO 21062 --- [ocp-async-8,a41503a5cfce2e4e,acacfb8a93f6ad63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:50.904 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:50.905 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:50.983 INFO 21062 --- [prometheus-minute-schedule-1,cd5cd723b570ec9f,4e1053bcb5933715] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:52.668 WARN 21062 --- [ocp-async-8,c97581e90b0a8480,d8b399e451b2cc81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:52.702 INFO 21062 --- [ocp-async-4,03445b161eed58bf,b75216d12a7fe843] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:53.905 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:53.906 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e35fd021a5fcd1b7,3629324836a61ffe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:54.559 INFO 21062 --- [ocp-schedules-3,de27957163350865,5ce6a15754295f8b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:56:54.669 WARN 21062 --- [ocp-async-4,97d93ac7a97b2782,7f3ce498a9f51256] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:54.702 INFO 21062 --- [ocp-async-3,050ea4c012eeb6cd,eccf9523e3547393] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:56.669 WARN 21062 --- [ocp-async-3,c6c77fd86a5acb62,8b9654e4a299759a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:56.702 INFO 21062 --- [ocp-async-10,5ef4c7a2cbe66459,f6adc39efb430fa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:56.906 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:56.907 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f5fdc78a0830e0ac,fa6eac73606f5727] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:56:58.500 INFO 21062 --- [ocp-schedules-1,2542d338d097f63c,5ee84228d31e1020] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:56:58.500 INFO 21062 --- [ocp-schedules-15,90d3f6e5aba44bcb,47ae1efd94e2890b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:56:58.502 INFO 21062 --- [ocp-schedules-14,fbe9344f5ded31ee,3fc386f935f5fa7b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:56:58.504 INFO 21062 --- [ocp-schedules-1,2542d338d097f63c,5ee84228d31e1020] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:58.504 INFO 21062 --- [ocp-schedules-15,90d3f6e5aba44bcb,47ae1efd94e2890b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:56:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:56:58.504 INFO 21062 --- [ocp-schedules-27,6215f0e1ef8dcd26,5067dc1efb418382] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:56:58.511 INFO 21062 --- [ocp-schedules-21,9a962f8338989c83,589cee5901767448] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:56:58.511 INFO 21062 --- [ocp-schedules-8,bb292b6839ab8f27,bc6acc90ac7c1d67] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:56:58.512 INFO 21062 --- [ocp-schedules-21,9a962f8338989c83,589cee5901767448] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:56:58.512 INFO 21062 --- [ocp-schedules-8,bb292b6839ab8f27,bc6acc90ac7c1d67] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:56:58.512 INFO 21062 --- [ocp-schedules-17,92730396dc4d7b69,afaa1575add12107] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:56:58.512 INFO 21062 --- [ocp-schedules-17,92730396dc4d7b69,afaa1575add12107] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:56:58.559 INFO 21062 --- [ocp-async-10,d3ff87e932dba04b,03820b186b8cabfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:56:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:56:58.587784+08:00, lastProcessTime=2024-11-04T12:56:48.589669+08:00 2024-11-04 12:56:58.648 INFO 21062 --- [ocp-schedules-4,b0a0cf1a665716ec,1bbe7e3f941ab9bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:56:58.669 WARN 21062 --- [ocp-async-8,0489ffe9914bbd2e,a1791366b8ff48a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:56:58.676 INFO 21062 --- [ocp-schedules-23,4fcfda862eee14ce,f8a8ed9b32d63196] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:56:58.702 INFO 21062 --- [ocp-async-2,f28d9b3a51c52d25,9c933b0f57ffb30e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:56:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:56:58.587784+08:00 2024-11-04 12:56:58.922 INFO 21062 --- [ocp-schedules-24,6a3ecd201d7305e0,b6942c69ccd78a87] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:56:59.907 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:56:59.908 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:56:59.983 INFO 21062 --- [prometheus-minute-schedule-1,68036a450f24b2f4,e0c13dab6c1861d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696220, interval=60 2024-11-04 12:57:00.185 INFO 21062 --- [ocp-schedules-10,7e76477ca4735aee,c9e3a79a9e639455] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:57:00.185 INFO 21062 --- [ocp-schedules-10,c65bf7216dfcf4b0,f2eeab73472c715d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:57:00.190 INFO 21062 --- [ocp-schedules-10,c65bf7216dfcf4b0,f2eeab73472c715d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:00.221 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:57:00.221 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:57:00.669 WARN 21062 --- [ocp-async-4,f83f77dce948269c,28bc584cbf6f9b2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:00.703 INFO 21062 --- [ocp-async-1,08d122a4318b18cf,7e24ec2fefc19078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:02.669 WARN 21062 --- [ocp-async-3,baa256f74c064e36,525ea3906d162bef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:02.703 INFO 21062 --- [ocp-async-6,b2e19220457950ea,41dec3b26be965a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:02.908 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:02.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:02.983 INFO 21062 --- [prometheus-minute-schedule-1,33cfe8c6ec6aa54a,e3885899e5111b4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:04.602 INFO 21062 --- [ocp-schedules-3,6f954351776064ad,737022d4dd621b2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:57:04.669 WARN 21062 --- [ocp-async-5,6aaf510869586186,09f2e83799c099ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:04.703 INFO 21062 --- [ocp-async-9,a7bbf6540af0d540,79d06bc0693731b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:05.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:05.910 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:05.983 INFO 21062 --- [prometheus-minute-schedule-1,bffc77b3a8a37c30,da5d3404b1891b6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:06.670 WARN 21062 --- [ocp-async-2,960f60bd80903fa7,3531498f23c061f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:06.703 INFO 21062 --- [ocp-async-7,f434588b28374db5,1cf88e40c18e136c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:08.505 INFO 21062 --- [ocp-schedules-15,90d3f6e5aba44bcb,47ae1efd94e2890b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:57:08.505 INFO 21062 --- [ocp-schedules-1,2542d338d097f63c,5ee84228d31e1020] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:57:08.559 INFO 21062 --- [ocp-async-7,280abbf4d2e9fa43,4d52fa09e6cab80f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:57:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:57:08.588113+08:00, lastProcessTime=2024-11-04T12:56:58.587784+08:00 2024-11-04 12:57:08.670 WARN 21062 --- [ocp-async-3,da3d2d8a49414e37,d6b5f00e54c978fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:08.703 INFO 21062 --- [ocp-async-6,a8d263bffd460d87,3174481d766d6f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:57:08.588113+08:00 2024-11-04 12:57:08.910 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:08.911 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:08.926 INFO 21062 --- [ocp-schedules-5,545f64015ae8cb6d,253d898911474961] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:57:08.983 INFO 21062 --- [prometheus-minute-schedule-1,bd355321ec76064f,6f774d11745d55a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:09.207 INFO 21062 --- [ocp-schedules-2,65ab8dcc52869a3e,9167fdc527c95030] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:57:09.214 INFO 21062 --- [ocp-schedules-2,65ab8dcc52869a3e,9167fdc527c95030] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:57:09.214 INFO 21062 --- [ocp-schedules-2,65ab8dcc52869a3e,9167fdc527c95030] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:57:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:57:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:57:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:57:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:57:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:57:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:57:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:57:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:57:10.672 WARN 21062 --- [ocp-async-5,742d254b239e3e3c,addb845fbc642c6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:10.704 INFO 21062 --- [ocp-async-9,7aaf3898b24a4224,d3a39ed7b6eeecc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:11.129 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=5, affectRows=5 2024-11-04 12:57:11.129 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=1, affectRows=1 2024-11-04 12:57:11.129 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=1, validCount=1 2024-11-04 12:57:11.129 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=11, affectRows=11 2024-11-04 12:57:11.129 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=5, validCount=5 2024-11-04 12:57:11.129 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=21, affectRows=21 2024-11-04 12:57:11.129 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=10, validCount=10 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=21, validCount=21 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=23, affectRows=23 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=10, affectRows=10 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=43, affectRows=43 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=38, affectRows=38 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=10, validCount=10 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=23, validCount=23 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=43, validCount=43 2024-11-04 12:57:11.130 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=38, validCount=38 2024-11-04 12:57:11.131 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=82, affectRows=82 2024-11-04 12:57:11.132 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=81, validCount=81 2024-11-04 12:57:11.132 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696160, nodeCount=88, affectRows=88 2024-11-04 12:57:11.132 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696160, nodeCount=87, validCount=87 2024-11-04 12:57:11.911 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:11.912 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b9cf126df5475274,1a847e36117f9b01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:12.672 WARN 21062 --- [ocp-async-2,e12e0f50a4ce9153,e8c0291b91105f35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:12.704 INFO 21062 --- [ocp-async-7,1504c2a8a43257ed,5430bc6a50e96fdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:13.984 INFO 21062 --- [prometheus-cache-monitor1,e71c21b11f344e07,b184951d547e6dbd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:57:13.984 INFO 21062 --- [prometheus-cache-monitor1,e71c21b11f344e07,b184951d547e6dbd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:57:14.252 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:53:14.215, end=2024-11-04T12:55:14.249} 2024-11-04 12:57:14.263 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:57:14.263 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:57:14.628 INFO 21062 --- [ocp-schedules-21,fa61fe16fdc27ec4,5a9cd965d60073da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:57:14.672 WARN 21062 --- [ocp-async-1,3f9cd63952fd7275,e247161e0aafef27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:14.704 INFO 21062 --- [ocp-async-10,02c11202b7bb4476,314298870066c773] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:14.912 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:14.916 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:14.983 INFO 21062 --- [prometheus-minute-schedule-1,31c82adc4bad4c99,7d8ee30ca113a4ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:16.672 WARN 21062 --- [ocp-async-6,68a4d13d8b749111,541d750abb24140b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:16.704 INFO 21062 --- [ocp-async-8,0a3f00a87747d219,887e42145afcdb79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:17.916 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:17.917 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:17.983 INFO 21062 --- [prometheus-minute-schedule-1,e789d84028e04f6d,01f4cb31a2edc8b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:18.559 INFO 21062 --- [ocp-async-8,6229a733db53c737,40efefe311c113ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:57:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:57:18.588460+08:00, lastProcessTime=2024-11-04T12:57:08.588113+08:00 2024-11-04 12:57:18.672 WARN 21062 --- [ocp-async-2,2c64c3b22beb469e,04726f9f9a0c6eec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:18.704 INFO 21062 --- [ocp-async-7,626c8be91dbdd9e2,d99b4be765d1c6b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:18.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:57:18.588460+08:00 2024-11-04 12:57:18.928 INFO 21062 --- [ocp-schedules-22,28834c8eb183707b,c7f3ba5b17e8c32e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:57:20.672 WARN 21062 --- [ocp-async-1,04ea3f8557719101,1873d25306360ab4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:20.704 INFO 21062 --- [ocp-async-10,34a961202c0be0fa,6ac45e6db469934d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:20.917 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:20.918 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b7d155aba120228d,705ee021a295cfed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:22.672 WARN 21062 --- [ocp-async-6,27fa3ae112ff6e56,16e9e0cdf875b04f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:22.705 INFO 21062 --- [ocp-async-8,1d07bf495003b997,2e220d1d9171642b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:23.918 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:23.919 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:23.983 INFO 21062 --- [prometheus-minute-schedule-1,160ad58af48356df,058897ca549283f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:24.670 INFO 21062 --- [ocp-schedules-6,cebdde48be4af8bd,d753eb658a1f6a3e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:57:24.673 WARN 21062 --- [ocp-async-9,73a5a0b515848b29,cafd240a36a02c90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:24.705 INFO 21062 --- [ocp-async-4,20b0220efae23ff7,75f98fed38229a40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:26.673 WARN 21062 --- [ocp-async-7,307586a2d8d24a38,41dcea1825380c19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:26.705 INFO 21062 --- [ocp-async-3,3be97a6913c46362,faeb88601b93603b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:26.919 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:26.920 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:26.983 INFO 21062 --- [prometheus-minute-schedule-1,c8ac943afbe76754,f3e0d9ec3fb09179] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c22df32265ec5e62,da462498017590cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:57:28.498 INFO 21062 --- [ocp-schedules-24,83d9cfe6fa67e427,27be7fe01cf4899d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:57:28.500 INFO 21062 --- [ocp-schedules-2,7a78b3e11eec2849,e1f7bc49b2110aba] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:57:28.500 INFO 21062 --- [ocp-schedules-17,c9ff5be9c15b66ab,0951f48834b93ab5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:57:28.500 INFO 21062 --- [ocp-schedules-11,9c73a6b099b620e0,6a93781663974b40] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:57:28.502 INFO 21062 --- [ocp-schedules-24,83d9cfe6fa67e427,27be7fe01cf4899d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:57:28.502 INFO 21062 --- [ocp-schedules-22,9295d369eac9cfc2,661106c1c17a6e19] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:57:28.504 INFO 21062 --- [ocp-schedules-2,7a78b3e11eec2849,e1f7bc49b2110aba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:28.504 INFO 21062 --- [ocp-schedules-11,9c73a6b099b620e0,6a93781663974b40] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:57:28.504 INFO 21062 --- [ocp-schedules-28,71eac2885fdd6875,e86346d22f42e8be] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:57:28.505 INFO 21062 --- [ocp-schedules-19,ff8cbb958333c708,b037b94aa06ad33c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 12:57:28.505 INFO 21062 --- [ocp-schedules-17,c9ff5be9c15b66ab,0951f48834b93ab5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:57:28.507 INFO 21062 --- [ocp-schedules-19,ff8cbb958333c708,b037b94aa06ad33c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:28.511 INFO 21062 --- [ocp-schedules-9,2f15210f79d52c49,6cd4a402b042c168] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:57:28.511 INFO 21062 --- [ocp-schedules-18,52b95fbc8ac18d62,f7a9e0f75d698a2c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:57:28.511 INFO 21062 --- [ocp-schedules-18,52b95fbc8ac18d62,f7a9e0f75d698a2c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:57:28.511 INFO 21062 --- [ocp-schedules-9,2f15210f79d52c49,6cd4a402b042c168] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:57:28.511 INFO 21062 --- [ocp-schedules-27,414854b739dbb63e,087ab01e41b8dd86] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:57:28.512 INFO 21062 --- [ocp-schedules-27,414854b739dbb63e,087ab01e41b8dd86] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:57:28.529 INFO 21062 --- [ocp-schedules-13,f79d657379aa5d16,54d128ec2dbd3d51] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 12:57:28.529 INFO 21062 --- [ocp-schedules-13,f79d657379aa5d16,54d128ec2dbd3d51] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 12:57:28.559 INFO 21062 --- [ocp-async-10,95a5944e289b1952,8158853d6c72bf62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:57:28.574 INFO 21062 --- [pool-12-thread-6,75720920c5019d3a,11cd2fe73a98eb8c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 12:57:28.574 INFO 21062 --- [pool-12-thread-8,75720920c5019d3a,6848df84b1019833] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 12:57:28.574 INFO 21062 --- [ocp-schedules-30,3a685ee4046d1e52,3f7f2770bea0cd7a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:57:28.575 INFO 21062 --- [ocp-schedules-30,3a685ee4046d1e52,3f7f2770bea0cd7a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 12:57:28.575 INFO 21062 --- [pool-12-thread-5,75720920c5019d3a,d72a34ab54d7d24f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 12:57:28.576 INFO 21062 --- [pool-12-thread-7,75720920c5019d3a,75ac133669f66cab] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 12:57:28.580 INFO 21062 --- [ForkJoinPool-11-worker-15,75720920c5019d3a,9536f9b2451e2d3a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 12:57:28.580 INFO 21062 --- [ForkJoinPool-11-worker-15,75720920c5019d3a,9536f9b2451e2d3a] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 12:57:28.580 INFO 21062 --- [ForkJoinPool-11-worker-15,75720920c5019d3a,9536f9b2451e2d3a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 12:57:28.580 INFO 21062 --- [pool-12-thread-6,75720920c5019d3a,11cd2fe73a98eb8c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 12:57:28.581 INFO 21062 --- [pool-12-thread-7,75720920c5019d3a,75ac133669f66cab] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 12:57:28.582 INFO 21062 --- [pool-12-thread-7,75720920c5019d3a,75ac133669f66cab] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 12:57:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:57:28.587839+08:00, lastProcessTime=2024-11-04T12:57:18.588460+08:00 2024-11-04 12:57:28.602 INFO 21062 --- [pool-12-thread-8,75720920c5019d3a,6848df84b1019833] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 12:57:28.606 INFO 21062 --- [pool-12-thread-8,75720920c5019d3a,6848df84b1019833] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 12:57:28.608 INFO 21062 --- [ForkJoinPool-11-worker-15,75720920c5019d3a,8d143218db932e3d] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 12:57:28.608 INFO 21062 --- [pool-12-thread-5,75720920c5019d3a,d72a34ab54d7d24f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 12:57:28.610 INFO 21062 --- [pool-12-thread-8,75720920c5019d3a,6848df84b1019833] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 12:57:28.619 INFO 21062 --- [ocp-schedules-6,b25a183293f17907,00c00001b2c9e3c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:57:28.625 INFO 21062 --- [ocp-schedules-17,5df91336a2f25fd3,36925974d99e44bf] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 12:57:28.637 INFO 21062 --- [ocp-schedules-17,5df91336a2f25fd3,36925974d99e44bf] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 12:57:28.667 INFO 21062 --- [ocp-schedules-22,abc8f2a704dc2fa0,e498ad9347e33d32] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:57:28.667 INFO 21062 --- [ocp-schedules-22,abc8f2a704dc2fa0,e498ad9347e33d32] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:57:28.673 WARN 21062 --- [ocp-async-8,15ad373ba8210645,890d8008d72b039d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:28.679 INFO 21062 --- [ocp-schedules-18,1baf0007a2314b6c,5b5e3c6995ece003] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:57:28.705 INFO 21062 --- [ocp-async-2,db6fdb1988dbcc43,88cc8fc950297eff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:28.724 INFO 21062 --- [ocp-schedules-1,67d0c4fdb754762c,68b7cb6e1cd72b04] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:57:28.729 INFO 21062 --- [ocp-schedules-1,67d0c4fdb754762c,68b7cb6e1cd72b04] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:57:28.729 INFO 21062 --- [ocp-schedules-1,67d0c4fdb754762c,68b7cb6e1cd72b04] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:57:28.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:57:28.587839+08:00 2024-11-04 12:57:28.787 INFO 21062 --- [ocp-schedules-26,d856d2a61be56347,13a9727643fdf4c6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 12:57:28.787 INFO 21062 --- [ocp-schedules-26,d856d2a61be56347,13a9727643fdf4c6] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 12:57:28.813 INFO 21062 --- [pool-24-thread-1,d856d2a61be56347,619e8320d9aa4ebd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 12:57:28.843 INFO 21062 --- [ocp-schedules-16,ca0f193c4f3d4bbe,f12096a5ed80dc2f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:57:28.843 INFO 21062 --- [ocp-schedules-16,ca0f193c4f3d4bbe,f12096a5ed80dc2f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:57:28.843 WARN 21062 --- [ocp-schedules-26,d856d2a61be56347,13a9727643fdf4c6] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 12:57:28.843 INFO 21062 --- [ocp-schedules-26,d856d2a61be56347,13a9727643fdf4c6] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 12:57:28.849 INFO 21062 --- [ocp-schedules-16,ca0f193c4f3d4bbe,f12096a5ed80dc2f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:57:28.849 INFO 21062 --- [ocp-schedules-16,ca0f193c4f3d4bbe,f12096a5ed80dc2f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:57:28.854 INFO 21062 --- [ocp-schedules-16,ca0f193c4f3d4bbe,f12096a5ed80dc2f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:57:28.854 INFO 21062 --- [ocp-schedules-16,ca0f193c4f3d4bbe,f12096a5ed80dc2f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:57:28.931 INFO 21062 --- [ocp-schedules-12,4bd2cc811a4dcafe,22b5e3d1e44e3f6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:57:29.204 INFO 21062 --- [ocp-schedules-15,44ba04a98d7d9b28,3325bf22746708e6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:57:29.204 INFO 21062 --- [ocp-schedules-15,44ba04a98d7d9b28,3325bf22746708e6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:57:29.216 INFO 21062 --- [ocp-schedules-29,64901b36a8d2157b,139329bde1c9e618] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:57:29.222 INFO 21062 --- [ocp-schedules-29,64901b36a8d2157b,139329bde1c9e618] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:57:29.222 INFO 21062 --- [ocp-schedules-29,64901b36a8d2157b,139329bde1c9e618] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:57:29.231 INFO 21062 --- [ocp-schedules-15,44ba04a98d7d9b28,3325bf22746708e6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:57:29.231 INFO 21062 --- [ocp-schedules-15,44ba04a98d7d9b28,3325bf22746708e6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:57:29.236 INFO 21062 --- [ocp-schedules-8,bc9bd71448d51761,021f73e73c19aa82] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:57:29.277 INFO 21062 --- [ocp-schedules-4,0e8e362f29a1efd2,9dbfaa6c78941299] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:57:29.277 INFO 21062 --- [ocp-schedules-4,0e8e362f29a1efd2,9dbfaa6c78941299] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:57:29.277 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:57:29.277 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:57:29.287 INFO 21062 --- [ocp-schedules-4,0e8e362f29a1efd2,9dbfaa6c78941299] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:57:29.287 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:57:29.287 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:57:29.289 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:57:29.289 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:57:29.289 WARN 21062 --- [ocp-schedules-4,0e8e362f29a1efd2,9dbfaa6c78941299] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:57:29.289 INFO 21062 --- [ocp-schedules-4,0e8e362f29a1efd2,9dbfaa6c78941299] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:57:29.289 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:57:29.289 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:57:29.291 WARN 21062 --- [ocp-schedules-4,0e8e362f29a1efd2,9dbfaa6c78941299] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:57:29.291 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:57:29.291 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:57:29.297 ERROR 21062 --- [ocp-schedules-4,0e8e362f29a1efd2,9dbfaa6c78941299] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 12:57:29.352 INFO 21062 --- [ocp-schedules-6,818b323f723cf41b,6ea6d020125ca5e5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:57:29.352+08:00, clusters=[1] 2024-11-04 12:57:29.354 INFO 21062 --- [ocp-schedules-6,818b323f723cf41b,6ea6d020125ca5e5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:57:29.434 INFO 21062 --- [ocp-schedules-28,2f8d8ca38bbc9a44,75c69852748cd2b7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:57:29.434+08:00, clusters=[1] 2024-11-04 12:57:29.438 INFO 21062 --- [ocp-schedules-28,2f8d8ca38bbc9a44,75c69852748cd2b7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:57:29.920 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:29.921 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:29.983 INFO 21062 --- [prometheus-minute-schedule-1,5267425d4b388003,b20142e7162f772e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:30.222 INFO 21062 --- [ocp-schedules-10,c65bf7216dfcf4b0,f2eeab73472c715d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:57:30.222 INFO 21062 --- [ocp-schedules-10,8546bf66f452b03c,034e744cff01e890] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:57:30.226 INFO 21062 --- [ocp-schedules-10,8546bf66f452b03c,034e744cff01e890] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:30.255 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:57:30.255 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:57:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:57:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:57:30.673 WARN 21062 --- [ocp-async-4,c03db9a2cceedd8e,4c799a25481d631f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:30.705 INFO 21062 --- [ocp-async-3,7281f264c5bb6893,d47702c5cfd1c55e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:30.789 INFO 21062 --- [ocp-schedules-25,d06a7fa0bda662c9,e907b0b1a7bd608a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:57:30.793 INFO 21062 --- [ocp-schedules-25,d06a7fa0bda662c9,e907b0b1a7bd608a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:30.880 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:57:30.881 ERROR 21062 --- [ocp-schedules-25,d06a7fa0bda662c9,e907b0b1a7bd608a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:57:30.881 INFO 21062 --- [ocp-schedules-25,d06a7fa0bda662c9,e907b0b1a7bd608a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:57:32.673 WARN 21062 --- [ocp-async-3,8a1c7978a0156c18,cd62be6b87d9caab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:32.705 INFO 21062 --- [ocp-async-10,401a2e761d6acbf1,700d31e4737e7250] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:32.921 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:32.922 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:32.983 INFO 21062 --- [prometheus-minute-schedule-1,2dabdbda15867d9b,035370b6754d8a44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:34.673 WARN 21062 --- [ocp-async-10,c8c51d67732c5061,15594190b249f29a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:34.694 INFO 21062 --- [ocp-schedules-5,3a897037fe9efa38,350e511c4497cd18] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:57:34.706 INFO 21062 --- [ocp-async-9,f57b9cdc8dee8c63,62f7abc1de052125] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:35.922 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:35.924 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:35.983 INFO 21062 --- [prometheus-minute-schedule-1,2357a3034a3318b4,de27e0f152fa47ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:36.674 WARN 21062 --- [ocp-async-9,56296745d0a58a6d,585529779ff0a153] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:36.706 INFO 21062 --- [ocp-async-7,08c50c4a588d827e,3c0f01955277a9f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:38.504 INFO 21062 --- [ocp-schedules-11,9c73a6b099b620e0,6a93781663974b40] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:57:38.505 INFO 21062 --- [ocp-schedules-2,7a78b3e11eec2849,e1f7bc49b2110aba] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:57:38.508 INFO 21062 --- [ocp-schedules-19,ff8cbb958333c708,b037b94aa06ad33c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 12:57:38.560 INFO 21062 --- [ocp-async-7,463c45faf2ad8a8b,121fb98163c5c5d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:57:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:57:38.588875+08:00, lastProcessTime=2024-11-04T12:57:28.587839+08:00 2024-11-04 12:57:38.674 WARN 21062 --- [ocp-async-1,31435f9c9fd44b6d,1c358876654b2235] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:38.706 INFO 21062 --- [ocp-async-5,e66a85de0d914153,d80687dc3d037792] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:57:38.588875+08:00 2024-11-04 12:57:38.924 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:38.925 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:38.934 INFO 21062 --- [ocp-schedules-15,8d3664bef0a99582,b16f02ae3c16c077] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:57:38.983 INFO 21062 --- [prometheus-minute-schedule-1,2776ccb7bf793366,1421ac69dfcaa572] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:40.002 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:57:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:57:40.674 WARN 21062 --- [ocp-async-5,2dbfb13d9c798e33,9349d9efa38b9769] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:40.706 INFO 21062 --- [ocp-async-9,e9e9851dff5df198,2f403180ce2572a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:41.925 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:41.926 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c64814fe9900b643,2049924f0e2d4708] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:42.674 WARN 21062 --- [ocp-async-9,07bb0a4f55e21e97,cb4abb2a5ad4c1fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:42.706 INFO 21062 --- [ocp-async-7,9fb6b9cca793df31,f76a2c4a5d90dcb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:44.674 WARN 21062 --- [ocp-async-7,f0e1c557942a5078,d2d7c624ab53a5c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:44.706 INFO 21062 --- [ocp-async-6,9db9454a415c870d,567e514de29eea9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:44.737 INFO 21062 --- [ocp-schedules-9,bb1f7bd8290101a3,059d4a9ab3b12b97] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:57:44.926 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:44.927 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:44.983 INFO 21062 --- [prometheus-minute-schedule-1,695aa1b0bf4445f1,b76469ec88eab96c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:46.675 WARN 21062 --- [ocp-async-6,a84304471607fa93,e7008260e97c1e52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:46.706 INFO 21062 --- [ocp-async-8,2347fc73742ed889,4629ff84f460323c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:47.927 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:47.929 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2c8524e1ea216fd8,60a8ddd02548af61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:48.560 INFO 21062 --- [ocp-async-2,59aaefc32dc988c9,156241286cdd4d10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:57:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:57:48.588476+08:00, lastProcessTime=2024-11-04T12:57:38.588875+08:00 2024-11-04 12:57:48.675 WARN 21062 --- [ocp-async-9,d8de36ca2bc58bb5,e168e4fe55d73800] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:48.707 INFO 21062 --- [ocp-async-3,0346e2c950ac7746,55213d338c1c8db8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:57:48.588476+08:00 2024-11-04 12:57:48.937 INFO 21062 --- [ocp-schedules-16,bc2c869763aba2ea,68f94e8a8a297e56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:57:49.225 INFO 21062 --- [ocp-schedules-5,74e1d75f22a83ca0,0d6418543a429f02] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:57:49.232 INFO 21062 --- [ocp-schedules-5,74e1d75f22a83ca0,0d6418543a429f02] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:57:49.232 INFO 21062 --- [ocp-schedules-5,74e1d75f22a83ca0,0d6418543a429f02] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:57:50.675 WARN 21062 --- [ocp-async-3,b6ea043364d8f6bf,ea932f04ee4c5650] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:50.707 INFO 21062 --- [ocp-async-1,475d3fdf53a5161d,82c84dd2a3815517] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:50.929 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:50.930 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:50.983 INFO 21062 --- [prometheus-minute-schedule-1,94aa560d4bb52611,dde91e97d9a6dcf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:52.675 WARN 21062 --- [ocp-async-1,54a2d52a7e5d2513,8b0f30868247a4f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:52.707 INFO 21062 --- [ocp-async-5,cf63c32294a20833,66141c5cfebb341b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:53.930 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:53.932 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6f922ad389e6a307,7cd677445d28d185] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:54.675 WARN 21062 --- [ocp-async-5,718fb4d82ab3dbb7,bbf74c989a55fb8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:54.707 INFO 21062 --- [ocp-async-9,c216885e3470921e,6578c136c36d7cf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:54.764 INFO 21062 --- [ocp-schedules-27,8e5f44777d19361f,fb2b4a840ab0dafa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:57:56.675 WARN 21062 --- [ocp-async-9,62eb26c6f06f331a,cc8119760c8e9fbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:56.707 INFO 21062 --- [ocp-async-3,96886637c5b5f418,f304208361c9ad9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:56.932 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:56.933 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e6ed36578b20c27a,c4f060f493b852db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:57:58.500 INFO 21062 --- [ocp-schedules-2,b828b1fb2567f3d4,96ab1f38a18387de] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:57:58.500 INFO 21062 --- [ocp-schedules-25,b5434a90ea210e59,d41664785759da3a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:57:58.502 INFO 21062 --- [ocp-schedules-9,be966dd19f6db211,76cf437b5fdf9a90] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:57:58.504 INFO 21062 --- [ocp-schedules-2,b828b1fb2567f3d4,96ab1f38a18387de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:58.505 INFO 21062 --- [ocp-schedules-25,b5434a90ea210e59,d41664785759da3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:57:58.507 INFO 21062 --- [ocp-schedules-23,84fb2f2c64de01c4,7d4b9c4d973482ec] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:57:58.512 INFO 21062 --- [ocp-schedules-27,688b2b5c4128647f,f8e64fd05c275ccb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:57:58.513 INFO 21062 --- [ocp-schedules-26,06f87df46efc52df,9126feb9aa4491bd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:57:58.513 INFO 21062 --- [ocp-schedules-27,688b2b5c4128647f,f8e64fd05c275ccb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:57:58.513 INFO 21062 --- [ocp-schedules-26,06f87df46efc52df,9126feb9aa4491bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:57:58.513 INFO 21062 --- [ocp-schedules-18,7da55f7c29a6f56e,baf6a479481f64e2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:57:58.513 INFO 21062 --- [ocp-schedules-18,7da55f7c29a6f56e,baf6a479481f64e2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:57:58.560 INFO 21062 --- [ocp-async-6,5e4151d7dc988b04,c485c9151ba61b9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:57:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:57:58.588474+08:00, lastProcessTime=2024-11-04T12:57:48.588476+08:00 2024-11-04 12:57:58.648 INFO 21062 --- [ocp-schedules-11,535e985cd386efdc,18eb6ba196689f77] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:57:58.652 INFO 21062 --- [ocp-schedules-11,535e985cd386efdc,18eb6ba196689f77] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:57:58.652 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:57:58.675 WARN 21062 --- [ocp-async-1,7cb654c6c0e321aa,03376c4fb680f54b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:57:58.681 INFO 21062 --- [ocp-schedules-15,8ae0fbdf947803e5,e3e8f3d4e3a7f307] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:57:58.708 INFO 21062 --- [ocp-async-8,b1c6a880cdbb8d7a,053786c3f4752143] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:57:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:57:58.588474+08:00 2024-11-04 12:57:58.941 INFO 21062 --- [ocp-schedules-5,24f370b5961fd10c,0020478e9832ae90] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:57:59.933 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:57:59.935 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:57:59.983 INFO 21062 --- [prometheus-minute-schedule-1,3d3f5d12bd89a489,63a4d25ada764c0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696280, interval=60 2024-11-04 12:58:00.256 INFO 21062 --- [ocp-schedules-10,8546bf66f452b03c,034e744cff01e890] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:58:00.256 INFO 21062 --- [ocp-schedules-10,83925f6586a37ce8,48cc1cf269fb68b2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:58:00.260 INFO 21062 --- [ocp-schedules-10,83925f6586a37ce8,48cc1cf269fb68b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:00.290 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:58:00.290 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:58:00.676 WARN 21062 --- [ocp-async-5,0453d265f8eab8be,dac78e2a162d825e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:00.708 INFO 21062 --- [ocp-async-7,3e502d9ca8514437,de79a38fa3bf5fbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:02.676 WARN 21062 --- [ocp-async-9,b1a1405804b79483,f02276ff79f8c48c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:02.708 INFO 21062 --- [ocp-async-10,d2e82c888bcbf593,70d6035d4a9767f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:02.935 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:02.937 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:02.983 INFO 21062 --- [prometheus-minute-schedule-1,646fcd997f4424d1,fcef694b437c279b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:04.676 WARN 21062 --- [ocp-async-3,ff224d4eb090745d,dab416bdb85900d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:04.708 INFO 21062 --- [ocp-async-1,42ab7b1d147aaa24,b12b94d16689f951] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:04.805 INFO 21062 --- [ocp-schedules-24,7df2943964cdb8f7,1326595480075957] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:58:05.937 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:05.939 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:05.983 INFO 21062 --- [prometheus-minute-schedule-1,66400a806b6deb1d,d3ed740192e5472d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:06.676 WARN 21062 --- [ocp-async-8,ebd2cfc8f8259538,33078935aa14feb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:06.708 INFO 21062 --- [ocp-async-5,bc98e59ad508211e,0309386db07c46d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:08.506 INFO 21062 --- [ocp-schedules-2,b828b1fb2567f3d4,96ab1f38a18387de] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:58:08.506 INFO 21062 --- [ocp-schedules-25,b5434a90ea210e59,d41664785759da3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:58:08.560 INFO 21062 --- [ocp-async-5,774e1fdba667a588,ba71d047371f1e12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:58:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:58:08.589297+08:00, lastProcessTime=2024-11-04T12:57:58.588474+08:00 2024-11-04 12:58:08.652 INFO 21062 --- [ocp-schedules-11,535e985cd386efdc,18eb6ba196689f77] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:58:08.676 WARN 21062 --- [ocp-async-9,9ff42ba1648b4cb0,0dd62a2c24da1da4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:08.709 INFO 21062 --- [ocp-async-6,3c43494ab43a0453,edb5d058eacb763d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:58:08.589297+08:00 2024-11-04 12:58:08.939 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:08.940 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:08.944 INFO 21062 --- [ocp-schedules-5,3232130731ce8436,aad78cfce1516163] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:58:08.983 INFO 21062 --- [prometheus-minute-schedule-1,feac2f14212ff700,dba4681a584f6fdf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:09.234 INFO 21062 --- [ocp-schedules-27,9e61e72358df7378,1c3fd554e2e7f51b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:58:09.242 INFO 21062 --- [ocp-schedules-27,9e61e72358df7378,1c3fd554e2e7f51b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:58:09.242 INFO 21062 --- [ocp-schedules-27,9e61e72358df7378,1c3fd554e2e7f51b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:58:10.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:58:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:58:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:58:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:58:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:58:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:58:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:58:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:58:10.676 WARN 21062 --- [ocp-async-3,93aaf19571ca136c,b3ae4f2712f8e421] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:10.709 INFO 21062 --- [ocp-async-2,76d4b3cccd4490be,b9c52c139e21443b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:11.135 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=24, affectRows=24 2024-11-04 12:58:11.135 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=32, affectRows=32 2024-11-04 12:58:11.135 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=24, validCount=24 2024-11-04 12:58:11.135 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=53, affectRows=53 2024-11-04 12:58:11.135 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=41, affectRows=41 2024-11-04 12:58:11.135 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=32, validCount=32 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=52, affectRows=52 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=31, affectRows=31 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=70, affectRows=70 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696220, nodeCount=19, affectRows=19 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=41, validCount=41 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=53, validCount=53 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=19, validCount=19 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=31, validCount=31 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=50, validCount=50 2024-11-04 12:58:11.136 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696220, nodeCount=69, validCount=69 2024-11-04 12:58:11.940 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:11.941 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:11.983 INFO 21062 --- [prometheus-minute-schedule-1,83152dc745cb7f95,669479cbbb1d1653] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:12.676 WARN 21062 --- [ocp-async-8,a0dcf1d497086b72,e485a07eaeefc10f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:12.709 INFO 21062 --- [ocp-async-4,40a975917feebf65,2edad492e24bed3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:13.932 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T12:52:13.904, end=2024-11-04T12:57:13.929} 2024-11-04 12:58:13.932 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T12:52:13.904, end=2024-11-04T12:57:13.929} 2024-11-04 12:58:13.936 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 12:58:13.936 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 12:58:13.936 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:58:13.936 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 12:58:13.984 INFO 21062 --- [prometheus-cache-monitor1,04c3a0fbe680a171,6201f2df2be9b337] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:58:13.984 INFO 21062 --- [prometheus-cache-monitor1,04c3a0fbe680a171,6201f2df2be9b337] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:58:14.677 WARN 21062 --- [ocp-async-7,5e5659f2462f6328,6b457fa366ef667e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:14.709 INFO 21062 --- [ocp-async-9,765e2b196c0c8625,be952fb2e559c34e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:14.832 INFO 21062 --- [ocp-schedules-22,0ec04106829c1725,6f4d2a0afe9c137d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:58:14.941 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:14.942 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f82ee35e580639f6,b8309b134385ffb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:16.677 WARN 21062 --- [ocp-async-6,2a286e606140e167,6643f950fc8933fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:16.709 INFO 21062 --- [ocp-async-3,0721841ad93c4670,f2211cf29705b800] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:17.942 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:17.943 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:17.983 INFO 21062 --- [prometheus-minute-schedule-1,bd601a075fcaee38,b16205340fe13e3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:18.560 INFO 21062 --- [ocp-async-3,b158c5869e9821d8,46daa94a120d39b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:58:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:58:18.588023+08:00, lastProcessTime=2024-11-04T12:58:08.589297+08:00 2024-11-04 12:58:18.677 WARN 21062 --- [ocp-async-8,5ec88cd014db6792,9b14da7560f96316] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:18.709 INFO 21062 --- [ocp-async-5,befecd5ead0f6f1d,4c8a54fdf7d74e8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:58:18.588023+08:00 2024-11-04 12:58:18.946 INFO 21062 --- [ocp-schedules-29,4bb8daabf169269c,229960c84e3f079d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:58:20.677 WARN 21062 --- [ocp-async-7,4dcd0996c09914a4,48eb0201be2b9527] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:20.710 INFO 21062 --- [ocp-async-10,6f3be40936da78c8,ffe297543aabe784] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:20.943 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:20.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:20.983 INFO 21062 --- [prometheus-minute-schedule-1,1bf50bf02b43778b,0c8acc3309cda35c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:22.677 WARN 21062 --- [ocp-async-6,f033c194cd29fe15,5a5e9fd64c775545] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:22.710 INFO 21062 --- [ocp-async-1,7161613123540508,f682f9d502b831a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:23.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:23.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:23.983 INFO 21062 --- [prometheus-minute-schedule-1,00484488c1d3a053,3ae97f34f9e328ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:24.678 WARN 21062 --- [ocp-async-2,b98b44705ef4cd36,272b4edce12761c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:24.710 INFO 21062 --- [ocp-async-8,a710660af7bddb75,de33cb1c06c06c58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:24.874 INFO 21062 --- [ocp-schedules-7,dfdb40130f095458,6dd365a3e520372b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:58:26.678 WARN 21062 --- [ocp-async-5,b79d8049920d3a9b,b7bf1b005f50a880] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:26.710 INFO 21062 --- [ocp-async-7,c4e22cf82bb76b04,16021420e7c590cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:26.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:26.946 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3a3d2fe78b7768cd,9822299e42dd3c43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:26.984 INFO 21062 --- [prometheus-hour-schedule-1,1ae9a7addf09db54,0caa8bd96b29930f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:58:28.498 INFO 21062 --- [ocp-schedules-16,508f212f7ca7f309,c01ab64def7154ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:58:28.499 INFO 21062 --- [ocp-schedules-1,a9749ea23348362d,2e85fa80a442771e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:58:28.500 INFO 21062 --- [ocp-schedules-12,135693bc1ee9edae,d3af407876d89314] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:58:28.500 INFO 21062 --- [ocp-schedules-11,b7f85d13c2593eeb,62246103c4ecfe7c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:58:28.502 INFO 21062 --- [ocp-schedules-16,508f212f7ca7f309,c01ab64def7154ae] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:58:28.502 INFO 21062 --- [ocp-schedules-22,468503e835153240,e54670a0ab9d5054] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:58:28.503 INFO 21062 --- [ocp-schedules-12,135693bc1ee9edae,d3af407876d89314] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:28.504 INFO 21062 --- [ocp-schedules-11,b7f85d13c2593eeb,62246103c4ecfe7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:58:28.504 INFO 21062 --- [ocp-schedules-23,32c1e4fd612691d9,173923a7d1f42c8b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:58:28.504 INFO 21062 --- [ocp-schedules-1,a9749ea23348362d,2e85fa80a442771e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:58:28.511 INFO 21062 --- [ocp-schedules-30,26f6ec5adfe3e322,19263f98e1e94e00] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:58:28.512 INFO 21062 --- [ocp-schedules-30,26f6ec5adfe3e322,19263f98e1e94e00] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:58:28.512 INFO 21062 --- [ocp-schedules-14,706c78c0d1fd176d,cc714f354e39166c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:58:28.512 INFO 21062 --- [ocp-schedules-14,706c78c0d1fd176d,cc714f354e39166c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:58:28.513 INFO 21062 --- [ocp-schedules-9,00f1115793d1af31,0352ca555c017f2c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:58:28.513 INFO 21062 --- [ocp-schedules-9,00f1115793d1af31,0352ca555c017f2c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:58:28.557 INFO 21062 --- [ocp-schedules-17,4148315547eb9685,dc44aacbb81a3276] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:58:28.557 INFO 21062 --- [ocp-schedules-17,4148315547eb9685,dc44aacbb81a3276] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:58:28.560 INFO 21062 --- [ocp-async-10,658018b86cf85712,05a887c5bce5ce1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:58:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:58:28.588025+08:00, lastProcessTime=2024-11-04T12:58:18.588023+08:00 2024-11-04 12:58:28.621 INFO 21062 --- [ocp-schedules-7,aa24f763d81120d8,4515bb93cda1000f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:58:28.670 INFO 21062 --- [ocp-schedules-6,593653bb2f51f939,e9e8bddfd86dc642] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:58:28.671 INFO 21062 --- [ocp-schedules-6,593653bb2f51f939,e9e8bddfd86dc642] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:58:28.678 WARN 21062 --- [ocp-async-1,938329c86cbe2cae,8feb50778460f70e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:28.683 INFO 21062 --- [ocp-schedules-20,f823cfb34abce3fa,c6ce5fdd1180d058] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:58:28.696 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 12:58:28.696 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 12:58:28.704 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 12:58:28.704 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 12:58:28.704 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 12:58:28.710 INFO 21062 --- [ocp-async-2,49aa3f9b9d0606a9,1828312e81771dfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:28.713 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 12:58:28.713 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 12:58:28.713 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 12:58:28.717 INFO 21062 --- [ocp-schedules-26,c1725468603251d5,3d6d8315a142c25e] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 12:58:28.729 INFO 21062 --- [ocp-schedules-30,2b4c0b0ea8534d35,7d89ef853523856d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:58:28.734 INFO 21062 --- [ocp-schedules-30,2b4c0b0ea8534d35,7d89ef853523856d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:58:28.734 INFO 21062 --- [ocp-schedules-30,2b4c0b0ea8534d35,7d89ef853523856d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:58:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:58:28.588025+08:00 2024-11-04 12:58:28.859 INFO 21062 --- [ocp-schedules-9,3b5f44efe9cf9960,1ea059f38786785b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:58:28.859 INFO 21062 --- [ocp-schedules-9,3b5f44efe9cf9960,1ea059f38786785b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:58:28.860 INFO 21062 --- [ocp-schedules-9,3b5f44efe9cf9960,1ea059f38786785b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:58:28.860 INFO 21062 --- [ocp-schedules-9,3b5f44efe9cf9960,1ea059f38786785b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:58:28.861 INFO 21062 --- [ocp-schedules-9,3b5f44efe9cf9960,1ea059f38786785b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:58:28.861 INFO 21062 --- [ocp-schedules-9,3b5f44efe9cf9960,1ea059f38786785b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:58:28.949 INFO 21062 --- [ocp-schedules-2,1f401e6b90aa334d,b69448983a3ac7d8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:58:29.232 INFO 21062 --- [ocp-schedules-24,d53abcd9e4fd40ce,72138ea396129210] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:58:29.232 INFO 21062 --- [ocp-schedules-24,d53abcd9e4fd40ce,72138ea396129210] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:58:29.244 INFO 21062 --- [ocp-schedules-27,c251fff2d1660107,cd3e15722c93935f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:58:29.250 INFO 21062 --- [ocp-schedules-27,c251fff2d1660107,cd3e15722c93935f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:58:29.250 INFO 21062 --- [ocp-schedules-27,c251fff2d1660107,cd3e15722c93935f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:58:29.262 INFO 21062 --- [ocp-schedules-24,d53abcd9e4fd40ce,72138ea396129210] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:58:29.262 INFO 21062 --- [ocp-schedules-24,d53abcd9e4fd40ce,72138ea396129210] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:58:29.262 INFO 21062 --- [ocp-schedules-28,73515bda518364a6,90eac1f804fdcbe7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:58:29.283 INFO 21062 --- [ocp-schedules-15,197410f407919feb,c09a2c54cbe9c995] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 12:58:29.286 INFO 21062 --- [ocp-schedules-15,197410f407919feb,c09a2c54cbe9c995] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:29.305 INFO 21062 --- [ocp-schedules-29,c9fe26ab39a3e389,23e71682f569019e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:58:29.305 INFO 21062 --- [ocp-schedules-29,c9fe26ab39a3e389,23e71682f569019e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:58:29.305 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:58:29.305 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:58:29.314 INFO 21062 --- [ocp-schedules-29,c9fe26ab39a3e389,23e71682f569019e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:58:29.314 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:58:29.315 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:58:29.316 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:58:29.316 WARN 21062 --- [ocp-schedules-29,c9fe26ab39a3e389,23e71682f569019e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:58:29.316 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:58:29.317 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:58:29.317 INFO 21062 --- [ocp-schedules-29,c9fe26ab39a3e389,23e71682f569019e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:58:29.317 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:58:29.318 WARN 21062 --- [ocp-schedules-29,c9fe26ab39a3e389,23e71682f569019e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:58:29.318 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:58:29.318 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:58:29.326 ERROR 21062 --- [ocp-schedules-29,c9fe26ab39a3e389,23e71682f569019e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:58:29.351 ERROR 21062 --- [ocp-schedules-15,197410f407919feb,c09a2c54cbe9c995] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:58:29.351 INFO 21062 --- [ocp-schedules-15,197410f407919feb,c09a2c54cbe9c995] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 12:58:29.380 INFO 21062 --- [ocp-schedules-19,c95ce68a306396d6,05dab15c610900e4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:58:29.380+08:00, clusters=[1] 2024-11-04 12:58:29.382 INFO 21062 --- [ocp-schedules-19,c95ce68a306396d6,05dab15c610900e4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:58:29.464 INFO 21062 --- [ocp-schedules-4,bf4e315b85d8ce27,51b74e18bb64353b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:58:29.464+08:00, clusters=[1] 2024-11-04 12:58:29.468 INFO 21062 --- [ocp-schedules-4,bf4e315b85d8ce27,51b74e18bb64353b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:58:29.946 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:29.947 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:29.983 INFO 21062 --- [prometheus-minute-schedule-1,afe0b96cbb3829af,8f9b9466a08dac72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:58:30.291 INFO 21062 --- [ocp-schedules-10,83925f6586a37ce8,48cc1cf269fb68b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:58:30.291 INFO 21062 --- [ocp-schedules-10,28ce4ff4c77d131a,7c88de74402f7aff] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:58:30.294 INFO 21062 --- [ocp-schedules-10,28ce4ff4c77d131a,7c88de74402f7aff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:58:30.322 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:58:30.322 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:58:30.678 WARN 21062 --- [ocp-async-2,79ae9078e7e8ad52,327717eadd3e456b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:30.711 INFO 21062 --- [ocp-async-5,f422174cea816b35,868fc99518905171] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:30.881 INFO 21062 --- [ocp-schedules-14,9e6d8e1d6d766736,93f829c252a4c03e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:58:30.885 INFO 21062 --- [ocp-schedules-14,9e6d8e1d6d766736,93f829c252a4c03e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:30.971 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:58:30.972 ERROR 21062 --- [ocp-schedules-14,9e6d8e1d6d766736,93f829c252a4c03e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:58:30.972 INFO 21062 --- [ocp-schedules-14,9e6d8e1d6d766736,93f829c252a4c03e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:58:32.678 WARN 21062 --- [ocp-async-5,8cde798837a5396f,48998f16e542e0c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:32.711 INFO 21062 --- [ocp-async-10,1d663c135bc57252,8330f966d2bf5eab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:32.947 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:32.949 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a51ecddc4b25c653,726772764af82f28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:34.679 WARN 21062 --- [ocp-async-10,643a25a88fac5714,0faaf9b581a11c69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:34.711 INFO 21062 --- [ocp-async-1,38aecf8008ae221d,ca19f9f4a268b0c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:34.900 INFO 21062 --- [ocp-schedules-6,9df05080446f2335,1d2f8746ba660d33] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:58:35.949 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:35.950 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4652ea84520f7e28,c0b516e49da33470] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:36.679 WARN 21062 --- [ocp-async-1,920f4e3d3df04313,a55e55d76ec53734] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:36.711 INFO 21062 --- [ocp-async-8,ce9c87e484232bae,0cf0dbff34b21fae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:38.505 INFO 21062 --- [ocp-schedules-12,135693bc1ee9edae,d3af407876d89314] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:58:38.505 INFO 21062 --- [ocp-schedules-11,b7f85d13c2593eeb,62246103c4ecfe7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:58:38.561 INFO 21062 --- [ocp-async-8,a1f4785902b5b706,5e6ff19961447b6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:58:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:58:38.588886+08:00, lastProcessTime=2024-11-04T12:58:28.588025+08:00 2024-11-04 12:58:38.679 WARN 21062 --- [ocp-async-9,90429637fd4a8187,7dd7e1b34f247d36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:38.711 INFO 21062 --- [ocp-async-10,20ced798ee33cecc,454da33fed76bfb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:58:38.588886+08:00 2024-11-04 12:58:38.951 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:38.951 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:38.952 INFO 21062 --- [ocp-schedules-12,385d81fe4e1dd3d3,191910e641108a51] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:58:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5d3cb677ec0cf902,ab7b49265846d854] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:58:40.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:58:40.679 WARN 21062 --- [ocp-async-10,2aa5f04ce8b3ab0f,98bb9e993c74fbbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:40.711 INFO 21062 --- [ocp-async-1,d3c961b5b194f10d,973cf26b2d816b87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:58:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:58:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 12:58:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 12:58:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 12:58:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 12:58:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 12:58:41.952 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:41.953 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:41.983 INFO 21062 --- [prometheus-minute-schedule-1,dafd728c39083d99,79c44fb7d11d4d6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:42.679 WARN 21062 --- [ocp-async-1,c94ef48c6a289585,0f9dee52ec2a26a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:42.712 INFO 21062 --- [ocp-async-8,fa19b890020cc192,a1ceb302ab7cdab5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:44.679 WARN 21062 --- [ocp-async-8,9f5bb666817abfff,e034a31877562df9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:44.712 INFO 21062 --- [ocp-async-6,b4258e8c6a308785,03512db5142bf382] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:44.942 INFO 21062 --- [ocp-schedules-9,8d884142db4b9052,a3502237d41e9f9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:58:44.953 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:44.954 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9f37078afc21e08a,52cc2635db6034c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:46.679 WARN 21062 --- [ocp-async-6,06ed4864e00c5a3b,04929312ddfc4538] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:46.712 INFO 21062 --- [ocp-async-4,5dbe91673bcfe6ec,12b33561f67267fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:47.954 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:47.955 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:47.983 INFO 21062 --- [prometheus-minute-schedule-1,0db46dc0939f3c17,42bb589bbf58b2c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:48.561 INFO 21062 --- [ocp-async-2,6497646706f42a63,f9696c65997c7922] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:58:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:58:48.589082+08:00, lastProcessTime=2024-11-04T12:58:38.588886+08:00 2024-11-04 12:58:48.680 WARN 21062 --- [ocp-async-1,0c32bcd9a5e15235,aacddc4f028546e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:48.712 INFO 21062 --- [ocp-async-5,30a675ac3585d1eb,4ea400b6e26552d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:48.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:58:48.589082+08:00 2024-11-04 12:58:48.954 INFO 21062 --- [ocp-schedules-16,5ebc10f346238336,ae865eb2bf5a89d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:58:49.253 INFO 21062 --- [ocp-schedules-17,8698a39abf606582,b593c8d506e5e379] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:58:49.259 INFO 21062 --- [ocp-schedules-17,8698a39abf606582,b593c8d506e5e379] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:58:49.260 INFO 21062 --- [ocp-schedules-17,8698a39abf606582,b593c8d506e5e379] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:58:50.680 WARN 21062 --- [ocp-async-5,e4ab914168c3d8de,fb4ae93913878f8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:50.712 INFO 21062 --- [ocp-async-9,7356b10405881a33,0cbc8b72bff41dbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:50.955 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:50.956 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a6ffbe59ee4ac69c,a76e1df3f37665f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:52.680 WARN 21062 --- [ocp-async-9,0c1d9bc62cc9f992,f0fec879d3492bbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:52.712 INFO 21062 --- [ocp-async-10,8bac88be772198f8,32563fae39490458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:53.956 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:53.957 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9c5dd3bad48d6e10,f8ec4ce440702042] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:54.680 WARN 21062 --- [ocp-async-10,1fe79cd6157bd50f,df5d085ef38fcf7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:54.713 INFO 21062 --- [ocp-async-1,9c699033ed0aec80,b6087a295dac16af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:54.967 INFO 21062 --- [ocp-schedules-2,60286640deeb0240,432ffd81413031ef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:58:56.680 WARN 21062 --- [ocp-async-1,3e647cc6dd9c2b31,059acae57e91365a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:56.713 INFO 21062 --- [ocp-async-5,0e426470204134ff,54f79f68bde07e35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:56.957 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:56.958 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:56.983 INFO 21062 --- [prometheus-minute-schedule-1,637eaa899cb8e545,4c36d8df411fdccd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:58:58.500 INFO 21062 --- [ocp-schedules-30,3598c3b3bf0def57,4eda200d7e5fc704] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:58:58.500 INFO 21062 --- [ocp-schedules-18,6f0d5268ba2800d1,1da094caf573807a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:58:58.502 INFO 21062 --- [ocp-schedules-3,aec43142ef92bccb,adef8e5394357034] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:58:58.504 INFO 21062 --- [ocp-schedules-18,6f0d5268ba2800d1,1da094caf573807a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:58.504 INFO 21062 --- [ocp-schedules-30,3598c3b3bf0def57,4eda200d7e5fc704] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:58:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:58:58.505 INFO 21062 --- [ocp-schedules-13,fa3a446fc4d47d12,12e63d828b4fa61b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:58:58.509 INFO 21062 --- [ocp-schedules-20,928b2e7d0d2aafba,77e08c52f2ba42d8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:58:58.509 INFO 21062 --- [ocp-schedules-20,928b2e7d0d2aafba,77e08c52f2ba42d8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:58:58.510 INFO 21062 --- [ocp-schedules-28,3e28d3543067671f,1a9abc42fe819d04] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:58:58.510 INFO 21062 --- [ocp-schedules-28,3e28d3543067671f,1a9abc42fe819d04] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:58:58.510 INFO 21062 --- [ocp-schedules-2,335df2c1775a2615,744f092e9a3ff2cc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:58:58.511 INFO 21062 --- [ocp-schedules-2,335df2c1775a2615,744f092e9a3ff2cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:58:58.561 INFO 21062 --- [ocp-async-3,091c40009bd383ea,11c7d8853d5e6cda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:58:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:58:58.587624+08:00, lastProcessTime=2024-11-04T12:58:48.589082+08:00 2024-11-04 12:58:58.680 WARN 21062 --- [ocp-async-9,141d496f43362f85,7e58df4798f367a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:58:58.686 INFO 21062 --- [ocp-schedules-19,6dbf8088684b1a2e,2644caddcef8b3e8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:58:58.713 INFO 21062 --- [ocp-async-4,27ed8923f96d414c,c803765ba9050e3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:58:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:58:58.587624+08:00 2024-11-04 12:58:58.957 INFO 21062 --- [ocp-schedules-22,c6604c51ef697a69,38b6c065b7aeb04d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:58:59.958 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:58:59.959 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:58:59.983 INFO 21062 --- [prometheus-minute-schedule-1,c6fd85762861de17,4b9a40a46e6e8149] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696340, interval=60 2024-11-04 12:59:00.323 INFO 21062 --- [ocp-schedules-10,28ce4ff4c77d131a,7c88de74402f7aff] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:59:00.323 INFO 21062 --- [ocp-schedules-10,7b222e7337675ff5,07694e61a9a6da01] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:59:00.327 INFO 21062 --- [ocp-schedules-10,7b222e7337675ff5,07694e61a9a6da01] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:00.357 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:59:00.357 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:59:00.681 WARN 21062 --- [ocp-async-10,13c308fb0a681b3d,a9743a4db1f7e6ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:00.713 INFO 21062 --- [ocp-async-7,9e46a795516699db,9593d5daf358b3a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:02.681 WARN 21062 --- [ocp-async-1,63a360aacd5081b2,a59f01f337591c68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:02.713 INFO 21062 --- [ocp-async-6,7e154486c2a1be58,1bc628506209d8f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:02.959 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:02.960 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:02.983 INFO 21062 --- [prometheus-minute-schedule-1,6d77b173caa13dd0,7317fa6e4ee94fec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:04.681 WARN 21062 --- [ocp-async-5,3a3c4a3c43e252f6,441710b00fda4dc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:04.713 INFO 21062 --- [ocp-async-9,abd3133da3257f4a,d96df6838255bbe2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:05.009 INFO 21062 --- [ocp-schedules-12,56680bcf66e4f4c8,4773348c476ced63] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:59:05.960 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:05.961 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:05.983 INFO 21062 --- [prometheus-minute-schedule-1,672014385000fbf9,581e0c12b97b8a08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:06.681 WARN 21062 --- [ocp-async-4,757f86de64657d0e,7e75e837c52de75c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:06.714 INFO 21062 --- [ocp-async-10,99c2ecffb606ca49,43f250967ace7e19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:08.505 INFO 21062 --- [ocp-schedules-18,6f0d5268ba2800d1,1da094caf573807a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:59:08.505 INFO 21062 --- [ocp-schedules-30,3598c3b3bf0def57,4eda200d7e5fc704] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:59:08.561 INFO 21062 --- [ocp-async-10,228dbb3273f37415,949231f6af350c23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:59:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:59:08.588817+08:00, lastProcessTime=2024-11-04T12:58:58.587624+08:00 2024-11-04 12:59:08.653 INFO 21062 --- [ocp-schedules-3,8da81a3e2a4192ac,eba623969fd964fd] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 12:59:08.656 INFO 21062 --- [ocp-schedules-3,8da81a3e2a4192ac,eba623969fd964fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:08.656 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 12:59:08.681 WARN 21062 --- [ocp-async-1,5392f0e9e255c5ac,26d0d10f92f89ba1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:08.714 INFO 21062 --- [ocp-async-3,50cdea75c1e9945d,d9f93494e011d9cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:59:08.588817+08:00 2024-11-04 12:59:08.961 INFO 21062 --- [ocp-schedules-14,1eaa59ac043f6df9,f807d01e485add8a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:59:08.962 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:08.962 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:08.983 INFO 21062 --- [prometheus-minute-schedule-1,c01c9edcdad5c909,0f291fe825f9041c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:09.262 INFO 21062 --- [ocp-schedules-16,52dc62c4d80b16f4,998410baa5079d62] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:59:09.267 INFO 21062 --- [ocp-schedules-16,52dc62c4d80b16f4,998410baa5079d62] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:59:09.267 INFO 21062 --- [ocp-schedules-16,52dc62c4d80b16f4,998410baa5079d62] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:59:10.002 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:59:10.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:59:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:59:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:59:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 12:59:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 12:59:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 12:59:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 12:59:10.681 WARN 21062 --- [ocp-async-5,8f60032bfe964f35,f5814176dc078ae9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:10.714 INFO 21062 --- [ocp-async-2,fae50e8305949491,3800d9f21a2aaf58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=10, affectRows=10 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=19, affectRows=19 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=10, validCount=10 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=19, validCount=19 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=4, affectRows=4 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=4, validCount=4 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=25, affectRows=25 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=8, affectRows=8 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=15, affectRows=15 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=20, affectRows=20 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=30, affectRows=30 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=18, affectRows=18 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=40, affectRows=40 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=8, validCount=8 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=15, validCount=15 2024-11-04 12:59:11.137 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=20, validCount=20 2024-11-04 12:59:11.136 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=25, validCount=25 2024-11-04 12:59:11.137 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=18, validCount=18 2024-11-04 12:59:11.137 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=29, validCount=29 2024-11-04 12:59:11.137 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=40, validCount=40 2024-11-04 12:59:11.137 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=64, affectRows=64 2024-11-04 12:59:11.137 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=64, validCount=64 2024-11-04 12:59:11.138 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696280, nodeCount=69, affectRows=69 2024-11-04 12:59:11.139 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696280, nodeCount=67, validCount=67 2024-11-04 12:59:11.962 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:11.964 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:11.983 INFO 21062 --- [prometheus-minute-schedule-1,afd6e1454b7ffbda,a60eb5a4492ed4c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:12.682 WARN 21062 --- [ocp-async-4,5256529209c55beb,22faf96e2672d6e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:12.714 INFO 21062 --- [ocp-async-8,6d39a5dcaa0b0a94,4c4e8a475881bd65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:13.984 INFO 21062 --- [prometheus-cache-monitor1,847c305f65990efd,fd3f0fb4bd6e67d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 12:59:13.984 INFO 21062 --- [prometheus-cache-monitor1,847c305f65990efd,fd3f0fb4bd6e67d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 12:59:14.274 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:55:14.249, end=2024-11-04T12:57:14.272} 2024-11-04 12:59:14.280 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 12:59:14.280 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 12:59:14.682 WARN 21062 --- [ocp-async-7,5382f5a90e642ce5,0dddebf510a67616] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:14.714 INFO 21062 --- [ocp-async-1,1a2126bd90a78047,4aab88e589fd190c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:14.964 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:14.965 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:14.983 INFO 21062 --- [prometheus-minute-schedule-1,7f315623b58aecdb,deb74420f660c7c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:15.036 INFO 21062 --- [ocp-schedules-17,b8b253ff4978f5b1,c6f4d6d9981c94e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:59:16.682 WARN 21062 --- [ocp-async-3,822049d6a8894db6,c5b5ac4dd3f6fd79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:16.715 INFO 21062 --- [ocp-async-5,e89ab9b5b96c16ce,a25148346272ecba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:17.965 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:17.966 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5e9edb07b5c85ec3,3871caf8f9076739] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:18.561 INFO 21062 --- [ocp-async-5,f3438920ffe9962a,88589c66188051f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:59:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:59:18.589479+08:00, lastProcessTime=2024-11-04T12:59:08.588817+08:00 2024-11-04 12:59:18.657 INFO 21062 --- [ocp-schedules-3,8da81a3e2a4192ac,eba623969fd964fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:59:18.682 WARN 21062 --- [ocp-async-4,43d0bf38e45e5514,638a4c2fb4ff9462] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:18.715 INFO 21062 --- [ocp-async-10,6033d516fa496ebe,a490afdb96da6e97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:59:18.589479+08:00 2024-11-04 12:59:18.964 INFO 21062 --- [ocp-schedules-23,8eeb1c2a0e624154,db9ec564719a4e58] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:59:20.682 WARN 21062 --- [ocp-async-7,8def68feb7049870,29b29c80a7e127ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:20.715 INFO 21062 --- [ocp-async-6,cd75db92be476a27,976930ee284d8a41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:20.966 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:20.967 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9779082e0d018ccd,9881275ab6a837ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:22.682 WARN 21062 --- [ocp-async-3,eabd02dde81cfddc,dcef8c6637e61ea3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:22.715 INFO 21062 --- [ocp-async-9,fca968f204c172d8,3113ffb3d58a7ed5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:23.967 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:23.969 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:23.983 INFO 21062 --- [prometheus-minute-schedule-1,1e067c238d00b77c,2e4238b2bb9ef4a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:24.683 WARN 21062 --- [ocp-async-2,8e28fa6486f5316b,4df0d9b3b6a03d2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:24.715 INFO 21062 --- [ocp-async-4,fd45f3bc9e65eee3,fa631b274569d43c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:25.077 INFO 21062 --- [ocp-schedules-23,4d648e57f7f9cc64,df4fd569e46c125e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:59:26.683 WARN 21062 --- [ocp-async-10,6bef09f209d75e58,e5443dbe656e615e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:26.715 INFO 21062 --- [ocp-async-7,5a3b17bb6747f979,312dbabf675f7379] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:26.970 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:26.971 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:26.983 INFO 21062 --- [prometheus-minute-schedule-1,a8cf5fd7bf873202,65deeb287f162e5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ffb76ff20d0396f8,bce70c19b6f51b04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=3600 2024-11-04 12:59:28.498 INFO 21062 --- [ocp-schedules-7,e7588f0cd2673949,73dc2decf10c39bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 12:59:28.499 INFO 21062 --- [ocp-schedules-8,1e1b34b6c9ede001,1af0981bd6a243ab] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 12:59:28.500 INFO 21062 --- [ocp-schedules-13,7acbf98eafbe8a35,6553988bfda681d8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:59:28.500 INFO 21062 --- [ocp-schedules-29,86b45ac58e129c1e,7ecbb9ee57c8860f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:59:28.502 INFO 21062 --- [ocp-schedules-7,e7588f0cd2673949,73dc2decf10c39bf] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 12:59:28.503 INFO 21062 --- [ocp-schedules-15,89a920621480cd9f,d02bbac42f5d40ea] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:59:28.503 INFO 21062 --- [ocp-schedules-13,7acbf98eafbe8a35,6553988bfda681d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:59:28.504 INFO 21062 --- [ocp-schedules-29,86b45ac58e129c1e,7ecbb9ee57c8860f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:28.505 INFO 21062 --- [ocp-schedules-17,465f7651b482b5b9,e4f23fded5312305] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:59:28.505 INFO 21062 --- [ocp-schedules-8,1e1b34b6c9ede001,1af0981bd6a243ab] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 12:59:28.512 INFO 21062 --- [ocp-schedules-23,d27cb7fddeef8e77,726f3e1f51e13874] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:59:28.512 INFO 21062 --- [ocp-schedules-21,bac3a74ec26f4e17,846463539a8f0062] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:59:28.512 INFO 21062 --- [ocp-schedules-3,af026c229f479c4b,8144957dc190abb5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:59:28.512 INFO 21062 --- [ocp-schedules-21,bac3a74ec26f4e17,846463539a8f0062] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:59:28.512 INFO 21062 --- [ocp-schedules-23,d27cb7fddeef8e77,726f3e1f51e13874] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:59:28.512 INFO 21062 --- [ocp-schedules-3,af026c229f479c4b,8144957dc190abb5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:59:28.540 INFO 21062 --- [ocp-schedules-20,8644c0ceacaf1635,dcd743e40cdc7738] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T12:59:28.540+08:00, tenantSize=0 2024-11-04 12:59:28.561 INFO 21062 --- [ocp-async-6,7ad5281316ba3d91,0844747829407fbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:59:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:59:28.588677+08:00, lastProcessTime=2024-11-04T12:59:18.589479+08:00 2024-11-04 12:59:28.623 INFO 21062 --- [ocp-schedules-19,1797ff88a53ea38b,57712b3cca684ccf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 12:59:28.675 INFO 21062 --- [ocp-schedules-12,a08bef551cc49de1,5ea1b241ab6ee9d2] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 12:59:28.675 INFO 21062 --- [ocp-schedules-12,a08bef551cc49de1,5ea1b241ab6ee9d2] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 12:59:28.683 WARN 21062 --- [ocp-async-9,12ebad801e98abe3,1d4e8186bd5be052] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:28.688 INFO 21062 --- [ocp-schedules-18,980119767af8630f,550f19177d8a81ef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:59:28.716 INFO 21062 --- [ocp-async-2,846a999c08d3ab56,e5693f2166f888ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:28.734 INFO 21062 --- [ocp-schedules-15,24a4fcefc8a85bf5,8838316a3e641966] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 12:59:28.739 INFO 21062 --- [ocp-schedules-15,24a4fcefc8a85bf5,8838316a3e641966] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 12:59:28.740 INFO 21062 --- [ocp-schedules-15,24a4fcefc8a85bf5,8838316a3e641966] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 12:59:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:59:28.588677+08:00 2024-11-04 12:59:28.862 INFO 21062 --- [ocp-schedules-21,d38341956637bcc2,25b25bbf3a619293] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 12:59:28.862 INFO 21062 --- [ocp-schedules-21,d38341956637bcc2,25b25bbf3a619293] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 12:59:28.862 INFO 21062 --- [ocp-schedules-21,d38341956637bcc2,25b25bbf3a619293] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 12:59:28.863 INFO 21062 --- [ocp-schedules-21,d38341956637bcc2,25b25bbf3a619293] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 12:59:28.863 INFO 21062 --- [ocp-schedules-21,d38341956637bcc2,25b25bbf3a619293] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 12:59:28.863 INFO 21062 --- [ocp-schedules-21,d38341956637bcc2,25b25bbf3a619293] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 12:59:28.967 INFO 21062 --- [ocp-schedules-3,24bd6da83b97d31a,9a4c2b1d2d0026b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:59:29.262 INFO 21062 --- [ocp-schedules-1,713b75b2a2163b7b,ba4adc0584537c32] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 12:59:29.262 INFO 21062 --- [ocp-schedules-1,713b75b2a2163b7b,ba4adc0584537c32] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 12:59:29.270 INFO 21062 --- [ocp-schedules-16,d421f5eecc685764,f0c17bbd126248a2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:59:29.276 INFO 21062 --- [ocp-schedules-16,d421f5eecc685764,f0c17bbd126248a2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:59:29.276 INFO 21062 --- [ocp-schedules-16,d421f5eecc685764,f0c17bbd126248a2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:59:29.290 INFO 21062 --- [ocp-schedules-1,713b75b2a2163b7b,ba4adc0584537c32] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 12:59:29.290 INFO 21062 --- [ocp-schedules-1,713b75b2a2163b7b,ba4adc0584537c32] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 12:59:29.291 INFO 21062 --- [ocp-schedules-28,0d2c1d89d7ec6e49,d5f24a75a72bde76] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 12:59:29.334 INFO 21062 --- [ocp-schedules-2,d17ee23ba923bea4,a467df7ec1e1051b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 12:59:29.334 INFO 21062 --- [ocp-schedules-2,d17ee23ba923bea4,a467df7ec1e1051b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 12:59:29.334 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 12:59:29.334 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 12:59:29.344 INFO 21062 --- [ocp-schedules-2,d17ee23ba923bea4,a467df7ec1e1051b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:59:29.345 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:59:29.346 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:59:29.346 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:59:29.346 WARN 21062 --- [ocp-schedules-2,d17ee23ba923bea4,a467df7ec1e1051b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:59:29.347 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 12:59:29.347 INFO 21062 --- [ocp-schedules-2,d17ee23ba923bea4,a467df7ec1e1051b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 12:59:29.347 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:59:29.348 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 12:59:29.348 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 12:59:29.349 WARN 21062 --- [ocp-schedules-2,d17ee23ba923bea4,a467df7ec1e1051b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 12:59:29.349 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 12:59:29.357 ERROR 21062 --- [ocp-schedules-2,d17ee23ba923bea4,a467df7ec1e1051b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 12:59:29.407 INFO 21062 --- [ocp-schedules-22,fd448df7557971d1,f614950542388710] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T12:59:29.407+08:00, clusters=[1] 2024-11-04 12:59:29.409 INFO 21062 --- [ocp-schedules-22,fd448df7557971d1,f614950542388710] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 12:59:29.494 INFO 21062 --- [ocp-schedules-9,e5008283fc7331dd,0b3acf0a0dc756de] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T12:59:29.494+08:00, clusters=[1] 2024-11-04 12:59:29.498 INFO 21062 --- [ocp-schedules-9,e5008283fc7331dd,0b3acf0a0dc756de] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 12:59:29.971 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:29.972 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:29.983 INFO 21062 --- [prometheus-minute-schedule-1,8a4e6206710e8981,32bd4048097facda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:59:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 12:59:30.357 INFO 21062 --- [ocp-schedules-10,7b222e7337675ff5,07694e61a9a6da01] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:59:30.357 INFO 21062 --- [ocp-schedules-10,6daca41ba0b67bae,9adfcc0ffc678f5b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 12:59:30.361 INFO 21062 --- [ocp-schedules-10,6daca41ba0b67bae,9adfcc0ffc678f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:30.390 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 12:59:30.390 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:59:30.683 WARN 21062 --- [ocp-async-4,ea59dca75c2e2210,a2f6bd3e8fa9b9cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:30.716 INFO 21062 --- [ocp-async-10,70a8909a0e19ab35,3c975866766efec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:30.973 INFO 21062 --- [ocp-schedules-8,c5da4ec4d2d05253,77f0b5a4fc5a2320] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 12:59:30.976 INFO 21062 --- [ocp-schedules-8,c5da4ec4d2d05253,77f0b5a4fc5a2320] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:31.062 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 12:59:31.064 ERROR 21062 --- [ocp-schedules-8,c5da4ec4d2d05253,77f0b5a4fc5a2320] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 12:59:31.064 INFO 21062 --- [ocp-schedules-8,c5da4ec4d2d05253,77f0b5a4fc5a2320] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 12:59:32.683 WARN 21062 --- [ocp-async-10,0ba4c57dec485881,588b3720745aa3ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:32.716 INFO 21062 --- [ocp-async-6,feeabdabd0b67356,b86f1c896945153c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:32.972 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:32.973 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:32.983 INFO 21062 --- [prometheus-minute-schedule-1,563a54dbc3121edf,f8e77fcad7dc6057] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:34.683 WARN 21062 --- [ocp-async-6,e5b1eab6decf6fc3,10fd62ae58f44a2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:34.716 INFO 21062 --- [ocp-async-9,bc7d54df73765354,c357024fe28aed25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:35.101 INFO 21062 --- [ocp-schedules-18,8744c420f182345a,196a59a86b4d6019] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:59:35.973 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:35.975 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:35.983 INFO 21062 --- [prometheus-minute-schedule-1,f1c8386d3c44cae5,1173bc6cc1f3df02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:36.684 WARN 21062 --- [ocp-async-9,3c6e21065745fe44,95a65acc9b95b1e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:36.716 INFO 21062 --- [ocp-async-4,82974dbb3b29e700,b3b56c40bd928d99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:38.504 INFO 21062 --- [ocp-schedules-13,7acbf98eafbe8a35,6553988bfda681d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 12:59:38.505 INFO 21062 --- [ocp-schedules-29,86b45ac58e129c1e,7ecbb9ee57c8860f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 12:59:38.562 INFO 21062 --- [ocp-async-4,88fc59b27e1e0600,06427e317a3c2eaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:59:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:59:38.589730+08:00, lastProcessTime=2024-11-04T12:59:28.588677+08:00 2024-11-04 12:59:38.684 WARN 21062 --- [ocp-async-1,8a001c2eba6c4660,8f861e5222ad62c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:38.717 INFO 21062 --- [ocp-async-6,844aaf2f4f877a67,1a1aaf928bdcf82c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:59:38.589730+08:00 2024-11-04 12:59:38.970 INFO 21062 --- [ocp-schedules-13,6bff1c0c8fe7958b,08389e6ce0a1dd97] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:59:38.975 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:38.976 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5c2a5af06d4110e2,aaa5f5bc4593c765] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 12:59:40.004 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 12:59:40.684 WARN 21062 --- [ocp-async-6,945cdd7463a1f7f7,5af4b425a12799b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:40.717 INFO 21062 --- [ocp-async-9,9e1074095ad7b718,ddb132594d86e933] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:41.976 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:41.978 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:41.983 INFO 21062 --- [prometheus-minute-schedule-1,674f8884c68b2676,4b679b0860f58852] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:42.684 WARN 21062 --- [ocp-async-9,34bb687108f74c57,936327226fdab333] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:42.717 INFO 21062 --- [ocp-async-4,4244f9a00edd2167,f30358d350b47682] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:44.684 WARN 21062 --- [ocp-async-4,a613d4220f545c4d,accbd122e07df650] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:44.717 INFO 21062 --- [ocp-async-3,a9154dee54254219,e9a7787f85fafa47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:44.978 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:44.979 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:44.983 INFO 21062 --- [prometheus-minute-schedule-1,dc9cfc35414569bf,3691020824186dbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:45.145 INFO 21062 --- [ocp-schedules-17,208186e0132d0ed3,9a360d2010525005] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:59:46.523 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 12:59:46.523 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 12:59:46.531 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:59:46.531 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:59:46.531 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:59:46.531 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:59:46.537 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 12:59:46.537 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 12:59:46.537 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 12:59:46.537 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 12:59:46.574 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:59:46.584 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:59:46.584 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:59:46.584 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:59:46.584 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:59:46.584 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:59:46.592 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:59:46.684 WARN 21062 --- [ocp-async-3,6595f69a356ab6a6,50f4fb2fb1e34cf0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:46.717 INFO 21062 --- [ocp-async-8,98fccf1c0836319f,f2f4ca54095b8998] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:47.150 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 12:59:47.718 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.180922713S 2024-11-04 12:59:47.719 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 12:59:47.815 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 12:59:47.815 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 12:59:47.845 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 12:59:47.903 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 12:59:47.903 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 12:59:47.903 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 12:59:47.903 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 12:59:47.903 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 12:59:47.903 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 12:59:47.914 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 12:59:47.979 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:47.981 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:47.983 INFO 21062 --- [prometheus-minute-schedule-1,598e7448e2aa1d5f,a1233f3b9e6f694e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:48.092 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] o.s.boot.SpringApplication : Started application in 1.56 seconds (JVM running for 1801.771) 2024-11-04 12:59:48.092 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.560745557S 2024-11-04 12:59:48.561 INFO 21062 --- [ocp-schedules-15,e13f2a16604b55a9,2a34dca504b90c27] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 12:59:48.562 INFO 21062 --- [ocp-async-2,93a18aa8dac42501,a3fa1ffb220f5e4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:59:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:59:48.588427+08:00, lastProcessTime=2024-11-04T12:59:38.589730+08:00 2024-11-04 12:59:48.685 WARN 21062 --- [ocp-async-9,e412b0cfae73f1e3,1c95fcac3f6b80b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:48.717 INFO 21062 --- [ocp-async-10,1468f0c547fd59d2,f23af56d3fca2935] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:48.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:59:48.588427+08:00 2024-11-04 12:59:48.972 INFO 21062 --- [ocp-schedules-15,af4ca37e420efedf,08b0851c4b0da18e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:59:49.278 INFO 21062 --- [ocp-schedules-11,bfbb1935baff480e,f1afae6c84f53ca8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 12:59:49.285 INFO 21062 --- [ocp-schedules-11,bfbb1935baff480e,f1afae6c84f53ca8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:59:49.285 INFO 21062 --- [ocp-schedules-11,bfbb1935baff480e,f1afae6c84f53ca8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 12:59:50.685 WARN 21062 --- [ocp-async-10,b05db8a350550d2a,106f5735d179d8a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:50.718 INFO 21062 --- [ocp-async-1,a5126d26c343582a,01b55a39674703ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:50.981 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:50.983 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:50.983 INFO 21062 --- [prometheus-minute-schedule-1,def5d9994d2959c4,15ad4c14c21b259d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:52.685 WARN 21062 --- [ocp-async-1,d921c48459ee2c99,c95681ec74947ae6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:52.718 INFO 21062 --- [ocp-async-6,6710ab80b27d37bf,1528c8a7a33dc383] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:53.983 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ad122c54a4cb5798,8f02d5162dbf0dba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:53.985 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:54.685 WARN 21062 --- [ocp-async-6,263d3e6bc7653be1,c267813aaeab76f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:54.718 INFO 21062 --- [ocp-async-9,cdcce2e5eeebb752,5c74db92948d48c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:55.172 INFO 21062 --- [ocp-schedules-22,15c2d940e2195634,79b1d93c9ab82e15] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 12:59:56.685 WARN 21062 --- [ocp-async-9,58f3791f888b669b,a8ed9048a327d938] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:56.718 INFO 21062 --- [ocp-async-10,65b97f0d5cbcb7b2,2cba8024e3a8d765] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:56.983 INFO 21062 --- [prometheus-minute-schedule-1,2b65518f051f3dd6,506f5a8f7a7a95e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:56.985 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:56.987 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 12:59:58.500 INFO 21062 --- [ocp-schedules-24,84f368d61a053b3c,0d225110214db791] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 12:59:58.500 INFO 21062 --- [ocp-schedules-22,d2e306235a734457,4f8f3021fd2841ba] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 12:59:58.502 INFO 21062 --- [ocp-schedules-13,03fec0598d450d95,9997e9eed058f9ab] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 12:59:58.504 INFO 21062 --- [ocp-schedules-24,84f368d61a053b3c,0d225110214db791] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 12:59:58.505 INFO 21062 --- [ocp-schedules-22,d2e306235a734457,4f8f3021fd2841ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 12:59:58.505 INFO 21062 --- [ocp-schedules-5,ce40cbbfa0d8a315,e681d4ac8d3d5d66] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 12:59:58.511 INFO 21062 --- [ocp-schedules-18,aabcc4ccdca89781,4509f4f399d5f450] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 12:59:58.511 INFO 21062 --- [ocp-schedules-15,a081672da6fe9d96,07b6585a6312b0e5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 12:59:58.511 INFO 21062 --- [ocp-schedules-18,aabcc4ccdca89781,4509f4f399d5f450] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 12:59:58.511 INFO 21062 --- [ocp-schedules-15,a081672da6fe9d96,07b6585a6312b0e5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 12:59:58.512 INFO 21062 --- [ocp-schedules-30,313f76ee83dfdd8c,2c5c6532ef09a19d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 12:59:58.512 INFO 21062 --- [ocp-schedules-30,313f76ee83dfdd8c,2c5c6532ef09a19d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 12:59:58.562 INFO 21062 --- [ocp-async-5,6de03bfbed5d5002,222625dd944485d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 12:59:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T12:59:58.588457+08:00, lastProcessTime=2024-11-04T12:59:48.588427+08:00 2024-11-04 12:59:58.685 WARN 21062 --- [ocp-async-1,e6cfe81ccc896e8b,eb2d88b07bdcb429] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 12:59:58.691 INFO 21062 --- [ocp-schedules-23,6e16f6286e369569,e4043c9d54ae63a3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 12:59:58.718 INFO 21062 --- [ocp-async-2,4ab2feed42333ab9,b49921401a7dd128] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 12:59:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T12:59:58.588457+08:00 2024-11-04 12:59:58.976 INFO 21062 --- [ocp-schedules-7,0731ebab99e8fa84,297eef633b1d538b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 12:59:59.983 INFO 21062 --- [prometheus-minute-schedule-1,3c277a4e2a3c08bc,173f26b0bbfed4c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696400, interval=60 2024-11-04 12:59:59.987 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 12:59:59.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:00.390 INFO 21062 --- [ocp-schedules-10,6daca41ba0b67bae,9adfcc0ffc678f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:00:00.390 INFO 21062 --- [ocp-schedules-10,87b8ec9959ff92ae,c52fd187083fd39e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:00:00.394 INFO 21062 --- [ocp-schedules-10,87b8ec9959ff92ae,c52fd187083fd39e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:00.425 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:00:00.425 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:00:00.685 WARN 21062 --- [ocp-async-6,5b6fd7e10ee3b75c,7d5a4466c07a449b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:00.719 INFO 21062 --- [ocp-async-7,82a4830f9018d06b,76aa7d65c792e46c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:00.799 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,6cab4e580ea2d3de] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:55:01.0, nextRun=2024-11-04 13:00:00.0, currentDate=Mon Nov 04 13:00:00 CST 2024 2024-11-04 13:00:00.799 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,6cab4e580ea2d3de] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 12:55:01.0, nextRunTime=2024-11-04 13:00:00.0} 2024-11-04 13:00:00.826 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,6cab4e580ea2d3de] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:00:00.899 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,6cab4e580ea2d3de] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:00:00.899 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,6cab4e580ea2d3de] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:00:02.686 WARN 21062 --- [ocp-async-9,c9a0792af33a6217,388ac2763e4f1203] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:02.719 INFO 21062 --- [ocp-async-3,e5166d1b0a0bbbcf,f4c999adece736a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:02.983 INFO 21062 --- [prometheus-minute-schedule-1,fb3d556ef7f89bbc,13419f7c0580714e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:02.989 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:02.989 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:04.686 WARN 21062 --- [ocp-async-10,04dc56be056e4241,6e2f07882db23ba1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:04.719 INFO 21062 --- [ocp-async-1,0808a046e4b008d9,c6d95f46f648a880] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:04.969 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,28813347a8e3f863] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 12:50:04.0, nextRun=2024-11-04 13:00:04.0, currentDate=Mon Nov 04 13:00:04 CST 2024 2024-11-04 13:00:04.970 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,28813347a8e3f863] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 12:50:04.0, nextRunTime=2024-11-04 13:00:04.0} 2024-11-04 13:00:04.977 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,28813347a8e3f863] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 13:00:04.977 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,28813347a8e3f863] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T13:00:04.977+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 13:00:05.218 INFO 21062 --- [ocp-schedules-16,0faeb24ecb4a954f,923dce64ca9afb64] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:00:05.367 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,000e909fc935ef57] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 71, operation:EXECUTE, state: RUNNING 2024-11-04 13:00:05.389 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=71, context=Context{parallelIdx=-1, stringMap={task_instance_id=5, task_operation=execute, latest_execution_start_time=2024-11-04T13:00:05.359+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=71}, listMap={}}, executor=10.46.128.24 2024-11-04 13:00:05.396 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:00:05.397 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:00:05.398 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.399 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:00:05.400 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.401 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.401 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.402 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:00:05.416 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.417 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.417 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.418 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:00:05.432 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.433 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.433 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.434 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:00:05.448 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.449 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.449 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.450 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:00:05.463 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.464 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.464 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.465 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:00:05.470 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.471 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.471 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.472 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:00:05.476 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.477 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.477 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.477 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:00:05.482 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.482 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.483 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.483 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:00:05.490 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.490 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.491 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.491 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:00:05.496 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.496 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.497 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.497 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:00:05.503 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.503 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.504 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.504 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:00:05.512 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.512 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.513 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.514 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:00:05.527 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.527 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.528 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.528 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:00:05.533 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.533 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.534 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.534 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:00:05.541 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.541 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.542 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.542 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:00:05.548 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.549 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.549 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.550 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:00:05.554 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.555 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.556 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.556 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:00:05.560 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.560 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.561 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.561 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:00:05.566 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.566 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.567 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.567 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:00:05.572 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.572 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.573 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.573 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:00:05.577 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.578 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.578 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.578 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:00:05.583 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.584 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.584 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.585 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:00:05.589 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.589 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.589 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.590 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:00:05.594 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.595 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.595 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.595 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:00:05.599 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.600 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.600 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.600 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:00:05.604 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.605 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.605 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.606 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:00:05.610 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.610 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.611 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.611 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:00:05.616 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.616 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.617 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.617 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:00:05.621 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.622 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.622 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.623 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:00:05.627 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.627 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.628 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.628 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:00:05.632 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.632 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.633 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.633 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:00:05.637 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.637 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.638 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.638 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:00:05.643 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.644 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.644 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.645 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:00:05.649 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.649 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.650 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.650 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:00:05.657 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.658 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.658 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.659 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:00:05.665 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.665 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.666 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.666 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:00:05.672 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.673 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.673 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.673 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:00:05.679 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.679 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.680 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.680 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:00:05.684 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.684 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.685 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.685 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:00:05.689 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.689 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.690 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.690 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:00:05.694 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.694 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.695 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.695 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:00:05.699 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.700 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.700 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.701 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:00:05.709 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.710 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.710 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.711 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:00:05.731 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.732 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.732 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.733 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:00:05.738 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.739 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.739 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.740 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:00:05.746 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.746 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.747 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.747 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:00:05.753 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.753 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.754 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.754 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:00:05.758 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.758 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.759 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.759 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:00:05.764 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.764 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.765 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.765 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:00:05.769 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:00:05.769 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:00:05.771 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.771 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:00:05.773 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.773 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.774 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.774 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:00:05.780 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.780 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:00:05.781 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:05.781 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:00:05.786 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:00:05.794 INFO 21062 --- [subtask-executor4,dd193f8f67a5a423,c6d73a4fb29d6fc3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 71, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 13:00:05.983 INFO 21062 --- [prometheus-minute-schedule-1,424adecad1c95331,3ac17b8c76876d84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:05.989 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:05.990 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:06.026 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:5, status:SUCCESS 2024-11-04 13:00:06.027 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:5, status:SUCCESS 2024-11-04 13:00:06.686 WARN 21062 --- [ocp-async-2,cdb431d8d94b2204,bf516953767f6fb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:06.719 INFO 21062 --- [ocp-async-6,6f3a199e3f390fe4,abe5881e57077179] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:08.505 INFO 21062 --- [ocp-schedules-24,84f368d61a053b3c,0d225110214db791] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:00:08.506 INFO 21062 --- [ocp-schedules-22,d2e306235a734457,4f8f3021fd2841ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:00:08.562 INFO 21062 --- [ocp-async-6,d8ee1d9aeee5a5db,d4f3bca3ed7229ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:00:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:00:08.587901+08:00, lastProcessTime=2024-11-04T12:59:58.588457+08:00 2024-11-04 13:00:08.686 WARN 21062 --- [ocp-async-9,02e80c040f46dcfd,24bb55d5534ff8b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:08.719 INFO 21062 --- [ocp-async-5,a03a3ae7be154f7e,bbd89ca160e43651] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:00:08.587901+08:00 2024-11-04 13:00:08.979 INFO 21062 --- [ocp-schedules-24,7bddb8d0d3ba7eff,7e010359d5cd12b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:00:08.983 INFO 21062 --- [prometheus-minute-schedule-1,5883b6bff1b894ac,192cf888c0457492] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:08.990 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:08.991 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:09.287 INFO 21062 --- [ocp-schedules-27,0954cdbbc0d04501,d32b6ed58d458485] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:00:09.292 INFO 21062 --- [ocp-schedules-27,0954cdbbc0d04501,d32b6ed58d458485] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:00:09.292 INFO 21062 --- [ocp-schedules-27,0954cdbbc0d04501,d32b6ed58d458485] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:00:10.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:00:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:00:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:00:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:00:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:00:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:00:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:00:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:00:10.686 WARN 21062 --- [ocp-async-10,12334fc575a3002a,7b9b9693f729d046] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:10.719 INFO 21062 --- [ocp-async-8,85d673976e155fa2,c8d50db3d91d86c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:11.146 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=2, affectRows=2 2024-11-04 13:00:11.146 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=2, validCount=2 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=14, affectRows=14 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=4, affectRows=4 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=4, validCount=4 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=14, validCount=14 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=2, affectRows=2 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=8, affectRows=8 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=2, validCount=2 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=8, affectRows=8 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=8, validCount=8 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=8, validCount=8 2024-11-04 13:00:11.147 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=25, affectRows=25 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=7, affectRows=7 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=25, validCount=25 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=6, validCount=6 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=49, affectRows=49 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=41, affectRows=41 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=28, affectRows=28 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=49, validCount=49 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=37, affectRows=37 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=40, validCount=40 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=28, validCount=28 2024-11-04 13:00:11.148 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=37, validCount=37 2024-11-04 13:00:11.149 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=35, affectRows=35 2024-11-04 13:00:11.149 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=35, validCount=35 2024-11-04 13:00:11.149 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=21, affectRows=21 2024-11-04 13:00:11.150 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=21, validCount=21 2024-11-04 13:00:11.150 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696340, nodeCount=41, affectRows=41 2024-11-04 13:00:11.150 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696340, nodeCount=40, validCount=40 2024-11-04 13:00:11.983 INFO 21062 --- [prometheus-minute-schedule-1,ccdf79c77e69a22d,f14d3920ccac076c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:11.991 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:11.992 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:12.687 WARN 21062 --- [ocp-async-2,997407b8b45dc016,cce7fb2d282aacbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:12.720 INFO 21062 --- [ocp-async-4,5334b54803c035ad,00e32725e033a000] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:13.984 INFO 21062 --- [prometheus-cache-monitor1,c9efef3fa9ef66bc,73d20e5d2fa9b020] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:00:13.984 INFO 21062 --- [prometheus-cache-monitor1,c9efef3fa9ef66bc,73d20e5d2fa9b020] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:00:14.687 WARN 21062 --- [ocp-async-7,40a440a94c76bd77,63b01cee7eabe1f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:14.720 INFO 21062 --- [ocp-async-9,012f6217cc082d62,a177395ec361589a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:14.983 INFO 21062 --- [prometheus-minute-schedule-1,4481f4ab4e1437c8,ae7fadb09a68bab4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:14.992 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:14.993 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:15.260 INFO 21062 --- [ocp-schedules-2,5616dd621fff3e81,bccd9ac7672ca293] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:00:16.687 WARN 21062 --- [ocp-async-5,854f1335a7817f3f,41ef19c8b6b24e13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:16.720 INFO 21062 --- [ocp-async-10,88dd237c6402393f,9c9d6087e7075c24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5eef6d8515dfadfb,e4c3cbe0ef117fc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:17.993 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:17.994 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:18.562 INFO 21062 --- [ocp-async-1,3c47761e9ecfb97b,8a485b3d915493c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:00:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:00:18.588463+08:00, lastProcessTime=2024-11-04T13:00:08.587901+08:00 2024-11-04 13:00:18.657 INFO 21062 --- [ocp-schedules-18,8448552aa380203f,b508943e2c34e5c4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:00:18.660 INFO 21062 --- [ocp-schedules-18,8448552aa380203f,b508943e2c34e5c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:18.661 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:00:18.687 WARN 21062 --- [ocp-async-2,c997a6f739406f3a,8b4c8ac374af0fc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:18.720 INFO 21062 --- [ocp-async-6,8fc72f6b72c79186,0bede6d6b6d0a8bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:00:18.588463+08:00 2024-11-04 13:00:18.981 INFO 21062 --- [ocp-schedules-28,5b3ba94ebeb317d8,177a1d512883238c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:00:20.687 WARN 21062 --- [ocp-async-7,169a744310c48505,5324764c4bbf57cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:20.720 INFO 21062 --- [ocp-async-3,e61c3fd919ab5202,16dee9bbce09788e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4ade5a60cee65e7a,23a4b17fa7e137db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:20.994 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:20.995 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:22.687 WARN 21062 --- [ocp-async-5,55a414be3c3d34b8,d0ae76f819dd6ee9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:22.720 INFO 21062 --- [ocp-async-10,9b1be2f030503d3c,c7499e58ccc35335] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:23.983 INFO 21062 --- [prometheus-minute-schedule-1,174b2797f91b88ee,1689deab5bc8d879] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:23.995 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:23.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:24.688 WARN 21062 --- [ocp-async-8,a2c703f28451320a,723462daf64801f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:24.721 INFO 21062 --- [ocp-async-2,e6c13be092398dff,5a357b245c38eb9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:25.311 INFO 21062 --- [ocp-schedules-4,61b70a8c39b4ed50,779076f50aa70d32] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:00:26.688 WARN 21062 --- [ocp-async-6,b0422654b0a88693,774830913fc622fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:26.721 INFO 21062 --- [ocp-async-9,e1ebf49f928eac96,d4b2cc5bbad27d52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:26.983 INFO 21062 --- [prometheus-minute-schedule-1,03b3cbe172e4041c,b9be419d8405c779] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a9b98f9d6a89e65c,52a06efe6dd7da08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:00:26.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:26.997 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:28.498 INFO 21062 --- [ocp-schedules-27,d7d07af4a62a4e25,cdc451c49a2c39a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:00:28.499 INFO 21062 --- [ocp-schedules-24,8e4752c4abcdfb3b,db0372fba9f49ebe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:00:28.500 INFO 21062 --- [ocp-schedules-30,70d30aa811ee4ca6,a39c9f01f0ccbaf2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:00:28.500 INFO 21062 --- [ocp-schedules-11,8077266544960b55,b68bf50f76be29a9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:00:28.502 INFO 21062 --- [ocp-schedules-13,96fe4a5d63cf004b,8cae195adeee554a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:00:28.504 INFO 21062 --- [ocp-schedules-27,d7d07af4a62a4e25,cdc451c49a2c39a3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:00:28.505 INFO 21062 --- [ocp-schedules-24,8e4752c4abcdfb3b,db0372fba9f49ebe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:00:28.505 INFO 21062 --- [ocp-schedules-30,70d30aa811ee4ca6,a39c9f01f0ccbaf2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:28.505 INFO 21062 --- [ocp-schedules-11,8077266544960b55,b68bf50f76be29a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:28.505 INFO 21062 --- [ocp-schedules-17,4fd191eb1f9208b1,361eec2dc4bee135] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:00:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:00:28.511 INFO 21062 --- [ocp-schedules-6,d341584fca22ade8,d1c4d24709fd8227] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:00:28.511 INFO 21062 --- [ocp-schedules-6,d341584fca22ade8,d1c4d24709fd8227] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:00:28.511 INFO 21062 --- [ocp-schedules-26,26a1548764b9a54d,ed4de94627daf3df] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:00:28.511 INFO 21062 --- [ocp-schedules-26,26a1548764b9a54d,ed4de94627daf3df] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:00:28.511 INFO 21062 --- [ocp-schedules-14,0b943b543645ead5,2a8a9330cc68a73d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:00:28.511 INFO 21062 --- [ocp-schedules-14,0b943b543645ead5,2a8a9330cc68a73d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:00:28.557 INFO 21062 --- [ocp-schedules-23,60eee8e5be18a37d,0515f2c53a15276f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:00:28.557 INFO 21062 --- [ocp-schedules-23,60eee8e5be18a37d,0515f2c53a15276f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:00:28.563 INFO 21062 --- [ocp-async-3,496b1ab0e6cb987f,a504f48bf09d463b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:00:28.576 INFO 21062 --- [ocp-schedules-20,abc6ed8c55db1ed1,ba6dc19db2dce557] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:00:28.577 INFO 21062 --- [ocp-schedules-20,abc6ed8c55db1ed1,ba6dc19db2dce557] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:00:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:00:28.589049+08:00, lastProcessTime=2024-11-04T13:00:18.588463+08:00 2024-11-04 13:00:28.625 INFO 21062 --- [ocp-schedules-12,08d6fba091d8d5e8,a571dba0fa0c4e6e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:00:28.661 INFO 21062 --- [ocp-schedules-18,8448552aa380203f,b508943e2c34e5c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:00:28.679 INFO 21062 --- [ocp-schedules-7,42bb62cfe2e55898,6d2fd241b0699309] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:00:28.679 INFO 21062 --- [ocp-schedules-7,42bb62cfe2e55898,6d2fd241b0699309] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:00:28.688 WARN 21062 --- [ocp-async-10,fe57b094f5b7cbdb,789a26016715bb9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:28.693 INFO 21062 --- [ocp-schedules-25,49eca7cd4cf2ec99,8d252864c961a486] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:00:28.721 INFO 21062 --- [ocp-async-4,1bea16383dce4811,81913ce61de1269a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:28.740 INFO 21062 --- [ocp-schedules-6,df2d9c25bf5187c6,f6d1e0acfbdd7383] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:00:28.746 INFO 21062 --- [ocp-schedules-6,df2d9c25bf5187c6,f6d1e0acfbdd7383] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:00:28.746 INFO 21062 --- [ocp-schedules-6,df2d9c25bf5187c6,f6d1e0acfbdd7383] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:00:28.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:00:28.589049+08:00 2024-11-04 13:00:28.864 INFO 21062 --- [ocp-schedules-14,1c48a90b15e48e6b,a594362731391489] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:00:28.864 INFO 21062 --- [ocp-schedules-14,1c48a90b15e48e6b,a594362731391489] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:00:28.870 INFO 21062 --- [ocp-schedules-14,1c48a90b15e48e6b,a594362731391489] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:00:28.870 INFO 21062 --- [ocp-schedules-14,1c48a90b15e48e6b,a594362731391489] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:00:28.877 INFO 21062 --- [ocp-schedules-14,1c48a90b15e48e6b,a594362731391489] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:00:28.877 INFO 21062 --- [ocp-schedules-14,1c48a90b15e48e6b,a594362731391489] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:00:28.983 INFO 21062 --- [ocp-schedules-1,7fe9afde6439f110,8756093169dcce6a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:00:29.291 INFO 21062 --- [ocp-schedules-20,37d5c5154cc22873,20eb59e08063609b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:00:29.291 INFO 21062 --- [ocp-schedules-20,37d5c5154cc22873,20eb59e08063609b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:00:29.294 INFO 21062 --- [ocp-schedules-28,73219289d9c2b726,0e037dbe97466feb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:00:29.301 INFO 21062 --- [ocp-schedules-28,73219289d9c2b726,0e037dbe97466feb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:00:29.301 INFO 21062 --- [ocp-schedules-28,73219289d9c2b726,0e037dbe97466feb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:00:29.319 INFO 21062 --- [ocp-schedules-20,37d5c5154cc22873,20eb59e08063609b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:00:29.319 INFO 21062 --- [ocp-schedules-20,37d5c5154cc22873,20eb59e08063609b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:00:29.321 INFO 21062 --- [ocp-schedules-9,fb8f975d8145a86f,91c25a12d142f828] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:00:29.352 INFO 21062 --- [ocp-schedules-19,0ca1c01b4cc7e10c,d653a3fca6edf627] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:00:29.355 INFO 21062 --- [ocp-schedules-19,0ca1c01b4cc7e10c,d653a3fca6edf627] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:29.367 INFO 21062 --- [ocp-schedules-8,2c952364f96b564d,144574a044580156] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:00:29.368 INFO 21062 --- [ocp-schedules-8,2c952364f96b564d,144574a044580156] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:00:29.368 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:00:29.368 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:00:29.377 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:00:29.378 INFO 21062 --- [ocp-schedules-8,2c952364f96b564d,144574a044580156] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:00:29.378 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:00:29.380 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:00:29.380 WARN 21062 --- [ocp-schedules-8,2c952364f96b564d,144574a044580156] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:00:29.380 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:00:29.380 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:00:29.380 INFO 21062 --- [ocp-schedules-8,2c952364f96b564d,144574a044580156] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:00:29.380 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:00:29.381 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:00:29.381 WARN 21062 --- [ocp-schedules-8,2c952364f96b564d,144574a044580156] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:00:29.381 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:00:29.388 ERROR 21062 --- [ocp-schedules-8,2c952364f96b564d,144574a044580156] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:00:29.417 ERROR 21062 --- [ocp-schedules-19,0ca1c01b4cc7e10c,d653a3fca6edf627] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:00:29.417 INFO 21062 --- [ocp-schedules-19,0ca1c01b4cc7e10c,d653a3fca6edf627] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:00:29.434 INFO 21062 --- [ocp-schedules-12,73d1c8b635f0540d,8a58e60b188ec03b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:00:29.434+08:00, clusters=[1] 2024-11-04 13:00:29.436 INFO 21062 --- [ocp-schedules-12,73d1c8b635f0540d,8a58e60b188ec03b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:00:29.523 INFO 21062 --- [ocp-schedules-7,d8f8956a82fdcdd0,932d2af0c15bd8d0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:00:29.523+08:00, clusters=[1] 2024-11-04 13:00:29.527 INFO 21062 --- [ocp-schedules-7,d8f8956a82fdcdd0,932d2af0c15bd8d0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:00:29.983 INFO 21062 --- [prometheus-minute-schedule-1,480783318233ac64,563d6cae7ace5879] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:29.997 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:29.998 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:00:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:00:30.425 INFO 21062 --- [ocp-schedules-10,87b8ec9959ff92ae,c52fd187083fd39e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:00:30.425 INFO 21062 --- [ocp-schedules-10,47fdf35fc632eea4,f2be9f13d0276e92] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:00:30.429 INFO 21062 --- [ocp-schedules-10,47fdf35fc632eea4,f2be9f13d0276e92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:30.458 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:00:30.458 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:00:30.688 WARN 21062 --- [ocp-async-4,7602a8b3f917175a,b9562205b90e8b41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:30.722 INFO 21062 --- [ocp-async-6,435ef8e1cbd2f035,cd7707ce02035e3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:31.065 INFO 21062 --- [ocp-schedules-6,ae89e5947b501acc,6b0993705561bb5b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:00:31.068 INFO 21062 --- [ocp-schedules-6,ae89e5947b501acc,6b0993705561bb5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:31.151 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:00:31.154 ERROR 21062 --- [ocp-schedules-6,ae89e5947b501acc,6b0993705561bb5b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:00:31.154 INFO 21062 --- [ocp-schedules-6,ae89e5947b501acc,6b0993705561bb5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:00:32.688 WARN 21062 --- [ocp-async-6,e8e50e2b47c97f7e,fb554055c4e996d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:32.722 INFO 21062 --- [ocp-async-5,cddd9cd483dd672d,ce17652527b66182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:32.983 INFO 21062 --- [prometheus-minute-schedule-1,608c93db8f15a7f4,663034655a497682] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:32.998 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:32.999 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:34.688 WARN 21062 --- [ocp-async-5,359b09699a8d4e5e,93026f77c85738d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:34.722 INFO 21062 --- [ocp-async-8,55c702285c1a89d6,4e67dc80faa83767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:35.348 INFO 21062 --- [ocp-schedules-17,8d81f4d98115b8f1,0dfc14ef209a0534] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:00:35.983 INFO 21062 --- [prometheus-minute-schedule-1,36e7035ff0c450f9,075876cae4d96baa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:35.999 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:36.000 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:36.689 WARN 21062 --- [ocp-async-8,2628db9b51686d42,2c2353210910208c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:36.722 INFO 21062 --- [ocp-async-9,5a9d795c05ce229b,c10c0caed2a3a3d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:38.506 INFO 21062 --- [ocp-schedules-11,8077266544960b55,b68bf50f76be29a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:00:38.506 INFO 21062 --- [ocp-schedules-30,70d30aa811ee4ca6,a39c9f01f0ccbaf2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:00:38.563 INFO 21062 --- [ocp-async-9,22fe7ea283c39fd4,16be9ab1587dc369] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:00:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:00:38.588894+08:00, lastProcessTime=2024-11-04T13:00:28.589049+08:00 2024-11-04 13:00:38.689 WARN 21062 --- [ocp-async-7,d10650358ab69b64,e8a03f0618fb9ad1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:38.722 INFO 21062 --- [ocp-async-1,854adbe3d6bce1d1,776d546f83f101c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:00:38.588894+08:00 2024-11-04 13:00:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0b26907b9409a7ed,6ae72aff25a4707d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:38.986 INFO 21062 --- [ocp-schedules-26,e80d2c8256557a8f,54af7b327edc971d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:00:39.000 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:39.001 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:00:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:00:40.689 WARN 21062 --- [ocp-async-1,577ca8796b2ec73f,2a17b95890188c32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:40.722 INFO 21062 --- [ocp-async-8,fc61a76993025f21,a8f24e7deb30a5f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:00:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:00:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:00:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:00:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:00:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:00:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d16ff9b5d573d13b,318d6c0a42d81dff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:42.001 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:42.002 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:42.689 WARN 21062 --- [ocp-async-8,5d1ce9067abd3c43,60e602133d191f99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:42.723 INFO 21062 --- [ocp-async-9,9847cd83df1a845b,0590be2696e16eb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:44.689 WARN 21062 --- [ocp-async-9,d298e26a399b6694,f6f6150b19dda4f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:44.723 INFO 21062 --- [ocp-async-3,6a49a1756ac5db69,c390a62ff90b7577] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d2111ffcc32827e4,d2801cbf63f87eb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:45.002 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:45.003 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:45.395 INFO 21062 --- [ocp-schedules-29,77bca28ccab2474d,ee1b7ebab1f5049b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:00:46.689 WARN 21062 --- [ocp-async-3,4bfa96d7bf475081,f7c0e5354653325a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:46.723 INFO 21062 --- [ocp-async-10,cfa271d965eabfdb,2bc4b0474f5d3bdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:47.983 INFO 21062 --- [prometheus-minute-schedule-1,ba1c5bc38bd392b5,47a525a4c45cc470] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:48.003 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:48.004 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:48.563 INFO 21062 --- [ocp-async-4,a9d9768c7da6f064,b2d6270d1e627eb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:00:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:00:48.591286+08:00, lastProcessTime=2024-11-04T13:00:38.588894+08:00 2024-11-04 13:00:48.690 WARN 21062 --- [ocp-async-8,117a5dd38757180d,bc0e7ff9deaf55cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:48.723 INFO 21062 --- [ocp-async-6,1b1936f0b4002382,0e82706ff544b788] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:00:48.591286+08:00 2024-11-04 13:00:48.990 INFO 21062 --- [ocp-schedules-17,dd01fe3c4d7eae01,ab98f8c8735be62a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:00:49.304 INFO 21062 --- [ocp-schedules-25,6fb3219eb4c95347,08b3a1ed901a4b57] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:00:49.311 INFO 21062 --- [ocp-schedules-25,6fb3219eb4c95347,08b3a1ed901a4b57] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:00:49.311 INFO 21062 --- [ocp-schedules-25,6fb3219eb4c95347,08b3a1ed901a4b57] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:00:50.690 WARN 21062 --- [ocp-async-6,ae6e9ab85982d65a,551f9e76a9ebc261] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:50.723 INFO 21062 --- [ocp-async-7,355a14d082db3da0,b13d1b530de7dedf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:50.983 INFO 21062 --- [prometheus-minute-schedule-1,07ece1276ebd391e,99209ee118e16fce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:51.004 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:51.005 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:52.690 WARN 21062 --- [ocp-async-7,1c3fbc213e12d44d,b1878c5a0c22a4ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:52.723 INFO 21062 --- [ocp-async-1,d14fd80359ac3d44,81a79998a8eaa561] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:53.983 INFO 21062 --- [prometheus-minute-schedule-1,df7defd6524e4970,79ed7078c2161914] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:54.005 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:54.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:54.690 WARN 21062 --- [ocp-async-1,f73b143f26a1eb8e,810f87e2f9d7cdbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:54.724 INFO 21062 --- [ocp-async-8,827100983a527263,a8d3601bfe64f11d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:55.425 INFO 21062 --- [ocp-schedules-6,b1e7ebf4dae9db86,913f3c06ea20ec0b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:00:56.690 WARN 21062 --- [ocp-async-8,dbd6e78a4e94ab90,4c760f72552bf656] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:56.724 INFO 21062 --- [ocp-async-6,4cf8531ccd029579,47f218c08a11514c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7d743b519b0e33df,3fc244973981e3c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:00:57.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:00:57.007 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:00:58.500 INFO 21062 --- [ocp-schedules-2,33fcff6c1e41fafd,5bd6c2201f85fca1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:00:58.500 INFO 21062 --- [ocp-schedules-19,71551fa4c0cb1ee0,92e46f525a4a94de] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:00:58.502 INFO 21062 --- [ocp-schedules-18,04eed13ec724898a,1255f25f441f251a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:00:58.504 INFO 21062 --- [ocp-schedules-2,33fcff6c1e41fafd,5bd6c2201f85fca1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:58.504 INFO 21062 --- [ocp-schedules-19,71551fa4c0cb1ee0,92e46f525a4a94de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:00:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:00:58.505 INFO 21062 --- [ocp-schedules-3,046e347630817cc7,8ac7c0a3495b2111] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:00:58.511 INFO 21062 --- [ocp-schedules-7,c9ddbaa62da3061b,8cabd6feb35c7f5b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:00:58.511 INFO 21062 --- [ocp-schedules-7,c9ddbaa62da3061b,8cabd6feb35c7f5b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:00:58.511 INFO 21062 --- [ocp-schedules-24,db5b63a3f66ac0c5,c7139a5446d278c4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:00:58.511 INFO 21062 --- [ocp-schedules-24,db5b63a3f66ac0c5,c7139a5446d278c4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:00:58.512 INFO 21062 --- [ocp-schedules-16,2daf0c08dd523498,16e00fdb10ccd5ec] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:00:58.512 INFO 21062 --- [ocp-schedules-16,2daf0c08dd523498,16e00fdb10ccd5ec] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:00:58.564 INFO 21062 --- [ocp-async-6,644217345e1eb28f,30daabb28745d830] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:00:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:00:58.588459+08:00, lastProcessTime=2024-11-04T13:00:48.591286+08:00 2024-11-04 13:00:58.690 WARN 21062 --- [ocp-async-7,88c87024b34d564a,6c5cd5af06c520b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:00:58.696 INFO 21062 --- [ocp-schedules-20,c06b7aaf827fa25f,116b12dc35e480aa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:00:58.724 INFO 21062 --- [ocp-async-10,30ecc18be5308a36,fcc40f78d533dfc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:00:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:00:58.588459+08:00 2024-11-04 13:00:58.993 INFO 21062 --- [ocp-schedules-12,6bc1328d696a65d8,12a2f17a4c56d394] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:00:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a4948f4ae28abbbe,07578ed96680d0dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696460, interval=60 2024-11-04 13:01:00.007 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:00.008 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:00.458 INFO 21062 --- [ocp-schedules-10,47fdf35fc632eea4,f2be9f13d0276e92] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:01:00.458 INFO 21062 --- [ocp-schedules-10,13f77fb25b3722e0,b70a8c4d91f94474] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:01:00.462 INFO 21062 --- [ocp-schedules-10,13f77fb25b3722e0,b70a8c4d91f94474] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:00.492 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:01:00.492 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:01:00.691 WARN 21062 --- [ocp-async-1,1155bcad9d5ad385,592084e535b21071] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:00.724 INFO 21062 --- [ocp-async-9,517fa5e62a9620a4,ca3b70498d335659] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:00.892 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ed438d12b5fc8f65] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2024-11-04 13:01:00.0, currentDate=Mon Nov 04 13:01:00 CST 2024 2024-11-04 13:01:00.892 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ed438d12b5fc8f65] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999995, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=null, nextRunTime=2024-11-04 13:01:00.0} 2024-11-04 13:01:00.895 WARN 21062 --- [task-coordinator3,dd193f8f67a5a423,ed438d12b5fc8f65] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-11-04 13:01:00.895 WARN 21062 --- [task-coordinator3,dd193f8f67a5a423,ed438d12b5fc8f65] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-11-04 13:01:00.896 WARN 21062 --- [task-coordinator3,dd193f8f67a5a423,ed438d12b5fc8f65] c.o.o.c.t.e.c.worker.ReadyTaskWorker : failed to get template for class, action=com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule java.lang.NullPointerException: null at com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule.ready(CollectClusterSnapshotSchedule.java:74) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.getTaskMeta(ReadyTaskWorker.java:366) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.checkAndScheduleCronTask(ReadyTaskWorker.java:173) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.lambda$loopScheduleTask$1(ReadyTaskWorker.java:134) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.task.engine.coordinator.worker.AbstractTaskWorker.transactionWithoutResult(AbstractTaskWorker.java:61) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.loopScheduleTask(ReadyTaskWorker.java:133) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.work(ReadyTaskWorker.java:102) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lockAndExecute(TaskCoordinator.java:137) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$startup$0(TaskCoordinator.java:75) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:01:00.901 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ed438d12b5fc8f65] c.o.o.c.t.e.c.worker.SubtaskGenerator : Split list is empty, name=Collect cluster hour metric, splitKey=cluster_ids, context=Argument{parallelIdx=-1, stringMap={}, listMap={}} 2024-11-04 13:01:00.904 WARN 21062 --- [task-coordinator3,dd193f8f67a5a423,ed438d12b5fc8f65] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Subtask is empty, mark task failed, name=Collect cluster hour metric, taskId=6 2024-11-04 13:01:02.691 WARN 21062 --- [ocp-async-8,0b3e5da5561031ad,c394bfb42e66cee6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:02.725 INFO 21062 --- [ocp-async-6,b5789cebac43b93a,f728e015d296f95c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:02.983 INFO 21062 --- [prometheus-minute-schedule-1,af5c934887373417,37ff0bf333e38ab6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:03.008 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:03.009 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:04.691 WARN 21062 --- [ocp-async-5,ec0824ec974b3b05,80c025ce3c3d933f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:04.725 INFO 21062 --- [ocp-async-4,0a027d71d5f881af,9ee738adcd5ae0c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:05.467 INFO 21062 --- [ocp-schedules-14,dc4b2d4c2f887143,78e9f2866117dacf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:01:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2f7a316548597aa9,e8c6c69c43643a15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:06.009 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:06.010 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:06.691 WARN 21062 --- [ocp-async-10,02e9916aace55d40,a8ad4f7aef7ae421] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:06.725 INFO 21062 --- [ocp-async-1,0debc2a227c40dd0,56ad3db01c46f305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:08.505 INFO 21062 --- [ocp-schedules-19,71551fa4c0cb1ee0,92e46f525a4a94de] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:01:08.505 INFO 21062 --- [ocp-schedules-2,33fcff6c1e41fafd,5bd6c2201f85fca1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:01:08.564 INFO 21062 --- [ocp-async-1,0a97125aacecc24d,327884b79c83ebfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:01:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:01:08.588681+08:00, lastProcessTime=2024-11-04T13:00:58.588459+08:00 2024-11-04 13:01:08.691 WARN 21062 --- [ocp-async-8,6bd69e863c562e89,f8f4e8e1635ee65c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:08.725 INFO 21062 --- [ocp-async-3,320508a7e7896aa0,8bf70124551dcf2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:01:08.588681+08:00 2024-11-04 13:01:08.983 INFO 21062 --- [prometheus-minute-schedule-1,da214df8793cb9ab,c9e199086df59ea2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:08.996 INFO 21062 --- [ocp-schedules-19,b983a193b24b18f0,0379e787e5391953] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:01:09.010 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:09.011 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:09.314 INFO 21062 --- [ocp-schedules-8,2abc6645358f8374,3bdaac38c9005ef4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:01:09.321 INFO 21062 --- [ocp-schedules-8,2abc6645358f8374,3bdaac38c9005ef4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:01:09.321 INFO 21062 --- [ocp-schedules-8,2abc6645358f8374,3bdaac38c9005ef4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:01:10.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:01:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:01:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:01:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:01:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:01:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:01:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:01:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:01:10.691 WARN 21062 --- [ocp-async-5,667220e87bdaff56,e42bfaa96d108a66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:10.725 INFO 21062 --- [ocp-async-7,0727a37bce39e831,c9a7c4d0ee4c5004] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:11.149 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696400, nodeCount=28, affectRows=28 2024-11-04 13:01:11.149 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696400, nodeCount=59, affectRows=59 2024-11-04 13:01:11.149 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696400, nodeCount=22, affectRows=22 2024-11-04 13:01:11.149 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696400, nodeCount=34, affectRows=34 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696400, nodeCount=28, validCount=28 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696400, nodeCount=22, validCount=22 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696400, nodeCount=34, validCount=34 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696400, nodeCount=66, affectRows=66 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696400, nodeCount=48, affectRows=48 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696400, nodeCount=65, affectRows=65 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696400, nodeCount=58, validCount=58 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696400, nodeCount=64, validCount=64 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696400, nodeCount=48, validCount=48 2024-11-04 13:01:11.150 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696400, nodeCount=65, validCount=65 2024-11-04 13:01:11.983 INFO 21062 --- [prometheus-minute-schedule-1,10420ff344eb3cc7,2005f6afcf40fe27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:12.011 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:12.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:12.691 WARN 21062 --- [ocp-async-10,11d4adeb59007b23,d04969fd7bd72210] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:12.725 INFO 21062 --- [ocp-async-2,d0ddf4262017faa2,f657caffc787bfa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:13.984 INFO 21062 --- [prometheus-cache-monitor1,2e9c5a4b9bb27e1a,16c0817409adc57d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":638} 2024-11-04 13:01:13.984 INFO 21062 --- [prometheus-cache-monitor1,2e9c5a4b9bb27e1a,16c0817409adc57d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:01:14.292 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:57:14.272, end=2024-11-04T12:59:14.289} 2024-11-04 13:01:14.303 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:01:14.303 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:01:14.692 WARN 21062 --- [ocp-async-9,30de35e6be3be2b8,e44b0a49fa8455da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:14.726 INFO 21062 --- [ocp-async-8,b0af14c3a62c6234,49b876b858d5e5f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:14.983 INFO 21062 --- [prometheus-minute-schedule-1,c19be3fa2a8c25d3,5ccc9b34e38c56f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:15.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:15.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:15.494 INFO 21062 --- [ocp-schedules-8,44b2b8dbb26936e0,8b0bda5ee4b6a781] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:01:16.692 WARN 21062 --- [ocp-async-3,57349fe574cbd8eb,95371a66d2faa471] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:16.726 INFO 21062 --- [ocp-async-4,b464cc43f01628d5,f4af5f46d340d091] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4fc4f084493262c0,4c065e19e585f0c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:18.013 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:18.013 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:18.564 INFO 21062 --- [ocp-async-5,c70b7cb52f3315ba,39e20445a8833be2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:01:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:01:18.589080+08:00, lastProcessTime=2024-11-04T13:01:08.588681+08:00 2024-11-04 13:01:18.692 WARN 21062 --- [ocp-async-10,2f4571eca30b6352,d7d591cb580e83f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:18.726 INFO 21062 --- [ocp-async-1,e855dbe77c7fc212,f41df11f35f43bd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:18.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:01:18.589080+08:00 2024-11-04 13:01:18.998 INFO 21062 --- [ocp-schedules-7,e578553406724c7a,18e3add6691dc6f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:01:20.692 WARN 21062 --- [ocp-async-9,7f362ac2a9fdd197,1ad673543f006dd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:20.726 INFO 21062 --- [ocp-async-6,ebd54c19b39ac408,2592ab4489fb7e29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:20.983 INFO 21062 --- [prometheus-minute-schedule-1,26453b37d0340271,89a4c54a9a0a65ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:21.013 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:21.014 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:22.692 WARN 21062 --- [ocp-async-3,289efda60c794f86,7f4b2d6b77d5a9d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:22.727 INFO 21062 --- [ocp-async-4,915ad8183f06f915,e99a112eff15e764] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:23.983 INFO 21062 --- [prometheus-minute-schedule-1,0abd1ad907635576,65d53eb833c76292] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:24.014 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:24.015 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:24.692 WARN 21062 --- [ocp-async-7,a1f9dc18d86e42f6,0b8046eebb92ac62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:24.727 INFO 21062 --- [ocp-async-10,9fa10be39255eda2,0d9de6f88911413a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:25.575 INFO 21062 --- [ocp-schedules-6,256df9824bc0edca,69eabb3de66aaf77] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:01:26.693 WARN 21062 --- [ocp-async-1,7a10410c43b61062,72f51c7521c889a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:26.727 INFO 21062 --- [ocp-async-9,bda239abe77dde0a,16945e345619cab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:26.983 INFO 21062 --- [prometheus-minute-schedule-1,92008e046ae49074,2f87be9d3ff6f774] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e67f860c05b082dc,ed1ff08a4f7f5f7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:01:27.015 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:27.016 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:28.498 INFO 21062 --- [ocp-schedules-20,822e0092ae4cb7fe,dd87652117cf77ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:01:28.499 INFO 21062 --- [ocp-schedules-25,fad08426a5c7b20a,6acc6fcb64420ffb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:01:28.500 INFO 21062 --- [ocp-schedules-17,b2e20cef858d2b98,709e27b9e893c82d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:01:28.500 INFO 21062 --- [ocp-schedules-16,3466b5253b3084a5,23745144e2118c53] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:01:28.502 INFO 21062 --- [ocp-schedules-20,822e0092ae4cb7fe,dd87652117cf77ad] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:01:28.502 INFO 21062 --- [ocp-schedules-18,8109c534efa97237,08711c7c852aef24] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:01:28.503 INFO 21062 --- [ocp-schedules-17,b2e20cef858d2b98,709e27b9e893c82d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:01:28.504 INFO 21062 --- [ocp-schedules-16,3466b5253b3084a5,23745144e2118c53] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:28.504 INFO 21062 --- [ocp-schedules-4,32570d528a7e863d,059c5c6d6fbbdfde] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:01:28.505 INFO 21062 --- [ocp-schedules-25,fad08426a5c7b20a,6acc6fcb64420ffb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:01:28.511 INFO 21062 --- [ocp-schedules-28,aefe4bb1b8fe1489,23e3ebfe4c5cd8b9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:01:28.511 INFO 21062 --- [ocp-schedules-28,aefe4bb1b8fe1489,23e3ebfe4c5cd8b9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:01:28.511 INFO 21062 --- [ocp-schedules-26,2aebc0208805be7b,73f03198a2decac9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:01:28.512 INFO 21062 --- [ocp-schedules-26,2aebc0208805be7b,73f03198a2decac9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:01:28.512 INFO 21062 --- [ocp-schedules-11,e6a9bf390b4804c1,380e366bbef87fad] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:01:28.512 INFO 21062 --- [ocp-schedules-11,e6a9bf390b4804c1,380e366bbef87fad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:01:28.564 INFO 21062 --- [ocp-async-6,f044537c885ed149,fdd3c26e25e53cd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:01:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:01:28.588801+08:00, lastProcessTime=2024-11-04T13:01:18.589080+08:00 2024-11-04 13:01:28.628 INFO 21062 --- [ocp-schedules-2,049f512ae90cd283,af0177afab74ac6a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:01:28.661 INFO 21062 --- [ocp-schedules-23,ff716fed4fdc00eb,f5c38024091b0a5e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:01:28.665 INFO 21062 --- [ocp-schedules-23,ff716fed4fdc00eb,f5c38024091b0a5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:28.666 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:01:28.685 INFO 21062 --- [ocp-schedules-15,9a75219d7fa874ae,5a20637f3a3efbb6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:01:28.685 INFO 21062 --- [ocp-schedules-15,9a75219d7fa874ae,5a20637f3a3efbb6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:01:28.693 WARN 21062 --- [ocp-async-4,fcad2d3b57c6a0a8,6fbe42cd666c6d73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:28.698 INFO 21062 --- [ocp-schedules-22,d3a21b5c7729f012,6c843d2dabc8c5ae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:01:28.717 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:01:28.717 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:01:28.724 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:01:28.724 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:01:28.724 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:01:28.727 INFO 21062 --- [ocp-async-7,7c2715f9acec9788,59311cc769d2b300] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:28.735 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:01:28.735 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:01:28.735 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:01:28.738 INFO 21062 --- [ocp-schedules-7,278e8279a17d66e5,151325fe1bd02799] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:01:28.746 INFO 21062 --- [ocp-schedules-28,d1da34ea2de8285e,f15693a7cc84ced0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:01:28.751 INFO 21062 --- [ocp-schedules-28,d1da34ea2de8285e,f15693a7cc84ced0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:01:28.751 INFO 21062 --- [ocp-schedules-28,d1da34ea2de8285e,f15693a7cc84ced0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:01:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:01:28.588801+08:00 2024-11-04 13:01:28.883 INFO 21062 --- [ocp-schedules-11,6d6ecc5e95225546,452525966cf81ea6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:01:28.883 INFO 21062 --- [ocp-schedules-11,6d6ecc5e95225546,452525966cf81ea6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:01:28.885 INFO 21062 --- [ocp-schedules-11,6d6ecc5e95225546,452525966cf81ea6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:01:28.885 INFO 21062 --- [ocp-schedules-11,6d6ecc5e95225546,452525966cf81ea6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:01:28.886 INFO 21062 --- [ocp-schedules-11,6d6ecc5e95225546,452525966cf81ea6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:01:28.886 INFO 21062 --- [ocp-schedules-11,6d6ecc5e95225546,452525966cf81ea6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:01:29.000 INFO 21062 --- [ocp-schedules-30,01012a201dcca28b,851a11d0212b9f39] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:01:29.319 INFO 21062 --- [ocp-schedules-29,1a44e57255272bd8,127b0b807e62f8bf] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:01:29.319 INFO 21062 --- [ocp-schedules-29,1a44e57255272bd8,127b0b807e62f8bf] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:01:29.323 INFO 21062 --- [ocp-schedules-6,1652248b07f356a8,51dfc957f09fdb47] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:01:29.329 INFO 21062 --- [ocp-schedules-6,1652248b07f356a8,51dfc957f09fdb47] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:01:29.329 INFO 21062 --- [ocp-schedules-6,1652248b07f356a8,51dfc957f09fdb47] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:01:29.346 INFO 21062 --- [ocp-schedules-29,1a44e57255272bd8,127b0b807e62f8bf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:01:29.346 INFO 21062 --- [ocp-schedules-29,1a44e57255272bd8,127b0b807e62f8bf] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:01:29.347 INFO 21062 --- [ocp-schedules-8,a4df01f3eb4b4631,54cd24cbd624ebb1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:01:29.396 INFO 21062 --- [ocp-schedules-13,43acbfb849b9a1ab,3f935948d32c2193] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:01:29.396 INFO 21062 --- [ocp-schedules-13,43acbfb849b9a1ab,3f935948d32c2193] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:01:29.396 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:01:29.396 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:01:29.406 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:01:29.406 INFO 21062 --- [ocp-schedules-13,43acbfb849b9a1ab,3f935948d32c2193] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:01:29.406 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:01:29.408 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:01:29.408 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:01:29.408 WARN 21062 --- [ocp-schedules-13,43acbfb849b9a1ab,3f935948d32c2193] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:01:29.409 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:01:29.409 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:01:29.409 INFO 21062 --- [ocp-schedules-13,43acbfb849b9a1ab,3f935948d32c2193] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:01:29.410 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:01:29.410 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:01:29.410 WARN 21062 --- [ocp-schedules-13,43acbfb849b9a1ab,3f935948d32c2193] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:01:29.417 ERROR 21062 --- [ocp-schedules-13,43acbfb849b9a1ab,3f935948d32c2193] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:01:29.461 INFO 21062 --- [ocp-schedules-15,9e4c81f62492c6a6,4f09dead64872430] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:01:29.461+08:00, clusters=[1] 2024-11-04 13:01:29.463 INFO 21062 --- [ocp-schedules-15,9e4c81f62492c6a6,4f09dead64872430] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:01:29.553 INFO 21062 --- [ocp-schedules-19,7527094f9042a063,78960b6679df4d76] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:01:29.553+08:00, clusters=[1] 2024-11-04 13:01:29.557 INFO 21062 --- [ocp-schedules-19,7527094f9042a063,78960b6679df4d76] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:01:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e69aed2e46d601ff,44298d55f98b8d57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:30.016 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:30.017 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:01:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:01:30.492 INFO 21062 --- [ocp-schedules-10,13f77fb25b3722e0,b70a8c4d91f94474] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:01:30.492 INFO 21062 --- [ocp-schedules-10,a8db816bef318a30,00c9bfcb49f1075e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:01:30.496 INFO 21062 --- [ocp-schedules-10,a8db816bef318a30,00c9bfcb49f1075e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:30.525 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:01:30.525 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:01:30.693 WARN 21062 --- [ocp-async-7,d272586d5b107d0a,3a7c0cf7783787e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:30.727 INFO 21062 --- [ocp-async-1,4b06e3c9ac7a5e42,59ca2349b8cb6545] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:31.154 INFO 21062 --- [ocp-schedules-14,fe1c8ae7f267c2ae,1b8ae954360210a3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:01:31.158 INFO 21062 --- [ocp-schedules-14,fe1c8ae7f267c2ae,1b8ae954360210a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:31.246 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:01:31.252 ERROR 21062 --- [ocp-schedules-14,fe1c8ae7f267c2ae,1b8ae954360210a3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:01:31.252 INFO 21062 --- [ocp-schedules-14,fe1c8ae7f267c2ae,1b8ae954360210a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:01:32.693 WARN 21062 --- [ocp-async-1,a41086a8f13f48fe,60d636d5a2e0b262] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:32.728 INFO 21062 --- [ocp-async-6,c3fc73d214f02225,42b9820232909861] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:32.983 INFO 21062 --- [prometheus-minute-schedule-1,ea2e462883177dd0,5c32d4ac65203ecd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:33.017 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:33.018 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:34.693 WARN 21062 --- [ocp-async-6,cc841d6ca4861beb,19e176a52d7d0c43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:34.728 INFO 21062 --- [ocp-async-4,deacf43a278ffe68,1f2efd97e990f1da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:35.601 INFO 21062 --- [ocp-schedules-9,33324d8870312124,e26808dd95d2d78a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:01:35.983 INFO 21062 --- [prometheus-minute-schedule-1,28eb8f3bc6123bbf,0f3c3eaf74ece1a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:36.018 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:36.019 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:36.693 WARN 21062 --- [ocp-async-4,e349316908cef613,520ad7eadb6083fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:36.728 INFO 21062 --- [ocp-async-10,8b68988b472c23a4,159612b4f251de9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:38.504 INFO 21062 --- [ocp-schedules-17,b2e20cef858d2b98,709e27b9e893c82d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:01:38.505 INFO 21062 --- [ocp-schedules-16,3466b5253b3084a5,23745144e2118c53] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:01:38.564 INFO 21062 --- [ocp-async-10,c472b5661b0ac85a,4307f64437c7dfcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:01:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:01:38.589600+08:00, lastProcessTime=2024-11-04T13:01:28.588801+08:00 2024-11-04 13:01:38.666 INFO 21062 --- [ocp-schedules-23,ff716fed4fdc00eb,f5c38024091b0a5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:01:38.694 WARN 21062 --- [ocp-async-8,12a6d0692a65538d,199162b151020468] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:38.728 INFO 21062 --- [ocp-async-6,14064e3f1662b840,8a965c449854aa9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:01:38.589600+08:00 2024-11-04 13:01:38.983 INFO 21062 --- [prometheus-minute-schedule-1,9ab296601e70e9fe,95406d47775fe47a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:39.003 INFO 21062 --- [ocp-schedules-6,f337990f28b7d8b7,64bbf1e1baa86748] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:01:39.019 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:39.020 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:01:40.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:01:40.694 WARN 21062 --- [ocp-async-6,3744efe84df6ffba,fd84484c41ce8f6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:40.728 INFO 21062 --- [ocp-async-7,3916559e7e2ee435,b837d98c28110240] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:41.983 INFO 21062 --- [prometheus-minute-schedule-1,3939b3bd7fba5664,0600c88f6bd0f709] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:42.020 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:42.021 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:42.694 WARN 21062 --- [ocp-async-7,7bd25efc51dd31aa,4a5eee84dbe5de84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:42.728 INFO 21062 --- [ocp-async-10,d2fb7258fb5f81c8,a8adb3ba15b6a688] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:44.694 WARN 21062 --- [ocp-async-10,564d99b4d662fc8c,daa932061512b034] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:44.728 INFO 21062 --- [ocp-async-3,ac96b69308070eba,422c47e7c7cc725e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e6ea3dbbbe0d0281,61eac2a57f38a74d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:45.021 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:45.022 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:45.642 INFO 21062 --- [ocp-schedules-20,c2e77f3e9fa93f0e,7cb961e5c8f11966] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:01:46.694 WARN 21062 --- [ocp-async-3,872fda2ee69faf87,305f14429c97fe4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:46.728 INFO 21062 --- [ocp-async-2,019d281eb7ab83f9,db23468f424bcd31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:47.983 INFO 21062 --- [prometheus-minute-schedule-1,3e624d1d72e20532,2fc052b81f4789da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:48.022 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:48.023 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:48.565 INFO 21062 --- [ocp-async-4,9d42888f2b865c98,a18b4a3f0cb0dced] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:01:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:01:48.590056+08:00, lastProcessTime=2024-11-04T13:01:38.589600+08:00 2024-11-04 13:01:48.695 WARN 21062 --- [ocp-async-7,4e8b2ba118361bac,fa30e0a1823e7f74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:48.729 INFO 21062 --- [ocp-async-1,6c0c2128640a10c1,73a0acdbb86dce6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:01:48.590056+08:00 2024-11-04 13:01:49.006 INFO 21062 --- [ocp-schedules-28,1736bba909ba7c25,e71993acbbc18371] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:01:49.331 INFO 21062 --- [ocp-schedules-1,c943225806b60123,417eddfd5f040a54] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:01:49.339 INFO 21062 --- [ocp-schedules-1,c943225806b60123,417eddfd5f040a54] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:01:49.339 INFO 21062 --- [ocp-schedules-1,c943225806b60123,417eddfd5f040a54] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:01:50.695 WARN 21062 --- [ocp-async-1,aa444d63aeb0b965,d32d723e8c26fead] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:50.729 INFO 21062 --- [ocp-async-8,49e5494d14dfd4fc,8c3a6363e8d40958] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a1207e99027359a6,6e65e51d50ec21eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:51.023 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:51.024 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:52.695 WARN 21062 --- [ocp-async-8,9a209780fde964dd,dab10c019953db8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:52.729 INFO 21062 --- [ocp-async-6,fc0de023fae867dc,062c353209d337a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:53.983 INFO 21062 --- [prometheus-minute-schedule-1,099550ceebbfdce5,090c5be38e8c8884] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:54.024 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:54.025 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:54.695 WARN 21062 --- [ocp-async-6,319cf6fdb967129e,539de6d4ac6fbcbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:54.729 INFO 21062 --- [ocp-async-7,fb094df028b26ecf,96f1306cb8307ded] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:55.669 INFO 21062 --- [ocp-schedules-16,521fe43b9ec5fae9,82e08fae7785485f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:01:56.695 WARN 21062 --- [ocp-async-7,491aa9771c3816db,397af3ab90fd88d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:56.729 INFO 21062 --- [ocp-async-1,e787a5cebb6911a7,c03dc98802a743b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:56.983 INFO 21062 --- [prometheus-minute-schedule-1,9281d039b4b967ef,5405d46feb167434] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:01:57.025 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:01:57.026 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:01:58.500 INFO 21062 --- [ocp-schedules-24,be686479bb63258e,7e4aa5329ff15145] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:01:58.500 INFO 21062 --- [ocp-schedules-27,7d787ab139c5bbf6,67d1cce25871504b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:01:58.502 INFO 21062 --- [ocp-schedules-13,f97ace95e8c8b518,76de091e7da2f93c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:01:58.504 INFO 21062 --- [ocp-schedules-24,be686479bb63258e,7e4aa5329ff15145] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:58.504 INFO 21062 --- [ocp-schedules-27,7d787ab139c5bbf6,67d1cce25871504b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:01:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:01:58.504 INFO 21062 --- [ocp-schedules-20,b92779347e220dac,7066f24ad10fa24a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:01:58.512 INFO 21062 --- [ocp-schedules-16,69791b8354fb9601,fd8fd1ab4266e742] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:01:58.512 INFO 21062 --- [ocp-schedules-7,47a6635894496433,35a2a806ef7da50e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:01:58.512 INFO 21062 --- [ocp-schedules-16,69791b8354fb9601,fd8fd1ab4266e742] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:01:58.512 INFO 21062 --- [ocp-schedules-7,47a6635894496433,35a2a806ef7da50e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:01:58.512 INFO 21062 --- [ocp-schedules-14,97c69475ef58e04c,acdadd482e05c54f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:01:58.512 INFO 21062 --- [ocp-schedules-14,97c69475ef58e04c,acdadd482e05c54f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:01:58.565 INFO 21062 --- [ocp-async-5,588d3ce8f55ac0c2,d37c7a1b8fdedb9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:01:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:01:58.589470+08:00, lastProcessTime=2024-11-04T13:01:48.590056+08:00 2024-11-04 13:01:58.695 WARN 21062 --- [ocp-async-8,76670fdd3a4b5bb5,672776d2aa17b2d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:01:58.700 INFO 21062 --- [ocp-schedules-19,5187e9616864dc6e,9d417d9090ef032d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:01:58.730 INFO 21062 --- [ocp-async-4,e8386671a1ad417e,506bdbc76af530a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:01:58.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:01:58.589470+08:00 2024-11-04 13:01:59.008 INFO 21062 --- [ocp-schedules-6,8e519b5168ee8645,7c3f54a101f16bf8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:01:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ff9140b14b5bb3df,857d87a2a499d151] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696520, interval=60 2024-11-04 13:02:00.026 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:00.027 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:00.525 INFO 21062 --- [ocp-schedules-10,a8db816bef318a30,00c9bfcb49f1075e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:02:00.525 INFO 21062 --- [ocp-schedules-10,78bf22c8ab6034d9,547da73b505e77bd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:02:00.530 INFO 21062 --- [ocp-schedules-10,78bf22c8ab6034d9,547da73b505e77bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:00.560 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:02:00.560 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:02:00.696 WARN 21062 --- [ocp-async-6,e592e49ebc41be54,fab9f60e78f9186e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:00.730 INFO 21062 --- [ocp-async-9,71c0eddb05dbc32c,a7c46449d112d390] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:02.696 WARN 21062 --- [ocp-async-7,cdda9701488d37fc,7227539ad15c5867] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:02.730 INFO 21062 --- [ocp-async-3,c2b2fa9ffba875af,e2186a859412ec56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a9a5f13b8881f1b9,e33280fcc3364d92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:03.027 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:03.028 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:04.696 WARN 21062 --- [ocp-async-1,07532828da1f7e94,aa79ef37c19ea26c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:04.730 INFO 21062 --- [ocp-async-8,6cee595bfae41865,91b9638c7091f24c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:05.708 INFO 21062 --- [ocp-schedules-6,f4ae948158cd402b,10369f8761899241] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:02:05.983 INFO 21062 --- [prometheus-minute-schedule-1,9c8685b3efcd890a,be898a72d0667562] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:06.028 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:06.029 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:06.696 WARN 21062 --- [ocp-async-4,84f6dbe802d3782d,dfcd09af5238043c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:06.730 INFO 21062 --- [ocp-async-10,69f2c77ecda38423,2ab0ec052e970460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:08.505 INFO 21062 --- [ocp-schedules-27,7d787ab139c5bbf6,67d1cce25871504b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:02:08.505 INFO 21062 --- [ocp-schedules-24,be686479bb63258e,7e4aa5329ff15145] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:02:08.565 INFO 21062 --- [ocp-async-10,78293e3164eb9273,b58e00834716717e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:02:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:02:08.588891+08:00, lastProcessTime=2024-11-04T13:01:58.589470+08:00 2024-11-04 13:02:08.696 WARN 21062 --- [ocp-async-7,b799ce8b596d6289,cc8bf5908917b14d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:08.731 INFO 21062 --- [ocp-async-3,65b27e4a8d343f4b,9dbbd02d183be5dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:02:08.588891+08:00 2024-11-04 13:02:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ca86a77a7f782f86,eef0c14afc7ce2b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:09.011 INFO 21062 --- [ocp-schedules-11,574ed9d8db568c0a,59ba45c764a4e52b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:02:09.029 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:09.030 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:09.342 INFO 21062 --- [ocp-schedules-21,fc93248035e1e66e,ae0e07779cd48283] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:02:09.348 INFO 21062 --- [ocp-schedules-21,fc93248035e1e66e,ae0e07779cd48283] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:02:09.348 INFO 21062 --- [ocp-schedules-21,fc93248035e1e66e,ae0e07779cd48283] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:02:10.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:02:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:02:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:02:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:02:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:02:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:02:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:02:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:02:10.696 WARN 21062 --- [ocp-async-1,fb6e546df069b461,616ff8b55b0ee29f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:10.731 INFO 21062 --- [ocp-async-8,fc8760fe7cd2a052,50c196b3e2e997c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:11.154 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696460, nodeCount=1, affectRows=1 2024-11-04 13:02:11.154 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696460, nodeCount=1, validCount=1 2024-11-04 13:02:11.155 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696460, nodeCount=24, affectRows=24 2024-11-04 13:02:11.155 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696460, nodeCount=14, affectRows=14 2024-11-04 13:02:11.155 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696460, nodeCount=14, validCount=14 2024-11-04 13:02:11.155 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696460, nodeCount=24, validCount=24 2024-11-04 13:02:11.156 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696460, nodeCount=44, affectRows=44 2024-11-04 13:02:11.156 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696460, nodeCount=44, validCount=44 2024-11-04 13:02:11.156 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696460, nodeCount=100, affectRows=100 2024-11-04 13:02:11.157 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696460, nodeCount=99, validCount=99 2024-11-04 13:02:11.158 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696460, nodeCount=139, affectRows=139 2024-11-04 13:02:11.158 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696460, nodeCount=137, validCount=137 2024-11-04 13:02:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9e1fa79023934abe,3f634f97cd7b773c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:12.030 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:12.031 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:12.697 WARN 21062 --- [ocp-async-4,eb5e4cb13efd25e0,58477a1afbb18717] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:12.731 INFO 21062 --- [ocp-async-10,2a0149104428151c,f74a99deee81c93a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:13.985 INFO 21062 --- [prometheus-cache-monitor1,4c9c02987e20a4a9,3fecaa223649d5f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":638} 2024-11-04 13:02:13.985 INFO 21062 --- [prometheus-cache-monitor1,4c9c02987e20a4a9,3fecaa223649d5f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:02:14.697 WARN 21062 --- [ocp-async-9,22d6f5fcf1a00289,cc5adab18d6e1830] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:14.731 INFO 21062 --- [ocp-async-5,9378e0286079ea22,8c37166c144e57b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5324e710796cef40,99bf49cdbe4e36a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:15.032 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:15.032 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:15.735 INFO 21062 --- [ocp-schedules-2,d159451200c2444f,0158fe578f001077] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:02:16.697 WARN 21062 --- [ocp-async-3,e61741a4ca985e1d,e798e2d520df47b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:16.731 INFO 21062 --- [ocp-async-2,f19425c63ddb67f5,174ead8c7339bab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7126dfacc3a0b247,685cb928671cbfe1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:18.033 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:18.034 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:18.565 INFO 21062 --- [ocp-async-1,a6cc5649f05cc5dc,324ab3517d0ea833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:02:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:02:18.589460+08:00, lastProcessTime=2024-11-04T13:02:08.588891+08:00 2024-11-04 13:02:18.697 WARN 21062 --- [ocp-async-4,eff5452578da6665,365312777c0023cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:18.732 INFO 21062 --- [ocp-async-10,94f613a74ffba7c0,35edfa44c83e416b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:02:18.589460+08:00 2024-11-04 13:02:19.015 INFO 21062 --- [ocp-schedules-29,433f9dcbeb9da93f,c019ea30bf1f737c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:02:20.697 WARN 21062 --- [ocp-async-9,fa7280429ada7fca,ebe7ae4a31694b74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:20.732 INFO 21062 --- [ocp-async-5,442740e133b861dd,d630e4d2d7b9b57d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4266f84aeb53adb1,3331744aa3072807] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:21.034 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:21.035 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:22.697 WARN 21062 --- [ocp-async-3,02ae131b6b7f0500,8d3496c7f2f50030] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:22.732 INFO 21062 --- [ocp-async-2,93c98c4c4458286e,3b5770ea691c59a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:23.983 INFO 21062 --- [prometheus-minute-schedule-1,d09b8a5d503f8523,6dcbd84bc74ea729] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:24.035 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:24.035 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:24.698 WARN 21062 --- [ocp-async-8,fcbad701f41cba09,d7e28d1aea7e0656] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:24.732 INFO 21062 --- [ocp-async-6,fc6504f9cfe21506,8ae83d03f9ca6ece] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:25.773 INFO 21062 --- [ocp-schedules-20,8903115055a04811,2e0e09f53f8a755c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:02:26.698 WARN 21062 --- [ocp-async-10,8520d485b481525f,cbb9099109798468] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:26.732 INFO 21062 --- [ocp-async-7,d09efe9ad3e605da,a1422c0ebf4942e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:26.983 INFO 21062 --- [prometheus-minute-schedule-1,d51bb5f2324145fe,cc48284acaabcb49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:26.984 INFO 21062 --- [prometheus-hour-schedule-1,707e98ee5363d13e,d95e9be4a517fa89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:02:27.036 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:27.036 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:28.498 INFO 21062 --- [ocp-schedules-29,8ba154fc40f40480,d52167fec1bfe41a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:02:28.499 INFO 21062 --- [ocp-schedules-4,bcd27fe9003b667f,f85febbdf2c937f1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:02:28.499 INFO 21062 --- [ocp-schedules-21,45f8093095976cd3,edc542165e7ac5f5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:02:28.500 INFO 21062 --- [ocp-schedules-27,83ab8f237474ddbb,d193dd1ab383f3e0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:02:28.502 INFO 21062 --- [ocp-schedules-29,8ba154fc40f40480,d52167fec1bfe41a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:02:28.502 INFO 21062 --- [ocp-schedules-25,feca0ccaf6f1f86d,ce8cda0e18d439d7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:02:28.503 INFO 21062 --- [ocp-schedules-21,45f8093095976cd3,edc542165e7ac5f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:28.504 INFO 21062 --- [ocp-schedules-4,bcd27fe9003b667f,f85febbdf2c937f1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:02:28.504 INFO 21062 --- [ocp-schedules-27,83ab8f237474ddbb,d193dd1ab383f3e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:28.504 INFO 21062 --- [ocp-schedules-26,e310eb237e2c2080,93f9598fb913d00a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:02:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:02:28.505 INFO 21062 --- [ocp-schedules-22,659ac5443a702353,c04ba2fdc75759a9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:02:28.508 INFO 21062 --- [ocp-schedules-22,659ac5443a702353,c04ba2fdc75759a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:28.510 INFO 21062 --- [ocp-schedules-6,e33e739748cf1a3c,a3e92af78b28fb29] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:02:28.510 INFO 21062 --- [ocp-schedules-18,96dc9beadc267a3d,21f86514d33b128c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:02:28.510 INFO 21062 --- [ocp-schedules-16,70a8fd16cbfc82fa,335d2354609b7247] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:02:28.510 INFO 21062 --- [ocp-schedules-6,e33e739748cf1a3c,a3e92af78b28fb29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:02:28.510 INFO 21062 --- [ocp-schedules-18,96dc9beadc267a3d,21f86514d33b128c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:02:28.510 INFO 21062 --- [ocp-schedules-16,70a8fd16cbfc82fa,335d2354609b7247] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:02:28.529 INFO 21062 --- [ocp-schedules-15,3c818ba65b4ccf26,2cccdb140c603ed0] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:02:28.529 INFO 21062 --- [ocp-schedules-15,3c818ba65b4ccf26,2cccdb140c603ed0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:02:28.558 INFO 21062 --- [ocp-schedules-30,8409df413a1e227d,bca91f28a48a168e] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:02:28.560 INFO 21062 --- [ocp-schedules-12,fa2af68d9d19101b,9a186e59426042f2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":638} 2024-11-04 13:02:28.560 INFO 21062 --- [ocp-schedules-12,fa2af68d9d19101b,9a186e59426042f2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:02:28.561 INFO 21062 --- [ocp-schedules-30,8409df413a1e227d,bca91f28a48a168e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:02:28.565 INFO 21062 --- [ocp-async-5,bcec2dfbbcb899ff,cdd49054bb107dfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:02:28.573 INFO 21062 --- [pool-12-thread-9,3d807fb3b50b7792,c2dd616af7d2ef34] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:02:28.573 INFO 21062 --- [pool-12-thread-11,3d807fb3b50b7792,a210dd8eb6f1fd19] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:02:28.574 INFO 21062 --- [pool-12-thread-11,3d807fb3b50b7792,a210dd8eb6f1fd19] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:02:28.574 INFO 21062 --- [pool-12-thread-12,3d807fb3b50b7792,ffbe7854710dde39] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:02:28.575 INFO 21062 --- [pool-12-thread-10,3d807fb3b50b7792,288044ed8998bcf0] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:02:28.577 INFO 21062 --- [pool-12-thread-11,3d807fb3b50b7792,a210dd8eb6f1fd19] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:02:28.580 INFO 21062 --- [ForkJoinPool-11-worker-1,3d807fb3b50b7792,22422ba5f3c83cc0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:02:28.580 INFO 21062 --- [ForkJoinPool-11-worker-1,3d807fb3b50b7792,22422ba5f3c83cc0] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:02:28.580 INFO 21062 --- [ForkJoinPool-11-worker-1,3d807fb3b50b7792,22422ba5f3c83cc0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:02:28.580 INFO 21062 --- [ForkJoinPool-11-worker-8,3d807fb3b50b7792,49a2f7860c335f3e] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:02:28.580 INFO 21062 --- [pool-12-thread-9,3d807fb3b50b7792,c2dd616af7d2ef34] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:02:28.580 INFO 21062 --- [pool-12-thread-12,3d807fb3b50b7792,ffbe7854710dde39] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:02:28.581 INFO 21062 --- [pool-12-thread-11,3d807fb3b50b7792,a210dd8eb6f1fd19] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:02:28.581 INFO 21062 --- [pool-12-thread-10,3d807fb3b50b7792,288044ed8998bcf0] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:02:28.581 INFO 21062 --- [pool-12-thread-10,3d807fb3b50b7792,288044ed8998bcf0] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:02:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:02:28.587850+08:00, lastProcessTime=2024-11-04T13:02:18.589460+08:00 2024-11-04 13:02:28.623 INFO 21062 --- [ocp-schedules-4,cae22f704f93e6e3,183db27901273a5b] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 13:02:28.623 INFO 21062 --- [ocp-schedules-4,cae22f704f93e6e3,183db27901273a5b] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 13:02:28.623 INFO 21062 --- [ocp-schedules-4,cae22f704f93e6e3,183db27901273a5b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=322 2024-11-04 13:02:28.632 INFO 21062 --- [ocp-schedules-26,4a3f0be705b03bab,287e9afc530d2d84] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:02:28.637 INFO 21062 --- [ocp-schedules-25,f129e6d482497f84,210c36219c16ffdd] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:02:28.649 INFO 21062 --- [ocp-schedules-4,cae22f704f93e6e3,183db27901273a5b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=322 2024-11-04 13:02:28.650 INFO 21062 --- [ocp-schedules-25,f129e6d482497f84,210c36219c16ffdd] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:02:28.688 INFO 21062 --- [ocp-schedules-6,7ba6efa77759169a,5862d44a89905005] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:02:28.688 INFO 21062 --- [ocp-schedules-6,7ba6efa77759169a,5862d44a89905005] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:02:28.698 WARN 21062 --- [ocp-async-1,2ac36bab89665307,50c1374c77a5ee29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:28.702 INFO 21062 --- [ocp-schedules-18,b510e1a5776866c5,30e31baebd63d4d8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:02:28.732 INFO 21062 --- [ocp-async-4,5fb04e268606cdc2,fb2b46e874fcc80f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:28.740 INFO 21062 --- [ocp-schedules-15,8335dba2b8b93caa,4ecc2fe69e650bec] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 13:02:28.743 INFO 21062 --- [ocp-schedules-15,8335dba2b8b93caa,4ecc2fe69e650bec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:28.751 INFO 21062 --- [ocp-schedules-13,2fe8dcc1a2bbddac,5e92af5738af3516] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:02:28.756 INFO 21062 --- [ocp-schedules-13,2fe8dcc1a2bbddac,5e92af5738af3516] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:02:28.756 INFO 21062 --- [ocp-schedules-13,2fe8dcc1a2bbddac,5e92af5738af3516] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:02:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:02:28.587850+08:00 2024-11-04 13:02:28.778 ERROR 21062 --- [ocp-schedules-15,8335dba2b8b93caa,4ecc2fe69e650bec] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:02:28.778 INFO 21062 --- [ocp-schedules-15,8335dba2b8b93caa,4ecc2fe69e650bec] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 13:02:28.843 INFO 21062 --- [ocp-schedules-2,be8a499342e1f241,1e4253d2e1dd5af9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:02:28.843 INFO 21062 --- [ocp-schedules-2,be8a499342e1f241,1e4253d2e1dd5af9] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:02:28.870 INFO 21062 --- [pool-25-thread-1,be8a499342e1f241,bfa9246018827cc5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:02:28.889 INFO 21062 --- [ocp-schedules-9,6df128acbcf74088,a9c6716fb9ae0bd3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:02:28.889 INFO 21062 --- [ocp-schedules-9,6df128acbcf74088,a9c6716fb9ae0bd3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:02:28.889 INFO 21062 --- [ocp-schedules-9,6df128acbcf74088,a9c6716fb9ae0bd3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:02:28.889 INFO 21062 --- [ocp-schedules-9,6df128acbcf74088,a9c6716fb9ae0bd3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:02:28.890 INFO 21062 --- [ocp-schedules-9,6df128acbcf74088,a9c6716fb9ae0bd3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:02:28.890 INFO 21062 --- [ocp-schedules-9,6df128acbcf74088,a9c6716fb9ae0bd3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:02:28.898 WARN 21062 --- [ocp-schedules-2,be8a499342e1f241,1e4253d2e1dd5af9] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:02:28.898 INFO 21062 --- [ocp-schedules-2,be8a499342e1f241,1e4253d2e1dd5af9] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:02:29.017 INFO 21062 --- [ocp-schedules-7,227d81db25a840b2,4a2a4c31a31c6ad9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:02:29.346 INFO 21062 --- [ocp-schedules-20,f8ef2e895db33f37,d51f28e859191703] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:02:29.346 INFO 21062 --- [ocp-schedules-20,f8ef2e895db33f37,d51f28e859191703] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:02:29.351 INFO 21062 --- [ocp-schedules-4,48a6ca5ed493eae5,35701f37c4765a0a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:02:29.355 INFO 21062 --- [ocp-schedules-4,48a6ca5ed493eae5,35701f37c4765a0a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:02:29.355 INFO 21062 --- [ocp-schedules-4,48a6ca5ed493eae5,35701f37c4765a0a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:02:29.373 INFO 21062 --- [ocp-schedules-20,f8ef2e895db33f37,d51f28e859191703] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:02:29.373 INFO 21062 --- [ocp-schedules-20,f8ef2e895db33f37,d51f28e859191703] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:02:29.373 INFO 21062 --- [ocp-schedules-5,a6357c33320b8826,bb04875ec5d47732] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:02:29.418 INFO 21062 --- [ocp-schedules-29,fec85aca0bd0ad34,76b853b9889f5e6c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:02:29.422 INFO 21062 --- [ocp-schedules-29,fec85aca0bd0ad34,76b853b9889f5e6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:29.425 INFO 21062 --- [ocp-schedules-6,2cfb817db477e4dc,49ed4d13e7864473] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:02:29.425 INFO 21062 --- [ocp-schedules-6,2cfb817db477e4dc,49ed4d13e7864473] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:02:29.425 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:02:29.425 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:02:29.435 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:02:29.435 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:02:29.435 INFO 21062 --- [ocp-schedules-6,2cfb817db477e4dc,49ed4d13e7864473] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:02:29.437 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:02:29.437 WARN 21062 --- [ocp-schedules-6,2cfb817db477e4dc,49ed4d13e7864473] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:02:29.437 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:02:29.438 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:02:29.438 INFO 21062 --- [ocp-schedules-6,2cfb817db477e4dc,49ed4d13e7864473] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:02:29.438 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:02:29.440 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:02:29.440 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:02:29.440 WARN 21062 --- [ocp-schedules-6,2cfb817db477e4dc,49ed4d13e7864473] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:02:29.446 ERROR 21062 --- [ocp-schedules-6,2cfb817db477e4dc,49ed4d13e7864473] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:02:29.478 ERROR 21062 --- [ocp-schedules-29,fec85aca0bd0ad34,76b853b9889f5e6c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:02:29.478 INFO 21062 --- [ocp-schedules-29,fec85aca0bd0ad34,76b853b9889f5e6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:02:29.489 INFO 21062 --- [ocp-schedules-16,a391f4ad75ed763e,302e26ce35402b76] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:02:29.489+08:00, clusters=[1] 2024-11-04 13:02:29.490 INFO 21062 --- [ocp-schedules-16,a391f4ad75ed763e,302e26ce35402b76] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:02:29.581 INFO 21062 --- [ocp-schedules-14,a02715174eb06a52,0a01bb3bcdfe3b67] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:02:29.581+08:00, clusters=[1] 2024-11-04 13:02:29.585 INFO 21062 --- [ocp-schedules-14,a02715174eb06a52,0a01bb3bcdfe3b67] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:02:29.983 INFO 21062 --- [prometheus-minute-schedule-1,40a920f7dd3b2258,18e857e2ba528ec2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:30.037 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:30.037 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:02:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:02:30.561 INFO 21062 --- [ocp-schedules-10,78bf22c8ab6034d9,547da73b505e77bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:02:30.562 INFO 21062 --- [ocp-schedules-10,4d964bbd3ff900f3,70290548c2c0ae30] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:02:30.566 INFO 21062 --- [ocp-schedules-10,4d964bbd3ff900f3,70290548c2c0ae30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:30.595 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:02:30.595 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:02:30.698 WARN 21062 --- [ocp-async-4,e03c463373123aed,5af77d062c752b64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:30.733 INFO 21062 --- [ocp-async-7,8798784cc77248aa,4b9ab0e7d9f5f5d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:31.252 INFO 21062 --- [ocp-schedules-19,e003d13f293cf66b,9a4cd7c50d8fc3cd] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:02:31.256 INFO 21062 --- [ocp-schedules-19,e003d13f293cf66b,9a4cd7c50d8fc3cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:31.339 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:02:31.341 ERROR 21062 --- [ocp-schedules-19,e003d13f293cf66b,9a4cd7c50d8fc3cd] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:02:31.341 INFO 21062 --- [ocp-schedules-19,e003d13f293cf66b,9a4cd7c50d8fc3cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:02:32.698 WARN 21062 --- [ocp-async-7,4ba09c4e5e5532c1,e600a651070ea44b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:32.733 INFO 21062 --- [ocp-async-3,a729eb329940e5a7,295f7ec4390d7866] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:32.983 INFO 21062 --- [prometheus-minute-schedule-1,e8edf45df7a2f104,6673757553956b80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:33.038 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:33.038 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:34.699 WARN 21062 --- [ocp-async-3,db972983041a35a3,296c162792c002be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:34.733 INFO 21062 --- [ocp-async-8,c19846d339ef0f9d,b0ad96ef2947e909] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:35.803 INFO 21062 --- [ocp-schedules-13,9a321f54458f25d0,9333565a9226f364] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:02:35.983 INFO 21062 --- [prometheus-minute-schedule-1,28673e44ba81178d,53366dfa89a40393] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:36.039 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:36.039 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:36.699 WARN 21062 --- [ocp-async-8,971a294cfe0cf63d,a2542c5fdb74bbc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:36.733 INFO 21062 --- [ocp-async-10,ba6766a61f7c8aeb,fea99b643bf0b727] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:38.505 INFO 21062 --- [ocp-schedules-27,83ab8f237474ddbb,d193dd1ab383f3e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:02:38.505 INFO 21062 --- [ocp-schedules-21,45f8093095976cd3,edc542165e7ac5f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:02:38.508 INFO 21062 --- [ocp-schedules-22,659ac5443a702353,c04ba2fdc75759a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:02:38.566 INFO 21062 --- [ocp-async-10,0293b2ec8014a0d1,e98474cb8217aae5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:02:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:02:38.589269+08:00, lastProcessTime=2024-11-04T13:02:28.587850+08:00 2024-11-04 13:02:38.666 INFO 21062 --- [ocp-schedules-29,36ae9bef95ca7735,f884eff296411f08] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:02:38.669 INFO 21062 --- [ocp-schedules-29,36ae9bef95ca7735,f884eff296411f08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:38.670 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:02:38.699 WARN 21062 --- [ocp-async-9,d3e4c770a495a1a0,d4f59d050ef02636] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:38.733 INFO 21062 --- [ocp-async-1,f9bbfcad7c471730,44c9cb490040c97f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:02:38.589269+08:00 2024-11-04 13:02:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5cca34f968f3be48,9b10bcb3e5cc8c8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:39.020 INFO 21062 --- [ocp-schedules-8,d934e7991842f9dd,a013101bc07604e6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:02:39.040 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:39.040 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:02:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:02:40.699 WARN 21062 --- [ocp-async-1,d30ddf9d81e794dc,71aa1349741794f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:40.733 INFO 21062 --- [ocp-async-6,252d54303545b48f,9219622e418b611d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:41.804 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:02:41.833 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:02:41.833 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-11-04 13:02:41.983 INFO 21062 --- [prometheus-minute-schedule-1,0ce0eba8213ffa30,7835fc2627e80507] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:42.041 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:42.042 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:42.699 WARN 21062 --- [ocp-async-6,ccbab5f50b7856ac,219bdb186fc2ebeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:42.733 INFO 21062 --- [ocp-async-7,8ca3183afb78b701,60292cae971e132c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:44.699 WARN 21062 --- [ocp-async-7,d2579132a05f706a,10a2ded0be82d9ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:44.734 INFO 21062 --- [ocp-async-3,29bf3b173d0b2c69,8e78f86f9ac99984] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:44.983 INFO 21062 --- [prometheus-minute-schedule-1,930c1deeb0a8029a,2ee45a342f611a11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:45.042 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:45.043 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:45.845 INFO 21062 --- [ocp-schedules-7,a62477bccc13e0b6,d4421df3c8f634b9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:02:46.699 WARN 21062 --- [ocp-async-3,d2c5dce3e3c8a51f,b1f646df1792bbf9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:46.734 INFO 21062 --- [ocp-async-8,736971135d5eb06b,1df2bc5e4b2880af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:47.983 INFO 21062 --- [prometheus-minute-schedule-1,1b9249df2e54676b,dd772c52c5c16896] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:48.043 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:48.044 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:48.561 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:02:48.561 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:02:48.566 INFO 21062 --- [ocp-async-2,07182ba299ce822c,de127ad057548fd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:02:48.569 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:02:48.569 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:02:48.569 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:02:48.569 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:02:48.575 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:02:48.575 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:02:48.575 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:02:48.575 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:02:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:02:48.589086+08:00, lastProcessTime=2024-11-04T13:02:38.589269+08:00 2024-11-04 13:02:48.628 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:02:48.643 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:02:48.643 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:02:48.644 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:02:48.644 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:02:48.644 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:02:48.658 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:02:48.670 INFO 21062 --- [ocp-schedules-29,36ae9bef95ca7735,f884eff296411f08] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:02:48.700 WARN 21062 --- [ocp-async-6,9cdc553d37241fdd,0de0890e4bfc6b02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:48.734 INFO 21062 --- [ocp-async-7,567dbaea71186183,0271b1bb5c756233] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:48.797 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:02:48.589086+08:00 2024-11-04 13:02:49.022 INFO 21062 --- [ocp-schedules-14,70008ba9c30f8c09,fb72ac10830f019a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:02:49.287 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:02:49.359 INFO 21062 --- [ocp-schedules-11,f01f8fbc8b59fe4a,ea00d0f9b5ffc62f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:02:49.365 INFO 21062 --- [ocp-schedules-11,f01f8fbc8b59fe4a,ea00d0f9b5ffc62f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:02:49.365 INFO 21062 --- [ocp-schedules-11,f01f8fbc8b59fe4a,ea00d0f9b5ffc62f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:02:49.858 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.283182571S 2024-11-04 13:02:49.858 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:02:49.964 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:02:49.964 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:02:49.991 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:02:50.086 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:02:50.087 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:02:50.087 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:02:50.087 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:02:50.087 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:02:50.087 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:02:50.098 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:02:50.275 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] o.s.boot.SpringApplication : Started application in 1.706 seconds (JVM running for 1983.955) 2024-11-04 13:02:50.275 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.707020505S 2024-11-04 13:02:50.700 WARN 21062 --- [ocp-async-7,8ae3c2dce89a4dad,52267817a65bc2a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:50.734 INFO 21062 --- [ocp-async-5,1fa7301239beefd5,ad9ec3ae2eefc122] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:50.764 INFO 21062 --- [ocp-schedules-21,31d5bc8579d06eaf,517286aef7edfea4] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:02:50.983 INFO 21062 --- [prometheus-minute-schedule-1,31dbcca620c04663,eb4dfddda5d53b0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:51.044 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:51.045 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:52.700 WARN 21062 --- [ocp-async-5,68f649b07b5c39be,9c6a5b26a0aeeb47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:52.734 INFO 21062 --- [ocp-async-2,4aef14021970bb84,6a595c4ddd2b9653] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4d4fda3255217acc,5e0c26266c0b0c9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:54.045 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:54.046 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:54.700 WARN 21062 --- [ocp-async-2,f559c0f66f09e33f,353c2452b224f5a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:54.734 INFO 21062 --- [ocp-async-4,078f275c8ccb4c16,3909a7ce350bfff5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:55.882 INFO 21062 --- [ocp-schedules-28,25c65b480b236993,12aca87d3b430080] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:02:56.701 WARN 21062 --- [ocp-async-4,9f0d40227da9de7f,1c32f7ffff2327b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:56.735 INFO 21062 --- [ocp-async-3,084474083926190a,91c5e02de001823f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:56.983 INFO 21062 --- [prometheus-minute-schedule-1,cfdc0d1f446f4c7c,77e27b068bb48df6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:02:57.046 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:02:57.047 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:02:58.500 INFO 21062 --- [ocp-schedules-18,116a48b15c1447fd,60521408c235c836] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:02:58.500 INFO 21062 --- [ocp-schedules-12,ed81ccae200dae9f,23e9cfaf84e1993a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:02:58.502 INFO 21062 --- [ocp-schedules-19,d4ad24c154f4bd05,434cfe26238a65df] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:02:58.504 INFO 21062 --- [ocp-schedules-18,116a48b15c1447fd,60521408c235c836] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:58.504 INFO 21062 --- [ocp-schedules-12,ed81ccae200dae9f,23e9cfaf84e1993a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:02:58.505 INFO 21062 --- [ocp-schedules-14,f6c56e7473af4eb5,aab2b43e59b867df] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:02:58.507 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:02:58.511 INFO 21062 --- [ocp-schedules-22,3213df9dce253fdb,38ffc7aa8c134ea0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:02:58.511 INFO 21062 --- [ocp-schedules-25,7c96996c952cfc91,35b88adc0587f852] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:02:58.511 INFO 21062 --- [ocp-schedules-24,db0e493cfd44fdd1,5a945c4d4142c055] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:02:58.511 INFO 21062 --- [ocp-schedules-22,3213df9dce253fdb,38ffc7aa8c134ea0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:02:58.511 INFO 21062 --- [ocp-schedules-24,db0e493cfd44fdd1,5a945c4d4142c055] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:02:58.511 INFO 21062 --- [ocp-schedules-25,7c96996c952cfc91,35b88adc0587f852] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:02:58.566 INFO 21062 --- [ocp-async-9,6a7855ae2cb809e8,6e7a5c6d947cee80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:02:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:02:58.589297+08:00, lastProcessTime=2024-11-04T13:02:48.589086+08:00 2024-11-04 13:02:58.701 WARN 21062 --- [ocp-async-5,9468549929fdf611,7732143a07d7d587] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:02:58.705 INFO 21062 --- [ocp-schedules-1,a254fd126054de18,2aea8b78003ab80d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:02:58.735 INFO 21062 --- [ocp-async-1,cc7e90d8c3f369b6,ec79ba1a5db5ff76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:02:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:02:58.589297+08:00 2024-11-04 13:02:59.025 INFO 21062 --- [ocp-schedules-20,7314a584bb43a8bc,480d67a12d9886b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:02:59.983 INFO 21062 --- [prometheus-minute-schedule-1,95226945d7679d9f,7cbf414d43b1fdcb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696580, interval=60 2024-11-04 13:03:00.047 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:00.048 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:00.596 INFO 21062 --- [ocp-schedules-10,4d964bbd3ff900f3,70290548c2c0ae30] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:03:00.596 INFO 21062 --- [ocp-schedules-10,2ac8758fe9db8bcf,2ce96eb9ec8dd6c0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:03:00.599 INFO 21062 --- [ocp-schedules-10,2ac8758fe9db8bcf,2ce96eb9ec8dd6c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:00.628 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:03:00.628 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:03:00.701 WARN 21062 --- [ocp-async-2,aef36855d76376e9,9b646a0fdea302aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:00.735 INFO 21062 --- [ocp-async-10,e90cf9551fbd8ce0,183902bc3dc78f81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:02.701 WARN 21062 --- [ocp-async-4,caaf39082b6ccb6a,b0aa947fdbf33b4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:02.735 INFO 21062 --- [ocp-async-9,0c2302f78adca4d6,44b4bae10726d6ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:02.983 INFO 21062 --- [prometheus-minute-schedule-1,b78893f6efbe7cc9,633d1b43bf65df13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:03.048 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:03.049 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-42,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=13, affectRows=13 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=13, validCount=13 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-39,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=11, affectRows=11 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=11, validCount=11 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-41,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=28, affectRows=28 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=28, validCount=28 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-48,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=26, affectRows=26 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=26, validCount=26 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-12,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=36, affectRows=36 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=36, validCount=36 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-34,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=34, affectRows=34 2024-11-04 13:03:04.143 INFO 21062 --- [metric-archive-minute-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=34, validCount=34 2024-11-04 13:03:04.144 INFO 21062 --- [metric-archive-minute-24,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=21, affectRows=21 2024-11-04 13:03:04.144 INFO 21062 --- [metric-archive-minute-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=21, validCount=21 2024-11-04 13:03:04.144 INFO 21062 --- [metric-archive-minute-37,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=32, affectRows=32 2024-11-04 13:03:04.144 INFO 21062 --- [metric-archive-minute-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=32, validCount=32 2024-11-04 13:03:04.144 INFO 21062 --- [metric-archive-minute-11,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=31, affectRows=31 2024-11-04 13:03:04.144 INFO 21062 --- [metric-archive-minute-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=31, validCount=31 2024-11-04 13:03:04.145 INFO 21062 --- [metric-archive-minute-29,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=27, affectRows=27 2024-11-04 13:03:04.145 INFO 21062 --- [metric-archive-minute-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=27, validCount=27 2024-11-04 13:03:04.147 INFO 21062 --- [metric-archive-minute-17,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=37, affectRows=37 2024-11-04 13:03:04.147 INFO 21062 --- [metric-archive-minute-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=37, validCount=37 2024-11-04 13:03:04.147 INFO 21062 --- [metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=23, affectRows=23 2024-11-04 13:03:04.147 INFO 21062 --- [metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730692800, nodeCount=23, validCount=23 2024-11-04 13:03:04.701 WARN 21062 --- [ocp-async-3,7323d7a5f72bf200,506ee46d2ceebb61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:04.735 INFO 21062 --- [ocp-async-8,f6de20721f07b202,6821fdeeb1beb1ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:05.908 INFO 21062 --- [ocp-schedules-19,7f7a8ffc2b0038a0,3eb65c3e6e0aba02] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:03:05.983 INFO 21062 --- [prometheus-minute-schedule-1,9f4461709820b18b,cbed6f066fdb8d97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:06.049 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:06.049 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:06.702 WARN 21062 --- [ocp-async-1,5b5b258914d89453,1841b17a06794eee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:06.736 INFO 21062 --- [ocp-async-6,1a8173542a814bcd,5bb1b64a6bc48126] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:08.508 INFO 21062 --- [ocp-schedules-12,ed81ccae200dae9f,23e9cfaf84e1993a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:03:08.508 INFO 21062 --- [ocp-schedules-18,116a48b15c1447fd,60521408c235c836] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:03:08.566 INFO 21062 --- [ocp-async-6,3bf86f77b81e95a4,1a6e3fb571385859] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:03:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:03:08.589298+08:00, lastProcessTime=2024-11-04T13:02:58.589297+08:00 2024-11-04 13:03:08.702 WARN 21062 --- [ocp-async-4,43393f243ffd44db,0352a68a6ea0b77a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:08.736 INFO 21062 --- [ocp-async-9,d5b173f08997142b,db6da03d6dc4e639] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:03:08.589298+08:00 2024-11-04 13:03:08.983 INFO 21062 --- [prometheus-minute-schedule-1,de73dada678c16bb,d268a9007c12358d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:09.028 INFO 21062 --- [ocp-schedules-26,535e1c1d8389cc5a,321e1809a3427aa7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:03:09.050 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:09.050 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:09.368 INFO 21062 --- [ocp-schedules-18,4180112bdd1b5f0a,3595bbdf35324322] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:03:09.375 INFO 21062 --- [ocp-schedules-18,4180112bdd1b5f0a,3595bbdf35324322] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:03:09.375 INFO 21062 --- [ocp-schedules-18,4180112bdd1b5f0a,3595bbdf35324322] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:03:10.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:03:10.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:03:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:03:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:03:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:03:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:03:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:03:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:03:10.702 WARN 21062 --- [ocp-async-3,6649ada9ade4d8b6,d310bdae82b66980] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:10.736 INFO 21062 --- [ocp-async-8,be621bb35aad9ded,b46eb44c494c9dc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:11.154 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=1, affectRows=1 2024-11-04 13:03:11.154 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=1, validCount=1 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=18, affectRows=18 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=18, affectRows=18 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=9, affectRows=9 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=25, affectRows=25 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=9, validCount=9 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=18, validCount=18 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=18, validCount=18 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=25, validCount=25 2024-11-04 13:03:11.155 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=46, affectRows=46 2024-11-04 13:03:11.156 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=43, affectRows=43 2024-11-04 13:03:11.156 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=46, validCount=46 2024-11-04 13:03:11.156 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=42, validCount=42 2024-11-04 13:03:11.156 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=52, affectRows=52 2024-11-04 13:03:11.156 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=43, affectRows=43 2024-11-04 13:03:11.156 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=51, validCount=51 2024-11-04 13:03:11.156 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=43, validCount=43 2024-11-04 13:03:11.157 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696520, nodeCount=67, affectRows=67 2024-11-04 13:03:11.157 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696520, nodeCount=66, validCount=66 2024-11-04 13:03:11.983 INFO 21062 --- [prometheus-minute-schedule-1,7fe3360309ec25fb,368f09590c136c15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:12.050 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:12.051 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:12.702 WARN 21062 --- [ocp-async-1,7ea7c179e953bd92,28794ca28cadf151] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:12.736 INFO 21062 --- [ocp-async-6,63eb30cef2abebdd,4f100f5fd56ac6f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:13.948 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T12:57:13.929, end=2024-11-04T13:02:13.946} 2024-11-04 13:03:13.948 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T12:57:13.929, end=2024-11-04T13:02:13.946} 2024-11-04 13:03:13.959 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:03:13.959 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:03:13.959 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:03:13.959 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:03:13.984 INFO 21062 --- [prometheus-cache-monitor1,6e722b3bb1f999e6,0099ceb18753090e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:03:13.984 INFO 21062 --- [prometheus-cache-monitor1,6e722b3bb1f999e6,0099ceb18753090e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:03:14.313 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T12:59:14.289, end=2024-11-04T13:01:14.31} 2024-11-04 13:03:14.316 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:03:14.316 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:03:14.702 WARN 21062 --- [ocp-async-10,44b81a1485b83ad2,ad9d579e663335f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:14.736 INFO 21062 --- [ocp-async-7,d5ca0c48ea242e2a,9a4c5a2d160c9425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:14.983 INFO 21062 --- [prometheus-minute-schedule-1,af73e77af0175a3a,9eaa5456987ba386] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:15.051 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:15.052 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:15.939 INFO 21062 --- [ocp-schedules-2,860088df1576d732,775cc41c4dc23280] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:03:16.702 WARN 21062 --- [ocp-async-9,62d70daca39946d1,1f3da527ac74cd0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:16.737 INFO 21062 --- [ocp-async-5,06022ca2b898e867,6923573a1657484c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:17.983 INFO 21062 --- [prometheus-minute-schedule-1,26189533d2479a10,aec5f45ee87dbd9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:18.053 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:18.053 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:18.567 INFO 21062 --- [ocp-async-3,6c9bf602db5866c3,19c5ba2ec1bd94d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:03:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:03:18.589297+08:00, lastProcessTime=2024-11-04T13:03:08.589298+08:00 2024-11-04 13:03:18.702 WARN 21062 --- [ocp-async-1,a9d43c83e926e8b9,d64d7aa04a1a70e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:18.737 INFO 21062 --- [ocp-async-6,b8504e757c8a82cf,2ebb6165415145b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:03:18.589297+08:00 2024-11-04 13:03:19.031 INFO 21062 --- [ocp-schedules-3,58bc980692d351b7,05d0473d7ac6385e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:03:20.702 WARN 21062 --- [ocp-async-10,ebb9a8fa614617df,1a21fc7f37f904c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:20.737 INFO 21062 --- [ocp-async-7,e591fea828169e1a,d76bf2ea7dbd9b11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:20.983 INFO 21062 --- [prometheus-minute-schedule-1,206d7aaf9b848e58,9948285316f6359a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:21.054 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:21.055 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:22.703 WARN 21062 --- [ocp-async-9,a2afb10929632984,64040d49b62bc8c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:22.737 INFO 21062 --- [ocp-async-5,e0f7dc0c824b7112,befa6db0e45d3ef0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:23.983 INFO 21062 --- [prometheus-minute-schedule-1,01771dfac64ee705,45f9df0654a8a677] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:24.055 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:24.056 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:24.703 WARN 21062 --- [ocp-async-8,9e697157e2eac460,a3f01078b0fcfe67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:24.737 INFO 21062 --- [ocp-async-2,1bced2ab23e4ce2d,dd82ca499de4ea7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:25.981 INFO 21062 --- [ocp-schedules-4,f286435fc02301ef,db39eff0e70853d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:03:26.703 WARN 21062 --- [ocp-async-6,02674ab5b39929a6,23abcdb3189010b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:26.737 INFO 21062 --- [ocp-async-4,c36c25ef101cff38,64758c48b26b58c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9ade358178564d73,24df7ec2f4ed4643] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:26.984 INFO 21062 --- [prometheus-hour-schedule-1,325f48dff02bc015,22efd11e457ebd08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:03:27.056 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:27.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:28.498 INFO 21062 --- [ocp-schedules-29,5d3dc5b4d44e6ce6,dd01cc63b3ae53af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:03:28.499 INFO 21062 --- [ocp-schedules-26,9d51c475bd32c4c4,c5b2e058c3f3b5a0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:03:28.500 INFO 21062 --- [ocp-schedules-8,b928624321221ce0,26d77215f7b0d8c4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:03:28.501 INFO 21062 --- [ocp-schedules-28,2cc10fea2dfbd6df,12f1dd87011c2368] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:03:28.502 INFO 21062 --- [ocp-schedules-18,3d11aaab3dd4c80b,bbf90f413bdeee09] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:03:28.503 INFO 21062 --- [ocp-schedules-29,5d3dc5b4d44e6ce6,dd01cc63b3ae53af] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:03:28.504 INFO 21062 --- [ocp-schedules-26,9d51c475bd32c4c4,c5b2e058c3f3b5a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:28.504 INFO 21062 --- [ocp-schedules-8,b928624321221ce0,26d77215f7b0d8c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:03:28.505 INFO 21062 --- [ocp-schedules-12,52c629d710820aac,6d55bd0b493a1324] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:03:28.506 INFO 21062 --- [ocp-schedules-28,2cc10fea2dfbd6df,12f1dd87011c2368] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:03:28.510 INFO 21062 --- [ocp-schedules-4,e5d7fd7feceae8fc,a93cad5efdb143d6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:03:28.510 INFO 21062 --- [ocp-schedules-4,e5d7fd7feceae8fc,a93cad5efdb143d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:03:28.510 INFO 21062 --- [ocp-schedules-24,175e54f380fc23a5,e777f1f68f262959] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:03:28.510 INFO 21062 --- [ocp-schedules-24,175e54f380fc23a5,e777f1f68f262959] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:03:28.511 INFO 21062 --- [ocp-schedules-15,e95f36f12d93aa7b,be3fc3d08f661daa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:03:28.511 INFO 21062 --- [ocp-schedules-15,e95f36f12d93aa7b,be3fc3d08f661daa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:03:28.550 INFO 21062 --- [ocp-schedules-7,e65603090c0f663c,ed016ffe437a0a66] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 13:03:28.550 INFO 21062 --- [ocp-schedules-7,e65603090c0f663c,ed016ffe437a0a66] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 13:03:28.567 INFO 21062 --- [ocp-async-7,d747d6bf5597dfd4,71411277437755a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:03:28.578 INFO 21062 --- [ocp-schedules-13,9bbfe9ab8217e74e,0ff409bfb1ba4b7c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:03:28.578 INFO 21062 --- [ocp-schedules-13,9bbfe9ab8217e74e,0ff409bfb1ba4b7c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:03:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:03:28.588638+08:00, lastProcessTime=2024-11-04T13:03:18.589297+08:00 2024-11-04 13:03:28.634 INFO 21062 --- [ocp-schedules-27,b12e4d483f50fb6f,abab32f6e4c6f6e5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:03:28.692 INFO 21062 --- [ocp-schedules-5,ef75e88476afa652,cd49209ffeb9fa5f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:03:28.692 INFO 21062 --- [ocp-schedules-5,ef75e88476afa652,cd49209ffeb9fa5f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:03:28.703 WARN 21062 --- [ocp-async-5,03c73c16dd5471a8,1ccbb1270e32dea4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:28.707 INFO 21062 --- [ocp-schedules-28,8ade71171875f182,8804bf7bbf00a52e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:03:28.737 INFO 21062 --- [ocp-async-1,8cf092b8148abac3,68b4129b97d8e661] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:28.753 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:03:28.588638+08:00 2024-11-04 13:03:28.756 INFO 21062 --- [ocp-schedules-18,385d6baa3a677555,79a5bc1804d323c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:03:28.762 INFO 21062 --- [ocp-schedules-18,385d6baa3a677555,79a5bc1804d323c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:03:28.762 INFO 21062 --- [ocp-schedules-18,385d6baa3a677555,79a5bc1804d323c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:03:28.891 INFO 21062 --- [ocp-schedules-29,7f28e9128bce690d,3871c758bdb277d7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:03:28.891 INFO 21062 --- [ocp-schedules-29,7f28e9128bce690d,3871c758bdb277d7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:03:28.898 INFO 21062 --- [ocp-schedules-29,7f28e9128bce690d,3871c758bdb277d7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:03:28.898 INFO 21062 --- [ocp-schedules-29,7f28e9128bce690d,3871c758bdb277d7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:03:28.903 INFO 21062 --- [ocp-schedules-29,7f28e9128bce690d,3871c758bdb277d7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:03:28.903 INFO 21062 --- [ocp-schedules-29,7f28e9128bce690d,3871c758bdb277d7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:03:29.033 INFO 21062 --- [ocp-schedules-20,aaf6dfc946de45de,1115bb3de6576601] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:03:29.373 INFO 21062 --- [ocp-schedules-22,2b997feb21d37b66,e0cac3cb0d1093c2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:03:29.373 INFO 21062 --- [ocp-schedules-22,2b997feb21d37b66,e0cac3cb0d1093c2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:03:29.377 INFO 21062 --- [ocp-schedules-17,05c4df8a99034af3,676b009a1ca162bc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:03:29.383 INFO 21062 --- [ocp-schedules-17,05c4df8a99034af3,676b009a1ca162bc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:03:29.383 INFO 21062 --- [ocp-schedules-17,05c4df8a99034af3,676b009a1ca162bc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:03:29.400 INFO 21062 --- [ocp-schedules-22,2b997feb21d37b66,e0cac3cb0d1093c2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:03:29.400 INFO 21062 --- [ocp-schedules-22,2b997feb21d37b66,e0cac3cb0d1093c2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:03:29.401 INFO 21062 --- [ocp-schedules-11,499fb1f197b7c035,6db12c303b4946cd] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:03:29.454 INFO 21062 --- [ocp-schedules-23,01ab177f3a78b137,26056b414505dbc8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:03:29.455 INFO 21062 --- [ocp-schedules-23,01ab177f3a78b137,26056b414505dbc8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:03:29.455 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:03:29.455 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:03:29.465 INFO 21062 --- [ocp-schedules-23,01ab177f3a78b137,26056b414505dbc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:03:29.465 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:03:29.466 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:03:29.467 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:03:29.467 WARN 21062 --- [ocp-schedules-23,01ab177f3a78b137,26056b414505dbc8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:03:29.467 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:03:29.467 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:03:29.468 INFO 21062 --- [ocp-schedules-23,01ab177f3a78b137,26056b414505dbc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:03:29.468 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:03:29.469 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:03:29.469 WARN 21062 --- [ocp-schedules-23,01ab177f3a78b137,26056b414505dbc8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:03:29.469 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:03:29.475 ERROR 21062 --- [ocp-schedules-23,01ab177f3a78b137,26056b414505dbc8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:03:29.515 INFO 21062 --- [ocp-schedules-19,7513feb101d37e73,64892b56466cd0f5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:03:29.515+08:00, clusters=[1] 2024-11-04 13:03:29.516 INFO 21062 --- [ocp-schedules-19,7513feb101d37e73,64892b56466cd0f5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:03:29.610 INFO 21062 --- [ocp-schedules-5,72bdfc1a0001ac4c,42a51c249b54c725] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:03:29.610+08:00, clusters=[1] 2024-11-04 13:03:29.613 INFO 21062 --- [ocp-schedules-5,72bdfc1a0001ac4c,42a51c249b54c725] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:03:29.983 INFO 21062 --- [prometheus-minute-schedule-1,24e6f799d147aebc,ceb90e38267e6ddc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:30.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:30.058 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:03:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:03:30.629 INFO 21062 --- [ocp-schedules-10,2ac8758fe9db8bcf,2ce96eb9ec8dd6c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:03:30.629 INFO 21062 --- [ocp-schedules-10,fbc531fa9815718f,777f5cea7256e010] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:03:30.632 INFO 21062 --- [ocp-schedules-10,fbc531fa9815718f,777f5cea7256e010] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:30.661 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:03:30.661 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:03:30.703 WARN 21062 --- [ocp-async-1,55366d9d0721b103,9ff0181ac248f43f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:30.738 INFO 21062 --- [ocp-async-10,07b1c0df8c0b13b4,ea9ea4aaafdd6b08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:31.341 INFO 21062 --- [ocp-schedules-22,f6b0ac3f07a2ee18,4337e126ebe799d5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:03:31.345 INFO 21062 --- [ocp-schedules-22,f6b0ac3f07a2ee18,4337e126ebe799d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:31.428 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:03:31.432 ERROR 21062 --- [ocp-schedules-22,f6b0ac3f07a2ee18,4337e126ebe799d5] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:03:31.432 INFO 21062 --- [ocp-schedules-22,f6b0ac3f07a2ee18,4337e126ebe799d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:03:32.704 WARN 21062 --- [ocp-async-10,a7df5c3e98fe3004,5c5bd2ca2b3889cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:32.738 INFO 21062 --- [ocp-async-9,eb8d51148328de0a,005ec214cdb94671] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3f0aa1464a104389,c29903c2a2ae2796] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:33.058 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:33.059 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:34.704 WARN 21062 --- [ocp-async-9,a7511741eb1c73af,011f1110be22c59d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:34.738 INFO 21062 --- [ocp-async-8,e1c032a1cdcd3d24,cef7b0475b49d53d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4606ee087243c712,e53ac42c3c791aac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:36.025 INFO 21062 --- [ocp-schedules-23,7d0707a48f04fc85,999b61e023d75c0d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:03:36.059 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:36.061 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:36.704 WARN 21062 --- [ocp-async-8,c1b27bc585a9fefe,b2dc0ac9806e976d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:36.738 INFO 21062 --- [ocp-async-4,104b479d47cf3b8d,58d2ee6e6edc34bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:38.505 INFO 21062 --- [ocp-schedules-26,9d51c475bd32c4c4,c5b2e058c3f3b5a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:03:38.505 INFO 21062 --- [ocp-schedules-8,b928624321221ce0,26d77215f7b0d8c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:03:38.567 INFO 21062 --- [ocp-async-4,a0c60b4c51c68057,cb2fc14e706d5f4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:03:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:03:38.589080+08:00, lastProcessTime=2024-11-04T13:03:28.588638+08:00 2024-11-04 13:03:38.704 WARN 21062 --- [ocp-async-6,10bca4c3f6a1d092,184c81fd752fc968] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:38.738 INFO 21062 --- [ocp-async-3,521990573a46b94d,f5d5fe1e534b3a5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:03:38.589080+08:00 2024-11-04 13:03:38.983 INFO 21062 --- [prometheus-minute-schedule-1,16720dba16614a4c,498f0957659516a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:39.036 INFO 21062 --- [ocp-schedules-6,ab52296149612fb2,25e5ff906e1b3aec] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:03:39.061 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:39.062 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:03:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:03:40.704 WARN 21062 --- [ocp-async-3,8aff23f3f753c96c,8e750f7732d9867a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:40.738 INFO 21062 --- [ocp-async-8,235e6922d2805d09,db24aea99c538a32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:41.983 INFO 21062 --- [prometheus-minute-schedule-1,add24c488ace9465,6be8a95255efd073] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:42.062 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:42.063 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:42.704 WARN 21062 --- [ocp-async-8,672f7a3f0eeaa6e6,62ea4cc1d4900b81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:42.738 INFO 21062 --- [ocp-async-4,4bfaa66ae08869fa,6c6609787ad61207] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:44.705 WARN 21062 --- [ocp-async-4,3d8640e82822ebac,baab8d484b295842] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:44.739 INFO 21062 --- [ocp-async-7,3972621f7f05f48b,2e52ffa154f9f1a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d61a89e3e8c5a254,24cb075ca676de48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:45.063 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:45.064 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:46.118 INFO 21062 --- [ocp-schedules-21,4ff1edc5a43338ae,b49e4553c5c1ecf6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:03:46.705 WARN 21062 --- [ocp-async-7,82f587858022e738,277a49d10a5247b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:46.739 INFO 21062 --- [ocp-async-5,277d93b1af7c9a3e,a7983efbba002411] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:47.983 INFO 21062 --- [prometheus-minute-schedule-1,89be685b717e8859,bbd7a7c65d50a1f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:48.064 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:48.065 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:48.567 INFO 21062 --- [ocp-async-2,992e7eeda028779e,4e28c0e42a9530a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:03:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:03:48.588827+08:00, lastProcessTime=2024-11-04T13:03:38.589080+08:00 2024-11-04 13:03:48.670 INFO 21062 --- [ocp-schedules-27,ff6e1e798955ae2b,eec6be5176b377b8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:03:48.674 INFO 21062 --- [ocp-schedules-27,ff6e1e798955ae2b,eec6be5176b377b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:48.675 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:03:48.705 WARN 21062 --- [ocp-async-8,138757df62485157,0328e48435e23ee4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:48.739 INFO 21062 --- [ocp-async-4,213e7dd7d1acb3cf,baa2a43e850bb89f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:48.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:03:48.588827+08:00 2024-11-04 13:03:49.039 INFO 21062 --- [ocp-schedules-2,ed0247a141236d16,49135cfa22dadb69] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:03:49.385 INFO 21062 --- [ocp-schedules-11,b58931caec442e51,f3fd2209b8fd80e5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:03:49.392 INFO 21062 --- [ocp-schedules-11,b58931caec442e51,f3fd2209b8fd80e5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:03:49.392 INFO 21062 --- [ocp-schedules-11,b58931caec442e51,f3fd2209b8fd80e5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:03:50.705 WARN 21062 --- [ocp-async-4,fb4a8875a0e39ad1,8512a15e9e5c8dbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:50.739 INFO 21062 --- [ocp-async-7,8558a3477e19d9ce,e42a94bb1fe9f2e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d8dbedc9c486417e,6627b8248a89c7e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:51.065 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:51.066 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:52.705 WARN 21062 --- [ocp-async-7,40fc5ecefe278aef,6cc1319732016cb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:52.739 INFO 21062 --- [ocp-async-2,0cf36cff2919358e,37aab962d08b6fd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:53.983 INFO 21062 --- [prometheus-minute-schedule-1,5f443f4708a694f7,a0d73b585616a093] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:54.066 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:54.067 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:54.705 WARN 21062 --- [ocp-async-2,f45b7dd4b7665fb4,6b8fccaf7ef7b110] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:54.740 INFO 21062 --- [ocp-async-1,2ecabee6684476b1,e0aab2521503b343] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:56.142 INFO 21062 --- [ocp-schedules-15,25253670cb4cdc55,2c1bb73408ffaf11] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:03:56.706 WARN 21062 --- [ocp-async-1,e64d08b3c7a864b0,a9ec45586dd6add2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:56.740 INFO 21062 --- [ocp-async-4,b4b584ae498bb9fb,4c3f7766ef0b8f57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:56.983 INFO 21062 --- [prometheus-minute-schedule-1,5888aa7e0072488b,1294f293a5b272a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:03:57.067 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:03:57.067 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:03:58.500 INFO 21062 --- [ocp-schedules-15,22fb84d2d2dc4c37,706447c256ae9c04] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:03:58.500 INFO 21062 --- [ocp-schedules-21,70e7e26bf8bce38e,82c189dd3cf70a0a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:03:58.502 INFO 21062 --- [ocp-schedules-20,774b729dabee8129,167e0bd2cf456ce5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:03:58.504 INFO 21062 --- [ocp-schedules-15,22fb84d2d2dc4c37,706447c256ae9c04] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:58.504 INFO 21062 --- [ocp-schedules-21,70e7e26bf8bce38e,82c189dd3cf70a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:03:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:03:58.506 INFO 21062 --- [ocp-schedules-9,2a4c035dd41a5ed0,40750c1804285c5f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:03:58.510 INFO 21062 --- [ocp-schedules-5,0a8013a61fa4f93a,6212ed3a03e42c49] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:03:58.510 INFO 21062 --- [ocp-schedules-5,0a8013a61fa4f93a,6212ed3a03e42c49] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:03:58.510 INFO 21062 --- [ocp-schedules-23,eb0fa2f26df47d32,8283a27c15659bc4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:03:58.510 INFO 21062 --- [ocp-schedules-23,eb0fa2f26df47d32,8283a27c15659bc4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:03:58.511 INFO 21062 --- [ocp-schedules-28,a561f6b7ed17b7a5,736cb570e18737c8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:03:58.511 INFO 21062 --- [ocp-schedules-28,a561f6b7ed17b7a5,736cb570e18737c8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:03:58.567 INFO 21062 --- [ocp-async-9,22e34e22217636bb,91506ac55bdd2e84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:03:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:03:58.589472+08:00, lastProcessTime=2024-11-04T13:03:48.588827+08:00 2024-11-04 13:03:58.675 INFO 21062 --- [ocp-schedules-27,ff6e1e798955ae2b,eec6be5176b377b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:03:58.706 WARN 21062 --- [ocp-async-7,80788176a23885aa,f4c758a8edfa6755] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:03:58.709 INFO 21062 --- [ocp-schedules-16,d44a7903746afa09,49b869be12f11b74] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:03:58.740 INFO 21062 --- [ocp-async-5,7be0f533fb589150,c33e4ea0d189b418] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:03:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:03:58.589472+08:00 2024-11-04 13:03:59.042 INFO 21062 --- [ocp-schedules-30,97288c25361fbff3,135a8bff5b9154ae] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:03:59.983 INFO 21062 --- [prometheus-minute-schedule-1,086f6e45f843b01a,8d0bf7ab0faeaab7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696640, interval=60 2024-11-04 13:04:00.067 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:00.068 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:00.662 INFO 21062 --- [ocp-schedules-10,fbc531fa9815718f,777f5cea7256e010] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:04:00.662 INFO 21062 --- [ocp-schedules-10,0a69192d58320de9,67009f88b7b512da] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:04:00.665 INFO 21062 --- [ocp-schedules-10,0a69192d58320de9,67009f88b7b512da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:00.696 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:04:00.696 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:04:00.706 WARN 21062 --- [ocp-async-2,5575860deab5a3f0,a1b4d29a9bd093d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:00.740 INFO 21062 --- [ocp-async-10,5e7326caf218485a,323ecb1bb018a09b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:02.706 WARN 21062 --- [ocp-async-1,ac66b8fcb341b440,19dd150eb2d8f6af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:02.740 INFO 21062 --- [ocp-async-6,a625efef61f7f399,ab5aed885440f1f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c3914d2c9b78c528,3dbfe0e3a3421576] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:03.068 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:03.069 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:04.706 WARN 21062 --- [ocp-async-4,70bac32345460661,e0ec180bb63ccef6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:04.740 INFO 21062 --- [ocp-async-7,c45115710f0c6f18,881010c030e48d30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:05.983 INFO 21062 --- [prometheus-minute-schedule-1,503000ca56929f3f,46004bcb8c994c4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:06.069 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:06.070 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:06.183 INFO 21062 --- [ocp-schedules-12,056fd260abbbdc62,fa15d83a453c5503] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:04:06.707 WARN 21062 --- [ocp-async-5,11045d3ac1039c2a,d66d340079f157a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:06.741 INFO 21062 --- [ocp-async-2,fbb1660ae4d44861,ddabb7dc51bd741a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:08.505 INFO 21062 --- [ocp-schedules-21,70e7e26bf8bce38e,82c189dd3cf70a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:04:08.505 INFO 21062 --- [ocp-schedules-15,22fb84d2d2dc4c37,706447c256ae9c04] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:04:08.568 INFO 21062 --- [ocp-async-8,d9ac87bc06914d8f,3fb59bfc1be52302] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:04:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:04:08.588849+08:00, lastProcessTime=2024-11-04T13:03:58.589472+08:00 2024-11-04 13:04:08.707 WARN 21062 --- [ocp-async-1,0fc723be9ca60968,7307b94015f4bf9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:08.741 INFO 21062 --- [ocp-async-6,1cb0173772582bdb,6a7ef279d8c464e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:04:08.588849+08:00 2024-11-04 13:04:08.983 INFO 21062 --- [prometheus-minute-schedule-1,9c463115869bc63c,97486401a20946cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:09.045 INFO 21062 --- [ocp-schedules-27,f23c828ec7abb20c,875646eca1bb7345] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:04:09.070 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:09.071 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:09.394 INFO 21062 --- [ocp-schedules-20,36e8962babf96f33,66083b4c7847c4e5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:04:09.399 INFO 21062 --- [ocp-schedules-20,36e8962babf96f33,66083b4c7847c4e5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:04:09.399 INFO 21062 --- [ocp-schedules-20,36e8962babf96f33,66083b4c7847c4e5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:04:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:04:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:04:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:04:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:04:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:04:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:04:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:04:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:04:10.707 WARN 21062 --- [ocp-async-4,9cd2100318841095,de23719b58f1850e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:10.741 INFO 21062 --- [ocp-async-7,6054a475ec554381,65cd759230faf68a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696580, nodeCount=31, affectRows=31 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696580, nodeCount=37, affectRows=37 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696580, nodeCount=24, affectRows=24 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696580, nodeCount=30, validCount=30 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696580, nodeCount=24, validCount=24 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696580, nodeCount=37, validCount=37 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696580, nodeCount=63, affectRows=63 2024-11-04 13:04:11.160 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696580, nodeCount=63, validCount=63 2024-11-04 13:04:11.161 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696580, nodeCount=86, affectRows=86 2024-11-04 13:04:11.161 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696580, nodeCount=81, affectRows=81 2024-11-04 13:04:11.161 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696580, nodeCount=86, validCount=86 2024-11-04 13:04:11.161 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696580, nodeCount=79, validCount=79 2024-11-04 13:04:11.983 INFO 21062 --- [prometheus-minute-schedule-1,45b3270ebe3522a4,d9f809fced36e6cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:12.071 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:12.072 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:12.707 WARN 21062 --- [ocp-async-5,34abfca56578f3b4,6b281bdda3abbe00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:12.741 INFO 21062 --- [ocp-async-8,23d0c83ccdecb4dc,df1c1d9c6a3f3add] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:13.984 INFO 21062 --- [prometheus-cache-monitor1,93a100b40e5b7cb2,ff456866d357e43b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:04:13.985 INFO 21062 --- [prometheus-cache-monitor1,93a100b40e5b7cb2,ff456866d357e43b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:04:14.707 WARN 21062 --- [ocp-async-10,11f1c41ebfc23c31,845c68b1868098ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:14.741 INFO 21062 --- [ocp-async-9,979b9aaacaca1625,43944f904858ef7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:14.983 INFO 21062 --- [prometheus-minute-schedule-1,65ef5edbf032d4b3,e34bfdbbc50e81f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:15.072 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:15.074 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:16.211 INFO 21062 --- [ocp-schedules-9,a26ab1fbf4f034f8,bdebe123ad012553] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:04:16.708 WARN 21062 --- [ocp-async-6,6aac8f7ef886a9d2,8ea341e1c48f453d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:16.742 INFO 21062 --- [ocp-async-3,caac91564c78e6b2,a0f13fe9fd96029e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7dc56f7186c30258,428885c974c2902a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:18.074 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:18.075 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:18.568 INFO 21062 --- [ocp-async-4,e074472f73155419,df1143c3e1e5ee8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:04:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:04:18.589079+08:00, lastProcessTime=2024-11-04T13:04:08.588849+08:00 2024-11-04 13:04:18.708 WARN 21062 --- [ocp-async-5,e2b0202e28e6341f,e127c786c724bc8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:18.742 INFO 21062 --- [ocp-async-2,a00ea20dad065e7b,17c42736b96c788c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:18.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:04:18.589079+08:00 2024-11-04 13:04:19.047 INFO 21062 --- [ocp-schedules-15,0237ae5b3da2e4cd,6bf71db8e1bce261] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:04:20.708 WARN 21062 --- [ocp-async-10,3c99fd2ef4ac4487,2d633334249db379] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:20.742 INFO 21062 --- [ocp-async-9,0e5f9179961a5151,9a6b2ceef2059083] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:20.983 INFO 21062 --- [prometheus-minute-schedule-1,91a63a247f6d4348,393d81a4dd89aa84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:21.075 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:21.076 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:22.708 WARN 21062 --- [ocp-async-6,14bb3497f51483ee,ecff327a61eab5b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:22.742 INFO 21062 --- [ocp-async-3,bd67e13bd23900b5,e730732aa2bde6b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:23.983 INFO 21062 --- [prometheus-minute-schedule-1,0d0a89467154e373,68d863f9482dcda6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:24.076 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:24.077 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:24.708 WARN 21062 --- [ocp-async-7,19a27be349b60575,510f04fa067dab38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:24.742 INFO 21062 --- [ocp-async-8,6ffdb1ea08da9294,0ffe9cb7a85cac23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:26.252 INFO 21062 --- [ocp-schedules-26,4d9fc3923e6aa394,31ef5090fb7176f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:04:26.708 WARN 21062 --- [ocp-async-2,25248bd00c86ccaf,e08cf4ae82110d96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:26.743 INFO 21062 --- [ocp-async-1,4e408bc2160d0424,fb314a52c8103598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:26.983 INFO 21062 --- [prometheus-minute-schedule-1,a194251195b4a633,b47c707a9df7997c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:26.984 INFO 21062 --- [prometheus-hour-schedule-1,50c964b03fa00f67,29abdaf7afe68465] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:04:27.077 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:27.078 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:28.498 INFO 21062 --- [ocp-schedules-26,a67223777c30eaa5,44f563dd105a79ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:04:28.500 INFO 21062 --- [ocp-schedules-18,48638e5d539f5719,ba5d2c18014136d8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:04:28.500 INFO 21062 --- [ocp-schedules-1,99fa877b1066d7dd,595970b0e5795023] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:04:28.500 INFO 21062 --- [ocp-schedules-17,b34fbb0369dc0716,c12a24fbbba29a6d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:04:28.502 INFO 21062 --- [ocp-schedules-26,a67223777c30eaa5,44f563dd105a79ec] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:04:28.502 INFO 21062 --- [ocp-schedules-9,27f7cc425c591305,419091019db4701e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:04:28.503 INFO 21062 --- [ocp-schedules-18,48638e5d539f5719,ba5d2c18014136d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:28.503 INFO 21062 --- [ocp-schedules-17,b34fbb0369dc0716,c12a24fbbba29a6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:04:28.505 INFO 21062 --- [ocp-schedules-28,6cd7cf518f9d7b4b,504777708daf8c8d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:04:28.506 INFO 21062 --- [ocp-schedules-1,99fa877b1066d7dd,595970b0e5795023] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:04:28.510 INFO 21062 --- [ocp-schedules-30,8b52744fb572a6c4,01801b4b5a171db0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:04:28.510 INFO 21062 --- [ocp-schedules-19,78fb8c167aee7d9f,87922007ff8d241a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:04:28.510 INFO 21062 --- [ocp-schedules-30,8b52744fb572a6c4,01801b4b5a171db0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:04:28.510 INFO 21062 --- [ocp-schedules-19,78fb8c167aee7d9f,87922007ff8d241a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:04:28.510 INFO 21062 --- [ocp-schedules-29,74ebe9f5c5608d6c,578fbf4ea2ce3a05] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:04:28.510 INFO 21062 --- [ocp-schedules-29,74ebe9f5c5608d6c,578fbf4ea2ce3a05] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:04:28.543 INFO 21062 --- [ocp-schedules-24,54136a5d81c2ee68,c2f56c9868605c5d] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:04:28.543+08:00, tenantSize=0 2024-11-04 13:04:28.561 INFO 21062 --- [ocp-schedules-21,c6455fe955a4d953,ed918a76e4c3f4e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:04:28.561 INFO 21062 --- [ocp-schedules-21,c6455fe955a4d953,ed918a76e4c3f4e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:04:28.568 INFO 21062 --- [ocp-async-9,14fb2a7805d0dafb,c7e4f21aa7fd9030] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:04:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:04:28.587798+08:00, lastProcessTime=2024-11-04T13:04:18.589079+08:00 2024-11-04 13:04:28.593 INFO 21062 --- [ocp-schedules-27,e0b66c19e8dd867f,f5fbefdc9d1862de] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 13:04:28.593 INFO 21062 --- [ocp-schedules-27,e0b66c19e8dd867f,f5fbefdc9d1862de] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 13:04:28.637 INFO 21062 --- [ocp-schedules-2,57842fdf1898476e,2b6fe362b9ea7bbb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:04:28.697 INFO 21062 --- [ocp-schedules-23,1306134630d57414,06273fa10947db28] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:04:28.697 INFO 21062 --- [ocp-schedules-23,1306134630d57414,06273fa10947db28] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:04:28.709 WARN 21062 --- [ocp-async-3,4f5663aa0dedb563,dbb4f02555f0c62a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:28.711 INFO 21062 --- [ocp-schedules-9,93dfa9b4ea3d2f92,866182e4046f1603] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:04:28.738 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:04:28.738 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:04:28.743 INFO 21062 --- [ocp-async-5,af8fb51912331ca8,a2b80554ae557b43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:28.749 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:04:28.749 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:04:28.749 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:04:28.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:04:28.587798+08:00 2024-11-04 13:04:28.759 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:04:28.759 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:04:28.759 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:04:28.762 INFO 21062 --- [ocp-schedules-19,1591e2e9f1d596e2,423ace678ba1f52c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:04:28.763 INFO 21062 --- [ocp-schedules-1,a30fbb7324a13d51,378d7b09903688f4] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:04:28.766 INFO 21062 --- [ocp-schedules-19,1591e2e9f1d596e2,423ace678ba1f52c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:04:28.766 INFO 21062 --- [ocp-schedules-19,1591e2e9f1d596e2,423ace678ba1f52c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:04:28.907 INFO 21062 --- [ocp-schedules-24,0c34b294c4dc9766,6acb5291f9800f4c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:04:28.907 INFO 21062 --- [ocp-schedules-24,0c34b294c4dc9766,6acb5291f9800f4c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:04:28.908 INFO 21062 --- [ocp-schedules-24,0c34b294c4dc9766,6acb5291f9800f4c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:04:28.908 INFO 21062 --- [ocp-schedules-24,0c34b294c4dc9766,6acb5291f9800f4c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:04:28.908 INFO 21062 --- [ocp-schedules-24,0c34b294c4dc9766,6acb5291f9800f4c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:04:28.908 INFO 21062 --- [ocp-schedules-24,0c34b294c4dc9766,6acb5291f9800f4c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:04:29.050 INFO 21062 --- [ocp-schedules-22,33613854110dea19,b33787efef105539] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:04:29.400 INFO 21062 --- [ocp-schedules-20,ca06e3d10bd7ba02,8c5947c0bd3fe83b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:04:29.400 INFO 21062 --- [ocp-schedules-20,ca06e3d10bd7ba02,8c5947c0bd3fe83b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:04:29.402 INFO 21062 --- [ocp-schedules-8,4d8b79cc97677c49,2f642c2fa09100d8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:04:29.408 INFO 21062 --- [ocp-schedules-8,4d8b79cc97677c49,2f642c2fa09100d8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:04:29.408 INFO 21062 --- [ocp-schedules-8,4d8b79cc97677c49,2f642c2fa09100d8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:04:29.427 INFO 21062 --- [ocp-schedules-11,8477b4570c443b89,efd934343f0a50db] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:04:29.427 INFO 21062 --- [ocp-schedules-20,ca06e3d10bd7ba02,8c5947c0bd3fe83b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:04:29.427 INFO 21062 --- [ocp-schedules-20,ca06e3d10bd7ba02,8c5947c0bd3fe83b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:04:29.479 INFO 21062 --- [ocp-schedules-27,f9fa3f2e817fb365,c6689e1c8abb8622] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:04:29.482 INFO 21062 --- [ocp-schedules-27,f9fa3f2e817fb365,c6689e1c8abb8622] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:29.483 INFO 21062 --- [ocp-schedules-12,28f1eb18fc8c179c,afad849ac9e10ec6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:04:29.483 INFO 21062 --- [ocp-schedules-12,28f1eb18fc8c179c,afad849ac9e10ec6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:04:29.483 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:04:29.483 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:04:29.492 INFO 21062 --- [ocp-schedules-12,28f1eb18fc8c179c,afad849ac9e10ec6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:04:29.492 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:04:29.493 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:04:29.494 WARN 21062 --- [ocp-schedules-12,28f1eb18fc8c179c,afad849ac9e10ec6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:04:29.494 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:04:29.494 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:04:29.494 INFO 21062 --- [ocp-schedules-12,28f1eb18fc8c179c,afad849ac9e10ec6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:04:29.495 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:04:29.495 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:04:29.496 WARN 21062 --- [ocp-schedules-12,28f1eb18fc8c179c,afad849ac9e10ec6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:04:29.496 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:04:29.496 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:04:29.503 ERROR 21062 --- [ocp-schedules-12,28f1eb18fc8c179c,afad849ac9e10ec6] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:04:29.536 ERROR 21062 --- [ocp-schedules-27,f9fa3f2e817fb365,c6689e1c8abb8622] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:04:29.536 INFO 21062 --- [ocp-schedules-27,f9fa3f2e817fb365,c6689e1c8abb8622] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:04:29.540 INFO 21062 --- [ocp-schedules-2,ce6d43850e774609,deec5678e575a76e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:04:29.540+08:00, clusters=[1] 2024-11-04 13:04:29.543 INFO 21062 --- [ocp-schedules-2,ce6d43850e774609,deec5678e575a76e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:04:29.638 INFO 21062 --- [ocp-schedules-4,eb810474fe5d9156,2439429adbad3990] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:04:29.638+08:00, clusters=[1] 2024-11-04 13:04:29.642 INFO 21062 --- [ocp-schedules-4,eb810474fe5d9156,2439429adbad3990] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:04:29.983 INFO 21062 --- [prometheus-minute-schedule-1,fb72612d90562453,9cc665bae0e0c8db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:30.078 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:30.079 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:04:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:04:30.697 INFO 21062 --- [ocp-schedules-10,0a69192d58320de9,67009f88b7b512da] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:04:30.697 INFO 21062 --- [ocp-schedules-10,2a0dbf3a8b879ff3,bbaa17a4d912746b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:04:30.701 INFO 21062 --- [ocp-schedules-10,2a0dbf3a8b879ff3,bbaa17a4d912746b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:30.709 WARN 21062 --- [ocp-async-8,0fa6de5c4fd0cf37,b30401acc7688947] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:30.730 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:04:30.730 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:04:30.743 INFO 21062 --- [ocp-async-1,00253ba70f91d0e8,c8d2d0e51e36d953] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:31.432 INFO 21062 --- [ocp-schedules-8,3c631e529731471b,306092f82622cefc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:04:31.436 INFO 21062 --- [ocp-schedules-8,3c631e529731471b,306092f82622cefc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:31.522 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:04:31.524 ERROR 21062 --- [ocp-schedules-8,3c631e529731471b,306092f82622cefc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:04:31.524 INFO 21062 --- [ocp-schedules-8,3c631e529731471b,306092f82622cefc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:04:32.709 WARN 21062 --- [ocp-async-1,19a65728e335c4ad,0e1e9426a89f959c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:32.743 INFO 21062 --- [ocp-async-6,278af6d9a3dc5db8,6882496edee1e7df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:32.983 INFO 21062 --- [prometheus-minute-schedule-1,4fdb4bcb7ac10ae4,e8e510f99bf02167] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:33.079 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:33.080 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:34.709 WARN 21062 --- [ocp-async-6,87ede8b5293f0e9f,3748a02c04ce172c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:34.743 INFO 21062 --- [ocp-async-7,83f22ec3e5f19f86,ef6b309aff8c5abc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:35.983 INFO 21062 --- [prometheus-minute-schedule-1,483cc1090789b230,dc636ca6b0bf3159] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:36.080 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:36.081 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:36.287 INFO 21062 --- [ocp-schedules-22,179bc032e12f16ee,f3da907530a2be6a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:04:36.709 WARN 21062 --- [ocp-async-7,bba31c58c377756d,cbae011b585ee064] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:36.743 INFO 21062 --- [ocp-async-10,b310ed9f67edb3e8,f772343f49b96b7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:38.504 INFO 21062 --- [ocp-schedules-17,b34fbb0369dc0716,c12a24fbbba29a6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:04:38.504 INFO 21062 --- [ocp-schedules-18,48638e5d539f5719,ba5d2c18014136d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:04:38.568 INFO 21062 --- [ocp-async-10,a9160cad1063699f,7270b9d8c71be239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:04:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:04:38.589516+08:00, lastProcessTime=2024-11-04T13:04:28.587798+08:00 2024-11-04 13:04:38.709 WARN 21062 --- [ocp-async-2,3594ca77d171146d,5c3958bd50fab884] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:38.744 INFO 21062 --- [ocp-async-4,40cdaee2c2d069f0,cb8743a4404c8c88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:04:38.589516+08:00 2024-11-04 13:04:38.983 INFO 21062 --- [prometheus-minute-schedule-1,9e0eeece59a67eda,a894cb1364133cd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:39.053 INFO 21062 --- [ocp-schedules-7,4003c8be4a8f551e,55f43b44180fb595] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:04:39.081 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:39.082 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:39.339 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:04:39.339 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:04:39.341 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.342 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:04:39.343 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.343 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.344 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.344 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:04:39.361 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.362 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.363 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.363 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:04:39.376 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.376 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.377 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.377 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:04:39.391 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.392 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.392 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.393 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:04:39.406 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.406 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.407 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.407 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:04:39.413 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.413 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.414 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.414 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:04:39.419 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.419 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.419 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.420 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:04:39.424 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.425 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.425 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.426 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:04:39.431 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.431 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.432 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.432 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:04:39.438 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.438 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.439 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.439 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:04:39.444 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.445 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.445 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.446 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:04:39.454 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.454 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.455 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.455 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:04:39.468 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.469 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.469 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.470 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:04:39.475 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.475 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.476 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.476 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:04:39.482 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.482 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.482 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.483 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:04:39.488 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.489 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.489 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.490 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:04:39.494 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.494 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.495 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.495 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:04:39.499 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.500 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.500 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.501 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:04:39.505 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.506 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.506 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.506 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:04:39.511 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.511 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.512 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.512 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:04:39.516 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.516 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.517 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.517 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:04:39.522 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.522 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.523 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.523 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:04:39.527 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.528 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.531 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.531 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:04:39.536 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.536 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.537 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.537 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:04:39.541 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.542 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.542 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.542 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:04:39.547 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.547 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.548 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.548 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:04:39.552 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.553 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.553 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.554 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:04:39.558 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.558 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.559 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.559 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:04:39.563 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.564 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.564 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.565 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:04:39.569 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.569 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.569 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.570 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:04:39.574 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.574 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.574 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.575 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:04:39.579 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.579 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.580 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.580 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:04:39.585 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.585 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.586 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.586 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:04:39.590 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.590 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.591 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.591 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:04:39.598 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.599 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.599 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.600 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:04:39.607 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.607 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.608 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.608 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:04:39.614 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.614 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.615 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.616 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:04:39.622 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.622 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.623 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.623 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:04:39.627 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.627 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.628 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.628 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:04:39.632 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.632 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.633 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.634 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:04:39.637 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.638 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.638 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.638 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:04:39.642 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.642 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.643 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.643 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:04:39.653 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.653 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.654 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.654 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:04:39.675 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.675 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.676 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.676 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:04:39.682 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.682 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.682 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.683 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:04:39.689 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.690 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.690 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.690 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:04:39.696 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.697 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.697 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.697 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:04:39.701 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.702 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.702 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.702 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:04:39.706 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.707 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.708 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.708 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:04:39.712 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:04:39.712 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:04:39.714 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.714 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:04:39.716 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.716 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.716 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.717 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:04:39.722 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.722 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:04:39.723 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:39.724 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:04:39.728 INFO 21062 --- [ocp-schedules-18,50a516b2ded12da4,bf9e9e851279a676] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:04:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:04:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:04:40.710 WARN 21062 --- [ocp-async-4,c56156d2848cb416,b1009a5a9a5b539a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:40.744 INFO 21062 --- [ocp-async-8,00ac0038296fc64b,8c27ca3417b9ec4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:04:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:04:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:04:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:04:41.853 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:04:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:04:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bef1a3fd76b0357c,62fa8f1bedf8acb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:42.082 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:42.083 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:42.710 WARN 21062 --- [ocp-async-8,67d637d248f6f8c3,e550729a95e84feb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:42.748 INFO 21062 --- [ocp-async-1,0a473e5954ddf9bd,45f8ea268219c586] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:44.710 WARN 21062 --- [ocp-async-1,3fe6162e0c436b19,77442717e9b3eb1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:44.748 INFO 21062 --- [ocp-async-6,71927869ac83d57e,7849b643f6a2f942] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:44.983 INFO 21062 --- [prometheus-minute-schedule-1,03daf1c44e9b5e07,6c6a436adaa3ab29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:45.084 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:45.085 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:46.328 INFO 21062 --- [ocp-schedules-23,871f8a92e70246cf,ef19689342c782a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:04:46.710 WARN 21062 --- [ocp-async-6,26a3c25ad5802a53,667ed7e1a02bc6fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:46.748 INFO 21062 --- [ocp-async-7,0451fdb31c381a77,2903171efe6241f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:47.983 INFO 21062 --- [prometheus-minute-schedule-1,744cbc4583f78dd1,e1e1beff45063892] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:48.085 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:48.086 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:48.569 INFO 21062 --- [ocp-async-3,d1d1b22c01594d81,edab3a028591f5d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:04:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:04:48.588620+08:00, lastProcessTime=2024-11-04T13:04:38.589516+08:00 2024-11-04 13:04:48.710 WARN 21062 --- [ocp-async-8,830639c462b2db0b,6fb5a15919d80507] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:48.748 INFO 21062 --- [ocp-async-1,893d300cbe146f70,2b8f958108eb3f9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:04:48.588620+08:00 2024-11-04 13:04:49.056 INFO 21062 --- [ocp-schedules-9,cd3ed65599798d47,ca1c701090c83755] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:04:49.411 INFO 21062 --- [ocp-schedules-25,65b643cfb6ac9acc,c1ff14079c853acb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:04:49.418 INFO 21062 --- [ocp-schedules-25,65b643cfb6ac9acc,c1ff14079c853acb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:04:49.418 INFO 21062 --- [ocp-schedules-25,65b643cfb6ac9acc,c1ff14079c853acb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:04:50.710 WARN 21062 --- [ocp-async-1,70d0699538557d9b,d3e165908db33358] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:50.749 INFO 21062 --- [ocp-async-6,eaeefe6b4c684a89,d5a58cea5bd011b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1a586b004963f8bc,126d2a0659f28be7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:51.086 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:51.087 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:52.711 WARN 21062 --- [ocp-async-6,ae4ca4f31f48cf96,b71ce19d32b4f305] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:52.749 INFO 21062 --- [ocp-async-7,86ca3603368acd8b,ea861393680702ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:53.983 INFO 21062 --- [prometheus-minute-schedule-1,5daf6ee522fee0e2,88717809878a76c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:54.087 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:54.088 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:54.711 WARN 21062 --- [ocp-async-7,a4d56c5ec78223f2,8969a005d6838313] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:54.749 INFO 21062 --- [ocp-async-10,0869e1418b1067a0,1d25b1e18845eec4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:56.353 INFO 21062 --- [ocp-schedules-15,715851eba6111c33,2447f72a728495e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:04:56.711 WARN 21062 --- [ocp-async-10,338c6056b63381ec,66cc3b331262c71e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:56.782 INFO 21062 --- [ocp-async-9,8ce66e5111500372,bdc8130dad7f96bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e97fb8f31ca4a00f,87e0ff4984a1eb4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:04:57.088 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:04:57.089 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:04:58.499 INFO 21062 --- [ocp-schedules-7,57b0db70829fb8d9,9bd612135ed25cc5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:04:58.500 INFO 21062 --- [ocp-schedules-20,9aa362a6b8f7dc57,c725bf4efade41ab] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:04:58.503 INFO 21062 --- [ocp-schedules-2,69336cfe56f4cf77,42d36ab55c1eaa44] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:04:58.503 INFO 21062 --- [ocp-schedules-7,57b0db70829fb8d9,9bd612135ed25cc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:04:58.505 INFO 21062 --- [ocp-schedules-20,9aa362a6b8f7dc57,c725bf4efade41ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:58.505 INFO 21062 --- [ocp-schedules-5,32db05f39a252daf,fc5bd6b0e2ee9c7a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:04:58.510 INFO 21062 --- [ocp-schedules-28,5150a8b977264f43,b419b712d69f9596] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:04:58.510 INFO 21062 --- [ocp-schedules-23,873f422d346fb5b2,91e2bb417f44db1f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:04:58.510 INFO 21062 --- [ocp-schedules-28,5150a8b977264f43,b419b712d69f9596] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:04:58.510 INFO 21062 --- [ocp-schedules-23,873f422d346fb5b2,91e2bb417f44db1f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:04:58.511 INFO 21062 --- [ocp-schedules-22,a51c362bb1a72468,c4452fe181a22df1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:04:58.511 INFO 21062 --- [ocp-schedules-22,a51c362bb1a72468,c4452fe181a22df1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:04:58.569 INFO 21062 --- [ocp-async-2,97d8c46984dc45f7,696cd1c0f6a274c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:04:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:04:58.589279+08:00, lastProcessTime=2024-11-04T13:04:48.588620+08:00 2024-11-04 13:04:58.675 INFO 21062 --- [ocp-schedules-6,2f94478278db3b36,fb5145e677e7135d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:04:58.678 INFO 21062 --- [ocp-schedules-6,2f94478278db3b36,fb5145e677e7135d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:04:58.679 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:04:58.711 WARN 21062 --- [ocp-async-6,cea10148ba8ad9aa,4209c99f8e991fd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:04:58.715 INFO 21062 --- [ocp-schedules-26,6e8b0b460a7a4d11,744897fe2e4a0932] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:04:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:04:58.589279+08:00 2024-11-04 13:04:58.784 INFO 21062 --- [ocp-async-3,96e4ef9287d9310f,235cb3a783dfb889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:04:59.059 INFO 21062 --- [ocp-schedules-25,3d07715c2505df2a,d68ff1782510817f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:04:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7a2b4588f9a7543e,81769c6bbd412f11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696700, interval=60 2024-11-04 13:05:00.089 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:00.090 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:00.711 WARN 21062 --- [ocp-async-3,923f9a5f30878623,1f0fc14d2c2f4ff3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:00.730 INFO 21062 --- [ocp-schedules-10,2a0dbf3a8b879ff3,bbaa17a4d912746b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:05:00.730 INFO 21062 --- [ocp-schedules-10,220980da593957e1,b41edf5ada6808a2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:05:00.734 INFO 21062 --- [ocp-schedules-10,220980da593957e1,b41edf5ada6808a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:05:00.752 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,b4e9f09c9dbd678a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:00:01.0, nextRun=2024-11-04 13:05:00.0, currentDate=Mon Nov 04 13:05:00 CST 2024 2024-11-04 13:05:00.753 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,b4e9f09c9dbd678a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:00:01.0, nextRunTime=2024-11-04 13:05:00.0} 2024-11-04 13:05:00.764 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:05:00.764 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:05:00.779 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,b4e9f09c9dbd678a] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:05:00.783 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,b4e9f09c9dbd678a] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:05:00.783 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,b4e9f09c9dbd678a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:05:00.784 INFO 21062 --- [ocp-async-8,b5e2a8baa5d89f87,9895b25797ebb0f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:02.711 WARN 21062 --- [ocp-async-8,0fac9cbc361e1bf0,5c35ac8b65a6b326] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:02.784 INFO 21062 --- [ocp-async-1,c3e7bac198bb18ff,45727a5865c99124] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c05427c398c078b2,9b9adc8ae73cc880] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:03.090 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:03.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:04.712 WARN 21062 --- [ocp-async-1,f613b199f12afc6a,6104205cc11c06f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:04.784 INFO 21062 --- [ocp-async-4,4800fe0d3116a9d3,7d0a04195d712295] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:05.983 INFO 21062 --- [prometheus-minute-schedule-1,3a0bd4b743b92066,3c0e4f73681b4198] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:06.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:06.093 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:06.394 INFO 21062 --- [ocp-schedules-13,9e65df540c4baf46,6b9b76f89877a018] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:05:06.712 WARN 21062 --- [ocp-async-4,5293553e88241542,37ff5a56a75e435a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:06.784 INFO 21062 --- [ocp-async-5,9ac0974b00b7905f,72d85d4ee39eb27a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:08.504 INFO 21062 --- [ocp-schedules-7,57b0db70829fb8d9,9bd612135ed25cc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:05:08.505 INFO 21062 --- [ocp-schedules-20,9aa362a6b8f7dc57,c725bf4efade41ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:05:08.569 INFO 21062 --- [ocp-async-3,d8d243e2075ad5ec,1f75bd7c9ea784b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:05:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:05:08.589101+08:00, lastProcessTime=2024-11-04T13:04:58.589279+08:00 2024-11-04 13:05:08.679 INFO 21062 --- [ocp-schedules-6,2f94478278db3b36,fb5145e677e7135d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:05:08.712 WARN 21062 --- [ocp-async-8,15c1bb82cfe8f202,9c26aad9f785b9f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:05:08.589101+08:00 2024-11-04 13:05:08.784 INFO 21062 --- [ocp-async-9,8ffeb240b35d2e20,d7fbb8e021fc5cae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:08.983 INFO 21062 --- [prometheus-minute-schedule-1,81e8aa9f0b68dbea,fcbe33bfd81189d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:09.062 INFO 21062 --- [ocp-schedules-2,a7ea71f5d45aab7a,1d86834280a59fe2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:05:09.093 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:09.094 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:09.423 INFO 21062 --- [ocp-schedules-24,dde1943fa3e79c61,ac7917320f6954f3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:05:09.428 INFO 21062 --- [ocp-schedules-24,dde1943fa3e79c61,ac7917320f6954f3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:05:09.428 INFO 21062 --- [ocp-schedules-24,dde1943fa3e79c61,ac7917320f6954f3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:05:10.002 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:05:10.002 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:05:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:05:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:05:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:05:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:05:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:05:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:05:10.712 WARN 21062 --- [ocp-async-1,5c253122f5d23613,ad4f17fbac478d4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:10.788 INFO 21062 --- [ocp-async-7,828d1072a9ac6055,0b66fdcdf18bcb74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=10, affectRows=10 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=10, validCount=10 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=17, affectRows=17 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=7, affectRows=7 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=26, affectRows=26 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=7, validCount=7 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=17, validCount=17 2024-11-04 13:05:11.163 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=26, validCount=26 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=30, affectRows=30 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=39, affectRows=39 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=48, affectRows=48 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=43, affectRows=43 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=44, affectRows=44 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=30, validCount=30 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=39, validCount=39 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696640, nodeCount=58, affectRows=58 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=43, validCount=43 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=48, validCount=48 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=44, validCount=44 2024-11-04 13:05:11.164 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696640, nodeCount=55, validCount=55 2024-11-04 13:05:11.983 INFO 21062 --- [prometheus-minute-schedule-1,35d6b74853da917d,0c7be5d6f3c52fd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:12.094 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:12.095 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:12.712 WARN 21062 --- [ocp-async-7,cf2446ccd34d16ed,11751be195fbb190] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:12.788 INFO 21062 --- [ocp-async-5,366ec24af08b0c7b,20459c8016e96884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:13.985 INFO 21062 --- [prometheus-cache-monitor1,204fb29fe43652a7,816d0fd610037987] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:05:13.985 INFO 21062 --- [prometheus-cache-monitor1,204fb29fe43652a7,816d0fd610037987] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:05:14.330 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:01:14.31, end=2024-11-04T13:03:14.327} 2024-11-04 13:05:14.336 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:05:14.336 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:05:14.712 WARN 21062 --- [ocp-async-5,aa1dc854e86fe7fd,1891a83d50a44c3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:14.788 INFO 21062 --- [ocp-async-2,81539060f01700cc,ade8830194da7b89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3e9f0e35fe55de52,d79007455ea9545d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:15.095 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:15.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:16.423 INFO 21062 --- [ocp-schedules-17,ca174f6ef720bff7,897604ee5ebb0086] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:05:16.713 WARN 21062 --- [ocp-async-2,e436ca3e76877c79,ca1ca7a9d1e83672] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:16.788 INFO 21062 --- [ocp-async-6,b28703711eb37986,b7b4c9218e8a5b83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d6cc7e387613648b,2f2ff4108f7868ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:18.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:18.097 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:18.569 INFO 21062 --- [ocp-async-1,807af0da119d035a,445416adf7cd3104] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:05:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:05:18.589079+08:00, lastProcessTime=2024-11-04T13:05:08.589101+08:00 2024-11-04 13:05:18.713 WARN 21062 --- [ocp-async-4,92fe95c7aa13d6ad,de25c4a85c82e48b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:05:18.589079+08:00 2024-11-04 13:05:18.789 INFO 21062 --- [ocp-async-5,ffbf0639dfd34a40,38087959d3e11894] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:19.065 INFO 21062 --- [ocp-schedules-28,30d080cf13b269d9,59aeef915044c446] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:05:20.713 WARN 21062 --- [ocp-async-5,769c4c63c622848c,50125c6b4841a6e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:20.789 INFO 21062 --- [ocp-async-2,3ee7142d47d534d9,a1871dfa0225c879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:20.983 INFO 21062 --- [prometheus-minute-schedule-1,d829eb4e85970101,3a2da93a13715fd9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:21.098 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:21.098 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:22.713 WARN 21062 --- [ocp-async-2,45cd69628f6a2312,e73d4a6f1210e2a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:22.789 INFO 21062 --- [ocp-async-6,8dd886e31bea6ab9,235d424ac7648e0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:23.983 INFO 21062 --- [prometheus-minute-schedule-1,bd7f99150ef67c45,c44c8c08c215a201] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:24.098 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:24.099 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:24.713 WARN 21062 --- [ocp-async-6,8ffa3218c7c1e09e,9d72109bb4457f28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:24.789 INFO 21062 --- [ocp-async-3,c60c3a7b18f77632,e44aed7ee92273a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:26.464 INFO 21062 --- [ocp-schedules-16,892a15f6ea30159c,7fe23d4498fa6fc2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:05:26.714 WARN 21062 --- [ocp-async-3,62ad14a7808bf0c0,3aedf2e707f02b39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:26.789 INFO 21062 --- [ocp-async-8,61c87e297c7af045,8ed656c8d2c5f3a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ac62fa330626a50f,4895e8b1cefb27f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:26.984 INFO 21062 --- [prometheus-hour-schedule-1,1347617c2c6b4683,006e04df95dbe339] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:05:27.099 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:27.100 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:28.498 INFO 21062 --- [ocp-schedules-3,87724f99c51b4359,5546506066b445a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:05:28.499 INFO 21062 --- [ocp-schedules-23,6c69e9c2a8d57b65,dc61e47c962f88ff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:05:28.500 INFO 21062 --- [ocp-schedules-15,2c7056519a94141c,0ff134c3b5de9c50] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:05:28.500 INFO 21062 --- [ocp-schedules-25,0ba8597ee748d213,9a84a89839de4a23] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:05:28.502 INFO 21062 --- [ocp-schedules-3,87724f99c51b4359,5546506066b445a7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:05:28.502 INFO 21062 --- [ocp-schedules-4,6d921beb167dd1d1,39c779ae5d37b982] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:05:28.503 INFO 21062 --- [ocp-schedules-23,6c69e9c2a8d57b65,dc61e47c962f88ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:05:28.503 INFO 21062 --- [ocp-schedules-25,0ba8597ee748d213,9a84a89839de4a23] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:05:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:05:28.504 INFO 21062 --- [ocp-schedules-9,8349c3dbac537317,5d4eb9fc5cfafae3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:05:28.506 INFO 21062 --- [ocp-schedules-15,2c7056519a94141c,0ff134c3b5de9c50] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:05:28.510 INFO 21062 --- [ocp-schedules-13,0ae798a88b86a4f5,90be389dc4b880cc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:05:28.510 INFO 21062 --- [ocp-schedules-29,048acd93295a3691,bfe7b85bd59ff5ac] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:05:28.510 INFO 21062 --- [ocp-schedules-18,adc1d4ac88ea3e1b,16cd6a31e71c7507] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:05:28.510 INFO 21062 --- [ocp-schedules-29,048acd93295a3691,bfe7b85bd59ff5ac] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:05:28.510 INFO 21062 --- [ocp-schedules-18,adc1d4ac88ea3e1b,16cd6a31e71c7507] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:05:28.510 INFO 21062 --- [ocp-schedules-13,0ae798a88b86a4f5,90be389dc4b880cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:05:28.569 INFO 21062 --- [ocp-async-2,3d51cfeb77aee605,0453d77db2f20b79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:05:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:05:28.588411+08:00, lastProcessTime=2024-11-04T13:05:18.589079+08:00 2024-11-04 13:05:28.640 INFO 21062 --- [ocp-schedules-12,00b9ea01e86e79ec,16eaeb0db57662ed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:05:28.701 INFO 21062 --- [ocp-schedules-16,64a4771bc67983a5,26252ec02ce8a4b5] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:05:28.701 INFO 21062 --- [ocp-schedules-16,64a4771bc67983a5,26252ec02ce8a4b5] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:05:28.714 WARN 21062 --- [ocp-async-1,11756cfaf9624855,cc52e08efc97d69e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:28.717 INFO 21062 --- [ocp-schedules-11,0055f3099dfd40ec,614b6897e33f85de] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:05:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:05:28.588411+08:00 2024-11-04 13:05:28.766 INFO 21062 --- [ocp-schedules-15,bb482705f09ddbd3,e7bccc7aed86f05e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:05:28.772 INFO 21062 --- [ocp-schedules-15,bb482705f09ddbd3,e7bccc7aed86f05e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:05:28.772 INFO 21062 --- [ocp-schedules-15,bb482705f09ddbd3,e7bccc7aed86f05e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:05:28.789 INFO 21062 --- [ocp-async-7,0b52d560fc9fd0f7,53a2ec227dc7ab3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:28.909 INFO 21062 --- [ocp-schedules-18,142d752db8fc09b9,fd06a6c8e76c9805] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:05:28.909 INFO 21062 --- [ocp-schedules-18,142d752db8fc09b9,fd06a6c8e76c9805] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:05:28.911 INFO 21062 --- [ocp-schedules-18,142d752db8fc09b9,fd06a6c8e76c9805] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:05:28.911 INFO 21062 --- [ocp-schedules-18,142d752db8fc09b9,fd06a6c8e76c9805] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:05:28.912 INFO 21062 --- [ocp-schedules-18,142d752db8fc09b9,fd06a6c8e76c9805] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:05:28.912 INFO 21062 --- [ocp-schedules-18,142d752db8fc09b9,fd06a6c8e76c9805] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:05:29.068 INFO 21062 --- [ocp-schedules-3,3968be63ff29a492,27a1555be84bf907] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:05:29.428 INFO 21062 --- [ocp-schedules-2,d1c91dba26ca0e3b,ab9a7911b5a51062] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:05:29.428 INFO 21062 --- [ocp-schedules-2,d1c91dba26ca0e3b,ab9a7911b5a51062] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:05:29.432 INFO 21062 --- [ocp-schedules-24,f8024c46093f32a1,20c06e5fba7f4d04] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:05:29.438 INFO 21062 --- [ocp-schedules-24,f8024c46093f32a1,20c06e5fba7f4d04] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:05:29.438 INFO 21062 --- [ocp-schedules-24,f8024c46093f32a1,20c06e5fba7f4d04] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:05:29.455 INFO 21062 --- [ocp-schedules-2,d1c91dba26ca0e3b,ab9a7911b5a51062] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:05:29.455 INFO 21062 --- [ocp-schedules-2,d1c91dba26ca0e3b,ab9a7911b5a51062] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:05:29.456 INFO 21062 --- [ocp-schedules-30,713ce05f4d23236d,947c63623ef54a2c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:05:29.512 INFO 21062 --- [ocp-schedules-21,a10722c8f23ce9d4,e08f58bb5f020541] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:05:29.512 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:05:29.512 INFO 21062 --- [ocp-schedules-21,a10722c8f23ce9d4,e08f58bb5f020541] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:05:29.515 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:05:29.521 INFO 21062 --- [ocp-schedules-21,a10722c8f23ce9d4,e08f58bb5f020541] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:05:29.522 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:05:29.523 WARN 21062 --- [ocp-schedules-21,a10722c8f23ce9d4,e08f58bb5f020541] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:05:29.524 INFO 21062 --- [ocp-schedules-21,a10722c8f23ce9d4,e08f58bb5f020541] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:05:29.524 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:05:29.525 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:05:29.525 WARN 21062 --- [ocp-schedules-21,a10722c8f23ce9d4,e08f58bb5f020541] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:05:29.526 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:05:29.526 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:05:29.526 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:05:29.527 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:05:29.528 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:05:29.532 ERROR 21062 --- [ocp-schedules-21,a10722c8f23ce9d4,e08f58bb5f020541] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:05:29.569 INFO 21062 --- [ocp-schedules-17,3d5793904ff1bf89,08d0d37c20ce9d10] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:05:29.569+08:00, clusters=[1] 2024-11-04 13:05:29.571 INFO 21062 --- [ocp-schedules-17,3d5793904ff1bf89,08d0d37c20ce9d10] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:05:29.669 INFO 21062 --- [ocp-schedules-16,227447394352a6fe,b56c88c6304e527f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:05:29.669+08:00, clusters=[1] 2024-11-04 13:05:29.672 INFO 21062 --- [ocp-schedules-16,227447394352a6fe,b56c88c6304e527f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:05:29.983 INFO 21062 --- [prometheus-minute-schedule-1,01d0670a3e50f727,9a22c6f61eba3b8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:30.101 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:30.102 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:05:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:05:30.714 WARN 21062 --- [ocp-async-7,34e219af3cb3c587,68353d824c3e543b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:30.764 INFO 21062 --- [ocp-schedules-10,220980da593957e1,b41edf5ada6808a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:05:30.764 INFO 21062 --- [ocp-schedules-10,8e4929ffd3ac9be8,1376325b2e667755] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:05:30.768 INFO 21062 --- [ocp-schedules-10,8e4929ffd3ac9be8,1376325b2e667755] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:05:30.789 INFO 21062 --- [ocp-async-5,09e70cb977b87237,10ffca9305ecd5fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:30.797 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:05:30.797 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:05:31.524 INFO 21062 --- [ocp-schedules-19,bf2b517470579984,f521edb895585aa3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:05:31.528 INFO 21062 --- [ocp-schedules-19,bf2b517470579984,f521edb895585aa3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:05:31.613 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:05:31.615 ERROR 21062 --- [ocp-schedules-19,bf2b517470579984,f521edb895585aa3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:05:31.615 INFO 21062 --- [ocp-schedules-19,bf2b517470579984,f521edb895585aa3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:05:32.714 WARN 21062 --- [ocp-async-5,14a61e5148e42a63,a5eaf7bf51aa9fa0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:32.790 INFO 21062 --- [ocp-async-9,6e8f6ddabaebfcac,beaeb55efcbea4e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b518c5c8dc5bf95f,a64208ddd06e581d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:33.103 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:33.103 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:34.714 WARN 21062 --- [ocp-async-9,34188959d89a7c48,604fd268459e6734] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:34.790 INFO 21062 --- [ocp-async-4,c56ab7cfa354496a,2c8b72ba64bb779c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:35.983 INFO 21062 --- [prometheus-minute-schedule-1,07db3cde166daa36,1b39712263475058] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:36.104 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:36.105 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:36.488 INFO 21062 --- [ocp-schedules-11,69fcd900f847b0d2,921363282356de40] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:05:36.714 WARN 21062 --- [ocp-async-4,d9df720ad36bc038,db41db408f85402d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:36.790 INFO 21062 --- [ocp-async-10,df794caf1a632cef,81df187a8e3f9b3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:38.504 INFO 21062 --- [ocp-schedules-23,6c69e9c2a8d57b65,dc61e47c962f88ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:05:38.504 INFO 21062 --- [ocp-schedules-25,0ba8597ee748d213,9a84a89839de4a23] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:05:38.570 INFO 21062 --- [ocp-async-10,a546f1fc2071a6df,b15adbfa6514101e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:05:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:05:38.588917+08:00, lastProcessTime=2024-11-04T13:05:28.588411+08:00 2024-11-04 13:05:38.715 WARN 21062 --- [ocp-async-8,72f25703f40d9a22,bc5c0bcf843bf2fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:05:38.588917+08:00 2024-11-04 13:05:38.790 INFO 21062 --- [ocp-async-6,255047b2afe515a3,4034f50c21cf7439] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:38.983 INFO 21062 --- [prometheus-minute-schedule-1,2b8155b54e18521b,ee78e9d25bd55995] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:39.073 INFO 21062 --- [ocp-schedules-26,dd1bc6bbaa7e1794,a93c320881e255fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:05:39.105 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:39.106 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:40.004 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:05:40.004 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:05:40.715 WARN 21062 --- [ocp-async-6,046c4d1c93acf915,ec94906f0ffc788d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:40.790 INFO 21062 --- [ocp-async-7,0ae818561bb72a87,3e43d4b84af61137] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:41.983 INFO 21062 --- [prometheus-minute-schedule-1,1c2fc3cad89e2dfd,2fcf8da212a7c67b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:42.106 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:42.107 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:42.715 WARN 21062 --- [ocp-async-7,ed92cdc5f127369d,673e573999aac9d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:42.791 INFO 21062 --- [ocp-async-5,af0e748b04f6771e,5b393a4e469cae61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:44.715 WARN 21062 --- [ocp-async-5,eb411a35bc7e2305,87c16f46737f2826] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:44.791 INFO 21062 --- [ocp-async-9,7cf26c6b6739862c,82db62ccd1527984] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:44.983 INFO 21062 --- [prometheus-minute-schedule-1,c3977b3e2eb2e511,665651b7e5157c6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:45.107 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:45.108 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:46.528 INFO 21062 --- [ocp-schedules-22,c7ba46a7351a8a6f,d842281ad38e52ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:05:46.715 WARN 21062 --- [ocp-async-9,1e46d0458be80819,411cc02e844b842b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:46.791 INFO 21062 --- [ocp-async-4,91ca3c5a1b31b8e8,fe4f2a2b764f89e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:47.983 INFO 21062 --- [prometheus-minute-schedule-1,56bb1692c4825c97,56b8d39c3be90471] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:48.108 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:48.109 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:48.570 INFO 21062 --- [ocp-async-6,36d61105501ccb23,19fb3ae36c7a6de2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:05:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:05:48.589079+08:00, lastProcessTime=2024-11-04T13:05:38.588917+08:00 2024-11-04 13:05:48.715 WARN 21062 --- [ocp-async-7,1bb36d1e51441dcf,61748c68cbc0ea97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:05:48.589079+08:00 2024-11-04 13:05:48.791 INFO 21062 --- [ocp-async-5,36d52b712f85b3cc,39051e794ba5d1a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:49.076 INFO 21062 --- [ocp-schedules-16,fcf9f4af572577dd,b376e7d34703c252] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:05:49.440 INFO 21062 --- [ocp-schedules-25,224978ae4834ae82,ed6dd806eff3bae7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:05:49.446 INFO 21062 --- [ocp-schedules-25,224978ae4834ae82,ed6dd806eff3bae7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:05:49.446 INFO 21062 --- [ocp-schedules-25,224978ae4834ae82,ed6dd806eff3bae7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:05:50.716 WARN 21062 --- [ocp-async-5,a4b09193c2e3404a,e12b803c727bf84d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:50.764 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:05:50.764 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:05:50.772 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:05:50.772 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:05:50.772 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:05:50.772 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:05:50.779 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:05:50.779 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:05:50.779 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:05:50.779 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:05:50.791 INFO 21062 --- [ocp-async-9,5553382517490546,da2b7227d690cb0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:50.819 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:05:50.827 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:05:50.827 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:05:50.827 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:05:50.827 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:05:50.827 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:05:50.835 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:05:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a8c41ac42581d576,cfdc5210431e0058] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:51.109 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:51.110 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:51.396 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:05:51.963 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.184428541S 2024-11-04 13:05:51.964 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:05:52.067 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:05:52.067 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:05:52.094 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:05:52.161 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:05:52.161 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:05:52.161 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:05:52.161 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:05:52.161 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:05:52.161 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:05:52.172 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:05:52.354 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] o.s.boot.SpringApplication : Started application in 1.582 seconds (JVM running for 2166.034) 2024-11-04 13:05:52.355 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.582595604S 2024-11-04 13:05:52.716 WARN 21062 --- [ocp-async-9,79340f64b9357f64,3a04ee88b40883e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:52.792 INFO 21062 --- [ocp-async-4,2ac7211c011584a1,01f81315d4bd5432] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:52.821 INFO 21062 --- [ocp-schedules-17,ba2aa9d4dbb7b963,9086492a45aadcf3] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:05:53.983 INFO 21062 --- [prometheus-minute-schedule-1,2e0ea79a95618267,00167af7d574f89f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:54.110 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:54.111 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:54.716 WARN 21062 --- [ocp-async-4,60476bf08bfbb9de,aa86cdb12b95caa8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:54.792 INFO 21062 --- [ocp-async-10,611895fe95b67d8c,01c6b799bb3cf24b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:56.553 INFO 21062 --- [ocp-schedules-9,5c8499c9bb92db74,60c15844caa1184b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:05:56.716 WARN 21062 --- [ocp-async-10,5c8c50b2b507f660,a1e81a29a6432ddb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:56.792 INFO 21062 --- [ocp-async-2,72fe56af9cc5fe22,8a4394235cddf2f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:56.983 INFO 21062 --- [prometheus-minute-schedule-1,bb875f6c0089abb5,b1a1443fe72856a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:05:57.112 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:05:57.112 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:05:58.500 INFO 21062 --- [ocp-schedules-28,62009c4f06a566e4,93e5a2a81eb06799] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:05:58.500 INFO 21062 --- [ocp-schedules-12,900b3a9b4b52b79c,a30ab997e9a96f5d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:05:58.502 INFO 21062 --- [ocp-schedules-22,96c905ebcdd2e5c9,5f0f28cc8f9c2326] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:05:58.504 INFO 21062 --- [ocp-schedules-28,62009c4f06a566e4,93e5a2a81eb06799] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:05:58.504 INFO 21062 --- [ocp-schedules-12,900b3a9b4b52b79c,a30ab997e9a96f5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:05:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:05:58.505 INFO 21062 --- [ocp-schedules-7,b79508c638322b56,737bba1708a23d9e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:05:58.513 INFO 21062 --- [ocp-schedules-27,eef6c801b212a189,184e4cdd38b9fe37] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:05:58.513 INFO 21062 --- [ocp-schedules-30,906a71dde786f8fd,98894ba11d19688a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:05:58.513 INFO 21062 --- [ocp-schedules-30,906a71dde786f8fd,98894ba11d19688a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:05:58.513 INFO 21062 --- [ocp-schedules-27,eef6c801b212a189,184e4cdd38b9fe37] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:05:58.514 INFO 21062 --- [ocp-schedules-21,bd4f6e0d3e9846c4,c467a8c8b4155c2b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:05:58.514 INFO 21062 --- [ocp-schedules-21,bd4f6e0d3e9846c4,c467a8c8b4155c2b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:05:58.570 INFO 21062 --- [ocp-async-5,1347edb56efda043,00d4af4853cbdeb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:05:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:05:58.590137+08:00, lastProcessTime=2024-11-04T13:05:48.589079+08:00 2024-11-04 13:05:58.716 WARN 21062 --- [ocp-async-9,0dd9b20bf63ef43c,6d88c1ca491350b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:05:58.719 INFO 21062 --- [ocp-schedules-5,88f09d41b2fbe145,47938089a8f8c427] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:05:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:05:58.590137+08:00 2024-11-04 13:05:58.792 INFO 21062 --- [ocp-async-6,343fcc6dcfec7158,a53eb2e6e243b560] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:05:59.079 INFO 21062 --- [ocp-schedules-23,24895346522ba6c3,b2df78ce9746f239] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:05:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e77475b9b8447328,82d80e950403b1e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696760, interval=60 2024-11-04 13:06:00.113 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:00.113 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:00.717 WARN 21062 --- [ocp-async-6,6d22e914dbb8c971,dbbe9bfcbf44c2b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:00.792 INFO 21062 --- [ocp-async-3,96b2be5347c259ca,6d14c15341af0c44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:00.798 INFO 21062 --- [ocp-schedules-10,8e4929ffd3ac9be8,1376325b2e667755] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:06:00.798 INFO 21062 --- [ocp-schedules-10,a37a2fda64f81922,fec214cff6c9991c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:06:00.801 INFO 21062 --- [ocp-schedules-10,a37a2fda64f81922,fec214cff6c9991c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:00.830 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:06:00.830 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:06:02.717 WARN 21062 --- [ocp-async-3,3f8013cde7b550f7,42ee8f2008ec56ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:02.793 INFO 21062 --- [ocp-async-8,f442943a2edfea61,6079271b448131b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:02.983 INFO 21062 --- [prometheus-minute-schedule-1,25a918ca656d36ef,a0156e1fadb9b415] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:03.114 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:03.115 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:04.717 WARN 21062 --- [ocp-async-8,34f715fe0c13a5da,db9ce3161194f62b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:04.793 INFO 21062 --- [ocp-async-1,d78757fa01ba63f0,2b195378101b318a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:05.983 INFO 21062 --- [prometheus-minute-schedule-1,022398076553f79e,bb4623707f013b43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:06.115 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:06.116 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:06.595 INFO 21062 --- [ocp-schedules-20,cac22ae4287be476,0a162e0470407ba3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:06:06.717 WARN 21062 --- [ocp-async-4,5ebfaf7aa9df034f,38ef393628235a3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:06.793 INFO 21062 --- [ocp-async-7,5025fb70bfa96906,70926c5707554ff9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:08.505 INFO 21062 --- [ocp-schedules-12,900b3a9b4b52b79c,a30ab997e9a96f5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:06:08.505 INFO 21062 --- [ocp-schedules-28,62009c4f06a566e4,93e5a2a81eb06799] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:06:08.570 INFO 21062 --- [ocp-async-6,d25359fefe6bc902,885e84a9bbb5cb03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:06:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:06:08.589806+08:00, lastProcessTime=2024-11-04T13:05:58.590137+08:00 2024-11-04 13:06:08.679 INFO 21062 --- [ocp-schedules-3,ef741a6a794ef3ed,56560e579fc5583c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:06:08.683 INFO 21062 --- [ocp-schedules-3,ef741a6a794ef3ed,56560e579fc5583c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:08.683 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:06:08.717 WARN 21062 --- [ocp-async-3,9c2312d66fff92e4,952d4958aa6ea1a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:06:08.589806+08:00 2024-11-04 13:06:08.793 INFO 21062 --- [ocp-async-2,91582556a483314d,509414c313e275fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:08.983 INFO 21062 --- [prometheus-minute-schedule-1,287d27f466f76734,07c2d55152f6454b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:09.082 INFO 21062 --- [ocp-schedules-23,9582875d619c4aae,fcec84c3e094d5cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:06:09.116 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:09.117 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:09.451 INFO 21062 --- [ocp-schedules-29,2cf64eb65b1bf695,392e125186f6f5b5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:06:09.457 INFO 21062 --- [ocp-schedules-29,2cf64eb65b1bf695,392e125186f6f5b5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:06:09.457 INFO 21062 --- [ocp-schedules-29,2cf64eb65b1bf695,392e125186f6f5b5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:06:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:06:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:06:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:06:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:06:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:06:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:06:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:06:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:06:10.717 WARN 21062 --- [ocp-async-8,2f25732139ce7f3f,d8a9585ed6965929] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:10.793 INFO 21062 --- [ocp-async-1,e247904805e8a338,3e2708ccba7e6728] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:11.166 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=6, affectRows=6 2024-11-04 13:06:11.166 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=6, validCount=6 2024-11-04 13:06:11.166 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=16, affectRows=16 2024-11-04 13:06:11.166 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=11, affectRows=11 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=11, validCount=11 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=16, validCount=16 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=29, affectRows=29 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=36, affectRows=36 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=45, affectRows=45 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=29, validCount=29 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=36, validCount=36 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=58, affectRows=58 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=45, validCount=45 2024-11-04 13:06:11.167 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=56, validCount=56 2024-11-04 13:06:11.168 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=52, affectRows=52 2024-11-04 13:06:11.168 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=52, validCount=52 2024-11-04 13:06:11.168 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696700, nodeCount=69, affectRows=69 2024-11-04 13:06:11.169 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696700, nodeCount=68, validCount=68 2024-11-04 13:06:11.983 INFO 21062 --- [prometheus-minute-schedule-1,740dad22af291abc,7a4bea0aec7a562d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:12.117 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:12.118 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:12.718 WARN 21062 --- [ocp-async-1,bfcd26d66bb2430a,54877cdced9b6f0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:12.794 INFO 21062 --- [ocp-async-7,29377a8e781a8e89,e782b15b58938fed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:13.984 INFO 21062 --- [prometheus-cache-monitor1,e8154b196c32a8d7,43cc0beb46b27a49] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:06:13.985 INFO 21062 --- [prometheus-cache-monitor1,e8154b196c32a8d7,43cc0beb46b27a49] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:06:14.718 WARN 21062 --- [ocp-async-7,9fe2738180b7f290,273e92f0e9613f2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:14.794 INFO 21062 --- [ocp-async-5,9cdb0752054607c6,672468d9ebc69407] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d61e7bb55befb947,e9ec8671c324f510] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:15.119 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:15.120 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:16.622 INFO 21062 --- [ocp-schedules-6,eb8aa3984ee57cf5,2a9f561eaf4279d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:06:16.718 WARN 21062 --- [ocp-async-5,65fae174af4509e2,3b89d8b59dfafa01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:16.794 INFO 21062 --- [ocp-async-9,929bc6b17bc893ff,aab3903277fae791] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:17.983 INFO 21062 --- [prometheus-minute-schedule-1,33853c0186e1618e,0bba48d00a89c756] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:18.120 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:18.121 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:18.570 INFO 21062 --- [ocp-async-8,ec13b7daddfe5d37,f05fe308f34e7730] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:06:18.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:06:18.588026+08:00, lastProcessTime=2024-11-04T13:06:08.589806+08:00 2024-11-04 13:06:18.684 INFO 21062 --- [ocp-schedules-3,ef741a6a794ef3ed,56560e579fc5583c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:06:18.718 WARN 21062 --- [ocp-async-1,35ecef9e503098d0,987d0646c2220fd1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:06:18.588026+08:00 2024-11-04 13:06:18.794 INFO 21062 --- [ocp-async-7,403073f0f8cb9f7e,d3a7728253308291] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:19.085 INFO 21062 --- [ocp-schedules-17,a865125064879820,975fd137b2553c03] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:06:20.718 WARN 21062 --- [ocp-async-7,bfdc899874947809,67c2ec8b9e98ee1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:20.795 INFO 21062 --- [ocp-async-5,bc78ced199a9e0fd,59359e74a652462d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a22c68438f81ccca,054214146b4088a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:21.121 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:21.122 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:22.718 WARN 21062 --- [ocp-async-5,80d852367bf70086,bb0c243084ad2b40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:22.795 INFO 21062 --- [ocp-async-9,363e7d1ad7d68ad3,887256df95bbd0e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ee001f8826a76212,a9bba94ef528c5b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:24.122 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:24.123 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:24.719 WARN 21062 --- [ocp-async-9,dc21ec7fa90407e5,5af64f41a944060e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:24.795 INFO 21062 --- [ocp-async-6,3708f534e7822292,2933c6e22a42b52c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:26.665 INFO 21062 --- [ocp-schedules-27,5e0394cc0ad04e34,878cb3d2e4408e69] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:06:26.719 WARN 21062 --- [ocp-async-6,ca8ef57f6ff7642d,1c991c54a7bcbf05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:26.795 INFO 21062 --- [ocp-async-3,94c4d43ada5b507e,646ad53f58515455] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:26.983 INFO 21062 --- [prometheus-minute-schedule-1,5330658fb4953fa5,fc4200f40f86ff1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:26.984 INFO 21062 --- [prometheus-hour-schedule-1,89683d66319f0273,6c5a2ecc9566d14d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:06:27.123 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:27.124 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:28.498 INFO 21062 --- [ocp-schedules-1,513462f5caa9a804,28c7a3eee109ab82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:06:28.499 INFO 21062 --- [ocp-schedules-20,304d8ec6ea31492c,41f5e681d43469d9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:06:28.500 INFO 21062 --- [ocp-schedules-28,190eb601cab3ea80,6f7b3417ff5cf0c7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:06:28.500 INFO 21062 --- [ocp-schedules-18,46bf87a19134f213,ea9c554dd96ca0e4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:06:28.502 INFO 21062 --- [ocp-schedules-7,1a5c49606e1800f2,019ac5c48f56289e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:06:28.503 INFO 21062 --- [ocp-schedules-1,513462f5caa9a804,28c7a3eee109ab82] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:06:28.503 INFO 21062 --- [ocp-schedules-28,190eb601cab3ea80,6f7b3417ff5cf0c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:28.504 INFO 21062 --- [ocp-schedules-18,46bf87a19134f213,ea9c554dd96ca0e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:28.504 INFO 21062 --- [ocp-schedules-27,1f58b7f6adfe1c8b,4708c250ad934b25] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:06:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:06:28.505 INFO 21062 --- [ocp-schedules-20,304d8ec6ea31492c,41f5e681d43469d9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:06:28.513 INFO 21062 --- [ocp-schedules-3,4d7d382884cf1453,18bdcf7ad809be03] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:06:28.513 INFO 21062 --- [ocp-schedules-30,1715ddb2395038b0,727c695032c1a4bd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:06:28.513 INFO 21062 --- [ocp-schedules-3,4d7d382884cf1453,18bdcf7ad809be03] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:06:28.513 INFO 21062 --- [ocp-schedules-30,1715ddb2395038b0,727c695032c1a4bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:06:28.514 INFO 21062 --- [ocp-schedules-24,2793d6d834ccf36f,07dddd0a928ea5b9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:06:28.514 INFO 21062 --- [ocp-schedules-24,2793d6d834ccf36f,07dddd0a928ea5b9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:06:28.561 INFO 21062 --- [ocp-schedules-25,fc37f779055443cd,2ba2def99c368696] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:06:28.561 INFO 21062 --- [ocp-schedules-25,fc37f779055443cd,2ba2def99c368696] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:06:28.571 INFO 21062 --- [ocp-async-5,ef0ab3ec555a0062,4429529feb8f881a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:06:28.571 INFO 21062 --- [ocp-schedules-26,9f6e996f31dd418d,91b38a3f6f5be19b] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:06:28.571 INFO 21062 --- [ocp-schedules-26,9f6e996f31dd418d,91b38a3f6f5be19b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:06:28.579 INFO 21062 --- [ocp-schedules-23,5d5640388e34aa1a,0e5cc23dc2c6490a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:06:28.580 INFO 21062 --- [ocp-schedules-23,5d5640388e34aa1a,0e5cc23dc2c6490a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:06:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:06:28.589041+08:00, lastProcessTime=2024-11-04T13:06:18.588026+08:00 2024-11-04 13:06:28.642 INFO 21062 --- [ocp-schedules-2,0676fa917deb9c10,da8456b2d0f04d27] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:06:28.709 INFO 21062 --- [ocp-schedules-15,62ba6c990ecb36a2,f1c20fbb9ef71b57] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:06:28.709 INFO 21062 --- [ocp-schedules-15,62ba6c990ecb36a2,f1c20fbb9ef71b57] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:06:28.719 WARN 21062 --- [ocp-async-8,de33ed90f421f7a6,8eaa7d0f34c34cbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:28.722 INFO 21062 --- [ocp-schedules-20,019f5452f752caad,eb8ac1410128bca0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:06:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:06:28.589041+08:00 2024-11-04 13:06:28.772 INFO 21062 --- [ocp-schedules-3,573a15f9e8d9e68f,f33daeb8d52d681d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:06:28.777 INFO 21062 --- [ocp-schedules-3,573a15f9e8d9e68f,f33daeb8d52d681d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:06:28.777 INFO 21062 --- [ocp-schedules-3,573a15f9e8d9e68f,f33daeb8d52d681d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:06:28.795 INFO 21062 --- [ocp-async-4,ee4e5e0496aaca35,a14262b909a4b6f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:28.913 INFO 21062 --- [ocp-schedules-16,b25926f84c368ecd,3f4e7060c6caa256] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:06:28.913 INFO 21062 --- [ocp-schedules-16,b25926f84c368ecd,3f4e7060c6caa256] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:06:28.919 INFO 21062 --- [ocp-schedules-16,b25926f84c368ecd,3f4e7060c6caa256] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:06:28.919 INFO 21062 --- [ocp-schedules-16,b25926f84c368ecd,3f4e7060c6caa256] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:06:28.925 INFO 21062 --- [ocp-schedules-16,b25926f84c368ecd,3f4e7060c6caa256] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:06:28.925 INFO 21062 --- [ocp-schedules-16,b25926f84c368ecd,3f4e7060c6caa256] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:06:29.087 INFO 21062 --- [ocp-schedules-25,444865d7a75103f3,62765a64a8ff2c71] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:06:29.455 INFO 21062 --- [ocp-schedules-26,0d6d93277f5fff29,83fa25bbe64fc00c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:06:29.455 INFO 21062 --- [ocp-schedules-26,0d6d93277f5fff29,83fa25bbe64fc00c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:06:29.459 INFO 21062 --- [ocp-schedules-22,290f94dbdb58121d,fc9a7e89aa0b4281] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:06:29.466 INFO 21062 --- [ocp-schedules-22,290f94dbdb58121d,fc9a7e89aa0b4281] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:06:29.466 INFO 21062 --- [ocp-schedules-22,290f94dbdb58121d,fc9a7e89aa0b4281] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:06:29.482 INFO 21062 --- [ocp-schedules-26,0d6d93277f5fff29,83fa25bbe64fc00c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:06:29.482 INFO 21062 --- [ocp-schedules-26,0d6d93277f5fff29,83fa25bbe64fc00c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:06:29.484 INFO 21062 --- [ocp-schedules-12,b74a73ad8e6e7519,6fce7f0f21b95254] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:06:29.536 INFO 21062 --- [ocp-schedules-11,67564136aed4a7f4,e8d796e8c70a1c27] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:06:29.540 INFO 21062 --- [ocp-schedules-11,67564136aed4a7f4,e8d796e8c70a1c27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:29.540 INFO 21062 --- [ocp-schedules-13,c8dd429598273c84,2a327596f2fd15c6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:06:29.540 INFO 21062 --- [ocp-schedules-13,c8dd429598273c84,2a327596f2fd15c6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:06:29.540 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:06:29.540 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:06:29.550 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:06:29.550 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:06:29.551 INFO 21062 --- [ocp-schedules-13,c8dd429598273c84,2a327596f2fd15c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:06:29.552 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:06:29.552 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:06:29.552 WARN 21062 --- [ocp-schedules-13,c8dd429598273c84,2a327596f2fd15c6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:06:29.553 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:06:29.553 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:06:29.553 INFO 21062 --- [ocp-schedules-13,c8dd429598273c84,2a327596f2fd15c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:06:29.554 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:06:29.554 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:06:29.554 WARN 21062 --- [ocp-schedules-13,c8dd429598273c84,2a327596f2fd15c6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:06:29.562 ERROR 21062 --- [ocp-schedules-13,c8dd429598273c84,2a327596f2fd15c6] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:06:29.595 INFO 21062 --- [ocp-schedules-19,d0238d91a9433fb7,19c0662985bf3a8b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:06:29.595+08:00, clusters=[1] 2024-11-04 13:06:29.596 INFO 21062 --- [ocp-schedules-19,d0238d91a9433fb7,19c0662985bf3a8b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:06:29.600 ERROR 21062 --- [ocp-schedules-11,67564136aed4a7f4,e8d796e8c70a1c27] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:06:29.600 INFO 21062 --- [ocp-schedules-11,67564136aed4a7f4,e8d796e8c70a1c27] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:06:29.698 INFO 21062 --- [ocp-schedules-21,259b8e769a55e807,493a148d3222e0fe] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:06:29.698+08:00, clusters=[1] 2024-11-04 13:06:29.701 INFO 21062 --- [ocp-schedules-21,259b8e769a55e807,493a148d3222e0fe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:06:29.983 INFO 21062 --- [prometheus-minute-schedule-1,912c40735d7871b5,339e0780562a13b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:30.124 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:30.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:06:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:06:30.719 WARN 21062 --- [ocp-async-4,6cef1f72e7fd0f9d,13fee7023369312e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:30.795 INFO 21062 --- [ocp-async-10,ac5ff34347e1af05,e35f78c6bbed2b16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:30.831 INFO 21062 --- [ocp-schedules-10,a37a2fda64f81922,fec214cff6c9991c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:06:30.831 INFO 21062 --- [ocp-schedules-10,46f4f4fdbc9f02a6,08c54b61bffc20cc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:06:30.834 INFO 21062 --- [ocp-schedules-10,46f4f4fdbc9f02a6,08c54b61bffc20cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:30.863 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:06:30.863 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:06:31.615 INFO 21062 --- [ocp-schedules-20,b433eceaad4eae94,438c483bc18502d1] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:06:31.619 INFO 21062 --- [ocp-schedules-20,b433eceaad4eae94,438c483bc18502d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:31.702 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:06:31.704 ERROR 21062 --- [ocp-schedules-20,b433eceaad4eae94,438c483bc18502d1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:06:31.704 INFO 21062 --- [ocp-schedules-20,b433eceaad4eae94,438c483bc18502d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:06:32.719 WARN 21062 --- [ocp-async-10,03b0f788bac769e5,752c2e74b4c4fef2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:32.796 INFO 21062 --- [ocp-async-2,17ee806c503f2f69,68d4addafc7792e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:32.983 INFO 21062 --- [prometheus-minute-schedule-1,4ebf1ca27a56f8fd,4904e429e1f6637a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:33.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:33.126 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:34.719 WARN 21062 --- [ocp-async-2,0ad7c17e5533bc0b,463e96afef48e6ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:34.796 INFO 21062 --- [ocp-async-9,5c6b3e45bd0735e3,dd99e94b9923e600] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:35.983 INFO 21062 --- [prometheus-minute-schedule-1,bdaf5ae5a8891bd5,3d4fb21582c91359] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:36.126 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:36.127 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:36.692 INFO 21062 --- [ocp-schedules-27,a52de80ca79a0035,baa3cba780de42ee] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:06:36.720 WARN 21062 --- [ocp-async-9,4b6827ac7656b7e1,0c6306bd9962bf28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:36.796 INFO 21062 --- [ocp-async-7,541a6f55134b97cd,db868616201e0e2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:38.505 INFO 21062 --- [ocp-schedules-18,46bf87a19134f213,ea9c554dd96ca0e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:06:38.506 INFO 21062 --- [ocp-schedules-28,190eb601cab3ea80,6f7b3417ff5cf0c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:06:38.571 INFO 21062 --- [ocp-async-6,21764974304f2b20,f465865a9a326364] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:06:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:06:38.588893+08:00, lastProcessTime=2024-11-04T13:06:28.589041+08:00 2024-11-04 13:06:38.720 WARN 21062 --- [ocp-async-3,80a5c891b9ac7bd6,641af748936fb4f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:06:38.588893+08:00 2024-11-04 13:06:38.796 INFO 21062 --- [ocp-async-8,ce1ff3317aafabbe,cdbe32ccc4b824c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:38.983 INFO 21062 --- [prometheus-minute-schedule-1,7642dfa1864d79c1,fd6092f95edee6e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:39.090 INFO 21062 --- [ocp-schedules-20,baad182a144cbf7a,1e5b8f441fb8e15a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:06:39.127 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:39.128 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:40.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:06:40.004 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:06:40.720 WARN 21062 --- [ocp-async-8,46a9e729b2ac9fda,6fffcf8c99c90e54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:40.796 INFO 21062 --- [ocp-async-4,9f8af2d5a9fce2ce,0410cf36e7456907] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:06:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:06:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:06:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:06:41.830 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:06:41.857 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:06:41.857 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:06:41.983 INFO 21062 --- [prometheus-minute-schedule-1,e5057de168f4bb9f,d074cf430722127c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:42.128 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:42.130 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:42.720 WARN 21062 --- [ocp-async-4,4148719accf7a108,60a17f68a06baac8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:42.796 INFO 21062 --- [ocp-async-10,08bf2d7205fb38fd,bc24c56aa7293c99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:44.720 WARN 21062 --- [ocp-async-10,44149c1fb41a2653,fa8c333b72202912] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:44.797 INFO 21062 --- [ocp-async-2,211abc68c04b4de4,03edbf3d4e3c1494] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:44.983 INFO 21062 --- [prometheus-minute-schedule-1,fcf2e4f07cdc56e3,42609ff7a6343bc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:45.130 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:45.131 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:46.720 WARN 21062 --- [ocp-async-2,f5d5f89220d5204b,cbb9d2d4af3f80fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:46.739 INFO 21062 --- [ocp-schedules-20,045b3637f50eba37,48eb37c323d52105] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:06:46.797 INFO 21062 --- [ocp-async-9,d65496fa171bdc5f,296b042f80e576e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:47.983 INFO 21062 --- [prometheus-minute-schedule-1,79786822fb74e887,ae2321d2a35ce401] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:48.131 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:48.132 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:48.571 INFO 21062 --- [ocp-async-8,f8a1294d040fe936,67be29b39f97e53e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:06:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:06:48.590838+08:00, lastProcessTime=2024-11-04T13:06:38.588893+08:00 2024-11-04 13:06:48.721 WARN 21062 --- [ocp-async-4,fd81c1c94d183650,cebbac8c22d07902] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:06:48.590838+08:00 2024-11-04 13:06:48.797 INFO 21062 --- [ocp-async-10,6b34891a90cd50bc,673510f0becebceb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:49.093 INFO 21062 --- [ocp-schedules-4,2262163be846e051,399d58f6a369b879] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:06:49.468 INFO 21062 --- [ocp-schedules-22,337d6b93802ffdc8,e3f9fa9a906fdea6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:06:49.474 INFO 21062 --- [ocp-schedules-22,337d6b93802ffdc8,e3f9fa9a906fdea6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:06:49.474 INFO 21062 --- [ocp-schedules-22,337d6b93802ffdc8,e3f9fa9a906fdea6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:06:50.721 WARN 21062 --- [ocp-async-10,b4c8762512fedccd,45036c0b67719533] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:50.797 INFO 21062 --- [ocp-async-2,5e499f32dcb057ce,c9ee8c01af6321a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a2fa3eeda7eec7f5,19d6d6329735561f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:51.132 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:51.133 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:52.721 WARN 21062 --- [ocp-async-2,ba4dde57091c32c7,3cae54279bb2cbed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:52.797 INFO 21062 --- [ocp-async-1,a03de1c2aec0eeed,c4ad807581d8d7c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:53.983 INFO 21062 --- [prometheus-minute-schedule-1,30a65175c5833fd6,5f86043171644858] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:54.133 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:54.134 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:54.721 WARN 21062 --- [ocp-async-1,5f9a9d5de74dbe93,cbe941482036a1cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:54.798 INFO 21062 --- [ocp-async-7,53724f75888033ec,c4e638a9bfc0e218] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:56.721 WARN 21062 --- [ocp-async-7,dc2f47a6a5c9e27f,7dd6ccaafac2bd74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:56.764 INFO 21062 --- [ocp-schedules-16,6b0105063afc5cad,fa8d75199c31ca67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:06:56.798 INFO 21062 --- [ocp-async-3,87aa1d1d9d8a927d,6c0db66ea2cc7ad8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:56.983 INFO 21062 --- [prometheus-minute-schedule-1,20e51821cdf4a691,22e5a87f6f1dccb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:06:57.134 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:06:57.135 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:06:58.500 INFO 21062 --- [ocp-schedules-2,4f82000aac0e02bc,0ddd40796ba0e1b6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:06:58.500 INFO 21062 --- [ocp-schedules-15,7321a39056f376f1,c80a4b204c87b2a8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:06:58.502 INFO 21062 --- [ocp-schedules-28,e6ee4643c760b845,3a30773b54297e80] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:06:58.504 INFO 21062 --- [ocp-schedules-2,4f82000aac0e02bc,0ddd40796ba0e1b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:06:58.504 INFO 21062 --- [ocp-schedules-15,7321a39056f376f1,c80a4b204c87b2a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:06:58.505 INFO 21062 --- [ocp-schedules-27,66382225d9f55bd8,2e20aea646e1c958] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:06:58.512 INFO 21062 --- [ocp-schedules-20,7bce84dd92f556b6,05c17d497b5c80d4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:06:58.513 INFO 21062 --- [ocp-schedules-3,96ab3cfa9bd0d85b,92d8eeb4cdb1afcb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:06:58.513 INFO 21062 --- [ocp-schedules-20,7bce84dd92f556b6,05c17d497b5c80d4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:06:58.513 INFO 21062 --- [ocp-schedules-3,96ab3cfa9bd0d85b,92d8eeb4cdb1afcb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:06:58.513 INFO 21062 --- [ocp-schedules-24,c28f4e0f7ed6ecaa,998cc391d714e660] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:06:58.513 INFO 21062 --- [ocp-schedules-24,c28f4e0f7ed6ecaa,998cc391d714e660] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:06:58.572 INFO 21062 --- [ocp-async-10,b09fb8cede4e34bd,e16e1f35a2a3805a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:06:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:06:58.588447+08:00, lastProcessTime=2024-11-04T13:06:48.590838+08:00 2024-11-04 13:06:58.721 WARN 21062 --- [ocp-async-2,0da1a77c551b5cde,78646d5310d1f13c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:06:58.725 INFO 21062 --- [ocp-schedules-8,b8375a9669fa047f,8981bdc84e5fe6b1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:06:58.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:06:58.588447+08:00 2024-11-04 13:06:58.798 INFO 21062 --- [ocp-async-9,a1969f927b804ea6,b3400ed619e8bbf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:06:59.096 INFO 21062 --- [ocp-schedules-19,647363b79704fcb7,a9880df2bc92b588] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:06:59.983 INFO 21062 --- [prometheus-minute-schedule-1,0bb889a63a77ec29,5f812699ffd54736] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696820, interval=60 2024-11-04 13:07:00.135 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:00.136 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:00.722 WARN 21062 --- [ocp-async-1,474eb6d6a7564dd2,ec6d5dd1083529c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:00.798 INFO 21062 --- [ocp-async-6,7f0d92dd41b80c32,54a61ede1bcd9ddf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:00.863 INFO 21062 --- [ocp-schedules-10,46f4f4fdbc9f02a6,08c54b61bffc20cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:07:00.864 INFO 21062 --- [ocp-schedules-10,7f550d642765113c,bf56c0f951ee4e6f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:07:00.867 INFO 21062 --- [ocp-schedules-10,7f550d642765113c,bf56c0f951ee4e6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:00.895 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:07:00.895 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:07:02.722 WARN 21062 --- [ocp-async-7,564f902f91e85aae,a2a7a898327fd964] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:02.798 INFO 21062 --- [ocp-async-5,d070b37123b12312,c7533c9f391df5b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:02.983 INFO 21062 --- [prometheus-minute-schedule-1,9e42479f9a46ac54,f9d8a7a5670ad57b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:03.136 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:03.137 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:04.722 WARN 21062 --- [ocp-async-3,eeee1327640a642c,62cae348ae8f7a4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:04.799 INFO 21062 --- [ocp-async-8,e42c27fcab064014,b67526a069cc345f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:05.983 INFO 21062 --- [prometheus-minute-schedule-1,52148c7f90fa1bb0,e8646722747c3e0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:06.137 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:06.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:06.722 WARN 21062 --- [ocp-async-8,a8e80af3b6dc67d6,06d01d6dca31c5c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:06.799 INFO 21062 --- [ocp-async-4,f42fde079076daef,c40e65647520cac3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:06.810 INFO 21062 --- [ocp-schedules-17,d76abd3bddae200e,954a4b57d3648082] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:07:08.505 INFO 21062 --- [ocp-schedules-2,4f82000aac0e02bc,0ddd40796ba0e1b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:07:08.505 INFO 21062 --- [ocp-schedules-15,7321a39056f376f1,c80a4b204c87b2a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:07:08.572 INFO 21062 --- [ocp-async-1,4674e93acde8b875,83d06be32ef72a27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:07:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:07:08.588460+08:00, lastProcessTime=2024-11-04T13:06:58.588447+08:00 2024-11-04 13:07:08.722 WARN 21062 --- [ocp-async-6,1db5bbe3984e603c,d4e8e85449f46437] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:07:08.588460+08:00 2024-11-04 13:07:08.799 INFO 21062 --- [ocp-async-5,2f8dc2c52a61863f,37831f66bad29548] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2122ed16aa9468c7,86d4601f84ba23ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:09.098 INFO 21062 --- [ocp-schedules-24,fa5ef9c56283ebb2,c90e0c359eb5a741] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:07:09.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:09.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:09.477 INFO 21062 --- [ocp-schedules-1,80cb5ba8ab7f3494,3fa41c2592d647a2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:07:09.483 INFO 21062 --- [ocp-schedules-1,80cb5ba8ab7f3494,3fa41c2592d647a2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:07:09.483 INFO 21062 --- [ocp-schedules-1,80cb5ba8ab7f3494,3fa41c2592d647a2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:07:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:07:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:07:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:07:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:07:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:07:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:07:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:07:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:07:10.722 WARN 21062 --- [ocp-async-5,bf5256c3413b1a12,e5783f233f023c9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:10.799 INFO 21062 --- [ocp-async-8,7a63e9f8f2fbbc93,45ec5d1e79884df4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:11.169 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=9, affectRows=9 2024-11-04 13:07:11.169 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=9, validCount=9 2024-11-04 13:07:11.170 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=24, affectRows=24 2024-11-04 13:07:11.170 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=23, validCount=23 2024-11-04 13:07:11.170 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=34, affectRows=34 2024-11-04 13:07:11.170 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=34, affectRows=34 2024-11-04 13:07:11.170 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=64, affectRows=64 2024-11-04 13:07:11.170 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=42, affectRows=42 2024-11-04 13:07:11.170 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=44, affectRows=44 2024-11-04 13:07:11.171 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=34, validCount=34 2024-11-04 13:07:11.171 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=34, validCount=34 2024-11-04 13:07:11.171 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=42, validCount=42 2024-11-04 13:07:11.171 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696760, nodeCount=71, affectRows=71 2024-11-04 13:07:11.171 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=44, validCount=44 2024-11-04 13:07:11.171 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=63, validCount=63 2024-11-04 13:07:11.171 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696760, nodeCount=70, validCount=70 2024-11-04 13:07:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6ec9dd608b8cd393,8292add879962f1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:12.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:12.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:12.723 WARN 21062 --- [ocp-async-8,2b5296d9b5671845,06cd651b714d8117] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:12.799 INFO 21062 --- [ocp-async-4,1c9617e6cbf0ade3,fe5f24b811063182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:13.984 INFO 21062 --- [prometheus-cache-monitor1,65021eb32b26f382,b62b97f95d73d05f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:07:13.984 INFO 21062 --- [prometheus-cache-monitor1,65021eb32b26f382,b62b97f95d73d05f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:07:14.348 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:03:14.327, end=2024-11-04T13:05:14.345} 2024-11-04 13:07:14.360 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:07:14.360 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:07:14.723 WARN 21062 --- [ocp-async-4,3b11ec0cf5bf369c,4aaecdda1a6adb49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:14.799 INFO 21062 --- [ocp-async-10,764bc5d3bf13faa2,17f2f14eaaf37127] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f3f05be6b1c4e5fb,49de88bfa968fa40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:15.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:15.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:16.723 WARN 21062 --- [ocp-async-10,34fbe576eb0f3b2e,68f5b86a16f1cc8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:16.800 INFO 21062 --- [ocp-async-2,2336ac1dc474102c,b716aaac27f46f29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:16.834 INFO 21062 --- [ocp-schedules-6,b4e2c3f59f2387f8,642c05c3e3f8e1c4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:07:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7063d438ab6408a2,7f83bcd25c1f27c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:18.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:18.143 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:18.572 INFO 21062 --- [ocp-async-3,d28f67f54547844a,598b7dea207419fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:07:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:07:18.588789+08:00, lastProcessTime=2024-11-04T13:07:08.588460+08:00 2024-11-04 13:07:18.684 INFO 21062 --- [ocp-schedules-14,9836d81e11dace56,6c4459b6ebb4a431] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:07:18.687 INFO 21062 --- [ocp-schedules-14,9836d81e11dace56,6c4459b6ebb4a431] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:18.688 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:07:18.723 WARN 21062 --- [ocp-async-8,aed9adcbd4412326,6c7851b2aeeb485b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:07:18.588789+08:00 2024-11-04 13:07:18.800 INFO 21062 --- [ocp-async-4,b967103c18ee9e8c,188717b4a1c7be13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:19.101 INFO 21062 --- [ocp-schedules-23,25e2a716bbafd501,b8e1b70b0e7a3d3b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:07:20.723 WARN 21062 --- [ocp-async-4,013329c3a89d122c,4ca434f8b52cd67f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:20.800 INFO 21062 --- [ocp-async-10,00413401336b0858,6cf85fbd70d24780] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:20.983 INFO 21062 --- [prometheus-minute-schedule-1,544fa51ec09046b9,3fb03e31ee1a9ea5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:21.143 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:21.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:22.724 WARN 21062 --- [ocp-async-10,d58f371cbbb3c77d,209bc4a3a5badd02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:22.800 INFO 21062 --- [ocp-async-2,807d0b0c3b31518b,29a01271635e4c4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:23.983 INFO 21062 --- [prometheus-minute-schedule-1,71feeb5b00634f44,e4bfd960ca6b45b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:24.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:24.145 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:24.724 WARN 21062 --- [ocp-async-2,22632e9a5b0ca19e,f5b7021a3447787e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:24.800 INFO 21062 --- [ocp-async-1,6eaeca45be123493,c48968c9a9cfb545] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:26.724 WARN 21062 --- [ocp-async-1,fc1da1ade8fc1246,fb88e6baebba10d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:26.800 INFO 21062 --- [ocp-async-7,17454fb9e7c005d4,374772c071e7948d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:26.873 INFO 21062 --- [ocp-schedules-20,fcbe878b359c6033,b12f135d46fed138] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:07:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f9fcffb2f91e39d3,400be7379a4ca7df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ac9434f96d437485,214a7e42f81c1c32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:07:27.145 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:27.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:28.498 INFO 21062 --- [ocp-schedules-2,04c75b307f34f3a7,9b6d9444fabb8255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:07:28.499 INFO 21062 --- [ocp-schedules-1,cf9cdee53502228c,594b4d7c4fb48bf8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:07:28.500 INFO 21062 --- [ocp-schedules-4,1ea8c41303a8562e,5f4e29adf7dfdd96] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:07:28.500 INFO 21062 --- [ocp-schedules-30,7de3c809257ac54d,4192f04d029387bb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:07:28.502 INFO 21062 --- [ocp-schedules-2,04c75b307f34f3a7,9b6d9444fabb8255] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:07:28.502 INFO 21062 --- [ocp-schedules-27,3b4c1043cf747e14,ec2fbacaa9216c11] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:07:28.503 INFO 21062 --- [ocp-schedules-4,1ea8c41303a8562e,5f4e29adf7dfdd96] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:28.503 INFO 21062 --- [ocp-schedules-30,7de3c809257ac54d,4192f04d029387bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:07:28.504 INFO 21062 --- [ocp-schedules-16,f701d1cc48040e1b,98d9fe9e9bb86b42] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:07:28.504 INFO 21062 --- [ocp-schedules-11,14c27264da9c2987,101c749cb0895e2f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:07:28.504 INFO 21062 --- [ocp-schedules-1,cf9cdee53502228c,594b4d7c4fb48bf8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:07:28.507 INFO 21062 --- [ocp-schedules-11,14c27264da9c2987,101c749cb0895e2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:28.511 INFO 21062 --- [ocp-schedules-21,7d20a3ae7cce9bee,ee5f59eac15e8cc5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:07:28.511 INFO 21062 --- [ocp-schedules-9,49341e1e43c723c4,aeffdcf76c6bb5a4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:07:28.511 INFO 21062 --- [ocp-schedules-13,82cf3578f250f141,c2c3dbfa5c20476a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:07:28.511 INFO 21062 --- [ocp-schedules-21,7d20a3ae7cce9bee,ee5f59eac15e8cc5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:07:28.511 INFO 21062 --- [ocp-schedules-9,49341e1e43c723c4,aeffdcf76c6bb5a4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:07:28.511 INFO 21062 --- [ocp-schedules-13,82cf3578f250f141,c2c3dbfa5c20476a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:07:28.529 INFO 21062 --- [ocp-schedules-3,14e31ced5219cece,d2e447d2d5b130e6] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:07:28.529 INFO 21062 --- [ocp-schedules-3,14e31ced5219cece,d2e447d2d5b130e6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:07:28.572 INFO 21062 --- [ocp-async-10,eb860c85cb7f973f,028c2059a846abf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:07:28.574 INFO 21062 --- [pool-12-thread-15,1a202df686dbd967,038d49b2a317bf66] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:07:28.574 INFO 21062 --- [pool-12-thread-13,1a202df686dbd967,71d36b179c37cc1d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:07:28.575 INFO 21062 --- [pool-12-thread-14,1a202df686dbd967,095fcc1c2046e625] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:07:28.575 INFO 21062 --- [pool-12-thread-14,1a202df686dbd967,095fcc1c2046e625] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:07:28.575 INFO 21062 --- [pool-12-thread-16,1a202df686dbd967,9de64570deaa6c9d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:07:28.578 INFO 21062 --- [pool-12-thread-14,1a202df686dbd967,095fcc1c2046e625] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:07:28.579 INFO 21062 --- [ForkJoinPool-11-worker-10,1a202df686dbd967,c7adc2ecaf0d8d57] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:07:28.579 INFO 21062 --- [ForkJoinPool-11-worker-10,1a202df686dbd967,c7adc2ecaf0d8d57] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:07:28.579 INFO 21062 --- [ForkJoinPool-11-worker-10,1a202df686dbd967,c7adc2ecaf0d8d57] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:07:28.580 INFO 21062 --- [ForkJoinPool-11-worker-3,1a202df686dbd967,bbd122156c436533] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:07:28.580 INFO 21062 --- [pool-12-thread-15,1a202df686dbd967,038d49b2a317bf66] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:07:28.580 INFO 21062 --- [pool-12-thread-13,1a202df686dbd967,71d36b179c37cc1d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:07:28.580 INFO 21062 --- [pool-12-thread-16,1a202df686dbd967,9de64570deaa6c9d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:07:28.580 INFO 21062 --- [pool-12-thread-16,1a202df686dbd967,9de64570deaa6c9d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:07:28.582 INFO 21062 --- [pool-12-thread-14,1a202df686dbd967,095fcc1c2046e625] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:07:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:07:28.587599+08:00, lastProcessTime=2024-11-04T13:07:18.588789+08:00 2024-11-04 13:07:28.644 INFO 21062 --- [ocp-schedules-8,42c1df9a1ac41865,f0b4f3e16a97de60] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:07:28.650 INFO 21062 --- [ocp-schedules-1,ab73755db749ec0e,2a8cfdec222651df] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:07:28.662 INFO 21062 --- [ocp-schedules-1,ab73755db749ec0e,2a8cfdec222651df] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:07:28.688 INFO 21062 --- [ocp-schedules-14,9836d81e11dace56,6c4459b6ebb4a431] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:07:28.713 INFO 21062 --- [ocp-schedules-27,bcd5e6a79f37a52c,a810cf3f5ea3fe17] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:07:28.713 INFO 21062 --- [ocp-schedules-27,bcd5e6a79f37a52c,a810cf3f5ea3fe17] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:07:28.724 WARN 21062 --- [ocp-async-3,e3e885913bfb31bb,e26728f9823115bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:28.727 INFO 21062 --- [ocp-schedules-21,d10c5c986cc87172,33d2af94b756b1be] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:07:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:07:28.587599+08:00 2024-11-04 13:07:28.763 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:07:28.763 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:07:28.771 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:07:28.771 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:07:28.771 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:07:28.777 INFO 21062 --- [ocp-schedules-6,c85cd8df5784715e,4c267f4810c74e34] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:07:28.782 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:07:28.782 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:07:28.782 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:07:28.782 INFO 21062 --- [ocp-schedules-6,c85cd8df5784715e,4c267f4810c74e34] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:07:28.782 INFO 21062 --- [ocp-schedules-6,c85cd8df5784715e,4c267f4810c74e34] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:07:28.785 INFO 21062 --- [ocp-schedules-3,950095242930348a,00e7b972cb889237] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:07:28.801 INFO 21062 --- [ocp-async-9,df49f0f1d0bdb184,761ca64113c49dd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:28.899 INFO 21062 --- [ocp-schedules-29,c889e65ab1cc3aba,dbaf6d43ce0b7500] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:07:28.899 INFO 21062 --- [ocp-schedules-29,c889e65ab1cc3aba,dbaf6d43ce0b7500] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:07:28.927 INFO 21062 --- [pool-26-thread-1,c889e65ab1cc3aba,256eff77f48988fe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:07:28.930 INFO 21062 --- [ocp-schedules-15,dea062c7e49d9d6d,9007e4fb68f259b6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:07:28.930 INFO 21062 --- [ocp-schedules-15,dea062c7e49d9d6d,9007e4fb68f259b6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:07:28.931 INFO 21062 --- [ocp-schedules-15,dea062c7e49d9d6d,9007e4fb68f259b6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:07:28.931 INFO 21062 --- [ocp-schedules-15,dea062c7e49d9d6d,9007e4fb68f259b6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:07:28.931 INFO 21062 --- [ocp-schedules-15,dea062c7e49d9d6d,9007e4fb68f259b6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:07:28.931 INFO 21062 --- [ocp-schedules-15,dea062c7e49d9d6d,9007e4fb68f259b6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:07:28.956 WARN 21062 --- [ocp-schedules-29,c889e65ab1cc3aba,dbaf6d43ce0b7500] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:07:28.956 INFO 21062 --- [ocp-schedules-29,c889e65ab1cc3aba,dbaf6d43ce0b7500] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:07:29.104 INFO 21062 --- [ocp-schedules-23,6781a3f72c39cd5f,1e71585f02daefd4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:07:29.482 INFO 21062 --- [ocp-schedules-14,e91691fb88f39eb7,d1c56737c69b42f3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:07:29.482 INFO 21062 --- [ocp-schedules-14,e91691fb88f39eb7,d1c56737c69b42f3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:07:29.485 INFO 21062 --- [ocp-schedules-18,7203b6dcfba345db,3577dbb3a7f14f7e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:07:29.491 INFO 21062 --- [ocp-schedules-18,7203b6dcfba345db,3577dbb3a7f14f7e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:07:29.491 INFO 21062 --- [ocp-schedules-18,7203b6dcfba345db,3577dbb3a7f14f7e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:07:29.508 INFO 21062 --- [ocp-schedules-14,e91691fb88f39eb7,d1c56737c69b42f3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:07:29.508 INFO 21062 --- [ocp-schedules-14,e91691fb88f39eb7,d1c56737c69b42f3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:07:29.509 INFO 21062 --- [ocp-schedules-9,32292e80cfaff27e,935b3c660ff3e67e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:07:29.570 INFO 21062 --- [ocp-schedules-21,67655acce04141bc,47599ffffd0acb60] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:07:29.571 INFO 21062 --- [ocp-schedules-21,67655acce04141bc,47599ffffd0acb60] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:07:29.571 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:07:29.571 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:07:29.580 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:07:29.580 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:07:29.581 INFO 21062 --- [ocp-schedules-21,67655acce04141bc,47599ffffd0acb60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:07:29.582 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:07:29.582 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:07:29.582 WARN 21062 --- [ocp-schedules-21,67655acce04141bc,47599ffffd0acb60] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:07:29.583 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:07:29.583 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:07:29.583 INFO 21062 --- [ocp-schedules-21,67655acce04141bc,47599ffffd0acb60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:07:29.584 WARN 21062 --- [ocp-schedules-21,67655acce04141bc,47599ffffd0acb60] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:07:29.584 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:07:29.584 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:07:29.592 ERROR 21062 --- [ocp-schedules-21,67655acce04141bc,47599ffffd0acb60] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:07:29.621 INFO 21062 --- [ocp-schedules-2,ae5eaaa0503f7688,c898beccfb0c7c1a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:07:29.621+08:00, clusters=[1] 2024-11-04 13:07:29.623 INFO 21062 --- [ocp-schedules-2,ae5eaaa0503f7688,c898beccfb0c7c1a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:07:29.726 INFO 21062 --- [ocp-schedules-3,d23fd8784f552e24,8323bb2f2b8f3058] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:07:29.726+08:00, clusters=[1] 2024-11-04 13:07:29.730 INFO 21062 --- [ocp-schedules-3,d23fd8784f552e24,8323bb2f2b8f3058] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:07:29.983 INFO 21062 --- [prometheus-minute-schedule-1,1b1c15a5d363b207,4dfb04e7093d8352] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:30.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:30.147 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:07:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:07:30.724 WARN 21062 --- [ocp-async-9,51c675dd95326b16,ac3de0a87cd678ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:30.801 INFO 21062 --- [ocp-async-6,fccafe934b616027,433e00ebea39eb07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:30.895 INFO 21062 --- [ocp-schedules-10,7f550d642765113c,bf56c0f951ee4e6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:07:30.896 INFO 21062 --- [ocp-schedules-10,14fdfe933cbcdcd0,8840cdc5dd72aa6b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:07:30.899 INFO 21062 --- [ocp-schedules-10,14fdfe933cbcdcd0,8840cdc5dd72aa6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:30.928 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:07:30.928 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:07:31.704 INFO 21062 --- [ocp-schedules-14,b87d677f098cdb05,153ec5cef4838b19] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:07:31.708 INFO 21062 --- [ocp-schedules-14,b87d677f098cdb05,153ec5cef4838b19] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:31.793 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:07:31.795 ERROR 21062 --- [ocp-schedules-14,b87d677f098cdb05,153ec5cef4838b19] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:07:31.795 INFO 21062 --- [ocp-schedules-14,b87d677f098cdb05,153ec5cef4838b19] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:07:32.725 WARN 21062 --- [ocp-async-6,ea0dea505f7b926c,bb36553eb11d5641] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:32.801 INFO 21062 --- [ocp-async-5,28a678b0d38fc205,ac3708a2794d8f87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:32.983 INFO 21062 --- [prometheus-minute-schedule-1,aece03b746be6d9f,9bd909eea3773bff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:33.147 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:33.148 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:34.725 WARN 21062 --- [ocp-async-5,391c908129b78da3,1e97d846b0207820] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:34.801 INFO 21062 --- [ocp-async-2,ed82bc04935f24cd,8ec841ccdb1fba9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:35.983 INFO 21062 --- [prometheus-minute-schedule-1,ec26c73b04c53741,60145245311257bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:36.149 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:36.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:36.725 WARN 21062 --- [ocp-async-2,67545f4a6cbdfbc0,c56e0f06ac66189d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:36.801 INFO 21062 --- [ocp-async-4,1ea4cc0eea9405c7,89f4b00089babde2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:36.899 INFO 21062 --- [ocp-schedules-26,db515f0c9ff6377c,34b4cfccee596019] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:07:38.504 INFO 21062 --- [ocp-schedules-4,1ea8c41303a8562e,5f4e29adf7dfdd96] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:07:38.504 INFO 21062 --- [ocp-schedules-30,7de3c809257ac54d,4192f04d029387bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:07:38.507 INFO 21062 --- [ocp-schedules-11,14c27264da9c2987,101c749cb0895e2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:07:38.572 INFO 21062 --- [ocp-async-1,41139ed9782d46d0,98bbd21dc147482c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:07:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:07:38.589026+08:00, lastProcessTime=2024-11-04T13:07:28.587599+08:00 2024-11-04 13:07:38.725 WARN 21062 --- [ocp-async-7,e4bc9f66f0fae7d2,8fd1793d3555cc28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:07:38.589026+08:00 2024-11-04 13:07:38.801 INFO 21062 --- [ocp-async-3,306ef5d800950821,20e47a8fc08d8c76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:38.983 INFO 21062 --- [prometheus-minute-schedule-1,b757a7cea97e1020,82e6a10ea3ecf199] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:39.107 INFO 21062 --- [ocp-schedules-11,e7cec9d1529bd189,df81758ab3e4bb1e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:07:39.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:39.151 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:07:40.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:07:40.725 WARN 21062 --- [ocp-async-3,043f8ac30dedbc7f,7cc6e8ed57aac142] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:40.802 INFO 21062 --- [ocp-async-9,6b765a5c8375d496,53c8ef5530b77868] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:41.983 INFO 21062 --- [prometheus-minute-schedule-1,fbd10ae82e55eff3,dbf8b9d25b0130b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:42.151 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:42.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:42.725 WARN 21062 --- [ocp-async-9,d08a022f7b7a9ef5,87cf58315cbcd193] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:42.802 INFO 21062 --- [ocp-async-6,8acc037b31f93720,3ec33487d0b3481b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:44.726 WARN 21062 --- [ocp-async-6,c14be8f299241d5c,5cd936342ab47dd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:44.802 INFO 21062 --- [ocp-async-5,ff7f3d60bf1c75e6,f9dad33ca4972aa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b3a053f836e4beda,d705710eaee442a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:45.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:45.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:46.726 WARN 21062 --- [ocp-async-5,c778f4cf1f856058,46e0f1db2cffda85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:46.802 INFO 21062 --- [ocp-async-2,f65f6d832731cd89,47d1c399ecb50155] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:46.940 INFO 21062 --- [ocp-schedules-3,b7d8ba80477c1977,c9a4d138d3a8c564] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:07:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2830ea23b048a4f1,22634265bffa9bf7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:48.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:48.154 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:48.572 INFO 21062 --- [ocp-async-3,b3241ecf2932decc,e86841a1bddc964d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:07:48.594 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:07:48.592505+08:00, lastProcessTime=2024-11-04T13:07:38.589026+08:00 2024-11-04 13:07:48.726 WARN 21062 --- [ocp-async-9,31d727de7faabb56,07de1e0b41acdcdf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:07:48.592505+08:00 2024-11-04 13:07:48.802 INFO 21062 --- [ocp-async-6,8faf8a0d1743037d,fae54b980f0294b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:49.109 INFO 21062 --- [ocp-schedules-26,dec3707772e1fdb5,054a5f32671950c3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:07:49.494 INFO 21062 --- [ocp-schedules-7,eed22412958acc31,7ee246f477e17e1c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:07:49.500 INFO 21062 --- [ocp-schedules-7,eed22412958acc31,7ee246f477e17e1c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:07:49.500 INFO 21062 --- [ocp-schedules-7,eed22412958acc31,7ee246f477e17e1c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:07:50.726 WARN 21062 --- [ocp-async-6,b7c1846688b3c37d,52ef3bfdb146359d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:50.802 INFO 21062 --- [ocp-async-5,74365cb5b3cd2257,7da9d311b581d239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b4459b0b101fd6f4,dd0be680022963f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:51.154 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:51.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:52.726 WARN 21062 --- [ocp-async-5,e6cf263f8467d21b,d28f395bc2492e9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:52.803 INFO 21062 --- [ocp-async-8,950c601d8f583137,fa9fde91188a9e23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4b0e147e68b3d3a3,d9618b1a931786b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:54.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:54.157 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:54.726 WARN 21062 --- [ocp-async-8,a9f124dc8b1dd321,4f38cf7b2076b3d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:54.803 INFO 21062 --- [ocp-async-4,43be597cbecd76d6,959db1bd25a584d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:56.726 WARN 21062 --- [ocp-async-4,39c3e7bd7a524740,a695d8c7a89c5e55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:56.803 INFO 21062 --- [ocp-async-7,e463bbb5d6c5da54,6bb1e7b1850246d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:56.968 INFO 21062 --- [ocp-schedules-19,59f2b8841d7bb7c8,29badeabf106592c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:07:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d4bf1c63c8f7757f,3572f6722f5dd1f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:07:57.157 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:07:57.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:07:58.500 INFO 21062 --- [ocp-schedules-21,0fb11aed5ae24610,e941a55505105442] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:07:58.500 INFO 21062 --- [ocp-schedules-17,b7d170ca2dc58e03,4674c8d9cf0ca4fe] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:07:58.503 INFO 21062 --- [ocp-schedules-4,34958cc4d470f2d4,61ca275ecbb04569] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:07:58.504 INFO 21062 --- [ocp-schedules-21,0fb11aed5ae24610,e941a55505105442] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:58.504 INFO 21062 --- [ocp-schedules-17,b7d170ca2dc58e03,4674c8d9cf0ca4fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:07:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:07:58.505 INFO 21062 --- [ocp-schedules-2,a57bd4a49258f830,b896a0ab43e92a8b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:07:58.511 INFO 21062 --- [ocp-schedules-19,e90c48e17c9d58fd,d482311f355fc7fe] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:07:58.511 INFO 21062 --- [ocp-schedules-30,90bbc1d8a20e9dfd,5c4131bd5a2be256] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:07:58.511 INFO 21062 --- [ocp-schedules-30,90bbc1d8a20e9dfd,5c4131bd5a2be256] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:07:58.511 INFO 21062 --- [ocp-schedules-19,e90c48e17c9d58fd,d482311f355fc7fe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:07:58.511 INFO 21062 --- [ocp-schedules-29,b990034c8cd4180e,69246241c0bf05bd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:07:58.511 INFO 21062 --- [ocp-schedules-29,b990034c8cd4180e,69246241c0bf05bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:07:58.573 INFO 21062 --- [ocp-async-6,3882dec80e529d42,391a0d14b50b6d85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:07:58.594 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:07:58.593303+08:00, lastProcessTime=2024-11-04T13:07:48.592505+08:00 2024-11-04 13:07:58.727 WARN 21062 --- [ocp-async-5,1d75c4a0c30670eb,a9c0340fc6e65761] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:07:58.729 INFO 21062 --- [ocp-schedules-9,993674188578623a,3ec1d4477a33a51d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:07:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:07:58.593303+08:00 2024-11-04 13:07:58.803 INFO 21062 --- [ocp-async-2,43629a1e0783f8dc,c615803f31bd6e6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:07:59.114 INFO 21062 --- [ocp-schedules-6,769ab18ee42f02f1,5790a7fffaf2e4f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:07:59.983 INFO 21062 --- [prometheus-minute-schedule-1,63c42cd1fdc4e528,7c0a031334789081] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696880, interval=60 2024-11-04 13:08:00.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:00.159 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:00.727 WARN 21062 --- [ocp-async-2,2c29044896853b55,613b0458c9456048] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:00.803 INFO 21062 --- [ocp-async-1,df14f9baac044440,ef06798014a16b74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:00.928 INFO 21062 --- [ocp-schedules-10,14fdfe933cbcdcd0,8840cdc5dd72aa6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:08:00.929 INFO 21062 --- [ocp-schedules-10,8d1990e310052444,c9c7a0417442f4ae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:08:00.932 INFO 21062 --- [ocp-schedules-10,8d1990e310052444,c9c7a0417442f4ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:00.964 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:08:00.964 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:08:02.727 WARN 21062 --- [ocp-async-1,30fd7623c6a16e01,c5552f60cea99fc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:02.804 INFO 21062 --- [ocp-async-10,9ca2ed597ccb60eb,dbf03d41380e0a78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:02.983 INFO 21062 --- [prometheus-minute-schedule-1,9cdc15bac217e3d7,27e1ab39c85228ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:03.159 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:03.160 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:04.727 WARN 21062 --- [ocp-async-7,40fc1d0a0886488e,a8322ec7008163f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:04.804 INFO 21062 --- [ocp-async-3,0569fa0c7a7b1cbd,d488e6b9dcecae52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:05.983 INFO 21062 --- [prometheus-minute-schedule-1,aeccc0c63f9302d2,c311f0819e480cf6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:06.160 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:06.162 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:06.728 WARN 21062 --- [ocp-async-8,f74277ebcec13fdf,44681c0ecc219325] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:06.804 INFO 21062 --- [ocp-async-9,ddcf0c800a0dc3f9,1c73d7cab96121cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:07.016 INFO 21062 --- [ocp-schedules-14,996fa351e7773720,4f4d5779f611bb54] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:08:08.505 INFO 21062 --- [ocp-schedules-21,0fb11aed5ae24610,e941a55505105442] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:08:08.505 INFO 21062 --- [ocp-schedules-17,b7d170ca2dc58e03,4674c8d9cf0ca4fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:08:08.573 INFO 21062 --- [ocp-async-2,8ab72648686c19b1,571549a6a3bec59c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:08:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:08:08.588870+08:00, lastProcessTime=2024-11-04T13:07:58.593303+08:00 2024-11-04 13:08:08.728 WARN 21062 --- [ocp-async-4,4671330f786178b8,34a4826de4b23df9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:08:08.588870+08:00 2024-11-04 13:08:08.804 INFO 21062 --- [ocp-async-10,6c869c9aaf79e128,e76796a1914aabce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:08.983 INFO 21062 --- [prometheus-minute-schedule-1,91e07c03ee2dbb93,9d1cc21e9c9ae0ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:09.117 INFO 21062 --- [ocp-schedules-21,f2b57046e0cc46e0,8ad895043dfba41d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:08:09.162 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:09.163 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:09.502 INFO 21062 --- [ocp-schedules-7,dde5b5652f2fc822,ca155c3f3a716c31] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:08:09.510 INFO 21062 --- [ocp-schedules-7,dde5b5652f2fc822,ca155c3f3a716c31] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:08:09.510 INFO 21062 --- [ocp-schedules-7,dde5b5652f2fc822,ca155c3f3a716c31] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:08:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:08:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:08:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:08:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:08:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:08:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:08:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:08:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:08:10.728 WARN 21062 --- [ocp-async-10,5db9df13e7ef939e,3d3e9a7e59bb6f0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:10.804 INFO 21062 --- [ocp-async-3,9a55f89125900a38,fccfce5b422f22da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:11.176 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=31, affectRows=31 2024-11-04 13:08:11.176 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=20, affectRows=20 2024-11-04 13:08:11.176 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=20, validCount=20 2024-11-04 13:08:11.176 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=31, validCount=31 2024-11-04 13:08:11.176 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=5, affectRows=5 2024-11-04 13:08:11.176 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=5, validCount=5 2024-11-04 13:08:11.177 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=16, affectRows=16 2024-11-04 13:08:11.177 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=15, validCount=15 2024-11-04 13:08:11.177 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=20, affectRows=20 2024-11-04 13:08:11.177 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=20, validCount=20 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=12, affectRows=12 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=12, validCount=12 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=42, affectRows=42 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=37, affectRows=37 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=37, validCount=37 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=42, validCount=42 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=37, affectRows=37 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=48, affectRows=48 2024-11-04 13:08:11.178 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=37, validCount=37 2024-11-04 13:08:11.179 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=48, validCount=48 2024-11-04 13:08:11.179 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696820, nodeCount=54, affectRows=54 2024-11-04 13:08:11.179 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696820, nodeCount=52, validCount=52 2024-11-04 13:08:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4f7ca48a4a22b1d0,33c992107adde55c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:12.164 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:12.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:12.728 WARN 21062 --- [ocp-async-3,d2010f7612ee5c48,6c4d2a4bc00259f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:12.805 INFO 21062 --- [ocp-async-9,0a09b8c01c3230ff,95461c1352a90fda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:13.973 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:02:13.946, end=2024-11-04T13:07:13.97} 2024-11-04 13:08:13.973 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:02:13.946, end=2024-11-04T13:07:13.97} 2024-11-04 13:08:13.977 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:08:13.977 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:08:13.977 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:08:13.977 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:08:13.984 INFO 21062 --- [prometheus-cache-monitor1,d2c365ea098181f0,2655e964a7643533] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:08:13.984 INFO 21062 --- [prometheus-cache-monitor1,d2c365ea098181f0,2655e964a7643533] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:08:14.728 WARN 21062 --- [ocp-async-9,10f0b14c32884064,ab1bf2bdce7fa054] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:14.805 INFO 21062 --- [ocp-async-6,2fa0d41c8ec8d92e,f5beb80871fe15b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:14.983 INFO 21062 --- [prometheus-minute-schedule-1,555241852dd93e1e,d05e26b661ac3123] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:15.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:15.166 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:16.728 WARN 21062 --- [ocp-async-6,279ade1ed74d3fc3,eec92c1284a0a326] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:16.805 INFO 21062 --- [ocp-async-5,4d358dfeb6b6e1d8,a2c76688f3af5af5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:17.043 INFO 21062 --- [ocp-schedules-8,4cdaf678ac9f79b4,a0ae87ca682c5568] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:08:17.983 INFO 21062 --- [prometheus-minute-schedule-1,392580293fc10f4f,0712eb79f66dff7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:18.166 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:18.167 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:18.573 INFO 21062 --- [ocp-async-7,820e000c2046386c,a91c1daa97993d0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:08:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:08:18.588446+08:00, lastProcessTime=2024-11-04T13:08:08.588870+08:00 2024-11-04 13:08:18.729 WARN 21062 --- [ocp-async-3,ac4ebf99f35bff4e,0ec70b52cf61c957] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:08:18.588446+08:00 2024-11-04 13:08:18.805 INFO 21062 --- [ocp-async-9,28480b765d8b84db,105ca4cdb84e570a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:19.120 INFO 21062 --- [ocp-schedules-8,7afb0790810f4e25,6b3a8bc9c46dc4ae] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:08:20.729 WARN 21062 --- [ocp-async-9,0a92aacfad82220d,f5088c8248f72a90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:20.805 INFO 21062 --- [ocp-async-6,81efe8540a11e561,5e9a12586d4ce707] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9d8b955eb9d8a21e,bdfa0257a4b19afe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:21.167 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:21.168 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:22.729 WARN 21062 --- [ocp-async-6,6167d56ef36b4834,90d8c0177203f45a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:22.806 INFO 21062 --- [ocp-async-5,ce46026dbedb61d1,e3394e98e824a305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:23.983 INFO 21062 --- [prometheus-minute-schedule-1,7debccec6c97ce67,2289ebb97071b6e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:24.168 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:24.169 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:24.729 WARN 21062 --- [ocp-async-5,c254adca995dfe00,4d9acd9225a9b774] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:24.806 INFO 21062 --- [ocp-async-2,fb5fa831d9755ce8,b62061ac836ab632] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:26.729 WARN 21062 --- [ocp-async-2,a6bd175722b941a6,d1ca6a8bd73e20eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:26.806 INFO 21062 --- [ocp-async-1,d4ee4d1c66e22a78,fdb8fbbcceac0546] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2864b2058df744f4,3fd8807a826570ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:26.984 INFO 21062 --- [prometheus-hour-schedule-1,36ab5141e139a845,63179ebc012e2550] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:08:27.081 INFO 21062 --- [ocp-schedules-23,fed72b547f7604bc,a3fee3800787dc87] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:08:27.169 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:27.170 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:28.498 INFO 21062 --- [ocp-schedules-25,e0abbbb85684a6ed,e6ef092a9b0286c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:08:28.500 INFO 21062 --- [ocp-schedules-12,99eff9d58449eb13,292b3c9fa0e7938c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:08:28.500 INFO 21062 --- [ocp-schedules-3,2eceaa80bb4b17ce,c4f4ef4d2e2f4710] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:08:28.500 INFO 21062 --- [ocp-schedules-1,d4dfa5392fdbd0f0,44677d25e6ef6625] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:08:28.502 INFO 21062 --- [ocp-schedules-25,e0abbbb85684a6ed,e6ef092a9b0286c6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:08:28.502 INFO 21062 --- [ocp-schedules-5,108a5efaf1951549,d2bf0bf2ccc86fd3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:08:28.503 INFO 21062 --- [ocp-schedules-12,99eff9d58449eb13,292b3c9fa0e7938c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:28.504 INFO 21062 --- [ocp-schedules-1,d4dfa5392fdbd0f0,44677d25e6ef6625] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:08:28.505 INFO 21062 --- [ocp-schedules-21,42285e30546c3195,13c45678c27988f6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:08:28.505 INFO 21062 --- [ocp-schedules-3,2eceaa80bb4b17ce,c4f4ef4d2e2f4710] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:08:28.512 INFO 21062 --- [ocp-schedules-27,b7cc7d41a4914868,4b8ce51f30b87fac] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:08:28.512 INFO 21062 --- [ocp-schedules-27,b7cc7d41a4914868,4b8ce51f30b87fac] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:08:28.512 INFO 21062 --- [ocp-schedules-15,2472e5ad82a1b31d,693233d2d5f30807] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:08:28.512 INFO 21062 --- [ocp-schedules-15,2472e5ad82a1b31d,693233d2d5f30807] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:08:28.513 INFO 21062 --- [ocp-schedules-23,412b488821760dac,ea41ab6670283bc0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:08:28.513 INFO 21062 --- [ocp-schedules-23,412b488821760dac,ea41ab6670283bc0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:08:28.561 INFO 21062 --- [ocp-schedules-24,5faec7c8842f6d4b,1fe098c2990f7dc8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:08:28.562 INFO 21062 --- [ocp-schedules-24,5faec7c8842f6d4b,1fe098c2990f7dc8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:08:28.573 INFO 21062 --- [ocp-async-6,73d99a59ebc137aa,b44dbabcb72d6a75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:08:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:08:28.588898+08:00, lastProcessTime=2024-11-04T13:08:18.588446+08:00 2024-11-04 13:08:28.647 INFO 21062 --- [ocp-schedules-11,6d2c1ddb770bbee4,8f0dbb5435fbe083] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:08:28.688 INFO 21062 --- [ocp-schedules-13,1b99af85943fa78b,e6525923eba00a0b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:08:28.691 INFO 21062 --- [ocp-schedules-13,1b99af85943fa78b,e6525923eba00a0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:28.692 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:08:28.720 INFO 21062 --- [ocp-schedules-19,26d3755c33a107f1,6cf38660825f37d0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:08:28.720 INFO 21062 --- [ocp-schedules-19,26d3755c33a107f1,6cf38660825f37d0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:08:28.729 WARN 21062 --- [ocp-async-7,f262a022ccd887e8,e42872e9ad898b39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:28.732 INFO 21062 --- [ocp-schedules-2,ce15ea7d6d29deb2,4d9bb611ffc3131d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:08:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:08:28.588898+08:00 2024-11-04 13:08:28.782 INFO 21062 --- [ocp-schedules-21,577e381c67803480,46737a8c3b57fdd0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:08:28.787 INFO 21062 --- [ocp-schedules-21,577e381c67803480,46737a8c3b57fdd0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:08:28.787 INFO 21062 --- [ocp-schedules-21,577e381c67803480,46737a8c3b57fdd0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:08:28.806 INFO 21062 --- [ocp-async-8,9fb521c38239b0a9,eb11f3c52ff32980] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:28.932 INFO 21062 --- [ocp-schedules-25,839c5c7ea4971019,73e9495a8bf7f214] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:08:28.932 INFO 21062 --- [ocp-schedules-25,839c5c7ea4971019,73e9495a8bf7f214] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:08:28.933 INFO 21062 --- [ocp-schedules-25,839c5c7ea4971019,73e9495a8bf7f214] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:08:28.933 INFO 21062 --- [ocp-schedules-25,839c5c7ea4971019,73e9495a8bf7f214] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:08:28.934 INFO 21062 --- [ocp-schedules-25,839c5c7ea4971019,73e9495a8bf7f214] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:08:28.934 INFO 21062 --- [ocp-schedules-25,839c5c7ea4971019,73e9495a8bf7f214] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:08:29.122 INFO 21062 --- [ocp-schedules-18,79664546898f74cb,557caf4bf3e6687b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:08:29.508 INFO 21062 --- [ocp-schedules-20,81d7627d72cba332,55f7a92067a4424c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:08:29.509 INFO 21062 --- [ocp-schedules-20,81d7627d72cba332,55f7a92067a4424c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:08:29.513 INFO 21062 --- [ocp-schedules-29,67d393734559fbf8,0270851f4daf49ea] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:08:29.520 INFO 21062 --- [ocp-schedules-29,67d393734559fbf8,0270851f4daf49ea] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:08:29.520 INFO 21062 --- [ocp-schedules-29,67d393734559fbf8,0270851f4daf49ea] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:08:29.536 INFO 21062 --- [ocp-schedules-20,81d7627d72cba332,55f7a92067a4424c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:08:29.536 INFO 21062 --- [ocp-schedules-20,81d7627d72cba332,55f7a92067a4424c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:08:29.536 INFO 21062 --- [ocp-schedules-24,d4d1565911514963,230a77e3333079b8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:08:29.600 INFO 21062 --- [ocp-schedules-30,8d0480f80e112e61,a911856ca3ee2622] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:08:29.601 INFO 21062 --- [ocp-schedules-11,5376f806701f893e,e11385d3c89a0962] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:08:29.601 INFO 21062 --- [ocp-schedules-11,5376f806701f893e,e11385d3c89a0962] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:08:29.601 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:08:29.601 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:08:29.604 INFO 21062 --- [ocp-schedules-30,8d0480f80e112e61,a911856ca3ee2622] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:29.613 INFO 21062 --- [ocp-schedules-11,5376f806701f893e,e11385d3c89a0962] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:08:29.613 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:08:29.614 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:08:29.615 WARN 21062 --- [ocp-schedules-11,5376f806701f893e,e11385d3c89a0962] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:08:29.615 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:08:29.616 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:08:29.616 INFO 21062 --- [ocp-schedules-11,5376f806701f893e,e11385d3c89a0962] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:08:29.616 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:08:29.616 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:08:29.617 WARN 21062 --- [ocp-schedules-11,5376f806701f893e,e11385d3c89a0962] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:08:29.617 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:08:29.617 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:08:29.624 ERROR 21062 --- [ocp-schedules-11,5376f806701f893e,e11385d3c89a0962] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:08:29.647 INFO 21062 --- [ocp-schedules-19,f7c7c61b003b1e50,4d26ae489cb0ce02] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:08:29.647+08:00, clusters=[1] 2024-11-04 13:08:29.648 INFO 21062 --- [ocp-schedules-19,f7c7c61b003b1e50,4d26ae489cb0ce02] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:08:29.662 ERROR 21062 --- [ocp-schedules-30,8d0480f80e112e61,a911856ca3ee2622] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:08:29.662 INFO 21062 --- [ocp-schedules-30,8d0480f80e112e61,a911856ca3ee2622] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:08:29.755 INFO 21062 --- [ocp-schedules-2,86adcaec791b2014,05cbf1a9e6c9ed8e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:08:29.755+08:00, clusters=[1] 2024-11-04 13:08:29.758 INFO 21062 --- [ocp-schedules-2,86adcaec791b2014,05cbf1a9e6c9ed8e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:08:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b581dfd09244bc44,8b6617c2ba755a78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:30.170 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:30.171 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:08:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:08:30.730 WARN 21062 --- [ocp-async-8,8d597342897bc61e,bb5140188f0bec1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:30.806 INFO 21062 --- [ocp-async-9,e3a3f66114217ccc,cbc6223f1ba88208] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:30.964 INFO 21062 --- [ocp-schedules-10,8d1990e310052444,c9c7a0417442f4ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:08:30.964 INFO 21062 --- [ocp-schedules-10,5d1a69577aff610d,8bdcf64aacb458d7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:08:30.968 INFO 21062 --- [ocp-schedules-10,5d1a69577aff610d,8bdcf64aacb458d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:30.999 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:08:30.999 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:08:31.795 INFO 21062 --- [ocp-schedules-27,28b63a9617310b23,a619e923e6d3985e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:08:31.799 INFO 21062 --- [ocp-schedules-27,28b63a9617310b23,a619e923e6d3985e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:31.882 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:08:31.884 ERROR 21062 --- [ocp-schedules-27,28b63a9617310b23,a619e923e6d3985e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:08:31.884 INFO 21062 --- [ocp-schedules-27,28b63a9617310b23,a619e923e6d3985e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:08:32.730 WARN 21062 --- [ocp-async-9,1b325bea02617cc2,7d71854b5c07a1f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:32.806 INFO 21062 --- [ocp-async-10,73f88ba4cc40fc9e,703d9aa575843646] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:32.983 INFO 21062 --- [prometheus-minute-schedule-1,0241a5a41a3d243e,dd609c55f4d0f442] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:33.171 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:33.172 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:34.730 WARN 21062 --- [ocp-async-10,400c4e8c9d007509,ea3f0c85263b690e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:34.807 INFO 21062 --- [ocp-async-5,ef2595c109162d0f,f00251d6f3a64b6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:35.983 INFO 21062 --- [prometheus-minute-schedule-1,f3dd1c0dd8234e2d,90291b0162ad6ac0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:36.172 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:36.173 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:36.730 WARN 21062 --- [ocp-async-5,08b9bb208cb7cfba,a7959120aceca982] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:36.807 INFO 21062 --- [ocp-async-4,88f188fc6fd58d0a,949cba4483d8bb3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:37.108 INFO 21062 --- [ocp-schedules-6,1b17ae081d567165,2eda982dbd9e6bc5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:08:38.504 INFO 21062 --- [ocp-schedules-12,99eff9d58449eb13,292b3c9fa0e7938c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:08:38.504 INFO 21062 --- [ocp-schedules-1,d4dfa5392fdbd0f0,44677d25e6ef6625] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:08:38.573 INFO 21062 --- [ocp-async-2,40d5d412a9979a88,876334a90c7cb60a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:08:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:08:38.588810+08:00, lastProcessTime=2024-11-04T13:08:28.588898+08:00 2024-11-04 13:08:38.692 INFO 21062 --- [ocp-schedules-13,1b99af85943fa78b,e6525923eba00a0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:08:38.730 WARN 21062 --- [ocp-async-1,913e49d0825dc6fe,aabc636dd38bb54c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:08:38.588810+08:00 2024-11-04 13:08:38.807 INFO 21062 --- [ocp-async-7,90d5dc56c967c7f1,92f8a4eb40a349c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:38.983 INFO 21062 --- [prometheus-minute-schedule-1,4138e0a45978fa5d,199750b805e8ae35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:39.125 INFO 21062 --- [ocp-schedules-30,06e55244cc7a2a47,e0b1313f0aa6b28c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:08:39.173 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:39.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:08:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:08:40.731 WARN 21062 --- [ocp-async-7,b35065b76e0c84c8,1b4db65de86fafd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:40.807 INFO 21062 --- [ocp-async-8,940aeb8bace6f01b,677808b5c1712951] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:08:41.829 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:08:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=32ms, period=120000ms 2024-11-04 13:08:41.983 INFO 21062 --- [prometheus-minute-schedule-1,1fbc92f6366e89de,9ea98a90c5f26024] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:42.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:42.176 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:42.731 WARN 21062 --- [ocp-async-8,4723cd4398cb91bc,dcd17d85f6ccfa49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:42.807 INFO 21062 --- [ocp-async-9,aeafd22d4ca55170,ee4960795f2eb387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:44.731 WARN 21062 --- [ocp-async-9,0fd26de5380294e2,046b53f5d83e5a6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:44.807 INFO 21062 --- [ocp-async-10,95b59df12b5a2d90,fbc161026c058568] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:44.983 INFO 21062 --- [prometheus-minute-schedule-1,89b088d99b1bcd33,0c0dd745ed4cc29e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:45.176 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:45.177 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:46.731 WARN 21062 --- [ocp-async-10,76202db9cdebd601,eb6ad3e22b725528] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:46.807 INFO 21062 --- [ocp-async-5,9052c355f26900e8,54bc7a571ec7c45b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:47.152 INFO 21062 --- [ocp-schedules-17,13cd335be37d6d85,44cef31045ebe003] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:08:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fd5ca5c1404a87f8,69d0a32dd8b9011c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:48.177 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:48.178 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:48.574 INFO 21062 --- [ocp-async-7,a2a2ad87d69d7630,1df0a15c6776b522] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:08:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:08:48.588408+08:00, lastProcessTime=2024-11-04T13:08:38.588810+08:00 2024-11-04 13:08:48.731 WARN 21062 --- [ocp-async-8,7aa3b1ca73c04cde,bb1f2d0a2f85da2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:08:48.588408+08:00 2024-11-04 13:08:48.808 INFO 21062 --- [ocp-async-9,c48f87050f33a616,68b3a1597bd1e7e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:49.127 INFO 21062 --- [ocp-schedules-12,94620723e23ae1a2,ccc556693eb18051] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:08:49.523 INFO 21062 --- [ocp-schedules-1,708e423d39012fd5,cc2b957a77cb6aad] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:08:49.529 INFO 21062 --- [ocp-schedules-1,708e423d39012fd5,cc2b957a77cb6aad] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:08:49.529 INFO 21062 --- [ocp-schedules-1,708e423d39012fd5,cc2b957a77cb6aad] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:08:50.732 WARN 21062 --- [ocp-async-9,56b4ae99fb515ca7,fbc34fde3309d5ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:50.808 INFO 21062 --- [ocp-async-10,ed59563bd369dc29,054da3038b4139be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:50.983 INFO 21062 --- [prometheus-minute-schedule-1,20a7d1dc0b96df37,23bfabc07cfc8c92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:51.178 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:51.179 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:52.732 WARN 21062 --- [ocp-async-10,6f6c2966c5f6285c,3a7f5ff5cf5d61b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:52.808 INFO 21062 --- [ocp-async-3,471ed0de0f69b774,2a3ec16d6bb0846d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:52.822 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:08:52.822 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:08:52.828 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:08:52.829 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:08:52.829 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:08:52.829 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:08:52.834 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:08:52.834 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:08:52.834 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:08:52.835 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:08:52.872 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:08:52.881 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:08:52.881 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:08:52.882 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:08:52.882 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:08:52.882 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:08:52.889 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:08:53.415 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:08:53.979 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.144520114S 2024-11-04 13:08:53.979 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:08:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c6df4509ced8c0c7,7db257256259f867] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:54.079 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:08:54.079 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:08:54.106 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:08:54.169 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:08:54.169 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:08:54.169 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:08:54.169 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:08:54.169 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:08:54.170 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:08:54.179 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:54.180 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:08:54.180 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:54.358 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] o.s.boot.SpringApplication : Started application in 1.529 seconds (JVM running for 2348.037) 2024-11-04 13:08:54.358 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.529956047S 2024-11-04 13:08:54.732 WARN 21062 --- [ocp-async-3,ff1148d1f56b7225,47910a195b046fda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:54.808 INFO 21062 --- [ocp-async-4,c4f352444062039e,45d0bf2247a6bda4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:54.850 INFO 21062 --- [ocp-schedules-24,fddcb95ea038c235,a5bd80267bee589a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:08:56.732 WARN 21062 --- [ocp-async-4,5f5a2fcad5be59b2,3934a93bdf74203c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:56.808 INFO 21062 --- [ocp-async-1,9b871a07b32b52ff,833cc6638060bdac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:56.983 INFO 21062 --- [prometheus-minute-schedule-1,6101e49c77f1ad6c,06334bfaab525766] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:08:57.175 INFO 21062 --- [ocp-schedules-3,e5bbf089a41b7b31,fa44c79e03de5fa5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:08:57.180 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:08:57.181 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:08:58.500 INFO 21062 --- [ocp-schedules-17,15f981dd2f284a66,d899d23910293c3c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:08:58.500 INFO 21062 --- [ocp-schedules-13,bc7564e9093fcd3f,d302ed638cc751d4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:08:58.502 INFO 21062 --- [ocp-schedules-21,bf58e6ea5aad259c,b2449dd0b4286762] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:08:58.504 INFO 21062 --- [ocp-schedules-17,15f981dd2f284a66,d899d23910293c3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:58.504 INFO 21062 --- [ocp-schedules-13,bc7564e9093fcd3f,d302ed638cc751d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:08:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:08:58.504 INFO 21062 --- [ocp-schedules-14,fb714bbb9b5e602d,5fca59749ef6a193] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:08:58.512 INFO 21062 --- [ocp-schedules-3,22df7dc6ff3a4fd0,a741f1cc172c0150] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:08:58.512 INFO 21062 --- [ocp-schedules-3,22df7dc6ff3a4fd0,a741f1cc172c0150] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:08:58.512 INFO 21062 --- [ocp-schedules-28,34d97f57415f304f,95dfe06bef0b1101] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:08:58.512 INFO 21062 --- [ocp-schedules-29,a556c87d2b48768a,5742a0e0578fd88d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:08:58.512 INFO 21062 --- [ocp-schedules-28,34d97f57415f304f,95dfe06bef0b1101] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:08:58.512 INFO 21062 --- [ocp-schedules-29,a556c87d2b48768a,5742a0e0578fd88d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:08:58.574 INFO 21062 --- [ocp-async-9,abc1ddf5e2c4ccf9,9211bd1b1dd90548] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:08:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:08:58.589140+08:00, lastProcessTime=2024-11-04T13:08:48.588408+08:00 2024-11-04 13:08:58.732 WARN 21062 --- [ocp-async-10,2351bda247433181,73c221ea9c288fca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:08:58.735 INFO 21062 --- [ocp-schedules-15,d91358d5c607481a,663cb6f5b4311350] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:08:58.745 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:08:58.589140+08:00 2024-11-04 13:08:58.808 INFO 21062 --- [ocp-async-5,caa1a44539d2c654,09daf559630da56d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:08:59.131 INFO 21062 --- [ocp-schedules-14,106a4dadeadb7f28,c1d78e23c9f5fa7d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:08:59.983 INFO 21062 --- [prometheus-minute-schedule-1,33edd9c24d7e42d9,f2cde90874a3a1c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730696940, interval=60 2024-11-04 13:09:00.181 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:00.182 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:00.732 WARN 21062 --- [ocp-async-3,892d2699cb152f7b,12a2a34bf7dc54d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:00.809 INFO 21062 --- [ocp-async-2,6b192e5339ac10bb,376e11895a6da76c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:00.999 INFO 21062 --- [ocp-schedules-10,5d1a69577aff610d,8bdcf64aacb458d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:09:00.999 INFO 21062 --- [ocp-schedules-10,a17c15c02423c48d,8d8f3a27285781bc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:09:01.003 INFO 21062 --- [ocp-schedules-10,a17c15c02423c48d,8d8f3a27285781bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:01.035 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:09:01.035 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:09:02.733 WARN 21062 --- [ocp-async-4,fe807a7a04f6bae9,8a39e2cbb95bc61d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:02.809 INFO 21062 --- [ocp-async-6,e7d4529f449a39d6,ed8d523c3d6752d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:02.983 INFO 21062 --- [prometheus-minute-schedule-1,8c6190fc8353bf27,184e3f05220e569e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:03.182 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:03.183 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:04.733 WARN 21062 --- [ocp-async-1,29cda99b3b09a256,f68f6fae2415645a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:04.809 INFO 21062 --- [ocp-async-7,9895404711388cb7,278fb1c20acc88c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0be1a6c0f269d331,2f3a1d1faf47ffda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:06.183 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:06.184 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:06.733 WARN 21062 --- [ocp-async-7,26643b8b9df4619f,ca1aaa38b4fc9450] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:06.809 INFO 21062 --- [ocp-async-8,42e4c07a01e7d0d3,ce35a32a89838ba8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:07.201 INFO 21062 --- [ocp-schedules-15,aaf7b256bff023df,ab93b16091d5fc21] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:09:08.504 INFO 21062 --- [ocp-schedules-13,bc7564e9093fcd3f,d302ed638cc751d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:09:08.505 INFO 21062 --- [ocp-schedules-17,15f981dd2f284a66,d899d23910293c3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:09:08.574 INFO 21062 --- [ocp-async-3,cf912a92f7127691,fd971dfbf7d34a79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:09:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:09:08.589642+08:00, lastProcessTime=2024-11-04T13:08:58.589140+08:00 2024-11-04 13:09:08.733 WARN 21062 --- [ocp-async-2,7c550d7de74ffa8e,52faec6f429b423e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:08.751 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:09:08.589642+08:00 2024-11-04 13:09:08.809 INFO 21062 --- [ocp-async-6,d72022cfcd3a6350,74d3ee1c68304cfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:08.983 INFO 21062 --- [prometheus-minute-schedule-1,da28ab10fa169aa7,e0bcebdc8e40bdd4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:09.134 INFO 21062 --- [ocp-schedules-13,10b2cf176af90161,ab2f8f87a3a98827] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:09:09.184 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:09.185 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:09.533 INFO 21062 --- [ocp-schedules-1,1229af878ecc6dd8,025547c6da9eaff5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:09:09.540 INFO 21062 --- [ocp-schedules-1,1229af878ecc6dd8,025547c6da9eaff5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:09:09.540 INFO 21062 --- [ocp-schedules-1,1229af878ecc6dd8,025547c6da9eaff5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:09:10.002 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:09:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:09:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:09:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:09:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:09:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:09:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:09:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:09:10.733 WARN 21062 --- [ocp-async-6,6602d283bdb8f245,ae007be67a7d49a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:10.809 INFO 21062 --- [ocp-async-7,b63e94ecc630b857,b6af6146fd72dd1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:11.176 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=21, affectRows=21 2024-11-04 13:09:11.176 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=14, affectRows=14 2024-11-04 13:09:11.176 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=20, affectRows=20 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=29, affectRows=29 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=21, validCount=21 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=13, validCount=13 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=20, validCount=20 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=29, validCount=29 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=74, affectRows=74 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=22, affectRows=22 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=73, validCount=73 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=22, validCount=22 2024-11-04 13:09:11.177 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=78, affectRows=78 2024-11-04 13:09:11.178 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696880, nodeCount=64, affectRows=64 2024-11-04 13:09:11.178 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=77, validCount=77 2024-11-04 13:09:11.178 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696880, nodeCount=64, validCount=64 2024-11-04 13:09:11.983 INFO 21062 --- [prometheus-minute-schedule-1,39af883c7112ba3e,a4101d902ca23ec0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:12.185 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:12.186 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:12.733 WARN 21062 --- [ocp-async-7,4a5bed4c3531e937,7d0801e31110bb23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:12.810 INFO 21062 --- [ocp-async-8,89ee9e88c9be832c,21b94cfcafc84b81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:13.984 INFO 21062 --- [prometheus-cache-monitor1,58b90b6d50efabe0,ed9129522420c54a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:09:13.984 INFO 21062 --- [prometheus-cache-monitor1,58b90b6d50efabe0,ed9129522420c54a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:09:14.373 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:05:14.345, end=2024-11-04T13:07:14.37} 2024-11-04 13:09:14.383 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:09:14.383 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:09:14.734 WARN 21062 --- [ocp-async-8,4089f47b97c296cb,710d7bbf6d08d44d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:14.810 INFO 21062 --- [ocp-async-9,8143a074fa92f0b1,56d254e667d64e21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:14.983 INFO 21062 --- [prometheus-minute-schedule-1,75ce1941f75412fe,de90b65ca37c5ac5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:15.186 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:15.187 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:16.734 WARN 21062 --- [ocp-async-9,bc8108ff49d9a168,29b8aa3016aea4ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:16.810 INFO 21062 --- [ocp-async-10,95f672f4c0f8de38,6aa3b47048098998] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:17.223 INFO 21062 --- [ocp-schedules-19,6494b834bb3cb084,c61bbf0f427f303d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:09:17.983 INFO 21062 --- [prometheus-minute-schedule-1,6d0f9adfb320d9bd,1e7727d6521e594f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:18.187 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:18.188 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:18.574 INFO 21062 --- [ocp-async-1,9444478d6b97976e,5e85322cd273b612] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:09:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:09:18.589080+08:00, lastProcessTime=2024-11-04T13:09:08.589642+08:00 2024-11-04 13:09:18.734 WARN 21062 --- [ocp-async-7,f161f835135366f8,301b80c4a5799e55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:09:18.589080+08:00 2024-11-04 13:09:18.810 INFO 21062 --- [ocp-async-8,4bcf57a1182fecea,76d75dffb45f71ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:19.136 INFO 21062 --- [ocp-schedules-27,a20b40b9da899717,765e7fabf2e171e3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:09:20.734 WARN 21062 --- [ocp-async-8,37ebb2e570caaccd,c82465303aa7612f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:20.810 INFO 21062 --- [ocp-async-9,787247886fdb672a,fe675bd8231e9df0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ad901d16ca1c0203,47121f4cebc75902] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:21.189 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:21.189 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:22.734 WARN 21062 --- [ocp-async-9,7094a74256ead509,6e00ed27875b061d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:22.811 INFO 21062 --- [ocp-async-10,f6b413c6c5afd906,93f92499f81bd87d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:23.983 INFO 21062 --- [prometheus-minute-schedule-1,81c1eee56a6cd25c,610445514fcc0192] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:24.189 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:24.190 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:24.734 WARN 21062 --- [ocp-async-10,ca3a617f769d07f7,c100d609f1e38a04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:24.811 INFO 21062 --- [ocp-async-3,72827f81eebcd537,81d41f1663915616] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:26.735 WARN 21062 --- [ocp-async-3,a8d91412729c0590,48994b51c40272f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:26.811 INFO 21062 --- [ocp-async-4,e9338a98c2afd1a5,e26f2dadb92f10d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:26.983 INFO 21062 --- [prometheus-minute-schedule-1,1a287e223a14a913,56342da6b70c43f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:26.984 INFO 21062 --- [prometheus-hour-schedule-1,cbfbc53741de9a60,1b16a7be43d629ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:09:27.191 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:27.191 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:27.264 INFO 21062 --- [ocp-schedules-1,58333dc9c9c7b46a,f12d0a1ef1babf72] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:09:28.498 INFO 21062 --- [ocp-schedules-17,32ec207917a9ac3e,dbf0a62f4dbb8a63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:09:28.500 INFO 21062 --- [ocp-schedules-7,7691d50f3b7f4c4e,b06a4b80fe3be2af] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:09:28.500 INFO 21062 --- [ocp-schedules-9,62b0a96a2f7f5ee5,f30e828dfa94532b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:09:28.500 INFO 21062 --- [ocp-schedules-6,03ba5dbefe288d56,8d9c51e96409f9ce] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:09:28.502 INFO 21062 --- [ocp-schedules-30,5547aee2c677d2fa,44378a970e656ffe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:09:28.503 INFO 21062 --- [ocp-schedules-17,32ec207917a9ac3e,dbf0a62f4dbb8a63] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:09:28.504 INFO 21062 --- [ocp-schedules-7,7691d50f3b7f4c4e,b06a4b80fe3be2af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:28.504 INFO 21062 --- [ocp-schedules-6,03ba5dbefe288d56,8d9c51e96409f9ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:09:28.505 INFO 21062 --- [ocp-schedules-2,50568ccb2e0bdd9d,c87f33b16b50c9bb] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:09:28.506 INFO 21062 --- [ocp-schedules-9,62b0a96a2f7f5ee5,f30e828dfa94532b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:09:28.512 INFO 21062 --- [ocp-schedules-20,8f274aca9b930a0b,254a2d0cc3b1d8d1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:09:28.512 INFO 21062 --- [ocp-schedules-20,8f274aca9b930a0b,254a2d0cc3b1d8d1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:09:28.512 INFO 21062 --- [ocp-schedules-24,4cc9691e06bd7e8c,ed6d09a51ccf0420] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:09:28.512 INFO 21062 --- [ocp-schedules-1,7a3d1cb3673e88fe,c49f2aecd02d7b91] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:09:28.512 INFO 21062 --- [ocp-schedules-24,4cc9691e06bd7e8c,ed6d09a51ccf0420] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:09:28.512 INFO 21062 --- [ocp-schedules-1,7a3d1cb3673e88fe,c49f2aecd02d7b91] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:09:28.546 INFO 21062 --- [ocp-schedules-12,5cf036731dc7d4a2,e903725a54469aa8] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:09:28.546+08:00, tenantSize=0 2024-11-04 13:09:28.574 INFO 21062 --- [ocp-async-9,006c690a698aaeca,b75ae8eb09635d87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:09:28.581 INFO 21062 --- [ocp-schedules-15,994e46c9ea8993b6,81048a0d4630557b] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:09:28.582 INFO 21062 --- [ocp-schedules-15,994e46c9ea8993b6,81048a0d4630557b] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:09:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:09:28.588809+08:00, lastProcessTime=2024-11-04T13:09:18.589080+08:00 2024-11-04 13:09:28.649 INFO 21062 --- [ocp-schedules-27,f399f7ac590bbe05,20036ad6b1f78821] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:09:28.724 INFO 21062 --- [ocp-schedules-4,1c2f43ff491a9e16,81bba5f8d69383fb] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:09:28.724 INFO 21062 --- [ocp-schedules-4,1c2f43ff491a9e16,81bba5f8d69383fb] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:09:28.735 WARN 21062 --- [ocp-async-1,e02396d3540e25a6,c3cd7029695a92ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:28.738 INFO 21062 --- [ocp-schedules-21,57b9b5168e82a567,60dda1105f36e330] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:09:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:09:28.588809+08:00 2024-11-04 13:09:28.787 INFO 21062 --- [ocp-schedules-2,520b3503bfc7e303,94c0fb85bdcdc5b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:09:28.794 INFO 21062 --- [ocp-schedules-2,520b3503bfc7e303,94c0fb85bdcdc5b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:09:28.795 INFO 21062 --- [ocp-schedules-2,520b3503bfc7e303,94c0fb85bdcdc5b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:09:28.811 INFO 21062 --- [ocp-async-5,4240d473b65ff97b,7e4b3f950edead9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:28.935 INFO 21062 --- [ocp-schedules-17,4cb250c95c5da25e,46795c7ff09b8338] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:09:28.935 INFO 21062 --- [ocp-schedules-17,4cb250c95c5da25e,46795c7ff09b8338] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:09:28.942 INFO 21062 --- [ocp-schedules-17,4cb250c95c5da25e,46795c7ff09b8338] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:09:28.942 INFO 21062 --- [ocp-schedules-17,4cb250c95c5da25e,46795c7ff09b8338] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:09:28.946 INFO 21062 --- [ocp-schedules-17,4cb250c95c5da25e,46795c7ff09b8338] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:09:28.946 INFO 21062 --- [ocp-schedules-17,4cb250c95c5da25e,46795c7ff09b8338] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:09:29.139 INFO 21062 --- [ocp-schedules-8,6e38a9ff4f807f4e,2dd15a0746c4c877] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:09:29.536 INFO 21062 --- [ocp-schedules-16,8c69ffccd9bbae73,d3df80ca1f25c749] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:09:29.536 INFO 21062 --- [ocp-schedules-16,8c69ffccd9bbae73,d3df80ca1f25c749] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:09:29.542 INFO 21062 --- [ocp-schedules-18,75f9a48aaca16055,2933595ffef65cc7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:09:29.548 INFO 21062 --- [ocp-schedules-18,75f9a48aaca16055,2933595ffef65cc7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:09:29.548 INFO 21062 --- [ocp-schedules-18,75f9a48aaca16055,2933595ffef65cc7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:09:29.564 INFO 21062 --- [ocp-schedules-16,8c69ffccd9bbae73,d3df80ca1f25c749] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:09:29.564 INFO 21062 --- [ocp-schedules-16,8c69ffccd9bbae73,d3df80ca1f25c749] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:09:29.567 INFO 21062 --- [ocp-schedules-23,5b5305199d284464,85efe5d540098a2d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:09:29.632 INFO 21062 --- [ocp-schedules-27,b039624e846c7095,2afc8a22b59587b5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:09:29.632 INFO 21062 --- [ocp-schedules-27,b039624e846c7095,2afc8a22b59587b5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:09:29.632 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:09:29.632 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:09:29.643 INFO 21062 --- [ocp-schedules-27,b039624e846c7095,2afc8a22b59587b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:09:29.643 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:09:29.643 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:09:29.644 WARN 21062 --- [ocp-schedules-27,b039624e846c7095,2afc8a22b59587b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:09:29.644 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:09:29.644 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:09:29.645 INFO 21062 --- [ocp-schedules-27,b039624e846c7095,2afc8a22b59587b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:09:29.645 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:09:29.645 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:09:29.646 WARN 21062 --- [ocp-schedules-27,b039624e846c7095,2afc8a22b59587b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:09:29.646 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:09:29.646 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:09:29.655 ERROR 21062 --- [ocp-schedules-27,b039624e846c7095,2afc8a22b59587b5] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:09:29.678 INFO 21062 --- [ocp-schedules-25,d83d5047b4f665bc,8cad8e320772a69a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:09:29.678+08:00, clusters=[1] 2024-11-04 13:09:29.679 INFO 21062 --- [ocp-schedules-25,d83d5047b4f665bc,8cad8e320772a69a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:09:29.785 INFO 21062 --- [ocp-schedules-5,837f82202a8751f3,7568d42c338f9225] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:09:29.785+08:00, clusters=[1] 2024-11-04 13:09:29.789 INFO 21062 --- [ocp-schedules-5,837f82202a8751f3,7568d42c338f9225] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:09:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b0beda2d4d3e3243,9b90b9b275b5a1b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:30.191 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:30.192 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:09:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:09:30.735 WARN 21062 --- [ocp-async-5,050a70aecb8f6cde,c05bcc31028b4d58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:30.812 INFO 21062 --- [ocp-async-8,f8720ee55b3ab1ff,b33dcf1ebb96dbf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:31.035 INFO 21062 --- [ocp-schedules-10,a17c15c02423c48d,8d8f3a27285781bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:09:31.035 INFO 21062 --- [ocp-schedules-10,26370e8225369812,2e7de417d1a1dc9e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:09:31.039 INFO 21062 --- [ocp-schedules-10,26370e8225369812,2e7de417d1a1dc9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:31.067 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:09:31.067 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:09:31.884 INFO 21062 --- [ocp-schedules-21,cfaaac083c18a4d2,ee076f28b8a3cba8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:09:31.888 INFO 21062 --- [ocp-schedules-21,cfaaac083c18a4d2,ee076f28b8a3cba8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:31.975 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:09:31.979 ERROR 21062 --- [ocp-schedules-21,cfaaac083c18a4d2,ee076f28b8a3cba8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:09:31.979 INFO 21062 --- [ocp-schedules-21,cfaaac083c18a4d2,ee076f28b8a3cba8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:09:32.735 WARN 21062 --- [ocp-async-8,75cb675ea05154de,7290b1b8b6702949] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:32.812 INFO 21062 --- [ocp-async-6,0644553c29f47da4,57448d1384b18d7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:32.983 INFO 21062 --- [prometheus-minute-schedule-1,10cd4e8e354c8be1,5d75339a58eaea91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:33.192 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:33.193 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:34.735 WARN 21062 --- [ocp-async-6,8bee62c012d6c8f7,0600ea62276e4acb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:34.812 INFO 21062 --- [ocp-async-10,0b26fccc4329a091,f5f6785769ebe76c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:35.983 INFO 21062 --- [prometheus-minute-schedule-1,21b0253237d1ccb7,0da2f98202fe956a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:36.193 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:36.194 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:36.735 WARN 21062 --- [ocp-async-10,a6440523156756ed,517d5eff220332df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:36.812 INFO 21062 --- [ocp-async-3,a13b4b63a36ea509,4a45f37a02c9e6e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:37.293 INFO 21062 --- [ocp-schedules-5,0fd7641762734888,6d235d50218e73cd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:09:38.504 INFO 21062 --- [ocp-schedules-6,03ba5dbefe288d56,8d9c51e96409f9ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:09:38.504 INFO 21062 --- [ocp-schedules-7,7691d50f3b7f4c4e,b06a4b80fe3be2af] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:09:38.575 INFO 21062 --- [ocp-async-2,4c62108880233e3f,51edcdfc00627d96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:09:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:09:38.588679+08:00, lastProcessTime=2024-11-04T13:09:28.588809+08:00 2024-11-04 13:09:38.692 INFO 21062 --- [ocp-schedules-16,dce6cd6e43b0045e,3206d1450bfe8742] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:09:38.695 INFO 21062 --- [ocp-schedules-16,dce6cd6e43b0045e,3206d1450bfe8742] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:38.696 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:09:38.735 WARN 21062 --- [ocp-async-4,d25a9ba26c32ec50,a7a0bddce5afa6de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:09:38.588679+08:00 2024-11-04 13:09:38.812 INFO 21062 --- [ocp-async-1,6e0e2b11271ad34f,3a3aced740054466] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:38.983 INFO 21062 --- [prometheus-minute-schedule-1,def23d9385975514,a79cd2a05c98be7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:39.142 INFO 21062 --- [ocp-schedules-26,b740bc602fffcdba,7f9fd8f9acebb4c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:09:39.194 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:39.195 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:40.003 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:09:40.004 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:09:40.736 WARN 21062 --- [ocp-async-1,3f7b0016024ba62f,055872aba3b4fe76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:40.812 INFO 21062 --- [ocp-async-5,bed756419de07024,803f928020340c3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:41.983 INFO 21062 --- [prometheus-minute-schedule-1,651d7ab04fa91ae8,6fc2467497416599] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:42.195 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:42.196 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:42.736 WARN 21062 --- [ocp-async-5,7ac0a608ee53dfae,9c6d28a7d141fe11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:42.812 INFO 21062 --- [ocp-async-8,58cb2274185d06cd,688b2d004a7baf30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:44.736 WARN 21062 --- [ocp-async-8,f2f25dc084f00731,3b7d4feda29fb8d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:44.813 INFO 21062 --- [ocp-async-6,797df6d07aca3057,cfcd29152ced9ef9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:44.983 INFO 21062 --- [prometheus-minute-schedule-1,eb439683de975ebc,16cd60b384b7e64c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:45.196 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:45.197 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:46.736 WARN 21062 --- [ocp-async-6,bca693aa0bacf89b,8f2eb3d44bc3ec65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:46.813 INFO 21062 --- [ocp-async-10,1cc8c268a4ae8c81,a21befab4be0d33e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:47.333 INFO 21062 --- [ocp-schedules-4,93b6fb768e2eea0e,52994b877342ff9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:09:47.983 INFO 21062 --- [prometheus-minute-schedule-1,702296eea89f4888,fef5f54aa7b1d16b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:48.197 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:48.198 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:48.575 INFO 21062 --- [ocp-async-1,cfa1e996bad7749d,054f7c96cbaadefc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:09:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:09:48.590694+08:00, lastProcessTime=2024-11-04T13:09:38.588679+08:00 2024-11-04 13:09:48.696 INFO 21062 --- [ocp-schedules-16,dce6cd6e43b0045e,3206d1450bfe8742] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:09:48.736 WARN 21062 --- [ocp-async-5,c48d2cc0fc39a6ce,fe1d8b5358312501] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:09:48.590694+08:00 2024-11-04 13:09:48.813 INFO 21062 --- [ocp-async-8,a59bf594c71b4c0a,c11ccdc78d65c4f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:49.144 INFO 21062 --- [ocp-schedules-21,6f529a778896cce3,685b1fae95c18948] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:09:49.550 INFO 21062 --- [ocp-schedules-15,46ea36fd00872d32,0dd7ceabf97fc94d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:09:49.556 INFO 21062 --- [ocp-schedules-15,46ea36fd00872d32,0dd7ceabf97fc94d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:09:49.556 INFO 21062 --- [ocp-schedules-15,46ea36fd00872d32,0dd7ceabf97fc94d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:09:50.737 WARN 21062 --- [ocp-async-8,0fd6097111b98b87,ae56868e96a3321f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:50.813 INFO 21062 --- [ocp-async-9,ca205504a8a17772,f725051580b1bdc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:50.983 INFO 21062 --- [prometheus-minute-schedule-1,725d3e44f3535550,7664885472b303e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:51.198 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:51.199 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:52.737 WARN 21062 --- [ocp-async-9,a71aa209642b4193,4a9e581176672063] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:52.813 INFO 21062 --- [ocp-async-7,030bcac77de5ab8f,84e1a2df9eca84dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:53.983 INFO 21062 --- [prometheus-minute-schedule-1,33ce56742fdfa5e2,bee7c1504676c30b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:54.199 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:54.200 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:54.737 WARN 21062 --- [ocp-async-7,35b163228d8744ab,7220eea71ae3c220] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:54.813 INFO 21062 --- [ocp-async-2,af3c5a62556c5744,68c8fe392cc84506] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:56.737 WARN 21062 --- [ocp-async-2,62aaebb2c3dce7f1,5fae68f1439832df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:56.814 INFO 21062 --- [ocp-async-4,2a5d5b8a0fe08a68,74098227a76d987a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:56.983 INFO 21062 --- [prometheus-minute-schedule-1,a24fddd5f059b0d1,fdb081a430d59b8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:09:57.200 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:09:57.201 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:09:57.360 INFO 21062 --- [ocp-schedules-1,64e5235d29083050,f6b036c1ca7996f3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:09:58.500 INFO 21062 --- [ocp-schedules-6,a4ac8c06c2a604da,e91f0a570f1d2fb9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:09:58.500 INFO 21062 --- [ocp-schedules-13,d019394743474cd3,72c32ba83ae0abfb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:09:58.503 INFO 21062 --- [ocp-schedules-20,4485eae151c4ad23,37c4991b1cd75a9e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:09:58.504 INFO 21062 --- [ocp-schedules-16,8c1fca580d71d74d,4b7194b0afce5ac8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:09:58.505 INFO 21062 --- [ocp-schedules-6,a4ac8c06c2a604da,e91f0a570f1d2fb9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:58.505 INFO 21062 --- [ocp-schedules-13,d019394743474cd3,72c32ba83ae0abfb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:09:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:09:58.513 INFO 21062 --- [ocp-schedules-9,599ffcda241c9e6a,26a1d3d001dd7954] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:09:58.513 INFO 21062 --- [ocp-schedules-8,0f28cba7d47ba9bd,188eec65609da660] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:09:58.513 INFO 21062 --- [ocp-schedules-9,599ffcda241c9e6a,26a1d3d001dd7954] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:09:58.513 INFO 21062 --- [ocp-schedules-8,0f28cba7d47ba9bd,188eec65609da660] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:09:58.513 INFO 21062 --- [ocp-schedules-24,a285423c7072c46d,1253159c07e84b61] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:09:58.513 INFO 21062 --- [ocp-schedules-24,a285423c7072c46d,1253159c07e84b61] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:09:58.575 INFO 21062 --- [ocp-async-8,7aef7844c8559513,ec73f134b4d4ed96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:09:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:09:58.589518+08:00, lastProcessTime=2024-11-04T13:09:48.590694+08:00 2024-11-04 13:09:58.737 WARN 21062 --- [ocp-async-9,80f135254b0dcc25,145b4caf6c5df402] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:09:58.740 INFO 21062 --- [ocp-schedules-25,0c76b153bfb109dd,7d3bf3423f8f89c4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:09:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:09:58.589518+08:00 2024-11-04 13:09:58.814 INFO 21062 --- [ocp-async-10,05713ad6f2d212ec,edd9624bc21b4102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:09:59.147 INFO 21062 --- [ocp-schedules-22,dbe9451c30726852,f7255f15333b2f12] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:09:59.983 INFO 21062 --- [prometheus-minute-schedule-1,03e9e4c70fac7538,f4a7961de190c91a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697000, interval=60 2024-11-04 13:10:00.201 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:00.202 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:00.609 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ab6ad690b8f43477] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:05:01.0, nextRun=2024-11-04 13:10:00.0, currentDate=Mon Nov 04 13:10:00 CST 2024 2024-11-04 13:10:00.610 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ab6ad690b8f43477] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:05:01.0, nextRunTime=2024-11-04 13:10:00.0} 2024-11-04 13:10:00.639 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ab6ad690b8f43477] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:10:00.673 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ab6ad690b8f43477] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:10:00.673 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,ab6ad690b8f43477] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:10:00.738 WARN 21062 --- [ocp-async-7,e992d989312206eb,68d6f690812dfe29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:00.814 INFO 21062 --- [ocp-async-3,692a520fe81feab1,5cb00ac1a3a9e77d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:01.068 INFO 21062 --- [ocp-schedules-10,26370e8225369812,2e7de417d1a1dc9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:10:01.068 INFO 21062 --- [ocp-schedules-10,9aedba645ca92ccc,6d290b2356ca7884] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:10:01.072 INFO 21062 --- [ocp-schedules-10,9aedba645ca92ccc,6d290b2356ca7884] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:01.102 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:10:01.102 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:10:02.738 WARN 21062 --- [ocp-async-2,71ac64e6d27b34d9,6dd7a06ad6fd67d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:02.814 INFO 21062 --- [ocp-async-6,ecd12e742fd9a29b,2fb468773d15c75b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5291d5027c0cf75b,a311314c03684a8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:03.202 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:03.203 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:04.738 WARN 21062 --- [ocp-async-6,84e5d3846c4f59b9,f9fe7157fc643478] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:04.744 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0557795ae864c0f6] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:00:05.0, nextRun=2024-11-04 13:10:04.0, currentDate=Mon Nov 04 13:10:04 CST 2024 2024-11-04 13:10:04.744 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0557795ae864c0f6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 13:00:05.0, nextRunTime=2024-11-04 13:10:04.0} 2024-11-04 13:10:04.749 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0557795ae864c0f6] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 13:10:04.750 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0557795ae864c0f6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T13:10:04.749+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 13:10:04.807 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,400fce9011937e00] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 72, operation:EXECUTE, state: RUNNING 2024-11-04 13:10:04.814 INFO 21062 --- [ocp-async-1,c73dd8cc0e5a82d4,1829dcbc08cfaf32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:04.825 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=72, context=Context{parallelIdx=-1, stringMap={task_instance_id=7, task_operation=execute, latest_execution_start_time=2024-11-04T13:10:04.799+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=72}, listMap={}}, executor=10.46.128.24 2024-11-04 13:10:04.830 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:10:04.830 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:10:04.832 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.832 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:10:04.834 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.834 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.834 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.835 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:10:04.850 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.851 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.852 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.852 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:10:04.865 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.865 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.866 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.866 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:10:04.880 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.881 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.882 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.882 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:10:04.895 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.895 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.896 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.896 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:10:04.902 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.902 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.902 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.903 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:10:04.907 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.908 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.908 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.909 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:10:04.913 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.913 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.914 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.914 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:10:04.920 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.921 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.921 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.921 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:10:04.926 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.927 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.927 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.928 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:10:04.933 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.934 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.935 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.935 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:10:04.944 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.944 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.945 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.945 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:10:04.959 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.960 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.960 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.961 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:10:04.966 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.966 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.967 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.967 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:10:04.973 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.973 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.974 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.974 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:10:04.980 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.980 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.981 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.981 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:10:04.985 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.986 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.986 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.988 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:10:04.993 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.993 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:04.994 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.994 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:10:04.998 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:04.999 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.000 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.000 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:10:05.005 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.005 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.006 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.006 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:10:05.011 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.011 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.011 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.012 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:10:05.016 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.017 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.017 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.018 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:10:05.022 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.022 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.023 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.023 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:10:05.028 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.028 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.029 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.029 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:10:05.033 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.034 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.035 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.035 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:10:05.039 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.039 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.040 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.040 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:10:05.045 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.045 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.046 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.046 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:10:05.051 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.051 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.052 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.053 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:10:05.057 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.057 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.058 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.058 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:10:05.062 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.063 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.063 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.063 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:10:05.068 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.068 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.069 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.069 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:10:05.074 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.075 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.076 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.076 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:10:05.080 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.081 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.081 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.082 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:10:05.086 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.086 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.087 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.087 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:10:05.095 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.095 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.096 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.096 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:10:05.102 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.103 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.103 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.104 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:10:05.109 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.110 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.110 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.111 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:10:05.116 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.116 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.117 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.117 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:10:05.121 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.121 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.122 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.122 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:10:05.126 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.126 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.127 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.127 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:10:05.130 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.131 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.132 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.132 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:10:05.136 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.136 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.137 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.137 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:10:05.146 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.147 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.147 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.148 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:10:05.168 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.168 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.169 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.169 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:10:05.175 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.176 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.177 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.177 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:10:05.183 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.183 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.183 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.184 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:10:05.189 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.190 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.190 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.191 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:10:05.194 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.195 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.195 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.196 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:10:05.200 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.200 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.201 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.201 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:10:05.205 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:10:05.205 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:10:05.207 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.207 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:10:05.208 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.209 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.209 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.209 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:10:05.215 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.215 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:10:05.216 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:05.216 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:10:05.220 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:10:05.229 INFO 21062 --- [subtask-executor5,dd193f8f67a5a423,4dcd7d63a6685174] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 72, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 13:10:05.799 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:7, status:SUCCESS 2024-11-04 13:10:05.800 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:7, status:SUCCESS 2024-11-04 13:10:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0247a058bac9e1cc,8828b9f9c78bacfb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:06.203 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:06.204 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:06.738 WARN 21062 --- [ocp-async-1,974086fb82a28768,3c2ad54709b6898d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:06.814 INFO 21062 --- [ocp-async-5,b8a11364198bbce1,d0b438b68b356dc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:07.400 INFO 21062 --- [ocp-schedules-15,3859b3cd6d49360d,647772c35c522a44] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:10:08.506 INFO 21062 --- [ocp-schedules-6,a4ac8c06c2a604da,e91f0a570f1d2fb9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:10:08.506 INFO 21062 --- [ocp-schedules-13,d019394743474cd3,72c32ba83ae0abfb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:10:08.575 INFO 21062 --- [ocp-async-7,e07aed84c8df7979,0dd106606d4b7d89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:10:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:10:08.588679+08:00, lastProcessTime=2024-11-04T13:09:58.589518+08:00 2024-11-04 13:10:08.738 WARN 21062 --- [ocp-async-3,b7ba4af119654988,d9d3efdda4b4fe83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:08.793 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:10:08.588679+08:00 2024-11-04 13:10:08.815 INFO 21062 --- [ocp-async-6,7a6b64c80e2b0605,fab83b1ca90be2fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2fbd0cc871d98a4e,569a4c6eeccf0e6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:09.149 INFO 21062 --- [ocp-schedules-13,9623cb28cca7cfc3,3b43a4f79d4d9740] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:10:09.204 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:09.205 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:09.559 INFO 21062 --- [ocp-schedules-29,6226ca9fbfde5986,58af251345936e84] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:10:09.566 INFO 21062 --- [ocp-schedules-29,6226ca9fbfde5986,58af251345936e84] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:10:09.566 INFO 21062 --- [ocp-schedules-29,6226ca9fbfde5986,58af251345936e84] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:10:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:10:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:10:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:10:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:10:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:10:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:10:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:10:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:10:10.738 WARN 21062 --- [ocp-async-4,8abe574f85861fb5,744de85e503d894e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:10.815 INFO 21062 --- [ocp-async-1,c09bde2e2c6591d5,1149c339c099736b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:11.187 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730696940, nodeCount=322, affectRows=322 2024-11-04 13:10:11.188 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730696940, nodeCount=319, validCount=319 2024-11-04 13:10:11.983 INFO 21062 --- [prometheus-minute-schedule-1,ae0350073af65393,499c9214b98818b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:12.205 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:12.206 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:12.738 WARN 21062 --- [ocp-async-1,5fee7a4ad9f30e7a,054f1275d0897661] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:12.815 INFO 21062 --- [ocp-async-5,532bfdf0978908fb,5ef520f8be4bae23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:13.984 INFO 21062 --- [prometheus-cache-monitor1,3c1991d62e552273,4e3e92789e42ccbe] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:10:13.984 INFO 21062 --- [prometheus-cache-monitor1,3c1991d62e552273,4e3e92789e42ccbe] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:10:14.739 WARN 21062 --- [ocp-async-5,58f6f566014703e7,f277fde101d04cba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:14.815 INFO 21062 --- [ocp-async-8,fce706ab1f9d25b7,691c0a655a1882cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:14.983 INFO 21062 --- [prometheus-minute-schedule-1,cb1b553ac690cde9,0b7c5e2f72655ac5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:15.206 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:15.207 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:16.739 WARN 21062 --- [ocp-async-8,482c43a18ab625e4,1ca4305eb7dbbb49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:16.815 INFO 21062 --- [ocp-async-9,d79c5223fc2e9c31,0c08d56b45906379] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:17.427 INFO 21062 --- [ocp-schedules-20,d39d9b65f636b409,87ec65ff91d3bbeb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:10:17.983 INFO 21062 --- [prometheus-minute-schedule-1,3b970fcbb7dd16d1,6a4c201556f71a27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:18.207 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:18.208 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:18.575 INFO 21062 --- [ocp-async-6,dc28ca126a2649e6,96677b2dbecae93e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:10:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:10:18.588888+08:00, lastProcessTime=2024-11-04T13:10:08.588679+08:00 2024-11-04 13:10:18.739 WARN 21062 --- [ocp-async-1,99445abbeb40cd9d,9af0e6474c81cdb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:10:18.588888+08:00 2024-11-04 13:10:18.815 INFO 21062 --- [ocp-async-5,3cb546c7a77b1141,d9cef1eb60f0c207] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:19.152 INFO 21062 --- [ocp-schedules-4,8fde489ecb74f584,9ee5ca2951022a90] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:10:20.739 WARN 21062 --- [ocp-async-5,48dcf44457ca8e40,6e331cd226c78c90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:20.816 INFO 21062 --- [ocp-async-8,aa06654129660fe7,49f99143067cdb92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:20.983 INFO 21062 --- [prometheus-minute-schedule-1,3dc9fec33cce0d6f,5781f554610aecc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:21.208 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:21.209 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:22.739 WARN 21062 --- [ocp-async-8,796e8962d4d4679f,a1c80ccaa4d4a04b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:22.816 INFO 21062 --- [ocp-async-9,730ca3c06f2563d4,e7551ae177b20612] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:23.983 INFO 21062 --- [prometheus-minute-schedule-1,eb48c025caa4f654,c8e78ed6d099924e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:24.209 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:24.210 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:24.739 WARN 21062 --- [ocp-async-9,6b8f8138bdd93d19,29a5e0e1c9bed5a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:24.816 INFO 21062 --- [ocp-async-7,093c64fcfc4bf2fd,fd50e6c9fd53fbaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:26.740 WARN 21062 --- [ocp-async-7,411571f2057cbb0c,3b23db11deb30557] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:26.816 INFO 21062 --- [ocp-async-3,65183ac873e1a38a,c3c5b993572181f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:26.983 INFO 21062 --- [prometheus-minute-schedule-1,922fcb93a4304146,7a2186b42e6865b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b6e40139f1ad28fe,53cd88f62e9f7011] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:10:27.210 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:27.210 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:27.469 INFO 21062 --- [ocp-schedules-12,94ac649f8d89c753,3371c54554e5d98f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:10:28.498 INFO 21062 --- [ocp-schedules-16,33d15e726c092f97,fe592abc60181ec0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:10:28.500 INFO 21062 --- [ocp-schedules-5,83116a5413422564,5bcc5a3a76339ee8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:10:28.500 INFO 21062 --- [ocp-schedules-1,f775d3b9a3614f3b,843898cbe12139e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:10:28.500 INFO 21062 --- [ocp-schedules-27,0cdaa6e04bfa9342,76ce6a927d88d658] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:10:28.502 INFO 21062 --- [ocp-schedules-12,20c830c6d583f89a,26afebfd2c07aac9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:10:28.503 INFO 21062 --- [ocp-schedules-16,33d15e726c092f97,fe592abc60181ec0] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:10:28.505 INFO 21062 --- [ocp-schedules-3,158d2dfad6db1c51,61eeff47b2fe044e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:10:28.505 INFO 21062 --- [ocp-schedules-1,f775d3b9a3614f3b,843898cbe12139e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:10:28.505 INFO 21062 --- [ocp-schedules-27,0cdaa6e04bfa9342,76ce6a927d88d658] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:28.505 INFO 21062 --- [ocp-schedules-5,83116a5413422564,5bcc5a3a76339ee8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:28.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:10:28.512 INFO 21062 --- [ocp-schedules-9,438aaa9f768f5891,f115230076500a8e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:10:28.512 INFO 21062 --- [ocp-schedules-25,9c911dd0e329a0ed,f34532a000df891a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:10:28.513 INFO 21062 --- [ocp-schedules-9,438aaa9f768f5891,f115230076500a8e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:10:28.513 INFO 21062 --- [ocp-schedules-25,9c911dd0e329a0ed,f34532a000df891a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:10:28.514 INFO 21062 --- [ocp-schedules-23,9d9a0fd2c7632649,c35204bc4f817979] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:10:28.514 INFO 21062 --- [ocp-schedules-23,9d9a0fd2c7632649,c35204bc4f817979] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:10:28.562 INFO 21062 --- [ocp-schedules-2,f2eeeb2bdc6d6c69,5adb749f441e4c25] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:10:28.562 INFO 21062 --- [ocp-schedules-2,f2eeeb2bdc6d6c69,5adb749f441e4c25] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:10:28.576 INFO 21062 --- [ocp-async-3,3ebb0e84e277ea7e,11e79f3d462538e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:10:28.582 INFO 21062 --- [ocp-schedules-11,7e2251f3b8743fea,a6d356148ec98ca5] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:10:28.582 INFO 21062 --- [ocp-schedules-11,7e2251f3b8743fea,a6d356148ec98ca5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:10:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:10:28.588434+08:00, lastProcessTime=2024-11-04T13:10:18.588888+08:00 2024-11-04 13:10:28.651 INFO 21062 --- [ocp-schedules-26,291595098da619b3,d9b74970ae9ff5c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:10:28.729 INFO 21062 --- [ocp-schedules-30,2b4e12d9910761c4,03c1889b91386df9] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:10:28.729 INFO 21062 --- [ocp-schedules-30,2b4e12d9910761c4,03c1889b91386df9] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:10:28.740 WARN 21062 --- [ocp-async-4,2b855db207732c96,551a08dd0fd39e70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:28.742 INFO 21062 --- [ocp-schedules-13,c66e7d18dfcd0674,f59efac8da4b5c71] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:10:28.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:10:28.588434+08:00 2024-11-04 13:10:28.786 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:10:28.786 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:10:28.794 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:10:28.794 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:10:28.794 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:10:28.795 INFO 21062 --- [ocp-schedules-9,33f6873833ab94ef,a553ad18a5f04a19] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:10:28.800 INFO 21062 --- [ocp-schedules-9,33f6873833ab94ef,a553ad18a5f04a19] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:10:28.800 INFO 21062 --- [ocp-schedules-9,33f6873833ab94ef,a553ad18a5f04a19] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:10:28.805 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:10:28.805 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:10:28.805 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:10:28.808 INFO 21062 --- [ocp-schedules-12,bb24437059e45469,986db2686e748b20] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:10:28.816 INFO 21062 --- [ocp-async-10,015d14a66d2e4257,ae212279e2214d72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:28.951 INFO 21062 --- [ocp-schedules-19,7d1deeab13d0fd86,a704039e02526879] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:10:28.951 INFO 21062 --- [ocp-schedules-19,7d1deeab13d0fd86,a704039e02526879] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:10:28.952 INFO 21062 --- [ocp-schedules-19,7d1deeab13d0fd86,a704039e02526879] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:10:28.952 INFO 21062 --- [ocp-schedules-19,7d1deeab13d0fd86,a704039e02526879] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:10:28.953 INFO 21062 --- [ocp-schedules-19,7d1deeab13d0fd86,a704039e02526879] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:10:28.953 INFO 21062 --- [ocp-schedules-19,7d1deeab13d0fd86,a704039e02526879] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:10:29.155 INFO 21062 --- [ocp-schedules-6,821db1e9034a9445,be14a6a598cbd802] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:10:29.564 INFO 21062 --- [ocp-schedules-11,5eaec1e0fad62a3e,2cb7d0bd74947248] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:10:29.564 INFO 21062 --- [ocp-schedules-11,5eaec1e0fad62a3e,2cb7d0bd74947248] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:10:29.569 INFO 21062 --- [ocp-schedules-22,ed2478de1d03133a,1070b414b6ecef15] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:10:29.575 INFO 21062 --- [ocp-schedules-22,ed2478de1d03133a,1070b414b6ecef15] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:10:29.575 INFO 21062 --- [ocp-schedules-22,ed2478de1d03133a,1070b414b6ecef15] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:10:29.594 INFO 21062 --- [ocp-schedules-11,5eaec1e0fad62a3e,2cb7d0bd74947248] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:10:29.594 INFO 21062 --- [ocp-schedules-11,5eaec1e0fad62a3e,2cb7d0bd74947248] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:10:29.595 INFO 21062 --- [ocp-schedules-14,1a18e3496bef4ef6,3fce5a8c5f09041e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:10:29.662 INFO 21062 --- [ocp-schedules-21,85f67bb965b98732,70a0c5437b6a8b8e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:10:29.664 INFO 21062 --- [ocp-schedules-26,08fabcdfe7c5d1ea,737a48ecda647bbc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:10:29.664 INFO 21062 --- [ocp-schedules-26,08fabcdfe7c5d1ea,737a48ecda647bbc] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:10:29.664 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:10:29.664 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:10:29.666 INFO 21062 --- [ocp-schedules-21,85f67bb965b98732,70a0c5437b6a8b8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:29.675 INFO 21062 --- [ocp-schedules-26,08fabcdfe7c5d1ea,737a48ecda647bbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:10:29.675 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:10:29.675 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:10:29.677 WARN 21062 --- [ocp-schedules-26,08fabcdfe7c5d1ea,737a48ecda647bbc] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:10:29.677 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:10:29.677 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:10:29.678 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:10:29.678 INFO 21062 --- [ocp-schedules-26,08fabcdfe7c5d1ea,737a48ecda647bbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:10:29.678 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:10:29.679 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:10:29.679 WARN 21062 --- [ocp-schedules-26,08fabcdfe7c5d1ea,737a48ecda647bbc] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:10:29.679 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:10:29.694 ERROR 21062 --- [ocp-schedules-26,08fabcdfe7c5d1ea,737a48ecda647bbc] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:10:29.713 INFO 21062 --- [ocp-schedules-30,d2b8b40443044e58,b3d177109c5b984b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:10:29.713+08:00, clusters=[1] 2024-11-04 13:10:29.716 INFO 21062 --- [ocp-schedules-30,d2b8b40443044e58,b3d177109c5b984b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:10:29.732 ERROR 21062 --- [ocp-schedules-21,85f67bb965b98732,70a0c5437b6a8b8e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:10:29.732 INFO 21062 --- [ocp-schedules-21,85f67bb965b98732,70a0c5437b6a8b8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:10:29.816 INFO 21062 --- [ocp-schedules-13,fafd7d9be6079e15,3c0057813a2c1415] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:10:29.816+08:00, clusters=[1] 2024-11-04 13:10:29.819 INFO 21062 --- [ocp-schedules-13,fafd7d9be6079e15,3c0057813a2c1415] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:10:29.983 INFO 21062 --- [prometheus-minute-schedule-1,562c392675184f8e,a512b4e395cf66d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:30.211 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:30.211 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:10:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:10:30.740 WARN 21062 --- [ocp-async-10,72b811df575a1312,01fd02659402429f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:30.817 INFO 21062 --- [ocp-async-5,b700d60dac4da2b9,2e8424b16637ef19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:31.102 INFO 21062 --- [ocp-schedules-10,9aedba645ca92ccc,6d290b2356ca7884] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:10:31.103 INFO 21062 --- [ocp-schedules-10,1657a8c6da40f8e2,ac0d0ddba9332e86] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:10:31.106 INFO 21062 --- [ocp-schedules-10,1657a8c6da40f8e2,ac0d0ddba9332e86] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:31.135 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:10:31.135 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:10:31.979 INFO 21062 --- [ocp-schedules-2,41b4a234c839006d,42cc8c5d7819c404] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:10:31.983 INFO 21062 --- [ocp-schedules-2,41b4a234c839006d,42cc8c5d7819c404] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:32.067 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:10:32.069 ERROR 21062 --- [ocp-schedules-2,41b4a234c839006d,42cc8c5d7819c404] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:10:32.069 INFO 21062 --- [ocp-schedules-2,41b4a234c839006d,42cc8c5d7819c404] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:10:32.740 WARN 21062 --- [ocp-async-5,81e3afbc1082fdc5,b4fa79a707527a1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:32.817 INFO 21062 --- [ocp-async-6,8645756df48dc365,e86c26dd089400e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a582f333741c3ebe,890ed7ca2fcca34d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:33.211 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:33.212 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:34.740 WARN 21062 --- [ocp-async-6,d27eb98494477f97,2e55fd088c55ddc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:34.817 INFO 21062 --- [ocp-async-1,bfbadf1c329b3fa0,b3e031156d396cac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:35.983 INFO 21062 --- [prometheus-minute-schedule-1,ca3ee5ac475ef462,3d052fc401564c93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:36.212 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:36.213 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:36.740 WARN 21062 --- [ocp-async-1,1c6111ddd30d3446,71b1b09c5beef91c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:36.817 INFO 21062 --- [ocp-async-2,edc2e10007fea611,36b59a23547348a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:37.494 INFO 21062 --- [ocp-schedules-2,afb480bb27d4deb7,3c94b37b05b01b95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:10:38.506 INFO 21062 --- [ocp-schedules-27,0cdaa6e04bfa9342,76ce6a927d88d658] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:10:38.506 INFO 21062 --- [ocp-schedules-5,83116a5413422564,5bcc5a3a76339ee8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:10:38.576 INFO 21062 --- [ocp-async-7,490515e35ce7c65c,6f44e22eb07ec945] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:10:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:10:38.589084+08:00, lastProcessTime=2024-11-04T13:10:28.588434+08:00 2024-11-04 13:10:38.740 WARN 21062 --- [ocp-async-8,961a5ebd9c166082,825b32d11dea9cad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:10:38.589084+08:00 2024-11-04 13:10:38.817 INFO 21062 --- [ocp-async-4,63832a661e33e33c,8adab6cc32b2e716] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:38.983 INFO 21062 --- [prometheus-minute-schedule-1,4e9f56d9af7b119b,47188995da7e45ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:39.158 INFO 21062 --- [ocp-schedules-1,c2d68fdd29e7a52a,4e4d037d55c2c870] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:10:39.214 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:39.214 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:10:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:10:40.741 WARN 21062 --- [ocp-async-4,2c2b0cf2d73c0a2e,b25f743555a5d07c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:40.818 INFO 21062 --- [ocp-async-10,41add124c622f1f0,c1f6a445480b3be9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:10:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:10:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:10:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:10:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:10:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:10:41.983 INFO 21062 --- [prometheus-minute-schedule-1,51791a51214e07f5,efd00b3d85c2d4ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:42.215 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:42.216 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:42.741 WARN 21062 --- [ocp-async-10,bb8183aeae1fc818,0e05a2e7475f83ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:42.818 INFO 21062 --- [ocp-async-5,7c1a1efce27b807d,0d2412a2aef1d974] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:44.741 WARN 21062 --- [ocp-async-5,26ea3e2773d11df6,649bf0fb9a2090b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:44.818 INFO 21062 --- [ocp-async-3,5fb7e699c8b57c8c,b56b8a1166cf7f9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:44.983 INFO 21062 --- [prometheus-minute-schedule-1,1c477c08a02f19a4,a97ed8544bf7f0fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:45.216 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:45.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:46.741 WARN 21062 --- [ocp-async-3,4e66fc7ccbc2d281,937bbe7b7fdea430] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:46.818 INFO 21062 --- [ocp-async-9,dce2fd2ad22a7d1b,1d0c3358d86b288f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:47.535 INFO 21062 --- [ocp-schedules-9,e09fd33a8d482b02,e59fdf18f7400524] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:10:47.983 INFO 21062 --- [prometheus-minute-schedule-1,f32a70d78b82358e,6b2110d0237f64a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:48.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:48.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:48.576 INFO 21062 --- [ocp-async-4,47446542a1ee987d,8cb066b62b7b90e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:10:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:10:48.590476+08:00, lastProcessTime=2024-11-04T13:10:38.589084+08:00 2024-11-04 13:10:48.696 INFO 21062 --- [ocp-schedules-2,3f848068b67572f4,1bfd7a147a422b18] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:10:48.702 INFO 21062 --- [ocp-schedules-2,3f848068b67572f4,1bfd7a147a422b18] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:48.702 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:10:48.741 WARN 21062 --- [ocp-async-10,eb36f3d777e23b43,45cb83428577247c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:48.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:10:48.590476+08:00 2024-11-04 13:10:48.818 INFO 21062 --- [ocp-async-2,fc66d9f7d9afde57,843c2060e88a3031] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:49.161 INFO 21062 --- [ocp-schedules-24,741ded27acba4de8,6feed7d96e10f11f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:10:49.577 INFO 21062 --- [ocp-schedules-20,e5935109c5cd252b,a99705c93c3639f2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:10:49.584 INFO 21062 --- [ocp-schedules-20,e5935109c5cd252b,a99705c93c3639f2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:10:49.584 INFO 21062 --- [ocp-schedules-20,e5935109c5cd252b,a99705c93c3639f2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:10:50.742 WARN 21062 --- [ocp-async-2,cfa48763001b3331,fbd85ab1d06e3299] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:50.819 INFO 21062 --- [ocp-async-6,bc5a2cd1bbf3886b,8c904ec71b9fcd04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:50.983 INFO 21062 --- [prometheus-minute-schedule-1,c265ed55026e8f45,20417e5433499e74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:51.218 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:51.219 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:52.742 WARN 21062 --- [ocp-async-3,d7219ffa9a977350,bde23cae823ca97e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:52.819 INFO 21062 --- [ocp-async-1,f8039143263c6dc2,3d1c1a4ce180ece5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:53.983 INFO 21062 --- [prometheus-minute-schedule-1,b35443e799ccabf3,2f4559aab797acf6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:54.219 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:54.220 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:54.742 WARN 21062 --- [ocp-async-1,58ba9f69e9a6727d,ccf2224a18868be1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:54.819 INFO 21062 --- [ocp-async-7,0405c6d2e4dd7f32,98e962c499870044] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:56.742 WARN 21062 --- [ocp-async-7,ef908011db504f0f,9ee6c187fc011243] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:56.819 INFO 21062 --- [ocp-async-8,de13010eebc4b1dd,a0c2a95934c89c64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d2c69263f20344cc,b15a2adf8c6beb06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:10:57.220 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:10:57.221 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:10:57.561 INFO 21062 --- [ocp-schedules-23,1a21488d09f1e4b1,a1886d3c37d117b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:10:58.499 INFO 21062 --- [ocp-schedules-25,4117327b6515aef4,c3eb7a6805a606fe] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:10:58.500 INFO 21062 --- [ocp-schedules-7,e9d594879fda6cdd,bf99614e1db6460a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:10:58.502 INFO 21062 --- [ocp-schedules-12,c921206aa76e3148,08058ae17e1253da] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:10:58.504 INFO 21062 --- [ocp-schedules-25,4117327b6515aef4,c3eb7a6805a606fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:58.504 INFO 21062 --- [ocp-schedules-7,e9d594879fda6cdd,bf99614e1db6460a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:10:58.504 INFO 21062 --- [ocp-schedules-22,a8130866e7707d56,cee52859866e4e12] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:10:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:10:58.511 INFO 21062 --- [ocp-schedules-27,34e0cb8a4342fb1f,2b58c51c0fa1e03a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:10:58.511 INFO 21062 --- [ocp-schedules-15,a20982d208943e51,212b7340fa330ab2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:10:58.511 INFO 21062 --- [ocp-schedules-15,a20982d208943e51,212b7340fa330ab2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:10:58.511 INFO 21062 --- [ocp-schedules-6,c945dbb1b78390de,9c6abc0bf9ac734b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:10:58.511 INFO 21062 --- [ocp-schedules-27,34e0cb8a4342fb1f,2b58c51c0fa1e03a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:10:58.511 INFO 21062 --- [ocp-schedules-6,c945dbb1b78390de,9c6abc0bf9ac734b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:10:58.576 INFO 21062 --- [ocp-async-2,b619e82cf47d3d49,a9d1752877a09f10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:10:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:10:58.588756+08:00, lastProcessTime=2024-11-04T13:10:48.590476+08:00 2024-11-04 13:10:58.703 INFO 21062 --- [ocp-schedules-2,3f848068b67572f4,1bfd7a147a422b18] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:10:58.742 WARN 21062 --- [ocp-async-6,f4e9b6a76b987d79,37ec344877047b93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:10:58.744 INFO 21062 --- [ocp-schedules-9,b5f468eb9d79b2a4,dee75a71c61ae0b6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:10:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:10:58.588756+08:00 2024-11-04 13:10:58.819 INFO 21062 --- [ocp-async-1,c75517f2438af019,36292c8d82bc9e7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:10:59.164 INFO 21062 --- [ocp-schedules-4,7cf6387ba5e7c96e,ba1916db2aecb555] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:10:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e368a71ca5b4d26b,8051e5a2b1c8773a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697060, interval=60 2024-11-04 13:11:00.221 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:00.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:00.742 WARN 21062 --- [ocp-async-1,2e454192f7e2e1e2,b68ef4918f1e6787] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:00.820 INFO 21062 --- [ocp-async-7,993f3678a7f5c65c,9561a588548bd43c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:01.135 INFO 21062 --- [ocp-schedules-10,1657a8c6da40f8e2,ac0d0ddba9332e86] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:11:01.135 INFO 21062 --- [ocp-schedules-10,ed7db9cf5fdac1ef,8ebf72539827bfdf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:11:01.139 INFO 21062 --- [ocp-schedules-10,ed7db9cf5fdac1ef,8ebf72539827bfdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:01.170 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:11:01.170 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:11:02.743 WARN 21062 --- [ocp-async-7,11b97406e4ca07cd,26e3454919ab746b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:02.820 INFO 21062 --- [ocp-async-8,4deb421bfbafe05c,83fc1a41d1496522] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d35d4a1c41b240d0,4a191937df948308] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:03.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:03.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:04.743 WARN 21062 --- [ocp-async-8,b54f31acb16c6139,a4636a88011fc903] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:04.820 INFO 21062 --- [ocp-async-4,5459eb553c6d1ee6,e13eeeac28e906c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:05.983 INFO 21062 --- [prometheus-minute-schedule-1,5cd5a0e0e5e05728,a182ea7635d79459] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:06.223 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:06.223 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:06.743 WARN 21062 --- [ocp-async-4,87a40854a87f2ff3,40cec52d85100487] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:06.820 INFO 21062 --- [ocp-async-10,e18dd866e12d2dd2,bf8cd8f713f4d477] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:07.603 INFO 21062 --- [ocp-schedules-8,64156cd5592bce87,480b1c86165f8aea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:11:08.505 INFO 21062 --- [ocp-schedules-7,e9d594879fda6cdd,bf99614e1db6460a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:11:08.505 INFO 21062 --- [ocp-schedules-25,4117327b6515aef4,c3eb7a6805a606fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:11:08.576 INFO 21062 --- [ocp-async-9,1259d452737bee1d,2d8a0d0bec123428] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:11:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:11:08.589881+08:00, lastProcessTime=2024-11-04T13:10:58.588756+08:00 2024-11-04 13:11:08.743 WARN 21062 --- [ocp-async-7,424b01af11c102e4,7865bc17cfff2409] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:08.777 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:11:08.589881+08:00 2024-11-04 13:11:08.820 INFO 21062 --- [ocp-async-8,7c58336da38d6818,f91a0120f257c468] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:08.983 INFO 21062 --- [prometheus-minute-schedule-1,424748025766e05b,94bed1e04dff0ba8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:09.167 INFO 21062 --- [ocp-schedules-11,1043e2cfc7c2d589,ac275b7a0d2656a9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:11:09.224 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:09.224 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:09.587 INFO 21062 --- [ocp-schedules-9,db73d99c0af978ff,338ff94ff9fafa6c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:11:09.593 INFO 21062 --- [ocp-schedules-9,db73d99c0af978ff,338ff94ff9fafa6c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:11:09.593 INFO 21062 --- [ocp-schedules-9,db73d99c0af978ff,338ff94ff9fafa6c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:11:10.004 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:11:10.004 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:11:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:11:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:11:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:11:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:11:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:11:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:11:10.743 WARN 21062 --- [ocp-async-8,25c0de6dacdcdf5e,762071e336903993] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:10.821 INFO 21062 --- [ocp-async-4,c145b8ca4b7fdf16,28be5ef9fa6a46f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:11.186 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=7, affectRows=7 2024-11-04 13:11:11.186 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=7, validCount=7 2024-11-04 13:11:11.186 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=17, affectRows=17 2024-11-04 13:11:11.186 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=28, affectRows=28 2024-11-04 13:11:11.186 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=9, affectRows=9 2024-11-04 13:11:11.186 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=14, affectRows=14 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=9, validCount=9 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=17, validCount=17 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=13, validCount=13 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=28, validCount=28 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=40, affectRows=40 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=15, affectRows=15 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=40, validCount=40 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=33, affectRows=33 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=15, validCount=15 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=33, validCount=33 2024-11-04 13:11:11.187 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=23, affectRows=23 2024-11-04 13:11:11.188 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=23, validCount=23 2024-11-04 13:11:11.188 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=14, affectRows=14 2024-11-04 13:11:11.188 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=43, affectRows=43 2024-11-04 13:11:11.188 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=14, validCount=14 2024-11-04 13:11:11.188 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=43, validCount=43 2024-11-04 13:11:11.188 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697000, nodeCount=79, affectRows=79 2024-11-04 13:11:11.189 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697000, nodeCount=77, validCount=77 2024-11-04 13:11:11.983 INFO 21062 --- [prometheus-minute-schedule-1,272e30b0049af115,18ebce41b5f5fd6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:12.225 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:12.226 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:12.744 WARN 21062 --- [ocp-async-4,f13dbc8f7a2e9dce,d461f4038233bcfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:12.821 INFO 21062 --- [ocp-async-10,828a415a0dad411e,b09e6bbad0cb2cad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:13.984 INFO 21062 --- [prometheus-cache-monitor1,f54cb8f7acfcb068,0dcf0d623e6e68fd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:11:13.984 INFO 21062 --- [prometheus-cache-monitor1,f54cb8f7acfcb068,0dcf0d623e6e68fd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:11:14.394 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:07:14.37, end=2024-11-04T13:09:14.391} 2024-11-04 13:11:14.400 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:11:14.400 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:11:14.744 WARN 21062 --- [ocp-async-10,60c630c1ae9365fe,11628050e744cc1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:14.821 INFO 21062 --- [ocp-async-2,858ca6ea592f7a08,5fd7a63f4836508d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:14.983 INFO 21062 --- [prometheus-minute-schedule-1,95e3b8f20e16f7a7,2cd1083a0538e291] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:15.226 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:15.227 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:16.744 WARN 21062 --- [ocp-async-2,2cedd794bc45d4d5,63bd424bb42945cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:16.821 INFO 21062 --- [ocp-async-3,8c4f6bd9705bb355,38b5f698c1fe06c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:17.627 INFO 21062 --- [ocp-schedules-14,70b8886b96e2a3ae,fa08a34ba5fc042b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:11:17.983 INFO 21062 --- [prometheus-minute-schedule-1,f627db096885ba72,c2df93de114addce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:18.227 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:18.228 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:18.577 INFO 21062 --- [ocp-async-8,4aa79704d779ddaa,38da092088d9e803] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:11:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:11:18.588463+08:00, lastProcessTime=2024-11-04T13:11:08.589881+08:00 2024-11-04 13:11:18.744 WARN 21062 --- [ocp-async-4,f5c927b9562c30e0,104e040fc6b55b3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:11:18.588463+08:00 2024-11-04 13:11:18.821 INFO 21062 --- [ocp-async-10,7a351d8c2fe96ee9,c329547cbbed0e40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:19.170 INFO 21062 --- [ocp-schedules-3,3f2f8fb7e4adede7,a2573665f9b635d4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:11:20.744 WARN 21062 --- [ocp-async-10,ffc44b26c0acbfaa,dd377f5cfe6d36aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:20.822 INFO 21062 --- [ocp-async-2,cc9b22f00c7d2479,98c8f42aee00d2d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:20.983 INFO 21062 --- [prometheus-minute-schedule-1,eb9901787f1cc422,ffb1776034b484b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:21.228 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:21.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:22.745 WARN 21062 --- [ocp-async-2,3741a0bd0e4c1121,8a68767fab3d4352] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:22.822 INFO 21062 --- [ocp-async-3,98dae74684b6de9c,caaba545385cbb3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:23.983 INFO 21062 --- [prometheus-minute-schedule-1,e74b4990185b259e,beda5b4b53cbc88c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:24.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:24.230 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:24.745 WARN 21062 --- [ocp-async-3,7db8cce2e30343fa,34193804bd8c11eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:24.822 INFO 21062 --- [ocp-async-1,3a317825c2414966,3558c915fc49c198] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:26.745 WARN 21062 --- [ocp-async-1,803a074cc8ee6044,3b4d9e1d7d02d0d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:26.822 INFO 21062 --- [ocp-async-5,bcdcd980d4e822ae,36ae5c5bee174846] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:26.983 INFO 21062 --- [prometheus-minute-schedule-1,8487a78c17b837a9,97ac2082f4ac0426] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:26.984 INFO 21062 --- [prometheus-hour-schedule-1,797b1312099ae028,2f3ef7f3a29c72f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:11:27.230 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:27.231 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:27.668 INFO 21062 --- [ocp-schedules-2,7f50d5b645ec93c4,5660ffa3bddfedb4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:11:28.498 INFO 21062 --- [ocp-schedules-1,664a88ffe281e7d2,70814009b17b5db9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:11:28.499 INFO 21062 --- [ocp-schedules-15,8a5c18ec17de6810,3f8f2156dcf03930] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:11:28.500 INFO 21062 --- [ocp-schedules-8,95c666a27e5fc4fa,164a4ff3ae295530] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:11:28.500 INFO 21062 --- [ocp-schedules-16,0989a1df5accc047,a474569e8b0f26b6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:11:28.502 INFO 21062 --- [ocp-schedules-1,664a88ffe281e7d2,70814009b17b5db9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:11:28.503 INFO 21062 --- [ocp-schedules-3,fd3f4de9e17d8d45,f3312338039b36a5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:11:28.503 INFO 21062 --- [ocp-schedules-8,95c666a27e5fc4fa,164a4ff3ae295530] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:28.503 INFO 21062 --- [ocp-schedules-16,0989a1df5accc047,a474569e8b0f26b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:11:28.504 INFO 21062 --- [ocp-schedules-9,286a5b711f0097c8,0b10951a274fb5b7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:11:28.505 INFO 21062 --- [ocp-schedules-15,8a5c18ec17de6810,3f8f2156dcf03930] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:11:28.510 INFO 21062 --- [ocp-schedules-26,7b49550efd739bc1,3ce45d91d0baaac3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:11:28.511 INFO 21062 --- [ocp-schedules-14,6278eb01d633f782,493f06d937b6a3d1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:11:28.511 INFO 21062 --- [ocp-schedules-30,6ad47a32090426b6,99e66154e206db8d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:11:28.511 INFO 21062 --- [ocp-schedules-14,6278eb01d633f782,493f06d937b6a3d1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:11:28.511 INFO 21062 --- [ocp-schedules-26,7b49550efd739bc1,3ce45d91d0baaac3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:11:28.511 INFO 21062 --- [ocp-schedules-30,6ad47a32090426b6,99e66154e206db8d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:11:28.577 INFO 21062 --- [ocp-async-5,24d7a3e3da13bafe,12f9dad33427eb67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:11:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:11:28.588425+08:00, lastProcessTime=2024-11-04T13:11:18.588463+08:00 2024-11-04 13:11:28.654 INFO 21062 --- [ocp-schedules-5,3158e44a80d9fad7,922a53e31457013e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:11:28.733 INFO 21062 --- [ocp-schedules-13,c7eb647b1043b203,d31f7e346fbc422f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:11:28.733 INFO 21062 --- [ocp-schedules-13,c7eb647b1043b203,d31f7e346fbc422f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:11:28.774 WARN 21062 --- [ocp-async-8,ae9948e0791f2865,a36bfc028e90c2e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:28.777 INFO 21062 --- [ocp-schedules-7,808734501d9630e2,1ef461729ccdff88] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:11:28.800 INFO 21062 --- [ocp-schedules-15,359f4fdb376f96fe,f1602f079d369fac] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:11:28.802 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:11:28.588425+08:00 2024-11-04 13:11:28.806 INFO 21062 --- [ocp-schedules-15,359f4fdb376f96fe,f1602f079d369fac] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:11:28.806 INFO 21062 --- [ocp-schedules-15,359f4fdb376f96fe,f1602f079d369fac] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:11:28.822 INFO 21062 --- [ocp-async-9,b3604c003514fa77,5cda1f4515452794] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:28.954 INFO 21062 --- [ocp-schedules-14,774d97461e8d300d,5a8332046a35f39a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:11:28.954 INFO 21062 --- [ocp-schedules-14,774d97461e8d300d,5a8332046a35f39a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:11:28.955 INFO 21062 --- [ocp-schedules-14,774d97461e8d300d,5a8332046a35f39a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:11:28.955 INFO 21062 --- [ocp-schedules-14,774d97461e8d300d,5a8332046a35f39a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:11:28.956 INFO 21062 --- [ocp-schedules-14,774d97461e8d300d,5a8332046a35f39a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:11:28.956 INFO 21062 --- [ocp-schedules-14,774d97461e8d300d,5a8332046a35f39a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:11:29.174 INFO 21062 --- [ocp-schedules-1,bdb9489f245551bd,1590718c5a62c70f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:11:29.594 INFO 21062 --- [ocp-schedules-17,9e2fd9185fd4cd03,f584d7c51a233c3c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:11:29.594 INFO 21062 --- [ocp-schedules-17,9e2fd9185fd4cd03,f584d7c51a233c3c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:11:29.597 INFO 21062 --- [ocp-schedules-11,60d623f6091dca0f,a3fb2c0de3ebcf2f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:11:29.606 INFO 21062 --- [ocp-schedules-11,60d623f6091dca0f,a3fb2c0de3ebcf2f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:11:29.606 INFO 21062 --- [ocp-schedules-11,60d623f6091dca0f,a3fb2c0de3ebcf2f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:11:29.623 INFO 21062 --- [ocp-schedules-17,9e2fd9185fd4cd03,f584d7c51a233c3c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:11:29.623 INFO 21062 --- [ocp-schedules-17,9e2fd9185fd4cd03,f584d7c51a233c3c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:11:29.623 INFO 21062 --- [ocp-schedules-24,e8bb9495cc3fcc06,c164593692c0eaa6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:11:29.702 INFO 21062 --- [ocp-schedules-5,4af7991732bd2f82,b549ac6ec943d5d2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:11:29.702 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:11:29.702 INFO 21062 --- [ocp-schedules-5,4af7991732bd2f82,b549ac6ec943d5d2] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:11:29.702 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:11:29.712 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:11:29.712 INFO 21062 --- [ocp-schedules-5,4af7991732bd2f82,b549ac6ec943d5d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:11:29.712 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:11:29.714 WARN 21062 --- [ocp-schedules-5,4af7991732bd2f82,b549ac6ec943d5d2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:11:29.714 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:11:29.714 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:11:29.714 INFO 21062 --- [ocp-schedules-5,4af7991732bd2f82,b549ac6ec943d5d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:11:29.714 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:11:29.715 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:11:29.715 WARN 21062 --- [ocp-schedules-5,4af7991732bd2f82,b549ac6ec943d5d2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:11:29.716 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:11:29.716 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:11:29.723 ERROR 21062 --- [ocp-schedules-5,4af7991732bd2f82,b549ac6ec943d5d2] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:11:29.742 INFO 21062 --- [ocp-schedules-23,8b177216d4363af4,5cb100340b6bc64d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:11:29.742+08:00, clusters=[1] 2024-11-04 13:11:29.743 INFO 21062 --- [ocp-schedules-23,8b177216d4363af4,5cb100340b6bc64d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:11:29.844 INFO 21062 --- [ocp-schedules-7,110dd31a4074b02f,dc0e621bb4d4c454] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:11:29.844+08:00, clusters=[1] 2024-11-04 13:11:29.847 INFO 21062 --- [ocp-schedules-7,110dd31a4074b02f,dc0e621bb4d4c454] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:11:29.983 INFO 21062 --- [prometheus-minute-schedule-1,83ddfae61264676b,c3cbdb3a8eff185b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:30.231 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:30.232 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:11:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:11:30.775 WARN 21062 --- [ocp-async-9,1e53cbd260a12321,5eb6be77428bea4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:30.822 INFO 21062 --- [ocp-async-1,79c74a8a16b347aa,5ca6862f406ac955] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:31.170 INFO 21062 --- [ocp-schedules-10,ed7db9cf5fdac1ef,8ebf72539827bfdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:11:31.170 INFO 21062 --- [ocp-schedules-10,9ea76102198aeca2,93118bb85daced06] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:11:31.174 INFO 21062 --- [ocp-schedules-10,9ea76102198aeca2,93118bb85daced06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:31.202 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:11:31.202 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:11:32.069 INFO 21062 --- [ocp-schedules-4,0c4be6f18ce784c7,2047d02f5fb30ef8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:11:32.074 INFO 21062 --- [ocp-schedules-4,0c4be6f18ce784c7,2047d02f5fb30ef8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:32.158 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:11:32.165 ERROR 21062 --- [ocp-schedules-4,0c4be6f18ce784c7,2047d02f5fb30ef8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:11:32.166 INFO 21062 --- [ocp-schedules-4,0c4be6f18ce784c7,2047d02f5fb30ef8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:11:32.775 WARN 21062 --- [ocp-async-1,66378f5a02771355,f87d162427ea602f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:32.823 INFO 21062 --- [ocp-async-6,39e76311a98b49c5,a74559637dc46232] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:32.983 INFO 21062 --- [prometheus-minute-schedule-1,add0588900a191a2,caa2aade0cc6cc20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:33.232 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:33.234 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:34.775 WARN 21062 --- [ocp-async-6,03764075b8a00670,e59ae18a514080a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:34.823 INFO 21062 --- [ocp-async-4,e1cce45ac926272c,41bed278d363bbbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5112285ef598b2f1,d993897327c8f79d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:36.234 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:36.235 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:36.775 WARN 21062 --- [ocp-async-4,9853acbd024708ea,3f98e48cb5d29c39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:36.823 INFO 21062 --- [ocp-async-10,cbe50d49c7d2642b,a8f2c64cf03efca8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:37.692 INFO 21062 --- [ocp-schedules-13,bc7d2fb7bcbb6cf1,2767dd19f5e0ffe1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:11:38.504 INFO 21062 --- [ocp-schedules-16,0989a1df5accc047,a474569e8b0f26b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:11:38.504 INFO 21062 --- [ocp-schedules-8,95c666a27e5fc4fa,164a4ff3ae295530] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:11:38.577 INFO 21062 --- [ocp-async-7,289bea68065c18f3,fc017a5a471ac45c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:11:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:11:38.589293+08:00, lastProcessTime=2024-11-04T13:11:28.588425+08:00 2024-11-04 13:11:38.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:11:38.589293+08:00 2024-11-04 13:11:38.775 WARN 21062 --- [ocp-async-5,736e5320800ce797,06ce4f46ae0f716b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:38.823 INFO 21062 --- [ocp-async-8,2e1db3fd1b32962b,53cfa06566dad6f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:38.983 INFO 21062 --- [prometheus-minute-schedule-1,b6d04f476a77a658,7fccaefe8179bf12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:39.177 INFO 21062 --- [ocp-schedules-23,6627e5702c53de3f,b7d1bbeb5255f92a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:11:39.235 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:39.236 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:11:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:11:40.775 WARN 21062 --- [ocp-async-8,97893ab3324594e0,d82190a3208a4ef7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:40.823 INFO 21062 --- [ocp-async-4,d6a241e5ea5e8856,40b4aa80b0bf40e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:41.983 INFO 21062 --- [prometheus-minute-schedule-1,380782d173c159aa,090d22a140382573] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:42.236 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:42.237 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:42.776 WARN 21062 --- [ocp-async-4,93ac3e42debdfe76,7bcfe76cc3d076ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:42.823 INFO 21062 --- [ocp-async-10,02d08f458954865c,9e127296207f8bf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:44.776 WARN 21062 --- [ocp-async-10,5fe6e1e8c0badbed,0bf81c0c1871b8df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:44.824 INFO 21062 --- [ocp-async-2,ae9ae66aaf29ce50,dce61cfc116d84ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:44.983 INFO 21062 --- [prometheus-minute-schedule-1,4f33924eb2db9159,deff7e57152f69af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:45.237 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:45.238 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:46.776 WARN 21062 --- [ocp-async-2,76769c8866e856ca,9c2193e83b4415ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:46.824 INFO 21062 --- [ocp-async-3,525145e09819fb26,00359dc65118bb06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:47.732 INFO 21062 --- [ocp-schedules-25,4e8f1d4488f494df,cd13fe225621d98b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:11:47.983 INFO 21062 --- [prometheus-minute-schedule-1,365953e54e47e76d,0e9fe4194955ab2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:48.238 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:48.239 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:48.577 INFO 21062 --- [ocp-async-8,5a01c1f268a633d8,7a8c106db2560a5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:11:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:11:48.589752+08:00, lastProcessTime=2024-11-04T13:11:38.589293+08:00 2024-11-04 13:11:48.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:11:48.589752+08:00 2024-11-04 13:11:48.776 WARN 21062 --- [ocp-async-4,4c0e4608700d45da,e42af96cd0113f10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:48.824 INFO 21062 --- [ocp-async-1,4d6b8f1e558645c0,176ed331ba37846d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:49.179 INFO 21062 --- [ocp-schedules-18,50d2d3309013facc,db04bb1b36f11f78] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:11:49.609 INFO 21062 --- [ocp-schedules-9,c0f1aca4d1716c6b,bcdd4d17a634e2e2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:11:49.616 INFO 21062 --- [ocp-schedules-9,c0f1aca4d1716c6b,bcdd4d17a634e2e2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:11:49.616 INFO 21062 --- [ocp-schedules-9,c0f1aca4d1716c6b,bcdd4d17a634e2e2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:11:50.776 WARN 21062 --- [ocp-async-10,9339f56605f43381,1bbcf3dec8939fdb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:50.824 INFO 21062 --- [ocp-async-6,319f4c7eb8c483ac,21e12cfe1286da7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:50.983 INFO 21062 --- [prometheus-minute-schedule-1,dbba997d4a419cb3,b5c515fb621fbfdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:51.239 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:51.240 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:52.776 WARN 21062 --- [ocp-async-2,2fbef0ffee0996b3,44381b18fdf432f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:52.824 INFO 21062 --- [ocp-async-9,1a304b6fcc9cf4c8,29c6656b230ecf53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9cb65596443ed235,377fa3452f0fbfc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:54.240 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:54.241 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:54.777 WARN 21062 --- [ocp-async-3,b92de1eb72638be9,b127130bfc28a1e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:54.825 INFO 21062 --- [ocp-async-7,93d41b89fc7419b1,ac8a1de07a9b01b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:54.850 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:11:54.850 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:11:54.858 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:11:54.858 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:11:54.858 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:11:54.858 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:11:54.864 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:11:54.864 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:11:54.864 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:11:54.864 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:11:54.913 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:11:54.921 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:11:54.921 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:11:54.922 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:11:54.922 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:11:54.922 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:11:54.929 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:11:55.460 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:11:56.024 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.160056144S 2024-11-04 13:11:56.024 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:11:56.153 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:11:56.153 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:11:56.183 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:11:56.239 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:11:56.239 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:11:56.239 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:11:56.239 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:11:56.239 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:11:56.239 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:11:56.249 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:11:56.419 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] o.s.boot.SpringApplication : Started application in 1.561 seconds (JVM running for 2530.099) 2024-11-04 13:11:56.420 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.561827693S 2024-11-04 13:11:56.777 WARN 21062 --- [ocp-async-1,6a78d6bea0eccf83,1faa1ab152baa8c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:56.810 INFO 21062 --- [ocp-schedules-14,a45ae239cb161ca3,8f5bec9184464e25] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:11:56.825 INFO 21062 --- [ocp-async-5,46bdde3ec7a1f8d0,28f3502b792883e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:56.983 INFO 21062 --- [prometheus-minute-schedule-1,48f201c76216b60e,35ac7f372c8ee6fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:11:57.241 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:11:57.242 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:11:57.759 INFO 21062 --- [ocp-schedules-2,eb4c23c310728b04,8a8e74371bd53891] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:11:58.500 INFO 21062 --- [ocp-schedules-18,c94d895bffc2b447,60256606bf744841] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:11:58.500 INFO 21062 --- [ocp-schedules-24,d525f2479d99bf15,03b484ad6e56da2a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:11:58.502 INFO 21062 --- [ocp-schedules-1,09d243dfdfea5e21,6566d531c23227a8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:11:58.504 INFO 21062 --- [ocp-schedules-18,c94d895bffc2b447,60256606bf744841] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:58.504 INFO 21062 --- [ocp-schedules-24,d525f2479d99bf15,03b484ad6e56da2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:11:58.505 INFO 21062 --- [ocp-schedules-19,d1fcb775275dfa40,543543c965612320] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:11:58.510 INFO 21062 --- [ocp-schedules-21,bcc5bba11cb73332,b3c7260d47bfff0e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:11:58.510 INFO 21062 --- [ocp-schedules-21,bcc5bba11cb73332,b3c7260d47bfff0e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:11:58.510 INFO 21062 --- [ocp-schedules-9,2711b94c0f5f548d,91afc62a50b853f1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:11:58.510 INFO 21062 --- [ocp-schedules-9,2711b94c0f5f548d,91afc62a50b853f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:11:58.512 INFO 21062 --- [ocp-schedules-22,59fdc21e8b7fb93b,b490c31b1d270814] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:11:58.512 INFO 21062 --- [ocp-schedules-22,59fdc21e8b7fb93b,b490c31b1d270814] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:11:58.578 INFO 21062 --- [ocp-async-10,da0ff8b9d7a6e4a7,cdaeb95411327b1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:11:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:11:58.589265+08:00, lastProcessTime=2024-11-04T13:11:48.589752+08:00 2024-11-04 13:11:58.703 INFO 21062 --- [ocp-schedules-26,54dd0a066e5d3afd,4fe2b58b9ab37040] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:11:58.706 INFO 21062 --- [ocp-schedules-26,54dd0a066e5d3afd,4fe2b58b9ab37040] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:11:58.707 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:11:58.744 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:11:58.589265+08:00 2024-11-04 13:11:58.777 WARN 21062 --- [ocp-async-2,9e895a30d2039ff0,979885383ea3b936] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:11:58.779 INFO 21062 --- [ocp-schedules-14,b0f2616db4de75fb,c3b153681f9ea1ef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:11:58.825 INFO 21062 --- [ocp-async-9,61aab2aa3a917b89,8186049f383a7ba2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:11:59.182 INFO 21062 --- [ocp-schedules-6,35a5c135ee36cf4e,3b74a9af27c6999c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:11:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ede48acb7ce25d13,72445fcf52ac038c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697120, interval=60 2024-11-04 13:12:00.242 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:00.243 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:00.777 WARN 21062 --- [ocp-async-9,5b6b5cb9b35b3098,6ef54e74f834524d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:00.825 INFO 21062 --- [ocp-async-7,6d50eed084f46851,d9bb3e763d95c4b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:01.202 INFO 21062 --- [ocp-schedules-10,9ea76102198aeca2,93118bb85daced06] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:12:01.202 INFO 21062 --- [ocp-schedules-10,4ab471c58591597f,37f16ffd5c7aee33] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:12:01.206 INFO 21062 --- [ocp-schedules-10,4ab471c58591597f,37f16ffd5c7aee33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:01.235 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:12:01.235 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:12:02.777 WARN 21062 --- [ocp-async-7,b844289b57550899,6b8750a3d64d5741] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:02.825 INFO 21062 --- [ocp-async-5,01c6f32facd2da43,23a0a1969970bfac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:02.983 INFO 21062 --- [prometheus-minute-schedule-1,9440dc78c2f9a3af,aa867e95d165e187] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:03.243 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:03.244 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:04.777 WARN 21062 --- [ocp-async-5,ca8cc6280eb91bfc,b9dd48c07485b2b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:04.825 INFO 21062 --- [ocp-async-8,4429101d5be9fe0e,f33500fda67c1071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:05.983 INFO 21062 --- [prometheus-minute-schedule-1,236803339f7d8219,7dedcbf3f6769b0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:06.244 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:06.245 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:06.778 WARN 21062 --- [ocp-async-8,162e2fccf69d8be0,0bd752e49199fcc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:06.825 INFO 21062 --- [ocp-async-4,3091ea85312c960d,5b51b9187e6a1a9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:07.783 INFO 21062 --- [ocp-schedules-22,4568937b0e1443e9,424d5113803094ad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:12:08.505 INFO 21062 --- [ocp-schedules-24,d525f2479d99bf15,03b484ad6e56da2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:12:08.505 INFO 21062 --- [ocp-schedules-18,c94d895bffc2b447,60256606bf744841] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:12:08.578 INFO 21062 --- [ocp-async-3,7eaeb5da9ae26b9e,fe7bdeef087ce0dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:12:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:12:08.589494+08:00, lastProcessTime=2024-11-04T13:11:58.589265+08:00 2024-11-04 13:12:08.707 INFO 21062 --- [ocp-schedules-26,54dd0a066e5d3afd,4fe2b58b9ab37040] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:12:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:12:08.589494+08:00 2024-11-04 13:12:08.778 WARN 21062 --- [ocp-async-7,3e186f587cc0c9d3,b29b363bc6718962] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:08.826 INFO 21062 --- [ocp-async-5,a56610b8da766cd9,9c512f91a77d11dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b5c849fb1e34b4f5,a60732a04c4bc16d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:09.185 INFO 21062 --- [ocp-schedules-3,a86db80b042ad4a2,ea781c5f00760eca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:12:09.245 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:09.246 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:09.619 INFO 21062 --- [ocp-schedules-15,4b2bcfec1121fab8,bab6679d27f1706a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:12:09.625 INFO 21062 --- [ocp-schedules-15,4b2bcfec1121fab8,bab6679d27f1706a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:12:09.625 INFO 21062 --- [ocp-schedules-15,4b2bcfec1121fab8,bab6679d27f1706a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:12:10.002 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:12:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:12:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:12:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:12:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:12:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:12:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:12:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:12:10.778 WARN 21062 --- [ocp-async-5,225c266f5682eca8,0b298a0f080cbb6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:10.826 INFO 21062 --- [ocp-async-8,7d78bb8cceb6ddbc,6131d36da74d695b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:11.190 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=1, affectRows=1 2024-11-04 13:12:11.190 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=1, validCount=1 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=27, affectRows=27 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=9, affectRows=9 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=17, affectRows=17 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=8, validCount=8 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=5, affectRows=5 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=27, validCount=27 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=17, validCount=17 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=5, validCount=5 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=33, affectRows=33 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=26, affectRows=26 2024-11-04 13:12:11.191 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=45, affectRows=45 2024-11-04 13:12:11.192 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=26, validCount=26 2024-11-04 13:12:11.192 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=33, validCount=33 2024-11-04 13:12:11.192 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=45, validCount=45 2024-11-04 13:12:11.192 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=74, affectRows=74 2024-11-04 13:12:11.192 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=74, validCount=74 2024-11-04 13:12:11.192 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697060, nodeCount=85, affectRows=85 2024-11-04 13:12:11.193 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697060, nodeCount=83, validCount=83 2024-11-04 13:12:11.983 INFO 21062 --- [prometheus-minute-schedule-1,d1ed27fdbbd98c11,093c4cdb4f995509] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:12.246 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:12.247 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:12.778 WARN 21062 --- [ocp-async-8,d2549ed6d5739ebf,459c12b3c2ad546c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:12.826 INFO 21062 --- [ocp-async-4,314fabdaa32f3cf2,5f4564cb74465df2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:13.985 INFO 21062 --- [prometheus-cache-monitor1,5c6f4f464eeb2204,42a18928cb4c77f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:12:13.985 INFO 21062 --- [prometheus-cache-monitor1,5c6f4f464eeb2204,42a18928cb4c77f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:12:14.779 WARN 21062 --- [ocp-async-4,3d51c77f8b5bff4b,c5e6505cd0254e16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:14.826 INFO 21062 --- [ocp-async-10,204719d19ec361f2,92c511649e9e02bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:14.983 INFO 21062 --- [prometheus-minute-schedule-1,afcbe533bf6cf84d,92530cfde1802cf6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:15.247 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:15.249 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:16.779 WARN 21062 --- [ocp-async-10,e4c7508225ec9a74,2e979b94635f7f65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:16.827 INFO 21062 --- [ocp-async-6,789dd1e7909cae2a,ddb51f278963632b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:17.808 INFO 21062 --- [ocp-schedules-24,47013ebb44951c52,92eb033fcaa818fb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:12:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ec2ef3ca7789da65,652520667bf6e374] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:18.249 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:18.250 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:18.578 INFO 21062 --- [ocp-async-5,70db0ab4ff886608,805400f3cfc3261b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:12:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:12:18.589638+08:00, lastProcessTime=2024-11-04T13:12:08.589494+08:00 2024-11-04 13:12:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:12:18.589638+08:00 2024-11-04 13:12:18.779 WARN 21062 --- [ocp-async-8,407ad49f8333150d,bcbc3ac712b382e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:18.827 INFO 21062 --- [ocp-async-4,3ffb3b00b7aa1ad3,29e5f85f09bc49f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:19.188 INFO 21062 --- [ocp-schedules-15,d07bb4ac587853b0,2bfee2c5ad3a40af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:12:20.779 WARN 21062 --- [ocp-async-4,d93c3ece325abdc3,1679efb9f697deb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:20.827 INFO 21062 --- [ocp-async-10,7519f28ec7299316,d367448efcc076b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b76f87476504743e,0a06de03acfe42bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:21.250 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:21.251 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:22.779 WARN 21062 --- [ocp-async-10,eb7e3876071165d9,8751134cc429d30e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:22.827 INFO 21062 --- [ocp-async-6,765ce9b4b9cf80da,7f28ca2ed8cedcca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:23.983 INFO 21062 --- [prometheus-minute-schedule-1,474c08ebf33fde22,21028bd3f4f5484f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:24.251 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:24.252 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:24.779 WARN 21062 --- [ocp-async-6,5a4f1ed47f4467e3,145a37776b510962] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:24.827 INFO 21062 --- [ocp-async-9,823a5ca71ec7db31,63e6227a7b877789] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:26.780 WARN 21062 --- [ocp-async-9,76ce6959ad7218bd,fefb829cb9d2104e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:26.827 INFO 21062 --- [ocp-async-1,fb28db870793e276,245b39a9bfa746cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:26.983 INFO 21062 --- [prometheus-minute-schedule-1,b31046bb9ccd7ce9,bdc775617f70e6d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2df89b49ccd2620d,7c64ce8af1b205e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:12:27.252 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:27.253 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:27.849 INFO 21062 --- [ocp-schedules-5,b44196456306fc11,e01a30f87d6be922] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:12:28.498 INFO 21062 --- [ocp-schedules-19,ee224e81a7dd8773,45579ecc0129a8b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:12:28.499 INFO 21062 --- [ocp-schedules-26,db7e695dedb0459d,4be3e0a913d726fb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:12:28.500 INFO 21062 --- [ocp-schedules-17,f4b73737ecac5a21,0ef3987fa592d52a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:12:28.500 INFO 21062 --- [ocp-schedules-25,4527bb18c19126c1,d0971ce916eb5e9d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:12:28.502 INFO 21062 --- [ocp-schedules-14,b1e64bfb2205bf8b,08a8a21f1906502a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:12:28.503 INFO 21062 --- [ocp-schedules-19,ee224e81a7dd8773,45579ecc0129a8b5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:12:28.503 INFO 21062 --- [ocp-schedules-17,f4b73737ecac5a21,0ef3987fa592d52a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:28.504 INFO 21062 --- [ocp-schedules-25,4527bb18c19126c1,d0971ce916eb5e9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:12:28.504 INFO 21062 --- [ocp-schedules-16,b447ede52f0f183d,29a520c040e186da] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:12:28.504 INFO 21062 --- [ocp-schedules-12,f4b03ef8f3005b80,fe1237e38f403acf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:12:28.505 INFO 21062 --- [ocp-schedules-26,db7e695dedb0459d,4be3e0a913d726fb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:12:28.507 INFO 21062 --- [ocp-schedules-12,f4b03ef8f3005b80,fe1237e38f403acf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:28.510 INFO 21062 --- [ocp-schedules-27,a4635b041c411668,dcfd1e9b92df6a85] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:12:28.510 INFO 21062 --- [ocp-schedules-29,96a46e95e4472aca,31b3f1ad81314863] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:12:28.510 INFO 21062 --- [ocp-schedules-27,a4635b041c411668,dcfd1e9b92df6a85] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:12:28.510 INFO 21062 --- [ocp-schedules-29,96a46e95e4472aca,31b3f1ad81314863] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:12:28.511 INFO 21062 --- [ocp-schedules-13,919559db3cf53ef4,81e69b0bc275cf5e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:12:28.511 INFO 21062 --- [ocp-schedules-13,919559db3cf53ef4,81e69b0bc275cf5e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:12:28.529 INFO 21062 --- [ocp-schedules-18,8cc15dba9b0f2a6a,bd972ab61d0b802b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:12:28.529 INFO 21062 --- [ocp-schedules-18,8cc15dba9b0f2a6a,bd972ab61d0b802b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:12:28.562 INFO 21062 --- [ocp-schedules-21,9a4d99dca240d4ac,75729957ea059226] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:12:28.562 INFO 21062 --- [ocp-schedules-21,9a4d99dca240d4ac,75729957ea059226] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:12:28.573 INFO 21062 --- [pool-12-thread-1,1089b1f50ac90309,b8a4afb675868d01] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:12:28.573 INFO 21062 --- [pool-12-thread-3,1089b1f50ac90309,eef821aaf3299f89] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:12:28.574 INFO 21062 --- [pool-12-thread-2,1089b1f50ac90309,17f49ddf9a0ae24f] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:12:28.574 INFO 21062 --- [pool-12-thread-4,1089b1f50ac90309,ea5f9811636218eb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:12:28.578 INFO 21062 --- [ocp-async-1,87d512f07cf76692,c03de1e716d476f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:12:28.580 INFO 21062 --- [ForkJoinPool-11-worker-12,1089b1f50ac90309,27d298ea7a74b721] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:12:28.580 INFO 21062 --- [ForkJoinPool-11-worker-12,1089b1f50ac90309,27d298ea7a74b721] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:12:28.580 INFO 21062 --- [ForkJoinPool-11-worker-12,1089b1f50ac90309,27d298ea7a74b721] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:12:28.580 INFO 21062 --- [pool-12-thread-4,1089b1f50ac90309,ea5f9811636218eb] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:12:28.582 INFO 21062 --- [pool-12-thread-2,1089b1f50ac90309,17f49ddf9a0ae24f] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:12:28.582 INFO 21062 --- [pool-12-thread-2,1089b1f50ac90309,17f49ddf9a0ae24f] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:12:28.582 INFO 21062 --- [ocp-schedules-9,8656407e33c251be,8da67fdbc71607c4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:12:28.583 INFO 21062 --- [ocp-schedules-9,8656407e33c251be,8da67fdbc71607c4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:12:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:12:28.588678+08:00, lastProcessTime=2024-11-04T13:12:18.589638+08:00 2024-11-04 13:12:28.601 INFO 21062 --- [pool-12-thread-3,1089b1f50ac90309,eef821aaf3299f89] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:12:28.604 INFO 21062 --- [pool-12-thread-3,1089b1f50ac90309,eef821aaf3299f89] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:12:28.606 INFO 21062 --- [ForkJoinPool-11-worker-12,1089b1f50ac90309,efca9b9578a061c8] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:12:28.606 INFO 21062 --- [pool-12-thread-1,1089b1f50ac90309,b8a4afb675868d01] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:12:28.608 INFO 21062 --- [pool-12-thread-3,1089b1f50ac90309,eef821aaf3299f89] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:12:28.653 INFO 21062 --- [ocp-schedules-14,e1b1f26ed3f41d53,4d1734a84bf2c847] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 13:12:28.653 INFO 21062 --- [ocp-schedules-14,e1b1f26ed3f41d53,4d1734a84bf2c847] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 13:12:28.653 INFO 21062 --- [ocp-schedules-14,e1b1f26ed3f41d53,4d1734a84bf2c847] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=322 2024-11-04 13:12:28.656 INFO 21062 --- [ocp-schedules-16,adfe0ab40a0eeafb,8b783a5108e70fc4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:12:28.663 INFO 21062 --- [ocp-schedules-27,6b860890ad8c8572,131e5ea71dde686f] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:12:28.674 INFO 21062 --- [ocp-schedules-14,e1b1f26ed3f41d53,4d1734a84bf2c847] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=322 2024-11-04 13:12:28.675 INFO 21062 --- [ocp-schedules-27,6b860890ad8c8572,131e5ea71dde686f] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:12:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:12:28.588678+08:00 2024-11-04 13:12:28.778 INFO 21062 --- [ocp-schedules-13,acc6862bdd9556d2,c486d79faca88d7d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 13:12:28.778 INFO 21062 --- [ocp-schedules-19,b02db5ca2dd1b8d0,bcce94fb36f5259a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:12:28.778 INFO 21062 --- [ocp-schedules-19,b02db5ca2dd1b8d0,bcce94fb36f5259a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:12:28.780 WARN 21062 --- [ocp-async-2,e8e225d858f8975e,303df8d4e107dd20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:28.781 INFO 21062 --- [ocp-schedules-13,acc6862bdd9556d2,c486d79faca88d7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:28.782 INFO 21062 --- [ocp-schedules-24,430eeb1d661cafe3,190650b65c9e1758] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:12:28.806 INFO 21062 --- [ocp-schedules-22,d57084470364e697,3941ed965b29f262] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:12:28.811 INFO 21062 --- [ocp-schedules-22,d57084470364e697,3941ed965b29f262] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:12:28.811 INFO 21062 --- [ocp-schedules-22,d57084470364e697,3941ed965b29f262] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:12:28.817 ERROR 21062 --- [ocp-schedules-13,acc6862bdd9556d2,c486d79faca88d7d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:12:28.817 INFO 21062 --- [ocp-schedules-13,acc6862bdd9556d2,c486d79faca88d7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 13:12:28.827 INFO 21062 --- [ocp-async-3,e5e753ad84005ff0,73c2391340f72b0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:28.957 INFO 21062 --- [ocp-schedules-3,613963e46d602675,a1401c5f9e74650b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:12:28.957 INFO 21062 --- [ocp-schedules-3,613963e46d602675,a1401c5f9e74650b] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:12:28.957 INFO 21062 --- [ocp-schedules-21,2069ca6ce4bca1c7,4e98bab6eb36f6d7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:12:28.957 INFO 21062 --- [ocp-schedules-21,2069ca6ce4bca1c7,4e98bab6eb36f6d7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:12:28.962 INFO 21062 --- [ocp-schedules-21,2069ca6ce4bca1c7,4e98bab6eb36f6d7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:12:28.962 INFO 21062 --- [ocp-schedules-21,2069ca6ce4bca1c7,4e98bab6eb36f6d7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:12:28.967 INFO 21062 --- [ocp-schedules-21,2069ca6ce4bca1c7,4e98bab6eb36f6d7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:12:28.967 INFO 21062 --- [ocp-schedules-21,2069ca6ce4bca1c7,4e98bab6eb36f6d7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:12:28.983 INFO 21062 --- [pool-27-thread-1,613963e46d602675,958238bb41fd5de0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:12:29.011 WARN 21062 --- [ocp-schedules-3,613963e46d602675,a1401c5f9e74650b] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:12:29.011 INFO 21062 --- [ocp-schedules-3,613963e46d602675,a1401c5f9e74650b] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:12:29.191 INFO 21062 --- [ocp-schedules-28,97af06dc190c6739,00795082581a958c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:12:29.623 INFO 21062 --- [ocp-schedules-23,8d86572825423daf,4e625aa8f8932042] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:12:29.623 INFO 21062 --- [ocp-schedules-23,8d86572825423daf,4e625aa8f8932042] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:12:29.627 INFO 21062 --- [ocp-schedules-27,d2a26f9dd867b15a,742ec29c6efc5d80] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:12:29.634 INFO 21062 --- [ocp-schedules-27,d2a26f9dd867b15a,742ec29c6efc5d80] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:12:29.634 INFO 21062 --- [ocp-schedules-27,d2a26f9dd867b15a,742ec29c6efc5d80] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:12:29.650 INFO 21062 --- [ocp-schedules-23,8d86572825423daf,4e625aa8f8932042] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:12:29.650 INFO 21062 --- [ocp-schedules-23,8d86572825423daf,4e625aa8f8932042] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:12:29.651 INFO 21062 --- [ocp-schedules-2,4c088e5baf3bbaf1,65557a9c8b5de404] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:12:29.732 INFO 21062 --- [ocp-schedules-24,fccb7fcf7e7239fd,64d7eace433df946] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:12:29.732 INFO 21062 --- [ocp-schedules-18,7361fa2391c3e5fa,f2b302b8c2066857] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:12:29.733 INFO 21062 --- [ocp-schedules-18,7361fa2391c3e5fa,f2b302b8c2066857] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:12:29.733 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:12:29.733 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:12:29.736 INFO 21062 --- [ocp-schedules-24,fccb7fcf7e7239fd,64d7eace433df946] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:29.743 INFO 21062 --- [ocp-schedules-18,7361fa2391c3e5fa,f2b302b8c2066857] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:12:29.743 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:12:29.743 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:12:29.745 WARN 21062 --- [ocp-schedules-18,7361fa2391c3e5fa,f2b302b8c2066857] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:12:29.745 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:12:29.745 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:12:29.745 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:12:29.746 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:12:29.746 INFO 21062 --- [ocp-schedules-18,7361fa2391c3e5fa,f2b302b8c2066857] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:12:29.747 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:12:29.747 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:12:29.747 WARN 21062 --- [ocp-schedules-18,7361fa2391c3e5fa,f2b302b8c2066857] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:12:29.754 ERROR 21062 --- [ocp-schedules-18,7361fa2391c3e5fa,f2b302b8c2066857] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:12:29.767 INFO 21062 --- [ocp-schedules-11,5bd049a3a73815ef,10d5785b3e396175] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:12:29.767+08:00, clusters=[1] 2024-11-04 13:12:29.768 INFO 21062 --- [ocp-schedules-11,5bd049a3a73815ef,10d5785b3e396175] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:12:29.793 ERROR 21062 --- [ocp-schedules-24,fccb7fcf7e7239fd,64d7eace433df946] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:12:29.793 INFO 21062 --- [ocp-schedules-24,fccb7fcf7e7239fd,64d7eace433df946] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:12:29.872 INFO 21062 --- [ocp-schedules-13,e74a51a6ca19b202,78f41ac9bda7832b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:12:29.872+08:00, clusters=[1] 2024-11-04 13:12:29.875 INFO 21062 --- [ocp-schedules-13,e74a51a6ca19b202,78f41ac9bda7832b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:12:29.983 INFO 21062 --- [prometheus-minute-schedule-1,32c9e84b641be154,25991f79b5cfa792] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:30.253 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:30.254 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:12:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:12:30.780 WARN 21062 --- [ocp-async-3,5a86217e8c4ad40d,3d75f099e46e30fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:30.828 INFO 21062 --- [ocp-async-9,5ada57969cf458ef,a06e01f7a33ee100] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:31.236 INFO 21062 --- [ocp-schedules-10,4ab471c58591597f,37f16ffd5c7aee33] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:12:31.236 INFO 21062 --- [ocp-schedules-10,91a9f541df8c8f64,b555c67fc6b82f30] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:12:31.240 INFO 21062 --- [ocp-schedules-10,91a9f541df8c8f64,b555c67fc6b82f30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:31.270 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:12:31.270 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:12:32.166 INFO 21062 --- [ocp-schedules-24,9fcd4b550fc14494,a7e796ef4f056589] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:12:32.170 INFO 21062 --- [ocp-schedules-24,9fcd4b550fc14494,a7e796ef4f056589] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:32.253 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:12:32.256 ERROR 21062 --- [ocp-schedules-24,9fcd4b550fc14494,a7e796ef4f056589] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:12:32.256 INFO 21062 --- [ocp-schedules-24,9fcd4b550fc14494,a7e796ef4f056589] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:12:32.780 WARN 21062 --- [ocp-async-9,eb01a58596cca23f,225393bb91c9fa7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:32.828 INFO 21062 --- [ocp-async-10,bd826d9ee6a9b80f,d306265fd3c2eac4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:32.983 INFO 21062 --- [prometheus-minute-schedule-1,c86b4deaea9f02aa,e531a587174420da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:33.254 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:33.255 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:34.780 WARN 21062 --- [ocp-async-10,26df1ce52616b973,f5db6c322485eab2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:34.828 INFO 21062 --- [ocp-async-6,8a7a94e144932327,04934cab30cc499a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:35.983 INFO 21062 --- [prometheus-minute-schedule-1,d7cebe6206fbf9bd,b43a78c6068448ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:36.255 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:36.256 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:36.780 WARN 21062 --- [ocp-async-6,c295f7c622071d0e,d3baeec0b59d2f5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:36.828 INFO 21062 --- [ocp-async-4,6590f70894461393,0c5e6418e1bae1ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:37.876 INFO 21062 --- [ocp-schedules-23,617ba3ddb2041754,3994262426a17984] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:12:38.504 INFO 21062 --- [ocp-schedules-17,f4b73737ecac5a21,0ef3987fa592d52a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:12:38.504 INFO 21062 --- [ocp-schedules-25,4527bb18c19126c1,d0971ce916eb5e9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:12:38.508 INFO 21062 --- [ocp-schedules-12,f4b03ef8f3005b80,fe1237e38f403acf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:12:38.579 INFO 21062 --- [ocp-async-7,96795a3e42df197d,ab2527c0ad860fc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:12:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:12:38.589954+08:00, lastProcessTime=2024-11-04T13:12:28.588678+08:00 2024-11-04 13:12:38.781 WARN 21062 --- [ocp-async-1,7dc2826dcccca463,16087dcc03d01fa8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:38.793 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:12:38.589954+08:00 2024-11-04 13:12:38.829 INFO 21062 --- [ocp-async-10,4abe43dff4b17d58,72e0394c942f8fdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c7d64aa31baa1bf0,d90040b127fb628f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:39.194 INFO 21062 --- [ocp-schedules-29,92d63b04348a7825,ff1628f9c0ece8a2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:12:39.256 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:39.256 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:12:40.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:12:40.781 WARN 21062 --- [ocp-async-2,a3fbf7552225be9d,15bc8e637f5e392e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:40.829 INFO 21062 --- [ocp-async-6,e218759d27e95d6a,73ae21942254bb6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:12:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:12:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:12:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:12:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:12:41.829 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:12:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 13:12:41.983 INFO 21062 --- [prometheus-minute-schedule-1,19c2b364ad22cfda,d745ffa8b715253f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:42.257 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:42.257 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:42.781 WARN 21062 --- [ocp-async-3,82d623f2658ead58,12f0767eb2785cb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:42.829 INFO 21062 --- [ocp-async-4,8bbef09b682c6cad,c278c780725e6cf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:44.781 WARN 21062 --- [ocp-async-9,8c88086815956a50,b4fe04d0fa212ce8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:44.829 INFO 21062 --- [ocp-async-5,4b2dd4a874a9017d,5cac0a96d5a9e76d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6a99db4f23c0ac86,214a29b556ac5b67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:45.258 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:45.259 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:46.781 WARN 21062 --- [ocp-async-10,ccc3c3a710a0ff7d,cb49c2885257e591] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:46.829 INFO 21062 --- [ocp-async-8,e6b26e0e18a90dca,8b2b1613491127d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:47.918 INFO 21062 --- [ocp-schedules-2,f26810e97af06e74,49aa3e0cd765a888] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:12:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e03668606c1c5904,f1f48934c60bc6c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:48.259 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:48.260 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:48.579 INFO 21062 --- [ocp-async-2,baf5c39ad5808513,1dd99448a4424ebc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:12:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:12:48.591754+08:00, lastProcessTime=2024-11-04T13:12:38.589954+08:00 2024-11-04 13:12:48.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:12:48.591754+08:00 2024-11-04 13:12:48.782 WARN 21062 --- [ocp-async-3,cb0951b1a2a8f9a9,d97b436a739323e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:48.829 INFO 21062 --- [ocp-async-4,ef2df133b95c5858,8602fab0df57d1bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:49.196 INFO 21062 --- [ocp-schedules-24,23331f324848bbdd,938200335cc2d845] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:12:49.637 INFO 21062 --- [ocp-schedules-11,2777dcf19f35bf1c,c42663b1468404e7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:12:49.644 INFO 21062 --- [ocp-schedules-11,2777dcf19f35bf1c,c42663b1468404e7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:12:49.644 INFO 21062 --- [ocp-schedules-11,2777dcf19f35bf1c,c42663b1468404e7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:12:50.782 WARN 21062 --- [ocp-async-9,a4f1bda311604f42,a656684c64fd5918] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:50.829 INFO 21062 --- [ocp-async-5,b086b9fc1b99f47a,dd2e613944c9c1e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:50.983 INFO 21062 --- [prometheus-minute-schedule-1,60d5419eaa0ab4b7,55e543ae1472553f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:51.260 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:51.261 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:52.782 WARN 21062 --- [ocp-async-10,84a916175a5c2cf1,7a177c947a58af0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:52.830 INFO 21062 --- [ocp-async-8,61dee20ba7d1842c,5dc3fc242ace6f28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a3f33106a3f513aa,69d5719f2635d762] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:54.261 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:54.262 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:54.782 WARN 21062 --- [ocp-async-6,e21ba233ffeb6c00,3307d6220afed460] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:54.830 INFO 21062 --- [ocp-async-7,fd907d95ed707350,a067bde9e0a20675] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:56.782 WARN 21062 --- [ocp-async-4,b7a9e3317d61dd0f,ffcd1d9063363424] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:56.830 INFO 21062 --- [ocp-async-1,522971f6b26220bf,9304fad26b944c3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d037c5395ce2926a,118a901d17ef6e3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:12:57.262 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:12:57.263 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:12:57.943 INFO 21062 --- [ocp-schedules-16,98a60c27aa3bbbef,eb1225494eb0e51b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:12:58.499 INFO 21062 --- [ocp-schedules-2,e2b2e5bf2015a199,cf8ba6096103371c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:12:58.500 INFO 21062 --- [ocp-schedules-12,965452f09eb86fbc,021d841daeba07de] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:12:58.502 INFO 21062 --- [ocp-schedules-26,d9f0a9974454794f,1f9094f70753326e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:12:58.504 INFO 21062 --- [ocp-schedules-12,965452f09eb86fbc,021d841daeba07de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:58.504 INFO 21062 --- [ocp-schedules-2,e2b2e5bf2015a199,cf8ba6096103371c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:12:58.505 INFO 21062 --- [ocp-schedules-27,3bf5ede224f44edd,5f5393558017a45b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:12:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:12:58.512 INFO 21062 --- [ocp-schedules-23,422912a30d57150b,c7bd4f248d381117] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:12:58.512 INFO 21062 --- [ocp-schedules-14,58da74e9dbc60504,f61038bda9d70f47] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:12:58.512 INFO 21062 --- [ocp-schedules-15,d5c7c2581ac83ed0,c9073291e4280be9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:12:58.512 INFO 21062 --- [ocp-schedules-15,d5c7c2581ac83ed0,c9073291e4280be9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:12:58.512 INFO 21062 --- [ocp-schedules-23,422912a30d57150b,c7bd4f248d381117] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:12:58.512 INFO 21062 --- [ocp-schedules-14,58da74e9dbc60504,f61038bda9d70f47] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:12:58.579 INFO 21062 --- [ocp-async-4,9eb53c10487a6539,5c3fa494e9f9a6a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:12:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:12:58.588463+08:00, lastProcessTime=2024-11-04T13:12:48.591754+08:00 2024-11-04 13:12:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:12:58.588463+08:00 2024-11-04 13:12:58.782 WARN 21062 --- [ocp-async-5,855764f148d04d3d,fbb7b63a6befc4d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:12:58.784 INFO 21062 --- [ocp-schedules-8,ea08c9fb9e63bd64,8d51d86d4b868f97] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:12:58.830 INFO 21062 --- [ocp-async-8,41ef64efb561b1de,0cde260ecd5eaf40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:12:59.199 INFO 21062 --- [ocp-schedules-28,f58f419cfe7714ba,345460bfea86b94b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:12:59.983 INFO 21062 --- [prometheus-minute-schedule-1,73c98b5d29163251,7338dbb9145088ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697180, interval=60 2024-11-04 13:13:00.263 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:00.264 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:00.783 WARN 21062 --- [ocp-async-8,c3daf2c858818c7d,eadfdc4ca4f35b94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:00.830 INFO 21062 --- [ocp-async-7,f60e622b247ef252,c1542117bce821cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:01.270 INFO 21062 --- [ocp-schedules-10,91a9f541df8c8f64,b555c67fc6b82f30] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:13:01.271 INFO 21062 --- [ocp-schedules-10,212a39fa338ea517,7552e753986c6b34] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:13:01.276 INFO 21062 --- [ocp-schedules-10,212a39fa338ea517,7552e753986c6b34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:01.306 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:13:01.306 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:13:02.783 WARN 21062 --- [ocp-async-7,b266091c91441fb0,64656064afe52e71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:02.831 INFO 21062 --- [ocp-async-1,5b55ba2d3496eef6,c3cb41ace9330e3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:02.983 INFO 21062 --- [prometheus-minute-schedule-1,10a2c172f77fca62,acbeca8960cb7e41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:03.264 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:03.265 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:04.783 WARN 21062 --- [ocp-async-1,2868640c1be47828,0d77bfe568ce024a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:04.831 INFO 21062 --- [ocp-async-2,f5f4529a5f4c7d20,1ddc3b4f449aba79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c5a35461761db740,fa910232b0ebf625] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:06.265 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:06.266 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:06.783 WARN 21062 --- [ocp-async-2,fcb879432b50254f,5df999a55155ad26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:06.831 INFO 21062 --- [ocp-async-3,cb56203295e87900,8672c3d43377f8c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:07.985 INFO 21062 --- [ocp-schedules-7,1a31537a7f846253,62956821557ce436] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:13:08.505 INFO 21062 --- [ocp-schedules-12,965452f09eb86fbc,021d841daeba07de] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:13:08.505 INFO 21062 --- [ocp-schedules-2,e2b2e5bf2015a199,cf8ba6096103371c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:13:08.579 INFO 21062 --- [ocp-async-8,b32e073237f569b9,63f46a44b563ee34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:13:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:13:08.589665+08:00, lastProcessTime=2024-11-04T13:12:58.588463+08:00 2024-11-04 13:13:08.707 INFO 21062 --- [ocp-schedules-4,c4582740f7bde441,05180d32f8e532f4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:13:08.710 INFO 21062 --- [ocp-schedules-4,c4582740f7bde441,05180d32f8e532f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:08.711 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:13:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:13:08.589665+08:00 2024-11-04 13:13:08.783 WARN 21062 --- [ocp-async-7,45dcd47c30d5ef09,a7e66ca37e033784] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:08.831 INFO 21062 --- [ocp-async-1,ec149fbe330a36a6,e41af83a9d08bd84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:08.983 INFO 21062 --- [prometheus-minute-schedule-1,41eb427fda217896,c6987e3f13ca6eec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:09.202 INFO 21062 --- [ocp-schedules-21,96d2aba8ff40140e,409965f66f074c5a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:13:09.266 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:09.267 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:09.647 INFO 21062 --- [ocp-schedules-11,520204777f656942,ca26f43e2eae3f1b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:13:09.653 INFO 21062 --- [ocp-schedules-11,520204777f656942,ca26f43e2eae3f1b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:13:09.653 INFO 21062 --- [ocp-schedules-11,520204777f656942,ca26f43e2eae3f1b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:13:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:13:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:13:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:13:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:13:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:13:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:13:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:13:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:13:10.784 WARN 21062 --- [ocp-async-1,5ac61e7e599392ba,e4a0265becdfd078] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:10.831 INFO 21062 --- [ocp-async-2,f55ac951a7dd7715,5c7c59c8a1004e4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:11.194 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=16, affectRows=16 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=16, validCount=16 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=13, affectRows=13 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=57, affectRows=57 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=3, affectRows=3 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=35, affectRows=35 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=3, validCount=3 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=13, validCount=13 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=35, validCount=35 2024-11-04 13:13:11.195 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=56, validCount=56 2024-11-04 13:13:11.196 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=52, affectRows=52 2024-11-04 13:13:11.196 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=51, validCount=51 2024-11-04 13:13:11.196 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=39, affectRows=39 2024-11-04 13:13:11.197 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=39, validCount=39 2024-11-04 13:13:11.197 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=67, affectRows=67 2024-11-04 13:13:11.197 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697120, nodeCount=40, affectRows=40 2024-11-04 13:13:11.197 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=39, validCount=39 2024-11-04 13:13:11.197 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697120, nodeCount=67, validCount=67 2024-11-04 13:13:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4ee889bd4e9bbf17,083dadffc7eadfae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:12.267 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:12.268 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:12.784 WARN 21062 --- [ocp-async-2,2fb62e852c27fc33,f5a107189255069f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:12.832 INFO 21062 --- [ocp-async-3,f7b2c2575ec79d2f,2a71d62fbb9e29d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:13.984 INFO 21062 --- [prometheus-cache-monitor1,0e9a9d942df74326,8ae0a4588d889151] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:13:13.985 INFO 21062 --- [prometheus-cache-monitor1,0e9a9d942df74326,8ae0a4588d889151] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:13:13.988 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:07:13.97, end=2024-11-04T13:12:13.985} 2024-11-04 13:13:13.989 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:07:13.97, end=2024-11-04T13:12:13.985} 2024-11-04 13:13:13.998 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:13:13.998 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:13:13.998 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:13:13.998 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:13:14.409 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:09:14.391, end=2024-11-04T13:11:14.407} 2024-11-04 13:13:14.413 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:13:14.413 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:13:14.784 WARN 21062 --- [ocp-async-3,cbcc319cee225c18,ac85e0fe7dae6b7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:14.832 INFO 21062 --- [ocp-async-9,2f1c50bee305fb6e,943887fcc5fe08cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:14.983 INFO 21062 --- [prometheus-minute-schedule-1,93a719015baf0c1d,e7658eda2a53cae7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:15.268 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:15.269 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:16.784 WARN 21062 --- [ocp-async-9,858c85807537eea1,28e6f4bf1dd04fee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:16.832 INFO 21062 --- [ocp-async-5,0e1bb41a51147a9d,00e01b674514c371] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ed34645d6789f82f,678a9a34d168f556] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:18.015 INFO 21062 --- [ocp-schedules-5,38b437c41ca8aa62,a0bb4599369904ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:13:18.269 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:18.270 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:18.580 INFO 21062 --- [ocp-async-1,4cfbd0c7ad4a905b,1871790a4f101cc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:13:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:13:18.589706+08:00, lastProcessTime=2024-11-04T13:13:08.589665+08:00 2024-11-04 13:13:18.711 INFO 21062 --- [ocp-schedules-4,c4582740f7bde441,05180d32f8e532f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:13:18.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:13:18.589706+08:00 2024-11-04 13:13:18.784 WARN 21062 --- [ocp-async-2,7e11ce3962677c36,368e40d41fc2049d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:18.832 INFO 21062 --- [ocp-async-3,999e4b8f07c1cb3f,3fff0997fde29d80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:19.205 INFO 21062 --- [ocp-schedules-2,c552b55e06770e69,e27f0c22da0251c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:13:20.784 WARN 21062 --- [ocp-async-3,8c887d182cd183eb,b0bd29d92f01823d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:20.832 INFO 21062 --- [ocp-async-9,759abb693137fad3,59ea7d5112c4b6c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9974dc51b8522195,39ddd1d7be9795df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:21.270 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:21.272 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:22.785 WARN 21062 --- [ocp-async-9,c2b07cfd6c20af72,accfa95442c49171] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:22.832 INFO 21062 --- [ocp-async-5,f76f339267afbdb5,9cadc97c3bbb0e6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b21dee2455bcee02,cf9d2b82f48f169c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:24.272 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:24.273 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:24.785 WARN 21062 --- [ocp-async-5,1c215ad8303ad9dc,c237b022f5dbfc81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:24.832 INFO 21062 --- [ocp-async-6,95a13e8cc2d22789,e4bbe63017e579dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:26.785 WARN 21062 --- [ocp-async-6,8faef20e31aa853f,c53c1c68cf9b08ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:26.833 INFO 21062 --- [ocp-async-7,295712922d0f3f37,5597a077ee9f3b1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9039f211dec981af,a36245381b463261] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4273449c1f2e61f6,c260e32c3dde4496] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:13:27.273 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:27.274 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:28.059 INFO 21062 --- [ocp-schedules-16,0ed7cf12230a2cb0,164ea22af8b3c943] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:13:28.498 INFO 21062 --- [ocp-schedules-23,7767d978fc17f1aa,f409ea99a7f1f79b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:13:28.499 INFO 21062 --- [ocp-schedules-5,271fcc2594ccb298,d980cfc7196458c5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:13:28.500 INFO 21062 --- [ocp-schedules-13,89cb55133d582771,3ceacea63d0413e3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:13:28.500 INFO 21062 --- [ocp-schedules-21,d05a29d2ce7c7bab,e82a6d8113e1518e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:13:28.502 INFO 21062 --- [ocp-schedules-25,2ae212e022dd1d19,77a32dc894d449db] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:13:28.503 INFO 21062 --- [ocp-schedules-23,7767d978fc17f1aa,f409ea99a7f1f79b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:13:28.503 INFO 21062 --- [ocp-schedules-13,89cb55133d582771,3ceacea63d0413e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:28.503 INFO 21062 --- [ocp-schedules-21,d05a29d2ce7c7bab,e82a6d8113e1518e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:13:28.504 INFO 21062 --- [ocp-schedules-30,d662c7cc7b84f998,8eb28cb5516cf040] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:13:28.505 INFO 21062 --- [ocp-schedules-5,271fcc2594ccb298,d980cfc7196458c5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:13:28.512 INFO 21062 --- [ocp-schedules-1,e8a6e6f041409e9c,f82da5f23fe79c1f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:13:28.512 INFO 21062 --- [ocp-schedules-17,981e08edd1997344,a14d598199ca3c74] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:13:28.512 INFO 21062 --- [ocp-schedules-1,e8a6e6f041409e9c,f82da5f23fe79c1f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:13:28.512 INFO 21062 --- [ocp-schedules-17,981e08edd1997344,a14d598199ca3c74] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:13:28.512 INFO 21062 --- [ocp-schedules-20,8be0cd8c4ae0a429,9abe472abdd19117] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:13:28.512 INFO 21062 --- [ocp-schedules-20,8be0cd8c4ae0a429,9abe472abdd19117] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:13:28.555 INFO 21062 --- [ocp-schedules-6,5dea9cbf342c8552,e8e818075489757d] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 13:13:28.555 INFO 21062 --- [ocp-schedules-6,5dea9cbf342c8552,e8e818075489757d] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 13:13:28.580 INFO 21062 --- [ocp-async-7,8d3fcb3188594a37,fc150c1ff26cede2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:13:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:13:28.588677+08:00, lastProcessTime=2024-11-04T13:13:18.589706+08:00 2024-11-04 13:13:28.658 INFO 21062 --- [ocp-schedules-15,2aeb97baad0d2e98,742fdb795996ba7d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:13:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:13:28.588677+08:00 2024-11-04 13:13:28.783 INFO 21062 --- [ocp-schedules-24,8659775fbcfd6fa9,17509ca6349d9a63] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:13:28.783 INFO 21062 --- [ocp-schedules-24,8659775fbcfd6fa9,17509ca6349d9a63] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:13:28.785 WARN 21062 --- [ocp-async-10,daa5deeaa1dd3ae0,04a10a6480d458ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:28.787 INFO 21062 --- [ocp-schedules-3,224d07c3413d7b78,b115320c85e7fd28] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:13:28.809 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:13:28.809 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:13:28.811 INFO 21062 --- [ocp-schedules-1,762f4f809dca597d,72fb5d15fc014ecf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:13:28.816 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:13:28.816 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:13:28.816 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:13:28.817 INFO 21062 --- [ocp-schedules-1,762f4f809dca597d,72fb5d15fc014ecf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:13:28.817 INFO 21062 --- [ocp-schedules-1,762f4f809dca597d,72fb5d15fc014ecf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:13:28.827 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:13:28.827 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:13:28.827 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:13:28.830 INFO 21062 --- [ocp-schedules-30,3f3124ea0227c01e,746933b0d0687857] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:13:28.833 INFO 21062 --- [ocp-async-2,453044ffd5b442a1,69322454e369c1bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:28.968 INFO 21062 --- [ocp-schedules-9,f08e4bcdcd4d3d49,3f1916c70c3a263a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:13:28.968 INFO 21062 --- [ocp-schedules-9,f08e4bcdcd4d3d49,3f1916c70c3a263a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:13:28.968 INFO 21062 --- [ocp-schedules-9,f08e4bcdcd4d3d49,3f1916c70c3a263a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:13:28.968 INFO 21062 --- [ocp-schedules-9,f08e4bcdcd4d3d49,3f1916c70c3a263a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:13:28.969 INFO 21062 --- [ocp-schedules-9,f08e4bcdcd4d3d49,3f1916c70c3a263a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:13:28.969 INFO 21062 --- [ocp-schedules-9,f08e4bcdcd4d3d49,3f1916c70c3a263a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:13:29.209 INFO 21062 --- [ocp-schedules-14,5cf8a066988a44df,41223877021c8a24] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:13:29.650 INFO 21062 --- [ocp-schedules-4,4288e25c421bfca0,5e586dd646463f28] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:13:29.650 INFO 21062 --- [ocp-schedules-4,4288e25c421bfca0,5e586dd646463f28] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:13:29.656 INFO 21062 --- [ocp-schedules-5,91e9d3deb8cb8b50,5708778f0e5fc6c0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:13:29.661 INFO 21062 --- [ocp-schedules-5,91e9d3deb8cb8b50,5708778f0e5fc6c0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:13:29.661 INFO 21062 --- [ocp-schedules-5,91e9d3deb8cb8b50,5708778f0e5fc6c0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:13:29.677 INFO 21062 --- [ocp-schedules-11,e3481fe7deb69e88,e1ab282d5660ba89] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:13:29.677 INFO 21062 --- [ocp-schedules-4,4288e25c421bfca0,5e586dd646463f28] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:13:29.677 INFO 21062 --- [ocp-schedules-4,4288e25c421bfca0,5e586dd646463f28] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:13:29.762 INFO 21062 --- [ocp-schedules-24,bdbbda90ce16b39c,c35466d47bb7719d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:13:29.763 INFO 21062 --- [ocp-schedules-24,bdbbda90ce16b39c,c35466d47bb7719d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:13:29.763 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:13:29.763 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:13:29.773 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:13:29.773 INFO 21062 --- [ocp-schedules-24,bdbbda90ce16b39c,c35466d47bb7719d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:13:29.773 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:13:29.775 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:13:29.775 WARN 21062 --- [ocp-schedules-24,bdbbda90ce16b39c,c35466d47bb7719d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:13:29.775 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:13:29.776 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:13:29.776 INFO 21062 --- [ocp-schedules-24,bdbbda90ce16b39c,c35466d47bb7719d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:13:29.776 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:13:29.777 WARN 21062 --- [ocp-schedules-24,bdbbda90ce16b39c,c35466d47bb7719d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:13:29.777 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:13:29.777 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:13:29.785 ERROR 21062 --- [ocp-schedules-24,bdbbda90ce16b39c,c35466d47bb7719d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:13:29.795 INFO 21062 --- [ocp-schedules-25,6ffa027d7dd90341,10284081e58df0bc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:13:29.795+08:00, clusters=[1] 2024-11-04 13:13:29.797 INFO 21062 --- [ocp-schedules-25,6ffa027d7dd90341,10284081e58df0bc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:13:29.900 INFO 21062 --- [ocp-schedules-17,5ad100efc07e514e,565d1b1cf96a1711] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:13:29.900+08:00, clusters=[1] 2024-11-04 13:13:29.904 INFO 21062 --- [ocp-schedules-17,5ad100efc07e514e,565d1b1cf96a1711] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:13:29.983 INFO 21062 --- [prometheus-minute-schedule-1,fbff93070eca67e0,b24ade140aad7f79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:30.274 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:30.275 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:13:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:13:30.785 WARN 21062 --- [ocp-async-2,7adf62802a5adc8d,6d42b3befbe19741] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:30.833 INFO 21062 --- [ocp-async-6,312131a2d9f0fda8,e92987aa579a189d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:31.306 INFO 21062 --- [ocp-schedules-10,212a39fa338ea517,7552e753986c6b34] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:13:31.307 INFO 21062 --- [ocp-schedules-10,6c3505621a738142,871e05e77151418b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:13:31.310 INFO 21062 --- [ocp-schedules-10,6c3505621a738142,871e05e77151418b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:31.340 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:13:31.340 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:13:32.257 INFO 21062 --- [ocp-schedules-4,3d084838e78d4405,e2889453b02862e0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:13:32.261 INFO 21062 --- [ocp-schedules-4,3d084838e78d4405,e2889453b02862e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:32.345 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:13:32.350 ERROR 21062 --- [ocp-schedules-4,3d084838e78d4405,e2889453b02862e0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:13:32.351 INFO 21062 --- [ocp-schedules-4,3d084838e78d4405,e2889453b02862e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:13:32.786 WARN 21062 --- [ocp-async-6,7f94376a93dc56e9,45f240825f319646] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:32.833 INFO 21062 --- [ocp-async-9,c7ad5ffc9082af94,16364637205895e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:32.983 INFO 21062 --- [prometheus-minute-schedule-1,497fb7596cd7046d,4a4a758960a2029a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:33.275 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:33.276 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:34.786 WARN 21062 --- [ocp-async-1,9cb1a7ec1f272401,6d3a9f2ce4281ebb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:34.833 INFO 21062 --- [ocp-async-5,14a1db807f9c2628,0ebb89ab00bb73a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:35.983 INFO 21062 --- [prometheus-minute-schedule-1,8435099e293b85fb,578999ef476dfc21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:36.276 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:36.277 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:36.786 WARN 21062 --- [ocp-async-8,7afb1a0b4893b013,010a16a0577d211a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:36.833 INFO 21062 --- [ocp-async-3,16f5f8f9797e5a3c,287fecf21ad140ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:38.084 INFO 21062 --- [ocp-schedules-7,3749dddd90dcd2fb,9114c02bffae0208] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:13:38.504 INFO 21062 --- [ocp-schedules-13,89cb55133d582771,3ceacea63d0413e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:13:38.504 INFO 21062 --- [ocp-schedules-21,d05a29d2ce7c7bab,e82a6d8113e1518e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:13:38.580 INFO 21062 --- [ocp-async-3,4199e84bbc6db09c,ffc9dc2270719b0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:13:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:13:38.589207+08:00, lastProcessTime=2024-11-04T13:13:28.588677+08:00 2024-11-04 13:13:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:13:38.589207+08:00 2024-11-04 13:13:38.786 WARN 21062 --- [ocp-async-7,2070aa29efb2d369,8637c232540d826f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:38.834 INFO 21062 --- [ocp-async-1,94bc72ea9296386e,e56ea36c19cdc56b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5f564a45727197fd,617b1e895a21cbf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:39.212 INFO 21062 --- [ocp-schedules-18,8804626bcd6b5d36,fd8a6f76dbc0fee5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:13:39.277 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:39.278 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:40.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:13:40.004 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:13:40.786 WARN 21062 --- [ocp-async-10,033ec4e447b147c9,a7bd53f7600e11bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:40.834 INFO 21062 --- [ocp-async-8,406b92068e353459,93d87dd23c7932a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:41.983 INFO 21062 --- [prometheus-minute-schedule-1,53c2c870f1e16e66,e5b6c2c2f60e3f89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:42.278 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:42.279 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:42.786 WARN 21062 --- [ocp-async-2,2b79a4b56b0db8ef,e6bacbf5d0a2d36a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:42.834 INFO 21062 --- [ocp-async-4,11959e30a41e7953,dc3fb57a2e177ba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:44.787 WARN 21062 --- [ocp-async-6,52f67b27e06271e5,e7682128f1937a7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:44.834 INFO 21062 --- [ocp-async-9,c69e6293b4d22a18,9395fbc06ab91284] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:44.983 INFO 21062 --- [prometheus-minute-schedule-1,356da8eab2188e5b,b27b335d849b5323] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:45.279 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:45.280 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:46.787 WARN 21062 --- [ocp-async-1,6afa8bcbd7ae12ed,b37e775076f33f64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:46.834 INFO 21062 --- [ocp-async-5,8d8053af2d15998a,141b77111936345e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:47.983 INFO 21062 --- [prometheus-minute-schedule-1,00199a4664b7806a,90393dcd37cb4954] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:48.130 INFO 21062 --- [ocp-schedules-18,f409d978eb8119d9,88127a7e2fd24597] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:13:48.280 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:48.281 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:48.580 INFO 21062 --- [ocp-async-10,ad180304bac83ff2,a79e930149182d64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:13:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:13:48.591556+08:00, lastProcessTime=2024-11-04T13:13:38.589207+08:00 2024-11-04 13:13:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:13:48.591556+08:00 2024-11-04 13:13:48.787 WARN 21062 --- [ocp-async-2,91ba23d3bb383a78,24f050594006a040] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:48.835 INFO 21062 --- [ocp-async-4,048c6b0f2db1a0ee,4e0104980c7bee59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:49.214 INFO 21062 --- [ocp-schedules-23,882fb024684552af,8619f8b3811fba9e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:13:49.664 INFO 21062 --- [ocp-schedules-14,0d3817ad0164358f,8857db19536e4bd2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:13:49.670 INFO 21062 --- [ocp-schedules-14,0d3817ad0164358f,8857db19536e4bd2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:13:49.670 INFO 21062 --- [ocp-schedules-14,0d3817ad0164358f,8857db19536e4bd2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:13:50.787 WARN 21062 --- [ocp-async-6,5f2b5361ebbc0c88,5cccb628e3c00444] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:50.835 INFO 21062 --- [ocp-async-9,8c9c856965c73894,b3e78e05d99ac455] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:50.983 INFO 21062 --- [prometheus-minute-schedule-1,fe58e267ff8f20e6,3fdce6364edcf8c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:51.281 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:51.281 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:52.787 WARN 21062 --- [ocp-async-1,73caddb230f8a094,6ad55ea127910369] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:52.835 INFO 21062 --- [ocp-async-5,f5427f41043f6f56,380b7dccedd5b516] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:53.983 INFO 21062 --- [prometheus-minute-schedule-1,7793cf7d12e0dffa,7a010dd787760d3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:54.282 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:54.282 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:54.787 WARN 21062 --- [ocp-async-8,52a759369baf6621,c143e5b4a05ddf7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:54.835 INFO 21062 --- [ocp-async-3,8a84441c4c53a901,1c5f1ea045a2b30b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:56.788 WARN 21062 --- [ocp-async-3,d4ede46bcbb91de2,4945049af22b7d33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:56.835 INFO 21062 --- [ocp-async-7,35bfc7a16bdd7f21,32269ca9b4307039] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e76ad9794c696781,cf14c2b5dc1eb47d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:13:57.282 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:13:57.283 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:13:58.155 INFO 21062 --- [ocp-schedules-25,42b771b5aa6c535f,185dd52c8c87973a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:13:58.500 INFO 21062 --- [ocp-schedules-4,e066368d49cdc94c,b3c873cf3ef635d9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:13:58.500 INFO 21062 --- [ocp-schedules-3,8320a6e6cb46d6bd,066c94b14d94de31] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:13:58.503 INFO 21062 --- [ocp-schedules-18,92d14533ace8759e,3f565cd11ad5b79c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:13:58.504 INFO 21062 --- [ocp-schedules-4,e066368d49cdc94c,b3c873cf3ef635d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:58.504 INFO 21062 --- [ocp-schedules-14,731681b2f3ce556f,01551026d917d106] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:13:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:13:58.504 INFO 21062 --- [ocp-schedules-3,8320a6e6cb46d6bd,066c94b14d94de31] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:13:58.512 INFO 21062 --- [ocp-schedules-26,b29cbc6f6a3963ee,9fdb25481544297c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:13:58.512 INFO 21062 --- [ocp-schedules-5,85ae1f8ae51e7a42,5fe38b0c9f80c594] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:13:58.512 INFO 21062 --- [ocp-schedules-5,85ae1f8ae51e7a42,5fe38b0c9f80c594] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:13:58.512 INFO 21062 --- [ocp-schedules-26,b29cbc6f6a3963ee,9fdb25481544297c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:13:58.513 INFO 21062 --- [ocp-schedules-1,e2d7168fcef28127,3e663179eecba518] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:13:58.513 INFO 21062 --- [ocp-schedules-1,e2d7168fcef28127,3e663179eecba518] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:13:58.581 INFO 21062 --- [ocp-async-4,81b52780ee4578ef,0cb3b04acec43369] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:13:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:13:58.589897+08:00, lastProcessTime=2024-11-04T13:13:48.591556+08:00 2024-11-04 13:13:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:13:58.589897+08:00 2024-11-04 13:13:58.788 WARN 21062 --- [ocp-async-9,6fae0c9d6e620daa,493d435978e8dd3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:13:58.790 INFO 21062 --- [ocp-schedules-9,420704013af15a6c,4581902545aa4130] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:13:58.835 INFO 21062 --- [ocp-async-5,ca46dcb6efe6ccea,c6b2aaebf688df7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:13:59.217 INFO 21062 --- [ocp-schedules-13,ecfd0f63f109380e,d8e6992968b2f43b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:13:59.983 INFO 21062 --- [prometheus-minute-schedule-1,664b168c5d703b54,8f588d4526a98485] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697240, interval=60 2024-11-04 13:14:00.283 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:00.284 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:00.788 WARN 21062 --- [ocp-async-5,2d8164011ed09d08,3dc59a7b7d551187] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:00.836 INFO 21062 --- [ocp-async-3,078d10ebe64de40f,707b923e4e2865af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:01.341 INFO 21062 --- [ocp-schedules-10,6c3505621a738142,871e05e77151418b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:14:01.341 INFO 21062 --- [ocp-schedules-10,c7c013d53ea90f63,7032095ce50a0a24] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:14:01.346 INFO 21062 --- [ocp-schedules-10,c7c013d53ea90f63,7032095ce50a0a24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:01.377 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:14:01.377 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:14:02.788 WARN 21062 --- [ocp-async-3,37f8a975c13553fd,0781a451f7f91300] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:02.836 INFO 21062 --- [ocp-async-7,260810c29a17232d,42bade5ef33fddfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:02.983 INFO 21062 --- [prometheus-minute-schedule-1,58e2ce99e0a20d11,e2365c4b2cd85235] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:03.285 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:03.285 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:04.788 WARN 21062 --- [ocp-async-7,dc7acae1f2c02b43,f78ca97f4451ef0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:04.836 INFO 21062 --- [ocp-async-10,2d211451c1db4766,3be446ae9be29793] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:05.983 INFO 21062 --- [prometheus-minute-schedule-1,5170e1b2cc961ee4,81f668cb0aaefa1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:06.285 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:06.286 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:06.788 WARN 21062 --- [ocp-async-10,5ca09157c3efa554,42c8648c95249340] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:06.836 INFO 21062 --- [ocp-async-8,22d720c775d6a6b1,ea7b6552c43a0c5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:08.197 INFO 21062 --- [ocp-schedules-16,b85255e50396756b,84985298c41bfa2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:14:08.505 INFO 21062 --- [ocp-schedules-3,8320a6e6cb46d6bd,066c94b14d94de31] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:14:08.505 INFO 21062 --- [ocp-schedules-4,e066368d49cdc94c,b3c873cf3ef635d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:14:08.581 INFO 21062 --- [ocp-async-5,346ab5ba33fcbe96,9e87c80df8ff2c58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:14:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:14:08.589520+08:00, lastProcessTime=2024-11-04T13:13:58.589897+08:00 2024-11-04 13:14:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:14:08.589520+08:00 2024-11-04 13:14:08.789 WARN 21062 --- [ocp-async-3,9592e11223552116,accc393eb640f94e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:08.836 INFO 21062 --- [ocp-async-7,e106288e8a3cad98,f704a715bfd104f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:08.983 INFO 21062 --- [prometheus-minute-schedule-1,cd0483dc2b1d90a6,2b9200209dfca738] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:09.220 INFO 21062 --- [ocp-schedules-14,9e3093b3d99e728b,1e970c372df1e4ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:14:09.286 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:09.287 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:09.673 INFO 21062 --- [ocp-schedules-5,e8b4e9aea95525de,9b016db42bb1318e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:14:09.680 INFO 21062 --- [ocp-schedules-5,e8b4e9aea95525de,9b016db42bb1318e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:14:09.680 INFO 21062 --- [ocp-schedules-5,e8b4e9aea95525de,9b016db42bb1318e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:14:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:14:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:14:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:14:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:14:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:14:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:14:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:14:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:14:10.789 WARN 21062 --- [ocp-async-7,9f0cf06d23d4adb0,1e5ad51447002871] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:10.836 INFO 21062 --- [ocp-async-10,958de69c28f39fbe,7be9d815763b44ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=1, affectRows=1 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=4, affectRows=4 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=2, affectRows=2 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=1, validCount=1 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=2, validCount=2 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=8, affectRows=8 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=4, validCount=4 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=11, affectRows=11 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=13, affectRows=13 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=8, validCount=8 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=11, validCount=11 2024-11-04 13:14:11.200 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=13, validCount=13 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=7, affectRows=7 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=14, affectRows=14 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=7, validCount=7 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=14, validCount=14 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=33, affectRows=33 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=36, affectRows=36 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=33, validCount=33 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=35, validCount=35 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=17, affectRows=17 2024-11-04 13:14:11.201 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=16, validCount=16 2024-11-04 13:14:11.202 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=26, affectRows=26 2024-11-04 13:14:11.202 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=26, validCount=26 2024-11-04 13:14:11.203 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=32, affectRows=32 2024-11-04 13:14:11.203 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=32, validCount=32 2024-11-04 13:14:11.204 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=37, affectRows=37 2024-11-04 13:14:11.204 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=39, affectRows=39 2024-11-04 13:14:11.204 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=37, validCount=37 2024-11-04 13:14:11.204 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=38, validCount=38 2024-11-04 13:14:11.204 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697180, nodeCount=42, affectRows=42 2024-11-04 13:14:11.204 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697180, nodeCount=42, validCount=42 2024-11-04 13:14:11.983 INFO 21062 --- [prometheus-minute-schedule-1,56e407eb50ce1ebf,9bc8d57faab4a0ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:12.287 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:12.288 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:12.789 WARN 21062 --- [ocp-async-10,1529eb12a646eb14,2e688b6f3002179e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:12.837 INFO 21062 --- [ocp-async-8,9c1f2056c42732c0,79ae449075bfbc2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:13.984 INFO 21062 --- [prometheus-cache-monitor1,793dbc949f83715b,1eaee600f9f3605c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:14:13.984 INFO 21062 --- [prometheus-cache-monitor1,793dbc949f83715b,1eaee600f9f3605c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:14:14.789 WARN 21062 --- [ocp-async-8,4d0050e555cc99d9,38e85b0891e4239c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:14.837 INFO 21062 --- [ocp-async-6,ddfd7d8a0ac300ff,a33e7be0c9b99784] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:14.983 INFO 21062 --- [prometheus-minute-schedule-1,cdf7d3c43705d5c8,cbd90ab2912f5236] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:15.288 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:15.289 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:16.789 WARN 21062 --- [ocp-async-6,081a6dec23a6ad1b,f2c5343066260a1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:16.837 INFO 21062 --- [ocp-async-1,f758fad2d6b8bcb4,e2fd9fd6a2b46edd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5c4b50a1a029a225,054b0b87e25fd8b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:18.225 INFO 21062 --- [ocp-schedules-27,6627d4af55260ca7,1cd9512310eb3cca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:14:18.289 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:18.290 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:18.581 INFO 21062 --- [ocp-async-7,15c841dceead39d4,7ce416b527d90c60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:14:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:14:18.588823+08:00, lastProcessTime=2024-11-04T13:14:08.589520+08:00 2024-11-04 13:14:18.712 INFO 21062 --- [ocp-schedules-21,b2845dbf50a4c8a4,d93e639b9bdb3eef] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:14:18.715 INFO 21062 --- [ocp-schedules-21,b2845dbf50a4c8a4,d93e639b9bdb3eef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:18.716 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:14:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:14:18.588823+08:00 2024-11-04 13:14:18.789 WARN 21062 --- [ocp-async-10,49e9e0abc7999545,82b01f5311059059] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:18.837 INFO 21062 --- [ocp-async-8,80128e35f981285a,9b470dbba39b2fe6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:19.224 INFO 21062 --- [ocp-schedules-12,7abaef4626636d3d,9733da13b66389b8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:14:20.790 WARN 21062 --- [ocp-async-8,21ac2b86a979f0a4,bd4ea9c03db9ba0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:20.837 INFO 21062 --- [ocp-async-6,219f5e350ed11cf9,62eeca1b1fbdce57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:20.983 INFO 21062 --- [prometheus-minute-schedule-1,c1d41f63f90f7957,4b45c7b7b9265fd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:21.290 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:21.291 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:22.790 WARN 21062 --- [ocp-async-6,bd6c441d42ba7270,48ec9f209ea988d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:22.837 INFO 21062 --- [ocp-async-1,83e500252789d0d7,2d4db753eb978c7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:23.983 INFO 21062 --- [prometheus-minute-schedule-1,58c81ef600af0048,9fe8e01ed46eadf4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:24.291 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:24.292 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:24.791 WARN 21062 --- [ocp-async-1,13c72ef4223a4a47,eacc06f25505aae2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:24.838 INFO 21062 --- [ocp-async-5,c277a2148c61c87a,f3bbf994daf1c194] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:26.792 WARN 21062 --- [ocp-async-5,cd0c19cb4b2cb773,07cf7157479ca18d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:26.838 INFO 21062 --- [ocp-async-3,870f404589b7c2b4,2242e4c2dd05eb6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:26.983 INFO 21062 --- [prometheus-minute-schedule-1,49d264203eafd78e,d108a3ebc4bdf8fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:26.984 INFO 21062 --- [prometheus-hour-schedule-1,34d66dfd21f01672,6c1fb602baae2278] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:14:27.292 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:27.293 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:28.268 INFO 21062 --- [ocp-schedules-20,0753cd23737fbcf8,b284d720a3d3fc9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:14:28.498 INFO 21062 --- [ocp-schedules-28,03c2865158560a70,b73db65db7d0c21c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:14:28.499 INFO 21062 --- [ocp-schedules-13,5998f98ef41fe416,90f0cb1365c4c860] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:14:28.500 INFO 21062 --- [ocp-schedules-27,438e67a12d7af501,4ff414c5a311369a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:14:28.500 INFO 21062 --- [ocp-schedules-25,e53807cb1186d5eb,518cd0840c17f1d0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:14:28.502 INFO 21062 --- [ocp-schedules-28,03c2865158560a70,b73db65db7d0c21c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:14:28.502 INFO 21062 --- [ocp-schedules-2,c64051f92844e6b8,6935ba9ba49a5867] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:14:28.503 INFO 21062 --- [ocp-schedules-27,438e67a12d7af501,4ff414c5a311369a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:28.503 INFO 21062 --- [ocp-schedules-25,e53807cb1186d5eb,518cd0840c17f1d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:14:28.504 INFO 21062 --- [ocp-schedules-22,e0572b4ab003dc93,4e87751502f38f72] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:14:28.504 INFO 21062 --- [ocp-schedules-13,5998f98ef41fe416,90f0cb1365c4c860] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:14:28.511 INFO 21062 --- [ocp-schedules-8,44445100a539e887,b3909a699ea5caa5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:14:28.511 INFO 21062 --- [ocp-schedules-14,d8a1cbe37aa454b1,cd13af76735fc818] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:14:28.511 INFO 21062 --- [ocp-schedules-8,44445100a539e887,b3909a699ea5caa5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:14:28.511 INFO 21062 --- [ocp-schedules-14,d8a1cbe37aa454b1,cd13af76735fc818] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:14:28.511 INFO 21062 --- [ocp-schedules-6,45800e7dd8b4c889,ef32c8e6e4dc7a18] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:14:28.511 INFO 21062 --- [ocp-schedules-6,45800e7dd8b4c889,ef32c8e6e4dc7a18] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:14:28.548 INFO 21062 --- [ocp-schedules-18,394868e0bb129298,54fb538e313168c5] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:14:28.548+08:00, tenantSize=0 2024-11-04 13:14:28.563 INFO 21062 --- [ocp-schedules-23,e43d2e5a79a04b39,2e294c9dcc993642] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:14:28.563 INFO 21062 --- [ocp-schedules-23,e43d2e5a79a04b39,2e294c9dcc993642] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:14:28.581 INFO 21062 --- [ocp-async-3,80a4f3b1a027da0b,75fe5e950218c0c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:14:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:14:28.588640+08:00, lastProcessTime=2024-11-04T13:14:18.588823+08:00 2024-11-04 13:14:28.592 INFO 21062 --- [ocp-schedules-17,c7dce3188e9881f5,11d34409a99ee5ea] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:14:28.593 INFO 21062 --- [ocp-schedules-17,c7dce3188e9881f5,11d34409a99ee5ea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:14:28.601 INFO 21062 --- [ocp-schedules-3,8b2ec0ffa51d02b1,9818e597319900c6] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 13:14:28.602 INFO 21062 --- [ocp-schedules-3,8b2ec0ffa51d02b1,9818e597319900c6] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 13:14:28.661 INFO 21062 --- [ocp-schedules-30,071628dba1cbe496,23fb0861704771d4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:14:28.716 INFO 21062 --- [ocp-schedules-21,b2845dbf50a4c8a4,d93e639b9bdb3eef] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:14:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:14:28.588640+08:00 2024-11-04 13:14:28.787 INFO 21062 --- [ocp-schedules-2,0527f1f6d7568316,870803111f8a712b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:14:28.787 INFO 21062 --- [ocp-schedules-2,0527f1f6d7568316,870803111f8a712b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:14:28.792 WARN 21062 --- [ocp-async-9,28de20e18edf392f,ce26bd420dcbbad9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:28.792 INFO 21062 --- [ocp-schedules-22,4d1e8f290b9c8bad,8ccbbf3e96cd14f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:14:28.817 INFO 21062 --- [ocp-schedules-6,e3750d8f9574b4e3,f2489fa438962f93] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:14:28.823 INFO 21062 --- [ocp-schedules-6,e3750d8f9574b4e3,f2489fa438962f93] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:14:28.823 INFO 21062 --- [ocp-schedules-6,e3750d8f9574b4e3,f2489fa438962f93] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:14:28.838 INFO 21062 --- [ocp-async-10,784b67760cc29b11,ad2d97ac3da64f51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:28.970 INFO 21062 --- [ocp-schedules-19,ab94678ced2b50a5,9cc7847e921c5717] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:14:28.970 INFO 21062 --- [ocp-schedules-19,ab94678ced2b50a5,9cc7847e921c5717] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:14:28.971 INFO 21062 --- [ocp-schedules-19,ab94678ced2b50a5,9cc7847e921c5717] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:14:28.971 INFO 21062 --- [ocp-schedules-19,ab94678ced2b50a5,9cc7847e921c5717] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:14:28.971 INFO 21062 --- [ocp-schedules-19,ab94678ced2b50a5,9cc7847e921c5717] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:14:28.971 INFO 21062 --- [ocp-schedules-19,ab94678ced2b50a5,9cc7847e921c5717] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:14:29.228 INFO 21062 --- [ocp-schedules-9,4aa7429d5aa58c11,e87956d8fcf56c57] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:14:29.678 INFO 21062 --- [ocp-schedules-3,f20b7267ee6e1740,0ea80e75934d4692] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:14:29.678 INFO 21062 --- [ocp-schedules-3,f20b7267ee6e1740,0ea80e75934d4692] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:14:29.683 INFO 21062 --- [ocp-schedules-26,c3d1e277d931282d,0e145b2114813f5a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:14:29.689 INFO 21062 --- [ocp-schedules-26,c3d1e277d931282d,0e145b2114813f5a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:14:29.689 INFO 21062 --- [ocp-schedules-26,c3d1e277d931282d,0e145b2114813f5a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:14:29.704 INFO 21062 --- [ocp-schedules-3,f20b7267ee6e1740,0ea80e75934d4692] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:14:29.704 INFO 21062 --- [ocp-schedules-3,f20b7267ee6e1740,0ea80e75934d4692] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:14:29.705 INFO 21062 --- [ocp-schedules-24,a43dbf83b57af14c,b096e9b1eee19349] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:14:29.793 INFO 21062 --- [ocp-schedules-8,cedc803e6c71a143,7e7c5e5e3cb167b7] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:14:29.794 INFO 21062 --- [ocp-schedules-13,55f3d8233bec88e5,ca1fbb557db78edf] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:14:29.794 INFO 21062 --- [ocp-schedules-13,55f3d8233bec88e5,ca1fbb557db78edf] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:14:29.794 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:14:29.794 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:14:29.796 INFO 21062 --- [ocp-schedules-8,cedc803e6c71a143,7e7c5e5e3cb167b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:29.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:14:29.804 INFO 21062 --- [ocp-schedules-13,55f3d8233bec88e5,ca1fbb557db78edf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:14:29.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:14:29.806 WARN 21062 --- [ocp-schedules-13,55f3d8233bec88e5,ca1fbb557db78edf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:14:29.806 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:14:29.806 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:14:29.807 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:14:29.807 INFO 21062 --- [ocp-schedules-13,55f3d8233bec88e5,ca1fbb557db78edf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:14:29.807 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:14:29.808 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:14:29.808 WARN 21062 --- [ocp-schedules-13,55f3d8233bec88e5,ca1fbb557db78edf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:14:29.808 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:14:29.817 ERROR 21062 --- [ocp-schedules-13,55f3d8233bec88e5,ca1fbb557db78edf] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:14:29.822 INFO 21062 --- [ocp-schedules-2,5e9399e92e470412,1c35bb513ec8bfbc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:14:29.822+08:00, clusters=[1] 2024-11-04 13:14:29.823 INFO 21062 --- [ocp-schedules-2,5e9399e92e470412,1c35bb513ec8bfbc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:14:29.853 ERROR 21062 --- [ocp-schedules-8,cedc803e6c71a143,7e7c5e5e3cb167b7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:14:29.853 INFO 21062 --- [ocp-schedules-8,cedc803e6c71a143,7e7c5e5e3cb167b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:14:29.929 INFO 21062 --- [ocp-schedules-14,f79a2c18b14587bc,78af42167878ccd4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:14:29.929+08:00, clusters=[1] 2024-11-04 13:14:29.933 INFO 21062 --- [ocp-schedules-14,f79a2c18b14587bc,78af42167878ccd4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:14:29.983 INFO 21062 --- [prometheus-minute-schedule-1,15f811ebfee527d5,5c5be8fae0f8be72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:14:30.293 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:30.294 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:14:30.792 WARN 21062 --- [ocp-async-10,4b5d807dfe5d5e87,bdb9e57c78fe2753] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:30.838 INFO 21062 --- [ocp-async-5,cea3f4d1f145d08b,67b088d4ac977617] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:31.378 INFO 21062 --- [ocp-schedules-10,c7c013d53ea90f63,7032095ce50a0a24] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:14:31.378 INFO 21062 --- [ocp-schedules-10,e013efe458aa35a0,f2d410d741474a0f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:14:31.382 INFO 21062 --- [ocp-schedules-10,e013efe458aa35a0,f2d410d741474a0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:31.411 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:14:31.412 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:14:32.351 INFO 21062 --- [ocp-schedules-8,bb0ceeb842ebe170,b9f6798047455fa6] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:14:32.355 INFO 21062 --- [ocp-schedules-8,bb0ceeb842ebe170,b9f6798047455fa6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:32.440 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:14:32.443 ERROR 21062 --- [ocp-schedules-8,bb0ceeb842ebe170,b9f6798047455fa6] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:14:32.443 INFO 21062 --- [ocp-schedules-8,bb0ceeb842ebe170,b9f6798047455fa6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:14:32.792 WARN 21062 --- [ocp-async-5,8a3b127f20ed0575,9b5b52d01944b9b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:32.838 INFO 21062 --- [ocp-async-6,e19aea58d83eb042,c066042f877c9cf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d1e4b13b8112514b,23c6f237989feded] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:33.294 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:33.295 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:34.792 WARN 21062 --- [ocp-async-7,63d989eea9bd3de8,38a0f823d79afc3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:34.838 INFO 21062 --- [ocp-async-1,31b9d8f6e7999516,511984646f720411] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:35.983 INFO 21062 --- [prometheus-minute-schedule-1,9b4092759f257225,d98d981babf8809a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:36.295 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:36.296 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:36.793 WARN 21062 --- [ocp-async-2,56c729bb9c9c88e5,bb1159619f07865a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:36.839 INFO 21062 --- [ocp-async-8,11c8f80b5c927bf7,e9909e27a0605aa9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:38.293 INFO 21062 --- [ocp-schedules-19,165289522ce4aa82,bddf3fe1f1515deb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:14:38.504 INFO 21062 --- [ocp-schedules-25,e53807cb1186d5eb,518cd0840c17f1d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:14:38.504 INFO 21062 --- [ocp-schedules-27,438e67a12d7af501,4ff414c5a311369a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:14:38.582 INFO 21062 --- [ocp-async-8,bf01b648bc8bd717,d8be1dc392462078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:14:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:14:38.589466+08:00, lastProcessTime=2024-11-04T13:14:28.588640+08:00 2024-11-04 13:14:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:14:38.589466+08:00 2024-11-04 13:14:38.793 WARN 21062 --- [ocp-async-3,ce1209cf819654b1,e571aabff4fa0a64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:38.839 INFO 21062 --- [ocp-async-7,c79a0041db49f31c,8188d020e91c24fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:38.983 INFO 21062 --- [prometheus-minute-schedule-1,d11986677bb6bb39,ad0e2634ea0b9dc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:39.231 INFO 21062 --- [ocp-schedules-22,3a651956a13afc0a,1c224ae69afe4336] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:14:39.296 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:39.297 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:39.735 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:14:39.736 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:14:39.737 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.738 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:14:39.739 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.739 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.740 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.740 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:14:39.756 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.757 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.757 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.758 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:14:39.770 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.771 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.771 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.771 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:14:39.785 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.786 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.786 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.787 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:14:39.799 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.799 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.800 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.800 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:14:39.806 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.806 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.807 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.807 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:14:39.812 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.813 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.813 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.814 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:14:39.819 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.820 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.820 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.820 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:14:39.827 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.827 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.828 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.828 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:14:39.833 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.834 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.834 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.834 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:14:39.840 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.840 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.841 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.841 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:14:39.850 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.850 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.851 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.851 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:14:39.864 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.864 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.865 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.866 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:14:39.871 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.871 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.872 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.872 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:14:39.879 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.879 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.880 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.880 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:14:39.887 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.887 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.888 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.888 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:14:39.892 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.893 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.893 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.894 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:14:39.898 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.898 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.899 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.899 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:14:39.903 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.904 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.904 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.904 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:14:39.909 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.910 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.910 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.910 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:14:39.915 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.915 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.915 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.916 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:14:39.920 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.921 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.922 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.922 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:14:39.926 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.927 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.927 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.928 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:14:39.932 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.932 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.933 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.933 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:14:39.938 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.938 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.939 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.939 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:14:39.943 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.943 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.944 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.944 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:14:39.949 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.949 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.950 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.950 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:14:39.955 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.955 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.956 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.956 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:14:39.960 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.960 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.961 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.961 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:14:39.965 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.966 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.966 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.967 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:14:39.971 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.971 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.972 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.972 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:14:39.976 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.977 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.977 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.978 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:14:39.982 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.983 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.983 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.984 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:14:39.988 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.988 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.989 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.989 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:14:39.997 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.997 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:39.998 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:39.998 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:14:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:14:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:14:40.003 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.004 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.004 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.005 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:14:40.010 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.011 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.011 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.011 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:14:40.017 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.018 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.018 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.019 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:14:40.022 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.022 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.023 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.023 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:14:40.028 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.028 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.029 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.029 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:14:40.033 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.033 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.034 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.035 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:14:40.038 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.039 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.039 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.040 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:14:40.049 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.049 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.049 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.050 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:14:40.071 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.071 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.072 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.072 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:14:40.077 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.078 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.078 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.079 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:14:40.084 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.084 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.085 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.085 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:14:40.092 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.092 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.093 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.093 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:14:40.097 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.098 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.098 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.099 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:14:40.103 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.103 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.104 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.104 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:14:40.108 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:14:40.108 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:14:40.110 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.111 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:14:40.112 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.112 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.113 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.113 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:14:40.118 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.119 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:14:40.120 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:14:40.120 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:14:40.124 INFO 21062 --- [ocp-schedules-29,18ac9dc080eb6ca6,dd163f9fe8f35f05] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:14:40.793 WARN 21062 --- [ocp-async-9,e99e4dd6a8d3fb2d,150a6ff7226ff601] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:40.839 INFO 21062 --- [ocp-async-2,261fdbe77451861b,3af799ccbc4762eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:14:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:14:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:14:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bc7037036f050788,432d179ada8a2805] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:42.297 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:42.298 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:42.793 WARN 21062 --- [ocp-async-10,bc3eabc565de6ad4,5454b84adb94d51b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:42.839 INFO 21062 --- [ocp-async-4,ff564bb1615710b2,178ee341e99cceb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:44.793 WARN 21062 --- [ocp-async-5,f99de5884e7dedce,1226e84b9301d86f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:44.839 INFO 21062 --- [ocp-async-6,f43c22614c6c53e6,f2407c081724b2ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:44.983 INFO 21062 --- [prometheus-minute-schedule-1,46266bbcd1be4eaa,3a6701cf9c5105af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:45.298 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:45.299 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:46.793 WARN 21062 --- [ocp-async-7,314b7182a51b1c73,3be5f72bee89c77b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:46.840 INFO 21062 --- [ocp-async-1,318911a37495bae6,d70e242e431d9b21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:47.983 INFO 21062 --- [prometheus-minute-schedule-1,39a256bb8f379873,7ea655553162291f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:48.299 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:48.301 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:48.336 INFO 21062 --- [ocp-schedules-4,2ba07ac8b59b508f,2644209393fb3c96] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:14:48.582 INFO 21062 --- [ocp-async-9,917ab0041370636e,fa12c66345dd7007] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:14:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:14:48.589683+08:00, lastProcessTime=2024-11-04T13:14:38.589466+08:00 2024-11-04 13:14:48.778 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:14:48.589683+08:00 2024-11-04 13:14:48.794 WARN 21062 --- [ocp-async-10,449c7382cb222596,f271dd676b9ae601] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:48.840 INFO 21062 --- [ocp-async-4,1407bcbb50b275d7,e16a56e527f63d81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:49.234 INFO 21062 --- [ocp-schedules-12,aea1fff859f4f85d,5bea65f5ff5b8dd5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:14:49.692 INFO 21062 --- [ocp-schedules-23,57687f16a5068fe1,5245ac443975f3df] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:14:49.697 INFO 21062 --- [ocp-schedules-23,57687f16a5068fe1,5245ac443975f3df] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:14:49.697 INFO 21062 --- [ocp-schedules-23,57687f16a5068fe1,5245ac443975f3df] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:14:50.794 WARN 21062 --- [ocp-async-5,afa08dd16eae57fe,ad8eacbacc31c339] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:50.840 INFO 21062 --- [ocp-async-6,825050f48c8a16ea,7b9f32a98e97408c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:50.983 INFO 21062 --- [prometheus-minute-schedule-1,95c748d88ce9433e,e1031b00c4009afb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:51.301 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:51.302 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:52.794 WARN 21062 --- [ocp-async-7,ee59649a0242df55,0dbd9bd011e10db2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:52.840 INFO 21062 --- [ocp-async-1,3ec03dbdc9d37833,308822d43e817ede] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e3c81fe8bdde0ad1,3eaa6036bcb5cd63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:54.302 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:54.303 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:54.794 WARN 21062 --- [ocp-async-2,7765753784385759,102634d4577c18d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:54.840 INFO 21062 --- [ocp-async-8,ca19efd453ff300f,eec79fc9115f763f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:56.794 WARN 21062 --- [ocp-async-8,365b4a682bee223d,5a22b46c4b926539] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:56.811 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:14:56.811 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:14:56.819 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:14:56.819 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:14:56.819 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:14:56.819 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:14:56.824 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:14:56.824 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:14:56.824 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:14:56.825 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:14:56.840 INFO 21062 --- [ocp-async-3,b78866dad8b7aeae,e436d463d12b1707] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:56.862 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:14:56.872 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:14:56.872 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:14:56.872 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:14:56.872 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:14:56.872 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:14:56.881 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:14:56.983 INFO 21062 --- [prometheus-minute-schedule-1,cc0f83e1af93a384,cd946ff82b30cd69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:14:57.303 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:14:57.304 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:14:57.409 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:14:58.040 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.214346697S 2024-11-04 13:14:58.041 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:14:58.138 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:14:58.138 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:14:58.165 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:14:58.223 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:14:58.223 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:14:58.223 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:14:58.223 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:14:58.223 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:14:58.223 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:14:58.234 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:14:58.360 INFO 21062 --- [ocp-schedules-19,92e68cf439702fa3,623c837b4b3e5423] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:14:58.421 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] o.s.boot.SpringApplication : Started application in 1.602 seconds (JVM running for 2712.101) 2024-11-04 13:14:58.422 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.603048805S 2024-11-04 13:14:58.500 INFO 21062 --- [ocp-schedules-13,da45c67c648b18bd,9e9b0ae6d0d5adb7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:14:58.500 INFO 21062 --- [ocp-schedules-30,00b9283f64de14f2,b08ae529695682ea] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:14:58.502 INFO 21062 --- [ocp-schedules-24,cdfa1d4dbaef5c09,2e905f6e7049dda2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:14:58.503 INFO 21062 --- [ocp-schedules-13,da45c67c648b18bd,9e9b0ae6d0d5adb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:58.503 INFO 21062 --- [ocp-schedules-30,00b9283f64de14f2,b08ae529695682ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:14:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:14:58.504 INFO 21062 --- [ocp-schedules-15,30f4dcacede28cb2,57f138ea4a7b5b29] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:14:58.511 INFO 21062 --- [ocp-schedules-27,c45abf26ae4b2496,76ffce87895e57b8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:14:58.511 INFO 21062 --- [ocp-schedules-27,c45abf26ae4b2496,76ffce87895e57b8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:14:58.511 INFO 21062 --- [ocp-schedules-9,4458cefd78e51804,1cdc7f2ce7c7a84e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:14:58.511 INFO 21062 --- [ocp-schedules-9,4458cefd78e51804,1cdc7f2ce7c7a84e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:14:58.512 INFO 21062 --- [ocp-schedules-4,a77cec50c41ae2d1,ad95aa7e5c710655] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:14:58.512 INFO 21062 --- [ocp-schedules-4,a77cec50c41ae2d1,ad95aa7e5c710655] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:14:58.582 INFO 21062 --- [ocp-async-4,3e480c278e0bd483,2d845a78c8515a92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:14:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:14:58.589904+08:00, lastProcessTime=2024-11-04T13:14:48.589683+08:00 2024-11-04 13:14:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:14:58.589904+08:00 2024-11-04 13:14:58.795 WARN 21062 --- [ocp-async-6,aa6a82a18dbd7d2b,2fc88bd777314507] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:14:58.795 INFO 21062 --- [ocp-schedules-29,9207dd56d355d2a4,555bf9a042361d74] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:14:58.841 INFO 21062 --- [ocp-async-1,4ae0c5553b080709,cb7fe9c4b6024f3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:14:58.932 INFO 21062 --- [ocp-schedules-16,25605a5bb57dc254,52e21941935da216] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:14:59.236 INFO 21062 --- [ocp-schedules-15,220be58fda48dffc,f89c9a3889218d43] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:14:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d89d78f3dcabb47b,5516e15989cd962f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697300, interval=60 2024-11-04 13:15:00.305 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:00.306 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:00.535 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4125c9d3727d4f82] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:10:01.0, nextRun=2024-11-04 13:15:00.0, currentDate=Mon Nov 04 13:15:00 CST 2024 2024-11-04 13:15:00.535 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4125c9d3727d4f82] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:10:01.0, nextRunTime=2024-11-04 13:15:00.0} 2024-11-04 13:15:00.562 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4125c9d3727d4f82] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:15:00.595 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4125c9d3727d4f82] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:15:00.595 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,4125c9d3727d4f82] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:15:00.795 WARN 21062 --- [ocp-async-1,e1a17ccb61bcc605,2edcdd8c746c4cad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:00.841 INFO 21062 --- [ocp-async-8,cc1d79c57bd21168,c9760f4039bb8aaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:01.412 INFO 21062 --- [ocp-schedules-10,e013efe458aa35a0,f2d410d741474a0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:15:01.412 INFO 21062 --- [ocp-schedules-10,232c375279080dc9,6758d5736aed3158] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:15:01.416 INFO 21062 --- [ocp-schedules-10,232c375279080dc9,6758d5736aed3158] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:01.446 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:15:01.446 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:15:02.795 WARN 21062 --- [ocp-async-8,67426016590abb51,8f863783b9931120] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:02.841 INFO 21062 --- [ocp-async-3,534e7ef5ca281385,fd2900fa2cdb9f39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:02.983 INFO 21062 --- [prometheus-minute-schedule-1,07a1c2af48b777e5,5c5d19c6782f69c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:03.306 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:03.307 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:04.795 WARN 21062 --- [ocp-async-3,7aaa5d20bbf9c8fc,0013af72e98a8989] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:04.841 INFO 21062 --- [ocp-async-7,158eefb99ee7f808,8ffd3e5b6240bb6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8bae1bcc31104e0b,b40a26803a695e0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:06.307 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:06.308 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:06.795 WARN 21062 --- [ocp-async-7,e9cf3d291a1bd2ed,fb188f31c39a41a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:06.841 INFO 21062 --- [ocp-async-2,a9e2155dffd51229,761ce08eea2505f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:08.402 INFO 21062 --- [ocp-schedules-24,3c0b8bec289c5e20,1becb7c1ae3822ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:15:08.504 INFO 21062 --- [ocp-schedules-30,00b9283f64de14f2,b08ae529695682ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:15:08.504 INFO 21062 --- [ocp-schedules-13,da45c67c648b18bd,9e9b0ae6d0d5adb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:15:08.583 INFO 21062 --- [ocp-async-1,75139fce6d691d8b,1a46d877e273d1ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:15:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:15:08.590132+08:00, lastProcessTime=2024-11-04T13:14:58.589904+08:00 2024-11-04 13:15:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:15:08.590132+08:00 2024-11-04 13:15:08.795 WARN 21062 --- [ocp-async-8,7c296c56e5808484,a19d470d518d52b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:08.841 INFO 21062 --- [ocp-async-3,4c6e1cefa5028cdf,4d405cac1806622b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d045802f6ac37d38,9700b1a68556fbb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:09.239 INFO 21062 --- [ocp-schedules-17,9ddf3c4edbe62918,4eac8f3c3aa7919f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:15:09.308 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:09.309 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:09.700 INFO 21062 --- [ocp-schedules-26,0e7486a32613b479,7ac99dc5307360b3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:15:09.706 INFO 21062 --- [ocp-schedules-26,0e7486a32613b479,7ac99dc5307360b3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:15:09.706 INFO 21062 --- [ocp-schedules-26,0e7486a32613b479,7ac99dc5307360b3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:15:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:15:10.003 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:15:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:15:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:15:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:15:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:15:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:15:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:15:10.796 WARN 21062 --- [ocp-async-3,bc2f7dd6671d89ef,405c5fa850c5893b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:10.841 INFO 21062 --- [ocp-async-7,ed17886a4ae33c5a,c2c953fa4872b56c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:11.200 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=1, affectRows=1 2024-11-04 13:15:11.200 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=1, validCount=1 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=12, affectRows=12 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=8, affectRows=8 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=8, validCount=8 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=12, validCount=12 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=27, affectRows=27 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=24, affectRows=24 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=33, affectRows=33 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=24, validCount=24 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=27, validCount=27 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=32, validCount=32 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=41, affectRows=41 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=54, affectRows=54 2024-11-04 13:15:11.202 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=41, validCount=41 2024-11-04 13:15:11.203 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=50, affectRows=50 2024-11-04 13:15:11.203 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=54, validCount=54 2024-11-04 13:15:11.203 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=48, validCount=48 2024-11-04 13:15:11.203 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697240, nodeCount=72, affectRows=72 2024-11-04 13:15:11.203 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697240, nodeCount=72, validCount=72 2024-11-04 13:15:11.983 INFO 21062 --- [prometheus-minute-schedule-1,1e92beee6f00ff81,33b98f324f8c0f8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:12.309 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:12.310 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:12.796 WARN 21062 --- [ocp-async-7,5ac3de151b9c3f65,31f95069a0cda366] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:12.842 INFO 21062 --- [ocp-async-2,779262fae66f464e,18d262a8a49a20a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:13.984 INFO 21062 --- [prometheus-cache-monitor1,ca008265811bcab3,1db5a948891dbff7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:15:13.984 INFO 21062 --- [prometheus-cache-monitor1,ca008265811bcab3,1db5a948891dbff7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:15:14.424 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:11:14.407, end=2024-11-04T13:13:14.422} 2024-11-04 13:15:14.436 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:15:14.436 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:15:14.796 WARN 21062 --- [ocp-async-2,b5009764ad22d120,89c58c9d63685c57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:14.842 INFO 21062 --- [ocp-async-4,3713ce47f55ce18c,ce0c184a26d31649] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:14.983 INFO 21062 --- [prometheus-minute-schedule-1,204027d717d38c13,17ea831c5f46f74a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:15.310 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:15.310 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:16.796 WARN 21062 --- [ocp-async-4,f433b08572a7e5e7,4495a9328e029ed8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:16.842 INFO 21062 --- [ocp-async-9,0ce90d4bcfe3ef84,7d9f407d4a355210] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d68e52712e2c2526,ce86b37ec989448a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:18.311 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:18.312 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:18.426 INFO 21062 --- [ocp-schedules-5,99c3844d20ef28a6,ea165b2b3a3824a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:15:18.583 INFO 21062 --- [ocp-async-3,cacf9b1c642e6ca1,1ca7ade5ebbe542c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:15:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:15:18.589299+08:00, lastProcessTime=2024-11-04T13:15:08.590132+08:00 2024-11-04 13:15:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:15:18.589299+08:00 2024-11-04 13:15:18.796 WARN 21062 --- [ocp-async-7,7ebb97c22d3de786,45176cf8e763b2c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:18.842 INFO 21062 --- [ocp-async-2,efe7c90e38896e4b,e7d7ef85801d9ae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:19.242 INFO 21062 --- [ocp-schedules-19,817cd71238c5627e,0855dbeb79872c65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:15:20.797 WARN 21062 --- [ocp-async-2,c21ac50ce06acc65,087cd952d21dd993] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:20.842 INFO 21062 --- [ocp-async-4,bf784ee0a5511cc4,382de79a75214809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ede736c1da5ede95,c3ee7c9d550db399] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:21.312 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:21.313 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:22.797 WARN 21062 --- [ocp-async-4,9039b41bc29696ef,beb25a75b9a516b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:22.842 INFO 21062 --- [ocp-async-6,266a3ca9d46b4e61,f83e9d3ec6a60127] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:23.983 INFO 21062 --- [prometheus-minute-schedule-1,d467c75cf1a6244d,e413a606e17e865b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:24.313 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:24.314 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:24.797 WARN 21062 --- [ocp-async-6,506c7dfa18721618,b9ce932f39c4db5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:24.843 INFO 21062 --- [ocp-async-1,1b51c910bd2be4a4,d4d3a53186950c93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:26.797 WARN 21062 --- [ocp-async-1,78e85b28fedf4392,9aa1d511f5d8ec5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:26.843 INFO 21062 --- [ocp-async-8,28a789ff10ef136f,67bb7515a1e13f2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2f1b767fac295126,408bd4dcb51b4db4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:26.984 INFO 21062 --- [prometheus-hour-schedule-1,d5083606e585e776,4ad70d61bc84f920] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:15:27.314 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:27.315 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:28.466 INFO 21062 --- [ocp-schedules-28,ebe9d0980001bd4b,61603d00ab2073f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:15:28.513 INFO 21062 --- [ocp-schedules-15,e9487f01c0660f26,a11953a6ed84539c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:15:28.514 INFO 21062 --- [ocp-schedules-19,f529c28545cbe9f3,14f8b69e4327ed0a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:15:28.514 INFO 21062 --- [ocp-schedules-7,36e743065388ee04,d2be4ee39cf8bd81] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:15:28.514 INFO 21062 --- [ocp-schedules-9,201e28e06773d188,c185df620268ef93] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:15:28.518 INFO 21062 --- [ocp-schedules-20,330e54e674b1a3fc,cb3a1bf56bd3f204] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:15:28.521 INFO 21062 --- [ocp-schedules-8,51f5109fbcafd9ff,d1fb3a4bf935dfab] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:15:28.521 INFO 21062 --- [ocp-schedules-9,201e28e06773d188,c185df620268ef93] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:28.521 INFO 21062 --- [ocp-schedules-7,36e743065388ee04,d2be4ee39cf8bd81] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:28.521 INFO 21062 --- [ocp-schedules-15,e9487f01c0660f26,a11953a6ed84539c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:15:28.521 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:15:28.528 INFO 21062 --- [ocp-schedules-20,330e54e674b1a3fc,cb3a1bf56bd3f204] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:15:28.529 INFO 21062 --- [ocp-schedules-23,27ecec559a76644f,cf8c18048fe54e43] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:15:28.529 INFO 21062 --- [ocp-schedules-23,27ecec559a76644f,cf8c18048fe54e43] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:15:28.529 INFO 21062 --- [ocp-schedules-29,bac0e30cac0dca62,47014db4bfdaa81a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:15:28.529 INFO 21062 --- [ocp-schedules-29,bac0e30cac0dca62,47014db4bfdaa81a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:15:28.529 INFO 21062 --- [ocp-schedules-22,5af66bd34b9346cb,43e57f6fcb4b37cf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:15:28.529 INFO 21062 --- [ocp-schedules-22,5af66bd34b9346cb,43e57f6fcb4b37cf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:15:28.583 INFO 21062 --- [ocp-async-8,bd91844fe3c3b602,0154dad845fbf033] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:15:28.584 INFO 21062 --- [ocp-schedules-21,09d6703bd8a135f9,90f79744d5c0bdcf] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:15:28.585 INFO 21062 --- [ocp-schedules-21,09d6703bd8a135f9,90f79744d5c0bdcf] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:15:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:15:28.588459+08:00, lastProcessTime=2024-11-04T13:15:18.589299+08:00 2024-11-04 13:15:28.663 INFO 21062 --- [ocp-schedules-12,758095076e241c8a,3ad9a396d93a0241] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:15:28.716 INFO 21062 --- [ocp-schedules-18,6f7404b6eed074f6,f65f2d7249acd290] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:15:28.720 INFO 21062 --- [ocp-schedules-18,6f7404b6eed074f6,f65f2d7249acd290] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:28.720 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:15:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:15:28.588459+08:00 2024-11-04 13:15:28.792 INFO 21062 --- [ocp-schedules-2,2fcb2ab47fba3ac4,be452ba7dc408a06] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:15:28.792 INFO 21062 --- [ocp-schedules-2,2fcb2ab47fba3ac4,be452ba7dc408a06] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:15:28.797 INFO 21062 --- [ocp-schedules-19,f1517aed89c07290,64416c6990349c4c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:15:28.797 WARN 21062 --- [ocp-async-9,76c484d30c60476b,6abfeb28a36437a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:28.823 INFO 21062 --- [ocp-schedules-23,9af8883a6ec2bb06,ce2f8c1eba904305] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:15:28.828 INFO 21062 --- [ocp-schedules-23,9af8883a6ec2bb06,ce2f8c1eba904305] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:15:28.828 INFO 21062 --- [ocp-schedules-23,9af8883a6ec2bb06,ce2f8c1eba904305] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:15:28.843 INFO 21062 --- [ocp-async-7,364602305438c11d,23b8d2455e5db4e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:28.972 INFO 21062 --- [ocp-schedules-25,7a3624ce8ed055be,5fb55351a08faaed] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:15:28.972 INFO 21062 --- [ocp-schedules-25,7a3624ce8ed055be,5fb55351a08faaed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:15:28.978 INFO 21062 --- [ocp-schedules-25,7a3624ce8ed055be,5fb55351a08faaed] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:15:28.978 INFO 21062 --- [ocp-schedules-25,7a3624ce8ed055be,5fb55351a08faaed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:15:28.982 INFO 21062 --- [ocp-schedules-25,7a3624ce8ed055be,5fb55351a08faaed] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:15:28.982 INFO 21062 --- [ocp-schedules-25,7a3624ce8ed055be,5fb55351a08faaed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:15:29.245 INFO 21062 --- [ocp-schedules-21,ae7e96cb6a59ec4a,c72b3553ebc8b6a5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:15:29.704 INFO 21062 --- [ocp-schedules-16,9b51cfec8869553d,62a5d784c3349912] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:15:29.704 INFO 21062 --- [ocp-schedules-16,9b51cfec8869553d,62a5d784c3349912] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:15:29.708 INFO 21062 --- [ocp-schedules-2,8728428ba3b0ee4e,3b939f2304f22a7d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:15:29.714 INFO 21062 --- [ocp-schedules-2,8728428ba3b0ee4e,3b939f2304f22a7d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:15:29.714 INFO 21062 --- [ocp-schedules-2,8728428ba3b0ee4e,3b939f2304f22a7d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:15:29.731 INFO 21062 --- [ocp-schedules-16,9b51cfec8869553d,62a5d784c3349912] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:15:29.731 INFO 21062 --- [ocp-schedules-16,9b51cfec8869553d,62a5d784c3349912] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:15:29.734 INFO 21062 --- [ocp-schedules-11,34307a3f0c1b7784,17bbc24571c181c5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:15:29.826 INFO 21062 --- [ocp-schedules-19,e086ba8e8c69f1d3,818a0cbd2a8aaf57] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:15:29.826 INFO 21062 --- [ocp-schedules-19,e086ba8e8c69f1d3,818a0cbd2a8aaf57] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:15:29.826 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:15:29.826 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:15:29.836 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:15:29.837 INFO 21062 --- [ocp-schedules-19,e086ba8e8c69f1d3,818a0cbd2a8aaf57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:15:29.837 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:15:29.838 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:15:29.838 WARN 21062 --- [ocp-schedules-19,e086ba8e8c69f1d3,818a0cbd2a8aaf57] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:15:29.839 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:15:29.839 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:15:29.839 INFO 21062 --- [ocp-schedules-19,e086ba8e8c69f1d3,818a0cbd2a8aaf57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:15:29.840 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:15:29.840 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:15:29.840 WARN 21062 --- [ocp-schedules-19,e086ba8e8c69f1d3,818a0cbd2a8aaf57] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:15:29.841 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:15:29.849 ERROR 21062 --- [ocp-schedules-19,e086ba8e8c69f1d3,818a0cbd2a8aaf57] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:15:29.853 INFO 21062 --- [ocp-schedules-20,966a12aa7ad2a30d,9e0c617221f100fe] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:15:29.853+08:00, clusters=[1] 2024-11-04 13:15:29.855 INFO 21062 --- [ocp-schedules-20,966a12aa7ad2a30d,9e0c617221f100fe] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:15:29.962 INFO 21062 --- [ocp-schedules-29,00748d18ce80af63,cfc15c4201f8e193] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:15:29.962+08:00, clusters=[1] 2024-11-04 13:15:29.966 INFO 21062 --- [ocp-schedules-29,00748d18ce80af63,cfc15c4201f8e193] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:15:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e161c827a0e488b4,a0a298a7c7f703e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:15:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:15:30.315 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:30.315 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:30.798 WARN 21062 --- [ocp-async-7,0c775ca5d662ebaa,0f5ae3d814614fe4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:30.843 INFO 21062 --- [ocp-async-1,19a7390b3b3dcb48,701a7026c13a0a41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:31.446 INFO 21062 --- [ocp-schedules-10,232c375279080dc9,6758d5736aed3158] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:15:31.446 INFO 21062 --- [ocp-schedules-10,459bc6df05e452fe,d4705a4e762bddfe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:15:31.450 INFO 21062 --- [ocp-schedules-10,459bc6df05e452fe,d4705a4e762bddfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:31.478 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:15:31.478 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:15:32.443 INFO 21062 --- [ocp-schedules-17,f5b80f9be06a486e,f2dd663386e2112a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:15:32.447 INFO 21062 --- [ocp-schedules-17,f5b80f9be06a486e,f2dd663386e2112a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:32.533 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:15:32.535 ERROR 21062 --- [ocp-schedules-17,f5b80f9be06a486e,f2dd663386e2112a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:15:32.535 INFO 21062 --- [ocp-schedules-17,f5b80f9be06a486e,f2dd663386e2112a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:15:32.798 WARN 21062 --- [ocp-async-5,814ed77239162016,3719040546347e47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:32.843 INFO 21062 --- [ocp-async-4,f6956929d7a0ed65,54db9a90266bf7dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:32.983 INFO 21062 --- [prometheus-minute-schedule-1,61ebc393b36fcc36,33f37a1168e257f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:33.316 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:33.316 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:34.798 WARN 21062 --- [ocp-async-3,6ac738df923883b1,25fca7bae59699dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:34.844 INFO 21062 --- [ocp-async-6,9716a521f6966e0d,bfefa3f3dbc7ef70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:35.983 INFO 21062 --- [prometheus-minute-schedule-1,a4d2c5d700d16eb0,780e39585d13a110] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:36.317 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:36.317 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:36.798 WARN 21062 --- [ocp-async-10,40ea118728e7744a,5a3f4953581521f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:36.844 INFO 21062 --- [ocp-async-2,ba33d70f2948b122,8c0c0db5d0cce845] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:38.521 INFO 21062 --- [ocp-schedules-9,201e28e06773d188,c185df620268ef93] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:15:38.521 INFO 21062 --- [ocp-schedules-7,36e743065388ee04,d2be4ee39cf8bd81] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:15:38.535 INFO 21062 --- [ocp-schedules-22,00b912369800a64d,a3b45f6192aa4546] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:15:38.583 INFO 21062 --- [ocp-async-2,ce2f10429cf4a384,090f27d0c82ff4d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:15:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:15:38.590719+08:00, lastProcessTime=2024-11-04T13:15:28.588459+08:00 2024-11-04 13:15:38.720 INFO 21062 --- [ocp-schedules-18,6f7404b6eed074f6,f65f2d7249acd290] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:15:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:15:38.590719+08:00 2024-11-04 13:15:38.799 WARN 21062 --- [ocp-async-8,1a2eabb2fd2d3f39,a9824776ab01488c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:38.844 INFO 21062 --- [ocp-async-3,d63a9f8c2924908e,a3a5c1349d5f948b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c07cbc009756e047,063343b200b8a4bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:39.248 INFO 21062 --- [ocp-schedules-6,8ef0edfb331a4c76,50fe4d8c28e27ae1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:15:39.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:39.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:15:40.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:15:40.799 WARN 21062 --- [ocp-async-9,183194f6be61e77a,d16ce56e0c5e66a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:40.844 INFO 21062 --- [ocp-async-10,7ae52b835565a959,2b80d4eccdd78992] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c179491631dc9a72,af6de744cac9be33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:42.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:42.319 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:42.799 WARN 21062 --- [ocp-async-7,b72f7dfac896f9a5,1f23b8e3eaad0073] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:42.844 INFO 21062 --- [ocp-async-1,de515da51fbb7ba4,9b531f882e19c79b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:44.799 WARN 21062 --- [ocp-async-5,f6ff35dd6e775cce,66c903059d9af97f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:44.845 INFO 21062 --- [ocp-async-4,90b2a2369e69b75c,de9c23b0d6a684c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:44.983 INFO 21062 --- [prometheus-minute-schedule-1,0c53aed60d376ec9,f0ff4f13eadcffdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:45.319 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:45.320 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:46.799 WARN 21062 --- [ocp-async-3,f53aba57a1af3de3,16bd37dbdfcfe646] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:46.845 INFO 21062 --- [ocp-async-6,9c3706d68ea03b73,3723bc6e6830da9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fb9f8ff76c0e4f67,cc0c24a67a3a83ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:48.320 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:48.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:48.581 INFO 21062 --- [ocp-schedules-17,3fc12a9484d879af,e8bf2039babefc47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:15:48.584 INFO 21062 --- [ocp-async-3,6d27a79e3204695b,9df459f653ef6643] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:15:48.594 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:15:48.593266+08:00, lastProcessTime=2024-11-04T13:15:38.590719+08:00 2024-11-04 13:15:48.781 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:15:48.593266+08:00 2024-11-04 13:15:48.799 WARN 21062 --- [ocp-async-7,81301b26f2acd5de,74a4704e61d3cf5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:48.845 INFO 21062 --- [ocp-async-1,cf09dfdd55034591,871ab29fffd75f77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:49.251 INFO 21062 --- [ocp-schedules-30,39211eeaf03cb0c6,9341fe651e66dc30] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:15:49.716 INFO 21062 --- [ocp-schedules-21,9b2bb66ba8556c24,660997ed974e0111] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:15:49.722 INFO 21062 --- [ocp-schedules-21,9b2bb66ba8556c24,660997ed974e0111] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:15:49.722 INFO 21062 --- [ocp-schedules-21,9b2bb66ba8556c24,660997ed974e0111] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:15:50.800 WARN 21062 --- [ocp-async-5,850c96b53a3c5093,afd397d83bc5a07d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:50.845 INFO 21062 --- [ocp-async-4,357abc0c4b85d730,9ac498c0a7b6711c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b16972d675d56514,af39b9e5eef95c39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:51.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:51.322 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:52.800 WARN 21062 --- [ocp-async-9,3b38718c6bcbd374,501bc9429ca9ce01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:52.846 INFO 21062 --- [ocp-async-6,66ed8b7a80223652,f9e5c550794e3b3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e22fdd2075fc735d,311fff1c7f98048c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:54.322 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:54.323 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:54.800 WARN 21062 --- [ocp-async-6,fca44abb2394887a,2e471e37deae5ab2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:54.846 INFO 21062 --- [ocp-async-2,1821f875f792bbe7,e30211f57f33256f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:56.800 WARN 21062 --- [ocp-async-2,74f0121ecf759f54,a0c45fb2c2d3b09c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:56.846 INFO 21062 --- [ocp-async-8,c199e6d8b36829bb,70208464cf670188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:56.983 INFO 21062 --- [prometheus-minute-schedule-1,370cabb850cdd519,87560479e88b91ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:15:57.323 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:15:57.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:15:58.500 INFO 21062 --- [ocp-schedules-28,9b3e6b91d1635601,eab2e1f8d97debbc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:15:58.500 INFO 21062 --- [ocp-schedules-23,784dc2fa4e48ea7e,c71e0b6a2ae5ee26] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:15:58.502 INFO 21062 --- [ocp-schedules-14,a568bd310613bb00,49d629cb710dc4a0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:15:58.504 INFO 21062 --- [ocp-schedules-23,784dc2fa4e48ea7e,c71e0b6a2ae5ee26] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:58.504 INFO 21062 --- [ocp-schedules-28,9b3e6b91d1635601,eab2e1f8d97debbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:15:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:15:58.505 INFO 21062 --- [ocp-schedules-4,47b213936e7e8ebf,247612320da6c7aa] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:15:58.510 INFO 21062 --- [ocp-schedules-17,c6a8d1a393ab13bf,5864f19689ffa6da] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:15:58.510 INFO 21062 --- [ocp-schedules-11,d8a72c014a5df017,53245b2a67bb71dd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:15:58.510 INFO 21062 --- [ocp-schedules-24,49afe065adf87361,cc147b24107a6d01] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:15:58.510 INFO 21062 --- [ocp-schedules-11,d8a72c014a5df017,53245b2a67bb71dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:15:58.510 INFO 21062 --- [ocp-schedules-24,49afe065adf87361,cc147b24107a6d01] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:15:58.510 INFO 21062 --- [ocp-schedules-17,c6a8d1a393ab13bf,5864f19689ffa6da] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:15:58.584 INFO 21062 --- [ocp-async-5,5fd312fb79b8e9a9,7ca630ab08f0a209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:15:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:15:58.589956+08:00, lastProcessTime=2024-11-04T13:15:48.593266+08:00 2024-11-04 13:15:58.623 INFO 21062 --- [ocp-schedules-20,ea5d70cbd43e0037,c77392804f126a02] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:15:58.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:15:58.589956+08:00 2024-11-04 13:15:58.800 INFO 21062 --- [ocp-schedules-9,ed3b1857351ba650,2ef83c770c66b6c5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:15:58.800 WARN 21062 --- [ocp-async-4,c26dc61ecfec3e48,fd42234247b80c5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:15:58.846 INFO 21062 --- [ocp-async-6,2a1cb6db03ce2649,ea6c7224fde41295] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:15:59.254 INFO 21062 --- [ocp-schedules-4,9d6b772aed3387d2,573213079ecee90b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:15:59.983 INFO 21062 --- [prometheus-minute-schedule-1,5a2416c69d0b4484,166786a6419338fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697360, interval=60 2024-11-04 13:16:00.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:00.325 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:00.801 WARN 21062 --- [ocp-async-6,7faef4a1ddb6ec82,86bb7f79bba01fc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:00.846 INFO 21062 --- [ocp-async-2,a699a9262aec813b,16bb1309c614e9dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:01.479 INFO 21062 --- [ocp-schedules-10,459bc6df05e452fe,d4705a4e762bddfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:16:01.479 INFO 21062 --- [ocp-schedules-10,f0b4a17ff44552e5,78060ed0723a68f4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:16:01.483 INFO 21062 --- [ocp-schedules-10,f0b4a17ff44552e5,78060ed0723a68f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:01.513 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:16:01.513 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:16:02.801 WARN 21062 --- [ocp-async-2,5b72824df81d9441,2c3d9a8cb0e19605] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:02.846 INFO 21062 --- [ocp-async-8,46af0cacc7708acd,b4fc0e1eccc7f9b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:02.983 INFO 21062 --- [prometheus-minute-schedule-1,bef9e8cddd6ba516,56e49edb7bc6b30a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:03.325 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:03.326 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:04.801 WARN 21062 --- [ocp-async-8,530690af56ec5498,727159615bd07762] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:04.847 INFO 21062 --- [ocp-async-9,5c4eff93f5e72ccc,72ea87ae6530eb34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:05.983 INFO 21062 --- [prometheus-minute-schedule-1,880cf7eb91df4ef7,273bb2cad1abb9d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:06.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:06.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:06.801 WARN 21062 --- [ocp-async-9,f83cfff7a6e94a8f,cf380f50576178f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:06.847 INFO 21062 --- [ocp-async-7,7e8f6c5324eb5d43,730a7177a309cb35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:08.505 INFO 21062 --- [ocp-schedules-23,784dc2fa4e48ea7e,c71e0b6a2ae5ee26] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:16:08.505 INFO 21062 --- [ocp-schedules-28,9b3e6b91d1635601,eab2e1f8d97debbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:16:08.584 INFO 21062 --- [ocp-async-6,7300960fe4962d8f,069009ff2b6b100f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:16:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:16:08.590681+08:00, lastProcessTime=2024-11-04T13:15:58.589956+08:00 2024-11-04 13:16:08.667 INFO 21062 --- [ocp-schedules-5,81f9ecabd1281fe7,c1968f49e9df5ddf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:16:08.775 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:16:08.590681+08:00 2024-11-04 13:16:08.801 WARN 21062 --- [ocp-async-2,23902d719b411886,06cd89ef04153bd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:08.847 INFO 21062 --- [ocp-async-8,010f1301bc5175ec,5653695328cf2354] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:08.983 INFO 21062 --- [prometheus-minute-schedule-1,cbbb340b5d2ff14c,e2565a7b549acd03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:09.257 INFO 21062 --- [ocp-schedules-9,9bb6227768de99bb,a2a858c8ad41065d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:16:09.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:09.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:09.724 INFO 21062 --- [ocp-schedules-29,8d6505a3bd8c98fa,47ee4b1aee188fd7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:16:09.731 INFO 21062 --- [ocp-schedules-29,8d6505a3bd8c98fa,47ee4b1aee188fd7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:16:09.731 INFO 21062 --- [ocp-schedules-29,8d6505a3bd8c98fa,47ee4b1aee188fd7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:16:10.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:16:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:16:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:16:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:16:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:16:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:16:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:16:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:16:10.802 WARN 21062 --- [ocp-async-8,a396a3d26b940bd5,dd86bfc6a6113ba1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:10.847 INFO 21062 --- [ocp-async-9,c0799172cddc631e,b3b8908758560540] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:11.207 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=1, affectRows=1 2024-11-04 13:16:11.207 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=1, validCount=1 2024-11-04 13:16:11.207 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=19, affectRows=19 2024-11-04 13:16:11.207 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=19, validCount=19 2024-11-04 13:16:11.207 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=18, affectRows=18 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=18, validCount=18 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=36, affectRows=36 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=9, affectRows=9 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=55, affectRows=55 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=9, validCount=9 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=45, affectRows=45 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=46, affectRows=46 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=36, validCount=36 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=38, affectRows=38 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=54, validCount=54 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=46, validCount=46 2024-11-04 13:16:11.208 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=44, validCount=44 2024-11-04 13:16:11.209 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=38, validCount=38 2024-11-04 13:16:11.209 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697300, nodeCount=55, affectRows=55 2024-11-04 13:16:11.209 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697300, nodeCount=54, validCount=54 2024-11-04 13:16:11.983 INFO 21062 --- [prometheus-minute-schedule-1,bc348776e77ebe91,6ddd357e48974c5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:12.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:12.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:12.802 WARN 21062 --- [ocp-async-9,2802f480915f05ca,c219ea51974892de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:12.847 INFO 21062 --- [ocp-async-7,ac761bf46d8f54c4,910899d4c7da48d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:13.984 INFO 21062 --- [prometheus-cache-monitor1,8528b52ac7142337,ab21afea87446f21] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:16:13.984 INFO 21062 --- [prometheus-cache-monitor1,8528b52ac7142337,ab21afea87446f21] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:16:14.802 WARN 21062 --- [ocp-async-7,ea68fac67fdb78df,9ab789786f5d6f10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:14.848 INFO 21062 --- [ocp-async-1,ea43f9577d95730c,df365c4b0334b1a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:14.983 INFO 21062 --- [prometheus-minute-schedule-1,33f5f784e5a0cad1,f43a8686132c1da3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:15.330 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:15.330 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:16.802 WARN 21062 --- [ocp-async-1,8b508d8c6237f274,8e1cfd519eaabab0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:16.848 INFO 21062 --- [ocp-async-3,4f81a6409b174ca0,fd31a23db7abfbd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8a3d0adb853c0def,ccad2c6da69791c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:18.331 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:18.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:18.585 INFO 21062 --- [ocp-async-8,0610bde2784070d6,51bb8db30ad09f0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:16:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:16:18.589735+08:00, lastProcessTime=2024-11-04T13:16:08.590681+08:00 2024-11-04 13:16:18.693 INFO 21062 --- [ocp-schedules-27,cdcb7c7516bdd649,24e5bc6f401c1d8c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:16:18.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:16:18.589735+08:00 2024-11-04 13:16:18.802 WARN 21062 --- [ocp-async-9,688e6b4797430a82,630ec8d7181f879c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:18.848 INFO 21062 --- [ocp-async-7,4f767a409480c0b5,0c30bc565a299fb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:19.260 INFO 21062 --- [ocp-schedules-24,bc79a925b4cce983,0efcaf5bffd6c237] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:16:20.802 WARN 21062 --- [ocp-async-7,83c6fafb6a2d16ac,9124eea42540d2a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:20.848 INFO 21062 --- [ocp-async-5,8c7a4c56c66e42e7,113d2c2b9eb647d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:20.983 INFO 21062 --- [prometheus-minute-schedule-1,165463bad5d045c8,1063e5f69bd5ec8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:21.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:21.333 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:22.802 WARN 21062 --- [ocp-async-5,6c71163af5bd67a2,36856891a6f0b563] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:22.848 INFO 21062 --- [ocp-async-4,6de405dfec127fce,04520990e4c57b55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:23.983 INFO 21062 --- [prometheus-minute-schedule-1,f549ddd0fc344515,5d3e14d8a22ed8a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:24.333 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:24.334 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:24.803 WARN 21062 --- [ocp-async-4,ce1b9134d5da7967,5bbc5bd178211f0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:24.848 INFO 21062 --- [ocp-async-6,fc69230880b3ae35,9ee9f0498ec8a768] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:26.803 WARN 21062 --- [ocp-async-6,edd66dfdd9634af7,5d3f5da06c5cd5ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:26.849 INFO 21062 --- [ocp-async-2,faddf8be94a3cdfc,3c99b34f1fa9c370] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:26.983 INFO 21062 --- [prometheus-minute-schedule-1,5c6a098175437532,c5cbb1b5719b4e05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:26.984 INFO 21062 --- [prometheus-hour-schedule-1,6ff6f7217301a1ba,ae12634e33b7ca4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:16:27.334 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:27.335 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:28.498 INFO 21062 --- [ocp-schedules-18,ba875c62ec55ec55,cd23131a3f162460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:16:28.499 INFO 21062 --- [ocp-schedules-19,4e1cd451e8027d4b,5e58d7fb9b8e11fd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:16:28.500 INFO 21062 --- [ocp-schedules-14,1c0af4d4fc015aa7,a7d8a884ebbd3791] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:16:28.500 INFO 21062 --- [ocp-schedules-13,8d030f29a823ce7a,5ee03cfce19daa82] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:16:28.502 INFO 21062 --- [ocp-schedules-2,c462747dab70f388,1f2166b429c3b0fc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:16:28.503 INFO 21062 --- [ocp-schedules-18,ba875c62ec55ec55,cd23131a3f162460] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:16:28.504 INFO 21062 --- [ocp-schedules-14,1c0af4d4fc015aa7,a7d8a884ebbd3791] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:28.504 INFO 21062 --- [ocp-schedules-15,3b4fd17bbd8a7004,eadda4de25a0ce5a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:16:28.504 INFO 21062 --- [ocp-schedules-13,8d030f29a823ce7a,5ee03cfce19daa82] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:16:28.505 INFO 21062 --- [ocp-schedules-19,4e1cd451e8027d4b,5e58d7fb9b8e11fd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:16:28.512 INFO 21062 --- [ocp-schedules-28,c1731c3aa3159c50,0c33152bddb57b62] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:16:28.512 INFO 21062 --- [ocp-schedules-27,e858c35051c05b72,72f7dfbbaa1d4fee] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:16:28.512 INFO 21062 --- [ocp-schedules-28,c1731c3aa3159c50,0c33152bddb57b62] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:16:28.512 INFO 21062 --- [ocp-schedules-27,e858c35051c05b72,72f7dfbbaa1d4fee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:16:28.512 INFO 21062 --- [ocp-schedules-24,67cdb805296e6306,305464dfd5ed6d63] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:16:28.512 INFO 21062 --- [ocp-schedules-24,67cdb805296e6306,305464dfd5ed6d63] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:16:28.563 INFO 21062 --- [ocp-schedules-29,8374b8db8d834287,76c15f33bf26e62b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:16:28.563 INFO 21062 --- [ocp-schedules-29,8374b8db8d834287,76c15f33bf26e62b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:16:28.585 INFO 21062 --- [ocp-async-1,a6f98c58acab8ae1,1b74741683d87523] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:16:28.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:16:28.590134+08:00, lastProcessTime=2024-11-04T13:16:18.589735+08:00 2024-11-04 13:16:28.666 INFO 21062 --- [ocp-schedules-21,411093fd0e81d142,0a2d01f145585aeb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:16:28.735 INFO 21062 --- [ocp-schedules-22,b716c6bebe13bab9,e5dd49d4ce030c41] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:16:28.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:16:28.590134+08:00 2024-11-04 13:16:28.796 INFO 21062 --- [ocp-schedules-7,ac6695033910ee8c,33aa61df7fa79ed0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:16:28.796 INFO 21062 --- [ocp-schedules-7,ac6695033910ee8c,33aa61df7fa79ed0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:16:28.803 WARN 21062 --- [ocp-async-3,2444106bbfb19a4b,cbea85550cb3261c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:28.803 INFO 21062 --- [ocp-schedules-8,5a520830212253f8,93ed9d42337faa39] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:16:28.828 INFO 21062 --- [ocp-schedules-2,d3334c216f894365,9c5eb9d399653888] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:16:28.831 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:16:28.831 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:16:28.834 INFO 21062 --- [ocp-schedules-2,d3334c216f894365,9c5eb9d399653888] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:16:28.834 INFO 21062 --- [ocp-schedules-2,d3334c216f894365,9c5eb9d399653888] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:16:28.840 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:16:28.840 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:16:28.841 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:16:28.849 INFO 21062 --- [ocp-async-9,ca21e68108ea407d,425d83ad678c2a37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:28.852 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:16:28.852 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:16:28.852 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:16:28.856 INFO 21062 --- [ocp-schedules-28,5c4c6134da2d0f34,0a94ecaa19306495] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:16:28.986 INFO 21062 --- [ocp-schedules-30,e1f3c0c6a4a2f12c,0992a36f0cf4b5de] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:16:28.986 INFO 21062 --- [ocp-schedules-30,e1f3c0c6a4a2f12c,0992a36f0cf4b5de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:16:28.987 INFO 21062 --- [ocp-schedules-30,e1f3c0c6a4a2f12c,0992a36f0cf4b5de] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:16:28.987 INFO 21062 --- [ocp-schedules-30,e1f3c0c6a4a2f12c,0992a36f0cf4b5de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:16:28.988 INFO 21062 --- [ocp-schedules-30,e1f3c0c6a4a2f12c,0992a36f0cf4b5de] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:16:28.988 INFO 21062 --- [ocp-schedules-30,e1f3c0c6a4a2f12c,0992a36f0cf4b5de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:16:29.262 INFO 21062 --- [ocp-schedules-20,7104ccc6948da76a,5431a89dc492bb8c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:16:29.732 INFO 21062 --- [ocp-schedules-12,4f21694a774fecda,1232f5fb75c6948e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:16:29.732 INFO 21062 --- [ocp-schedules-12,4f21694a774fecda,1232f5fb75c6948e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:16:29.735 INFO 21062 --- [ocp-schedules-6,9202c2db34ae1e38,4ccae0c63f4e138e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:16:29.741 INFO 21062 --- [ocp-schedules-6,9202c2db34ae1e38,4ccae0c63f4e138e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:16:29.741 INFO 21062 --- [ocp-schedules-6,9202c2db34ae1e38,4ccae0c63f4e138e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:16:29.758 INFO 21062 --- [ocp-schedules-12,4f21694a774fecda,1232f5fb75c6948e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:16:29.759 INFO 21062 --- [ocp-schedules-12,4f21694a774fecda,1232f5fb75c6948e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:16:29.760 INFO 21062 --- [ocp-schedules-7,983c9cc80b459d62,406e9ff9dc0e8224] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:16:29.853 INFO 21062 --- [ocp-schedules-15,3668cffa0e94c0da,9d3941bac5a6daa0] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:16:29.857 INFO 21062 --- [ocp-schedules-15,3668cffa0e94c0da,9d3941bac5a6daa0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:29.857 INFO 21062 --- [ocp-schedules-8,abc310fe0cdfc615,6f22c0acb7b6b510] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:16:29.858 INFO 21062 --- [ocp-schedules-8,abc310fe0cdfc615,6f22c0acb7b6b510] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:16:29.858 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:16:29.858 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:16:29.868 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:16:29.868 INFO 21062 --- [ocp-schedules-8,abc310fe0cdfc615,6f22c0acb7b6b510] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:16:29.868 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:16:29.869 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:16:29.869 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:16:29.869 WARN 21062 --- [ocp-schedules-8,abc310fe0cdfc615,6f22c0acb7b6b510] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:16:29.870 INFO 21062 --- [ocp-schedules-8,abc310fe0cdfc615,6f22c0acb7b6b510] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:16:29.870 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:16:29.870 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:16:29.871 WARN 21062 --- [ocp-schedules-8,abc310fe0cdfc615,6f22c0acb7b6b510] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:16:29.871 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:16:29.871 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:16:29.880 ERROR 21062 --- [ocp-schedules-8,abc310fe0cdfc615,6f22c0acb7b6b510] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:16:29.881 INFO 21062 --- [ocp-schedules-2,a5b5b36418b0c21a,8142e8b7dd493aa2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:16:29.881+08:00, clusters=[1] 2024-11-04 13:16:29.883 INFO 21062 --- [ocp-schedules-2,a5b5b36418b0c21a,8142e8b7dd493aa2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:16:29.914 ERROR 21062 --- [ocp-schedules-15,3668cffa0e94c0da,9d3941bac5a6daa0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:16:29.914 INFO 21062 --- [ocp-schedules-15,3668cffa0e94c0da,9d3941bac5a6daa0] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:16:29.983 INFO 21062 --- [prometheus-minute-schedule-1,dffaaea985791de5,6466a4e79d93a7e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:29.992 INFO 21062 --- [ocp-schedules-27,c520b8d3d1e175bc,c9fb3f5f2d88a5b8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:16:29.992+08:00, clusters=[1] 2024-11-04 13:16:29.996 INFO 21062 --- [ocp-schedules-27,c520b8d3d1e175bc,c9fb3f5f2d88a5b8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:16:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:16:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:16:30.335 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:30.336 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:30.803 WARN 21062 --- [ocp-async-10,4bdefed014078449,e490acdb46f02320] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:30.849 INFO 21062 --- [ocp-async-6,aa40422bf1234c37,8417dfd45f86bb1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:31.513 INFO 21062 --- [ocp-schedules-10,f0b4a17ff44552e5,78060ed0723a68f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:16:31.513 INFO 21062 --- [ocp-schedules-10,51b7bef9260bb3c8,ac02d90bc169b266] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:16:31.519 INFO 21062 --- [ocp-schedules-10,51b7bef9260bb3c8,ac02d90bc169b266] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:31.548 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:16:31.548 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:16:32.535 INFO 21062 --- [ocp-schedules-17,63be9b092310a915,8b60e92402baab2c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:16:32.538 INFO 21062 --- [ocp-schedules-17,63be9b092310a915,8b60e92402baab2c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:32.625 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:16:32.627 ERROR 21062 --- [ocp-schedules-17,63be9b092310a915,8b60e92402baab2c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:16:32.627 INFO 21062 --- [ocp-schedules-17,63be9b092310a915,8b60e92402baab2c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:16:32.803 WARN 21062 --- [ocp-async-2,fabafadf5e69ebeb,d015cac58e15264c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:32.849 INFO 21062 --- [ocp-async-5,5095580fd3dcbd66,efb96b5c958b8e8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:32.983 INFO 21062 --- [prometheus-minute-schedule-1,233bd7f1481c15b8,fea97a4b29e435f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:33.336 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:33.337 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:34.803 WARN 21062 --- [ocp-async-8,1808b58d6abee960,92ddd727cbaad42b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:34.849 INFO 21062 --- [ocp-async-4,529ee10bee4effbc,fad21bc9af6d70c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:35.983 INFO 21062 --- [prometheus-minute-schedule-1,eb1556b0fbcc0083,8006deccfab11e17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:36.337 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:36.338 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:36.803 WARN 21062 --- [ocp-async-9,c3c464fbe946c3c8,7735b0a6807ac6bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:36.849 INFO 21062 --- [ocp-async-7,8ffdda62833123ae,cb4726430ec28474] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:38.505 INFO 21062 --- [ocp-schedules-13,8d030f29a823ce7a,5ee03cfce19daa82] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:16:38.505 INFO 21062 --- [ocp-schedules-14,1c0af4d4fc015aa7,a7d8a884ebbd3791] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:16:38.585 INFO 21062 --- [ocp-async-7,05620f7762450356,d315fb6dca3a58c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:16:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:16:38.589300+08:00, lastProcessTime=2024-11-04T13:16:28.590134+08:00 2024-11-04 13:16:38.738 INFO 21062 --- [ocp-schedules-17,7d78b2352d64985b,0b141a0c3f9fe415] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:16:38.743 INFO 21062 --- [ocp-schedules-17,7d78b2352d64985b,0b141a0c3f9fe415] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:38.744 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:16:38.759 INFO 21062 --- [ocp-schedules-23,bf6f483abf103cb3,094922d02db546b0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:16:38.799 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:16:38.589300+08:00 2024-11-04 13:16:38.804 WARN 21062 --- [ocp-async-1,ff860ca3e3b6b15a,373e6e9b0324d2cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:38.850 INFO 21062 --- [ocp-async-8,d651ca222969fc45,6a96536fad7553bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5e2a3fe9d281e501,5f69b426d28820b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:39.265 INFO 21062 --- [ocp-schedules-4,e423bce948561fb1,e605b9328f7f0f45] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:16:39.338 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:39.339 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:16:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:16:40.804 WARN 21062 --- [ocp-async-3,16430c2f82ffc12d,a65daaecfc6f2021] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:40.850 INFO 21062 --- [ocp-async-9,339262fd355a887c,3105e056c05a1289] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:16:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:16:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:16:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:16:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:16:41.829 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:16:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 13:16:41.983 INFO 21062 --- [prometheus-minute-schedule-1,25c57a84f50eaed2,56d16b8d1f59cfbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:42.339 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:42.340 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:42.804 WARN 21062 --- [ocp-async-10,4d33c27ca202fb57,f1528f3b0a313105] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:42.850 INFO 21062 --- [ocp-async-6,843613ac18c5c092,5876d7d46375a28d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:44.804 WARN 21062 --- [ocp-async-2,4db4413466bf0d6e,b208840285dd4f96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:44.850 INFO 21062 --- [ocp-async-5,890808abbd4e2009,a08160b79cd958e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e9c3310c8e4bf548,a478f3fa5450e3dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:45.340 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:45.341 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:46.804 WARN 21062 --- [ocp-async-8,30bb7ab3a20b8166,599810361b9800ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:46.850 INFO 21062 --- [ocp-async-4,af58275b79107a5f,dfff595d1a0722b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:47.983 INFO 21062 --- [prometheus-minute-schedule-1,db012c0771d06a3d,baad30ab559c38b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:48.341 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:48.342 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:48.585 INFO 21062 --- [ocp-async-8,f8085981c96c12bf,96da3c8b45fc344c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:16:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:16:48.591647+08:00, lastProcessTime=2024-11-04T13:16:38.589300+08:00 2024-11-04 13:16:48.744 INFO 21062 --- [ocp-schedules-17,7d78b2352d64985b,0b141a0c3f9fe415] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:16:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:16:48.591647+08:00 2024-11-04 13:16:48.800 INFO 21062 --- [ocp-schedules-24,a4c1b2623882e9f1,1102d60ac5bebe09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:16:48.804 WARN 21062 --- [ocp-async-10,593f4944fe42d589,fcfa066eb75fb63a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:48.850 INFO 21062 --- [ocp-async-6,1e4ac78f2d2fb13d,381dedc80a3c8029] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:49.267 INFO 21062 --- [ocp-schedules-7,413673a77c81d735,c5cb7d712b569864] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:16:49.744 INFO 21062 --- [ocp-schedules-22,5e83606ca0bc21a1,a341650c3fb69a50] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:16:49.750 INFO 21062 --- [ocp-schedules-22,5e83606ca0bc21a1,a341650c3fb69a50] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:16:49.750 INFO 21062 --- [ocp-schedules-22,5e83606ca0bc21a1,a341650c3fb69a50] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:16:50.805 WARN 21062 --- [ocp-async-6,c68503151fc150c2,5b1aa313a54083f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:50.851 INFO 21062 --- [ocp-async-5,411e4a5df66a2d2a,29b198415d01a8c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8f1b7496d4742f2c,704f20bf0ccafc69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:51.342 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:51.343 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:52.805 WARN 21062 --- [ocp-async-5,3b9ae637eac4d460,2f44e38b111fdac5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:52.851 INFO 21062 --- [ocp-async-4,d0d000a45f14e8d8,4f174554b62b4531] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:53.983 INFO 21062 --- [prometheus-minute-schedule-1,925e46d3e107ed53,10b71f8707a38647] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:54.343 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:54.344 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:54.805 WARN 21062 --- [ocp-async-4,433fb13a17980416,5d19d65a08de5d75] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:54.851 INFO 21062 --- [ocp-async-7,fb5686f2f120dbf8,785a3cc3d2712542] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:56.805 WARN 21062 --- [ocp-async-7,4df8fda5c90a4974,12bd9d5b2fd0d073] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:56.851 INFO 21062 --- [ocp-async-1,058ee6a60244b632,69b8ab12305c1133] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:56.983 INFO 21062 --- [prometheus-minute-schedule-1,3ed0bcc239821c90,6ced895a27d60065] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:16:57.344 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:16:57.345 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:16:58.499 INFO 21062 --- [ocp-schedules-7,3cafd484bb5612fe,56bb2ffd2126637a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:16:58.500 INFO 21062 --- [ocp-schedules-27,be0ce98bafc7678e,bfc938c2d81dd5cd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:16:58.503 INFO 21062 --- [ocp-schedules-15,98eec01903fad911,c5aa01bca8c99478] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:16:58.504 INFO 21062 --- [ocp-schedules-27,be0ce98bafc7678e,bfc938c2d81dd5cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:58.504 INFO 21062 --- [ocp-schedules-7,3cafd484bb5612fe,56bb2ffd2126637a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:16:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:16:58.505 INFO 21062 --- [ocp-schedules-21,e9f8341e531f79fe,1dc786f45a5bd1e5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:16:58.512 INFO 21062 --- [ocp-schedules-13,ad70a23660bd21af,c7a9a0e263dffa87] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:16:58.512 INFO 21062 --- [ocp-schedules-14,16339f99664bb08f,9ac6d892fbfc5e19] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:16:58.512 INFO 21062 --- [ocp-schedules-14,16339f99664bb08f,9ac6d892fbfc5e19] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:16:58.512 INFO 21062 --- [ocp-schedules-13,ad70a23660bd21af,c7a9a0e263dffa87] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:16:58.512 INFO 21062 --- [ocp-schedules-1,6c977c7e37248cb5,3d42c7e6b33e1730] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:16:58.512 INFO 21062 --- [ocp-schedules-1,6c977c7e37248cb5,3d42c7e6b33e1730] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:16:58.586 INFO 21062 --- [ocp-async-6,f52a63bbac3b5845,fa550a3314f35d8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:16:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:16:58.589648+08:00, lastProcessTime=2024-11-04T13:16:48.591647+08:00 2024-11-04 13:16:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:16:58.589648+08:00 2024-11-04 13:16:58.806 INFO 21062 --- [ocp-schedules-19,597752e6f9f8ae8c,f192879f7b62842d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:16:58.806 WARN 21062 --- [ocp-async-5,a88c3cb411405213,a12a9f431bfac9e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:16:58.825 INFO 21062 --- [ocp-schedules-8,069e75452b0c427d,635235fb71763821] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:16:58.852 INFO 21062 --- [ocp-async-4,5ece747c7614b91a,a8601e9c530325e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:16:59.270 INFO 21062 --- [ocp-schedules-22,b249fe85a6d950bc,f53ba62b5af2da37] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:16:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1ea1d8ae216b4b29,6f3578544f7f3473] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697420, interval=60 2024-11-04 13:17:00.345 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:00.346 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:00.806 WARN 21062 --- [ocp-async-4,cca1dd14ef3b82ba,8612499863c4757d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:00.852 INFO 21062 --- [ocp-async-7,78239e429fe339b6,f95e34ba184d71fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:01.549 INFO 21062 --- [ocp-schedules-10,51b7bef9260bb3c8,ac02d90bc169b266] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:17:01.549 INFO 21062 --- [ocp-schedules-10,eb8a03369c9582f9,28eef4bd84ba0793] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:17:01.552 INFO 21062 --- [ocp-schedules-10,eb8a03369c9582f9,28eef4bd84ba0793] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:01.581 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:17:01.581 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:17:02.806 WARN 21062 --- [ocp-async-7,1ddedb23fae1ebc9,9da89bf34e0aa508] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:02.852 INFO 21062 --- [ocp-async-1,a6f806acad2f8b15,cb05dbf2c02ce2e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1af28153ab3657b3,45e218f48a580534] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:03.346 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:03.347 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:04.806 WARN 21062 --- [ocp-async-1,a7e9532bc4a7e462,184ad6121f7d869d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:04.852 INFO 21062 --- [ocp-async-8,3c6f298897f3bd71,0a7e2df37bcfbb91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:05.983 INFO 21062 --- [prometheus-minute-schedule-1,9520e1c5adb05402,cefd8aab3eed1d84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:06.347 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:06.349 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:06.806 WARN 21062 --- [ocp-async-8,f627a85c0d23179e,209e03e7cbd06aec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:06.852 INFO 21062 --- [ocp-async-9,6ebb97a23b45fd56,d0732968bb6dc2a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:08.505 INFO 21062 --- [ocp-schedules-27,be0ce98bafc7678e,bfc938c2d81dd5cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:17:08.505 INFO 21062 --- [ocp-schedules-7,3cafd484bb5612fe,56bb2ffd2126637a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:17:08.586 INFO 21062 --- [ocp-async-4,515f2cdd75bc6dd1,3902f9728a3950af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:17:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:17:08.590786+08:00, lastProcessTime=2024-11-04T13:16:58.589648+08:00 2024-11-04 13:17:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:17:08.590786+08:00 2024-11-04 13:17:08.807 WARN 21062 --- [ocp-async-7,09819f2e3c98e9fb,2cf23d9a1752f66e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:08.853 INFO 21062 --- [ocp-async-1,90e0eeec8228359d,c6549514bf156004] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:08.872 INFO 21062 --- [ocp-schedules-1,0eb351033f8d1221,46e365d50b7cfb29] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:17:08.983 INFO 21062 --- [prometheus-minute-schedule-1,3acf304d088dae4c,01cd9f50f7743632] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:09.273 INFO 21062 --- [ocp-schedules-7,11a5d0a195f8fb82,103ac25790727bb7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:17:09.349 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:09.351 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:09.752 INFO 21062 --- [ocp-schedules-23,a4290e0963627c02,a53e85d82db10c2f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:17:09.758 INFO 21062 --- [ocp-schedules-23,a4290e0963627c02,a53e85d82db10c2f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:17:09.758 INFO 21062 --- [ocp-schedules-23,a4290e0963627c02,a53e85d82db10c2f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:17:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:17:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:17:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:17:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:17:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:17:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:17:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:17:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:17:10.807 WARN 21062 --- [ocp-async-1,748b337d678be6f3,434b8c0b983b9ce6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:10.853 INFO 21062 --- [ocp-async-8,1148fa600e6f7fc9,ae6a90a73ab5b3b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:11.210 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=5, affectRows=5 2024-11-04 13:17:11.210 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=5, validCount=5 2024-11-04 13:17:11.211 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=24, affectRows=24 2024-11-04 13:17:11.211 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=24, validCount=24 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=12, affectRows=12 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=18, affectRows=18 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=39, affectRows=39 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=11, validCount=11 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=34, affectRows=34 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=55, affectRows=55 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=18, validCount=18 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=71, affectRows=71 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=39, validCount=39 2024-11-04 13:17:11.212 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=34, validCount=34 2024-11-04 13:17:11.213 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697360, nodeCount=64, affectRows=64 2024-11-04 13:17:11.213 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=55, validCount=55 2024-11-04 13:17:11.213 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=69, validCount=69 2024-11-04 13:17:11.213 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697360, nodeCount=64, validCount=64 2024-11-04 13:17:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e7f79626e8175988,f444f7b7a38ba880] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:12.351 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:12.352 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:12.807 WARN 21062 --- [ocp-async-8,f15d9476774d4e71,539536502fea5ec4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:12.853 INFO 21062 --- [ocp-async-9,1a25d17fdc789008,099ab14546d694f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:13.984 INFO 21062 --- [prometheus-cache-monitor1,37e97f88a5b28613,4538003ba382f1ff] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:17:13.984 INFO 21062 --- [prometheus-cache-monitor1,37e97f88a5b28613,4538003ba382f1ff] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:17:14.447 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:13:14.422, end=2024-11-04T13:15:14.445} 2024-11-04 13:17:14.452 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:17:14.452 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:17:14.807 WARN 21062 --- [ocp-async-9,33a1685a2b4b7889,25176c8fbe537f41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:14.853 INFO 21062 --- [ocp-async-2,05e323bc6e7609b0,c3c2600d05460663] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:14.983 INFO 21062 --- [prometheus-minute-schedule-1,64eca4787eaae648,d50d23af5b3e8d74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:15.352 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:15.353 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:16.807 WARN 21062 --- [ocp-async-2,a8c8d00ff10c551e,486d615d3a032ccc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:16.853 INFO 21062 --- [ocp-async-3,255ec1b565d89dde,5860ce6a660c825d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:17.983 INFO 21062 --- [prometheus-minute-schedule-1,37da1135791dfddd,131b4ad1ffc2ee4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:18.353 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:18.354 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:18.586 INFO 21062 --- [ocp-async-1,939654fa49d0adfb,4cb8015132c447ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:17:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:17:18.591109+08:00, lastProcessTime=2024-11-04T13:17:08.590786+08:00 2024-11-04 13:17:18.777 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:17:18.591109+08:00 2024-11-04 13:17:18.807 WARN 21062 --- [ocp-async-8,493e1f4ed153274e,b7ad42e16a9c8268] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:18.854 INFO 21062 --- [ocp-async-9,fa9e7827f7afe62c,95460efd44362cd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:18.902 INFO 21062 --- [ocp-schedules-8,44f1ee61388c78df,724cd18e7493f74a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:17:19.276 INFO 21062 --- [ocp-schedules-15,c996c0f7deb8c7ba,1abe04af8ecfcdca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:17:20.808 WARN 21062 --- [ocp-async-9,44b30bcf3d770b0d,e40ea163af4d8908] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:20.854 INFO 21062 --- [ocp-async-6,129183f23ebb20ba,7d5eed655b6b09ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5fff139d1da17f84,df3cd55f132c45d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:21.355 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:21.355 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:22.808 WARN 21062 --- [ocp-async-6,96504a792f99e724,7318dea3237e5525] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:22.854 INFO 21062 --- [ocp-async-5,2c98093175466b54,d5b522e20f68e830] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:23.983 INFO 21062 --- [prometheus-minute-schedule-1,cdff01fcf25f3128,71d4f6bd5c1ed3d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:24.355 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:24.356 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:24.808 WARN 21062 --- [ocp-async-5,c1d89df8cb4c08eb,71c5b543663f307e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:24.854 INFO 21062 --- [ocp-async-4,986a1d5b069bac70,e4379e865d40fe14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:26.808 WARN 21062 --- [ocp-async-4,11510cdcd19ec5fe,d96047050b50b365] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:26.854 INFO 21062 --- [ocp-async-7,7363f1a1e7b53321,d9c7754f7d209489] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:26.983 INFO 21062 --- [prometheus-minute-schedule-1,512b3b507867e9e9,530481d1861056c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:26.984 INFO 21062 --- [prometheus-hour-schedule-1,1f9eeaa8f992f002,02c06a167bff646b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:17:27.356 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:27.357 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:28.498 INFO 21062 --- [ocp-schedules-5,4449db6e5d053b56,b014b2fdc397ed28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:17:28.499 INFO 21062 --- [ocp-schedules-22,0e720e5ad60b2ea0,6ce456acd9dfce88] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:17:28.500 INFO 21062 --- [ocp-schedules-23,12cf313a2feea9e3,6f2342e9593ce9e4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:17:28.500 INFO 21062 --- [ocp-schedules-28,8108d091352f5016,dc067be1f41c0e93] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:17:28.502 INFO 21062 --- [ocp-schedules-12,32a7121d5fb870cb,28555308f9750871] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:17:28.503 INFO 21062 --- [ocp-schedules-5,4449db6e5d053b56,b014b2fdc397ed28] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:17:28.503 INFO 21062 --- [ocp-schedules-23,12cf313a2feea9e3,6f2342e9593ce9e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:28.504 INFO 21062 --- [ocp-schedules-28,8108d091352f5016,dc067be1f41c0e93] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:17:28.504 INFO 21062 --- [ocp-schedules-3,bfdd5afc2513e113,d0481e961ed76f2f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:17:28.504 INFO 21062 --- [ocp-schedules-29,bc79c783a2cf0eb7,b152eccfdc5305d8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:17:28.505 INFO 21062 --- [ocp-schedules-22,0e720e5ad60b2ea0,6ce456acd9dfce88] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:17:28.507 INFO 21062 --- [ocp-schedules-29,bc79c783a2cf0eb7,b152eccfdc5305d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:28.510 INFO 21062 --- [ocp-schedules-20,8cddae3fec8f2191,042aad2143ad0009] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:17:28.510 INFO 21062 --- [ocp-schedules-20,8cddae3fec8f2191,042aad2143ad0009] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:17:28.510 INFO 21062 --- [ocp-schedules-11,8114f34271af3949,956d7dc5c8d4504f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:17:28.510 INFO 21062 --- [ocp-schedules-15,316107c75adca62f,5319a29a80087efd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:17:28.510 INFO 21062 --- [ocp-schedules-11,8114f34271af3949,956d7dc5c8d4504f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:17:28.510 INFO 21062 --- [ocp-schedules-15,316107c75adca62f,5319a29a80087efd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:17:28.530 INFO 21062 --- [ocp-schedules-16,9cd7b9c5aa9f3296,6f8bc2ae03a11011] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:17:28.530 INFO 21062 --- [ocp-schedules-16,9cd7b9c5aa9f3296,6f8bc2ae03a11011] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:17:28.575 INFO 21062 --- [pool-12-thread-5,a922a1737bd5a7f8,347e928d863112a6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:17:28.575 INFO 21062 --- [pool-12-thread-8,a922a1737bd5a7f8,db80159bf90697a3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:17:28.575 INFO 21062 --- [pool-12-thread-6,a922a1737bd5a7f8,7d363bc3a235b99e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:17:28.575 INFO 21062 --- [pool-12-thread-6,a922a1737bd5a7f8,7d363bc3a235b99e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:17:28.576 INFO 21062 --- [pool-12-thread-7,a922a1737bd5a7f8,3f9f760443b6a512] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:17:28.579 INFO 21062 --- [pool-12-thread-6,a922a1737bd5a7f8,7d363bc3a235b99e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:17:28.580 INFO 21062 --- [ForkJoinPool-11-worker-5,a922a1737bd5a7f8,a039d14ade4d2466] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:17:28.580 INFO 21062 --- [ForkJoinPool-11-worker-14,a922a1737bd5a7f8,b0575640825c3c86] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:17:28.580 INFO 21062 --- [pool-12-thread-8,a922a1737bd5a7f8,db80159bf90697a3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:17:28.580 INFO 21062 --- [ForkJoinPool-11-worker-14,a922a1737bd5a7f8,b0575640825c3c86] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:17:28.581 INFO 21062 --- [ForkJoinPool-11-worker-14,a922a1737bd5a7f8,b0575640825c3c86] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:17:28.581 INFO 21062 --- [pool-12-thread-5,a922a1737bd5a7f8,347e928d863112a6] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:17:28.582 INFO 21062 --- [pool-12-thread-6,a922a1737bd5a7f8,7d363bc3a235b99e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:17:28.585 INFO 21062 --- [pool-12-thread-7,a922a1737bd5a7f8,3f9f760443b6a512] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:17:28.585 INFO 21062 --- [pool-12-thread-7,a922a1737bd5a7f8,3f9f760443b6a512] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:17:28.586 INFO 21062 --- [ocp-async-2,dc27c5b92dbe2c84,81a9499016509ebd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:17:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:17:28.589022+08:00, lastProcessTime=2024-11-04T13:17:18.591109+08:00 2024-11-04 13:17:28.669 INFO 21062 --- [ocp-schedules-17,7c0349e6dea70da4,6c69f3d1b9f232f6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:17:28.675 INFO 21062 --- [ocp-schedules-13,4eff09cb42323d51,2b77c917043087b9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:17:28.688 INFO 21062 --- [ocp-schedules-13,4eff09cb42323d51,2b77c917043087b9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:17:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:17:28.589022+08:00 2024-11-04 13:17:28.801 INFO 21062 --- [ocp-schedules-3,9d78f2e045d7d698,de121918e6f4c86a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:17:28.801 INFO 21062 --- [ocp-schedules-3,9d78f2e045d7d698,de121918e6f4c86a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:17:28.808 INFO 21062 --- [ocp-schedules-12,459f90471eab008c,f0f8ea6cb4d5e190] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:17:28.808 WARN 21062 --- [ocp-async-3,5ecf56bf61e91eba,65ad94fddf9d32f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:28.834 INFO 21062 --- [ocp-schedules-15,2ee3e86dc48b440a,db4faad1c55baa14] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:17:28.839 INFO 21062 --- [ocp-schedules-15,2ee3e86dc48b440a,db4faad1c55baa14] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:17:28.839 INFO 21062 --- [ocp-schedules-15,2ee3e86dc48b440a,db4faad1c55baa14] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:17:28.855 INFO 21062 --- [ocp-async-8,a3055bc80b11fc03,4288059a420220ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:28.943 INFO 21062 --- [ocp-schedules-18,9ce2199226d02576,0ee9588fa0ed8aaf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:17:28.989 INFO 21062 --- [ocp-schedules-2,0a599daaf741044b,ebfc33f191e1dcc9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:17:28.989 INFO 21062 --- [ocp-schedules-2,0a599daaf741044b,ebfc33f191e1dcc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:17:28.990 INFO 21062 --- [ocp-schedules-2,0a599daaf741044b,ebfc33f191e1dcc9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:17:28.990 INFO 21062 --- [ocp-schedules-2,0a599daaf741044b,ebfc33f191e1dcc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:17:28.990 INFO 21062 --- [ocp-schedules-2,0a599daaf741044b,ebfc33f191e1dcc9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:17:28.990 INFO 21062 --- [ocp-schedules-2,0a599daaf741044b,ebfc33f191e1dcc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:17:29.011 INFO 21062 --- [ocp-schedules-4,451a1c00b7e0cde6,945dd1e98f4a1c5f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:17:29.011 INFO 21062 --- [ocp-schedules-4,451a1c00b7e0cde6,945dd1e98f4a1c5f] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:17:29.040 INFO 21062 --- [pool-28-thread-1,451a1c00b7e0cde6,e2f88223cb607fc4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:17:29.070 WARN 21062 --- [ocp-schedules-4,451a1c00b7e0cde6,945dd1e98f4a1c5f] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:17:29.070 INFO 21062 --- [ocp-schedules-4,451a1c00b7e0cde6,945dd1e98f4a1c5f] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:17:29.278 INFO 21062 --- [ocp-schedules-6,fd2c428a70d6fb11,94f2cfe8a925a77e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:17:29.759 INFO 21062 --- [ocp-schedules-7,078ff371f8558c3f,6b85d879a225b36d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:17:29.759 INFO 21062 --- [ocp-schedules-7,078ff371f8558c3f,6b85d879a225b36d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:17:29.761 INFO 21062 --- [ocp-schedules-21,a8f60c1b12e4969f,bff295989215bbd5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:17:29.767 INFO 21062 --- [ocp-schedules-21,a8f60c1b12e4969f,bff295989215bbd5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:17:29.767 INFO 21062 --- [ocp-schedules-21,a8f60c1b12e4969f,bff295989215bbd5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:17:29.786 INFO 21062 --- [ocp-schedules-7,078ff371f8558c3f,6b85d879a225b36d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:17:29.786 INFO 21062 --- [ocp-schedules-7,078ff371f8558c3f,6b85d879a225b36d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:17:29.789 INFO 21062 --- [ocp-schedules-3,b1f989436a6d036d,81d7db5c74272800] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:17:29.891 INFO 21062 --- [ocp-schedules-15,093085f1ee029206,c12c0372c97dca97] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:17:29.891 INFO 21062 --- [ocp-schedules-15,093085f1ee029206,c12c0372c97dca97] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:17:29.891 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:17:29.891 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:17:29.900 INFO 21062 --- [ocp-schedules-15,093085f1ee029206,c12c0372c97dca97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:17:29.900 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:17:29.900 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:17:29.902 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:17:29.902 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:17:29.902 WARN 21062 --- [ocp-schedules-15,093085f1ee029206,c12c0372c97dca97] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:17:29.903 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:17:29.903 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:17:29.903 INFO 21062 --- [ocp-schedules-15,093085f1ee029206,c12c0372c97dca97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:17:29.904 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:17:29.904 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:17:29.904 WARN 21062 --- [ocp-schedules-15,093085f1ee029206,c12c0372c97dca97] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:17:29.908 INFO 21062 --- [ocp-schedules-16,2bd4ea0ab3259ffa,9fd708b5324ce899] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:17:29.908+08:00, clusters=[1] 2024-11-04 13:17:29.909 INFO 21062 --- [ocp-schedules-16,2bd4ea0ab3259ffa,9fd708b5324ce899] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:17:29.911 ERROR 21062 --- [ocp-schedules-15,093085f1ee029206,c12c0372c97dca97] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:17:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b4c608d9574304b1,f9a778b1e95fd5e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:30.022 INFO 21062 --- [ocp-schedules-19,8fda1bca5d7c614b,8fb7533ae9dafcf1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:17:30.022+08:00, clusters=[1] 2024-11-04 13:17:30.025 INFO 21062 --- [ocp-schedules-19,8fda1bca5d7c614b,8fb7533ae9dafcf1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:17:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:17:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:17:30.357 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:30.358 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:30.808 WARN 21062 --- [ocp-async-10,64e30f806c4036e5,b13c59193f610c9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:30.855 INFO 21062 --- [ocp-async-4,6e70f065325e21f4,e1be823638086466] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:31.582 INFO 21062 --- [ocp-schedules-10,eb8a03369c9582f9,28eef4bd84ba0793] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:17:31.582 INFO 21062 --- [ocp-schedules-10,bb7cf52a8b153631,5cf2e248fdb89cdc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:17:31.585 INFO 21062 --- [ocp-schedules-10,bb7cf52a8b153631,5cf2e248fdb89cdc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:31.616 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:17:31.616 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:17:32.627 INFO 21062 --- [ocp-schedules-1,87bc3e044c9bb43d,74faaf3074c20c53] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:17:32.631 INFO 21062 --- [ocp-schedules-1,87bc3e044c9bb43d,74faaf3074c20c53] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:32.720 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:17:32.722 ERROR 21062 --- [ocp-schedules-1,87bc3e044c9bb43d,74faaf3074c20c53] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:17:32.722 INFO 21062 --- [ocp-schedules-1,87bc3e044c9bb43d,74faaf3074c20c53] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:17:32.809 WARN 21062 --- [ocp-async-7,05d34e7cdbaf67fb,a4a0fad04866cd9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:32.855 INFO 21062 --- [ocp-async-6,854d48c0f5dcd824,74b84aa125ebe497] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7381cbb15bae5f1e,6d0435e8ce74b6b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:33.358 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:33.360 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:34.809 WARN 21062 --- [ocp-async-1,8ce20828029b0a89,ce23284674dc7c3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:34.855 INFO 21062 --- [ocp-async-5,1abff3791ae12475,67a66e1c6585d110] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:35.983 INFO 21062 --- [prometheus-minute-schedule-1,55c0a41aaa8e7702,c7a48cc82919e677] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:36.360 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:36.361 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:36.809 WARN 21062 --- [ocp-async-8,dd2b30faa32ab10a,7f3944fb8fc8d9ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:36.855 INFO 21062 --- [ocp-async-9,21f227bca7f5faca,d6a74d47d2eb1c10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:38.504 INFO 21062 --- [ocp-schedules-23,12cf313a2feea9e3,6f2342e9593ce9e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:17:38.504 INFO 21062 --- [ocp-schedules-28,8108d091352f5016,dc067be1f41c0e93] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:17:38.508 INFO 21062 --- [ocp-schedules-29,bc79c783a2cf0eb7,b152eccfdc5305d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:17:38.587 INFO 21062 --- [ocp-async-10,36f83b093b004f6a,3924ebe9ec9714df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:17:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:17:38.591401+08:00, lastProcessTime=2024-11-04T13:17:28.589022+08:00 2024-11-04 13:17:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:17:38.591401+08:00 2024-11-04 13:17:38.809 WARN 21062 --- [ocp-async-2,7b8b46c1bb35dba0,888c2cb59600179c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:38.855 INFO 21062 --- [ocp-async-1,ba044cbc2803a916,8e6da938e8f3ce50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:38.968 INFO 21062 --- [ocp-schedules-22,f52fd23a7272e987,239ea0a822c07574] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:17:38.983 INFO 21062 --- [prometheus-minute-schedule-1,f5b57f070fb03344,7c8799cb74ffdaea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:39.281 INFO 21062 --- [ocp-schedules-1,6728658e4a8d4c1b,57323b2970971b29] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:17:39.361 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:39.362 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:17:40.003 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:17:40.809 WARN 21062 --- [ocp-async-3,89e2e882ae696d29,6bd8fe884b2b70af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:40.856 INFO 21062 --- [ocp-async-8,bb0684b250623a0a,929e0d3d92fa5dd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c32a5b56afde9bca,825b6bead303ef69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:42.362 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:42.363 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:42.809 WARN 21062 --- [ocp-async-9,5580c42fa3b1e2dd,e4f5a4be8e0313df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:42.856 INFO 21062 --- [ocp-async-4,439960b12593eba5,36e73e1fa352a118] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:44.810 WARN 21062 --- [ocp-async-7,518dc9f487ecc40c,81043435440c0bc5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:44.856 INFO 21062 --- [ocp-async-6,1dae0c75a20030bd,14c3f982199c2755] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6d3e41131e9bc7af,6fc60fd5ee7637a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:45.363 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:45.364 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:46.810 WARN 21062 --- [ocp-async-1,c0ceac5bae2e22b2,aeb9d9364dd85161] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:46.856 INFO 21062 --- [ocp-async-5,61de97b4df6ab4ce,32a8aa778ef273e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:47.983 INFO 21062 --- [prometheus-minute-schedule-1,cfaf1cfe01ddba92,d94e6a89d15bbc66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:48.364 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:48.365 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:48.587 INFO 21062 --- [ocp-async-1,bf701a7c38dfc688,cd79730d6edc3f67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:17:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:17:48.590715+08:00, lastProcessTime=2024-11-04T13:17:38.591401+08:00 2024-11-04 13:17:48.744 INFO 21062 --- [ocp-schedules-1,ff601441f928ef36,8b4a5e8706d4983c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:17:48.750 INFO 21062 --- [ocp-schedules-1,ff601441f928ef36,8b4a5e8706d4983c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:48.751 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:17:48.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:17:48.590715+08:00 2024-11-04 13:17:48.810 WARN 21062 --- [ocp-async-9,f6e855c045fb12fe,f92c7a338e115ff3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:48.856 INFO 21062 --- [ocp-async-4,94a08a4ec16b702a,a3cd5a832c8a33f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:49.008 INFO 21062 --- [ocp-schedules-8,6034ede9ff558329,7c9baad29cc005e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:17:49.283 INFO 21062 --- [ocp-schedules-26,0c269dbf1e4f2937,3915255c070cdffd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:17:49.770 INFO 21062 --- [ocp-schedules-19,626c452372a6282e,712da7eebad4ea21] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:17:49.777 INFO 21062 --- [ocp-schedules-19,626c452372a6282e,712da7eebad4ea21] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:17:49.777 INFO 21062 --- [ocp-schedules-19,626c452372a6282e,712da7eebad4ea21] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:17:50.810 WARN 21062 --- [ocp-async-7,f2397727e86d72da,a68dc305015bbfce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:50.857 INFO 21062 --- [ocp-async-6,744e9908e2bfdf0c,71958a05f8958c7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:50.983 INFO 21062 --- [prometheus-minute-schedule-1,aa2fee8c1500c95a,0d88f2d5debeb714] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:51.365 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:51.366 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:52.810 WARN 21062 --- [ocp-async-6,867dfd35223906f0,bfe6f4aa0a474258] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:52.857 INFO 21062 --- [ocp-async-5,18da770594240601,f6a0dbc6b6ac6191] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:53.983 INFO 21062 --- [prometheus-minute-schedule-1,28d2d1662ccfcbf0,bfe5909dcb416424] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:54.366 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:54.367 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:54.810 WARN 21062 --- [ocp-async-5,c63d98d339f40df9,0a3524fd5ddf4a27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:54.857 INFO 21062 --- [ocp-async-10,61ca885bef69037a,fa189e10770f1fd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:56.811 WARN 21062 --- [ocp-async-10,9255de058c7039d3,e718162c09433d90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:56.857 INFO 21062 --- [ocp-async-2,00cc5810787658ff,ae9eb67de50fb709] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7a8400863a33fe95,039567e3348ed0f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:17:57.367 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:17:57.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:17:58.500 INFO 21062 --- [ocp-schedules-15,e67fc7187816f346,37f1e4fff1819fd6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:17:58.500 INFO 21062 --- [ocp-schedules-14,e6fca81ff90dc1ba,06043e48f2c7a36d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:17:58.502 INFO 21062 --- [ocp-schedules-3,fe7c16f6ac14ea3e,7eb8d9d6160d2873] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:17:58.504 INFO 21062 --- [ocp-schedules-15,e67fc7187816f346,37f1e4fff1819fd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:17:58.505 INFO 21062 --- [ocp-schedules-14,e6fca81ff90dc1ba,06043e48f2c7a36d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:17:58.508 INFO 21062 --- [ocp-schedules-7,2fe4f916a6205b3e,953681e2af7de785] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:17:58.511 INFO 21062 --- [ocp-schedules-30,8f0fea699b6da1f7,f1279a098ad8c497] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:17:58.511 INFO 21062 --- [ocp-schedules-30,8f0fea699b6da1f7,f1279a098ad8c497] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:17:58.512 INFO 21062 --- [ocp-schedules-13,1cda00b2dcda8cab,15b778a6b44009cc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:17:58.512 INFO 21062 --- [ocp-schedules-13,1cda00b2dcda8cab,15b778a6b44009cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:17:58.512 INFO 21062 --- [ocp-schedules-29,6cc21af5ebc45ffb,a7d8a39692458c1b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:17:58.512 INFO 21062 --- [ocp-schedules-29,6cc21af5ebc45ffb,a7d8a39692458c1b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:17:58.587 INFO 21062 --- [ocp-async-7,4de6cdfced3809cf,cf76af3091af9f6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:17:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:17:58.590134+08:00, lastProcessTime=2024-11-04T13:17:48.590715+08:00 2024-11-04 13:17:58.751 INFO 21062 --- [ocp-schedules-1,ff601441f928ef36,8b4a5e8706d4983c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:17:58.795 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:17:58.590134+08:00 2024-11-04 13:17:58.811 WARN 21062 --- [ocp-async-6,8cf9653f4a6d6863,73d98180c95d9379] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:17:58.811 INFO 21062 --- [ocp-schedules-28,d42d309530611a82,f08686610b1e7a03] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:17:58.857 INFO 21062 --- [ocp-async-5,cb7d0e5f83594872,97de9bb9ef6f60d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:17:58.935 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:17:58.935 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:17:58.941 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:17:58.941 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:17:58.941 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:17:58.941 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:17:58.949 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:17:58.949 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:17:58.949 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:17:58.949 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:17:58.987 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:17:58.995 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:17:58.995 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:17:58.995 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:17:58.995 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:17:58.996 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:17:59.003 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:17:59.034 INFO 21062 --- [ocp-schedules-24,92f5848b84c4b056,529afa6c536ae4d6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:17:59.286 INFO 21062 --- [ocp-schedules-30,c43a50163a740bb1,2724296bed9e5dba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:17:59.546 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:17:59.983 INFO 21062 --- [prometheus-minute-schedule-1,921415255ae734a5,cce565776c80af79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697480, interval=60 2024-11-04 13:18:00.102 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.153136689S 2024-11-04 13:18:00.103 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:18:00.199 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:18:00.199 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:18:00.225 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:18:00.288 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:18:00.288 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:18:00.288 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:18:00.288 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:18:00.288 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:18:00.288 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:18:00.300 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:18:00.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:00.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:00.475 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] o.s.boot.SpringApplication : Started application in 1.533 seconds (JVM running for 2894.154) 2024-11-04 13:18:00.475 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.533948192S 2024-11-04 13:18:00.811 WARN 21062 --- [ocp-async-5,60ddb8b23938a23b,fa128fa4f6a590b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:00.858 INFO 21062 --- [ocp-async-10,c45ec973263cbba1,66d0a3b6a36525c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:00.894 INFO 21062 --- [ocp-schedules-5,3868f9256a02c3e8,c23da600c4eda7c0] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:18:01.617 INFO 21062 --- [ocp-schedules-10,bb7cf52a8b153631,5cf2e248fdb89cdc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:18:01.617 INFO 21062 --- [ocp-schedules-10,97fd111aed9b154a,8fa4cb118fd1e2db] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:18:01.620 INFO 21062 --- [ocp-schedules-10,97fd111aed9b154a,8fa4cb118fd1e2db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:01.650 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:18:01.650 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:18:02.811 WARN 21062 --- [ocp-async-10,509e038c7d38312f,4147354775b257a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:02.858 INFO 21062 --- [ocp-async-2,ae8bc22a0a692426,e666f5f7a3499b5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:02.983 INFO 21062 --- [prometheus-minute-schedule-1,68cb2526705d7506,2ab4d6d60a79cc7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:03.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:03.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:04.811 WARN 21062 --- [ocp-async-2,a2086a5f937625b5,2ff5fa5e3cf49174] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:04.858 INFO 21062 --- [ocp-async-3,bcb35c9d481aac6c,e20d1888772882be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d690fb9f3457a3a6,3431b36bed6de830] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:06.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:06.371 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:06.812 WARN 21062 --- [ocp-async-3,1f50e4350a48a9db,35cf2b8b99d14648] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:06.858 INFO 21062 --- [ocp-async-9,582f5f3c9ff6dd61,4a0126fdc61c5723] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:08.505 INFO 21062 --- [ocp-schedules-15,e67fc7187816f346,37f1e4fff1819fd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:18:08.507 INFO 21062 --- [ocp-schedules-14,e6fca81ff90dc1ba,06043e48f2c7a36d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:18:08.587 INFO 21062 --- [ocp-async-5,2b0e3f79b6edfe84,3bec6d90b08ee00f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:18:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:18:08.589516+08:00, lastProcessTime=2024-11-04T13:17:58.590134+08:00 2024-11-04 13:18:08.751 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:18:08.589516+08:00 2024-11-04 13:18:08.812 WARN 21062 --- [ocp-async-10,295ab46bc67dd16b,51dc63989a3b5ba4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:08.858 INFO 21062 --- [ocp-async-2,d65011a6a6f29611,98320cee2262dd1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8d5c7338fcbb9386,2a669b4c34dac15b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:09.078 INFO 21062 --- [ocp-schedules-3,c54ca0cd4b965527,6585be87a4cb713f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:18:09.289 INFO 21062 --- [ocp-schedules-20,9ee1508cbd90d132,6b3054b049538aa4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:18:09.371 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:09.372 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:09.781 INFO 21062 --- [ocp-schedules-16,c8755209022cd855,c6e1364383cc5563] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:18:09.788 INFO 21062 --- [ocp-schedules-16,c8755209022cd855,c6e1364383cc5563] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:18:09.788 INFO 21062 --- [ocp-schedules-16,c8755209022cd855,c6e1364383cc5563] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:18:10.002 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:18:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:18:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:18:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:18:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:18:10.320 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:18:10.320 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:18:10.320 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:18:10.812 WARN 21062 --- [ocp-async-2,b44b0d6e8b6d0124,328570fb5617cb03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:10.859 INFO 21062 --- [ocp-async-3,d84f32e0631c9fee,6b264efc9591e821] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=5, affectRows=5 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=5, validCount=5 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=23, affectRows=23 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=23, validCount=23 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=10, affectRows=10 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=17, affectRows=17 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=28, affectRows=28 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=10, validCount=10 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=17, validCount=17 2024-11-04 13:18:11.214 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=28, validCount=28 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=46, affectRows=46 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=26, affectRows=26 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=26, validCount=26 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=46, validCount=46 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=50, affectRows=50 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=68, affectRows=68 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=49, validCount=49 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=66, validCount=66 2024-11-04 13:18:11.215 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697420, nodeCount=49, affectRows=49 2024-11-04 13:18:11.216 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697420, nodeCount=49, validCount=49 2024-11-04 13:18:11.983 INFO 21062 --- [prometheus-minute-schedule-1,5e070d9698283da6,c125fe4bd0af42c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:12.372 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:12.373 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:12.812 WARN 21062 --- [ocp-async-3,4768af4c94894ba5,c1bb7385c94ea82d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:12.859 INFO 21062 --- [ocp-async-9,f76f33712a64a2fb,abec812c6a320796] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:13.984 INFO 21062 --- [prometheus-cache-monitor1,a632671b4ac3e5aa,74ffd39749095ccf] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:18:13.984 INFO 21062 --- [prometheus-cache-monitor1,a632671b4ac3e5aa,74ffd39749095ccf] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:18:14.009 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:12:13.985, end=2024-11-04T13:17:14.006} 2024-11-04 13:18:14.009 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:12:13.985, end=2024-11-04T13:17:14.006} 2024-11-04 13:18:14.018 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:18:14.018 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:18:14.018 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:18:14.018 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:18:14.812 WARN 21062 --- [ocp-async-9,a1436146608ee3ff,9109d0c50a19086c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:14.859 INFO 21062 --- [ocp-async-4,8ef79c60f91c31b8,984091722e23e26f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:14.983 INFO 21062 --- [prometheus-minute-schedule-1,7af52d50e12966b2,cd62634d9e850430] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:15.373 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:15.374 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:16.812 WARN 21062 --- [ocp-async-4,a03e4bb07d9262a5,1aa89b6f140a9444] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:16.859 INFO 21062 --- [ocp-async-1,24b683212b57627a,248efd2655256ca6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:17.983 INFO 21062 --- [prometheus-minute-schedule-1,06bd3b98045afde0,412baef3a99bb63b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:18.374 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:18.375 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:18.587 INFO 21062 --- [ocp-async-2,7f3a0f39d899fd40,d718fea06e9a6f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:18:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:18:18.590518+08:00, lastProcessTime=2024-11-04T13:18:08.589516+08:00 2024-11-04 13:18:18.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:18:18.590518+08:00 2024-11-04 13:18:18.813 WARN 21062 --- [ocp-async-3,106f87d2c913860f,2f9de782b22e7905] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:18.859 INFO 21062 --- [ocp-async-9,de97b2fbf3313cd7,9e1d6f577b16c8a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:19.104 INFO 21062 --- [ocp-schedules-24,6d8eed290db87736,d64409cbbccaed65] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:18:19.291 INFO 21062 --- [ocp-schedules-2,b336db51672a8fd7,cf2dc781146b72bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:18:20.813 WARN 21062 --- [ocp-async-9,d6438c41a3a8fe87,b5252793aae8a4d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:20.859 INFO 21062 --- [ocp-async-7,3ad9a0684123ea85,70492cff2195a2c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:20.983 INFO 21062 --- [prometheus-minute-schedule-1,104e36995bdb53c7,fd1505ede13f4ab4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:21.375 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:21.376 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:22.813 WARN 21062 --- [ocp-async-7,7d24276832c82ff5,b448b6e61fb09e70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:22.860 INFO 21062 --- [ocp-async-6,12c8909192cbdedc,fc7772168d6fbd76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:23.983 INFO 21062 --- [prometheus-minute-schedule-1,75f1acf4390ea6e0,8f43694a27656581] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:24.376 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:24.377 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:24.813 WARN 21062 --- [ocp-async-6,bda7318455fe154b,c1dc4ca118c5dccd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:24.860 INFO 21062 --- [ocp-async-5,51cd2ebd1093beae,0eb2368c94291aa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:26.814 WARN 21062 --- [ocp-async-5,ffba34d37bdedf61,9863c5b69f0df5f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:26.860 INFO 21062 --- [ocp-async-10,9d04768197307a21,3792bf493c308711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2102a3e2588b283d,1d68418447de83b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5a89004f54ada587,8e387deca9847f03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:18:27.377 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:27.378 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:28.498 INFO 21062 --- [ocp-schedules-25,c5a94902f5f66608,f272f6ca45644863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:18:28.499 INFO 21062 --- [ocp-schedules-11,e22509cc53eb003a,347f0ec273a67fd2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:18:28.500 INFO 21062 --- [ocp-schedules-1,9f164e844ba0a489,c66eb9bb346213fb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:18:28.500 INFO 21062 --- [ocp-schedules-23,a04bc50ccfab8f88,9aa7639cc220e07f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:18:28.502 INFO 21062 --- [ocp-schedules-2,6496e4318953db15,54f4fa65c27e8a00] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:18:28.503 INFO 21062 --- [ocp-schedules-25,c5a94902f5f66608,f272f6ca45644863] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:18:28.504 INFO 21062 --- [ocp-schedules-1,9f164e844ba0a489,c66eb9bb346213fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:28.504 INFO 21062 --- [ocp-schedules-23,a04bc50ccfab8f88,9aa7639cc220e07f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:18:28.505 INFO 21062 --- [ocp-schedules-30,2c145b8924edbd07,e69081de6151eb5f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:18:28.506 INFO 21062 --- [ocp-schedules-11,e22509cc53eb003a,347f0ec273a67fd2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:18:28.512 INFO 21062 --- [ocp-schedules-24,11c1738c4e4160e5,68837a75d0e65ad1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:18:28.512 INFO 21062 --- [ocp-schedules-16,908c80f9dc2a3e84,c75e0ba9ef0d895b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:18:28.512 INFO 21062 --- [ocp-schedules-24,11c1738c4e4160e5,68837a75d0e65ad1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:18:28.512 INFO 21062 --- [ocp-schedules-16,908c80f9dc2a3e84,c75e0ba9ef0d895b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:18:28.512 INFO 21062 --- [ocp-schedules-18,412333579bd4e525,8c77536356784bee] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:18:28.512 INFO 21062 --- [ocp-schedules-18,412333579bd4e525,8c77536356784bee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:18:28.564 INFO 21062 --- [ocp-schedules-29,eed69ee3bbf39558,cce027428ad5af69] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:18:28.564 INFO 21062 --- [ocp-schedules-29,eed69ee3bbf39558,cce027428ad5af69] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:18:28.588 INFO 21062 --- [ocp-async-10,2a982a6bf858bb24,31c57eeea82c4b6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:18:28.590 INFO 21062 --- [ocp-schedules-28,d69272bd1cd597f4,d9e5357ae03560d7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:18:28.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:18:28.590917+08:00, lastProcessTime=2024-11-04T13:18:18.590518+08:00 2024-11-04 13:18:28.591 INFO 21062 --- [ocp-schedules-28,d69272bd1cd597f4,d9e5357ae03560d7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:18:28.603 INFO 21062 --- [ocp-schedules-21,5a49da76ae1c488a,f87f41732021004f] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:18:28.603 INFO 21062 --- [ocp-schedules-21,5a49da76ae1c488a,f87f41732021004f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:18:28.671 INFO 21062 --- [ocp-schedules-8,e90eb781cc51a5ad,f12689c68af07752] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:18:28.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:18:28.590917+08:00 2024-11-04 13:18:28.805 INFO 21062 --- [ocp-schedules-11,b8cde200700dddbf,bbe7c951cf9df402] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:18:28.805 INFO 21062 --- [ocp-schedules-11,b8cde200700dddbf,bbe7c951cf9df402] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:18:28.813 INFO 21062 --- [ocp-schedules-2,7b183667de6f3e70,17445d908b6f5a42] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:18:28.814 WARN 21062 --- [ocp-async-1,964cd78ac17840f2,481f9090329eb586] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:28.839 INFO 21062 --- [ocp-schedules-24,3a1b3f110f23e564,1172b171173e60b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:18:28.845 INFO 21062 --- [ocp-schedules-24,3a1b3f110f23e564,1172b171173e60b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:18:28.845 INFO 21062 --- [ocp-schedules-24,3a1b3f110f23e564,1172b171173e60b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:18:28.860 INFO 21062 --- [ocp-async-3,d8117cfc597adcd2,fcd6d4cf8a691764] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:28.991 INFO 21062 --- [ocp-schedules-29,a24a2802c98ea30e,b66e80756fb85670] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:18:28.991 INFO 21062 --- [ocp-schedules-29,a24a2802c98ea30e,b66e80756fb85670] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:18:28.998 INFO 21062 --- [ocp-schedules-29,a24a2802c98ea30e,b66e80756fb85670] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:18:28.998 INFO 21062 --- [ocp-schedules-29,a24a2802c98ea30e,b66e80756fb85670] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:18:29.002 INFO 21062 --- [ocp-schedules-29,a24a2802c98ea30e,b66e80756fb85670] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:18:29.002 INFO 21062 --- [ocp-schedules-29,a24a2802c98ea30e,b66e80756fb85670] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:18:29.146 INFO 21062 --- [ocp-schedules-7,901cb36c147f2c25,dd4ef3d2df17602b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:18:29.294 INFO 21062 --- [ocp-schedules-9,bbd24f0ed3eb8c0e,838f7e8cba1b951b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:18:29.786 INFO 21062 --- [ocp-schedules-2,dfdbf15d4259aff6,dad7ec2b711ddd60] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:18:29.787 INFO 21062 --- [ocp-schedules-2,dfdbf15d4259aff6,dad7ec2b711ddd60] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:18:29.791 INFO 21062 --- [ocp-schedules-30,ef41723eb017f87f,fca75bdefca486cb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:18:29.797 INFO 21062 --- [ocp-schedules-30,ef41723eb017f87f,fca75bdefca486cb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:18:29.797 INFO 21062 --- [ocp-schedules-30,ef41723eb017f87f,fca75bdefca486cb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:18:29.812 INFO 21062 --- [ocp-schedules-2,dfdbf15d4259aff6,dad7ec2b711ddd60] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:18:29.812 INFO 21062 --- [ocp-schedules-2,dfdbf15d4259aff6,dad7ec2b711ddd60] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:18:29.817 INFO 21062 --- [ocp-schedules-16,ebe94481ee657d70,861fb27ad772d8d6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:18:29.914 INFO 21062 --- [ocp-schedules-17,404728685cdb0916,2e926a532178a66d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:18:29.918 INFO 21062 --- [ocp-schedules-17,404728685cdb0916,2e926a532178a66d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:29.918 INFO 21062 --- [ocp-schedules-18,5ea26f0da5b1356e,18fe6649e65a3cfa] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:18:29.919 INFO 21062 --- [ocp-schedules-18,5ea26f0da5b1356e,18fe6649e65a3cfa] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:18:29.919 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:18:29.919 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:18:29.928 INFO 21062 --- [ocp-schedules-18,5ea26f0da5b1356e,18fe6649e65a3cfa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:18:29.929 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:18:29.929 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:18:29.930 WARN 21062 --- [ocp-schedules-18,5ea26f0da5b1356e,18fe6649e65a3cfa] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:18:29.930 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:18:29.930 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:18:29.931 INFO 21062 --- [ocp-schedules-18,5ea26f0da5b1356e,18fe6649e65a3cfa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:18:29.931 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:18:29.931 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:18:29.932 WARN 21062 --- [ocp-schedules-18,5ea26f0da5b1356e,18fe6649e65a3cfa] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:18:29.932 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:18:29.932 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:18:29.935 INFO 21062 --- [ocp-schedules-25,0bca1c3ece71ae9e,f36c4956472b5efc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:18:29.935+08:00, clusters=[1] 2024-11-04 13:18:29.937 INFO 21062 --- [ocp-schedules-25,0bca1c3ece71ae9e,f36c4956472b5efc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:18:29.938 ERROR 21062 --- [ocp-schedules-18,5ea26f0da5b1356e,18fe6649e65a3cfa] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:18:29.973 ERROR 21062 --- [ocp-schedules-17,404728685cdb0916,2e926a532178a66d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:18:29.974 INFO 21062 --- [ocp-schedules-17,404728685cdb0916,2e926a532178a66d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:18:29.983 INFO 21062 --- [prometheus-minute-schedule-1,62b60963a9720f76,335d19e0d700bd7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:30.051 INFO 21062 --- [ocp-schedules-6,ee0dbe705f727fe4,1bd255eb9e33d46a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:18:30.051+08:00, clusters=[1] 2024-11-04 13:18:30.054 INFO 21062 --- [ocp-schedules-6,ee0dbe705f727fe4,1bd255eb9e33d46a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:18:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:18:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:18:30.378 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:30.379 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:30.814 WARN 21062 --- [ocp-async-8,b0ca5d49c78f75ad,2b62407a7e0f5859] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:30.860 INFO 21062 --- [ocp-async-5,148662e03f3c8ff7,8e38d9ce9203e2fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:31.650 INFO 21062 --- [ocp-schedules-10,97fd111aed9b154a,8fa4cb118fd1e2db] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:18:31.650 INFO 21062 --- [ocp-schedules-10,5aa0ddaed68426bd,ec1caaa9a832f8cc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:18:31.654 INFO 21062 --- [ocp-schedules-10,5aa0ddaed68426bd,ec1caaa9a832f8cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:31.683 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:18:31.683 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:18:32.722 INFO 21062 --- [ocp-schedules-15,8ae83c6ea0403a9f,4335e208003a37bf] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:18:32.726 INFO 21062 --- [ocp-schedules-15,8ae83c6ea0403a9f,4335e208003a37bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:32.811 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:18:32.813 ERROR 21062 --- [ocp-schedules-15,8ae83c6ea0403a9f,4335e208003a37bf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:18:32.813 INFO 21062 --- [ocp-schedules-15,8ae83c6ea0403a9f,4335e208003a37bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:18:32.814 WARN 21062 --- [ocp-async-4,1790d0560aa49424,d35918bf59925929] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:32.860 INFO 21062 --- [ocp-async-7,28185e0af5407944,a6424dea3e141bdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:32.983 INFO 21062 --- [prometheus-minute-schedule-1,5ecf785088986cfa,589b16169452cffc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:33.379 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:33.380 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:34.814 WARN 21062 --- [ocp-async-2,5fe0f74fed97afb0,dc00098f93f338d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:34.861 INFO 21062 --- [ocp-async-6,deac365cae6daaad,820d75d965706332] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5a42f25937eeb499,db19b923ad25459d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:36.381 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:36.381 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:36.814 WARN 21062 --- [ocp-async-3,b41d281e51c9f4ce,372cad5a03b3c55e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:36.861 INFO 21062 --- [ocp-async-9,c551ecdd37b8d8f1,1ed4c223b1d8813a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:38.505 INFO 21062 --- [ocp-schedules-23,a04bc50ccfab8f88,9aa7639cc220e07f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:18:38.505 INFO 21062 --- [ocp-schedules-1,9f164e844ba0a489,c66eb9bb346213fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:18:38.588 INFO 21062 --- [ocp-async-8,bf0d5a6e5ebf4b58,2b9b037fdcc4e95a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:18:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:18:38.590879+08:00, lastProcessTime=2024-11-04T13:18:28.590917+08:00 2024-11-04 13:18:38.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:18:38.590879+08:00 2024-11-04 13:18:38.814 WARN 21062 --- [ocp-async-10,2797cbbf1864d3e6,4463c9628ef36b9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:38.861 INFO 21062 --- [ocp-async-2,38637ce0b7468e52,2e60fd98b3fdc3b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:38.983 INFO 21062 --- [prometheus-minute-schedule-1,d575b6837f480248,09ed74762ce1fa1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:39.171 INFO 21062 --- [ocp-schedules-28,73f0f1cc749608c6,a1f72d0de1e6aee4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:18:39.297 INFO 21062 --- [ocp-schedules-25,1a423ed8dcfbbc41,d4516569b34eaf96] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:18:39.381 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:39.382 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:18:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:18:40.815 WARN 21062 --- [ocp-async-1,7004df9b51359104,2d8e921b1397d1ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:40.861 INFO 21062 --- [ocp-async-3,bd8f66ab98357f1c,5aff0d3e7b5178a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:18:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:18:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:18:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:18:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:18:41.855 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:18:41.855 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:18:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d15f975c27c7fff0,850cabe5f7d885dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:42.382 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:42.383 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:42.815 WARN 21062 --- [ocp-async-9,57f1642f7e0335e3,a4c2ffd03dbdfdbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:42.861 INFO 21062 --- [ocp-async-5,bd51dce03328e1bf,21e85d2249fc907a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:44.815 WARN 21062 --- [ocp-async-4,433bb198c9ef8f6a,589bfdb452941f26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:44.862 INFO 21062 --- [ocp-async-7,6d742842fb4dff8e,e4ae145d8ec1d038] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e9d442db5bda7c69,71d58bac2b0a3a3b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:45.383 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:45.384 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:46.815 WARN 21062 --- [ocp-async-2,6561da2d7239af81,7e0ed2c17d8b3670] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:46.862 INFO 21062 --- [ocp-async-6,06018ce3f041aaa3,abf8e2006fb6f597] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2a85a54e0dc43475,9d4816cf8535de8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:48.385 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:48.386 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:48.588 INFO 21062 --- [ocp-async-2,d79c9fa58d46ecfe,ce21532b0d01d5a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:18:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:18:48.590608+08:00, lastProcessTime=2024-11-04T13:18:38.590879+08:00 2024-11-04 13:18:48.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:18:48.590608+08:00 2024-11-04 13:18:48.815 WARN 21062 --- [ocp-async-9,9f9b401b4cbeac2d,cd7d5e02e444208e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:48.862 INFO 21062 --- [ocp-async-5,20989520f024df33,c9f66063a486a2f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:49.214 INFO 21062 --- [ocp-schedules-29,4849057118fd7916,4a3185198adb8e02] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:18:49.299 INFO 21062 --- [ocp-schedules-3,8afd7079c62e2933,67e1814d713e5a8b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:18:49.800 INFO 21062 --- [ocp-schedules-16,2003140defae509e,f37aab1a31b12550] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:18:49.805 INFO 21062 --- [ocp-schedules-16,2003140defae509e,f37aab1a31b12550] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:18:49.805 INFO 21062 --- [ocp-schedules-16,2003140defae509e,f37aab1a31b12550] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:18:50.815 WARN 21062 --- [ocp-async-4,d24eb6f5894f3126,6c276bb6f4940f96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:50.862 INFO 21062 --- [ocp-async-7,394161618ff85763,b302cc158edb02cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4fb9dd9496d62a26,45c75332f89ff8ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:51.386 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:51.387 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:52.816 WARN 21062 --- [ocp-async-7,138d553d9ecc3d57,0871dd32a1af5a80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:52.862 INFO 21062 --- [ocp-async-6,93b3d634742c1163,e3f4d1663acf3013] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ab1f18c85fa65b56,d92efac97a855897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:54.387 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:54.388 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:54.816 WARN 21062 --- [ocp-async-6,e8a6629bcc2c1f7f,c670567d3b4c77fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:54.863 INFO 21062 --- [ocp-async-8,ed9eb403f4d34423,7f89949aee2cd6c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:56.816 WARN 21062 --- [ocp-async-8,d485f3b8bc7b08e9,3dc00815008f5324] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:56.863 INFO 21062 --- [ocp-async-10,e7525fa204395b2b,533b6136d984b241] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:56.983 INFO 21062 --- [prometheus-minute-schedule-1,963a161f58cee7bb,e6a28f285fa4bfe1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:18:57.388 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:18:57.389 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:18:58.500 INFO 21062 --- [ocp-schedules-2,bf80f393780e5a77,683bd0991a5a0c6c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:18:58.500 INFO 21062 --- [ocp-schedules-26,9feb02310034ca48,d453bbcf58a32764] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:18:58.502 INFO 21062 --- [ocp-schedules-30,f61ba189c1e6fef0,1f6543c463a3f3f1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:18:58.504 INFO 21062 --- [ocp-schedules-2,bf80f393780e5a77,683bd0991a5a0c6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:58.504 INFO 21062 --- [ocp-schedules-26,9feb02310034ca48,d453bbcf58a32764] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:58.505 INFO 21062 --- [ocp-schedules-22,9c0ba163f00b4348,62bb08cf4d4d848e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:18:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:18:58.511 INFO 21062 --- [ocp-schedules-20,3c909e7c65c627c0,4b54803702e23be5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:18:58.511 INFO 21062 --- [ocp-schedules-4,4cd6f64f05f5032c,4f506ad11b48a03e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:18:58.511 INFO 21062 --- [ocp-schedules-20,3c909e7c65c627c0,4b54803702e23be5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:18:58.511 INFO 21062 --- [ocp-schedules-4,4cd6f64f05f5032c,4f506ad11b48a03e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:18:58.511 INFO 21062 --- [ocp-schedules-19,43e6e4dc1fff3116,2f9665cc3958938b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:18:58.511 INFO 21062 --- [ocp-schedules-19,43e6e4dc1fff3116,2f9665cc3958938b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:18:58.588 INFO 21062 --- [ocp-async-4,e6776ab768ee5593,04049213851a4379] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:18:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:18:58.588892+08:00, lastProcessTime=2024-11-04T13:18:48.590608+08:00 2024-11-04 13:18:58.751 INFO 21062 --- [ocp-schedules-5,02923a40b12d851c,84979ddc67cb3bc6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:18:58.755 INFO 21062 --- [ocp-schedules-5,02923a40b12d851c,84979ddc67cb3bc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:18:58.756 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:18:58.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:18:58.588892+08:00 2024-11-04 13:18:58.816 INFO 21062 --- [ocp-schedules-13,0de2c29fbcd0b759,ba3ec9b33ae1d41f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:18:58.816 WARN 21062 --- [ocp-async-7,52ed66bad6f20c47,ff90a16e335b4ed7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:18:58.863 INFO 21062 --- [ocp-async-6,b8c64e72680b454f,1094f63d63bddcd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:18:59.238 INFO 21062 --- [ocp-schedules-22,343eaaeb12002c8a,7e03c85db97d46c7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:18:59.302 INFO 21062 --- [ocp-schedules-20,b8c8976f8c4c93d5,37671a3e8ab0bc61] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:18:59.983 INFO 21062 --- [prometheus-minute-schedule-1,16123168ea651c70,b0525b5dafe2899b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697540, interval=60 2024-11-04 13:19:00.389 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:00.390 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:00.816 WARN 21062 --- [ocp-async-6,f0427a08ab3bd932,c030c47339b0a3dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:00.863 INFO 21062 --- [ocp-async-8,539169ec4cdcfbf0,66001ec7f1fff14a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:01.684 INFO 21062 --- [ocp-schedules-10,5aa0ddaed68426bd,ec1caaa9a832f8cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:19:01.684 INFO 21062 --- [ocp-schedules-10,54174546ad941411,3572bf1b52edf698] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:19:01.688 INFO 21062 --- [ocp-schedules-10,54174546ad941411,3572bf1b52edf698] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:19:01.718 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:19:01.718 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:19:02.816 WARN 21062 --- [ocp-async-8,99911434ba03da83,668511b1b625e755] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:02.863 INFO 21062 --- [ocp-async-10,11997c2d3847a245,d5e67a2da87c5f30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:02.983 INFO 21062 --- [prometheus-minute-schedule-1,383efe11ea2e2761,3657c164d99b7b08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:03.390 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:03.391 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:04.817 WARN 21062 --- [ocp-async-10,dc3d19c9636aaf38,d9c9841efd887640] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:04.863 INFO 21062 --- [ocp-async-1,f72b4dbf5d6cb884,e9872470573d6cd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:05.983 INFO 21062 --- [prometheus-minute-schedule-1,b2d84f89659aadce,a300f1de6d8e5fe6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:06.391 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:06.392 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:06.817 WARN 21062 --- [ocp-async-1,13531b6ee20858ae,e81d1b9c2b862b37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:06.864 INFO 21062 --- [ocp-async-9,c41cfe7450fcbca5,90b1bf0e093c0e27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:08.505 INFO 21062 --- [ocp-schedules-26,9feb02310034ca48,d453bbcf58a32764] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:19:08.505 INFO 21062 --- [ocp-schedules-2,bf80f393780e5a77,683bd0991a5a0c6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:19:08.588 INFO 21062 --- [ocp-async-6,094d604269305122,764b84bc55c0cba2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:19:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:19:08.588892+08:00, lastProcessTime=2024-11-04T13:18:58.588892+08:00 2024-11-04 13:19:08.756 INFO 21062 --- [ocp-schedules-5,02923a40b12d851c,84979ddc67cb3bc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:19:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:19:08.588892+08:00 2024-11-04 13:19:08.817 WARN 21062 --- [ocp-async-8,51108b734a3a0a75,a60ad6b8d42ef452] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:08.864 INFO 21062 --- [ocp-async-10,b856f2624f7cd464,a7976fb82233f52c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:08.983 INFO 21062 --- [prometheus-minute-schedule-1,085b4b9061365802,2a71ac9149a3e871] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:09.280 INFO 21062 --- [ocp-schedules-20,358744a48714dfa7,7ed5069e27a3c8d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:19:09.304 INFO 21062 --- [ocp-schedules-22,2493014741c9e535,d7d8aced5f0f2f20] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:19:09.392 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:09.393 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:09.808 INFO 21062 --- [ocp-schedules-11,4640024d93f19f3f,ed45942fb8969259] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:19:09.814 INFO 21062 --- [ocp-schedules-11,4640024d93f19f3f,ed45942fb8969259] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:19:09.814 INFO 21062 --- [ocp-schedules-11,4640024d93f19f3f,ed45942fb8969259] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:19:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:19:10.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:19:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:19:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:19:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:19:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:19:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:19:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:19:10.817 WARN 21062 --- [ocp-async-10,b2ce148f55f4d207,f753a9fc1e579fc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:10.864 INFO 21062 --- [ocp-async-1,6077f99bd7fe5bcb,5ea91663170220f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=2, affectRows=2 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=2, validCount=2 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=14, affectRows=14 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=8, affectRows=8 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=30, affectRows=30 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=38, affectRows=38 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=8, validCount=8 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=14, validCount=14 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=29, validCount=29 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=58, affectRows=58 2024-11-04 13:19:11.218 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=59, affectRows=59 2024-11-04 13:19:11.219 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=38, validCount=38 2024-11-04 13:19:11.219 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=56, validCount=56 2024-11-04 13:19:11.219 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=59, validCount=59 2024-11-04 13:19:11.219 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=51, affectRows=51 2024-11-04 13:19:11.219 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697480, nodeCount=62, affectRows=62 2024-11-04 13:19:11.219 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=62, validCount=62 2024-11-04 13:19:11.219 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697480, nodeCount=51, validCount=51 2024-11-04 13:19:11.983 INFO 21062 --- [prometheus-minute-schedule-1,93ef6a20fae75fe9,47d410c3463891b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:12.393 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:12.394 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:12.817 WARN 21062 --- [ocp-async-1,059b2f6d19e47496,d095d8dd1d14555e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:12.864 INFO 21062 --- [ocp-async-9,a91647147d3dd68c,346ce0ecd98869d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:13.984 INFO 21062 --- [prometheus-cache-monitor1,2aedf9baa2438465,195d103e55c7e99e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:19:13.984 INFO 21062 --- [prometheus-cache-monitor1,2aedf9baa2438465,195d103e55c7e99e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:19:14.464 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:15:14.445, end=2024-11-04T13:17:14.462} 2024-11-04 13:19:14.469 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:19:14.469 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:19:14.817 WARN 21062 --- [ocp-async-9,b6f3acec9a418e5b,3ecf70be17428d94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:14.864 INFO 21062 --- [ocp-async-5,6baf94f785a7c509,77095a5ba4d907a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:14.983 INFO 21062 --- [prometheus-minute-schedule-1,93ef1f4d5439e38d,1716dae2444e5eab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:15.394 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:15.395 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:16.818 WARN 21062 --- [ocp-async-5,c27eb3a4ba006c41,0bd4b933ec2c6c9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:16.865 INFO 21062 --- [ocp-async-2,2ccb5fd82c3ee7f1,ccd5bcc0b7d89728] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4dde4e334b34063f,7dc0e46597a02c79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:18.395 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:18.396 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:18.588 INFO 21062 --- [ocp-async-10,f04ee455be175136,55ae6baf37c841e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:19:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:19:18.590991+08:00, lastProcessTime=2024-11-04T13:19:08.588892+08:00 2024-11-04 13:19:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:19:18.590991+08:00 2024-11-04 13:19:18.818 WARN 21062 --- [ocp-async-1,a8b1d2ad6c22c402,aeb67592dca14a47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:18.865 INFO 21062 --- [ocp-async-9,22c597821e27089b,eb0c8b2336ea4c38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:19.306 INFO 21062 --- [ocp-schedules-5,9623d95ad7236e6c,cd542b6a3fc0d077] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:19:19.306 INFO 21062 --- [ocp-schedules-6,eaeed9abe43001c8,5622c9dae1e3fa32] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:19:20.818 WARN 21062 --- [ocp-async-9,d7da5fabc4c0bc00,051e74ed0acf0d8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:20.865 INFO 21062 --- [ocp-async-4,24c56f8ef3938288,e8c898e713041816] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:20.983 INFO 21062 --- [prometheus-minute-schedule-1,fb49bb460a126b5f,71d73af23a031301] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:21.397 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:21.397 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:22.818 WARN 21062 --- [ocp-async-4,d80c401714324eed,69c8b41950c5e472] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:22.865 INFO 21062 --- [ocp-async-7,acf463bf3091ea3a,10572b88a6ebe99e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b1ffaf22006a3111,cc8be8e2305e0c5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:24.398 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:24.398 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:24.818 WARN 21062 --- [ocp-async-7,a5ed083837456ef0,c1f8bae3c515ac24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:24.865 INFO 21062 --- [ocp-async-6,bf5cdd143b034365,5aa5d16e58686b3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:26.818 WARN 21062 --- [ocp-async-6,88a584b34b08317f,51599000f689678e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:26.865 INFO 21062 --- [ocp-async-8,3ac098ba6afac7b1,2741e36ad9cbd735] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:26.983 INFO 21062 --- [prometheus-minute-schedule-1,5c42b68ef1c811f1,554343fee7d08648] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9ffd203c182819be,54d054e360501256] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:19:27.399 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:27.402 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:28.498 INFO 21062 --- [ocp-schedules-20,d8f5f1b5743ada3d,0e9c32235e4756fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:19:28.500 INFO 21062 --- [ocp-schedules-17,d89dffb65c53e1e4,509e944abc59d442] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:19:28.500 INFO 21062 --- [ocp-schedules-25,246944dc912ac887,18795622ea6874c9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:19:28.500 INFO 21062 --- [ocp-schedules-16,77b6588c162868d9,6a8e8ff73e399d7b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:19:28.502 INFO 21062 --- [ocp-schedules-18,e5d22bdf13a5f1e9,97dc91e67c31a703] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:19:28.503 INFO 21062 --- [ocp-schedules-20,d8f5f1b5743ada3d,0e9c32235e4756fa] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:19:28.503 INFO 21062 --- [ocp-schedules-17,d89dffb65c53e1e4,509e944abc59d442] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:19:28.504 INFO 21062 --- [ocp-schedules-16,77b6588c162868d9,6a8e8ff73e399d7b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:19:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:19:28.504 INFO 21062 --- [ocp-schedules-29,d4fc4258e7600ca9,84e195cda7b873bf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:19:28.505 INFO 21062 --- [ocp-schedules-25,246944dc912ac887,18795622ea6874c9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:19:28.510 INFO 21062 --- [ocp-schedules-14,5187b7db8886d78e,abd0061d2c0d19a5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:19:28.510 INFO 21062 --- [ocp-schedules-8,5d7cf21ba1819ed0,e1374628f868296f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:19:28.510 INFO 21062 --- [ocp-schedules-14,5187b7db8886d78e,abd0061d2c0d19a5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:19:28.510 INFO 21062 --- [ocp-schedules-8,5d7cf21ba1819ed0,e1374628f868296f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:19:28.510 INFO 21062 --- [ocp-schedules-30,ba4bff879ab0b1f4,c1b481ed880cb57c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:19:28.510 INFO 21062 --- [ocp-schedules-30,ba4bff879ab0b1f4,c1b481ed880cb57c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:19:28.550 INFO 21062 --- [ocp-schedules-23,19f9500d2aff608f,f72dcb9f91e60ecf] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:19:28.550+08:00, tenantSize=0 2024-11-04 13:19:28.589 INFO 21062 --- [ocp-async-8,731674b2c3836b1f,949e6daade68e106] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:19:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:19:28.589464+08:00, lastProcessTime=2024-11-04T13:19:18.590991+08:00 2024-11-04 13:19:28.674 INFO 21062 --- [ocp-schedules-19,cb5d5cb44a1d0b87,a6793bbf198f854f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:19:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:19:28.589464+08:00 2024-11-04 13:19:28.810 INFO 21062 --- [ocp-schedules-2,b37133dc643c7efe,b4321c2f6776d7be] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:19:28.810 INFO 21062 --- [ocp-schedules-2,b37133dc643c7efe,b4321c2f6776d7be] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:19:28.818 INFO 21062 --- [ocp-schedules-26,71ea14037d748d2e,da97d613d813d23c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:19:28.819 WARN 21062 --- [ocp-async-2,53495a6fbe4e1fa5,f9ce0034fa9caac7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:28.845 INFO 21062 --- [ocp-schedules-29,1375eef97618f492,167ea3b03399d5b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:19:28.850 INFO 21062 --- [ocp-schedules-29,1375eef97618f492,167ea3b03399d5b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:19:28.850 INFO 21062 --- [ocp-schedules-29,1375eef97618f492,167ea3b03399d5b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:19:28.859 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:19:28.859 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:19:28.866 INFO 21062 --- [ocp-async-1,24d3b01fc1c8aa27,5f6de22c44c2ef72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:28.867 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:19:28.867 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:19:28.867 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:19:28.877 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:19:28.877 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:19:28.877 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:19:28.880 INFO 21062 --- [ocp-schedules-18,bce23ceaabbd61eb,2dc6bf4321489c9e] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:19:29.007 INFO 21062 --- [ocp-schedules-11,39f3ceed6416d7b9,b0f7d138be7fef14] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:19:29.007 INFO 21062 --- [ocp-schedules-11,39f3ceed6416d7b9,b0f7d138be7fef14] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:19:29.008 INFO 21062 --- [ocp-schedules-11,39f3ceed6416d7b9,b0f7d138be7fef14] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:19:29.008 INFO 21062 --- [ocp-schedules-11,39f3ceed6416d7b9,b0f7d138be7fef14] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:19:29.009 INFO 21062 --- [ocp-schedules-11,39f3ceed6416d7b9,b0f7d138be7fef14] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:19:29.009 INFO 21062 --- [ocp-schedules-11,39f3ceed6416d7b9,b0f7d138be7fef14] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:19:29.310 INFO 21062 --- [ocp-schedules-21,b8a2dfb857d99769,a2b7697e7dae942c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:19:29.349 INFO 21062 --- [ocp-schedules-4,5fbcfdfe0ebabf62,8067cb50bc80846d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:19:29.813 INFO 21062 --- [ocp-schedules-2,a34da3d27aa12fc8,22a4b52d7d53a767] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:19:29.813 INFO 21062 --- [ocp-schedules-2,a34da3d27aa12fc8,22a4b52d7d53a767] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:19:29.817 INFO 21062 --- [ocp-schedules-26,21a5382f1ffc673f,0799ca53c0a4d724] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:19:29.823 INFO 21062 --- [ocp-schedules-26,21a5382f1ffc673f,0799ca53c0a4d724] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:19:29.823 INFO 21062 --- [ocp-schedules-26,21a5382f1ffc673f,0799ca53c0a4d724] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:19:29.840 INFO 21062 --- [ocp-schedules-2,a34da3d27aa12fc8,22a4b52d7d53a767] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:19:29.840 INFO 21062 --- [ocp-schedules-2,a34da3d27aa12fc8,22a4b52d7d53a767] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:19:29.843 INFO 21062 --- [ocp-schedules-25,c3b6c34f2c21dd19,8107b6383a57aff4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:19:29.947 INFO 21062 --- [ocp-schedules-14,1c92d624627fd9ea,c050e4c799aa0e30] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:19:29.947 INFO 21062 --- [ocp-schedules-14,1c92d624627fd9ea,c050e4c799aa0e30] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:19:29.947 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:19:29.947 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:19:29.957 INFO 21062 --- [ocp-schedules-14,1c92d624627fd9ea,c050e4c799aa0e30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:19:29.957 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:19:29.957 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:19:29.958 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:19:29.958 WARN 21062 --- [ocp-schedules-14,1c92d624627fd9ea,c050e4c799aa0e30] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:19:29.958 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:19:29.959 INFO 21062 --- [ocp-schedules-14,1c92d624627fd9ea,c050e4c799aa0e30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:19:29.959 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:19:29.959 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:19:29.961 WARN 21062 --- [ocp-schedules-14,1c92d624627fd9ea,c050e4c799aa0e30] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:19:29.961 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:19:29.961 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:19:29.961 INFO 21062 --- [ocp-schedules-8,6fcfcb6c9dd72281,ea84e2274e78591c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:19:29.961+08:00, clusters=[1] 2024-11-04 13:19:29.963 INFO 21062 --- [ocp-schedules-8,6fcfcb6c9dd72281,ea84e2274e78591c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:19:29.968 ERROR 21062 --- [ocp-schedules-14,1c92d624627fd9ea,c050e4c799aa0e30] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:19:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b8442e17f45ce51b,e3f7a78fdd3e3a43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:30.079 INFO 21062 --- [ocp-schedules-18,319044ae3d7b1306,ca4597d23c8259bc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:19:30.079+08:00, clusters=[1] 2024-11-04 13:19:30.082 INFO 21062 --- [ocp-schedules-18,319044ae3d7b1306,ca4597d23c8259bc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:19:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:19:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:19:30.403 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:30.403 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:30.819 WARN 21062 --- [ocp-async-3,063aa9a92d3c1695,f0bbd7cdbdcb795e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:30.866 INFO 21062 --- [ocp-async-6,2feda042daa93a94,150fbd9aca5c2d6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:31.719 INFO 21062 --- [ocp-schedules-10,54174546ad941411,3572bf1b52edf698] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:19:31.719 INFO 21062 --- [ocp-schedules-10,e0a6e004f1988f68,c565233f06af521c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:19:31.724 INFO 21062 --- [ocp-schedules-10,e0a6e004f1988f68,c565233f06af521c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:19:31.754 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:19:31.755 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:19:32.813 INFO 21062 --- [ocp-schedules-1,5118a353ce167fa0,fd8c98ae559fe823] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:19:32.818 INFO 21062 --- [ocp-schedules-1,5118a353ce167fa0,fd8c98ae559fe823] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:19:32.819 WARN 21062 --- [ocp-async-5,ff8b40920e2b37b3,de32395791580c1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:32.868 INFO 21062 --- [ocp-async-4,b7f3d342bde88b57,0d10342b984d9d20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:32.906 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:19:32.909 ERROR 21062 --- [ocp-schedules-1,5118a353ce167fa0,fd8c98ae559fe823] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:19:32.909 INFO 21062 --- [ocp-schedules-1,5118a353ce167fa0,fd8c98ae559fe823] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:19:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f232c2ae8ef40aa8,ccdac174e8a939ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:33.403 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:33.404 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:34.819 WARN 21062 --- [ocp-async-10,9a02292956d0b7e4,db76aaedd96628af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:34.868 INFO 21062 --- [ocp-async-7,e8b3559d7f010fd5,bb9a157de150a1c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:35.983 INFO 21062 --- [prometheus-minute-schedule-1,f9fc6b6392da0465,1c83ea550dbe4a65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:36.405 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:36.406 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:36.819 WARN 21062 --- [ocp-async-1,ecdf4fb63970a7d4,b25f56b04abe1520] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:36.868 INFO 21062 --- [ocp-async-9,fc35f8023580b6c5,049b41c9e8a72c3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:38.504 INFO 21062 --- [ocp-schedules-16,77b6588c162868d9,6a8e8ff73e399d7b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:19:38.504 INFO 21062 --- [ocp-schedules-17,d89dffb65c53e1e4,509e944abc59d442] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:19:38.589 INFO 21062 --- [ocp-async-3,8eb544734153e668,b60deb2f0267351a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:19:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:19:38.591205+08:00, lastProcessTime=2024-11-04T13:19:28.589464+08:00 2024-11-04 13:19:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:19:38.591205+08:00 2024-11-04 13:19:38.819 WARN 21062 --- [ocp-async-8,4705dd59600af94e,cf1704176e8338a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:38.868 INFO 21062 --- [ocp-async-10,e1a4d093ce656031,23d9185cf0ea0b87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ec4d723635da48a8,ee4ed7def0277975] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:39.312 INFO 21062 --- [ocp-schedules-23,1b73b5ddf2ee3333,64e21247eca053d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:19:39.374 INFO 21062 --- [ocp-schedules-17,1db7f0a45c3d4d71,8dd6ef40fb1ef0e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:19:39.406 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:39.407 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:19:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:19:40.820 WARN 21062 --- [ocp-async-2,bb81877e701a8b4a,e0cb87f225345f4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:40.868 INFO 21062 --- [ocp-async-1,228a14c9812c0ce7,eb16dd13f65f3c07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a75e1d2b9ff24bb8,20d48e891b66aeb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:42.407 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:42.409 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:42.820 WARN 21062 --- [ocp-async-9,6a24c015ceee8eac,10368b5225d9656b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:42.868 INFO 21062 --- [ocp-async-6,2d209f95cfa70fbe,eff17ac9f6022226] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:44.820 WARN 21062 --- [ocp-async-5,2df9d1a7578d7616,c395e1b626258503] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:44.869 INFO 21062 --- [ocp-async-4,b4c68c5460c6dc69,481847099342042b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:44.983 INFO 21062 --- [prometheus-minute-schedule-1,7ebc75a77ba44e05,945358f878c97942] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:45.409 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:45.410 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:46.820 WARN 21062 --- [ocp-async-10,8851295419307180,6551757898ee86cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:46.869 INFO 21062 --- [ocp-async-7,63f2611edcd65319,60e2e15da9f06ba3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:47.983 INFO 21062 --- [prometheus-minute-schedule-1,1e67669729c2ae4e,51fb7094836f747c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:48.410 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:48.411 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:48.589 INFO 21062 --- [ocp-async-10,4c65fb3fda8e8ec7,1fd26ecaf053f91c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:19:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:19:48.589720+08:00, lastProcessTime=2024-11-04T13:19:38.591205+08:00 2024-11-04 13:19:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:19:48.589720+08:00 2024-11-04 13:19:48.820 WARN 21062 --- [ocp-async-9,3700705ea64b7c7f,895dba805ec6ccf6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:48.869 INFO 21062 --- [ocp-async-6,047cd85571e296fe,6d01b78e215a40d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:49.315 INFO 21062 --- [ocp-schedules-9,ffd701271b1d2087,8c7de38199c20c3b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:19:49.419 INFO 21062 --- [ocp-schedules-26,1858566226b6d8a6,34ae421593c3d92f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:19:49.826 INFO 21062 --- [ocp-schedules-23,e578816397c414f6,280ff9e576a42b6b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:19:49.833 INFO 21062 --- [ocp-schedules-23,e578816397c414f6,280ff9e576a42b6b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:19:49.833 INFO 21062 --- [ocp-schedules-23,e578816397c414f6,280ff9e576a42b6b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:19:50.820 WARN 21062 --- [ocp-async-5,f943407ce7ece7d9,bebcddb563ffb2ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:50.869 INFO 21062 --- [ocp-async-4,668c54ca5be3e709,92aa2f779d86822d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:50.983 INFO 21062 --- [prometheus-minute-schedule-1,73aafdb1cc387f04,e277f3fa46ce5c29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:51.411 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:51.412 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:52.821 WARN 21062 --- [ocp-async-4,bdbc3a3e599ba013,a3d16453344945b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:52.869 INFO 21062 --- [ocp-async-7,b0afd23e19e52d89,ba3bf22c732671ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c288223d8db8a805,1bf42799891ea935] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:54.412 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:54.413 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:54.821 WARN 21062 --- [ocp-async-7,1df9dacde2e18fc0,c2b4052879ce2e64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:54.870 INFO 21062 --- [ocp-async-3,8e43c5d07ac2290a,7353144a77c93436] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:56.821 WARN 21062 --- [ocp-async-3,3009b0b45387610c,192a82ecd286f7ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:56.870 INFO 21062 --- [ocp-async-8,47c0a3d842bc59c3,993cabe93d54d8e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c7b0d6a8d37fcaa0,ad8af7896ebd4946] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:19:57.413 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:19:57.414 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:19:58.500 INFO 21062 --- [ocp-schedules-22,441a403bc7c40959,635f5bc6fab81352] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:19:58.500 INFO 21062 --- [ocp-schedules-3,a9e7b8b757ed1049,32472e8e8e65792e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:19:58.502 INFO 21062 --- [ocp-schedules-19,354d7e12e6927c35,719e7766a62039ca] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:19:58.504 INFO 21062 --- [ocp-schedules-22,441a403bc7c40959,635f5bc6fab81352] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:19:58.504 INFO 21062 --- [ocp-schedules-3,a9e7b8b757ed1049,32472e8e8e65792e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:19:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:19:58.505 INFO 21062 --- [ocp-schedules-9,174d2bd7bb28602e,ce31079c9c4efc9c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:19:58.512 INFO 21062 --- [ocp-schedules-11,0722bad13589677d,f48df156a9070091] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:19:58.512 INFO 21062 --- [ocp-schedules-11,0722bad13589677d,f48df156a9070091] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:19:58.512 INFO 21062 --- [ocp-schedules-20,f9feae8f9303e38e,20add342156a3ba7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:19:58.512 INFO 21062 --- [ocp-schedules-20,f9feae8f9303e38e,20add342156a3ba7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:19:58.513 INFO 21062 --- [ocp-schedules-5,237bf076377bcd05,70b56a517bc9d178] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:19:58.513 INFO 21062 --- [ocp-schedules-5,237bf076377bcd05,70b56a517bc9d178] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:19:58.589 INFO 21062 --- [ocp-async-5,95b3bb2a017e9cb6,634c487fa360d580] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:19:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:19:58.589519+08:00, lastProcessTime=2024-11-04T13:19:48.589720+08:00 2024-11-04 13:19:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:19:58.589519+08:00 2024-11-04 13:19:58.821 INFO 21062 --- [ocp-schedules-2,678bcea14850a831,23e0a5fdd061934a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:19:58.821 WARN 21062 --- [ocp-async-4,66a378cb6e168559,743d000862a06467] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:19:58.870 INFO 21062 --- [ocp-async-7,3c57811887a1d65b,d1a945a086a581ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:19:59.318 INFO 21062 --- [ocp-schedules-21,84b187901c68cac8,a24da761125ec2e7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:19:59.445 INFO 21062 --- [ocp-schedules-5,ed7f14dac0633e91,a0e2fd50a7bddf32] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:19:59.983 INFO 21062 --- [prometheus-minute-schedule-1,74c237a73820d60a,8df4d4f03d9585fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697600, interval=60 2024-11-04 13:20:00.371 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a4ff426acf7c6715] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:15:01.0, nextRun=2024-11-04 13:20:00.0, currentDate=Mon Nov 04 13:20:00 CST 2024 2024-11-04 13:20:00.371 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a4ff426acf7c6715] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:15:01.0, nextRunTime=2024-11-04 13:20:00.0} 2024-11-04 13:20:00.399 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a4ff426acf7c6715] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:20:00.404 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a4ff426acf7c6715] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:20:00.404 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a4ff426acf7c6715] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:20:00.414 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:00.415 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:00.821 WARN 21062 --- [ocp-async-7,f5f125ad3b3f0abe,287f2373ec8a14e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:00.870 INFO 21062 --- [ocp-async-3,005b92af2031f476,9b95c69ffbec8677] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:01.755 INFO 21062 --- [ocp-schedules-10,e0a6e004f1988f68,c565233f06af521c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:20:01.755 INFO 21062 --- [ocp-schedules-10,e3a1dd18db44aafe,53d72858a8cf748d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:20:01.759 INFO 21062 --- [ocp-schedules-10,e3a1dd18db44aafe,53d72858a8cf748d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:01.791 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:20:01.791 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:20:02.822 WARN 21062 --- [ocp-async-3,c54c3a380f420171,85cd2b42a952f884] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:02.870 INFO 21062 --- [ocp-async-8,232a9febf3469559,1959ea8128df250c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4c14ad72e93c9100,81b60b076044313d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:03.415 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:03.416 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:04.473 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a6a9c429af0f0038] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:10:05.0, nextRun=2024-11-04 13:20:04.0, currentDate=Mon Nov 04 13:20:04 CST 2024 2024-11-04 13:20:04.473 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a6a9c429af0f0038] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 13:10:05.0, nextRunTime=2024-11-04 13:20:04.0} 2024-11-04 13:20:04.478 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a6a9c429af0f0038] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 13:20:04.478 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a6a9c429af0f0038] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T13:20:04.478+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 13:20:04.822 WARN 21062 --- [ocp-async-8,0ed2fb6541364198,7174e45267ab34fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:04.870 INFO 21062 --- [ocp-async-2,4b2e77c239ad0f57,383b64ae441004a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:05.288 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,6c2b545c81ffb9b9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 73, operation:EXECUTE, state: RUNNING 2024-11-04 13:20:05.309 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=73, context=Context{parallelIdx=-1, stringMap={task_instance_id=8, task_operation=execute, latest_execution_start_time=2024-11-04T13:20:05.280+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=73}, listMap={}}, executor=10.46.128.24 2024-11-04 13:20:05.314 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:20:05.314 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:20:05.316 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.316 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:20:05.317 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.318 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.318 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.319 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:20:05.335 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.335 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.336 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.336 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:20:05.348 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.349 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.349 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.350 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:20:05.363 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.364 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.364 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.365 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:20:05.377 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.378 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.378 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.379 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:20:05.385 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.385 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.386 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.386 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:20:05.390 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.390 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.391 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.391 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:20:05.396 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.396 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.397 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.397 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:20:05.403 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.403 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.403 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.404 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:20:05.409 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.410 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.410 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.411 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:20:05.416 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.417 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.417 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.418 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:20:05.426 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.426 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.427 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.427 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:20:05.440 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.441 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.442 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.442 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:20:05.447 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.447 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.448 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.448 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:20:05.454 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.454 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.455 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.455 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:20:05.462 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.462 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.463 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.463 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:20:05.468 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.468 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.469 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.469 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:20:05.473 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.474 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.475 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.475 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:20:05.479 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.480 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.480 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.480 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:20:05.486 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.486 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.486 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.487 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:20:05.491 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.491 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.491 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.492 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:20:05.497 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.497 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.498 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.498 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:20:05.502 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.502 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.503 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.503 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:20:05.508 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.508 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.509 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.509 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:20:05.513 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.513 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.514 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.515 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:20:05.519 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.519 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.519 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.520 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:20:05.524 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.525 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.525 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.526 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:20:05.530 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.531 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.531 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.532 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:20:05.536 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.536 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.537 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.537 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:20:05.542 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.543 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.543 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.544 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:20:05.548 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.548 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.549 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.549 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:20:05.554 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.555 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.555 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.556 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:20:05.560 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.560 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.561 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.561 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:20:05.565 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.566 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.566 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.567 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:20:05.573 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.574 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.574 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.575 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:20:05.581 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.581 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.582 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.582 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:20:05.588 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.589 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.589 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.590 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:20:05.596 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.596 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.597 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.597 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:20:05.601 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.601 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.602 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.602 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:20:05.606 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.607 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.608 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.608 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:20:05.612 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.612 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.613 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.613 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:20:05.617 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.617 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.618 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.618 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:20:05.627 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.628 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.629 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.629 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:20:05.649 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.650 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.650 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.651 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:20:05.657 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.657 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.658 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.658 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:20:05.664 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.664 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.665 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.665 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:20:05.671 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.672 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.672 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.673 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:20:05.677 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.677 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.678 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.678 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:20:05.682 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.683 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.683 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.684 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:20:05.688 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:20:05.688 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:20:05.689 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.689 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:20:05.691 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.691 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.691 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.692 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:20:05.697 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.698 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:20:05.698 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:20:05.699 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:20:05.703 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:20:05.711 INFO 21062 --- [subtask-executor6,dd193f8f67a5a423,b4c7cd64ccdb8933] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 73, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 13:20:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e0ff2cacbae13ded,258439fa4c629bed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:06.416 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:06.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:06.630 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8, status:SUCCESS 2024-11-04 13:20:06.631 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:8, status:SUCCESS 2024-11-04 13:20:06.822 WARN 21062 --- [ocp-async-2,5d7d8589793e2f5e,8d8a97c9f9562c68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:06.871 INFO 21062 --- [ocp-async-9,4ca6bc7e215519b1,5fa170aecb36b740] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:08.505 INFO 21062 --- [ocp-schedules-3,a9e7b8b757ed1049,32472e8e8e65792e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:20:08.506 INFO 21062 --- [ocp-schedules-22,441a403bc7c40959,635f5bc6fab81352] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:20:08.590 INFO 21062 --- [ocp-async-7,ac8e13f04ec61754,90a3466ea45b8009] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:20:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:20:08.590063+08:00, lastProcessTime=2024-11-04T13:19:58.589519+08:00 2024-11-04 13:20:08.756 INFO 21062 --- [ocp-schedules-12,4d5118a1ed141aa4,3ef9d354dc0fa951] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:20:08.762 INFO 21062 --- [ocp-schedules-12,4d5118a1ed141aa4,3ef9d354dc0fa951] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:08.763 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:20:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:20:08.590063+08:00 2024-11-04 13:20:08.822 WARN 21062 --- [ocp-async-3,ac66d33238cc661c,9d8ad7f3acab95e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:08.871 INFO 21062 --- [ocp-async-8,4445d9bed051974d,59f27c9993c485c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:08.983 INFO 21062 --- [prometheus-minute-schedule-1,24811e303ab6b09a,df024cb5e4c22f5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:09.322 INFO 21062 --- [ocp-schedules-27,77db652713e509b4,9c7cf25d8b490ee0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:20:09.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:09.419 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:09.489 INFO 21062 --- [ocp-schedules-15,37341e2284c298cb,55f51578a14a6499] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:20:09.836 INFO 21062 --- [ocp-schedules-21,8b03d8d3954bd0b8,c3d02cd4b25a3b16] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:20:09.843 INFO 21062 --- [ocp-schedules-21,8b03d8d3954bd0b8,c3d02cd4b25a3b16] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:20:09.843 INFO 21062 --- [ocp-schedules-21,8b03d8d3954bd0b8,c3d02cd4b25a3b16] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:20:10.002 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:20:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:20:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:20:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:20:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:20:10.333 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:20:10.333 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:20:10.334 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:20:10.822 WARN 21062 --- [ocp-async-8,0f92410b9c99d83c,97da498b8d7ea3a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:10.871 INFO 21062 --- [ocp-async-2,4b0f74f1e3ec338e,c868fdee59b10f1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:11.221 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=3, affectRows=3 2024-11-04 13:20:11.221 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=3, validCount=3 2024-11-04 13:20:11.221 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=13, affectRows=13 2024-11-04 13:20:11.222 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=13, validCount=13 2024-11-04 13:20:11.222 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=27, affectRows=27 2024-11-04 13:20:11.222 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=27, validCount=27 2024-11-04 13:20:11.222 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=41, affectRows=41 2024-11-04 13:20:11.222 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=71, affectRows=71 2024-11-04 13:20:11.222 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=49, affectRows=49 2024-11-04 13:20:11.223 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=41, validCount=41 2024-11-04 13:20:11.223 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=29, affectRows=29 2024-11-04 13:20:11.223 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=49, validCount=49 2024-11-04 13:20:11.223 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=29, validCount=29 2024-11-04 13:20:11.223 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=69, validCount=69 2024-11-04 13:20:11.223 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697540, nodeCount=89, affectRows=89 2024-11-04 13:20:11.223 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697540, nodeCount=88, validCount=88 2024-11-04 13:20:11.983 INFO 21062 --- [prometheus-minute-schedule-1,ec3c1b87956f1b03,dc401fdd8b55bee1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:12.419 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:12.420 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:12.823 WARN 21062 --- [ocp-async-2,4bffb04f7e0ff513,8e7e5904ee37efbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:12.871 INFO 21062 --- [ocp-async-9,07f3f1d12c56920d,76a1b848822391ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:13.984 INFO 21062 --- [prometheus-cache-monitor1,e5bcc48fa1c49325,9f248281a776d06c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:20:13.985 INFO 21062 --- [prometheus-cache-monitor1,e5bcc48fa1c49325,9f248281a776d06c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:20:14.823 WARN 21062 --- [ocp-async-9,43103ae1391127b4,638128537ca1222a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:14.871 INFO 21062 --- [ocp-async-6,bd1d73ef9788145e,0288c6919a012f8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:14.983 INFO 21062 --- [prometheus-minute-schedule-1,24b5624bcd15c106,d91f1a87ff2bc0cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:15.420 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:15.421 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:16.823 WARN 21062 --- [ocp-async-6,3a74708926f64ad5,db59ceb029e5187b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:16.872 INFO 21062 --- [ocp-async-10,52a2a48a865efcff,e68442ef175a6bb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:17.983 INFO 21062 --- [prometheus-minute-schedule-1,e4aee0562010086b,8782c25af36508f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:18.421 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:18.422 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:18.590 INFO 21062 --- [ocp-async-8,c526c7f721e42a7f,aa273e0cb5186778] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:20:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:20:18.589777+08:00, lastProcessTime=2024-11-04T13:20:08.590063+08:00 2024-11-04 13:20:18.763 INFO 21062 --- [ocp-schedules-12,4d5118a1ed141aa4,3ef9d354dc0fa951] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:20:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:20:18.589777+08:00 2024-11-04 13:20:18.823 WARN 21062 --- [ocp-async-2,e87f86198c74161f,99fbfb95bf64b1a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:18.872 INFO 21062 --- [ocp-async-9,bd680fe179a50d15,ad39d85fe010a2f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:19.325 INFO 21062 --- [ocp-schedules-29,b1150acdce7753eb,c392342ba1039e56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:20:19.515 INFO 21062 --- [ocp-schedules-1,37b7100f2bd0e376,27186ebf7cfcfe8b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:20:20.823 WARN 21062 --- [ocp-async-9,4daf458a7573cd48,a917a9b42f2f0207] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:20.872 INFO 21062 --- [ocp-async-5,6c3879b65d56368d,490c47b635d03ad6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5f634e21c305b053,d3d37758b7616712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:21.422 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:21.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:22.823 WARN 21062 --- [ocp-async-5,ad4be011f86b7af4,01007a9c720bd726] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:22.872 INFO 21062 --- [ocp-async-4,7bcb99be11d48f6e,9c19fccafb85cc24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:23.983 INFO 21062 --- [prometheus-minute-schedule-1,7f846f518bc87f90,f21137c998a85240] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:24.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:24.425 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:24.824 WARN 21062 --- [ocp-async-4,ec1aea39c97f32aa,73e46a1a54f5a152] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:24.872 INFO 21062 --- [ocp-async-7,e4526a8f2d26c205,ef62b1cca25c0f2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:26.824 WARN 21062 --- [ocp-async-7,3fd1c5bbf7f9af59,d5539d3984db92c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:26.872 INFO 21062 --- [ocp-async-3,cd290d2b01de6acd,6fe0a848a64e365c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2b2e51de01d47a59,fce190acc4e30973] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a17049e1ee7a3096,74f07321a02509f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:20:27.425 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:27.426 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:28.498 INFO 21062 --- [ocp-schedules-6,1626066a6508c9b8,9bec446bb119da1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:20:28.499 INFO 21062 --- [ocp-schedules-23,c5723491ff642c50,7f79c178248a4566] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:20:28.499 INFO 21062 --- [ocp-schedules-16,ef4bd155bcb881da,45784cb260b140a3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:20:28.500 INFO 21062 --- [ocp-schedules-14,cfe4bf93b1a9f545,a09f6b8f2046ea11] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:20:28.502 INFO 21062 --- [ocp-schedules-6,1626066a6508c9b8,9bec446bb119da1f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:20:28.502 INFO 21062 --- [ocp-schedules-18,a3e0e399e3b874b9,a477e2e5bef3d438] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:20:28.503 INFO 21062 --- [ocp-schedules-14,cfe4bf93b1a9f545,a09f6b8f2046ea11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:28.503 INFO 21062 --- [ocp-schedules-16,ef4bd155bcb881da,45784cb260b140a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:20:28.505 INFO 21062 --- [ocp-schedules-23,c5723491ff642c50,7f79c178248a4566] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:20:28.505 INFO 21062 --- [ocp-schedules-28,206be39e7b598d0e,8ad39d3334e55693] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:20:28.510 INFO 21062 --- [ocp-schedules-22,f40c3f2ad9c16e89,264c6d4c83121de0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:20:28.510 INFO 21062 --- [ocp-schedules-26,6be8f8f41e9b8bd6,ed48817131ad4583] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:20:28.510 INFO 21062 --- [ocp-schedules-26,6be8f8f41e9b8bd6,ed48817131ad4583] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:20:28.510 INFO 21062 --- [ocp-schedules-22,f40c3f2ad9c16e89,264c6d4c83121de0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:20:28.511 INFO 21062 --- [ocp-schedules-12,aedd4a1604d7c6c3,379ba210426b5fb5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:20:28.511 INFO 21062 --- [ocp-schedules-12,aedd4a1604d7c6c3,379ba210426b5fb5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:20:28.564 INFO 21062 --- [ocp-schedules-20,e61e81cf795d747e,ef9d72ccf1aba576] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:20:28.564 INFO 21062 --- [ocp-schedules-20,e61e81cf795d747e,ef9d72ccf1aba576] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:20:28.590 INFO 21062 --- [ocp-async-6,9e8f96a71d72215b,152ce711613855cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:20:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:20:28.589904+08:00, lastProcessTime=2024-11-04T13:20:18.589777+08:00 2024-11-04 13:20:28.676 INFO 21062 --- [ocp-schedules-9,d40df5ab74641091,6c1f1317603b7ce4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:20:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:20:28.589904+08:00 2024-11-04 13:20:28.814 INFO 21062 --- [ocp-schedules-8,5230f864f28ed488,17fef0beb717c5ca] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:20:28.814 INFO 21062 --- [ocp-schedules-8,5230f864f28ed488,17fef0beb717c5ca] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:20:28.824 INFO 21062 --- [ocp-schedules-17,f920f308324e292a,4004510c73812140] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:20:28.824 WARN 21062 --- [ocp-async-10,8efea19f261d5cea,4b937d3e23eb20dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:28.851 INFO 21062 --- [ocp-schedules-18,260a7dadfdc9cc44,137d6717bdbc182d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:20:28.855 INFO 21062 --- [ocp-schedules-18,260a7dadfdc9cc44,137d6717bdbc182d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:20:28.855 INFO 21062 --- [ocp-schedules-18,260a7dadfdc9cc44,137d6717bdbc182d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:20:28.872 INFO 21062 --- [ocp-async-2,0445461c6243561e,2206f1a3420a12b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:29.010 INFO 21062 --- [ocp-schedules-22,849225037486c0ba,c0714dbe8449771a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:20:29.010 INFO 21062 --- [ocp-schedules-22,849225037486c0ba,c0714dbe8449771a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:20:29.011 INFO 21062 --- [ocp-schedules-22,849225037486c0ba,c0714dbe8449771a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:20:29.011 INFO 21062 --- [ocp-schedules-22,849225037486c0ba,c0714dbe8449771a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:20:29.012 INFO 21062 --- [ocp-schedules-22,849225037486c0ba,c0714dbe8449771a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:20:29.012 INFO 21062 --- [ocp-schedules-22,849225037486c0ba,c0714dbe8449771a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:20:29.328 INFO 21062 --- [ocp-schedules-29,4bd4efdd41a332e8,6775b471ce89a8bf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:20:29.554 INFO 21062 --- [ocp-schedules-4,8a5a766eda2e1dc5,4bd8244f174731d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:20:29.841 INFO 21062 --- [ocp-schedules-25,8c5ab186edcfc197,53265822a1a63c56] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:20:29.841 INFO 21062 --- [ocp-schedules-25,8c5ab186edcfc197,53265822a1a63c56] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:20:29.846 INFO 21062 --- [ocp-schedules-7,98f639b1c0f979c0,abfdce618c0b9037] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:20:29.852 INFO 21062 --- [ocp-schedules-7,98f639b1c0f979c0,abfdce618c0b9037] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:20:29.852 INFO 21062 --- [ocp-schedules-7,98f639b1c0f979c0,abfdce618c0b9037] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:20:29.868 INFO 21062 --- [ocp-schedules-25,8c5ab186edcfc197,53265822a1a63c56] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:20:29.868 INFO 21062 --- [ocp-schedules-25,8c5ab186edcfc197,53265822a1a63c56] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:20:29.870 INFO 21062 --- [ocp-schedules-17,9b79742eb2a28b81,c03370a0a35458d6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:20:29.974 INFO 21062 --- [ocp-schedules-26,35dcde5cfccd6477,eb574228137b8a77] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:20:29.977 INFO 21062 --- [ocp-schedules-18,f4d1f3fd62dbaa34,e6ab6d97df54ea80] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:20:29.977 INFO 21062 --- [ocp-schedules-18,f4d1f3fd62dbaa34,e6ab6d97df54ea80] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:20:29.977 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:20:29.977 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:20:29.977 INFO 21062 --- [ocp-schedules-26,35dcde5cfccd6477,eb574228137b8a77] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7b458bf114631c53,97ace31335ba314b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:29.987 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:20:29.987 INFO 21062 --- [ocp-schedules-18,f4d1f3fd62dbaa34,e6ab6d97df54ea80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:20:29.987 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:20:29.989 WARN 21062 --- [ocp-schedules-18,f4d1f3fd62dbaa34,e6ab6d97df54ea80] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:20:29.989 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:20:29.989 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:20:29.989 INFO 21062 --- [ocp-schedules-23,ab41b47b01f389d9,29c13a98d17a3bf9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:20:29.989+08:00, clusters=[1] 2024-11-04 13:20:29.989 INFO 21062 --- [ocp-schedules-18,f4d1f3fd62dbaa34,e6ab6d97df54ea80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:20:29.989 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:20:29.989 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:20:29.990 INFO 21062 --- [ocp-schedules-23,ab41b47b01f389d9,29c13a98d17a3bf9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:20:29.991 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:20:29.991 WARN 21062 --- [ocp-schedules-18,f4d1f3fd62dbaa34,e6ab6d97df54ea80] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:20:29.991 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:20:30.002 ERROR 21062 --- [ocp-schedules-18,f4d1f3fd62dbaa34,e6ab6d97df54ea80] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:20:30.038 ERROR 21062 --- [ocp-schedules-26,35dcde5cfccd6477,eb574228137b8a77] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:20:30.038 INFO 21062 --- [ocp-schedules-26,35dcde5cfccd6477,eb574228137b8a77] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:20:30.109 INFO 21062 --- [ocp-schedules-6,a3ec087392a6f861,7dc4ded3482205ff] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:20:30.109+08:00, clusters=[1] 2024-11-04 13:20:30.112 INFO 21062 --- [ocp-schedules-6,a3ec087392a6f861,7dc4ded3482205ff] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:20:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:20:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:20:30.426 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:30.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:30.824 WARN 21062 --- [ocp-async-1,d7539c5378cf24b8,aa89bbfeaff07e2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:30.873 INFO 21062 --- [ocp-async-7,73ed9cba5335378f,86bb1cec715e9caf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:31.792 INFO 21062 --- [ocp-schedules-10,e3a1dd18db44aafe,53d72858a8cf748d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:20:31.792 INFO 21062 --- [ocp-schedules-10,0e5c92700ef27ae8,a2f496ea1dc333c1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:20:31.796 INFO 21062 --- [ocp-schedules-10,0e5c92700ef27ae8,a2f496ea1dc333c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:31.825 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:20:31.825 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:20:32.825 WARN 21062 --- [ocp-async-3,2cb3e282245f36cd,3e36663804bf1d67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:32.873 INFO 21062 --- [ocp-async-5,aeef09d0baad0089,2c63ea211bb74c93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:32.909 INFO 21062 --- [ocp-schedules-3,c97df29fb2fec075,e4f8a3ec0e3053dc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:20:32.912 INFO 21062 --- [ocp-schedules-3,c97df29fb2fec075,e4f8a3ec0e3053dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:32.983 INFO 21062 --- [prometheus-minute-schedule-1,2c8b1b08511639da,bcaee7e71b0d7971] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:32.999 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:20:33.001 ERROR 21062 --- [ocp-schedules-3,c97df29fb2fec075,e4f8a3ec0e3053dc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:20:33.001 INFO 21062 --- [ocp-schedules-3,c97df29fb2fec075,e4f8a3ec0e3053dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:20:33.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:33.428 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:34.825 WARN 21062 --- [ocp-async-8,3fcf734b7ab3eb1a,22b3256a273a8365] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:34.873 INFO 21062 --- [ocp-async-4,a97a0b7024a1a698,5d882bb47684df6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:35.983 INFO 21062 --- [prometheus-minute-schedule-1,757eea21ba1bd0f8,2a06a71cc7ca04dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:36.428 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:36.430 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:36.825 WARN 21062 --- [ocp-async-2,5e4171b32e96d649,39dd1e21a62804b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:36.873 INFO 21062 --- [ocp-async-9,a4db9120eb47e05b,ee2f01e38d0fc4cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:38.504 INFO 21062 --- [ocp-schedules-14,cfe4bf93b1a9f545,a09f6b8f2046ea11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:20:38.505 INFO 21062 --- [ocp-schedules-16,ef4bd155bcb881da,45784cb260b140a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:20:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:20:38.589697+08:00, lastProcessTime=2024-11-04T13:20:28.589904+08:00 2024-11-04 13:20:38.590 INFO 21062 --- [ocp-async-1,f8f4a1f8b3542bdb,b2101acbeac51023] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:20:38.799 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:20:38.589697+08:00 2024-11-04 13:20:38.825 WARN 21062 --- [ocp-async-6,944af6fb524556f8,9a38f069d3d4bff7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:38.873 INFO 21062 --- [ocp-async-8,ea62eebeb385d58e,b208e7f04ee6daad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:38.983 INFO 21062 --- [prometheus-minute-schedule-1,597bd540904f0eb0,dd2b66841a112fb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:39.331 INFO 21062 --- [ocp-schedules-15,a87f507e816c98d5,d1c5e07f2a7a222e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:20:39.430 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:39.431 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:39.580 INFO 21062 --- [ocp-schedules-9,6a3f0dcbfb9f408c,00fc0e3115b79c8d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:20:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:20:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:20:40.825 WARN 21062 --- [ocp-async-10,becfab27158426f9,ca847a685aa9577c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:40.874 INFO 21062 --- [ocp-async-2,90d5f6a367bd36b6,1b8a5f4943e49912] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:20:41.826 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:20:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-11-04 13:20:41.983 INFO 21062 --- [prometheus-minute-schedule-1,73d26c7e793b3c0b,c27d1b8af6150149] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:42.431 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:42.432 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:42.825 WARN 21062 --- [ocp-async-1,7052bfbba32c2cc9,a0b8e80780bff250] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:42.874 INFO 21062 --- [ocp-async-7,89e0399ec86c6c4f,b6c22f8163b59847] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:44.826 WARN 21062 --- [ocp-async-3,5938c04f62278a89,e3285f23c7f6d358] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:44.874 INFO 21062 --- [ocp-async-5,57cba253612b41d9,37d0eab186c3dfa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:44.983 INFO 21062 --- [prometheus-minute-schedule-1,72401fb2eeef786f,c223a2ade86ab4d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:45.432 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:45.433 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:46.826 WARN 21062 --- [ocp-async-8,46027a6fb1f84192,a5406a620bc661c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:46.874 INFO 21062 --- [ocp-async-4,f42254f78e9a5fbf,55eb14b65cc6babc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:47.983 INFO 21062 --- [prometheus-minute-schedule-1,50fd3be8e0a493b4,09448728649de5fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:48.433 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:48.435 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:48.590 INFO 21062 --- [ocp-async-8,571e5a97ee0f10ea,9e0ef53fdeeb9c34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:20:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:20:48.590911+08:00, lastProcessTime=2024-11-04T13:20:38.589697+08:00 2024-11-04 13:20:48.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:20:48.590911+08:00 2024-11-04 13:20:48.826 WARN 21062 --- [ocp-async-1,efbfe75502e7e7cf,18c65ca49b331b47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:48.874 INFO 21062 --- [ocp-async-7,cb475684635a6881,884b9f19d68c9cb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:49.334 INFO 21062 --- [ocp-schedules-22,6c467136b756912e,bfb5938b22234045] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:20:49.623 INFO 21062 --- [ocp-schedules-30,e7d075967161bec5,451d51027fcc77fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:20:49.855 INFO 21062 --- [ocp-schedules-18,a06c95f3ab43d39e,4e460240ee07f38e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:20:49.862 INFO 21062 --- [ocp-schedules-18,a06c95f3ab43d39e,4e460240ee07f38e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:20:49.862 INFO 21062 --- [ocp-schedules-18,a06c95f3ab43d39e,4e460240ee07f38e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:20:50.826 WARN 21062 --- [ocp-async-3,72c3816a96d0d2c8,c6eb427c0aaaed03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:50.875 INFO 21062 --- [ocp-async-5,6d72093d190ccc0c,c6a992508a9a43c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:50.983 INFO 21062 --- [prometheus-minute-schedule-1,47a3a088037c0762,b8b5944550d397bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:51.435 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:51.436 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:52.826 WARN 21062 --- [ocp-async-5,5e865cf9b6a0bb49,4aad689cb563bb0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:52.875 INFO 21062 --- [ocp-async-4,2f22d043d9a848a0,11a12a0a18d6656c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d32192e7873b4d85,7081fb3538451656] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:54.436 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:54.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:54.826 WARN 21062 --- [ocp-async-4,857847ec6052de3b,c251c30844f908fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:54.875 INFO 21062 --- [ocp-async-9,d6e95e041e8f5cdb,8a9b7dcc52d0dfe4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:56.827 WARN 21062 --- [ocp-async-9,4b421fd34fb12891,76c864de3cbb8d6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:56.875 INFO 21062 --- [ocp-async-6,2ca747799c8ca3cf,31c9fd46df575152] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fb200c6b4fd4c4ad,637c8dcb792b4af9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:20:57.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:20:57.438 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:20:58.500 INFO 21062 --- [ocp-schedules-17,04066486ab54bd8d,5b0b10814f5e0696] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:20:58.500 INFO 21062 --- [ocp-schedules-2,b3633215448652ac,b0839f6b66462423] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:20:58.502 INFO 21062 --- [ocp-schedules-13,cdd6c18410b0014b,c5003e8404cbe97d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:20:58.504 INFO 21062 --- [ocp-schedules-2,b3633215448652ac,b0839f6b66462423] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:58.504 INFO 21062 --- [ocp-schedules-17,04066486ab54bd8d,5b0b10814f5e0696] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:20:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:20:58.505 INFO 21062 --- [ocp-schedules-14,dd39c15120203715,ce9bdc4619c531df] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:20:58.510 INFO 21062 --- [ocp-schedules-7,833bc777a61d3a0c,030b45e84c9efcce] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:20:58.510 INFO 21062 --- [ocp-schedules-30,6ec079178efb2794,5835d53214f6c3bb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:20:58.510 INFO 21062 --- [ocp-schedules-29,a57d33dbb815edd0,2cea349e80547258] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:20:58.510 INFO 21062 --- [ocp-schedules-30,6ec079178efb2794,5835d53214f6c3bb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:20:58.510 INFO 21062 --- [ocp-schedules-7,833bc777a61d3a0c,030b45e84c9efcce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:20:58.510 INFO 21062 --- [ocp-schedules-29,a57d33dbb815edd0,2cea349e80547258] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:20:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:20:58.588901+08:00, lastProcessTime=2024-11-04T13:20:48.590911+08:00 2024-11-04 13:20:58.591 INFO 21062 --- [ocp-async-3,3b0854e0db9dc88d,78184fa34ea7fb77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:20:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:20:58.588901+08:00 2024-11-04 13:20:58.827 INFO 21062 --- [ocp-schedules-25,321e17a975005384,f6c57f1864d0fa76] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:20:58.827 WARN 21062 --- [ocp-async-5,7ebb6ccbba219234,8eec9ecf15dec95b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:20:58.875 INFO 21062 --- [ocp-async-4,2f49b4abab10cecb,666592c34e776f9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:20:59.337 INFO 21062 --- [ocp-schedules-22,c95bf1c43fd197b2,7e64bb6df441975b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:20:59.647 INFO 21062 --- [ocp-schedules-1,05597a0c486a6766,4dda6407392cd5a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:20:59.983 INFO 21062 --- [prometheus-minute-schedule-1,71ada082459600ea,d9737d8bd903916f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697660, interval=60 2024-11-04 13:21:00.438 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:00.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:00.827 WARN 21062 --- [ocp-async-4,83a43dc9206bad73,47ef1e24a5f62739] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:00.876 INFO 21062 --- [ocp-async-9,dcd94d5efae05aec,8938a1b53a573bd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:00.894 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:21:00.894 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:21:00.902 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:21:00.902 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:21:00.902 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:21:00.902 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:21:00.908 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:21:00.908 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:21:00.908 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:21:00.908 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:21:00.946 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:21:00.955 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:21:00.955 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:21:00.955 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:21:00.955 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:21:00.955 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:21:00.963 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:21:01.494 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:21:01.825 INFO 21062 --- [ocp-schedules-10,0e5c92700ef27ae8,a2f496ea1dc333c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:21:01.825 INFO 21062 --- [ocp-schedules-10,de241e2ca1923cdf,62225b214dc8157b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:21:01.829 INFO 21062 --- [ocp-schedules-10,de241e2ca1923cdf,62225b214dc8157b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:01.858 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:21:01.858 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:21:02.083 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.17504169S 2024-11-04 13:21:02.083 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:21:02.185 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:21:02.185 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:21:02.218 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:21:02.305 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:21:02.305 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:21:02.305 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:21:02.305 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:21:02.305 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:21:02.306 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:21:02.317 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:21:02.489 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] o.s.boot.SpringApplication : Started application in 1.587 seconds (JVM running for 3076.168) 2024-11-04 13:21:02.489 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.587426659S 2024-11-04 13:21:02.827 WARN 21062 --- [ocp-async-9,538442be0e55d82b,76e48de9907a3ffa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:02.876 INFO 21062 --- [ocp-async-6,1d239980d700d489,31aff3d41912f139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:02.939 INFO 21062 --- [ocp-schedules-13,fa9df33f008ee6a2,906a7ef83b3b514f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:21:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7f1d92b2644d86b7,985dc059d90d1629] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:03.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:03.440 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:04.827 WARN 21062 --- [ocp-async-6,c2aa44b4dc7b6509,b7df1db60a6027fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:04.876 INFO 21062 --- [ocp-async-10,736a536eabd62146,ad6595cef4d71198] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f2a4d32c61f9feba,cdd7b17bd0d69a1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:06.440 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:06.441 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:06.827 WARN 21062 --- [ocp-async-10,e24f28695f0c18e4,30ac79c4320370fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:06.876 INFO 21062 --- [ocp-async-2,3327718347c5e54d,6d14539793146ffc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:08.505 INFO 21062 --- [ocp-schedules-2,b3633215448652ac,b0839f6b66462423] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:21:08.505 INFO 21062 --- [ocp-schedules-17,04066486ab54bd8d,5b0b10814f5e0696] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:21:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:21:08.590559+08:00, lastProcessTime=2024-11-04T13:20:58.588901+08:00 2024-11-04 13:21:08.591 INFO 21062 --- [ocp-async-4,9d22d5faa1d87240,679fe4ef9351b24d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:21:08.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:21:08.590559+08:00 2024-11-04 13:21:08.828 WARN 21062 --- [ocp-async-9,17f46d1cfc7dbc3a,7931d125a1252605] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:08.876 INFO 21062 --- [ocp-async-7,a1e923b301c3479a,063a5bcfcc7bbcd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f81cd664d5b9ca96,846544c0bde6c292] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:09.341 INFO 21062 --- [ocp-schedules-27,634597de990464c3,a029653679bef4c1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:21:09.441 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:09.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:09.692 INFO 21062 --- [ocp-schedules-4,d0aacea35832aee1,dd39bdb49fc00b85] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:21:09.865 INFO 21062 --- [ocp-schedules-29,51558e659802e884,c313e03067629bea] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:21:09.872 INFO 21062 --- [ocp-schedules-29,51558e659802e884,c313e03067629bea] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:21:09.872 INFO 21062 --- [ocp-schedules-29,51558e659802e884,c313e03067629bea] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:21:10.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:21:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:21:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:21:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:21:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:21:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:21:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:21:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:21:10.828 WARN 21062 --- [ocp-async-7,ff87a7c56877f96e,3e384bb8806cf0e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:10.877 INFO 21062 --- [ocp-async-8,ddb799e6be49391d,690257770af2a8d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=23, affectRows=23 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=33, affectRows=33 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=41, affectRows=41 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=23, validCount=23 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=10, affectRows=10 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=10, validCount=10 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=52, affectRows=52 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=9, affectRows=9 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=33, validCount=33 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=39, validCount=39 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=9, validCount=9 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=19, affectRows=19 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=11, affectRows=11 2024-11-04 13:21:11.225 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=5, affectRows=5 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=52, affectRows=52 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=5, validCount=5 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=11, validCount=11 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=19, validCount=19 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=23, affectRows=23 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=51, validCount=51 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=23, validCount=23 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=52, validCount=52 2024-11-04 13:21:11.226 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697600, nodeCount=44, affectRows=44 2024-11-04 13:21:11.227 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697600, nodeCount=44, validCount=44 2024-11-04 13:21:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8946663e6b3a017b,423502cc3e8e345c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:12.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:12.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:12.828 WARN 21062 --- [ocp-async-8,1a19a2b5031846de,25727b4341ef82ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:12.877 INFO 21062 --- [ocp-async-1,de5a08a93643383a,2de706b7c3518e69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:13.984 INFO 21062 --- [prometheus-cache-monitor1,50e2e836498a9f6c,5db670dddac112bf] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:21:13.984 INFO 21062 --- [prometheus-cache-monitor1,50e2e836498a9f6c,5db670dddac112bf] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:21:14.480 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:17:14.462, end=2024-11-04T13:19:14.477} 2024-11-04 13:21:14.489 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:21:14.489 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:21:14.828 WARN 21062 --- [ocp-async-1,92ec676589449e3f,e5a05d284cb27ae4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:14.877 INFO 21062 --- [ocp-async-3,533a5a859dcdef8b,791936cbd5388d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:14.983 INFO 21062 --- [prometheus-minute-schedule-1,509e35fcdd3d9af7,ce93cc62d46173bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:15.443 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:15.444 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:16.828 WARN 21062 --- [ocp-async-3,f88f155d22a9eedd,2675626451b5113f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:16.877 INFO 21062 --- [ocp-async-5,ba95996c22d04c2d,5b003311862afdf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:17.983 INFO 21062 --- [prometheus-minute-schedule-1,431f10e98db9fa24,44038d9302f7b939] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:18.444 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:18.445 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:21:18.589944+08:00, lastProcessTime=2024-11-04T13:21:08.590559+08:00 2024-11-04 13:21:18.591 INFO 21062 --- [ocp-async-3,d976805e07b6c8d5,cb88d90885319285] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:21:18.763 INFO 21062 --- [ocp-schedules-21,9a83a9c2aef1b146,eeb66a58feb17549] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:21:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:21:18.589944+08:00 2024-11-04 13:21:18.766 INFO 21062 --- [ocp-schedules-21,9a83a9c2aef1b146,eeb66a58feb17549] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:18.767 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:21:18.829 WARN 21062 --- [ocp-async-8,bf31b74485b0dede,ba115172b8bd17c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:18.877 INFO 21062 --- [ocp-async-2,5b868f225bd01ef0,4b1a586c7b4035fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:19.344 INFO 21062 --- [ocp-schedules-14,3a6ca2abb85dd4a1,21cf61eef536e147] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:21:19.720 INFO 21062 --- [ocp-schedules-20,ebaa1cd2ead5e193,fdeb6408ef658cf3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:21:20.829 WARN 21062 --- [ocp-async-1,d557a9ab755050d6,29ba375b1eaf2460] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:20.878 INFO 21062 --- [ocp-async-6,fb69d1263b6f2bc2,11fd5fc0547cc486] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5a63360010d07d0e,fbc0ddcceed038ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:21.445 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:21.446 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:22.829 WARN 21062 --- [ocp-async-7,99fd579d653c600b,65c5a4bc07c5f279] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:22.878 INFO 21062 --- [ocp-async-5,be40cba62691fa4d,7479f48d491cc577] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:23.983 INFO 21062 --- [prometheus-minute-schedule-1,017fe7bdfb0ebd32,7278ee8f11382fd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:24.446 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:24.447 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:24.829 WARN 21062 --- [ocp-async-10,f91d4204e0c91fff,9c16a4b82e575aaf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:24.878 INFO 21062 --- [ocp-async-4,26f084078bb5ba1f,cdd18fa33892dbff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:26.829 WARN 21062 --- [ocp-async-2,e85792a4581f2229,145d667198e09723] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:26.878 INFO 21062 --- [ocp-async-9,d371adeae238f21f,329df1aa0bfe1fa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:26.983 INFO 21062 --- [prometheus-minute-schedule-1,42057e816d8640f0,3a7cc04a0bf0df1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a503db3f10930da4,a3b03f0774f48093] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:21:27.447 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:27.448 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:28.498 INFO 21062 --- [ocp-schedules-25,af334149fe4e0707,af7dfebb80f1d069] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:21:28.499 INFO 21062 --- [ocp-schedules-28,10ec7eec25589b61,8b687e36525e71da] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:21:28.500 INFO 21062 --- [ocp-schedules-19,76b86647d01a1ac7,42a964cbc95fc1b0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:21:28.500 INFO 21062 --- [ocp-schedules-24,341a4c5509d8b572,6e21b555b539e8ea] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:21:28.502 INFO 21062 --- [ocp-schedules-2,2ad741aa84b0fea2,ad84f91e8d7773b1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:21:28.503 INFO 21062 --- [ocp-schedules-25,af334149fe4e0707,af7dfebb80f1d069] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:21:28.504 INFO 21062 --- [ocp-schedules-19,76b86647d01a1ac7,42a964cbc95fc1b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:28.504 INFO 21062 --- [ocp-schedules-24,341a4c5509d8b572,6e21b555b539e8ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:21:28.504 INFO 21062 --- [ocp-schedules-28,10ec7eec25589b61,8b687e36525e71da] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:21:28.505 INFO 21062 --- [ocp-schedules-17,e9bf47d0611a934a,205b260b57ff2cb9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:21:28.510 INFO 21062 --- [ocp-schedules-5,423f772becb2625a,ba629be882d846af] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:21:28.510 INFO 21062 --- [ocp-schedules-5,423f772becb2625a,ba629be882d846af] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:21:28.510 INFO 21062 --- [ocp-schedules-14,5fbec7288b3c2017,78a48a6e1d60560c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:21:28.510 INFO 21062 --- [ocp-schedules-30,104b7e04bb89768c,0567eb03e98514c5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:21:28.510 INFO 21062 --- [ocp-schedules-14,5fbec7288b3c2017,78a48a6e1d60560c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:21:28.510 INFO 21062 --- [ocp-schedules-30,104b7e04bb89768c,0567eb03e98514c5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:21:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:21:28.589727+08:00, lastProcessTime=2024-11-04T13:21:18.589944+08:00 2024-11-04 13:21:28.591 INFO 21062 --- [ocp-async-9,e0939f08f0952849,64c688e1bd9de0da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:21:28.593 INFO 21062 --- [ocp-schedules-18,b988db179f529892,3988c2a408a60b6a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:21:28.594 INFO 21062 --- [ocp-schedules-18,b988db179f529892,3988c2a408a60b6a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:21:28.678 INFO 21062 --- [ocp-schedules-27,683d895e8f592c51,de82d22b8294d740] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:21:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:21:28.589727+08:00 2024-11-04 13:21:28.767 INFO 21062 --- [ocp-schedules-21,9a83a9c2aef1b146,eeb66a58feb17549] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:21:28.818 INFO 21062 --- [ocp-schedules-26,ebc6398881061bf7,058eba667c284051] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:21:28.818 INFO 21062 --- [ocp-schedules-26,ebc6398881061bf7,058eba667c284051] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:21:28.829 INFO 21062 --- [ocp-schedules-7,9652774ec2577c73,7dd3a4927d6d4707] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:21:28.829 WARN 21062 --- [ocp-async-5,157b7bf3af42ce74,5a82599a4d3c0985] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:28.856 INFO 21062 --- [ocp-schedules-17,76192dea067a1175,febeb4441a3ef92d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:21:28.861 INFO 21062 --- [ocp-schedules-17,76192dea067a1175,febeb4441a3ef92d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:21:28.861 INFO 21062 --- [ocp-schedules-17,76192dea067a1175,febeb4441a3ef92d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:21:28.878 INFO 21062 --- [ocp-async-8,9c93ac1945608622,2aa749cf2962f5cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:29.013 INFO 21062 --- [ocp-schedules-8,2993f20a061ab3f1,2bafa41a93214e18] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:21:29.013 INFO 21062 --- [ocp-schedules-8,2993f20a061ab3f1,2bafa41a93214e18] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:21:29.019 INFO 21062 --- [ocp-schedules-8,2993f20a061ab3f1,2bafa41a93214e18] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:21:29.019 INFO 21062 --- [ocp-schedules-8,2993f20a061ab3f1,2bafa41a93214e18] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:21:29.025 INFO 21062 --- [ocp-schedules-8,2993f20a061ab3f1,2bafa41a93214e18] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:21:29.025 INFO 21062 --- [ocp-schedules-8,2993f20a061ab3f1,2bafa41a93214e18] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:21:29.347 INFO 21062 --- [ocp-schedules-23,41b621b420613724,278e482a874dda6c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:21:29.758 INFO 21062 --- [ocp-schedules-21,005721521065fa87,93e8ccd35d1ee89d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:21:29.868 INFO 21062 --- [ocp-schedules-26,079d84681d9dd255,608efe44f39fab9d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:21:29.868 INFO 21062 --- [ocp-schedules-26,079d84681d9dd255,608efe44f39fab9d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:21:29.874 INFO 21062 --- [ocp-schedules-28,dc0a1b7090731af9,c1195809d2577d21] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:21:29.880 INFO 21062 --- [ocp-schedules-28,dc0a1b7090731af9,c1195809d2577d21] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:21:29.880 INFO 21062 --- [ocp-schedules-28,dc0a1b7090731af9,c1195809d2577d21] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:21:29.895 INFO 21062 --- [ocp-schedules-26,079d84681d9dd255,608efe44f39fab9d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:21:29.895 INFO 21062 --- [ocp-schedules-26,079d84681d9dd255,608efe44f39fab9d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:21:29.898 INFO 21062 --- [ocp-schedules-7,889aa0012b189f78,4a5ae2f1368eea9b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:21:29.983 INFO 21062 --- [prometheus-minute-schedule-1,8f8e5c4e1e1b2d29,e283b9cd54a327ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:30.010 INFO 21062 --- [ocp-schedules-14,b766602b34da9f29,c9e88e1aee9aec55] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:21:30.010 INFO 21062 --- [ocp-schedules-14,b766602b34da9f29,c9e88e1aee9aec55] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:21:30.010 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:21:30.010 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:21:30.017 INFO 21062 --- [ocp-schedules-17,02c52ec6b7a28b1e,01e2a94edaedb1a8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:21:30.017+08:00, clusters=[1] 2024-11-04 13:21:30.018 INFO 21062 --- [ocp-schedules-17,02c52ec6b7a28b1e,01e2a94edaedb1a8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:21:30.020 INFO 21062 --- [ocp-schedules-14,b766602b34da9f29,c9e88e1aee9aec55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:21:30.020 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:21:30.020 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:21:30.022 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:21:30.022 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:21:30.022 WARN 21062 --- [ocp-schedules-14,b766602b34da9f29,c9e88e1aee9aec55] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:21:30.022 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:21:30.022 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:21:30.023 INFO 21062 --- [ocp-schedules-14,b766602b34da9f29,c9e88e1aee9aec55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:21:30.024 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:21:30.024 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:21:30.024 WARN 21062 --- [ocp-schedules-14,b766602b34da9f29,c9e88e1aee9aec55] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:21:30.032 ERROR 21062 --- [ocp-schedules-14,b766602b34da9f29,c9e88e1aee9aec55] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:21:30.139 INFO 21062 --- [ocp-schedules-30,8ad17a03dbd6d80d,3ddf13eab2187540] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:21:30.139+08:00, clusters=[1] 2024-11-04 13:21:30.142 INFO 21062 --- [ocp-schedules-30,8ad17a03dbd6d80d,3ddf13eab2187540] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:21:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:21:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:21:30.448 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:30.450 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:30.830 WARN 21062 --- [ocp-async-2,60456ff5cd34c115,be79bd51d9b90867] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:30.878 INFO 21062 --- [ocp-async-1,4c9294b816407ff7,f7b6ddef03f62720] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:31.858 INFO 21062 --- [ocp-schedules-10,de241e2ca1923cdf,62225b214dc8157b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:21:31.859 INFO 21062 --- [ocp-schedules-10,4b3fe13f6357afac,d8367b80dad6c40f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:21:31.862 INFO 21062 --- [ocp-schedules-10,4b3fe13f6357afac,d8367b80dad6c40f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:31.890 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:21:31.890 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:21:32.830 WARN 21062 --- [ocp-async-9,7503f399762189c2,7a191f5a8dce2cac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:32.879 INFO 21062 --- [ocp-async-7,b7c491f32ccb8501,a95d62539e7e211d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8af7a28b628e36da,7ec146c2c67f551c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:33.002 INFO 21062 --- [ocp-schedules-18,cb8ddb5546e834d6,764bf533448e04ae] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:21:33.005 INFO 21062 --- [ocp-schedules-18,cb8ddb5546e834d6,764bf533448e04ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:33.089 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:21:33.092 ERROR 21062 --- [ocp-schedules-18,cb8ddb5546e834d6,764bf533448e04ae] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:21:33.092 INFO 21062 --- [ocp-schedules-18,cb8ddb5546e834d6,764bf533448e04ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:21:33.450 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:33.451 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:34.830 WARN 21062 --- [ocp-async-3,4b2b8d259036e2d5,b4ede10ded4f887f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:34.879 INFO 21062 --- [ocp-async-10,fa671fed5dee65c0,eeca112aea33c2a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c6c89e7300ad91d1,c1cfe77c7e6cd5ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:36.451 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:36.452 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:36.830 WARN 21062 --- [ocp-async-8,4eb350a4eae7b407,34c4b4f96450d40b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:36.879 INFO 21062 --- [ocp-async-4,e650658f444be856,13f985c7e647b1e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:38.505 INFO 21062 --- [ocp-schedules-19,76b86647d01a1ac7,42a964cbc95fc1b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:21:38.505 INFO 21062 --- [ocp-schedules-24,341a4c5509d8b572,6e21b555b539e8ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:21:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:21:38.589722+08:00, lastProcessTime=2024-11-04T13:21:28.589727+08:00 2024-11-04 13:21:38.591 INFO 21062 --- [ocp-async-2,95c5d7f81b121583,3ecc26a9e7a0e6cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:21:38.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:21:38.589722+08:00 2024-11-04 13:21:38.830 WARN 21062 --- [ocp-async-9,fec212f90c58bf8e,74fbf3db7ab0be42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:38.879 INFO 21062 --- [ocp-async-3,500ca24542d00dab,f8f995ca0b0c9b45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c9f3535ec8574d1a,3385d13017401991] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:39.350 INFO 21062 --- [ocp-schedules-23,fc78e09848003d18,a6c20dd50ac92dfe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:21:39.452 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:39.453 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:39.784 INFO 21062 --- [ocp-schedules-25,90bd47cb9b1ede70,9280325bee972d57] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:21:40.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:21:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:21:40.830 WARN 21062 --- [ocp-async-3,ebbb7956db821a8e,327b725c81e2c00a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:40.880 INFO 21062 --- [ocp-async-10,74a51c7422a585da,5188f6342696445f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bf8b313d1131ca04,882c8e98731d67cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:42.453 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:42.454 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:42.831 WARN 21062 --- [ocp-async-10,b5e65f4d65bf7208,091465612a1d4455] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:42.880 INFO 21062 --- [ocp-async-1,7e59cb5fa148a572,e3dae067bb61c331] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:44.831 WARN 21062 --- [ocp-async-1,aac4a22a66663c5c,628bc9c8f941b0d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:44.880 INFO 21062 --- [ocp-async-7,a28e01813f5eb25e,257c6fc1c7fa0f95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:44.983 INFO 21062 --- [prometheus-minute-schedule-1,79138c0587edb4c1,364990e3b1dcaa30] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:45.454 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:45.455 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:46.831 WARN 21062 --- [ocp-async-7,52134572667bcdee,13710bc4d537cab9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:46.880 INFO 21062 --- [ocp-async-8,d6377a749a1f700a,fe780371a921ff80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:47.983 INFO 21062 --- [prometheus-minute-schedule-1,75176ea983d0c936,e8f4ee2b9e8a3f29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:48.456 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:48.457 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:48.591 INFO 21062 --- [ocp-async-3,8e596c8bce054bf5,d8ff8382d203b11d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:21:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:21:48.591164+08:00, lastProcessTime=2024-11-04T13:21:38.589722+08:00 2024-11-04 13:21:48.775 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:21:48.591164+08:00 2024-11-04 13:21:48.831 WARN 21062 --- [ocp-async-10,30232b5a47406264,95d7a71e9267e15e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:48.880 INFO 21062 --- [ocp-async-1,3bdc94ff4130a5b2,f4698d15bb200e15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:49.353 INFO 21062 --- [ocp-schedules-30,7fbedaa39ea5f4d1,6017a58fd8f88220] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:21:49.826 INFO 21062 --- [ocp-schedules-18,ec77275d5c7aac62,5630dc06e7daf85c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:21:49.883 INFO 21062 --- [ocp-schedules-27,9f320caaafea5dc3,79c2d55b2c025ae6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:21:49.889 INFO 21062 --- [ocp-schedules-27,9f320caaafea5dc3,79c2d55b2c025ae6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:21:49.889 INFO 21062 --- [ocp-schedules-27,9f320caaafea5dc3,79c2d55b2c025ae6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:21:50.832 WARN 21062 --- [ocp-async-1,b2cd69f56e92d81b,a411074e434a2cf5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:50.880 INFO 21062 --- [ocp-async-7,603be1eb343b4086,7db4b255b0f9299c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:50.983 INFO 21062 --- [prometheus-minute-schedule-1,9c9a10bdd8ed0df2,76b696f73d086896] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:51.457 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:51.458 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:52.832 WARN 21062 --- [ocp-async-7,472fd8322776345e,01f88f8d268d946e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:52.880 INFO 21062 --- [ocp-async-8,001ccace4a78af99,29f1a13f96530b72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:53.983 INFO 21062 --- [prometheus-minute-schedule-1,fa54abd69808278a,b66de6b5b909a0a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:54.458 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:54.459 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:54.832 WARN 21062 --- [ocp-async-8,8112b6f5e41161e5,051b4be354840ffc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:54.881 INFO 21062 --- [ocp-async-2,ad381c2c14db39e2,684b7043a78551c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:56.832 WARN 21062 --- [ocp-async-2,af14ade2c865de5d,d92478beff151cce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:56.881 INFO 21062 --- [ocp-async-6,fc429a2c7578248b,b4ba6e2805256a8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:56.983 INFO 21062 --- [prometheus-minute-schedule-1,a6d05f633bb7f70b,6f09efaa79a19559] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:21:57.459 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:21:57.460 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:21:58.500 INFO 21062 --- [ocp-schedules-14,42e9384d107b6be5,c6c58f5788813009] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:21:58.500 INFO 21062 --- [ocp-schedules-2,4d8ad794b35e9d5f,262c0ac93e0c34c3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:21:58.503 INFO 21062 --- [ocp-schedules-11,77712494d654ef2b,6de9c9a56df8b116] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:21:58.504 INFO 21062 --- [ocp-schedules-14,42e9384d107b6be5,c6c58f5788813009] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:58.504 INFO 21062 --- [ocp-schedules-2,4d8ad794b35e9d5f,262c0ac93e0c34c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:21:58.504 INFO 21062 --- [ocp-schedules-25,c4a521ac433860e4,92aa706289603aa9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:21:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:21:58.510 INFO 21062 --- [ocp-schedules-20,f6b917d095736979,e5ff0adbc3d1799f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:21:58.510 INFO 21062 --- [ocp-schedules-16,fcd12ea9781f38de,9a8c990d5d8d87f3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:21:58.510 INFO 21062 --- [ocp-schedules-24,2093d5e8230499af,d73b231ac38a37b4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:21:58.510 INFO 21062 --- [ocp-schedules-20,f6b917d095736979,e5ff0adbc3d1799f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:21:58.510 INFO 21062 --- [ocp-schedules-16,fcd12ea9781f38de,9a8c990d5d8d87f3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:21:58.510 INFO 21062 --- [ocp-schedules-24,2093d5e8230499af,d73b231ac38a37b4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:21:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:21:58.590260+08:00, lastProcessTime=2024-11-04T13:21:48.591164+08:00 2024-11-04 13:21:58.592 INFO 21062 --- [ocp-async-1,b7d6b90ece93670f,c993efd793ddd14c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:21:58.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:21:58.590260+08:00 2024-11-04 13:21:58.831 INFO 21062 --- [ocp-schedules-28,48167978a333514a,bd195f2aca546fc8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:21:58.832 WARN 21062 --- [ocp-async-7,22bb143f4526a48c,9a5b232c1d42fed1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:21:58.881 INFO 21062 --- [ocp-async-8,396502c104a0ea99,3e26076fe4f837ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:21:59.357 INFO 21062 --- [ocp-schedules-6,18a6021a1acf047f,190df8df3b65d73b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:21:59.853 INFO 21062 --- [ocp-schedules-24,283912201057c7d3,53e16a08c3120911] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:21:59.983 INFO 21062 --- [prometheus-minute-schedule-1,3ec06754c78e98d5,0527cbc65e423d02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697720, interval=60 2024-11-04 13:22:00.460 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:00.461 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:00.832 WARN 21062 --- [ocp-async-8,f6897a3f0826aa25,b3b99d777b926437] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:00.881 INFO 21062 --- [ocp-async-2,681da197dcbe5bf4,a8ec99efb69db4ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:01.891 INFO 21062 --- [ocp-schedules-10,4b3fe13f6357afac,d8367b80dad6c40f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:22:01.891 INFO 21062 --- [ocp-schedules-10,ea963cad02f146a0,00766e17ca927d51] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:22:01.896 INFO 21062 --- [ocp-schedules-10,ea963cad02f146a0,00766e17ca927d51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:01.926 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:22:01.926 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:22:02.833 WARN 21062 --- [ocp-async-2,aa535339442c6444,dc6d42b29ec64759] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:02.881 INFO 21062 --- [ocp-async-6,b1f665ac88d13441,2cb9f06d6ae50532] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4dd9a9613a12cdb7,3f572891623e2980] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:03.461 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:03.462 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:04.833 WARN 21062 --- [ocp-async-6,17a442018c8c1beb,1f54722661a3b20d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:04.881 INFO 21062 --- [ocp-async-5,4da584f314a4ebfd,8989253b66042946] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:05.983 INFO 21062 --- [prometheus-minute-schedule-1,752f0a3a113a0fe3,7be964ace4ad6a45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:06.462 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:06.463 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:06.833 WARN 21062 --- [ocp-async-5,f08f429022d799d7,24f9d5afc477e78b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:06.882 INFO 21062 --- [ocp-async-4,fb303018774c322f,af077c895e9f1f24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:08.505 INFO 21062 --- [ocp-schedules-2,4d8ad794b35e9d5f,262c0ac93e0c34c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:22:08.505 INFO 21062 --- [ocp-schedules-14,42e9384d107b6be5,c6c58f5788813009] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:22:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:22:08.590956+08:00, lastProcessTime=2024-11-04T13:21:58.590260+08:00 2024-11-04 13:22:08.592 INFO 21062 --- [ocp-async-8,8e0eccc6e8f4dc2d,612e5752039c8d6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:22:08.787 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:22:08.590956+08:00 2024-11-04 13:22:08.833 WARN 21062 --- [ocp-async-2,7e3bfd84584b74c4,6fbe79cd2d9bb897] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:08.882 INFO 21062 --- [ocp-async-9,139d44147a43362c,78974af2f43aed3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:08.983 INFO 21062 --- [prometheus-minute-schedule-1,fd987c590b072977,f1a076b7e75586e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:09.360 INFO 21062 --- [ocp-schedules-29,c7cce24a3d3448e3,e7ae675344bfaf65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:22:09.463 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:09.464 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:09.892 INFO 21062 --- [ocp-schedules-22,d5f8e61842cfb913,21900142d5d09c1b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:22:09.898 INFO 21062 --- [ocp-schedules-22,d5f8e61842cfb913,21900142d5d09c1b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:22:09.898 INFO 21062 --- [ocp-schedules-22,d5f8e61842cfb913,21900142d5d09c1b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:22:09.931 INFO 21062 --- [ocp-schedules-15,5efe9d6861e752eb,b8c64998af8fa260] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:22:10.003 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:22:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:22:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:22:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:22:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:22:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:22:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:22:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:22:10.833 WARN 21062 --- [ocp-async-9,5ac81164b9387f2d,508d953c364b2a26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:10.882 INFO 21062 --- [ocp-async-3,5a6ba8df919edf24,8f26919a7016cd40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=3, affectRows=3 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=3, validCount=3 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=35, affectRows=35 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=13, affectRows=13 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=47, affectRows=47 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=10, affectRows=10 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=13, validCount=13 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=10, validCount=10 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=35, validCount=35 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=63, affectRows=63 2024-11-04 13:22:11.229 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=38, affectRows=38 2024-11-04 13:22:11.230 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=47, validCount=47 2024-11-04 13:22:11.230 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=38, validCount=38 2024-11-04 13:22:11.230 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=86, affectRows=86 2024-11-04 13:22:11.230 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=62, validCount=62 2024-11-04 13:22:11.230 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=85, validCount=85 2024-11-04 13:22:11.231 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697660, nodeCount=27, affectRows=27 2024-11-04 13:22:11.231 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697660, nodeCount=26, validCount=26 2024-11-04 13:22:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b088b77c8db92901,765b29bd61175154] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:12.464 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:12.465 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:12.833 WARN 21062 --- [ocp-async-3,125e41cfb3ba2b52,445edf2d8ca16881] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:12.882 INFO 21062 --- [ocp-async-10,d6022dc30bcc33b1,50f5f3a1a3889989] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:13.984 INFO 21062 --- [prometheus-cache-monitor1,440e5b50bc5f01c6,bee280d6ce7c54ee] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:22:13.985 INFO 21062 --- [prometheus-cache-monitor1,440e5b50bc5f01c6,bee280d6ce7c54ee] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:22:14.834 WARN 21062 --- [ocp-async-10,212855a349c18d84,61db6574e5a00d2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:14.882 INFO 21062 --- [ocp-async-1,5afe2500193ad13c,a0f28c17ac36498f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:14.983 INFO 21062 --- [prometheus-minute-schedule-1,24242d563a460d84,be529ef5cd470284] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:15.465 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:15.466 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:16.834 WARN 21062 --- [ocp-async-1,5fbb6471c0da8088,08e6c6715e58864b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:16.883 INFO 21062 --- [ocp-async-7,314a5b535a7bf2d4,21d77695a47f2e3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:17.983 INFO 21062 --- [prometheus-minute-schedule-1,0661639a81afe979,20df1d0c42a1bc2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:18.466 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:18.467 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:22:18.591007+08:00, lastProcessTime=2024-11-04T13:22:08.590956+08:00 2024-11-04 13:22:18.592 INFO 21062 --- [ocp-async-1,32358506878251ab,082b9d717e4b98dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:22:18.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:22:18.591007+08:00 2024-11-04 13:22:18.834 WARN 21062 --- [ocp-async-3,26898e8af81bbabd,334b64b07cfa330d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:18.883 INFO 21062 --- [ocp-async-4,7d536f1afa60620b,44df48796060b1d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:19.363 INFO 21062 --- [ocp-schedules-5,6cc1b3b50a9813ca,311a28dd66683d1f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:22:19.956 INFO 21062 --- [ocp-schedules-16,4e7abe3759e842ca,30c72f7b23ce944a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:22:20.834 WARN 21062 --- [ocp-async-10,412700abbfb225b3,109f61433277f51f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:20.883 INFO 21062 --- [ocp-async-6,e4bfebb4af4341ad,b9e5e14fb7c1dc23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4a85b0b5046e3768,d6b60cf3047cea5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:21.467 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:21.468 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:22.834 WARN 21062 --- [ocp-async-9,879bb84da7b4cfea,ceb6320a46c92709] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:22.883 INFO 21062 --- [ocp-async-7,b4248c72b325beb2,6bb87893fd1d748f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c94ed273193771aa,097ad38d1e648c54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:24.468 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:24.470 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:24.834 WARN 21062 --- [ocp-async-5,bb96f155f39fd517,fae9fd9c085616db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:24.883 INFO 21062 --- [ocp-async-8,abcf6b464548a6f8,9e58d75fab92bc91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:26.834 WARN 21062 --- [ocp-async-4,d263ae062e13393f,7bd962e637e42c6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:26.883 INFO 21062 --- [ocp-async-2,4c257d3f37f0f04d,b22736369464cb15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2ccb52eafcd127c1,800297ae3213cbb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:26.984 INFO 21062 --- [prometheus-hour-schedule-1,56469282e79ed306,d87c4ae9208a8911] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:22:27.470 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:27.470 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:28.498 INFO 21062 --- [ocp-schedules-16,71dbd066cd77d596,dc7a47cec318b6d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:22:28.500 INFO 21062 --- [ocp-schedules-4,6fc9653cab010629,69c1c9e49e26cb25] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:22:28.500 INFO 21062 --- [ocp-schedules-7,b3d20b6da67c222f,43395f469b27362c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:22:28.501 INFO 21062 --- [ocp-schedules-1,85c70ac4bac45df0,79f5d6d0bfdac911] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:22:28.502 INFO 21062 --- [ocp-schedules-20,a38fdd8f0ce5bae8,62bb7333a4ffb29e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:22:28.502 INFO 21062 --- [ocp-schedules-16,71dbd066cd77d596,dc7a47cec318b6d3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:22:28.504 INFO 21062 --- [ocp-schedules-7,b3d20b6da67c222f,43395f469b27362c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:28.504 INFO 21062 --- [ocp-schedules-4,6fc9653cab010629,69c1c9e49e26cb25] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:28.504 INFO 21062 --- [ocp-schedules-13,02843872685273ad,9f1eb665e90e635e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:22:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:22:28.505 INFO 21062 --- [ocp-schedules-5,6a922754122da26d,90a097a221dc64ab] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:22:28.506 INFO 21062 --- [ocp-schedules-1,85c70ac4bac45df0,79f5d6d0bfdac911] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:22:28.508 INFO 21062 --- [ocp-schedules-5,6a922754122da26d,90a097a221dc64ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:28.511 INFO 21062 --- [ocp-schedules-18,9bcaafa6e8fe730a,b915fb753f1df2b6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:22:28.511 INFO 21062 --- [ocp-schedules-9,79e039c7ee288159,2f932d38960b10a9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:22:28.511 INFO 21062 --- [ocp-schedules-9,79e039c7ee288159,2f932d38960b10a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:22:28.511 INFO 21062 --- [ocp-schedules-18,9bcaafa6e8fe730a,b915fb753f1df2b6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:22:28.512 INFO 21062 --- [ocp-schedules-11,af699a1abeaed903,3561d737fa3255dd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:22:28.512 INFO 21062 --- [ocp-schedules-11,af699a1abeaed903,3561d737fa3255dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:22:28.530 INFO 21062 --- [ocp-schedules-22,baab39be3bbce3ee,8fdcc1e794dbb9bc] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:22:28.530 INFO 21062 --- [ocp-schedules-22,baab39be3bbce3ee,8fdcc1e794dbb9bc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:22:28.564 INFO 21062 --- [ocp-schedules-17,0b827f26596ea8a4,fe77dd374df96cb3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:22:28.565 INFO 21062 --- [ocp-schedules-17,0b827f26596ea8a4,fe77dd374df96cb3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:22:28.575 INFO 21062 --- [pool-12-thread-10,ad810d0f88deeada,25ffd217cd3bd142] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:22:28.575 INFO 21062 --- [pool-12-thread-12,ad810d0f88deeada,76ec6c65b3ef1b59] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:22:28.576 INFO 21062 --- [pool-12-thread-11,ad810d0f88deeada,2135314ee76d329b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:22:28.576 INFO 21062 --- [pool-12-thread-9,ad810d0f88deeada,4ba1d96603e613e8] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:22:28.581 INFO 21062 --- [pool-12-thread-12,ad810d0f88deeada,76ec6c65b3ef1b59] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:22:28.581 INFO 21062 --- [pool-12-thread-12,ad810d0f88deeada,76ec6c65b3ef1b59] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:22:28.581 INFO 21062 --- [ForkJoinPool-11-worker-7,ad810d0f88deeada,084b5744eda6f917] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:22:28.581 INFO 21062 --- [ForkJoinPool-11-worker-7,ad810d0f88deeada,084b5744eda6f917] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:22:28.581 INFO 21062 --- [ForkJoinPool-11-worker-7,ad810d0f88deeada,084b5744eda6f917] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:22:28.581 INFO 21062 --- [pool-12-thread-11,ad810d0f88deeada,2135314ee76d329b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:22:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:22:28.588244+08:00, lastProcessTime=2024-11-04T13:22:18.591007+08:00 2024-11-04 13:22:28.592 INFO 21062 --- [ocp-async-2,ac8781d6d20d6756,cf05d0c53bdc3bfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:22:28.604 INFO 21062 --- [pool-12-thread-9,ad810d0f88deeada,4ba1d96603e613e8] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:22:28.616 INFO 21062 --- [pool-12-thread-9,ad810d0f88deeada,4ba1d96603e613e8] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:22:28.618 INFO 21062 --- [ForkJoinPool-11-worker-7,ad810d0f88deeada,220928aaa47437a7] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:22:28.618 INFO 21062 --- [pool-12-thread-10,ad810d0f88deeada,25ffd217cd3bd142] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:22:28.619 INFO 21062 --- [pool-12-thread-9,ad810d0f88deeada,4ba1d96603e613e8] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:22:28.622 INFO 21062 --- [ocp-schedules-21,3167b76048b949fb,45bf0c050d8f1724] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:22:28.622 INFO 21062 --- [ocp-schedules-21,3167b76048b949fb,45bf0c050d8f1724] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:22:28.678 INFO 21062 --- [ocp-schedules-1,c1f9050a9b4be9a7,19150564eb585417] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 13:22:28.679 INFO 21062 --- [ocp-schedules-1,c1f9050a9b4be9a7,19150564eb585417] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 13:22:28.679 INFO 21062 --- [ocp-schedules-1,c1f9050a9b4be9a7,19150564eb585417] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=322 2024-11-04 13:22:28.681 INFO 21062 --- [ocp-schedules-20,09615b314c86bba1,6ee6f8b7ef4de5cf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:22:28.688 INFO 21062 --- [ocp-schedules-18,6e2614e1715326b5,e38fc4019d42dd26] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:22:28.693 INFO 21062 --- [ocp-schedules-1,c1f9050a9b4be9a7,19150564eb585417] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=322 2024-11-04 13:22:28.700 INFO 21062 --- [ocp-schedules-18,6e2614e1715326b5,e38fc4019d42dd26] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:22:28.767 INFO 21062 --- [ocp-schedules-11,8c20ca22a210541d,7e1912834f3e9a38] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:22:28.770 INFO 21062 --- [ocp-schedules-11,8c20ca22a210541d,7e1912834f3e9a38] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:28.771 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:22:28.786 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:22:28.588244+08:00 2024-11-04 13:22:28.818 INFO 21062 --- [ocp-schedules-16,4f70fd4c28e98e6d,eb764da47d33d058] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 13:22:28.821 INFO 21062 --- [ocp-schedules-16,4f70fd4c28e98e6d,eb764da47d33d058] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:28.822 INFO 21062 --- [ocp-schedules-22,2d41ae0a14123d94,22e5d94be4c260f6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:22:28.822 INFO 21062 --- [ocp-schedules-22,2d41ae0a14123d94,22e5d94be4c260f6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:22:28.834 INFO 21062 --- [ocp-schedules-27,c9a209899859b677,29ef115dbfdf64e3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:22:28.835 WARN 21062 --- [ocp-async-7,ba9096e84389e6b4,5c47d490b011e3e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:28.856 ERROR 21062 --- [ocp-schedules-16,4f70fd4c28e98e6d,eb764da47d33d058] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:22:28.856 INFO 21062 --- [ocp-schedules-16,4f70fd4c28e98e6d,eb764da47d33d058] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 13:22:28.861 INFO 21062 --- [ocp-schedules-28,ef95da08abe35135,03489e7b93ea2a5f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:22:28.865 INFO 21062 --- [ocp-schedules-28,ef95da08abe35135,03489e7b93ea2a5f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:22:28.865 INFO 21062 --- [ocp-schedules-28,ef95da08abe35135,03489e7b93ea2a5f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:22:28.880 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:22:28.880 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:22:28.884 INFO 21062 --- [ocp-async-3,a9c7f2546a55e3e1,a28fa002e71b8c47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:28.887 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:22:28.887 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:22:28.887 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:22:28.898 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:22:28.898 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:22:28.898 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:22:28.901 INFO 21062 --- [ocp-schedules-8,faab6e9e23700319,45b8044a958fa737] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:22:29.029 INFO 21062 --- [ocp-schedules-3,8e1d6aad98b7d1c9,11b651686c61d619] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:22:29.029 INFO 21062 --- [ocp-schedules-3,8e1d6aad98b7d1c9,11b651686c61d619] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:22:29.030 INFO 21062 --- [ocp-schedules-3,8e1d6aad98b7d1c9,11b651686c61d619] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:22:29.030 INFO 21062 --- [ocp-schedules-3,8e1d6aad98b7d1c9,11b651686c61d619] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:22:29.031 INFO 21062 --- [ocp-schedules-3,8e1d6aad98b7d1c9,11b651686c61d619] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:22:29.031 INFO 21062 --- [ocp-schedules-3,8e1d6aad98b7d1c9,11b651686c61d619] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:22:29.070 INFO 21062 --- [ocp-schedules-14,c45460cb352837f6,3d0cc8e7cdb62dc8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:22:29.070 INFO 21062 --- [ocp-schedules-14,c45460cb352837f6,3d0cc8e7cdb62dc8] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:22:29.097 INFO 21062 --- [pool-29-thread-1,c45460cb352837f6,bc546a9cf23c08a0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:22:29.126 WARN 21062 --- [ocp-schedules-14,c45460cb352837f6,3d0cc8e7cdb62dc8] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:22:29.126 INFO 21062 --- [ocp-schedules-14,c45460cb352837f6,3d0cc8e7cdb62dc8] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:22:29.367 INFO 21062 --- [ocp-schedules-24,90c07acfc7e2b178,c7f2d3b15000d487] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:22:29.895 INFO 21062 --- [ocp-schedules-22,9a186c571c8e84fd,9aa1eb303689e14f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:22:29.896 INFO 21062 --- [ocp-schedules-22,9a186c571c8e84fd,9aa1eb303689e14f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:22:29.900 INFO 21062 --- [ocp-schedules-28,0d51d709010d7f52,7be0d07814327355] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:22:29.906 INFO 21062 --- [ocp-schedules-28,0d51d709010d7f52,7be0d07814327355] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:22:29.906 INFO 21062 --- [ocp-schedules-28,0d51d709010d7f52,7be0d07814327355] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:22:29.921 INFO 21062 --- [ocp-schedules-22,9a186c571c8e84fd,9aa1eb303689e14f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:22:29.921 INFO 21062 --- [ocp-schedules-22,9a186c571c8e84fd,9aa1eb303689e14f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:22:29.922 INFO 21062 --- [ocp-schedules-16,f2fb4c5c259a0cdd,28c93ebbb77cf120] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:22:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7254848535593ef6,83eb6d7c4f2fa016] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:29.996 INFO 21062 --- [ocp-schedules-17,869d99664cf8beb7,e70c1e1a8e06fcf0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:22:30.038 INFO 21062 --- [ocp-schedules-26,0d41db1b71940b62,730165e33624e436] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:22:30.040 INFO 21062 --- [ocp-schedules-30,7d5587bb7cce9259,6739862406a4150f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:22:30.040 INFO 21062 --- [ocp-schedules-30,7d5587bb7cce9259,6739862406a4150f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:22:30.040 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:22:30.040 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:22:30.041 INFO 21062 --- [ocp-schedules-26,0d41db1b71940b62,730165e33624e436] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:30.043 INFO 21062 --- [ocp-schedules-8,d50ebc28e3dba933,7336eb910f38a0e8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:22:30.043+08:00, clusters=[1] 2024-11-04 13:22:30.045 INFO 21062 --- [ocp-schedules-8,d50ebc28e3dba933,7336eb910f38a0e8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:22:30.049 INFO 21062 --- [ocp-schedules-30,7d5587bb7cce9259,6739862406a4150f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:22:30.049 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:22:30.050 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:22:30.051 WARN 21062 --- [ocp-schedules-30,7d5587bb7cce9259,6739862406a4150f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:22:30.051 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:22:30.051 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:22:30.051 INFO 21062 --- [ocp-schedules-30,7d5587bb7cce9259,6739862406a4150f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:22:30.051 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:22:30.052 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:22:30.053 WARN 21062 --- [ocp-schedules-30,7d5587bb7cce9259,6739862406a4150f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:22:30.053 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:22:30.053 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:22:30.061 ERROR 21062 --- [ocp-schedules-30,7d5587bb7cce9259,6739862406a4150f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:22:30.100 ERROR 21062 --- [ocp-schedules-26,0d41db1b71940b62,730165e33624e436] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:22:30.100 INFO 21062 --- [ocp-schedules-26,0d41db1b71940b62,730165e33624e436] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:22:30.167 INFO 21062 --- [ocp-schedules-23,1c0b2bd83d46e6a1,997b85d588360581] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:22:30.167+08:00, clusters=[1] 2024-11-04 13:22:30.170 INFO 21062 --- [ocp-schedules-23,1c0b2bd83d46e6a1,997b85d588360581] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:22:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:22:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:22:30.471 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:30.471 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:30.835 WARN 21062 --- [ocp-async-4,e9144273f392325c,0f381c77480221de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:30.884 INFO 21062 --- [ocp-async-10,db4a94a37bbe38d5,5937bbf4e91e7ba1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:31.927 INFO 21062 --- [ocp-schedules-10,ea963cad02f146a0,00766e17ca927d51] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:22:31.927 INFO 21062 --- [ocp-schedules-10,8f2c9c46b0560d58,4511de665f13c828] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:22:31.931 INFO 21062 --- [ocp-schedules-10,8f2c9c46b0560d58,4511de665f13c828] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:31.960 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:22:31.960 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:22:32.835 WARN 21062 --- [ocp-async-2,63162573be2955da,607068f74c6c8197] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:32.884 INFO 21062 --- [ocp-async-9,8260815a90fe701b,a9aac26576307301] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:32.983 INFO 21062 --- [prometheus-minute-schedule-1,97d4eb6b35c47754,fbce093b6921a321] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:33.092 INFO 21062 --- [ocp-schedules-6,023514cfb8974ea6,04e4fcda92455054] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:22:33.096 INFO 21062 --- [ocp-schedules-6,023514cfb8974ea6,04e4fcda92455054] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:33.186 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:22:33.188 ERROR 21062 --- [ocp-schedules-6,023514cfb8974ea6,04e4fcda92455054] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:22:33.188 INFO 21062 --- [ocp-schedules-6,023514cfb8974ea6,04e4fcda92455054] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:22:33.472 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:33.472 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:34.835 WARN 21062 --- [ocp-async-1,e53661652711c53c,fa1a3ad3be7e375f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:34.884 INFO 21062 --- [ocp-async-5,5ece6f53f0274b67,65d9c6d74ad4ae3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5cc17285436bd47c,c3d126e2084ba3bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:36.473 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:36.473 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:36.835 WARN 21062 --- [ocp-async-3,643b30a410afbcbe,262f0777975e58df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:36.884 INFO 21062 --- [ocp-async-8,9f9f1228cbf88479,3c06118cfcd8209d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:38.504 INFO 21062 --- [ocp-schedules-7,b3d20b6da67c222f,43395f469b27362c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:22:38.505 INFO 21062 --- [ocp-schedules-4,6fc9653cab010629,69c1c9e49e26cb25] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:22:38.508 INFO 21062 --- [ocp-schedules-5,6a922754122da26d,90a097a221dc64ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:22:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:22:38.589917+08:00, lastProcessTime=2024-11-04T13:22:28.588244+08:00 2024-11-04 13:22:38.592 INFO 21062 --- [ocp-async-4,6de1281774456bfb,ddc54f75740318ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:22:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:22:38.589917+08:00 2024-11-04 13:22:38.771 INFO 21062 --- [ocp-schedules-11,8c20ca22a210541d,7e1912834f3e9a38] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:22:38.836 WARN 21062 --- [ocp-async-2,ac6b581c6acfb2da,8f69afeb6df72feb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:38.884 INFO 21062 --- [ocp-async-1,5377fa315eaaaefc,db44b787c2e3f4b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:38.983 INFO 21062 --- [prometheus-minute-schedule-1,17d627d85cfbd4d1,0d06caa3206517ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:39.370 INFO 21062 --- [ocp-schedules-16,0a158404076ba8f5,585dc5af6b8227f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:22:39.473 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:39.474 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:40.002 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:22:40.004 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:22:40.023 INFO 21062 --- [ocp-schedules-6,bd7d99f1b3e02ef5,77af25fb1ec01147] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:22:40.836 WARN 21062 --- [ocp-async-1,85e44c1920d33871,ef395a15f76add14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:40.884 INFO 21062 --- [ocp-async-5,6fd01fc38b6c087a,c4857469696f872a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:22:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:22:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:22:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:22:41.802 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:22:41.831 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:22:41.831 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-11-04 13:22:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d8cfe0f56c78aa5e,a7300eff74898b7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:42.475 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:42.476 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:42.836 WARN 21062 --- [ocp-async-5,7781d79ca245f227,dc9ffab85d32edcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:42.885 INFO 21062 --- [ocp-async-10,2f9309cddf578833,7a576fe12b1bd18b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:44.836 WARN 21062 --- [ocp-async-10,d65498668c868800,113cbce4d23438c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:44.885 INFO 21062 --- [ocp-async-9,7c80bf608d31ba3e,0e4e6be674050d11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:44.983 INFO 21062 --- [prometheus-minute-schedule-1,585d63f12b550a79,09bcc0ac2ade0fc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:45.476 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:45.477 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:46.836 WARN 21062 --- [ocp-async-9,c10af454796604e8,9f2f5ef8468621ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:46.885 INFO 21062 --- [ocp-async-3,a41c5e57a6cf8689,1bf94aaa9a66c262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:47.983 INFO 21062 --- [prometheus-minute-schedule-1,f80674292612272e,728d5ff840224fff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:48.477 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:48.478 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:22:48.590113+08:00, lastProcessTime=2024-11-04T13:22:38.589917+08:00 2024-11-04 13:22:48.593 INFO 21062 --- [ocp-async-1,6afc35fa40017ac8,0e6b04c51d11f92d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:22:48.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:22:48.590113+08:00 2024-11-04 13:22:48.836 WARN 21062 --- [ocp-async-5,2e43883facec65f2,da3fc164dd57a897] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:48.885 INFO 21062 --- [ocp-async-10,95032f7d554e106a,d87810839e3f675e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:49.373 INFO 21062 --- [ocp-schedules-4,c478b54cf965ab01,cf90e71604ce1c64] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:22:49.908 INFO 21062 --- [ocp-schedules-1,7fca6a7e06e0bee0,1293ab17b0f5084b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:22:49.914 INFO 21062 --- [ocp-schedules-1,7fca6a7e06e0bee0,1293ab17b0f5084b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:22:49.914 INFO 21062 --- [ocp-schedules-1,7fca6a7e06e0bee0,1293ab17b0f5084b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:22:50.065 INFO 21062 --- [ocp-schedules-6,af11aa77d80bd63f,ddc913fa6d9df6f2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:22:50.837 WARN 21062 --- [ocp-async-10,33421c353907824b,ae7c2e1be72afa41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:50.885 INFO 21062 --- [ocp-async-9,6fc258f8155ad1de,bcab52b945fb25f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:50.983 INFO 21062 --- [prometheus-minute-schedule-1,bf89570b6171f491,3a748f6e99b52f64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:51.478 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:51.479 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:52.837 WARN 21062 --- [ocp-async-9,2abd1fdaec4273e4,02b90d627bd0ff50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:52.886 INFO 21062 --- [ocp-async-3,33f04b42bb42913c,2220a3e0686f9bf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:53.983 INFO 21062 --- [prometheus-minute-schedule-1,36ca9c5440ea4ea4,5b6a22a12ae3603c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:54.479 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:54.480 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:54.837 WARN 21062 --- [ocp-async-3,07235de750cfa818,980c8f32e2fa5192] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:54.886 INFO 21062 --- [ocp-async-4,f581a862509bd207,a6e52454e76af15b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:56.837 WARN 21062 --- [ocp-async-4,6a17b698503fcbac,0a4c1ca6be0ea46c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:56.886 INFO 21062 --- [ocp-async-2,75a2e04807022b95,5670a0195b4c7f6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:56.983 INFO 21062 --- [prometheus-minute-schedule-1,dd33f2bd3523df86,4ba8ef4134b08281] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:22:57.480 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:22:57.481 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:22:58.500 INFO 21062 --- [ocp-schedules-6,dc46f659bfa8bb86,79aa8539f1888e40] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:22:58.500 INFO 21062 --- [ocp-schedules-19,0b201849a1af9da7,62d3c5d70ffea6af] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:22:58.502 INFO 21062 --- [ocp-schedules-9,700566384572e0b4,06fff6fd741d60ee] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:22:58.504 INFO 21062 --- [ocp-schedules-19,0b201849a1af9da7,62d3c5d70ffea6af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:58.504 INFO 21062 --- [ocp-schedules-6,dc46f659bfa8bb86,79aa8539f1888e40] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:22:58.506 INFO 21062 --- [ocp-schedules-20,f7f74ba6dd1170b3,a3045b3380a9ec06] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:22:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:22:58.510 INFO 21062 --- [ocp-schedules-8,ebafd72bde260a54,d45ae546d2ce40aa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:22:58.510 INFO 21062 --- [ocp-schedules-13,2199adaba1e11be9,ac929e878a513fda] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:22:58.510 INFO 21062 --- [ocp-schedules-3,699100f9133ec065,ed21544a05da6e90] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:22:58.510 INFO 21062 --- [ocp-schedules-8,ebafd72bde260a54,d45ae546d2ce40aa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:22:58.510 INFO 21062 --- [ocp-schedules-3,699100f9133ec065,ed21544a05da6e90] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:22:58.510 INFO 21062 --- [ocp-schedules-13,2199adaba1e11be9,ac929e878a513fda] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:22:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:22:58.589730+08:00, lastProcessTime=2024-11-04T13:22:48.590113+08:00 2024-11-04 13:22:58.593 INFO 21062 --- [ocp-async-10,b16f0a136427f5f1,fb66f06da4aae3d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:22:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:22:58.589730+08:00 2024-11-04 13:22:58.837 INFO 21062 --- [ocp-schedules-22,33c9f63b08e34fd3,02ece53b5470b94d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:22:58.837 WARN 21062 --- [ocp-async-9,193fb7569e9c34f4,bdb2faa60c25399e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:22:58.886 INFO 21062 --- [ocp-async-3,d87ef9c925d51697,6bbaf2515f585751] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:22:59.376 INFO 21062 --- [ocp-schedules-26,1f060bc6d60990f2,01b92704b68c8980] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:22:59.983 INFO 21062 --- [prometheus-minute-schedule-1,4cbae73d29d3c45f,2d2659a329c223b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697780, interval=60 2024-11-04 13:23:00.092 INFO 21062 --- [ocp-schedules-22,6eff18fd2829b374,b20e5ea08e3dfae7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:23:00.481 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:00.482 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:00.838 WARN 21062 --- [ocp-async-3,0628e32a52ebc9a3,1f952b785832e054] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:00.886 INFO 21062 --- [ocp-async-4,f1e1fc7634eaf77b,85f6d9e3f84b1dd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:01.961 INFO 21062 --- [ocp-schedules-10,8f2c9c46b0560d58,4511de665f13c828] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:23:01.961 INFO 21062 --- [ocp-schedules-10,9993a23013fe164c,02d964f2a0f91e22] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:23:01.965 INFO 21062 --- [ocp-schedules-10,9993a23013fe164c,02d964f2a0f91e22] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:01.996 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:23:01.996 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:23:02.838 WARN 21062 --- [ocp-async-4,5759c759e8d97b25,fb4c58e847b71971] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:02.887 INFO 21062 --- [ocp-async-2,f6fbf48a6085ea63,2c61de85da365fb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d1492dc68980aee4,e5a2f400ded629d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:03.482 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:03.483 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:04.838 WARN 21062 --- [ocp-async-2,d3e8ead712042861,923c8334ddf89888] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:04.887 INFO 21062 --- [ocp-async-7,19658c7962fe68e9,1691e664e00c5155] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:05.983 INFO 21062 --- [prometheus-minute-schedule-1,30164fdff4abaca7,91e46595fa4dcd31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:06.483 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:06.484 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:06.838 WARN 21062 --- [ocp-async-7,e51171535851b052,fc60ddca8433a759] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:06.887 INFO 21062 --- [ocp-async-8,3b7917f66028990b,ae7159f100224d01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:08.506 INFO 21062 --- [ocp-schedules-19,0b201849a1af9da7,62d3c5d70ffea6af] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:23:08.507 INFO 21062 --- [ocp-schedules-6,dc46f659bfa8bb86,79aa8539f1888e40] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:23:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:23:08.590952+08:00, lastProcessTime=2024-11-04T13:22:58.589730+08:00 2024-11-04 13:23:08.593 INFO 21062 --- [ocp-async-3,4ab631168f0b626d,5eb568e5d362674d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:23:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:23:08.590952+08:00 2024-11-04 13:23:08.838 WARN 21062 --- [ocp-async-4,8cced763fbd43858,bf8d2dba77f81aab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:08.887 INFO 21062 --- [ocp-async-6,3b16000e2e4c43b1,939e0f5b26038201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2210af415d1c6969,1174183b09060153] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:09.379 INFO 21062 --- [ocp-schedules-12,c5c269191d8e3631,c064a69e3d51bafa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:23:09.484 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:09.485 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:09.917 INFO 21062 --- [ocp-schedules-20,68ac902cac3a0760,87edf458da900a14] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:23:09.923 INFO 21062 --- [ocp-schedules-20,68ac902cac3a0760,87edf458da900a14] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:23:09.923 INFO 21062 --- [ocp-schedules-20,68ac902cac3a0760,87edf458da900a14] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:23:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:23:10.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:23:10.132 INFO 21062 --- [ocp-schedules-4,08042028ad04ac48,2020e52c48a78310] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:23:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:23:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:23:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:23:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:23:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:23:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:23:10.838 WARN 21062 --- [ocp-async-6,dbe419c3da2bf9bd,f5e327dcdb4a0db6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:10.887 INFO 21062 --- [ocp-async-1,9a677216cfc17f4b,8a8a8acabf053c24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:11.231 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=3, affectRows=3 2024-11-04 13:23:11.231 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=5, affectRows=5 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=3, validCount=3 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=5, validCount=5 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=19, affectRows=19 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=31, affectRows=31 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=11, affectRows=11 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=21, affectRows=21 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=11, affectRows=11 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=7, affectRows=7 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=11, validCount=11 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=11, validCount=11 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=19, validCount=19 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=9, affectRows=9 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=6, validCount=6 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=31, validCount=31 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=21, validCount=21 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=9, validCount=9 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=43, affectRows=43 2024-11-04 13:23:11.232 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=35, affectRows=35 2024-11-04 13:23:11.233 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=43, validCount=43 2024-11-04 13:23:11.233 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=35, validCount=35 2024-11-04 13:23:11.233 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=54, affectRows=54 2024-11-04 13:23:11.233 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697720, nodeCount=73, affectRows=73 2024-11-04 13:23:11.233 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=54, validCount=54 2024-11-04 13:23:11.233 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697720, nodeCount=71, validCount=71 2024-11-04 13:23:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e4d20fc0aaa2eda4,d40547397f7235e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:12.485 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:12.486 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:12.839 WARN 21062 --- [ocp-async-1,874bb97db0d1c803,ce4baf50cfbd7368] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:12.887 INFO 21062 --- [ocp-async-5,3c4c769d0298de49,b4f1bf6a786dae46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:13.984 INFO 21062 --- [prometheus-cache-monitor1,ed895d7acefbdd9b,695f4b8aea20c9c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:23:13.985 INFO 21062 --- [prometheus-cache-monitor1,ed895d7acefbdd9b,695f4b8aea20c9c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:23:14.029 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:17:14.006, end=2024-11-04T13:22:14.026} 2024-11-04 13:23:14.029 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:17:14.006, end=2024-11-04T13:22:14.026} 2024-11-04 13:23:14.033 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:23:14.033 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:23:14.033 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:23:14.033 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:23:14.499 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:19:14.477, end=2024-11-04T13:21:14.497} 2024-11-04 13:23:14.503 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:23:14.503 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:23:14.839 WARN 21062 --- [ocp-async-5,61d8f27a2c3fbfa0,a6d97fd59c8e7b7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:14.887 INFO 21062 --- [ocp-async-10,c652b9e7ab143e46,1ccdb88507191294] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:14.983 INFO 21062 --- [prometheus-minute-schedule-1,47a4b9f24fa34eaf,bedfe6ace8db553c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:15.486 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:15.487 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:16.839 WARN 21062 --- [ocp-async-10,99ca5d35727dd598,72cfe8dcba185d59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:16.888 INFO 21062 --- [ocp-async-9,d9b64f5df7b29752,7c6591b5ca58fb42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5b5ed4e98571f96a,e19b9a6ac89994f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:18.487 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:18.488 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:23:18.589728+08:00, lastProcessTime=2024-11-04T13:23:08.590952+08:00 2024-11-04 13:23:18.593 INFO 21062 --- [ocp-async-10,8073060ea783334f,e82fb95cb01aafa9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:23:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:23:18.589728+08:00 2024-11-04 13:23:18.839 WARN 21062 --- [ocp-async-1,5c5f36f68468e486,9a112a48f4a6b719] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:18.888 INFO 21062 --- [ocp-async-8,3e589b3c2c4638bd,8810a8de55e495e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:19.383 INFO 21062 --- [ocp-schedules-1,7e6c3b09d2d32b4f,b01901e6dd632f74] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:23:20.157 INFO 21062 --- [ocp-schedules-13,608fd677117f3f51,153890534c3eef67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:23:20.839 WARN 21062 --- [ocp-async-5,8c9c87d3cbc9069d,0416be95a4c9ab82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:20.888 INFO 21062 --- [ocp-async-2,5247fc2e8d10a93f,0878712253cb8d71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ec0aa70dc7bd6bfc,10726a5ec3cea520] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:21.488 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:21.489 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:22.839 WARN 21062 --- [ocp-async-10,6d8ca25f900d3d9d,13185302bb9eae71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:22.888 INFO 21062 --- [ocp-async-9,6d1da904c5879d9e,a16888dc73df386c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:23.983 INFO 21062 --- [prometheus-minute-schedule-1,edc139e014f5b685,4552c4d8c006d6a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:24.490 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:24.491 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:24.839 WARN 21062 --- [ocp-async-7,bee1c7c50e1a9cd9,67dbc05a96043dea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:24.888 INFO 21062 --- [ocp-async-3,b9059dad6f5f0f9c,7319c4a4412cd67b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:26.840 WARN 21062 --- [ocp-async-8,9a72ddde95f6e38d,09f4fff4960c0809] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:26.889 INFO 21062 --- [ocp-async-4,28b35534ef6c01fa,3c95d04260d2dcbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:26.983 INFO 21062 --- [prometheus-minute-schedule-1,038407e4dee0673b,18ddbc495b7078f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e3465be4c2e8e318,454d7f55e41009aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:23:27.491 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:27.492 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:28.498 INFO 21062 --- [ocp-schedules-13,38e663f36f31e829,31d880bd0e1a9a23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:23:28.500 INFO 21062 --- [ocp-schedules-20,c6bf38ca573eeef6,feeea007b887d580] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:23:28.500 INFO 21062 --- [ocp-schedules-6,3bc8fc82349e12cc,f691928f57f68521] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:23:28.500 INFO 21062 --- [ocp-schedules-15,62b4a4ec2372d88e,1c58dca1a9754484] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:23:28.502 INFO 21062 --- [ocp-schedules-13,38e663f36f31e829,31d880bd0e1a9a23] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:23:28.502 INFO 21062 --- [ocp-schedules-14,1bdb9105c802d9f8,a1a47de5b9cee00c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:23:28.503 INFO 21062 --- [ocp-schedules-6,3bc8fc82349e12cc,f691928f57f68521] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:28.504 INFO 21062 --- [ocp-schedules-15,62b4a4ec2372d88e,1c58dca1a9754484] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:23:28.505 INFO 21062 --- [ocp-schedules-19,5b5376760a397ae3,e9520b9769d74c1c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:23:28.506 INFO 21062 --- [ocp-schedules-20,c6bf38ca573eeef6,feeea007b887d580] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:23:28.512 INFO 21062 --- [ocp-schedules-24,acf1b4fe0bb29e1f,62e2ad0cb9135faf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:23:28.512 INFO 21062 --- [ocp-schedules-3,c25b7dae491d6ac6,d4289e2212961c1b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:23:28.512 INFO 21062 --- [ocp-schedules-24,acf1b4fe0bb29e1f,62e2ad0cb9135faf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:23:28.512 INFO 21062 --- [ocp-schedules-3,c25b7dae491d6ac6,d4289e2212961c1b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:23:28.512 INFO 21062 --- [ocp-schedules-4,71a5c722645e1731,eb29bf9675d1fb61] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:23:28.512 INFO 21062 --- [ocp-schedules-4,71a5c722645e1731,eb29bf9675d1fb61] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:23:28.558 INFO 21062 --- [ocp-schedules-27,0fac3cfa43852270,644546951995299a] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 13:23:28.558 INFO 21062 --- [ocp-schedules-27,0fac3cfa43852270,644546951995299a] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 13:23:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:23:28.589455+08:00, lastProcessTime=2024-11-04T13:23:18.589728+08:00 2024-11-04 13:23:28.593 INFO 21062 --- [ocp-async-4,6b6795459263bceb,2df53012b87710c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:23:28.683 INFO 21062 --- [ocp-schedules-11,42d0837776be0a46,473656988e763966] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:23:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:23:28.589455+08:00 2024-11-04 13:23:28.826 INFO 21062 --- [ocp-schedules-23,c2afae198d0e6d82,d37b825a93777742] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:23:28.826 INFO 21062 --- [ocp-schedules-23,c2afae198d0e6d82,d37b825a93777742] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:23:28.839 INFO 21062 --- [ocp-schedules-20,c5edc57c2bcf6efc,28ad46d93ad7fd57] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:23:28.840 WARN 21062 --- [ocp-async-9,57ffe6d2ae89b77b,67d988b4e090e168] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:28.865 INFO 21062 --- [ocp-schedules-24,4eacdef3bc52ed3e,b900b5562dd5b3f4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:23:28.870 INFO 21062 --- [ocp-schedules-24,4eacdef3bc52ed3e,b900b5562dd5b3f4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:23:28.870 INFO 21062 --- [ocp-schedules-24,4eacdef3bc52ed3e,b900b5562dd5b3f4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:23:28.889 INFO 21062 --- [ocp-async-1,a9bd74ebd27c0599,66668a7879c5d338] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:29.031 INFO 21062 --- [ocp-schedules-29,ebed255a38ffb204,7fa501b8dcccfa6b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:23:29.031 INFO 21062 --- [ocp-schedules-29,ebed255a38ffb204,7fa501b8dcccfa6b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:23:29.032 INFO 21062 --- [ocp-schedules-29,ebed255a38ffb204,7fa501b8dcccfa6b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:23:29.032 INFO 21062 --- [ocp-schedules-29,ebed255a38ffb204,7fa501b8dcccfa6b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:23:29.033 INFO 21062 --- [ocp-schedules-29,ebed255a38ffb204,7fa501b8dcccfa6b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:23:29.033 INFO 21062 --- [ocp-schedules-29,ebed255a38ffb204,7fa501b8dcccfa6b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:23:29.386 INFO 21062 --- [ocp-schedules-28,1841407281b58efa,269937662a6c566a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:23:29.922 INFO 21062 --- [ocp-schedules-8,33dcad6381d7363d,557e71a1db83c3f1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:23:29.922 INFO 21062 --- [ocp-schedules-8,33dcad6381d7363d,557e71a1db83c3f1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:23:29.926 INFO 21062 --- [ocp-schedules-23,a56b2da5fcc0dd0a,36484b21131f8903] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:23:29.932 INFO 21062 --- [ocp-schedules-23,a56b2da5fcc0dd0a,36484b21131f8903] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:23:29.932 INFO 21062 --- [ocp-schedules-23,a56b2da5fcc0dd0a,36484b21131f8903] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:23:29.949 INFO 21062 --- [ocp-schedules-8,33dcad6381d7363d,557e71a1db83c3f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:23:29.949 INFO 21062 --- [ocp-schedules-8,33dcad6381d7363d,557e71a1db83c3f1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:23:29.949 INFO 21062 --- [ocp-schedules-22,338896c74d88bdb6,f2c35c54f09deff2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:23:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e820101a90e11920,73a8090cf7e9fe06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:30.068 INFO 21062 --- [ocp-schedules-3,72ea8fa1a25510e4,7d89946fd0257c13] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:23:30.068 INFO 21062 --- [ocp-schedules-3,72ea8fa1a25510e4,7d89946fd0257c13] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:23:30.068 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:23:30.068 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:23:30.071 INFO 21062 --- [ocp-schedules-24,16f5c77769ec6b8d,36ddcfb046bdbbad] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:23:30.071+08:00, clusters=[1] 2024-11-04 13:23:30.073 INFO 21062 --- [ocp-schedules-24,16f5c77769ec6b8d,36ddcfb046bdbbad] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:23:30.078 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:23:30.078 INFO 21062 --- [ocp-schedules-3,72ea8fa1a25510e4,7d89946fd0257c13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:23:30.079 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:23:30.080 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:23:30.080 WARN 21062 --- [ocp-schedules-3,72ea8fa1a25510e4,7d89946fd0257c13] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:23:30.080 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:23:30.080 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:23:30.080 INFO 21062 --- [ocp-schedules-3,72ea8fa1a25510e4,7d89946fd0257c13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:23:30.081 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:23:30.082 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:23:30.082 WARN 21062 --- [ocp-schedules-3,72ea8fa1a25510e4,7d89946fd0257c13] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:23:30.082 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:23:30.090 ERROR 21062 --- [ocp-schedules-3,72ea8fa1a25510e4,7d89946fd0257c13] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:23:30.199 INFO 21062 --- [ocp-schedules-16,506b561fcfe40d7c,4eac174a4f424372] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:23:30.199+08:00, clusters=[1] 2024-11-04 13:23:30.202 INFO 21062 --- [ocp-schedules-16,506b561fcfe40d7c,4eac174a4f424372] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:23:30.246 INFO 21062 --- [ocp-schedules-26,3e77792ec8c8ae5e,b817c51277ddba66] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:23:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:23:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:23:30.492 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:30.492 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:30.840 WARN 21062 --- [ocp-async-8,87f5d1d953bdd45f,4a500debc5c9d7b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:30.889 INFO 21062 --- [ocp-async-5,a426c1b41a2489e3,970e46b7f736a2c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:31.996 INFO 21062 --- [ocp-schedules-10,9993a23013fe164c,02d964f2a0f91e22] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:23:31.997 INFO 21062 --- [ocp-schedules-10,5bbb71cfe2f4e1ed,f6fc638c31cce9ff] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:23:32.001 INFO 21062 --- [ocp-schedules-10,5bbb71cfe2f4e1ed,f6fc638c31cce9ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:32.033 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:23:32.033 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:23:32.840 WARN 21062 --- [ocp-async-4,93e5c5b66c898c09,f7061ff01fcfc4db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:32.889 INFO 21062 --- [ocp-async-10,d921b79b342b0b3b,def97c54abd51920] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:32.983 INFO 21062 --- [prometheus-minute-schedule-1,fe97db1dffb51bda,4df0a3e0900a7eae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:33.189 INFO 21062 --- [ocp-schedules-8,cecd9a47ab32c641,db40d4f868b05786] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:23:33.192 INFO 21062 --- [ocp-schedules-8,cecd9a47ab32c641,db40d4f868b05786] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:33.277 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:23:33.285 ERROR 21062 --- [ocp-schedules-8,cecd9a47ab32c641,db40d4f868b05786] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:23:33.285 INFO 21062 --- [ocp-schedules-8,cecd9a47ab32c641,db40d4f868b05786] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:23:33.493 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:33.493 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:34.840 WARN 21062 --- [ocp-async-6,94936e6a63da8649,8316ce61b05e5ecf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:34.889 INFO 21062 --- [ocp-async-7,e6988e8ac97b90cf,b0b9532a87331d06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b34118b57e7ed8ad,79ae9b8e3bed7ce9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:36.493 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:36.494 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:36.841 WARN 21062 --- [ocp-async-7,f07c3a8b06404fbc,0dd152fc3cfb286a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:36.889 INFO 21062 --- [ocp-async-3,3cbbbee1c3fce5c7,411a2e7a6c6d1674] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:38.504 INFO 21062 --- [ocp-schedules-15,62b4a4ec2372d88e,1c58dca1a9754484] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:23:38.505 INFO 21062 --- [ocp-schedules-6,3bc8fc82349e12cc,f691928f57f68521] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:23:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:23:38.588891+08:00, lastProcessTime=2024-11-04T13:23:28.589455+08:00 2024-11-04 13:23:38.593 INFO 21062 --- [ocp-async-8,685ff5e595fd601f,c80de4396efe94f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:23:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:23:38.588891+08:00 2024-11-04 13:23:38.771 INFO 21062 --- [ocp-schedules-1,229d779244fe3c3a,82c5bb1120cad78a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:23:38.774 INFO 21062 --- [ocp-schedules-1,229d779244fe3c3a,82c5bb1120cad78a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:38.775 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:23:38.841 WARN 21062 --- [ocp-async-4,1e56d83a37734b61,ec5041bfa4c1b064] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:38.889 INFO 21062 --- [ocp-async-6,161ff6662acfe47a,2a73ecce7da14ae7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:38.983 INFO 21062 --- [prometheus-minute-schedule-1,23eb26a38e15d2cd,75e5f4ae14543145] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:39.389 INFO 21062 --- [ocp-schedules-4,e492d37d01b31f15,3bf40e6a8a472520] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:23:39.494 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:39.495 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:23:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:23:40.270 INFO 21062 --- [ocp-schedules-16,11630bf797105882,a4f36d299ca1bd65] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:23:40.841 WARN 21062 --- [ocp-async-6,369fc5abbf020b7e,9ab2d69b59d42efe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:40.890 INFO 21062 --- [ocp-async-1,51e42da7af55fd00,44bf72abc9d7b9e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:41.983 INFO 21062 --- [prometheus-minute-schedule-1,b0ef2ca99843ab4c,f975f1514ed5afcb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:42.495 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:42.496 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:42.841 WARN 21062 --- [ocp-async-1,e10cb99356d803d7,c8f8b4fe2ac0db93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:42.890 INFO 21062 --- [ocp-async-5,5ebddcbcee2137f8,5471e5b7d3da0d5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:44.841 WARN 21062 --- [ocp-async-5,32c3b4141b601f88,4b4cbcf6c8ff9348] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:44.890 INFO 21062 --- [ocp-async-10,597c734289710da7,96f8d6d2771dc7f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a0246dfd7ee5ee48,1db44da07e19bf97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:45.497 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:45.498 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:46.841 WARN 21062 --- [ocp-async-10,f2f3a32d737edb3f,ff8afe9cf1f176cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:46.890 INFO 21062 --- [ocp-async-9,961eccf36b4fc8b3,d2c5cbb50e753072] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:47.983 INFO 21062 --- [prometheus-minute-schedule-1,69736d421dc58b19,15002c7c045c7d96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:48.498 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:48.499 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:23:48.590507+08:00, lastProcessTime=2024-11-04T13:23:38.588891+08:00 2024-11-04 13:23:48.594 INFO 21062 --- [ocp-async-6,971761b83c9b2579,58eacfa5b282398e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:23:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:23:48.590507+08:00 2024-11-04 13:23:48.775 INFO 21062 --- [ocp-schedules-1,229d779244fe3c3a,82c5bb1120cad78a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:23:48.842 WARN 21062 --- [ocp-async-1,a5d367c0c72def3a,5032f2514d67b88d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:48.890 INFO 21062 --- [ocp-async-5,b92fa74a8a9a7266,81e5c7ba0fbb7c83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:49.392 INFO 21062 --- [ocp-schedules-22,81c3f19fbd72c77a,c2fd8af904da38b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:23:49.935 INFO 21062 --- [ocp-schedules-21,7ea4c02d45495d3f,75a1bf2de6c21ea5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:23:49.941 INFO 21062 --- [ocp-schedules-21,7ea4c02d45495d3f,75a1bf2de6c21ea5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:23:49.941 INFO 21062 --- [ocp-schedules-21,7ea4c02d45495d3f,75a1bf2de6c21ea5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:23:50.314 INFO 21062 --- [ocp-schedules-1,35b6af442e6c9d74,d5084c2719b72612] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:23:50.842 WARN 21062 --- [ocp-async-5,3410930a19f22c86,0276b2c4f78c7e54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:50.891 INFO 21062 --- [ocp-async-10,42da87949ce1d14b,cba6481755fc2c30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:50.983 INFO 21062 --- [prometheus-minute-schedule-1,63453905bdfcc42f,68c81d43d2edb597] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:51.499 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:51.500 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:52.842 WARN 21062 --- [ocp-async-10,5642235eaaefaed8,b0426675afe8a0b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:52.891 INFO 21062 --- [ocp-async-9,7ac5a1bd87ec7e90,d47d6ffa4976d025] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a4f8cefd8711ca94,c3f4af6548b2b2d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:54.500 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:54.501 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:54.842 WARN 21062 --- [ocp-async-9,4c18536fdcfce95a,54f16e671c95c1b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:54.891 INFO 21062 --- [ocp-async-8,0567c7af5472ec58,9177b87aa8b78f2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:56.842 WARN 21062 --- [ocp-async-8,a0dcbe06b88a450b,c00b1d262eba8d1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:56.891 INFO 21062 --- [ocp-async-2,bc8c2b2213847add,527898374a85a1bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:56.983 INFO 21062 --- [prometheus-minute-schedule-1,169fd06d37e17ad5,8c715f3eb8322130] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:23:57.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:23:57.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:23:58.500 INFO 21062 --- [ocp-schedules-26,ad6d79d9914af75b,2a072b6fa5cbd0d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:23:58.500 INFO 21062 --- [ocp-schedules-8,819430024c7c1af7,4b1bde9178925c20] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:23:58.502 INFO 21062 --- [ocp-schedules-9,914ac1c33022172a,eecbcec23daff90d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:23:58.504 INFO 21062 --- [ocp-schedules-8,819430024c7c1af7,4b1bde9178925c20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:58.505 INFO 21062 --- [ocp-schedules-26,ad6d79d9914af75b,2a072b6fa5cbd0d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:23:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:23:58.505 INFO 21062 --- [ocp-schedules-27,71a4c04479db7447,cb198eb33274650e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:23:58.512 INFO 21062 --- [ocp-schedules-20,49fd4c8cf0820419,33f1e3c21a365094] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:23:58.512 INFO 21062 --- [ocp-schedules-20,49fd4c8cf0820419,33f1e3c21a365094] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:23:58.512 INFO 21062 --- [ocp-schedules-14,321cac93dcf5c379,d783ce5f7e1f7404] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:23:58.512 INFO 21062 --- [ocp-schedules-14,321cac93dcf5c379,d783ce5f7e1f7404] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:23:58.512 INFO 21062 --- [ocp-schedules-21,5226b10b2ab962cf,91906000664fb114] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:23:58.512 INFO 21062 --- [ocp-schedules-21,5226b10b2ab962cf,91906000664fb114] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:23:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:23:58.589893+08:00, lastProcessTime=2024-11-04T13:23:48.590507+08:00 2024-11-04 13:23:58.594 INFO 21062 --- [ocp-async-5,a4422044963bd1f0,b65f53636f4a32d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:23:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:23:58.589893+08:00 2024-11-04 13:23:58.842 INFO 21062 --- [ocp-schedules-28,f3c57b220cf29991,20d65f7eedd9121b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:23:58.842 WARN 21062 --- [ocp-async-10,b2470d4b8fdf906f,059daf5c8e751a73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:23:58.891 INFO 21062 --- [ocp-async-9,33749bc9da914404,28af946e144724fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:23:59.395 INFO 21062 --- [ocp-schedules-5,5eb56f4fdf74a6d9,4bb316b3cfc5d0dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:23:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f696083a866a3b6f,3d4b853080855d35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697840, interval=60 2024-11-04 13:24:00.340 INFO 21062 --- [ocp-schedules-13,b9331bc402e4ef54,0348f058b6953dd2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:24:00.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:00.504 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:00.843 WARN 21062 --- [ocp-async-9,808700b1a1614bc5,2af6f25181edd965] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:00.892 INFO 21062 --- [ocp-async-8,eccef4459ced9eab,e3e736db5cf037b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:02.033 INFO 21062 --- [ocp-schedules-10,5bbb71cfe2f4e1ed,f6fc638c31cce9ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:24:02.034 INFO 21062 --- [ocp-schedules-10,ac0541beb7bb170b,7f658024112fd2c1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:24:02.038 INFO 21062 --- [ocp-schedules-10,ac0541beb7bb170b,7f658024112fd2c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:02.070 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:24:02.070 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:24:02.843 WARN 21062 --- [ocp-async-8,b50bcd7b23b811a4,f70e0a4a152d4504] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:02.892 INFO 21062 --- [ocp-async-2,a0d58c38537533de,29d64d66c170595d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:02.939 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:24:02.939 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:24:02.948 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:24:02.948 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:24:02.948 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:24:02.948 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:24:02.954 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:24:02.954 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:24:02.954 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:24:02.954 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:24:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2201b0cb4626f00d,4a1f3f25981fc7e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:02.998 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:24:03.007 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:24:03.007 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:24:03.007 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:24:03.007 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:24:03.007 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:24:03.015 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:24:03.504 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:03.505 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:03.527 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:24:04.125 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.170875873S 2024-11-04 13:24:04.126 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:24:04.225 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:24:04.226 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:24:04.257 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:24:04.313 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:24:04.313 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:24:04.313 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:24:04.313 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:24:04.313 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:24:04.313 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:24:04.324 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:24:04.494 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] o.s.boot.SpringApplication : Started application in 1.546 seconds (JVM running for 3258.174) 2024-11-04 13:24:04.495 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.546933021S 2024-11-04 13:24:04.843 WARN 21062 --- [ocp-async-2,de45a3b0a13a6ee5,fae31c7a8c8fdaff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:04.892 INFO 21062 --- [ocp-async-6,58afe248e042306c,24e5e44f90735e88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:04.921 INFO 21062 --- [ocp-schedules-5,089d198d807f12eb,76f720839e81d49e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:24:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2cbb92a59932a211,ae13972902a2449d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:06.505 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:06.506 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:06.843 WARN 21062 --- [ocp-async-6,ab31a40cd37f1ef6,419cd9a412d00798] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:06.892 INFO 21062 --- [ocp-async-1,1deb3db4bf9ee72e,b587ea550a347b0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:08.505 INFO 21062 --- [ocp-schedules-8,819430024c7c1af7,4b1bde9178925c20] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:24:08.506 INFO 21062 --- [ocp-schedules-26,ad6d79d9914af75b,2a072b6fa5cbd0d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:24:08.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:24:08.592025+08:00, lastProcessTime=2024-11-04T13:23:58.589893+08:00 2024-11-04 13:24:08.594 INFO 21062 --- [ocp-async-6,b93c905c6237bd1a,ef821eae82d7842f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:24:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:24:08.592025+08:00 2024-11-04 13:24:08.843 WARN 21062 --- [ocp-async-8,8aa6157c5ba3bd46,5f8e63bb72d37927] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:08.892 INFO 21062 --- [ocp-async-4,c59d99bf2ff400b5,833c53b9c95cd23c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:08.983 INFO 21062 --- [prometheus-minute-schedule-1,846516f656b7887f,a64cfc2df7423708] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:09.400 INFO 21062 --- [ocp-schedules-30,afe190a392d6af55,642700f53a2586a2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:24:09.506 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:09.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:09.944 INFO 21062 --- [ocp-schedules-16,4a0b6ffc279d590c,ef0bd561104ab458] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:24:09.950 INFO 21062 --- [ocp-schedules-16,4a0b6ffc279d590c,ef0bd561104ab458] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:24:09.950 INFO 21062 --- [ocp-schedules-16,4a0b6ffc279d590c,ef0bd561104ab458] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:24:10.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:24:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:24:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:24:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:24:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:24:10.319 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:24:10.319 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:24:10.320 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:24:10.382 INFO 21062 --- [ocp-schedules-12,04f7513427ee893d,7f8f9dd5bdb95815] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:24:10.844 WARN 21062 --- [ocp-async-4,d70c18fb377f7cc1,f30351f629fd92fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:10.892 INFO 21062 --- [ocp-async-7,b7bba1faa7d5e502,8435cfe493c9d0b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=19, affectRows=19 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=33, affectRows=33 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=20, affectRows=20 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=35, affectRows=35 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=23, affectRows=23 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=38, affectRows=38 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=19, validCount=19 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=8, affectRows=8 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=8, validCount=8 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=23, validCount=23 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=26, affectRows=26 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=20, validCount=20 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=33, validCount=33 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=38, validCount=38 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=35, validCount=35 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=23, affectRows=23 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=26, validCount=26 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=23, validCount=23 2024-11-04 13:24:11.235 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=51, affectRows=51 2024-11-04 13:24:11.236 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697780, nodeCount=46, affectRows=46 2024-11-04 13:24:11.236 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=44, validCount=44 2024-11-04 13:24:11.236 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697780, nodeCount=50, validCount=50 2024-11-04 13:24:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c1138e7b601f42da,7292531036b2850c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:12.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:12.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:12.844 WARN 21062 --- [ocp-async-9,3dbe70eb619f5c15,7a2e0d7f84fcedfc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:12.892 INFO 21062 --- [ocp-async-3,3a879e59cda41387,9ead145afaf36477] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:13.984 INFO 21062 --- [prometheus-cache-monitor1,3d81d8037d24178e,654179e348428d64] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:24:13.985 INFO 21062 --- [prometheus-cache-monitor1,3d81d8037d24178e,654179e348428d64] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:24:14.844 WARN 21062 --- [ocp-async-1,200642b677fc8020,f763107f453b4d1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:14.893 INFO 21062 --- [ocp-async-5,90971832a2f74154,e7a310528aa2a753] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9571d249360839e1,ef7d02bedb514291] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:15.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:15.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:16.844 WARN 21062 --- [ocp-async-2,4b4eb355abb57771,54a554abb88218c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:16.893 INFO 21062 --- [ocp-async-10,f257480d372a9aec,aabf823def6cd617] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:17.983 INFO 21062 --- [prometheus-minute-schedule-1,a254e613462e6af8,739f26273430a964] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:18.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:18.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:24:18.589695+08:00, lastProcessTime=2024-11-04T13:24:08.592025+08:00 2024-11-04 13:24:18.594 INFO 21062 --- [ocp-async-2,1e44cb1d5337ff71,07bc890cbbe91485] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:24:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:24:18.589695+08:00 2024-11-04 13:24:18.844 WARN 21062 --- [ocp-async-9,8489c400be35c911,46fa60b04601c186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:18.893 INFO 21062 --- [ocp-async-3,8c2e91d0a136c5ab,84efe03938493866] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:19.402 INFO 21062 --- [ocp-schedules-13,77ff0f9e12df52a9,6b425a609492cd44] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:24:20.407 INFO 21062 --- [ocp-schedules-22,17188d5eda8eb324,e9afc474b976f56e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:24:20.845 WARN 21062 --- [ocp-async-1,11b5e9aef3c989e2,5bcb1aeb13f9cc6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:20.893 INFO 21062 --- [ocp-async-5,0a13ff4e5ce4335f,453e09e87309cc71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:20.983 INFO 21062 --- [prometheus-minute-schedule-1,da1ccc1ad9da26a6,8a160b19f12a3a38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:21.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:21.510 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:22.845 WARN 21062 --- [ocp-async-2,84a47d6dda2bfdee,81b3b6f9b5da3da0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:22.894 INFO 21062 --- [ocp-async-10,57aa74a12ce56b16,a195d2ac169c86bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ceb8781362d8d603,4fbe5ab74ba6e1d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:24.511 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:24.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:24.845 WARN 21062 --- [ocp-async-7,e1d00ad70bd04193,540b078d2c3f4371] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:24.894 INFO 21062 --- [ocp-async-6,dff31c05eafd903e,7116275a7a53c286] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:26.845 WARN 21062 --- [ocp-async-3,75560a1472e054e0,51745ddb875fdeee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:26.894 INFO 21062 --- [ocp-async-8,a240875e0f68ce57,0e2734f5d4f081fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:26.983 INFO 21062 --- [prometheus-minute-schedule-1,051044c3957a7d35,650ca3dda303d6bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c69fb62d7d58d751,1a9ae5760fe25a1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:24:27.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:27.513 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:28.498 INFO 21062 --- [ocp-schedules-8,763991888904343d,890915b6dad93eb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:24:28.499 INFO 21062 --- [ocp-schedules-18,2a593384e08b97fa,ae9362edc326bd84] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:24:28.500 INFO 21062 --- [ocp-schedules-13,1a4fe9f359986c3e,4f2c064e32897962] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:24:28.500 INFO 21062 --- [ocp-schedules-14,7fd0f1c808625129,b4547843d1d5babe] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:24:28.502 INFO 21062 --- [ocp-schedules-17,2b4761007ca9d861,fdb7fb713f8cf05e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:24:28.503 INFO 21062 --- [ocp-schedules-8,763991888904343d,890915b6dad93eb7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:24:28.503 INFO 21062 --- [ocp-schedules-13,1a4fe9f359986c3e,4f2c064e32897962] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:28.504 INFO 21062 --- [ocp-schedules-14,7fd0f1c808625129,b4547843d1d5babe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:24:28.505 INFO 21062 --- [ocp-schedules-11,7870ac165d5c96c7,ccf5b7ea7cc2a5ae] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:24:28.505 INFO 21062 --- [ocp-schedules-18,2a593384e08b97fa,ae9362edc326bd84] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:24:28.511 INFO 21062 --- [ocp-schedules-29,ad4a357441a7e183,4b6b337bf917d36f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:24:28.511 INFO 21062 --- [ocp-schedules-29,ad4a357441a7e183,4b6b337bf917d36f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:24:28.511 INFO 21062 --- [ocp-schedules-21,42a76126b1d612df,9e68702bc200ea67] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:24:28.512 INFO 21062 --- [ocp-schedules-21,42a76126b1d612df,9e68702bc200ea67] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:24:28.512 INFO 21062 --- [ocp-schedules-1,86df9deb37322917,bca0b1b1d3a20765] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:24:28.512 INFO 21062 --- [ocp-schedules-1,86df9deb37322917,bca0b1b1d3a20765] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:24:28.553 INFO 21062 --- [ocp-schedules-12,252cebc2428148b8,081da77edb4bc301] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:24:28.553+08:00, tenantSize=0 2024-11-04 13:24:28.565 INFO 21062 --- [ocp-schedules-6,7bad6c6e4d795d68,5240e9b05930c39e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:24:28.565 INFO 21062 --- [ocp-schedules-6,7bad6c6e4d795d68,5240e9b05930c39e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:24:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:24:28.589581+08:00, lastProcessTime=2024-11-04T13:24:18.589695+08:00 2024-11-04 13:24:28.594 INFO 21062 --- [ocp-async-8,4d3e755b8ef6f683,f1b28e8413f8ac9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:24:28.594 INFO 21062 --- [ocp-schedules-4,31755eb2be957ad5,60047d099b3f1558] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:24:28.595 INFO 21062 --- [ocp-schedules-4,31755eb2be957ad5,60047d099b3f1558] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:24:28.610 INFO 21062 --- [ocp-schedules-25,a739b72f16cae329,1730b7f2cae0ad79] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 13:24:28.610 INFO 21062 --- [ocp-schedules-25,a739b72f16cae329,1730b7f2cae0ad79] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 13:24:28.685 INFO 21062 --- [ocp-schedules-30,ff29cfb35b6c143e,11d4d482b365c786] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:24:28.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:24:28.589581+08:00 2024-11-04 13:24:28.831 INFO 21062 --- [ocp-schedules-18,86ec1168a7068ab2,0de3d02aa36dca23] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:24:28.831 INFO 21062 --- [ocp-schedules-18,86ec1168a7068ab2,0de3d02aa36dca23] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:24:28.844 INFO 21062 --- [ocp-schedules-17,f2fa12092ec3c31b,d801d17a4da965d1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:24:28.845 WARN 21062 --- [ocp-async-10,be8a9377a81cbfa8,b15a103789f6a10c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:28.871 INFO 21062 --- [ocp-schedules-21,8588a828513a4590,66ed21bbb410b592] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:24:28.876 INFO 21062 --- [ocp-schedules-21,8588a828513a4590,66ed21bbb410b592] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:24:28.876 INFO 21062 --- [ocp-schedules-21,8588a828513a4590,66ed21bbb410b592] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:24:28.894 INFO 21062 --- [ocp-async-9,a8529fdcb96dd5ac,8143168368c64470] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:29.034 INFO 21062 --- [ocp-schedules-22,36748f11a007afb0,939e0dc5133bb75f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:24:29.034 INFO 21062 --- [ocp-schedules-22,36748f11a007afb0,939e0dc5133bb75f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:24:29.041 INFO 21062 --- [ocp-schedules-22,36748f11a007afb0,939e0dc5133bb75f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:24:29.041 INFO 21062 --- [ocp-schedules-22,36748f11a007afb0,939e0dc5133bb75f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:24:29.046 INFO 21062 --- [ocp-schedules-22,36748f11a007afb0,939e0dc5133bb75f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:24:29.046 INFO 21062 --- [ocp-schedules-22,36748f11a007afb0,939e0dc5133bb75f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:24:29.405 INFO 21062 --- [ocp-schedules-3,08ea36877233cfb2,61503fe23ec5c340] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:24:29.949 INFO 21062 --- [ocp-schedules-18,8d7a50ca977e2108,152e6fc25e266eef] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:24:29.949 INFO 21062 --- [ocp-schedules-18,8d7a50ca977e2108,152e6fc25e266eef] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:24:29.953 INFO 21062 --- [ocp-schedules-17,9e362a605358fdd6,e4d84e4857d2acef] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:24:29.960 INFO 21062 --- [ocp-schedules-17,9e362a605358fdd6,e4d84e4857d2acef] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:24:29.960 INFO 21062 --- [ocp-schedules-17,9e362a605358fdd6,e4d84e4857d2acef] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:24:29.976 INFO 21062 --- [ocp-schedules-18,8d7a50ca977e2108,152e6fc25e266eef] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:24:29.976 INFO 21062 --- [ocp-schedules-18,8d7a50ca977e2108,152e6fc25e266eef] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:24:29.978 INFO 21062 --- [ocp-schedules-20,20a8f33a6ef5cea5,cb69ecea322e1d8f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:24:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e4221a5c5b6ff717,0816887de6f6a54b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:30.099 INFO 21062 --- [ocp-schedules-21,ee7bafcccbfe6b2d,f95accead4b153df] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:24:30.099 INFO 21062 --- [ocp-schedules-21,ee7bafcccbfe6b2d,f95accead4b153df] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:24:30.099 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:24:30.099 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:24:30.099 INFO 21062 --- [ocp-schedules-1,1f0cdaebfa512021,08f894725dd78c78] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:24:30.099+08:00, clusters=[1] 2024-11-04 13:24:30.100 INFO 21062 --- [ocp-schedules-27,85e3fcd77ebfcf8e,c8f5416a11d459da] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:24:30.101 INFO 21062 --- [ocp-schedules-1,1f0cdaebfa512021,08f894725dd78c78] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:24:30.104 INFO 21062 --- [ocp-schedules-27,85e3fcd77ebfcf8e,c8f5416a11d459da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:30.108 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:24:30.108 INFO 21062 --- [ocp-schedules-21,ee7bafcccbfe6b2d,f95accead4b153df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:24:30.109 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:24:30.110 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:24:30.110 WARN 21062 --- [ocp-schedules-21,ee7bafcccbfe6b2d,f95accead4b153df] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:24:30.110 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:24:30.111 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:24:30.111 INFO 21062 --- [ocp-schedules-21,ee7bafcccbfe6b2d,f95accead4b153df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:24:30.111 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:24:30.112 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:24:30.112 WARN 21062 --- [ocp-schedules-21,ee7bafcccbfe6b2d,f95accead4b153df] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:24:30.112 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:24:30.120 ERROR 21062 --- [ocp-schedules-21,ee7bafcccbfe6b2d,f95accead4b153df] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:24:30.159 ERROR 21062 --- [ocp-schedules-27,85e3fcd77ebfcf8e,c8f5416a11d459da] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:24:30.159 INFO 21062 --- [ocp-schedules-27,85e3fcd77ebfcf8e,c8f5416a11d459da] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:24:30.228 INFO 21062 --- [ocp-schedules-2,d36799966c9089fc,1abad189b08d746c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:24:30.228+08:00, clusters=[1] 2024-11-04 13:24:30.232 INFO 21062 --- [ocp-schedules-2,d36799966c9089fc,1abad189b08d746c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:24:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:24:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:24:30.449 INFO 21062 --- [ocp-schedules-12,3300a33c2797c966,a556d975ec7e8989] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:24:30.513 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:30.514 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:30.845 WARN 21062 --- [ocp-async-9,4b1f678342bba657,4f48f7cd48463852] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:30.894 INFO 21062 --- [ocp-async-1,f6d2c392083d7c3d,e234bfa2addbcda4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:32.070 INFO 21062 --- [ocp-schedules-10,ac0541beb7bb170b,7f658024112fd2c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:24:32.070 INFO 21062 --- [ocp-schedules-10,f4c542df7fb1d1b4,daa86aab91fe42af] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:24:32.074 INFO 21062 --- [ocp-schedules-10,f4c542df7fb1d1b4,daa86aab91fe42af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:32.103 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:24:32.103 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:24:32.846 WARN 21062 --- [ocp-async-1,56e4f3f3f6c42fcc,08921c2235908585] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:32.894 INFO 21062 --- [ocp-async-2,e7be12a969175462,feb35bfede5f3c5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:32.983 INFO 21062 --- [prometheus-minute-schedule-1,bbe5d6eb921c8613,ca5f84584b9e3fbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:33.285 INFO 21062 --- [ocp-schedules-20,5e9a725263bf8c81,94791765de816c86] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:24:33.289 INFO 21062 --- [ocp-schedules-20,5e9a725263bf8c81,94791765de816c86] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:33.376 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:24:33.378 ERROR 21062 --- [ocp-schedules-20,5e9a725263bf8c81,94791765de816c86] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:24:33.378 INFO 21062 --- [ocp-schedules-20,5e9a725263bf8c81,94791765de816c86] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:24:33.514 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:33.515 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:34.846 WARN 21062 --- [ocp-async-2,e0cf4490a665d1c5,6df605a31913a49e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:34.895 INFO 21062 --- [ocp-async-7,d2b19828adde2c8c,eadc54c9ffd7533c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:35.983 INFO 21062 --- [prometheus-minute-schedule-1,446ab498d192bb4a,5c066a0a31b296d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:36.515 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:36.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:36.846 WARN 21062 --- [ocp-async-7,f5e7b8670ff8c637,dccfe35839218a31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:36.895 INFO 21062 --- [ocp-async-6,4baa69f8ca46334f,0c6a154b9b5e7b05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:38.504 INFO 21062 --- [ocp-schedules-14,7fd0f1c808625129,b4547843d1d5babe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:24:38.504 INFO 21062 --- [ocp-schedules-13,1a4fe9f359986c3e,4f2c064e32897962] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:24:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:24:38.589737+08:00, lastProcessTime=2024-11-04T13:24:28.589581+08:00 2024-11-04 13:24:38.594 INFO 21062 --- [ocp-async-9,ce177e195bff241e,26c19dc1f5669fb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:24:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:24:38.589737+08:00 2024-11-04 13:24:38.846 WARN 21062 --- [ocp-async-8,bd2358559e866f8e,ba3a5688e13f4a7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:38.895 INFO 21062 --- [ocp-async-10,bec9c7a287dbe54d,981226da1670f18f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:38.983 INFO 21062 --- [prometheus-minute-schedule-1,908dcea3ea9bbdc5,2fe4d912cd810cf3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:39.409 INFO 21062 --- [ocp-schedules-9,b2c5a224db2571af,1941bda20748f8e7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:24:39.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:39.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:24:40.004 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:24:40.131 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:24:40.131 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:24:40.133 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.134 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:24:40.135 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.135 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.136 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.136 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:24:40.151 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.152 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.152 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.153 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:24:40.166 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.167 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.167 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.167 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:24:40.180 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.181 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.181 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.182 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:24:40.195 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.196 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.196 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.197 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:24:40.202 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.203 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.203 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.204 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:24:40.208 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.209 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.209 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.210 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:24:40.214 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.215 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.216 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.216 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:24:40.222 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.222 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.223 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.223 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:24:40.229 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.229 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.230 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.230 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:24:40.235 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.235 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.236 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.237 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:24:40.246 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.246 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.247 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.247 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:24:40.260 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.260 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.261 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.261 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:24:40.267 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.267 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.267 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.268 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:24:40.274 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.274 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.275 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.275 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:24:40.280 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.281 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.281 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.282 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:24:40.286 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.287 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.287 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.288 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:24:40.292 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.292 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.293 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.293 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:24:40.297 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.298 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.298 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.299 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:24:40.307 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.307 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.308 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.308 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:24:40.312 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.313 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.313 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.314 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:24:40.319 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.319 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.319 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.320 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:24:40.324 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.324 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.325 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.325 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:24:40.330 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.330 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.330 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.331 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:24:40.335 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.336 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.337 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.337 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:24:40.341 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.341 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.342 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.342 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:24:40.347 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.348 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.348 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.349 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:24:40.353 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.353 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.354 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.354 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:24:40.359 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.359 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.360 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.360 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:24:40.365 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.365 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.366 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.366 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:24:40.370 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.371 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.371 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.372 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:24:40.376 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.377 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.377 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.377 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:24:40.382 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.382 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.383 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.383 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:24:40.387 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.388 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.388 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.389 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:24:40.396 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.396 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.397 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.397 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:24:40.403 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.404 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.404 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.405 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:24:40.410 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.411 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.411 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.412 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:24:40.417 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.417 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.418 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.418 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:24:40.422 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.422 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.423 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.423 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:24:40.428 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.428 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.429 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.430 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:24:40.434 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.434 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.435 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.435 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:24:40.439 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.440 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.440 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.440 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:24:40.449 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.450 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.450 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.451 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:24:40.470 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.471 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.471 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.472 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:24:40.473 INFO 21062 --- [ocp-schedules-18,bddf9183cedafdaf,67afb78b27f8d47f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:24:40.478 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.478 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.479 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.479 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:24:40.486 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.486 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.487 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.487 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:24:40.494 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.494 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.495 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.495 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:24:40.500 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.500 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.501 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.501 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:24:40.505 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.506 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.506 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.507 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:24:40.511 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:24:40.511 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:24:40.512 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.513 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:24:40.514 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.515 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.515 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.516 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:24:40.522 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.522 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:24:40.523 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:40.523 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:24:40.528 INFO 21062 --- [ocp-schedules-21,669d0d27fc7846b9,c00e388c83d7554b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:24:40.847 WARN 21062 --- [ocp-async-10,821cf46582104516,aed041aae3020ed0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:40.895 INFO 21062 --- [ocp-async-3,968aec9672988d89,50eca4f1ff40966c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:24:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:24:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:24:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:24:41.854 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:24:41.854 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 13:24:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c762d3704b93f282,c63ad5214715f4e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:42.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:42.518 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:42.847 WARN 21062 --- [ocp-async-3,7b97eb74ff243790,16998135e93b1604] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:42.895 INFO 21062 --- [ocp-async-1,a54d9807c12870c8,45678753e23ba795] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:44.847 WARN 21062 --- [ocp-async-1,8b6b946bffb9ef1e,2b9fae4555ba8811] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:44.896 INFO 21062 --- [ocp-async-2,b1f7796850fd2de2,497887f0793fb071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:44.983 INFO 21062 --- [prometheus-minute-schedule-1,c9a1c475737b0b9b,71f683dfe99c8864] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:45.518 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:45.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:46.847 WARN 21062 --- [ocp-async-2,c69bea6498abf88a,b6ac75f8c89782c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:46.896 INFO 21062 --- [ocp-async-7,db71b9d03e50e67c,5e702689ceccf73a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:47.983 INFO 21062 --- [prometheus-minute-schedule-1,a0b85d723e71e2db,edcb40c46597ef19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:48.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:48.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:24:48.591007+08:00, lastProcessTime=2024-11-04T13:24:38.589737+08:00 2024-11-04 13:24:48.594 INFO 21062 --- [ocp-async-10,085b8ba5a4e25565,37dc01b2b8ac3749] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:24:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:24:48.591007+08:00 2024-11-04 13:24:48.776 INFO 21062 --- [ocp-schedules-23,9d7f8d522a33530c,d22adefe71814efc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:24:48.779 INFO 21062 --- [ocp-schedules-23,9d7f8d522a33530c,d22adefe71814efc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:48.780 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:24:48.847 WARN 21062 --- [ocp-async-3,1c233198b77a600b,d0e86b20b1a6ce53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:48.896 INFO 21062 --- [ocp-async-1,22d3f633b33818cd,165d91718ad3c087] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:49.411 INFO 21062 --- [ocp-schedules-18,5d6f25d7f39e0bde,ac1abcebae749fe3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:24:49.963 INFO 21062 --- [ocp-schedules-16,ea4c806798bc07bf,cf2b5e0995c826a2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:24:49.968 INFO 21062 --- [ocp-schedules-16,ea4c806798bc07bf,cf2b5e0995c826a2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:24:49.968 INFO 21062 --- [ocp-schedules-16,ea4c806798bc07bf,cf2b5e0995c826a2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:24:50.514 INFO 21062 --- [ocp-schedules-12,c9d9e09d13aa92d9,537810acd0dab1b2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:24:50.847 WARN 21062 --- [ocp-async-1,57d8fc6db1d7c0c3,08b4f43f03b2311d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:50.896 INFO 21062 --- [ocp-async-2,bb7c2970b8aa3101,56c3fd2201b40aab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a7b82187536e77c7,fdd3a47834089343] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:51.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:51.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:52.848 WARN 21062 --- [ocp-async-2,5c758eac7e6b9d37,1fba68998e59c1dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:52.896 INFO 21062 --- [ocp-async-7,972d70fc226b4a48,39e424fff8e6ec40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9ede214eac85965c,9fbdc34f71ecf178] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:54.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:54.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:54.848 WARN 21062 --- [ocp-async-7,cfa9183d4f7967b9,90367dd6f8538a70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:54.896 INFO 21062 --- [ocp-async-6,101771cde86d3392,18564abf0b1909a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:56.848 WARN 21062 --- [ocp-async-6,0acd8e2c78c9b2b3,4f90c04b1177cdd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:56.896 INFO 21062 --- [ocp-async-8,1517fa2c93a0cd3d,1e035a87a22eb1dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d2a97325df09f842,4f72972f2562532c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:24:57.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:24:57.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:24:58.500 INFO 21062 --- [ocp-schedules-6,9c438e33cdafd676,a314baffc518bfa4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:24:58.500 INFO 21062 --- [ocp-schedules-4,3aafea0153eaaf9f,ea02853b7eb11ff0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:24:58.502 INFO 21062 --- [ocp-schedules-13,a8557b2aae9d8575,af022cbcf5971e6d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:24:58.504 INFO 21062 --- [ocp-schedules-6,9c438e33cdafd676,a314baffc518bfa4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:58.504 INFO 21062 --- [ocp-schedules-4,3aafea0153eaaf9f,ea02853b7eb11ff0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:24:58.504 INFO 21062 --- [ocp-schedules-20,1b1b1b749e8ee69b,fca23ec1fe4bbd4a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:24:58.507 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:24:58.510 INFO 21062 --- [ocp-schedules-21,d8f262078cd4cfbd,7e3ae567982751c7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:24:58.510 INFO 21062 --- [ocp-schedules-14,befe78c1d3ed774f,d2032ea833cda4ef] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:24:58.510 INFO 21062 --- [ocp-schedules-14,befe78c1d3ed774f,d2032ea833cda4ef] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:24:58.510 INFO 21062 --- [ocp-schedules-21,d8f262078cd4cfbd,7e3ae567982751c7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:24:58.510 INFO 21062 --- [ocp-schedules-12,131c96428bdcabcb,550196c39b4971eb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:24:58.511 INFO 21062 --- [ocp-schedules-12,131c96428bdcabcb,550196c39b4971eb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:24:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:24:58.588636+08:00, lastProcessTime=2024-11-04T13:24:48.591007+08:00 2024-11-04 13:24:58.595 INFO 21062 --- [ocp-async-6,c4d53e50ff2e3e48,6feeb7ec9b2f314b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:24:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:24:58.588636+08:00 2024-11-04 13:24:58.780 INFO 21062 --- [ocp-schedules-23,9d7f8d522a33530c,d22adefe71814efc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:24:58.847 INFO 21062 --- [ocp-schedules-28,b71eb7ce78692dbb,67e58def0214410d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:24:58.848 WARN 21062 --- [ocp-async-2,48530eb5e3ec9fc4,d4700c26545a676b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:24:58.897 INFO 21062 --- [ocp-async-4,77435e28dfd86a06,0fbbdb59c627501f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:24:59.414 INFO 21062 --- [ocp-schedules-30,78fce623660161f4,4ff64cee035ee4e3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:24:59.983 INFO 21062 --- [prometheus-minute-schedule-1,91c4ddf316495c65,8b942bca655bb9f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697900, interval=60 2024-11-04 13:25:00.203 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,813d3e425308a92f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:20:00.0, nextRun=2024-11-04 13:25:00.0, currentDate=Mon Nov 04 13:25:00 CST 2024 2024-11-04 13:25:00.204 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,813d3e425308a92f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:20:00.0, nextRunTime=2024-11-04 13:25:00.0} 2024-11-04 13:25:00.229 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,813d3e425308a92f] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:25:00.233 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,813d3e425308a92f] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:25:00.233 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,813d3e425308a92f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:25:00.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:00.523 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:00.538 INFO 21062 --- [ocp-schedules-13,a90d68b31a4ae5fb,e88f64faa0c12195] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:25:00.848 WARN 21062 --- [ocp-async-4,09a63b45df5d909f,8c52cce3da20558f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:00.897 INFO 21062 --- [ocp-async-9,9ccc2db36e3b8078,1dd306822b1860d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:02.103 INFO 21062 --- [ocp-schedules-10,f4c542df7fb1d1b4,daa86aab91fe42af] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:25:02.103 INFO 21062 --- [ocp-schedules-10,54121dda27fe464b,c44b492b0dba0944] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:25:02.107 INFO 21062 --- [ocp-schedules-10,54121dda27fe464b,c44b492b0dba0944] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:02.136 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:25:02.136 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:25:02.848 WARN 21062 --- [ocp-async-9,7b88c599e5bb822c,25b5e0bf765cb846] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:02.897 INFO 21062 --- [ocp-async-5,229772c01c9f5e81,aa385659821a53b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1d970b9969ea4553,a16c306018d15af4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:03.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:03.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:04.849 WARN 21062 --- [ocp-async-5,9b4b856752732ad6,e5510216cb060039] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:04.897 INFO 21062 --- [ocp-async-10,13a6b9b9797c40e7,775732deb846ce87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:05.983 INFO 21062 --- [prometheus-minute-schedule-1,248da184b98d8d1e,11b64293a06fcc7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:06.525 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:06.525 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:06.849 WARN 21062 --- [ocp-async-7,cd81d9a93b1d6d3c,9c72f79eedd881a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:06.897 INFO 21062 --- [ocp-async-3,b281fc5b9b6150b4,42b92394aacd4bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:08.508 INFO 21062 --- [ocp-schedules-4,3aafea0153eaaf9f,ea02853b7eb11ff0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:25:08.508 INFO 21062 --- [ocp-schedules-6,9c438e33cdafd676,a314baffc518bfa4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:25:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:25:08.589950+08:00, lastProcessTime=2024-11-04T13:24:58.588636+08:00 2024-11-04 13:25:08.595 INFO 21062 --- [ocp-async-7,8d2bdd64b85e4564,4b3cf6297f9c09ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:25:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:25:08.589950+08:00 2024-11-04 13:25:08.849 WARN 21062 --- [ocp-async-9,988cbd6036fc2142,da2e68e695263ef4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:08.898 INFO 21062 --- [ocp-async-8,42f3987a99fafc70,312dcd35c2868fa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:08.983 INFO 21062 --- [prometheus-minute-schedule-1,bd6a7efc447c54f0,1272089561c66993] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:09.417 INFO 21062 --- [ocp-schedules-9,b4f570348bbb5c7c,9de36bc9b6176527] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:25:09.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:09.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:09.971 INFO 21062 --- [ocp-schedules-17,4ff8aaebe567ef70,331718b868dcbeb8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:25:09.977 INFO 21062 --- [ocp-schedules-17,4ff8aaebe567ef70,331718b868dcbeb8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:25:09.977 INFO 21062 --- [ocp-schedules-17,4ff8aaebe567ef70,331718b868dcbeb8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:25:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:25:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:25:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:25:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:25:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:25:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:25:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:25:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:25:10.579 INFO 21062 --- [ocp-schedules-11,eeebaae15ad21b7e,f21b45557a45f917] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:25:10.849 WARN 21062 --- [ocp-async-5,ecab2f16c92f62b5,1b7bd695c2b7e912] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:10.898 INFO 21062 --- [ocp-async-10,78637e76bf311f7b,03bc01a7cd6f996b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=4, affectRows=4 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=3, validCount=3 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=8, affectRows=8 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=9, affectRows=9 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=8, validCount=8 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=9, validCount=9 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=19, affectRows=19 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=19, validCount=19 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=13, affectRows=13 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=29, affectRows=29 2024-11-04 13:25:11.239 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=13, validCount=13 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=29, validCount=29 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=24, affectRows=24 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=24, validCount=24 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=21, affectRows=21 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=38, affectRows=38 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=21, validCount=21 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=37, validCount=37 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=61, affectRows=61 2024-11-04 13:25:11.240 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=45, affectRows=45 2024-11-04 13:25:11.241 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=45, validCount=45 2024-11-04 13:25:11.241 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=61, validCount=61 2024-11-04 13:25:11.242 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697840, nodeCount=51, affectRows=51 2024-11-04 13:25:11.242 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697840, nodeCount=50, validCount=50 2024-11-04 13:25:11.983 INFO 21062 --- [prometheus-minute-schedule-1,2ee3c4b5a989370b,4285096afc67e918] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:12.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:12.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:12.849 WARN 21062 --- [ocp-async-7,9ecbf2eae904dc18,8642cd58319f614f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:12.898 INFO 21062 --- [ocp-async-3,a8e3fe469cd40842,1e404e7ca268338c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:13.984 INFO 21062 --- [prometheus-cache-monitor1,549569f4c21d15c6,9277a04551cf6c08] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:25:13.985 INFO 21062 --- [prometheus-cache-monitor1,549569f4c21d15c6,9277a04551cf6c08] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:25:14.514 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:21:14.497, end=2024-11-04T13:23:14.512} 2024-11-04 13:25:14.524 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:25:14.524 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:25:14.850 WARN 21062 --- [ocp-async-1,36dde75209f4b1d9,57a38d176cc74ad7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:14.898 INFO 21062 --- [ocp-async-6,868140539b78c5b0,ed96d9259e3cbbac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:14.983 INFO 21062 --- [prometheus-minute-schedule-1,52e2a911c0e5d0c5,d352e3c9447386f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:15.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:15.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:16.850 WARN 21062 --- [ocp-async-8,072cc6555d53ef3c,67525d0c3c83fbc3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:16.898 INFO 21062 --- [ocp-async-2,f91215d7a58b4d6d,43428b56e93b165d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:17.983 INFO 21062 --- [prometheus-minute-schedule-1,28c8d422aee7df8f,20b9da730200bddf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:18.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:18.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:25:18.590332+08:00, lastProcessTime=2024-11-04T13:25:08.589950+08:00 2024-11-04 13:25:18.595 INFO 21062 --- [ocp-async-8,f8f49bb23cf3e707,44fa8b64e636c3c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:25:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:25:18.590332+08:00 2024-11-04 13:25:18.850 WARN 21062 --- [ocp-async-7,3cc6901d55ccb167,06cc69cac932f1b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:18.899 INFO 21062 --- [ocp-async-3,da3b4c0a734bc8a5,37ad7ffd4337af87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:19.420 INFO 21062 --- [ocp-schedules-16,b470b16b86577a22,5c1f447aa2ba4dbd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:25:20.605 INFO 21062 --- [ocp-schedules-8,4dc50e786644db75,247ba9ff8189c2a6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:25:20.850 WARN 21062 --- [ocp-async-1,c581474026da107b,b39b307fd6958e81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:20.899 INFO 21062 --- [ocp-async-6,ff093cf79b500200,1b19859e9ee64250] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:20.983 INFO 21062 --- [prometheus-minute-schedule-1,33a8b79942ef0d72,0bafeec46ad88d62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:21.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:21.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:22.850 WARN 21062 --- [ocp-async-8,58805bd90279715e,2e1400273e7601a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:22.899 INFO 21062 --- [ocp-async-2,3fcabbe5d478dc27,029e89b6b48a184c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:23.983 INFO 21062 --- [prometheus-minute-schedule-1,0fb26e74f24740f7,0aa7d438e091a37e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:24.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:24.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:24.850 WARN 21062 --- [ocp-async-10,3fc638a7c5e03b1b,f2613189a45847ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:24.899 INFO 21062 --- [ocp-async-4,2a6b4735007377ab,6374e5b3df12f280] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:26.851 WARN 21062 --- [ocp-async-3,c64b04cbf1739b6d,52d810ab9ea727ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:26.899 INFO 21062 --- [ocp-async-9,5332ee773a75af5d,7b98cef3a3c5c7d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:26.983 INFO 21062 --- [prometheus-minute-schedule-1,83480b435cb30b7d,c3d4d8466b0b81d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4fdf7d98208a3d6b,f9e7f3a424f37c9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:25:27.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:27.532 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:28.498 INFO 21062 --- [ocp-schedules-7,b0650b35de493bfd,1295a3b8777a5663] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:25:28.499 INFO 21062 --- [ocp-schedules-30,7b210bfcbab86c9e,a4f986f2632c1b58] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:25:28.500 INFO 21062 --- [ocp-schedules-9,8e6d7b4f91ffcd43,ce71201ac2014e4e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:25:28.500 INFO 21062 --- [ocp-schedules-8,a397f4e223ffccb9,3d4d433879b58dc3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:25:28.502 INFO 21062 --- [ocp-schedules-7,b0650b35de493bfd,1295a3b8777a5663] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:25:28.502 INFO 21062 --- [ocp-schedules-4,2dcc535aa9beb77d,15226e51d8728d9a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:25:28.503 INFO 21062 --- [ocp-schedules-9,8e6d7b4f91ffcd43,ce71201ac2014e4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:25:28.504 INFO 21062 --- [ocp-schedules-8,a397f4e223ffccb9,3d4d433879b58dc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:28.504 INFO 21062 --- [ocp-schedules-16,57268fc3df92887f,276303083c5d8bd8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:25:28.505 INFO 21062 --- [ocp-schedules-30,7b210bfcbab86c9e,a4f986f2632c1b58] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:25:28.510 INFO 21062 --- [ocp-schedules-13,3def12673a147ac4,6b0b10df2435fd2e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:25:28.510 INFO 21062 --- [ocp-schedules-13,3def12673a147ac4,6b0b10df2435fd2e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:25:28.510 INFO 21062 --- [ocp-schedules-24,d4a2718cefcb9e89,491cd39e1493f1f8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:25:28.510 INFO 21062 --- [ocp-schedules-24,d4a2718cefcb9e89,491cd39e1493f1f8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:25:28.511 INFO 21062 --- [ocp-schedules-12,8aedc93e24257259,c22b6e3650379cc4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:25:28.511 INFO 21062 --- [ocp-schedules-12,8aedc93e24257259,c22b6e3650379cc4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:25:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:25:28.588461+08:00, lastProcessTime=2024-11-04T13:25:18.590332+08:00 2024-11-04 13:25:28.595 INFO 21062 --- [ocp-async-9,6f98a493a35832d9,f3fd3fe75375b411] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:25:28.688 INFO 21062 --- [ocp-schedules-2,bc09b8908ab78649,7d056b27713541e2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:25:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:25:28.588461+08:00 2024-11-04 13:25:28.835 INFO 21062 --- [ocp-schedules-22,c9d4a018de85e377,a905421fbf2e7244] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:25:28.835 INFO 21062 --- [ocp-schedules-22,c9d4a018de85e377,a905421fbf2e7244] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:25:28.849 INFO 21062 --- [ocp-schedules-26,175c02d1efaeb73a,a063e92d5a579b86] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:25:28.851 WARN 21062 --- [ocp-async-5,326a0305c2feddeb,dcf4976089cef9da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:28.876 INFO 21062 --- [ocp-schedules-30,939d4c557ee287df,3692457ef817a831] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:25:28.881 INFO 21062 --- [ocp-schedules-30,939d4c557ee287df,3692457ef817a831] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:25:28.881 INFO 21062 --- [ocp-schedules-30,939d4c557ee287df,3692457ef817a831] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:25:28.899 INFO 21062 --- [ocp-async-7,a44a368da94fb3cb,5694ebf83973a84b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:28.902 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:25:28.902 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:25:28.910 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:25:28.910 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:25:28.910 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:25:28.921 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:25:28.921 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:25:28.921 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:25:28.925 INFO 21062 --- [ocp-schedules-16,d60fa4779201ed7d,e7b771937847bb82] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:25:29.050 INFO 21062 --- [ocp-schedules-12,2d5083ca60b58337,8935797c212817c2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:25:29.050 INFO 21062 --- [ocp-schedules-12,2d5083ca60b58337,8935797c212817c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:25:29.051 INFO 21062 --- [ocp-schedules-12,2d5083ca60b58337,8935797c212817c2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:25:29.051 INFO 21062 --- [ocp-schedules-12,2d5083ca60b58337,8935797c212817c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:25:29.052 INFO 21062 --- [ocp-schedules-12,2d5083ca60b58337,8935797c212817c2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:25:29.052 INFO 21062 --- [ocp-schedules-12,2d5083ca60b58337,8935797c212817c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:25:29.423 INFO 21062 --- [ocp-schedules-25,62387d670495942a,50230162f343dc6d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:25:29.976 INFO 21062 --- [ocp-schedules-20,0b5c64ba154e2959,be3c0c92ba8f0492] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:25:29.976 INFO 21062 --- [ocp-schedules-20,0b5c64ba154e2959,be3c0c92ba8f0492] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:25:29.980 INFO 21062 --- [ocp-schedules-17,9f25ab67467625b6,27d99501be85058c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:25:29.983 INFO 21062 --- [prometheus-minute-schedule-1,98642a170e4e4f38,ac0c1888294a5f02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:29.987 INFO 21062 --- [ocp-schedules-17,9f25ab67467625b6,27d99501be85058c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:25:29.987 INFO 21062 --- [ocp-schedules-17,9f25ab67467625b6,27d99501be85058c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:25:30.004 INFO 21062 --- [ocp-schedules-20,0b5c64ba154e2959,be3c0c92ba8f0492] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:25:30.004 INFO 21062 --- [ocp-schedules-20,0b5c64ba154e2959,be3c0c92ba8f0492] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:25:30.004 INFO 21062 --- [ocp-schedules-30,1ea7624ab3b36b66,e363727a40e67f8d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:25:30.127 INFO 21062 --- [ocp-schedules-16,6bfd7fc25b73eed0,10db6ba1c886e2e8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:25:30.127 INFO 21062 --- [ocp-schedules-16,6bfd7fc25b73eed0,10db6ba1c886e2e8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:25:30.127 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:25:30.128 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:25:30.131 INFO 21062 --- [ocp-schedules-13,12f5f3a2092473a0,f58bb2691c7f0b24] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:25:30.131+08:00, clusters=[1] 2024-11-04 13:25:30.132 INFO 21062 --- [ocp-schedules-13,12f5f3a2092473a0,f58bb2691c7f0b24] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:25:30.137 INFO 21062 --- [ocp-schedules-16,6bfd7fc25b73eed0,10db6ba1c886e2e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:25:30.138 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:25:30.138 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:25:30.139 WARN 21062 --- [ocp-schedules-16,6bfd7fc25b73eed0,10db6ba1c886e2e8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:25:30.139 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:25:30.139 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:25:30.139 INFO 21062 --- [ocp-schedules-16,6bfd7fc25b73eed0,10db6ba1c886e2e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:25:30.140 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:25:30.140 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:25:30.141 WARN 21062 --- [ocp-schedules-16,6bfd7fc25b73eed0,10db6ba1c886e2e8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:25:30.141 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:25:30.141 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:25:30.147 ERROR 21062 --- [ocp-schedules-16,6bfd7fc25b73eed0,10db6ba1c886e2e8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:25:30.258 INFO 21062 --- [ocp-schedules-14,f970dcdda6f6631a,2c506063fe663dbc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:25:30.258+08:00, clusters=[1] 2024-11-04 13:25:30.261 INFO 21062 --- [ocp-schedules-14,f970dcdda6f6631a,2c506063fe663dbc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:25:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:25:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:25:30.532 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:30.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:30.643 INFO 21062 --- [ocp-schedules-27,1405db4ead0b0107,944368c45ccd7597] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:25:30.851 WARN 21062 --- [ocp-async-7,14f6268b75b0cbdc,7a3d3619ecb0d630] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:30.900 INFO 21062 --- [ocp-async-1,70db0b68a0fda2b8,1a7bb150fd273114] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:32.137 INFO 21062 --- [ocp-schedules-10,54121dda27fe464b,c44b492b0dba0944] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:25:32.137 INFO 21062 --- [ocp-schedules-10,d4b987d3d83d6e6f,a3196322f36f1db3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:25:32.141 INFO 21062 --- [ocp-schedules-10,d4b987d3d83d6e6f,a3196322f36f1db3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:32.172 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:25:32.172 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:25:32.851 WARN 21062 --- [ocp-async-1,f17ac07eeb779e5f,327d7352c6f866cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:32.900 INFO 21062 --- [ocp-async-8,55c452027c404f26,b0de548d4eaee619] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:32.983 INFO 21062 --- [prometheus-minute-schedule-1,2428a0ea2c4b03b0,71d16e9e9cf60f31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:33.378 INFO 21062 --- [ocp-schedules-17,096f1ff5af316832,c351313223f86d9e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:25:33.383 INFO 21062 --- [ocp-schedules-17,096f1ff5af316832,c351313223f86d9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:33.469 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:25:33.472 ERROR 21062 --- [ocp-schedules-17,096f1ff5af316832,c351313223f86d9e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:25:33.472 INFO 21062 --- [ocp-schedules-17,096f1ff5af316832,c351313223f86d9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:25:33.533 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:33.534 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:34.851 WARN 21062 --- [ocp-async-8,5e0e3274e04b6b88,aed7cd1efb25038a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:34.900 INFO 21062 --- [ocp-async-10,fbe4245960d26b38,dd1f80e12b932f2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:35.983 INFO 21062 --- [prometheus-minute-schedule-1,da5349453ca9c936,62af3720ef793c93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:36.534 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:36.535 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:36.852 WARN 21062 --- [ocp-async-10,b69c03f9476c85d2,a08eb7e4aeeb1a1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:36.900 INFO 21062 --- [ocp-async-4,58ad1a4073b47aa0,0a705382b0968798] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:38.504 INFO 21062 --- [ocp-schedules-9,8e6d7b4f91ffcd43,ce71201ac2014e4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:25:38.505 INFO 21062 --- [ocp-schedules-8,a397f4e223ffccb9,3d4d433879b58dc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:25:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:25:38.588175+08:00, lastProcessTime=2024-11-04T13:25:28.588461+08:00 2024-11-04 13:25:38.595 INFO 21062 --- [ocp-async-3,814dc73185412f91,0fe237fa4c36db3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:25:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:25:38.588175+08:00 2024-11-04 13:25:38.852 WARN 21062 --- [ocp-async-6,e75211a0f1a6ff1a,b169b0ce6034ced2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:38.900 INFO 21062 --- [ocp-async-2,beef5bc546f38fd7,c2edcabede0a3b5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:38.983 INFO 21062 --- [prometheus-minute-schedule-1,9a55254229b026e4,df08c8de83181d98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:39.426 INFO 21062 --- [ocp-schedules-30,c2f8e839c5d8da9e,811bf909aab32240] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:25:39.535 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:39.536 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:25:40.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:25:40.667 INFO 21062 --- [ocp-schedules-18,1cfdcf052593b703,695db8a59f42db2e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:25:40.852 WARN 21062 --- [ocp-async-2,e641c8cf32f71b0e,ba519f096a4925aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:40.900 INFO 21062 --- [ocp-async-7,78bb44987fe80af9,ec80e2c93a8c2648] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d8759ad055cb6859,eb9f73d41afb4214] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:42.536 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:42.537 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:42.852 WARN 21062 --- [ocp-async-7,6156f408f1edddd2,96fa58e4a0e51ba0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:42.900 INFO 21062 --- [ocp-async-1,106a23d3ffbed5a4,37e1473010545d5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:44.852 WARN 21062 --- [ocp-async-1,bb18ed6e1a903ee7,8aa044e27bc7a1c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:44.901 INFO 21062 --- [ocp-async-8,27f3f1283726735f,b0656932e0a0f5bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:44.983 INFO 21062 --- [prometheus-minute-schedule-1,ca7a736e58bed2c9,bcac22777bc5b88a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:45.537 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:45.538 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:46.852 WARN 21062 --- [ocp-async-8,a8a8f14d5670e54c,a14a2ef500ffb51a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:46.901 INFO 21062 --- [ocp-async-10,56a4f7554f9ed624,6860cc2069d0fcde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:47.983 INFO 21062 --- [prometheus-minute-schedule-1,f19b6eb883bac3e8,a35361ea354ca6b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:48.538 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:48.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:25:48.590120+08:00, lastProcessTime=2024-11-04T13:25:38.588175+08:00 2024-11-04 13:25:48.595 INFO 21062 --- [ocp-async-2,ec8810a1b7773538,c98a02bbcf71aa4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:25:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:25:48.590120+08:00 2024-11-04 13:25:48.853 WARN 21062 --- [ocp-async-7,f09a1c14c79c7cc3,acfe4b3bc7d2ef60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:48.901 INFO 21062 --- [ocp-async-1,84eff4f17c0bc184,7ae20595073909cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:49.429 INFO 21062 --- [ocp-schedules-21,2a53c4363b59dadf,71cd2a617aff6a33] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:25:49.990 INFO 21062 --- [ocp-schedules-18,33f5c991362ccd5c,ea9202c509b6df62] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:25:49.996 INFO 21062 --- [ocp-schedules-18,33f5c991362ccd5c,ea9202c509b6df62] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:25:49.996 INFO 21062 --- [ocp-schedules-18,33f5c991362ccd5c,ea9202c509b6df62] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:25:50.713 INFO 21062 --- [ocp-schedules-16,223170fdaba403a1,30bdca9a465cafb1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:25:50.853 WARN 21062 --- [ocp-async-1,b1a920c577c6dc6b,301c224571d3a0f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:50.901 INFO 21062 --- [ocp-async-8,00feceee09f266cc,802fab69296bc38c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ce5b49e0a2cb50be,4010edbcd6e66e07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:51.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:51.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:52.853 WARN 21062 --- [ocp-async-8,4e54eca2fa98c83f,2b795345ea2f1763] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:52.902 INFO 21062 --- [ocp-async-10,1b92342c6c3b7536,aa792111e5664cbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:53.983 INFO 21062 --- [prometheus-minute-schedule-1,8f52ba8d57827b43,a05cdd1a42e6c250] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:54.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:54.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:54.853 WARN 21062 --- [ocp-async-10,d5a2b7d2551c2fe3,18b54cb5ed552809] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:54.902 INFO 21062 --- [ocp-async-4,a6e0a89459267059,1f74ec517a2a5b07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:56.853 WARN 21062 --- [ocp-async-4,e88150bcdded4bd8,b48fe2632838c722] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:56.902 INFO 21062 --- [ocp-async-6,80c69f5e3db963c3,0051c8e7cbd9bbad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:56.983 INFO 21062 --- [prometheus-minute-schedule-1,4ad4ca7b32e51e96,d00f797e18a12ea9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:25:57.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:25:57.541 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:25:58.500 INFO 21062 --- [ocp-schedules-18,21bf70371424ac55,427fcfde3b258747] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:25:58.500 INFO 21062 --- [ocp-schedules-25,1b87af0464482827,09e22b82d0ede175] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:25:58.502 INFO 21062 --- [ocp-schedules-23,0401aa3def9c34a2,78b0dceb05d724c0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:25:58.504 INFO 21062 --- [ocp-schedules-18,21bf70371424ac55,427fcfde3b258747] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:58.504 INFO 21062 --- [ocp-schedules-25,1b87af0464482827,09e22b82d0ede175] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:58.504 INFO 21062 --- [ocp-schedules-4,53d3c9735082a5d9,1830f9ad68d3e522] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:25:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:25:58.511 INFO 21062 --- [ocp-schedules-17,6b0729484b740c00,77cfa98b39fb2e00] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:25:58.511 INFO 21062 --- [ocp-schedules-7,7cb86a9c73af8a60,f943985dd9e826ed] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:25:58.511 INFO 21062 --- [ocp-schedules-7,7cb86a9c73af8a60,f943985dd9e826ed] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:25:58.511 INFO 21062 --- [ocp-schedules-17,6b0729484b740c00,77cfa98b39fb2e00] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:25:58.511 INFO 21062 --- [ocp-schedules-11,52a34bb524045062,d54109aaa26e00ec] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:25:58.512 INFO 21062 --- [ocp-schedules-11,52a34bb524045062,d54109aaa26e00ec] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:25:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:25:58.590134+08:00, lastProcessTime=2024-11-04T13:25:48.590120+08:00 2024-11-04 13:25:58.596 INFO 21062 --- [ocp-async-4,b37f2c28baaf81d6,7f8601736c42d69d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:25:58.780 INFO 21062 --- [ocp-schedules-24,9e49a17ec8e7e397,8d1ae79251fc3504] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:25:58.784 INFO 21062 --- [ocp-schedules-24,9e49a17ec8e7e397,8d1ae79251fc3504] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:25:58.784 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:25:58.805 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:25:58.590134+08:00 2024-11-04 13:25:58.852 INFO 21062 --- [ocp-schedules-19,e282b4e9f25ea536,469f60047f2190a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:25:58.854 WARN 21062 --- [ocp-async-8,63c81a01a6237836,61da3235b6c288d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:25:58.902 INFO 21062 --- [ocp-async-5,4a54014172eb704c,bee9cdede388fda6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:25:59.432 INFO 21062 --- [ocp-schedules-11,894261d3eedc0628,f959791bf0a9b216] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:25:59.983 INFO 21062 --- [prometheus-minute-schedule-1,185956f8e4a628b2,a222b2c40f1c6674] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730697960, interval=60 2024-11-04 13:26:00.541 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:00.542 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:00.739 INFO 21062 --- [ocp-schedules-20,97c9d5798cee12d3,1e140340051ba60f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:26:00.854 WARN 21062 --- [ocp-async-5,ef5aeb215901ce20,3873eb811d32e977] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:00.902 INFO 21062 --- [ocp-async-3,1ebb865326887d0f,59b78363e3117c1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:02.172 INFO 21062 --- [ocp-schedules-10,d4b987d3d83d6e6f,a3196322f36f1db3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:26:02.173 INFO 21062 --- [ocp-schedules-10,c8088ee6d844745d,9baaae90a1a0b0de] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:26:02.177 INFO 21062 --- [ocp-schedules-10,c8088ee6d844745d,9baaae90a1a0b0de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:02.208 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:26:02.208 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:26:02.854 WARN 21062 --- [ocp-async-3,fa39c90d3079cb1c,797277de853362b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:02.902 INFO 21062 --- [ocp-async-9,f51efae82f288364,ca1e5de4bb0c763b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:02.983 INFO 21062 --- [prometheus-minute-schedule-1,141a625e12050c77,943bc1b73c57082f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:03.542 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:03.543 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:04.854 WARN 21062 --- [ocp-async-9,7817e622bdf4f1ea,050e8492b19b0007] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:04.903 INFO 21062 --- [ocp-async-2,8d35bfecde132d67,ba502ac7b70d52cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d671893440af6b0f,8fac0045e671de6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:06.543 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:06.544 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:06.854 WARN 21062 --- [ocp-async-10,2d3f8c3db9a52039,4672d4b2855874e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:06.903 INFO 21062 --- [ocp-async-7,828a2bcda6ca6da6,ee4154ce3bd3a561] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:08.505 INFO 21062 --- [ocp-schedules-25,1b87af0464482827,09e22b82d0ede175] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:26:08.505 INFO 21062 --- [ocp-schedules-18,21bf70371424ac55,427fcfde3b258747] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:26:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:26:08.590134+08:00, lastProcessTime=2024-11-04T13:25:58.590134+08:00 2024-11-04 13:26:08.596 INFO 21062 --- [ocp-async-10,b1ce5b0e094d6541,391cbc555d43a7d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:26:08.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:26:08.590134+08:00 2024-11-04 13:26:08.784 INFO 21062 --- [ocp-schedules-24,9e49a17ec8e7e397,8d1ae79251fc3504] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:26:08.854 WARN 21062 --- [ocp-async-3,34cc81a4858d9297,e364ab9e2ccc6bb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:08.903 INFO 21062 --- [ocp-async-6,8902056a968c3231,4b3ab12c4e8bd905] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:08.983 INFO 21062 --- [prometheus-minute-schedule-1,210f0e4e382dcaf9,bfdbf9867328f87d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:09.435 INFO 21062 --- [ocp-schedules-26,cea60b4fac9f5554,74285048cf6fd364] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:26:09.544 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:09.545 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:09.998 INFO 21062 --- [ocp-schedules-3,a302303d460bebd1,40f7eb438e167f51] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:26:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:26:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:26:10.004 INFO 21062 --- [ocp-schedules-3,a302303d460bebd1,40f7eb438e167f51] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:26:10.004 INFO 21062 --- [ocp-schedules-3,a302303d460bebd1,40f7eb438e167f51] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:26:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:26:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:26:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:26:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:26:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:26:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:26:10.777 INFO 21062 --- [ocp-schedules-26,89ca678d317fd3ca,cd8c468b37f34f09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:26:10.855 WARN 21062 --- [ocp-async-9,0350fb4e786a7707,95c7583030c38c45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:10.903 INFO 21062 --- [ocp-async-2,50455ebbd471bcc9,3e7093694c4bafa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=2, affectRows=2 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=2, validCount=2 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=15, affectRows=15 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=15, validCount=15 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=15, affectRows=15 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=22, affectRows=22 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=7, affectRows=7 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=7, validCount=7 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=22, affectRows=22 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=22, validCount=22 2024-11-04 13:26:11.243 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=15, validCount=15 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=22, validCount=22 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=43, affectRows=43 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=46, affectRows=46 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=34, affectRows=34 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=51, affectRows=51 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=43, validCount=43 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=34, validCount=34 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=50, validCount=50 2024-11-04 13:26:11.244 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=44, validCount=44 2024-11-04 13:26:11.245 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=37, affectRows=37 2024-11-04 13:26:11.245 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=37, validCount=37 2024-11-04 13:26:11.245 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697900, nodeCount=28, affectRows=28 2024-11-04 13:26:11.245 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697900, nodeCount=28, validCount=28 2024-11-04 13:26:11.983 INFO 21062 --- [prometheus-minute-schedule-1,bfbd86d537df5edf,22dc7fd292085793] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:12.545 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:12.546 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:12.855 WARN 21062 --- [ocp-async-10,d1ea93677d7edcfe,b41761340eaf1691] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:12.903 INFO 21062 --- [ocp-async-7,0cc80311504f692d,7d40ed26891320b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:13.985 INFO 21062 --- [prometheus-cache-monitor1,86580c2e1a80f2a0,f1b40695a5dc36cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:26:13.985 INFO 21062 --- [prometheus-cache-monitor1,86580c2e1a80f2a0,f1b40695a5dc36cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:26:14.855 WARN 21062 --- [ocp-async-1,2a2e56bcc75e1636,08d44b81f8a73c3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:14.903 INFO 21062 --- [ocp-async-4,0df0132314fc0b09,39a5595f89b1464d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:14.983 INFO 21062 --- [prometheus-minute-schedule-1,e0c83cfe4ad2f158,590fb8558f4bdf7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:15.546 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:15.547 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:16.855 WARN 21062 --- [ocp-async-6,a8aa4ad4a4ee883d,d853ca39d047ba64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:16.904 INFO 21062 --- [ocp-async-8,c8af44f8e7f5e526,a6b5307aa57078bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4698293e6663003d,ad1fa8267bcdd740] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:18.547 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:18.548 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:26:18.589295+08:00, lastProcessTime=2024-11-04T13:26:08.590134+08:00 2024-11-04 13:26:18.596 INFO 21062 --- [ocp-async-6,7bcf0257f1719039,2aa2ff5746a345b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:26:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:26:18.589295+08:00 2024-11-04 13:26:18.855 WARN 21062 --- [ocp-async-10,5ad1c3c8adc048bc,6936ac9e15311a8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:18.904 INFO 21062 --- [ocp-async-7,375de81d11f4c561,beecfda56b7f15d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:19.438 INFO 21062 --- [ocp-schedules-20,a6d0f02dd7efdd9b,919c9396235bbcd9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:26:20.802 INFO 21062 --- [ocp-schedules-18,34819910b182bcc4,f6b8523f44991c07] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:26:20.855 WARN 21062 --- [ocp-async-1,68c379979c3098a7,79e63e6b656e1267] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:20.904 INFO 21062 --- [ocp-async-4,6af720f4b15b5a01,978a1c38a687fbf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9ecb856bc9df1ac2,f9d6710e401d7329] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:21.548 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:21.549 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:22.856 WARN 21062 --- [ocp-async-6,0741160b3469abb9,0949bb375c314f26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:22.904 INFO 21062 --- [ocp-async-8,ce163c716f5afc88,204bbf51df062156] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a5af465dd5364fa6,1bf951b1655389fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:24.549 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:24.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:24.856 WARN 21062 --- [ocp-async-2,1ec127136063efa5,56be957fa6e76408] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:24.905 INFO 21062 --- [ocp-async-5,0af93542aebb10b1,6cb8ccc60ee07202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:26.856 WARN 21062 --- [ocp-async-7,365b8ca7b2ce9c41,a5d90ceed9fa6d36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:26.905 INFO 21062 --- [ocp-async-3,2a5ad79df2ca99e3,ada694aaf7a18cdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:26.983 INFO 21062 --- [prometheus-minute-schedule-1,8b9494dc447096db,5c1bdeed3f3530b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ce0799feb6de4c63,dc38c12dad0ef1f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:26:27.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:27.551 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:28.498 INFO 21062 --- [ocp-schedules-28,2563620d7f12233c,19131c94d95eba03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:26:28.499 INFO 21062 --- [ocp-schedules-30,65f3b0097081ab68,f0b895736b2f52bb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:26:28.500 INFO 21062 --- [ocp-schedules-21,5f488a89674bf293,392b112b8955aebe] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:26:28.500 INFO 21062 --- [ocp-schedules-3,e487db1b18c635e5,6a0e7896b4549ef3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:26:28.502 INFO 21062 --- [ocp-schedules-28,2563620d7f12233c,19131c94d95eba03] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:26:28.502 INFO 21062 --- [ocp-schedules-15,54b78c4bd2d79a1e,47a9efe5f4f8653f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:26:28.503 INFO 21062 --- [ocp-schedules-21,5f488a89674bf293,392b112b8955aebe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:28.503 INFO 21062 --- [ocp-schedules-3,e487db1b18c635e5,6a0e7896b4549ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:26:28.504 INFO 21062 --- [ocp-schedules-25,a18604c5c2ab66a5,4d6a016be503ce18] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:26:28.505 INFO 21062 --- [ocp-schedules-30,65f3b0097081ab68,f0b895736b2f52bb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:26:28.510 INFO 21062 --- [ocp-schedules-20,7bfb4e4af7c4b09c,770c4de313213333] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:26:28.510 INFO 21062 --- [ocp-schedules-20,7bfb4e4af7c4b09c,770c4de313213333] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:26:28.510 INFO 21062 --- [ocp-schedules-9,af50cc26a35934da,2dbaa9b6388ace0e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:26:28.510 INFO 21062 --- [ocp-schedules-9,af50cc26a35934da,2dbaa9b6388ace0e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:26:28.511 INFO 21062 --- [ocp-schedules-29,59223b9153974ea4,55373ec3ce38d918] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:26:28.511 INFO 21062 --- [ocp-schedules-29,59223b9153974ea4,55373ec3ce38d918] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:26:28.565 INFO 21062 --- [ocp-schedules-23,9034c701853a6afa,3e7f3a7a6842fc75] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:26:28.565 INFO 21062 --- [ocp-schedules-23,9034c701853a6afa,3e7f3a7a6842fc75] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:26:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:26:28.589949+08:00, lastProcessTime=2024-11-04T13:26:18.589295+08:00 2024-11-04 13:26:28.596 INFO 21062 --- [ocp-async-3,b5596a6516b6a0d3,a399b45043f88208] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:26:28.631 INFO 21062 --- [ocp-schedules-6,79ca670f288d0a8c,a70214516ab8535d] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:26:28.632 INFO 21062 --- [ocp-schedules-6,79ca670f288d0a8c,a70214516ab8535d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:26:28.690 INFO 21062 --- [ocp-schedules-26,8d1445400e8cb0ac,4d7269e3149af8ab] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:26:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:26:28.589949+08:00 2024-11-04 13:26:28.840 INFO 21062 --- [ocp-schedules-14,ef8521343bb7c22b,bb77e41f827a6388] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:26:28.840 INFO 21062 --- [ocp-schedules-14,ef8521343bb7c22b,bb77e41f827a6388] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:26:28.855 INFO 21062 --- [ocp-schedules-2,b390d5a24ac7ea20,5f0cfc99f3bb8329] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:26:28.856 WARN 21062 --- [ocp-async-9,252fbfb067d0e6b1,2dd3e886ec92a8f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:28.882 INFO 21062 --- [ocp-schedules-20,a8ba8cea3239ee3e,394690089cdfe10f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:26:28.887 INFO 21062 --- [ocp-schedules-20,a8ba8cea3239ee3e,394690089cdfe10f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:26:28.887 INFO 21062 --- [ocp-schedules-20,a8ba8cea3239ee3e,394690089cdfe10f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:26:28.905 INFO 21062 --- [ocp-async-10,b91e24c53e85bfdc,c61e8389b1529f96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:29.053 INFO 21062 --- [ocp-schedules-13,1533ed1efe35d4cf,4ef0d73dc635cd39] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:26:29.053 INFO 21062 --- [ocp-schedules-13,1533ed1efe35d4cf,4ef0d73dc635cd39] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:26:29.053 INFO 21062 --- [ocp-schedules-13,1533ed1efe35d4cf,4ef0d73dc635cd39] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:26:29.053 INFO 21062 --- [ocp-schedules-13,1533ed1efe35d4cf,4ef0d73dc635cd39] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:26:29.054 INFO 21062 --- [ocp-schedules-13,1533ed1efe35d4cf,4ef0d73dc635cd39] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:26:29.054 INFO 21062 --- [ocp-schedules-13,1533ed1efe35d4cf,4ef0d73dc635cd39] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:26:29.441 INFO 21062 --- [ocp-schedules-22,068124efc62189cc,b772a82d10a9a221] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:26:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7dfac6470af4b5af,66f7f87f8deab408] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:30.004 INFO 21062 --- [ocp-schedules-14,79ceb52e6551a7c2,32cc3eb177f0cee5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:26:30.004 INFO 21062 --- [ocp-schedules-14,79ceb52e6551a7c2,32cc3eb177f0cee5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:26:30.007 INFO 21062 --- [ocp-schedules-30,f8d11cb76a3ee967,76da2289b299f504] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:26:30.012 INFO 21062 --- [ocp-schedules-30,f8d11cb76a3ee967,76da2289b299f504] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:26:30.012 INFO 21062 --- [ocp-schedules-30,f8d11cb76a3ee967,76da2289b299f504] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:26:30.031 INFO 21062 --- [ocp-schedules-14,79ceb52e6551a7c2,32cc3eb177f0cee5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:26:30.031 INFO 21062 --- [ocp-schedules-14,79ceb52e6551a7c2,32cc3eb177f0cee5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:26:30.032 INFO 21062 --- [ocp-schedules-25,441bfd6ca21371af,2622089ed4898ba6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:26:30.155 INFO 21062 --- [ocp-schedules-9,00a31b06022a020b,e18f1a8701dd7451] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:26:30.155 INFO 21062 --- [ocp-schedules-9,00a31b06022a020b,e18f1a8701dd7451] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:26:30.155 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:26:30.155 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:26:30.158 INFO 21062 --- [ocp-schedules-20,171ee6ebcb19f875,db2ae0dbbf64beee] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:26:30.158+08:00, clusters=[1] 2024-11-04 13:26:30.159 INFO 21062 --- [ocp-schedules-28,951a8f6f7a12545d,a34ef69f087b50f1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:26:30.160 INFO 21062 --- [ocp-schedules-20,171ee6ebcb19f875,db2ae0dbbf64beee] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:26:30.163 INFO 21062 --- [ocp-schedules-28,951a8f6f7a12545d,a34ef69f087b50f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:30.165 INFO 21062 --- [ocp-schedules-9,00a31b06022a020b,e18f1a8701dd7451] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:26:30.165 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:26:30.165 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:26:30.167 WARN 21062 --- [ocp-schedules-9,00a31b06022a020b,e18f1a8701dd7451] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:26:30.167 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:26:30.167 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:26:30.167 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:26:30.167 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:26:30.167 INFO 21062 --- [ocp-schedules-9,00a31b06022a020b,e18f1a8701dd7451] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:26:30.168 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:26:30.169 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:26:30.169 WARN 21062 --- [ocp-schedules-9,00a31b06022a020b,e18f1a8701dd7451] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:26:30.178 ERROR 21062 --- [ocp-schedules-9,00a31b06022a020b,e18f1a8701dd7451] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:26:30.220 ERROR 21062 --- [ocp-schedules-28,951a8f6f7a12545d,a34ef69f087b50f1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:26:30.220 INFO 21062 --- [ocp-schedules-28,951a8f6f7a12545d,a34ef69f087b50f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:26:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:26:30.286 INFO 21062 --- [ocp-schedules-17,e1109c6ad6adc595,d0762757ca0500a9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:26:30.286+08:00, clusters=[1] 2024-11-04 13:26:30.290 INFO 21062 --- [ocp-schedules-17,e1109c6ad6adc595,d0762757ca0500a9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:26:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:26:30.551 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:30.552 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:30.841 INFO 21062 --- [ocp-schedules-18,ff3920e621425335,e1f437daa67a79b9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:26:30.856 WARN 21062 --- [ocp-async-10,b4c172eb86da1b08,6a57195060893909] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:30.905 INFO 21062 --- [ocp-async-1,8a42ae7020a4a7bf,a5ba93d1265473a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:32.208 INFO 21062 --- [ocp-schedules-10,c8088ee6d844745d,9baaae90a1a0b0de] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:26:32.208 INFO 21062 --- [ocp-schedules-10,8a4e35158bc39938,42cac5b6b1ae8f20] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:26:32.212 INFO 21062 --- [ocp-schedules-10,8a4e35158bc39938,42cac5b6b1ae8f20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:32.241 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:26:32.241 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:26:32.856 WARN 21062 --- [ocp-async-1,717e72e0b1f6ed74,a1a9a28e51c9182b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:32.905 INFO 21062 --- [ocp-async-6,9558cde376c9fd7f,f032a41e22d095f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:32.983 INFO 21062 --- [prometheus-minute-schedule-1,76ffc79b13023694,02dac31e1a8907e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:33.472 INFO 21062 --- [ocp-schedules-17,964ad544f3b8a738,0cec067066752f43] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:26:33.476 INFO 21062 --- [ocp-schedules-17,964ad544f3b8a738,0cec067066752f43] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:33.552 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:33.553 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:33.563 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:26:33.565 ERROR 21062 --- [ocp-schedules-17,964ad544f3b8a738,0cec067066752f43] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:26:33.565 INFO 21062 --- [ocp-schedules-17,964ad544f3b8a738,0cec067066752f43] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:26:34.857 WARN 21062 --- [ocp-async-6,264b34f2c5bfe482,a9aee20a04f0b134] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:34.905 INFO 21062 --- [ocp-async-2,77393c6be36c46e2,1af9c5960745d7d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e235795ea04fd6c6,9f32dcd7fca35d55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:36.553 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:36.554 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:36.857 WARN 21062 --- [ocp-async-2,64edef20482ea0e7,3b180cf9a14aca47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:36.906 INFO 21062 --- [ocp-async-5,be7489319662a9d2,b4484fb668d01ed9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:38.504 INFO 21062 --- [ocp-schedules-3,e487db1b18c635e5,6a0e7896b4549ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:26:38.504 INFO 21062 --- [ocp-schedules-21,5f488a89674bf293,392b112b8955aebe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:26:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:26:38.590348+08:00, lastProcessTime=2024-11-04T13:26:28.589949+08:00 2024-11-04 13:26:38.596 INFO 21062 --- [ocp-async-7,7a182c756078aff9,3083cd6175456eb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:26:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:26:38.590348+08:00 2024-11-04 13:26:38.857 WARN 21062 --- [ocp-async-3,e706853a075ac709,a9a3c5ebe2af0b71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:38.906 INFO 21062 --- [ocp-async-8,1b5a3e2a5a863980,b2e598fb59de269f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5a5d662f8d6b76a2,ec14003cc5366b25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:39.444 INFO 21062 --- [ocp-schedules-1,02e4549e9f31b096,4c87d40023351db4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:26:39.554 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:39.555 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:40.002 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:26:40.002 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:26:40.857 WARN 21062 --- [ocp-async-8,73faa4ae7879d684,915df8a65fe518d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:40.870 INFO 21062 --- [ocp-schedules-29,4c65f485a509ec28,31b72fe98aa10996] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:26:40.906 INFO 21062 --- [ocp-async-10,b0a56f7f476cee07,18392a5b8e9444c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:26:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:26:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 13:26:41.983 INFO 21062 --- [prometheus-minute-schedule-1,8982253885dd0bb8,0929ca7cdabe7ba9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:42.555 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:42.556 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:42.858 WARN 21062 --- [ocp-async-10,f4bfda3ed75ef4a7,fa2e7f4b47e738e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:42.906 INFO 21062 --- [ocp-async-1,42ff5212d2e33d5d,dfdd764df79c89bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:44.858 WARN 21062 --- [ocp-async-1,cfb37e591383a75f,9522b03da1bd42ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:44.906 INFO 21062 --- [ocp-async-6,4419883fa512c5cd,e3f522b26d4007e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:44.983 INFO 21062 --- [prometheus-minute-schedule-1,4d7a49eaeb3dee40,5d495fbcac34bf68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:45.556 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:45.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:46.858 WARN 21062 --- [ocp-async-6,f1c394c812cfb1cf,b807eb54b371a5f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:46.907 INFO 21062 --- [ocp-async-2,bbfd562a4d3391cf,8aa487fe82be057d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:47.983 INFO 21062 --- [prometheus-minute-schedule-1,f9c87af0b8d53f76,0085b13bcb25b761] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:48.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:48.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:26:48.591393+08:00, lastProcessTime=2024-11-04T13:26:38.590348+08:00 2024-11-04 13:26:48.597 INFO 21062 --- [ocp-async-8,2b2b6f9a3344f9cc,704813044700b459] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:26:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:26:48.591393+08:00 2024-11-04 13:26:48.858 WARN 21062 --- [ocp-async-10,11fb90cf4bf05aa3,2c597761ff740d90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:48.907 INFO 21062 --- [ocp-async-1,62ea38b0ecabb082,ac3c4971bb2e34b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:49.446 INFO 21062 --- [ocp-schedules-17,0c34b9e7ba9fe5df,063c22a73da61246] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:26:50.015 INFO 21062 --- [ocp-schedules-22,8943869b3aa56e0f,f764d30dc1a03169] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:26:50.023 INFO 21062 --- [ocp-schedules-22,8943869b3aa56e0f,f764d30dc1a03169] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:26:50.023 INFO 21062 --- [ocp-schedules-22,8943869b3aa56e0f,f764d30dc1a03169] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:26:50.858 WARN 21062 --- [ocp-async-1,0ceb6ecb6d536ab9,1d7e20549a31e716] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:50.907 INFO 21062 --- [ocp-async-6,0806ab5b8785c3cb,f0df5670d7c1941c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:50.915 INFO 21062 --- [ocp-schedules-5,47aae77ce174b2df,ad0744490f8cef4b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:26:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b6fad7c59a19849f,3c0844adecda85b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:51.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:51.560 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:52.858 WARN 21062 --- [ocp-async-6,61fc256a2e36b4ac,ce03c0e9d712e490] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:52.907 INFO 21062 --- [ocp-async-2,d2e14850d2e11265,d04dc35788a1c71c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:53.983 INFO 21062 --- [prometheus-minute-schedule-1,0b4be8e3da0e9cd9,c61536ae006f46c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:54.560 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:54.562 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:54.859 WARN 21062 --- [ocp-async-2,88da331888098f99,4674fa2ead7ec3a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:54.907 INFO 21062 --- [ocp-async-5,a7203d1c13993e00,5557b8f00f801f18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:56.859 WARN 21062 --- [ocp-async-5,c6eef170adee29e6,d25b339bdc738fa7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:56.907 INFO 21062 --- [ocp-async-4,b5fc92b52ef01246,35aad8764f0bd75d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:56.983 INFO 21062 --- [prometheus-minute-schedule-1,6be9c4a79ba74d42,41d59b9384f45c4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:26:57.562 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:26:57.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:26:58.500 INFO 21062 --- [ocp-schedules-21,6f51a23d7b0304d3,a002a601dae401e0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:26:58.500 INFO 21062 --- [ocp-schedules-5,faab7c8d3ff4bbda,d50dc390c62399bf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:26:58.503 INFO 21062 --- [ocp-schedules-23,c61a8a267647d6ab,3d401989e83747aa] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:26:58.504 INFO 21062 --- [ocp-schedules-21,6f51a23d7b0304d3,a002a601dae401e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:58.504 INFO 21062 --- [ocp-schedules-5,faab7c8d3ff4bbda,d50dc390c62399bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:26:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:26:58.505 INFO 21062 --- [ocp-schedules-29,e5b86d72fdb1515d,04efc9ff9f7b48ce] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:26:58.513 INFO 21062 --- [ocp-schedules-2,538a08465fe862fe,3e741a9d539195d3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:26:58.513 INFO 21062 --- [ocp-schedules-24,3426242de09e7c78,c143b9082a6e9c8c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:26:58.513 INFO 21062 --- [ocp-schedules-2,538a08465fe862fe,3e741a9d539195d3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:26:58.513 INFO 21062 --- [ocp-schedules-24,3426242de09e7c78,c143b9082a6e9c8c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:26:58.514 INFO 21062 --- [ocp-schedules-3,9d35df86ce52c5c0,63051488ee92e859] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:26:58.514 INFO 21062 --- [ocp-schedules-3,9d35df86ce52c5c0,63051488ee92e859] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:26:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:26:58.589733+08:00, lastProcessTime=2024-11-04T13:26:48.591393+08:00 2024-11-04 13:26:58.597 INFO 21062 --- [ocp-async-5,c505932db6220280,4c3e8ff751372319] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:26:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:26:58.589733+08:00 2024-11-04 13:26:58.857 INFO 21062 --- [ocp-schedules-16,1faba51bf39ffb54,aa463871b7411689] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:26:58.859 WARN 21062 --- [ocp-async-6,e3bb812f510e2a9a,1186ae65c01c356f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:26:58.908 INFO 21062 --- [ocp-async-9,7ce254fabed73890,2cf6d380512d320e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:26:59.449 INFO 21062 --- [ocp-schedules-7,1df6002025b5c1d8,d6acf3da913e7da7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:26:59.983 INFO 21062 --- [prometheus-minute-schedule-1,49d3451e8d41ced6,e1cb8619328458e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698020, interval=60 2024-11-04 13:27:00.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:00.564 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:00.859 WARN 21062 --- [ocp-async-9,56c3934007311222,735f4a4040b89118] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:00.908 INFO 21062 --- [ocp-async-7,b3b4fa68b92cd9c0,7038bc3b390ad9cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:00.942 INFO 21062 --- [ocp-schedules-17,52e79bd3b2b90cfc,60d268084c2657f7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:27:02.242 INFO 21062 --- [ocp-schedules-10,8a4e35158bc39938,42cac5b6b1ae8f20] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:27:02.242 INFO 21062 --- [ocp-schedules-10,6594ac359d308d3f,1b54b0995361f47f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:27:02.245 INFO 21062 --- [ocp-schedules-10,6594ac359d308d3f,1b54b0995361f47f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:02.273 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:27:02.273 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:27:02.859 WARN 21062 --- [ocp-async-7,262bc98f4c754af6,de6caa5fd7b41f3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:02.908 INFO 21062 --- [ocp-async-3,cd533631ba324473,2fe9749e4566209a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c8795001f7a60fbb,ffa44ed869ba1678] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:03.564 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:03.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:04.859 WARN 21062 --- [ocp-async-3,e9d5464e98c8569c,8a3d099eeeed358a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:04.908 INFO 21062 --- [ocp-async-8,85f542abadfcddbb,4baf6053a9f2d3bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:04.921 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:27:04.921 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:27:04.928 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:27:04.928 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:27:04.928 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:27:04.929 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:27:04.934 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:27:04.934 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:27:04.934 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:27:04.934 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:27:04.970 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:27:04.978 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:27:04.978 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:27:04.978 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:27:04.978 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:27:04.978 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:27:04.987 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:27:05.543 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:27:05.983 INFO 21062 --- [prometheus-minute-schedule-1,96a3fdb806135b19,52e5f2d168cf5852] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:06.096 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.162001072S 2024-11-04 13:27:06.097 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:27:06.190 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:27:06.190 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:27:06.218 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:27:06.285 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:27:06.285 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:27:06.285 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:27:06.285 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:27:06.285 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:27:06.285 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:27:06.297 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:27:06.475 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] o.s.boot.SpringApplication : Started application in 1.547 seconds (JVM running for 3440.155) 2024-11-04 13:27:06.476 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.547617361S 2024-11-04 13:27:06.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:06.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:06.860 WARN 21062 --- [ocp-async-8,641f31364379af8a,0aef3aed5834857f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:06.908 INFO 21062 --- [ocp-async-10,64691eb2dfe0741d,b7529392d0c97c78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:06.944 INFO 21062 --- [ocp-schedules-30,1b4063b34345eb83,726cb4903b3adc9b] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:27:08.505 INFO 21062 --- [ocp-schedules-5,faab7c8d3ff4bbda,d50dc390c62399bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:27:08.506 INFO 21062 --- [ocp-schedules-21,6f51a23d7b0304d3,a002a601dae401e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:27:08.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:27:08.593298+08:00, lastProcessTime=2024-11-04T13:26:58.589733+08:00 2024-11-04 13:27:08.597 INFO 21062 --- [ocp-async-8,ef3b0751e6b74edd,d8c8ee60bb41f82c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:27:08.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:27:08.593298+08:00 2024-11-04 13:27:08.785 INFO 21062 --- [ocp-schedules-16,277be6b74e7e65c8,2826ed1a434e64ed] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:27:08.788 INFO 21062 --- [ocp-schedules-16,277be6b74e7e65c8,2826ed1a434e64ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:08.789 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:27:08.860 WARN 21062 --- [ocp-async-7,7d1132e1a8682cb8,d3fbc5b6d9e24e2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:08.909 INFO 21062 --- [ocp-async-4,5bf7d578a1fc6f56,bb9114640aed3ec2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:08.983 INFO 21062 --- [prometheus-minute-schedule-1,77ae73821718f567,13f750b6edbb2ba5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:09.452 INFO 21062 --- [ocp-schedules-27,97d78e4fa2a802b7,977dffb9764c3aea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:27:09.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:09.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:10.002 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:27:10.002 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:27:10.025 INFO 21062 --- [ocp-schedules-26,714e104e60384aa9,38dffca24170bfa6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:27:10.032 INFO 21062 --- [ocp-schedules-26,714e104e60384aa9,38dffca24170bfa6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:27:10.032 INFO 21062 --- [ocp-schedules-26,714e104e60384aa9,38dffca24170bfa6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:27:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:27:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:27:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:27:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:27:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:27:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:27:10.860 WARN 21062 --- [ocp-async-3,121d0e6302efc8db,d64478c98b6a0f80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:10.909 INFO 21062 --- [ocp-async-2,93e19b15e0f9b4c8,2eedaa1362912b86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:10.981 INFO 21062 --- [ocp-schedules-6,365ba37b5307b0ce,47df72b32e106f25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:27:11.246 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=3, affectRows=3 2024-11-04 13:27:11.246 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=11, affectRows=11 2024-11-04 13:27:11.246 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=12, affectRows=12 2024-11-04 13:27:11.246 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=3, validCount=3 2024-11-04 13:27:11.246 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=10, validCount=10 2024-11-04 13:27:11.246 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=12, validCount=12 2024-11-04 13:27:11.247 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=21, affectRows=21 2024-11-04 13:27:11.247 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=21, validCount=21 2024-11-04 13:27:11.247 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=39, affectRows=39 2024-11-04 13:27:11.248 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=39, validCount=39 2024-11-04 13:27:11.248 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=80, affectRows=80 2024-11-04 13:27:11.248 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=62, affectRows=62 2024-11-04 13:27:11.248 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=79, validCount=79 2024-11-04 13:27:11.248 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=62, validCount=62 2024-11-04 13:27:11.249 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730697960, nodeCount=94, affectRows=94 2024-11-04 13:27:11.249 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730697960, nodeCount=93, validCount=93 2024-11-04 13:27:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a5d53784c26189c5,344f81b311bc2cb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:12.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:12.569 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:12.860 WARN 21062 --- [ocp-async-8,079d7ee7429b6522,93f081cdd090fd2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:12.909 INFO 21062 --- [ocp-async-5,2668f1c259e06403,806eaa7335660881] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:13.984 INFO 21062 --- [prometheus-cache-monitor1,eae718b83251865a,f3ee25e6e68f9acd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:27:13.985 INFO 21062 --- [prometheus-cache-monitor1,eae718b83251865a,f3ee25e6e68f9acd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:27:14.536 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:23:14.512, end=2024-11-04T13:25:14.533} 2024-11-04 13:27:14.544 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:27:14.544 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:27:14.860 WARN 21062 --- [ocp-async-10,7cdfffec690cc27a,96b2610d045e54c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:14.909 INFO 21062 --- [ocp-async-1,9e500209c1ec2eed,fc8d3910062d11eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:14.983 INFO 21062 --- [prometheus-minute-schedule-1,dc0ffbc8649d1f9d,2467373f2ef2533c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:15.569 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:15.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:16.860 WARN 21062 --- [ocp-async-4,d04ba7e14d268849,423cb61ce9645775] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:16.909 INFO 21062 --- [ocp-async-6,38010095651a0adf,8c9738efb55e464f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:17.983 INFO 21062 --- [prometheus-minute-schedule-1,a8748761aea53cb9,07ea202e8e443c59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:18.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:18.571 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:27:18.589502+08:00, lastProcessTime=2024-11-04T13:27:08.593298+08:00 2024-11-04 13:27:18.597 INFO 21062 --- [ocp-async-4,110f5be40299f80a,dcdd6beb7063c8bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:27:18.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:27:18.589502+08:00 2024-11-04 13:27:18.789 INFO 21062 --- [ocp-schedules-16,277be6b74e7e65c8,2826ed1a434e64ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:27:18.861 WARN 21062 --- [ocp-async-8,6baabdfa17929bcf,061d91c16e0633c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:18.909 INFO 21062 --- [ocp-async-5,64aa8b607773473b,05110e02f94d2b1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:19.456 INFO 21062 --- [ocp-schedules-26,7326ebb6024b6b53,707938bee88e37ce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:27:20.861 WARN 21062 --- [ocp-async-10,7071ef69a3488ad0,9c733b8649714f26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:20.910 INFO 21062 --- [ocp-async-1,042673ff5a6e054b,1c30bb2862926fe5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:20.983 INFO 21062 --- [prometheus-minute-schedule-1,7f3d489133b7a5be,95e44f6a9b733bab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:21.005 INFO 21062 --- [ocp-schedules-9,9a05d6830949551e,bd65aa1f7b8c82dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:27:21.571 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:21.572 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:22.861 WARN 21062 --- [ocp-async-4,dbec527c66e6b30c,d2077f00deb3c21d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:22.910 INFO 21062 --- [ocp-async-6,1d25730619b32e93,810d5ccafdfc237c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:23.983 INFO 21062 --- [prometheus-minute-schedule-1,7fd69b4272d1af39,d907dcc7c48e23f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:24.572 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:24.573 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:24.861 WARN 21062 --- [ocp-async-2,c28dd986355b2903,d2a99627a21cf558] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:24.910 INFO 21062 --- [ocp-async-9,9d205e276bf25d66,ddaba7daaa228675] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:26.861 WARN 21062 --- [ocp-async-9,04f9914ce0078072,e7eb8f6927ab4f8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:26.910 INFO 21062 --- [ocp-async-7,1f7c99f728fd8277,7cb97a83f6e0ddef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9d3bcf6a4660a959,abe367361be92541] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:26.984 INFO 21062 --- [prometheus-hour-schedule-1,26a5a68a78247097,9ffabae264f46bf9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:27:27.573 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:27.574 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:28.498 INFO 21062 --- [ocp-schedules-2,34beed562ed0eecb,5d09aee2fef47928] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:27:28.499 INFO 21062 --- [ocp-schedules-20,ab9f72064023cbe3,0f8fb5ed95c84a29] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:27:28.499 INFO 21062 --- [ocp-schedules-26,e45d0ccfb8902a67,707c991b6dc08ef3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:27:28.500 INFO 21062 --- [ocp-schedules-17,7c03d771dea2a1de,3db67be22ca04e83] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:27:28.502 INFO 21062 --- [ocp-schedules-2,34beed562ed0eecb,5d09aee2fef47928] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:27:28.502 INFO 21062 --- [ocp-schedules-8,4e37b847365d8b7a,ad85020c3dd2ce19] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:27:28.503 INFO 21062 --- [ocp-schedules-26,e45d0ccfb8902a67,707c991b6dc08ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:28.504 INFO 21062 --- [ocp-schedules-17,7c03d771dea2a1de,3db67be22ca04e83] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:27:28.505 INFO 21062 --- [ocp-schedules-15,fef91995628af0af,9db8265348d404ed] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:27:28.505 INFO 21062 --- [ocp-schedules-29,34d06f5a5cbdeb66,952acd4d84cdfea9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:27:28.505 INFO 21062 --- [ocp-schedules-20,ab9f72064023cbe3,0f8fb5ed95c84a29] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:27:28.508 INFO 21062 --- [ocp-schedules-29,34d06f5a5cbdeb66,952acd4d84cdfea9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:28.510 INFO 21062 --- [ocp-schedules-11,242490bab66ee267,53e3b578452af653] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:27:28.510 INFO 21062 --- [ocp-schedules-3,f1f213322dc9d01f,84bdfe2dbf096247] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:27:28.510 INFO 21062 --- [ocp-schedules-3,f1f213322dc9d01f,84bdfe2dbf096247] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:27:28.510 INFO 21062 --- [ocp-schedules-11,242490bab66ee267,53e3b578452af653] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:27:28.511 INFO 21062 --- [ocp-schedules-5,f3c7da08c015145a,cffbaf2b139ab82b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:27:28.511 INFO 21062 --- [ocp-schedules-5,f3c7da08c015145a,cffbaf2b139ab82b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:27:28.531 INFO 21062 --- [ocp-schedules-23,54230c5a36ee4f76,fedbc8d72005801c] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:27:28.531 INFO 21062 --- [ocp-schedules-23,54230c5a36ee4f76,fedbc8d72005801c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:27:28.575 INFO 21062 --- [pool-12-thread-15,57f249d70b5822aa,73ba6341ea52955d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:27:28.575 INFO 21062 --- [pool-12-thread-14,57f249d70b5822aa,d6087a63818493e3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:27:28.575 INFO 21062 --- [pool-12-thread-16,57f249d70b5822aa,21d1c6753adbde32] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:27:28.576 INFO 21062 --- [pool-12-thread-13,57f249d70b5822aa,dda4b5a2d932deda] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:27:28.580 INFO 21062 --- [ForkJoinPool-11-worker-0,57f249d70b5822aa,90fed54a4cb1616f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:27:28.580 INFO 21062 --- [ForkJoinPool-11-worker-0,57f249d70b5822aa,90fed54a4cb1616f] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:27:28.580 INFO 21062 --- [ForkJoinPool-11-worker-0,57f249d70b5822aa,90fed54a4cb1616f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:27:28.580 INFO 21062 --- [pool-12-thread-16,57f249d70b5822aa,21d1c6753adbde32] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:27:28.581 INFO 21062 --- [pool-12-thread-13,57f249d70b5822aa,dda4b5a2d932deda] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:27:28.581 INFO 21062 --- [pool-12-thread-13,57f249d70b5822aa,dda4b5a2d932deda] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:27:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:27:28.588024+08:00, lastProcessTime=2024-11-04T13:27:18.589502+08:00 2024-11-04 13:27:28.596 INFO 21062 --- [ocp-schedules-7,03e12631962cd0be,5c2794aa7ad6b285] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:27:28.597 INFO 21062 --- [ocp-schedules-7,03e12631962cd0be,5c2794aa7ad6b285] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:27:28.597 INFO 21062 --- [ocp-async-7,16aa5ccd90a8c30c,b31fecae8eae4009] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:27:28.600 INFO 21062 --- [pool-12-thread-15,57f249d70b5822aa,73ba6341ea52955d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:27:28.603 INFO 21062 --- [pool-12-thread-15,57f249d70b5822aa,73ba6341ea52955d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:27:28.605 INFO 21062 --- [ForkJoinPool-11-worker-0,57f249d70b5822aa,db5bc57eba265a99] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:27:28.605 INFO 21062 --- [pool-12-thread-14,57f249d70b5822aa,d6087a63818493e3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:27:28.606 INFO 21062 --- [pool-12-thread-15,57f249d70b5822aa,73ba6341ea52955d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:27:28.693 INFO 21062 --- [ocp-schedules-20,de2a55e60405a9d4,5feb8a08ec186328] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:27:28.700 INFO 21062 --- [ocp-schedules-8,ccfb0a2f4f3482c5,ead81d67c33c3e6a] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:27:28.712 INFO 21062 --- [ocp-schedules-8,ccfb0a2f4f3482c5,ead81d67c33c3e6a] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:27:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:27:28.588024+08:00 2024-11-04 13:27:28.844 INFO 21062 --- [ocp-schedules-3,82efde1b81673d5a,cb7b7c35ec66ea5f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:27:28.844 INFO 21062 --- [ocp-schedules-3,82efde1b81673d5a,cb7b7c35ec66ea5f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:27:28.860 INFO 21062 --- [ocp-schedules-11,a8f74af231bfc21b,b51858eeae819b38] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:27:28.861 WARN 21062 --- [ocp-async-3,e811a17a285d19f2,1b57d5075301b3dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:28.887 INFO 21062 --- [ocp-schedules-23,f5b221de34bbd37d,30a6f885333eff67] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:27:28.891 INFO 21062 --- [ocp-schedules-23,f5b221de34bbd37d,30a6f885333eff67] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:27:28.892 INFO 21062 --- [ocp-schedules-23,f5b221de34bbd37d,30a6f885333eff67] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:27:28.910 INFO 21062 --- [ocp-async-8,226423ce229be485,b091a3b96da92f7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:29.055 INFO 21062 --- [ocp-schedules-9,9173ca9a1833de5c,b468a707509d4d1e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:27:29.055 INFO 21062 --- [ocp-schedules-9,9173ca9a1833de5c,b468a707509d4d1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:27:29.060 INFO 21062 --- [ocp-schedules-9,9173ca9a1833de5c,b468a707509d4d1e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:27:29.060 INFO 21062 --- [ocp-schedules-9,9173ca9a1833de5c,b468a707509d4d1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:27:29.065 INFO 21062 --- [ocp-schedules-9,9173ca9a1833de5c,b468a707509d4d1e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:27:29.065 INFO 21062 --- [ocp-schedules-9,9173ca9a1833de5c,b468a707509d4d1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:27:29.126 INFO 21062 --- [ocp-schedules-27,b21a64c6268c63e8,8544a065c0480670] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:27:29.126 INFO 21062 --- [ocp-schedules-27,b21a64c6268c63e8,8544a065c0480670] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:27:29.154 INFO 21062 --- [pool-30-thread-1,b21a64c6268c63e8,275b31fc5316ad1e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:27:29.183 WARN 21062 --- [ocp-schedules-27,b21a64c6268c63e8,8544a065c0480670] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:27:29.183 INFO 21062 --- [ocp-schedules-27,b21a64c6268c63e8,8544a065c0480670] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:27:29.458 INFO 21062 --- [ocp-schedules-7,715d57c6e2b03327,984f73eec0318ed3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:27:29.983 INFO 21062 --- [prometheus-minute-schedule-1,8bee508419e690ad,f83de412212f639e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:30.031 INFO 21062 --- [ocp-schedules-3,77eb10bdc23400ad,b0c5943f4cd26327] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:27:30.031 INFO 21062 --- [ocp-schedules-3,77eb10bdc23400ad,b0c5943f4cd26327] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:27:30.034 INFO 21062 --- [ocp-schedules-2,147248c6fdc94e9f,236498733f3f17ef] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:27:30.039 INFO 21062 --- [ocp-schedules-2,147248c6fdc94e9f,236498733f3f17ef] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:27:30.039 INFO 21062 --- [ocp-schedules-2,147248c6fdc94e9f,236498733f3f17ef] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:27:30.057 INFO 21062 --- [ocp-schedules-3,77eb10bdc23400ad,b0c5943f4cd26327] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:27:30.057 INFO 21062 --- [ocp-schedules-3,77eb10bdc23400ad,b0c5943f4cd26327] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:27:30.058 INFO 21062 --- [ocp-schedules-5,19a5d234cd11ad3b,109f613054be9114] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:27:30.185 INFO 21062 --- [ocp-schedules-23,c63833ca279e6290,ea4cfa2b8bea7159] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:27:30.185+08:00, clusters=[1] 2024-11-04 13:27:30.186 INFO 21062 --- [ocp-schedules-14,0c9af32a392ce4cc,9a767a46ce3111a1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:27:30.187 INFO 21062 --- [ocp-schedules-14,0c9af32a392ce4cc,9a767a46ce3111a1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:27:30.187 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:27:30.187 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:27:30.187 INFO 21062 --- [ocp-schedules-23,c63833ca279e6290,ea4cfa2b8bea7159] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:27:30.196 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:27:30.196 INFO 21062 --- [ocp-schedules-14,0c9af32a392ce4cc,9a767a46ce3111a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:27:30.198 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:27:30.198 WARN 21062 --- [ocp-schedules-14,0c9af32a392ce4cc,9a767a46ce3111a1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:27:30.198 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:27:30.199 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:27:30.199 INFO 21062 --- [ocp-schedules-14,0c9af32a392ce4cc,9a767a46ce3111a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:27:30.199 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:27:30.200 WARN 21062 --- [ocp-schedules-14,0c9af32a392ce4cc,9a767a46ce3111a1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:27:30.200 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:27:30.200 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:27:30.201 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:27:30.209 ERROR 21062 --- [ocp-schedules-14,0c9af32a392ce4cc,9a767a46ce3111a1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:27:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:27:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:27:30.316 INFO 21062 --- [ocp-schedules-13,8f31a88970c685fb,7b1ffb20cd27bffc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:27:30.316+08:00, clusters=[1] 2024-11-04 13:27:30.319 INFO 21062 --- [ocp-schedules-13,8f31a88970c685fb,7b1ffb20cd27bffc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:27:30.575 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:30.576 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:30.862 WARN 21062 --- [ocp-async-8,517d138011f54633,9c8d433364260f2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:30.910 INFO 21062 --- [ocp-async-10,ce8514c2d0cff5c8,317485a21ba7c34b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:31.042 INFO 21062 --- [ocp-schedules-18,288d0d06394d341b,8750f233516364c5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:27:32.274 INFO 21062 --- [ocp-schedules-10,6594ac359d308d3f,1b54b0995361f47f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:27:32.274 INFO 21062 --- [ocp-schedules-10,708ff275cf992934,8ea88c9c77a5bcd1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:27:32.278 INFO 21062 --- [ocp-schedules-10,708ff275cf992934,8ea88c9c77a5bcd1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:32.306 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:27:32.306 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:27:32.862 WARN 21062 --- [ocp-async-10,0787e8d2329c3a5c,10b9603423e56ab1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:32.911 INFO 21062 --- [ocp-async-4,877170fd0768eb88,8adaebf109416adc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7d0f4e9bbadc6d63,2524c5151706044c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:33.566 INFO 21062 --- [ocp-schedules-14,c4593087c8a391cd,e9dfacfb1937cf43] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:27:33.569 INFO 21062 --- [ocp-schedules-14,c4593087c8a391cd,e9dfacfb1937cf43] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:33.576 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:33.576 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:33.652 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:27:33.655 ERROR 21062 --- [ocp-schedules-14,c4593087c8a391cd,e9dfacfb1937cf43] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:27:33.655 INFO 21062 --- [ocp-schedules-14,c4593087c8a391cd,e9dfacfb1937cf43] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:27:34.862 WARN 21062 --- [ocp-async-4,e928749dae4e3b89,8074e10d05635a6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:34.911 INFO 21062 --- [ocp-async-2,504e8d4195cd2fa0,3971152e0885621f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1812110f923d9bea,b8d9e71be28e36d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:36.577 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:36.577 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:36.862 WARN 21062 --- [ocp-async-2,5ed3c0f6bddf47c1,db645f69c23950f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:36.911 INFO 21062 --- [ocp-async-9,a874cd563c3da4fd,1f54ff989dc6dbcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:38.504 INFO 21062 --- [ocp-schedules-26,e45d0ccfb8902a67,707c991b6dc08ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:27:38.505 INFO 21062 --- [ocp-schedules-17,7c03d771dea2a1de,3db67be22ca04e83] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:27:38.508 INFO 21062 --- [ocp-schedules-29,34d06f5a5cbdeb66,952acd4d84cdfea9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:27:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:27:38.589942+08:00, lastProcessTime=2024-11-04T13:27:28.588024+08:00 2024-11-04 13:27:38.598 INFO 21062 --- [ocp-async-8,41a63a092abd565c,ede55b9b01db6ed9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:27:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:27:38.589942+08:00 2024-11-04 13:27:38.862 WARN 21062 --- [ocp-async-7,f4ed441a1cba9a65,6559629690120e1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:38.911 INFO 21062 --- [ocp-async-3,b0cc998b7751fa73,997922336fb0ff87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:38.983 INFO 21062 --- [prometheus-minute-schedule-1,127a77116ded5aae,46819d84d537b266] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:39.461 INFO 21062 --- [ocp-schedules-18,5fea7b505dfa74d6,b0dadf233b31211e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:27:39.578 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:39.579 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:40.004 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:27:40.004 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:27:40.863 WARN 21062 --- [ocp-async-3,2e17418d4f8b39dc,07fe00e4010c6633] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:40.911 INFO 21062 --- [ocp-async-5,8d5ad59bc3f929ca,718f5d1b43a7f237] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:41.066 INFO 21062 --- [ocp-schedules-16,a823cfb5514fbea6,d4e031df12abb142] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:27:41.983 INFO 21062 --- [prometheus-minute-schedule-1,aad68c00594d7e27,71486a7079a0296f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:42.579 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:42.580 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:42.863 WARN 21062 --- [ocp-async-5,c9af22e5d52b89df,940e2a39e26eda4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:42.912 INFO 21062 --- [ocp-async-10,de1e23665f20a547,51a3c3ddd0314284] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:44.863 WARN 21062 --- [ocp-async-10,77bb1a93d491c2a1,a61d8311ccb8f1ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:44.912 INFO 21062 --- [ocp-async-4,79771a81c62fa88b,4b74bc96c96421ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:44.983 INFO 21062 --- [prometheus-minute-schedule-1,0372b5535e948a5f,5818ddb3ebaf55df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:45.580 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:45.581 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:46.863 WARN 21062 --- [ocp-async-4,4135ed0e0f404413,05335ce7e456be27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:46.912 INFO 21062 --- [ocp-async-6,6de4516f36695943,96eb43d07b5d28f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fe5ed934b65d75f3,87f6ed1cb81b2307] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:48.581 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:48.582 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:27:48.590420+08:00, lastProcessTime=2024-11-04T13:27:38.589942+08:00 2024-11-04 13:27:48.598 INFO 21062 --- [ocp-async-4,676c9c5d720c8939,1c3f3954dd4f7b39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:27:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:27:48.590420+08:00 2024-11-04 13:27:48.863 WARN 21062 --- [ocp-async-5,1da1901f8094fb37,6642b2a3dde2e050] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:48.912 INFO 21062 --- [ocp-async-9,411d47213e237f46,7292be9184018bcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:49.464 INFO 21062 --- [ocp-schedules-20,42c0c536da514697,28b88ae2e66ab57a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:27:50.045 INFO 21062 --- [ocp-schedules-2,8571e99eb5781f10,55db888356ef2b71] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:27:50.052 INFO 21062 --- [ocp-schedules-2,8571e99eb5781f10,55db888356ef2b71] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:27:50.052 INFO 21062 --- [ocp-schedules-2,8571e99eb5781f10,55db888356ef2b71] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:27:50.864 WARN 21062 --- [ocp-async-9,b13b611fbb60360f,f412bf7db768f1b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:50.912 INFO 21062 --- [ocp-async-1,edb1826399f17925,6e5f6d710e67f0de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1ed17f0ea0d3bbe1,c809d4b19383d1c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:51.108 INFO 21062 --- [ocp-schedules-30,17e662cec2eccbf6,f3173a97d3d0538f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:27:51.582 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:51.583 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:52.864 WARN 21062 --- [ocp-async-1,1222d4017d6bb35d,dadcbd49b52d7834] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:52.912 INFO 21062 --- [ocp-async-2,c019b0e8b71dab29,89965d86a4960968] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:53.983 INFO 21062 --- [prometheus-minute-schedule-1,83fd8bfa748dca98,f45066557a83e336] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:54.583 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:54.584 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:54.864 WARN 21062 --- [ocp-async-6,b5809e5420b14dbf,6f946875ad140c4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:54.913 INFO 21062 --- [ocp-async-8,5f1b4df6f7eb4c08,33bb78ed4497302a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:56.864 WARN 21062 --- [ocp-async-10,0d3a7d75d3c39e69,50286065640c7e87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:56.913 INFO 21062 --- [ocp-async-7,1197e7c2f71675b4,9d9fcfb7e0f71531] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:56.983 INFO 21062 --- [prometheus-minute-schedule-1,cb38976f2ec77970,2229a634e6de1eda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:27:57.584 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:27:57.585 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:27:58.500 INFO 21062 --- [ocp-schedules-30,a81e17840a95e1bb,68945fc23effc938] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:27:58.500 INFO 21062 --- [ocp-schedules-2,faeee01dd91aad8a,c8c3cdae8e3f8f9a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:27:58.502 INFO 21062 --- [ocp-schedules-22,5bd6a94c4b5fbc76,7a7dd1a9929a19ce] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:27:58.503 INFO 21062 --- [ocp-schedules-2,faeee01dd91aad8a,c8c3cdae8e3f8f9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:58.503 INFO 21062 --- [ocp-schedules-30,a81e17840a95e1bb,68945fc23effc938] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:27:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:27:58.504 INFO 21062 --- [ocp-schedules-19,1f9172ec1e5e61d2,02c1c7f72652ac0f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:27:58.510 INFO 21062 --- [ocp-schedules-14,ac2f17e6d8e1a3be,9653c0cf7a211a14] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:27:58.510 INFO 21062 --- [ocp-schedules-14,ac2f17e6d8e1a3be,9653c0cf7a211a14] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:27:58.510 INFO 21062 --- [ocp-schedules-9,5c019e28f73c296d,954a3e2ae43b5084] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:27:58.510 INFO 21062 --- [ocp-schedules-9,5c019e28f73c296d,954a3e2ae43b5084] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:27:58.510 INFO 21062 --- [ocp-schedules-16,5b407cf1ac772ea4,4b45ea4d6f858e1f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:27:58.510 INFO 21062 --- [ocp-schedules-16,5b407cf1ac772ea4,4b45ea4d6f858e1f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:27:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:27:58.589725+08:00, lastProcessTime=2024-11-04T13:27:48.590420+08:00 2024-11-04 13:27:58.598 INFO 21062 --- [ocp-async-10,fc9ddbdc7e74ec70,0d995142afa6f7e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:27:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:27:58.589725+08:00 2024-11-04 13:27:58.863 INFO 21062 --- [ocp-schedules-29,41a56c6ba2ea127d,c66ff45e125b4e43] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:27:58.864 WARN 21062 --- [ocp-async-1,ba702d66ad0c10a4,d738608ff28b2aec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:27:58.913 INFO 21062 --- [ocp-async-2,0aa74e9cea2633cb,6cec83971ae717cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:27:59.467 INFO 21062 --- [ocp-schedules-9,be77545741e8c044,6452288e07355edb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:27:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a39517b58982a09a,ed7c34bf198f9e71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698080, interval=60 2024-11-04 13:28:00.585 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:00.586 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:00.865 WARN 21062 --- [ocp-async-6,674a7181f755fe16,73eff23aab64cd42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:00.913 INFO 21062 --- [ocp-async-8,bd1d972fcca5694e,9e939c614abf7204] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:01.133 INFO 21062 --- [ocp-schedules-23,cf03fc48e66c8d53,ac686122b63c42f0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:28:02.306 INFO 21062 --- [ocp-schedules-10,708ff275cf992934,8ea88c9c77a5bcd1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:28:02.307 INFO 21062 --- [ocp-schedules-10,c8c6a4cceb914f4e,09951cc1e8b0723b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:28:02.310 INFO 21062 --- [ocp-schedules-10,c8c6a4cceb914f4e,09951cc1e8b0723b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:02.340 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:28:02.340 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:28:02.865 WARN 21062 --- [ocp-async-10,768689bc60a9c4ea,01a0f66ffbfaf8ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:02.913 INFO 21062 --- [ocp-async-7,3f65d722b7f562cb,825c010dbb40a851] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:02.983 INFO 21062 --- [prometheus-minute-schedule-1,8f4488db6b47e797,07c2142b20d6bdcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:03.586 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:03.587 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:04.865 WARN 21062 --- [ocp-async-4,3295aef60ede8bfb,9a7f35b84731225c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:04.914 INFO 21062 --- [ocp-async-3,6f98831754739360,d080bff720e2f367] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:05.983 INFO 21062 --- [prometheus-minute-schedule-1,cf67483161afc661,231729babe062dd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:06.587 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:06.588 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:06.865 WARN 21062 --- [ocp-async-2,34c783bf96c1cda8,256ec5c75d114cfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:06.914 INFO 21062 --- [ocp-async-5,b72d492cb6f92244,d7b6d05062277832] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:08.504 INFO 21062 --- [ocp-schedules-2,faeee01dd91aad8a,c8c3cdae8e3f8f9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:28:08.505 INFO 21062 --- [ocp-schedules-30,a81e17840a95e1bb,68945fc23effc938] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:28:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:28:08.590545+08:00, lastProcessTime=2024-11-04T13:27:58.589725+08:00 2024-11-04 13:28:08.598 INFO 21062 --- [ocp-async-2,75914d875e9d165c,18a636e236b70202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:28:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:28:08.590545+08:00 2024-11-04 13:28:08.865 WARN 21062 --- [ocp-async-10,336b2eeb272f8f5e,646333972482dcca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:08.914 INFO 21062 --- [ocp-async-7,269d26500e9f3619,1b0ab59da55d8346] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:08.983 INFO 21062 --- [prometheus-minute-schedule-1,e0af4ded17b4f16c,d7c6440ae5b03ea4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:09.470 INFO 21062 --- [ocp-schedules-4,f6b96082a1f691b5,d725b3082483212d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:28:09.588 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:09.589 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:28:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:28:10.055 INFO 21062 --- [ocp-schedules-22,4fce89feda578c14,a31becd65b62468e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:28:10.062 INFO 21062 --- [ocp-schedules-22,4fce89feda578c14,a31becd65b62468e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:28:10.062 INFO 21062 --- [ocp-schedules-22,4fce89feda578c14,a31becd65b62468e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:28:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:28:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:28:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:28:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:28:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:28:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:28:10.865 WARN 21062 --- [ocp-async-4,e7d922c5b807ad15,d70b012d7d026382] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:10.914 INFO 21062 --- [ocp-async-3,ab8593f83f4d0185,6148a2c549f9d2f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:11.173 INFO 21062 --- [ocp-schedules-6,3a51352b7602aed4,16004e36cc440f21] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:28:11.250 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=11, affectRows=11 2024-11-04 13:28:11.250 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=11, validCount=11 2024-11-04 13:28:11.250 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=3, affectRows=3 2024-11-04 13:28:11.250 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=15, affectRows=15 2024-11-04 13:28:11.250 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=3, validCount=3 2024-11-04 13:28:11.250 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=15, validCount=15 2024-11-04 13:28:11.250 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=46, affectRows=46 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=11, affectRows=11 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=11, validCount=11 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=39, affectRows=39 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=44, validCount=44 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=49, affectRows=49 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=26, affectRows=26 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=39, validCount=39 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=36, affectRows=36 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=26, affectRows=26 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=31, affectRows=31 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=25, validCount=25 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698020, nodeCount=29, affectRows=29 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=49, validCount=49 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=36, validCount=36 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=26, validCount=26 2024-11-04 13:28:11.251 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=31, validCount=31 2024-11-04 13:28:11.252 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698020, nodeCount=29, validCount=29 2024-11-04 13:28:11.983 INFO 21062 --- [prometheus-minute-schedule-1,053b29dff43516d7,4f540e9050405883] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:12.589 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:12.590 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:12.865 WARN 21062 --- [ocp-async-2,f2154c1cf529ba6c,b4ee591547f75967] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:12.914 INFO 21062 --- [ocp-async-5,19b317ae0305828d,a658e600abc65bae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:13.985 INFO 21062 --- [prometheus-cache-monitor1,e3cda4aff1b6e3d2,eece188b8d1324aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:28:13.985 INFO 21062 --- [prometheus-cache-monitor1,e3cda4aff1b6e3d2,eece188b8d1324aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:28:14.044 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:22:14.026, end=2024-11-04T13:27:14.041} 2024-11-04 13:28:14.044 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:22:14.026, end=2024-11-04T13:27:14.041} 2024-11-04 13:28:14.048 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:28:14.048 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:28:14.048 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:28:14.048 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:28:14.866 WARN 21062 --- [ocp-async-8,b1b8c80bba3e3d5b,9e03efb8d1d1c40d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:14.914 INFO 21062 --- [ocp-async-9,09bc6d375cf9c404,f90ede6e8d15ce8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9eb64cd3b91dcf95,8b1ff706dc1c5a1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:15.590 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:15.591 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:16.866 WARN 21062 --- [ocp-async-9,365e8c5c881b28a8,41e9e40ee2c1b159] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:16.915 INFO 21062 --- [ocp-async-1,f484f11e8ccbabbc,41ddec4ff50ab7af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:17.983 INFO 21062 --- [prometheus-minute-schedule-1,93c873e5ad56c6f1,00cf9f6fcd8b896c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:28:18.590194+08:00, lastProcessTime=2024-11-04T13:28:08.590545+08:00 2024-11-04 13:28:18.591 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:18.592 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:18.598 INFO 21062 --- [ocp-async-7,aaf9e832c14a68ed,9c9ff39e2acc04a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:28:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:28:18.590194+08:00 2024-11-04 13:28:18.789 INFO 21062 --- [ocp-schedules-27,a01d9879ad47e5c2,1c3490710762e979] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:28:18.793 INFO 21062 --- [ocp-schedules-27,a01d9879ad47e5c2,1c3490710762e979] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:18.793 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:28:18.866 WARN 21062 --- [ocp-async-3,58a46147a1a69598,063ba34853a131d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:18.915 INFO 21062 --- [ocp-async-5,bd7a26c7aa5be83e,f2b4b6c4a9ad381c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:19.473 INFO 21062 --- [ocp-schedules-25,5bf072642ab4d45b,4a14c14401bc17b1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:28:20.866 WARN 21062 --- [ocp-async-5,1c4717a080fa9cbb,7b7a8fe35ac29486] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:20.915 INFO 21062 --- [ocp-async-9,efa5d85fd6581eaf,d1b5aae1455359f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:20.983 INFO 21062 --- [prometheus-minute-schedule-1,0ca05ff3b9f40d06,33d0357c11732221] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:21.198 INFO 21062 --- [ocp-schedules-22,ad337390c90d75b5,6d054bb3681affc6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:28:21.592 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:21.594 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:22.866 WARN 21062 --- [ocp-async-9,f9a777558f65b3cc,da1c718ae193b8a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:22.915 INFO 21062 --- [ocp-async-1,d4d50b2167c71355,c467667f68658e9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a6a1eef46eaebe6e,8f2a47f9d1fe111a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:24.594 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:24.595 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:24.867 WARN 21062 --- [ocp-async-1,cd8a64d50868215e,6ff5939ec2d66718] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:24.915 INFO 21062 --- [ocp-async-6,b13b76dd5b815a6c,608a87ff34fa113e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:26.867 WARN 21062 --- [ocp-async-6,8f4dd3b3f6f7cd44,48da1f89be353b60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:26.915 INFO 21062 --- [ocp-async-10,afb1746a605d20d5,9cae4134d8c7ff9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3a86fc5a3d35c1ce,c70e8348c0be6b0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:26.984 INFO 21062 --- [prometheus-hour-schedule-1,709d709c78511c20,92b88324d0fc9653] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:28:27.595 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:27.596 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:28.498 INFO 21062 --- [ocp-schedules-5,bf8e59f92853fdc4,9a611e5653856904] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:28:28.499 INFO 21062 --- [ocp-schedules-18,0d0b5e471f8a5067,bd341309cf98f82c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:28:28.500 INFO 21062 --- [ocp-schedules-8,688a11e6a1e4b7bd,51793c937dac286a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:28:28.500 INFO 21062 --- [ocp-schedules-30,9e32d927b7216319,886643b60d500d92] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:28:28.502 INFO 21062 --- [ocp-schedules-5,bf8e59f92853fdc4,9a611e5653856904] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:28:28.502 INFO 21062 --- [ocp-schedules-12,ad9d0052ab490ea1,a11ad947dec241bb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:28:28.504 INFO 21062 --- [ocp-schedules-18,0d0b5e471f8a5067,bd341309cf98f82c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:28:28.504 INFO 21062 --- [ocp-schedules-8,688a11e6a1e4b7bd,51793c937dac286a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:28.504 INFO 21062 --- [ocp-schedules-23,680ba36387ad1900,6a6ceb56fa4f67c5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:28:28.505 INFO 21062 --- [ocp-schedules-30,9e32d927b7216319,886643b60d500d92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:28:28.511 INFO 21062 --- [ocp-schedules-22,1dd5c40539a6b4a9,4445cf1d58f821f0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:28:28.511 INFO 21062 --- [ocp-schedules-22,1dd5c40539a6b4a9,4445cf1d58f821f0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:28:28.512 INFO 21062 --- [ocp-schedules-9,1ec6c9f392fc8903,7684c8633506f1bc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:28:28.512 INFO 21062 --- [ocp-schedules-9,1ec6c9f392fc8903,7684c8633506f1bc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:28:28.513 INFO 21062 --- [ocp-schedules-24,83123016fe42d782,dad79ff33eee0de7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:28:28.513 INFO 21062 --- [ocp-schedules-24,83123016fe42d782,dad79ff33eee0de7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:28:28.565 INFO 21062 --- [ocp-schedules-11,35cecf319570d74e,7cfe316d35397a99] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:28:28.565 INFO 21062 --- [ocp-schedules-11,35cecf319570d74e,7cfe316d35397a99] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:28:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:28:28.589730+08:00, lastProcessTime=2024-11-04T13:28:18.590194+08:00 2024-11-04 13:28:28.598 INFO 21062 --- [ocp-async-10,f2573b41a989b849,028ae82c4aa726f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:28:28.695 INFO 21062 --- [ocp-schedules-17,902c9839d60ba4d5,e0b914f2032ac28a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:28:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:28:28.589730+08:00 2024-11-04 13:28:28.794 INFO 21062 --- [ocp-schedules-27,a01d9879ad47e5c2,1c3490710762e979] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:28:28.849 INFO 21062 --- [ocp-schedules-14,0dd0f1ea0d4f7b64,ec23600105d8e555] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:28:28.849 INFO 21062 --- [ocp-schedules-14,0dd0f1ea0d4f7b64,ec23600105d8e555] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:28:28.866 INFO 21062 --- [ocp-schedules-19,14ed1785766dafd5,b0a465fe4ba60d50] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:28:28.867 WARN 21062 --- [ocp-async-7,10853c4daf5c9521,3dd3f2cefde6f82c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:28.892 INFO 21062 --- [ocp-schedules-23,8ee7011044b605b4,e8430fc4e46104f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:28:28.899 INFO 21062 --- [ocp-schedules-23,8ee7011044b605b4,e8430fc4e46104f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:28:28.899 INFO 21062 --- [ocp-schedules-23,8ee7011044b605b4,e8430fc4e46104f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:28:28.916 INFO 21062 --- [ocp-async-2,cf5ae8aaeba148e1,f163764638ee83e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:28.925 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:28:28.925 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:28:28.934 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:28:28.934 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:28:28.934 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:28:28.945 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:28:28.945 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:28:28.945 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:28:28.949 INFO 21062 --- [ocp-schedules-24,fce3d79b166f446b,ce2f86c370026623] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:28:29.069 INFO 21062 --- [ocp-schedules-4,b7ab664a354b9d56,af62b73c0d04b8d1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:28:29.069 INFO 21062 --- [ocp-schedules-4,b7ab664a354b9d56,af62b73c0d04b8d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:28:29.070 INFO 21062 --- [ocp-schedules-4,b7ab664a354b9d56,af62b73c0d04b8d1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:28:29.070 INFO 21062 --- [ocp-schedules-4,b7ab664a354b9d56,af62b73c0d04b8d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:28:29.071 INFO 21062 --- [ocp-schedules-4,b7ab664a354b9d56,af62b73c0d04b8d1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:28:29.071 INFO 21062 --- [ocp-schedules-4,b7ab664a354b9d56,af62b73c0d04b8d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:28:29.476 INFO 21062 --- [ocp-schedules-15,0ac2b7aa890f397a,0e340b9a201b0487] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:28:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b56ab2a0ff193a2d,61f1371dde28d55c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:30.057 INFO 21062 --- [ocp-schedules-26,2772938b4f96d8aa,ea5ded9251a4aebd] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:28:30.057 INFO 21062 --- [ocp-schedules-26,2772938b4f96d8aa,ea5ded9251a4aebd] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:28:30.064 INFO 21062 --- [ocp-schedules-12,14b9b43f93a08bca,7b23f4179c60fe8a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:28:30.070 INFO 21062 --- [ocp-schedules-12,14b9b43f93a08bca,7b23f4179c60fe8a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:28:30.070 INFO 21062 --- [ocp-schedules-12,14b9b43f93a08bca,7b23f4179c60fe8a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:28:30.084 INFO 21062 --- [ocp-schedules-26,2772938b4f96d8aa,ea5ded9251a4aebd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:28:30.084 INFO 21062 --- [ocp-schedules-26,2772938b4f96d8aa,ea5ded9251a4aebd] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:28:30.084 INFO 21062 --- [ocp-schedules-18,2f15f792342d77d1,8dca08f60663076a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:28:30.215 INFO 21062 --- [ocp-schedules-22,68d60415e34c1e01,3839d9b783190029] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:28:30.215+08:00, clusters=[1] 2024-11-04 13:28:30.217 INFO 21062 --- [ocp-schedules-22,68d60415e34c1e01,3839d9b783190029] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:28:30.217 INFO 21062 --- [ocp-schedules-24,95c1274ea24121a1,cc7613f21167d4ea] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:28:30.217 INFO 21062 --- [ocp-schedules-24,95c1274ea24121a1,cc7613f21167d4ea] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:28:30.218 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:28:30.218 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:28:30.220 INFO 21062 --- [ocp-schedules-5,21c6a56a35f92e98,2bbda7a3f09c1f0f] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:28:30.224 INFO 21062 --- [ocp-schedules-5,21c6a56a35f92e98,2bbda7a3f09c1f0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:30.227 INFO 21062 --- [ocp-schedules-24,95c1274ea24121a1,cc7613f21167d4ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:28:30.228 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:28:30.228 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:28:30.230 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:28:30.230 WARN 21062 --- [ocp-schedules-24,95c1274ea24121a1,cc7613f21167d4ea] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:28:30.230 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:28:30.230 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:28:30.230 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:28:30.230 INFO 21062 --- [ocp-schedules-24,95c1274ea24121a1,cc7613f21167d4ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:28:30.232 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:28:30.232 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:28:30.232 WARN 21062 --- [ocp-schedules-24,95c1274ea24121a1,cc7613f21167d4ea] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:28:30.241 ERROR 21062 --- [ocp-schedules-24,95c1274ea24121a1,cc7613f21167d4ea] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:28:30.280 ERROR 21062 --- [ocp-schedules-5,21c6a56a35f92e98,2bbda7a3f09c1f0f] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:28:30.280 INFO 21062 --- [ocp-schedules-5,21c6a56a35f92e98,2bbda7a3f09c1f0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:28:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:28:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:28:30.344 INFO 21062 --- [ocp-schedules-28,ca0f4e3bf11c7a4c,7acacfbd98ea7b9e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:28:30.344+08:00, clusters=[1] 2024-11-04 13:28:30.348 INFO 21062 --- [ocp-schedules-28,ca0f4e3bf11c7a4c,7acacfbd98ea7b9e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:28:30.597 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:30.597 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:30.867 WARN 21062 --- [ocp-async-2,1ffc6d2db95c8aaa,63a0e8c729085dbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:30.916 INFO 21062 --- [ocp-async-8,4a132f1501c1cdf4,14e7158e0a2c636e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:31.237 INFO 21062 --- [ocp-schedules-27,df12354122306ab7,7c64623a1a2880ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:28:32.341 INFO 21062 --- [ocp-schedules-10,c8c6a4cceb914f4e,09951cc1e8b0723b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:28:32.341 INFO 21062 --- [ocp-schedules-10,f10cd0683772a69c,357f83f9633d90c3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:28:32.345 INFO 21062 --- [ocp-schedules-10,f10cd0683772a69c,357f83f9633d90c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:32.375 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:28:32.375 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:28:32.867 WARN 21062 --- [ocp-async-8,583ccded657abc39,14d75bec677726ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:32.916 INFO 21062 --- [ocp-async-4,bd0c035de9bc6811,1205fb14d8c7801a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7aa90aa5deef590b,86fe1fa4d2acab13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:33.597 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:33.598 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:33.656 INFO 21062 --- [ocp-schedules-12,34e0b12052b4516d,5f222ac7d73abecf] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:28:33.660 INFO 21062 --- [ocp-schedules-12,34e0b12052b4516d,5f222ac7d73abecf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:33.748 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:28:33.750 ERROR 21062 --- [ocp-schedules-12,34e0b12052b4516d,5f222ac7d73abecf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:28:33.750 INFO 21062 --- [ocp-schedules-12,34e0b12052b4516d,5f222ac7d73abecf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:28:34.867 WARN 21062 --- [ocp-async-4,4d43be61f09daaff,8e0f37f33bd9ff3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:34.916 INFO 21062 --- [ocp-async-3,0586e247b270515f,87009171883d95e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:35.983 INFO 21062 --- [prometheus-minute-schedule-1,fe9427526a0240c0,9809b3abfa01dbcd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:36.598 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:36.599 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:36.867 WARN 21062 --- [ocp-async-3,8f0ff6707054227d,a0c308c106f84112] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:36.916 INFO 21062 --- [ocp-async-6,36d45f8b3c65033e,d4ec2262dd33d733] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:38.505 INFO 21062 --- [ocp-schedules-30,9e32d927b7216319,886643b60d500d92] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:28:38.506 INFO 21062 --- [ocp-schedules-8,688a11e6a1e4b7bd,51793c937dac286a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:28:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:28:38.589906+08:00, lastProcessTime=2024-11-04T13:28:28.589730+08:00 2024-11-04 13:28:38.599 INFO 21062 --- [ocp-async-2,2ce53ba70234ae62,aee5d280bbb5e437] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:28:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:28:38.589906+08:00 2024-11-04 13:28:38.868 WARN 21062 --- [ocp-async-10,b211960abf1bd87e,3c4910aee074c9f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:38.916 INFO 21062 --- [ocp-async-7,b3052c84d09b81ab,86495e7879e2947e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:38.983 INFO 21062 --- [prometheus-minute-schedule-1,856e1b2f7886ce90,20e851d723000cfd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:39.480 INFO 21062 --- [ocp-schedules-6,5aa886ae655cf382,b7fa104b1bd6c750] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:28:39.599 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:39.600 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:28:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:28:40.868 WARN 21062 --- [ocp-async-7,8e0d4ca395cb09cf,0c420562ec64a6a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:40.917 INFO 21062 --- [ocp-async-5,09cd35cc3a2b9375,3138455ebcacc9be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:41.261 INFO 21062 --- [ocp-schedules-21,235637fd9816ec8f,9e6d3d4b0f361b22] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:28:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:28:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:28:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:28:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:28:41.802 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:28:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:28:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:28:41.983 INFO 21062 --- [prometheus-minute-schedule-1,5b0564eefccf72ea,8e501cadc11a929f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:42.601 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:42.603 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:42.868 WARN 21062 --- [ocp-async-5,841dbcfeab4d2051,91f6220cf5aa9837] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:42.917 INFO 21062 --- [ocp-async-8,d4bf9a0042a247d5,434c9a2e68eeaf13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:44.868 WARN 21062 --- [ocp-async-8,fe6d969a72153dc0,e4bb11068beec801] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:44.917 INFO 21062 --- [ocp-async-4,4b8df55fdd5559bb,76ead3a596d87de7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:44.983 INFO 21062 --- [prometheus-minute-schedule-1,049af19e531d83e7,c68598f35301db81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:45.604 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:45.605 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:46.868 WARN 21062 --- [ocp-async-4,b136d235ad31266e,e8135b8b70491dab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:46.917 INFO 21062 --- [ocp-async-1,5a4dfe8ced1a8cf5,64ddae6bb9fd4ea4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:47.983 INFO 21062 --- [prometheus-minute-schedule-1,0dd08c807fa603f6,e5a7a2629578bc2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:28:48.589909+08:00, lastProcessTime=2024-11-04T13:28:38.589906+08:00 2024-11-04 13:28:48.599 INFO 21062 --- [ocp-async-4,84be40d26cf5d45b,5d44ac0ebfb4f3d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:28:48.605 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:48.605 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:28:48.589909+08:00 2024-11-04 13:28:48.868 WARN 21062 --- [ocp-async-5,7606935da92a2e9b,55ec1f31b553d835] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:48.917 INFO 21062 --- [ocp-async-6,a25dc39fed8410f8,0b91cb30451d1ff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:49.482 INFO 21062 --- [ocp-schedules-11,d05159dad8f94be6,4ed58111a3b1a168] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:28:50.073 INFO 21062 --- [ocp-schedules-15,a1b2f5c42c016cd2,ce1f3daebbd09932] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:28:50.079 INFO 21062 --- [ocp-schedules-15,a1b2f5c42c016cd2,ce1f3daebbd09932] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:28:50.079 INFO 21062 --- [ocp-schedules-15,a1b2f5c42c016cd2,ce1f3daebbd09932] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:28:50.869 WARN 21062 --- [ocp-async-6,7e8cb4dfa5292908,4ea17a0cd1628227] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:50.918 INFO 21062 --- [ocp-async-9,d01fde9f4d6a0777,110b57800395d38c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:50.983 INFO 21062 --- [prometheus-minute-schedule-1,39bc1c9618bbadc7,35dad4f7b16eb98b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:51.304 INFO 21062 --- [ocp-schedules-7,681fcdd52f59c242,a8dac2079b1e4e9c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:28:51.605 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:51.606 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:52.869 WARN 21062 --- [ocp-async-9,abc267ae8bbd1d2c,f1b1235fea282c89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:52.918 INFO 21062 --- [ocp-async-3,4818af8db41389d4,1deedb230e96a384] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:53.983 INFO 21062 --- [prometheus-minute-schedule-1,8e269e2db4d471ad,be67b8cae62448ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:54.607 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:54.607 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:54.869 WARN 21062 --- [ocp-async-3,6fd37943dbb6f61c,9a57d143466b2b05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:54.918 INFO 21062 --- [ocp-async-2,d2e225d02bd9adeb,3a9a228d64ebb5cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:56.869 WARN 21062 --- [ocp-async-8,6417987d99ed96a8,85542685f08c9230] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:56.918 INFO 21062 --- [ocp-async-10,ad7f3ca78f39da66,13d56a503aa02ea8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:56.983 INFO 21062 --- [prometheus-minute-schedule-1,596d3394c4ea44c0,54164cba7164d681] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:28:57.607 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:28:57.608 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:28:58.500 INFO 21062 --- [ocp-schedules-13,d518ac14bcf6ce06,5e6f5c8734fc0eb7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:28:58.500 INFO 21062 --- [ocp-schedules-26,33c98fad9a06e58a,26d4dc36a01efdfc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:28:58.502 INFO 21062 --- [ocp-schedules-16,cb4ebdd9e7bde256,fd0c9b71a233bbd1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:28:58.504 INFO 21062 --- [ocp-schedules-13,d518ac14bcf6ce06,5e6f5c8734fc0eb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:58.504 INFO 21062 --- [ocp-schedules-26,33c98fad9a06e58a,26d4dc36a01efdfc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:28:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:28:58.506 INFO 21062 --- [ocp-schedules-25,f7cdce146e23c17a,6f556d2a61285276] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:28:58.509 INFO 21062 --- [ocp-schedules-1,50f8822916ccfd4e,19fbb3114e8c1cae] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:28:58.509 INFO 21062 --- [ocp-schedules-1,50f8822916ccfd4e,19fbb3114e8c1cae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:28:58.510 INFO 21062 --- [ocp-schedules-23,808603b14fbe7e02,b3233c4d2ad8e003] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:28:58.510 INFO 21062 --- [ocp-schedules-23,808603b14fbe7e02,b3233c4d2ad8e003] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:28:58.510 INFO 21062 --- [ocp-schedules-15,34591caf4a231995,56c80b4001c7f0dd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:28:58.510 INFO 21062 --- [ocp-schedules-15,34591caf4a231995,56c80b4001c7f0dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:28:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:28:58.590117+08:00, lastProcessTime=2024-11-04T13:28:48.589909+08:00 2024-11-04 13:28:58.599 INFO 21062 --- [ocp-async-8,b6d33fdaabdbf7d5,c1bfe35510e2458c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:28:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:28:58.590117+08:00 2024-11-04 13:28:58.868 INFO 21062 --- [ocp-schedules-27,c791a3a4522c2c85,6df3a315005dccf5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:28:58.869 WARN 21062 --- [ocp-async-9,909e31ee6ea731e2,7a65b4463cdacdb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:28:58.918 INFO 21062 --- [ocp-async-1,648c515788d9f235,1b1b59e186914aa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:28:59.485 INFO 21062 --- [ocp-schedules-4,7ba562ebd81f6c2e,d8a1a4ae1ba63368] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:28:59.983 INFO 21062 --- [prometheus-minute-schedule-1,958e65a0caf67129,022beb8f6f4505f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698140, interval=60 2024-11-04 13:29:00.608 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:00.609 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:00.870 WARN 21062 --- [ocp-async-3,898e05907a065747,4c56a7c2e1a37d7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:00.919 INFO 21062 --- [ocp-async-2,4c0c5a4cbdfe3ee7,3b3627df128d806e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:01.329 INFO 21062 --- [ocp-schedules-14,b3ecbaafa298ede7,008364a0afd0a9e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:29:02.376 INFO 21062 --- [ocp-schedules-10,f10cd0683772a69c,357f83f9633d90c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:29:02.376 INFO 21062 --- [ocp-schedules-10,7e2e536463f25f8c,593ebf49f9ed3d9a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:29:02.380 INFO 21062 --- [ocp-schedules-10,7e2e536463f25f8c,593ebf49f9ed3d9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:02.409 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:29:02.409 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:29:02.870 WARN 21062 --- [ocp-async-8,c9de7d8d8acb7035,543a016a9f3a86a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:02.919 INFO 21062 --- [ocp-async-10,68ecbb987eb15a3d,3bb50c88457cf62e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7a5cb65414a44f2b,2ef8a0ff627f6646] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:03.610 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:03.611 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:04.870 WARN 21062 --- [ocp-async-4,21989b86cf4390a3,15a8d10fbaf9335c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:04.919 INFO 21062 --- [ocp-async-7,7dd945bc217c94a2,63d553da58e68647] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2e6b54b7cc47f500,1410367be46c05f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:06.611 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:06.611 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:06.870 WARN 21062 --- [ocp-async-1,3dd64ff094f0b88a,9e49c57a951bc529] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:06.919 INFO 21062 --- [ocp-async-5,a152471dc3096b94,e44ca7daf7025664] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:08.505 INFO 21062 --- [ocp-schedules-26,33c98fad9a06e58a,26d4dc36a01efdfc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:29:08.505 INFO 21062 --- [ocp-schedules-13,d518ac14bcf6ce06,5e6f5c8734fc0eb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:29:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:29:08.590783+08:00, lastProcessTime=2024-11-04T13:28:58.590117+08:00 2024-11-04 13:29:08.599 INFO 21062 --- [ocp-async-1,d74f5ef3fd990fa8,4f4fe1876de35fad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:29:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:29:08.590783+08:00 2024-11-04 13:29:08.870 WARN 21062 --- [ocp-async-8,648373d303645f97,83fd8115ff91c57e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:08.920 INFO 21062 --- [ocp-async-10,8299b60a0f95e4de,8d4b511706186e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8e58502887009302,c6430c588a091181] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:09.490 INFO 21062 --- [ocp-schedules-17,1201ba34a8b60cbc,779a3f19d9fa62bb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:29:09.612 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:09.612 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:29:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:29:10.082 INFO 21062 --- [ocp-schedules-6,3cc49b7896b3e7be,6162ac4fa66a6021] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:29:10.088 INFO 21062 --- [ocp-schedules-6,3cc49b7896b3e7be,6162ac4fa66a6021] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:29:10.088 INFO 21062 --- [ocp-schedules-6,3cc49b7896b3e7be,6162ac4fa66a6021] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:29:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:29:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:29:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:29:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:29:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:29:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:29:10.871 WARN 21062 --- [ocp-async-4,370be3aca5ce6088,fb3217ffdcd6acb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:10.920 INFO 21062 --- [ocp-async-7,fe82df96488123fb,ecc4b7c12a094fab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:11.252 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=1, affectRows=1 2024-11-04 13:29:11.252 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=1, validCount=1 2024-11-04 13:29:11.253 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=1, affectRows=1 2024-11-04 13:29:11.253 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=1, validCount=1 2024-11-04 13:29:11.253 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=25, affectRows=25 2024-11-04 13:29:11.253 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=9, affectRows=9 2024-11-04 13:29:11.253 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=25, validCount=25 2024-11-04 13:29:11.253 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=9, validCount=9 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=45, affectRows=45 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=56, affectRows=56 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=17, affectRows=17 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=17, validCount=17 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=45, validCount=45 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=53, affectRows=53 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=56, validCount=56 2024-11-04 13:29:11.254 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=52, validCount=52 2024-11-04 13:29:11.255 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=71, affectRows=71 2024-11-04 13:29:11.255 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=69, validCount=69 2024-11-04 13:29:11.255 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698080, nodeCount=44, affectRows=44 2024-11-04 13:29:11.255 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698080, nodeCount=44, validCount=44 2024-11-04 13:29:11.373 INFO 21062 --- [ocp-schedules-28,56cef894da5c29ca,0f5cac3d0d0995fc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:29:11.983 INFO 21062 --- [prometheus-minute-schedule-1,d455e3690803291a,9689e42bf0f3f3af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:12.613 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:12.614 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:12.871 WARN 21062 --- [ocp-async-1,9ab7f024f462ec83,77b59e3972cf33aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:12.920 INFO 21062 --- [ocp-async-5,73083605c665e962,4a3fd9a28315f828] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:13.985 INFO 21062 --- [prometheus-cache-monitor1,0188c86cc566329c,27d74cc9e754b63d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:29:13.985 INFO 21062 --- [prometheus-cache-monitor1,0188c86cc566329c,27d74cc9e754b63d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:29:14.556 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:25:14.533, end=2024-11-04T13:27:14.553} 2024-11-04 13:29:14.560 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:29:14.560 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:29:14.871 WARN 21062 --- [ocp-async-2,6bae49eff21d3760,8f17e7911e46dff9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:14.920 INFO 21062 --- [ocp-async-6,da2008e9de047b84,298d78711ce1028f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3f21e6d37955575c,364f8d74d326527a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:15.614 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:15.615 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:16.871 WARN 21062 --- [ocp-async-10,0345d1543dee8292,fe075c6c86ce746e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:16.920 INFO 21062 --- [ocp-async-9,b32ed2df5bbaec4a,befa8aebd9968a54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ec915344b46237dc,261ce6f7bb07050a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:29:18.590056+08:00, lastProcessTime=2024-11-04T13:29:08.590783+08:00 2024-11-04 13:29:18.599 INFO 21062 --- [ocp-async-10,0d950b76ff7e1962,50946687dea4907a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:29:18.615 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:18.615 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:29:18.590056+08:00 2024-11-04 13:29:18.871 WARN 21062 --- [ocp-async-7,6c77a4a4da6ed7f0,13152e25928100f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:18.920 INFO 21062 --- [ocp-async-5,97f6a4648d80ba8e,1d46f299c3f22812] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:19.493 INFO 21062 --- [ocp-schedules-15,6cf370be3e457224,bd883d059125b3a9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:29:20.872 WARN 21062 --- [ocp-async-5,339203ef19ea4111,1c8b2d28ac1508b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:20.921 INFO 21062 --- [ocp-async-6,afad384bd7b1acc3,66e18f26b7082318] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f3109904f9edaf88,3665c1e4a25928f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:21.401 INFO 21062 --- [ocp-schedules-1,cd1fabe168b84bdf,3ed47c34388cc047] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:29:21.616 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:21.616 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:22.872 WARN 21062 --- [ocp-async-6,6793580452e06239,006f9a9ead2e6c23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:22.921 INFO 21062 --- [ocp-async-9,216104812e3787c3,03bb972f4e2dd780] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4647c8da1733fc0e,fb18229b90856a7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:24.616 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:24.617 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:24.872 WARN 21062 --- [ocp-async-9,52d578bb1ebe5bbe,a3964f34012d9e23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:24.921 INFO 21062 --- [ocp-async-3,870aff37d43d3e40,0bedbc58d030b31a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:26.872 WARN 21062 --- [ocp-async-3,010dbe636b59555d,ed33a08c436af9f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:26.921 INFO 21062 --- [ocp-async-8,30c20028e482c5d0,9c65383dded029df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:26.983 INFO 21062 --- [prometheus-minute-schedule-1,cabab4babf583c08,d9f051f384433898] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f0e12775f2472c1a,4f938c207c3f926a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:29:27.617 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:27.618 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:28.498 INFO 21062 --- [ocp-schedules-12,076c7c76d8139099,0c1cb669ab7dc8a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:29:28.499 INFO 21062 --- [ocp-schedules-9,895351aacd78325f,cfbdd9be33f1e539] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:29:28.500 INFO 21062 --- [ocp-schedules-11,0a6c4f21897568f4,4b3c5fed637d58d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:29:28.500 INFO 21062 --- [ocp-schedules-7,17dd0f6e6720b6d2,71a580738838b866] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:29:28.502 INFO 21062 --- [ocp-schedules-29,653ca8d00b1877bb,183cec27930ef5ff] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:29:28.503 INFO 21062 --- [ocp-schedules-12,076c7c76d8139099,0c1cb669ab7dc8a7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:29:28.504 INFO 21062 --- [ocp-schedules-11,0a6c4f21897568f4,4b3c5fed637d58d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:28.504 INFO 21062 --- [ocp-schedules-7,17dd0f6e6720b6d2,71a580738838b866] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:28.504 INFO 21062 --- [ocp-schedules-27,5a61d312090f0ab7,a12e18b1f9b9ad47] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:29:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:29:28.506 INFO 21062 --- [ocp-schedules-9,895351aacd78325f,cfbdd9be33f1e539] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:29:28.512 INFO 21062 --- [ocp-schedules-6,d4bf44b316d43950,698caae48a781359] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:29:28.512 INFO 21062 --- [ocp-schedules-25,2385b2718c63cf19,d992ba815d3a5d54] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:29:28.512 INFO 21062 --- [ocp-schedules-6,d4bf44b316d43950,698caae48a781359] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:29:28.512 INFO 21062 --- [ocp-schedules-25,2385b2718c63cf19,d992ba815d3a5d54] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:29:28.512 INFO 21062 --- [ocp-schedules-23,93a7ab5829617c7c,5da578c0bab590c0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:29:28.512 INFO 21062 --- [ocp-schedules-23,93a7ab5829617c7c,5da578c0bab590c0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:29:28.556 INFO 21062 --- [ocp-schedules-30,2d6b12898db9ad1b,aac54b7a4e765193] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:29:28.556+08:00, tenantSize=0 2024-11-04 13:29:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:29:28.589081+08:00, lastProcessTime=2024-11-04T13:29:18.590056+08:00 2024-11-04 13:29:28.599 INFO 21062 --- [ocp-async-8,816831e51f45452b,c860a9e0c5ad942c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:29:28.697 INFO 21062 --- [ocp-schedules-24,5e966ea118bf7b31,d74510765a3ed1be] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:29:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:29:28.589081+08:00 2024-11-04 13:29:28.794 INFO 21062 --- [ocp-schedules-20,c17fe938bc874416,07d386fe373a4ba9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:29:28.798 INFO 21062 --- [ocp-schedules-20,c17fe938bc874416,07d386fe373a4ba9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:28.798 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:29:28.853 INFO 21062 --- [ocp-schedules-5,541555c3874e401e,af2dddc22fbdf70c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:29:28.853 INFO 21062 --- [ocp-schedules-5,541555c3874e401e,af2dddc22fbdf70c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:29:28.871 INFO 21062 --- [ocp-schedules-1,7651fae56e1d64a4,3958ce35768c4b81] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:29:28.872 WARN 21062 --- [ocp-async-10,ac0e950c624ee7d1,eefd604db146366a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:28.899 INFO 21062 --- [ocp-schedules-27,074231200b1f2cb3,c857ac9c40eb311d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:29:28.904 INFO 21062 --- [ocp-schedules-27,074231200b1f2cb3,c857ac9c40eb311d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:29:28.904 INFO 21062 --- [ocp-schedules-27,074231200b1f2cb3,c857ac9c40eb311d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:29:28.921 INFO 21062 --- [ocp-async-1,42143021f5a1d3ad,ddabeea9c0953498] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:29.072 INFO 21062 --- [ocp-schedules-15,0d3c294f1f1c94a6,f6e41d6992cd3eec] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:29:29.072 INFO 21062 --- [ocp-schedules-15,0d3c294f1f1c94a6,f6e41d6992cd3eec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:29:29.073 INFO 21062 --- [ocp-schedules-15,0d3c294f1f1c94a6,f6e41d6992cd3eec] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:29:29.073 INFO 21062 --- [ocp-schedules-15,0d3c294f1f1c94a6,f6e41d6992cd3eec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:29:29.074 INFO 21062 --- [ocp-schedules-15,0d3c294f1f1c94a6,f6e41d6992cd3eec] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:29:29.074 INFO 21062 --- [ocp-schedules-15,0d3c294f1f1c94a6,f6e41d6992cd3eec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:29:29.496 INFO 21062 --- [ocp-schedules-18,86f94c696eb7bdb8,aa861e5b60a3075f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:29:29.983 INFO 21062 --- [prometheus-minute-schedule-1,82f41acfa8b31c78,f60e5e83b1975be2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:30.084 INFO 21062 --- [ocp-schedules-25,95d3b42f4f872aa2,00706f87b7b2f10a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:29:30.084 INFO 21062 --- [ocp-schedules-25,95d3b42f4f872aa2,00706f87b7b2f10a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:29:30.090 INFO 21062 --- [ocp-schedules-9,85a91c986a223982,97bcc8ca140bdc8d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:29:30.096 INFO 21062 --- [ocp-schedules-9,85a91c986a223982,97bcc8ca140bdc8d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:29:30.096 INFO 21062 --- [ocp-schedules-9,85a91c986a223982,97bcc8ca140bdc8d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:29:30.111 INFO 21062 --- [ocp-schedules-25,95d3b42f4f872aa2,00706f87b7b2f10a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:29:30.111 INFO 21062 --- [ocp-schedules-25,95d3b42f4f872aa2,00706f87b7b2f10a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:29:30.113 INFO 21062 --- [ocp-schedules-27,954620c13065c409,3bf8ffb24b96b431] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:29:30.242 INFO 21062 --- [ocp-schedules-2,2edbe2d02060a19f,a82f3f38a4525ca9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:29:30.242+08:00, clusters=[1] 2024-11-04 13:29:30.243 INFO 21062 --- [ocp-schedules-2,2edbe2d02060a19f,a82f3f38a4525ca9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:29:30.248 INFO 21062 --- [ocp-schedules-12,1be6acdb9d1d7185,e55b0a8556ff4c43] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:29:30.248 INFO 21062 --- [ocp-schedules-12,1be6acdb9d1d7185,e55b0a8556ff4c43] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:29:30.248 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:29:30.248 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:29:30.258 INFO 21062 --- [ocp-schedules-12,1be6acdb9d1d7185,e55b0a8556ff4c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:29:30.259 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:29:30.259 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:29:30.260 WARN 21062 --- [ocp-schedules-12,1be6acdb9d1d7185,e55b0a8556ff4c43] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:29:30.261 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:29:30.261 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:29:30.261 INFO 21062 --- [ocp-schedules-12,1be6acdb9d1d7185,e55b0a8556ff4c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:29:30.261 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:29:30.261 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:29:30.262 WARN 21062 --- [ocp-schedules-12,1be6acdb9d1d7185,e55b0a8556ff4c43] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:29:30.262 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:29:30.263 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:29:30.271 ERROR 21062 --- [ocp-schedules-12,1be6acdb9d1d7185,e55b0a8556ff4c43] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:29:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:29:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:29:30.374 INFO 21062 --- [ocp-schedules-4,fee84194a2a7c217,898f2de24ba03690] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:29:30.374+08:00, clusters=[1] 2024-11-04 13:29:30.377 INFO 21062 --- [ocp-schedules-4,fee84194a2a7c217,898f2de24ba03690] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:29:30.618 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:30.619 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:30.873 WARN 21062 --- [ocp-async-1,245f9ec37a8a98ab,a86be7a28ffff25f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:30.921 INFO 21062 --- [ocp-async-2,5b8fd24c4a029e88,8c3ad4476afbbf4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:31.443 INFO 21062 --- [ocp-schedules-28,360c77abd87f8abf,899735345380aa9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:29:32.410 INFO 21062 --- [ocp-schedules-10,7e2e536463f25f8c,593ebf49f9ed3d9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:29:32.410 INFO 21062 --- [ocp-schedules-10,7bd6a930aa45cee0,f9fc34f446bdb629] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:29:32.414 INFO 21062 --- [ocp-schedules-10,7bd6a930aa45cee0,f9fc34f446bdb629] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:32.443 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:29:32.443 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:29:32.873 WARN 21062 --- [ocp-async-2,a7ed1224e63cd898,ea59d728cfc2ed7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:32.922 INFO 21062 --- [ocp-async-4,bf0911d420a4daa6,3e19da1f79065d94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:32.983 INFO 21062 --- [prometheus-minute-schedule-1,cc38199c65b7510d,6e9fb5c0a2d33302] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:33.619 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:33.620 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:33.750 INFO 21062 --- [ocp-schedules-13,97942b38014810c2,5761c6ccdf0e4cc1] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:29:33.754 INFO 21062 --- [ocp-schedules-13,97942b38014810c2,5761c6ccdf0e4cc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:33.848 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:29:33.850 ERROR 21062 --- [ocp-schedules-13,97942b38014810c2,5761c6ccdf0e4cc1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:29:33.851 INFO 21062 --- [ocp-schedules-13,97942b38014810c2,5761c6ccdf0e4cc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:29:34.873 WARN 21062 --- [ocp-async-4,c6411383155ba0a9,162f3e9da2f413a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:34.922 INFO 21062 --- [ocp-async-7,7435977e4b891d8c,899b61ca7f07330b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0478c92d8b6a15c3,7d2cd3b55b3ffaec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:36.620 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:36.622 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:36.873 WARN 21062 --- [ocp-async-7,886638899da8bd40,4a9f9add1192b282] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:36.922 INFO 21062 --- [ocp-async-3,4bea410347ea6c42,77fb3dc3812d3a6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:38.504 INFO 21062 --- [ocp-schedules-7,17dd0f6e6720b6d2,71a580738838b866] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:29:38.505 INFO 21062 --- [ocp-schedules-11,0a6c4f21897568f4,4b3c5fed637d58d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:29:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:29:38.589902+08:00, lastProcessTime=2024-11-04T13:29:28.589081+08:00 2024-11-04 13:29:38.600 INFO 21062 --- [ocp-async-1,42eca2554e803c16,034451638697e095] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:29:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:29:38.589902+08:00 2024-11-04 13:29:38.799 INFO 21062 --- [ocp-schedules-20,c17fe938bc874416,07d386fe373a4ba9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:29:38.873 WARN 21062 --- [ocp-async-8,22f791a2a4209990,677907e148a6632b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:38.922 INFO 21062 --- [ocp-async-10,e43941506f0eaddd,3bf4a3e5c1dd8ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:38.983 INFO 21062 --- [prometheus-minute-schedule-1,f04a27524c86f543,ff07fd2a27f7f051] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:39.499 INFO 21062 --- [ocp-schedules-11,6cf5bb97705cdbbb,d00d3d24fd8c5457] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:29:39.622 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:39.623 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:40.004 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:29:40.004 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:29:40.873 WARN 21062 --- [ocp-async-10,882a80e6120bdb90,77c1f52f663fe60a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:40.922 INFO 21062 --- [ocp-async-5,297dea5f1d7418ef,8dfe3889dd9e67c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:41.466 INFO 21062 --- [ocp-schedules-16,1250cbc4223c0c00,27924b944e28aee3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:29:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6a22e656328f0230,ed3a934940e9fe98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:42.623 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:42.625 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:42.873 WARN 21062 --- [ocp-async-5,87b420bbd112014c,0791c81f8f434a61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:42.922 INFO 21062 --- [ocp-async-2,0aa67a48e7199781,dbd3cfabb9743376] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:44.874 WARN 21062 --- [ocp-async-2,a29c14a03981d812,791877c839dc36a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:44.923 INFO 21062 --- [ocp-async-4,5aeecfd11a9684ee,b9bf34ceedd490f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b5655ac988b52299,0e3b2b0ea07d341a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:45.625 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:45.627 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:46.874 WARN 21062 --- [ocp-async-4,881148d826f5b03b,9e551bc325e1a50c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:46.923 INFO 21062 --- [ocp-async-7,b37a6e917d52110e,1971a57043328806] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b6817e9618d0041b,00cad300d910ef0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:48.594 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:29:48.593578+08:00, lastProcessTime=2024-11-04T13:29:38.589902+08:00 2024-11-04 13:29:48.600 INFO 21062 --- [ocp-async-4,b500ffc2ec40f681,7f323856987b5b69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:29:48.627 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:48.627 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:29:48.593578+08:00 2024-11-04 13:29:48.874 WARN 21062 --- [ocp-async-5,928a6edb671c9162,663db62695ec08e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:48.923 INFO 21062 --- [ocp-async-3,b96fe7446fa61633,ceb82d0778c18613] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:49.503 INFO 21062 --- [ocp-schedules-25,bc2c3d9ca73d0524,24166bb753c1503d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:29:50.098 INFO 21062 --- [ocp-schedules-24,1467de42fea634a1,ae24901c88ab47cf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:29:50.104 INFO 21062 --- [ocp-schedules-24,1467de42fea634a1,ae24901c88ab47cf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:29:50.104 INFO 21062 --- [ocp-schedules-24,1467de42fea634a1,ae24901c88ab47cf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:29:50.874 WARN 21062 --- [ocp-async-3,09a4dcfecf5d8926,4b51b7697aad45b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:50.923 INFO 21062 --- [ocp-async-6,3361ce390926ba89,95eab21ac65c93de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:50.983 INFO 21062 --- [prometheus-minute-schedule-1,2a60cf18c00aa74f,4699fa5f772cd568] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:51.510 INFO 21062 --- [ocp-schedules-13,f254d515e9988be0,9e91c29d595dd4e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:29:51.627 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:51.628 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:52.874 WARN 21062 --- [ocp-async-6,1896391c2e272472,f92259bfc7d3bb90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:52.923 INFO 21062 --- [ocp-async-9,bc488b9aa256f026,fe1319581f551f21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:53.983 INFO 21062 --- [prometheus-minute-schedule-1,048f6dc5fd0cdcd2,b4c1d57b53da71ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:54.628 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:54.629 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:54.874 WARN 21062 --- [ocp-async-9,a3b4c2e45917bb92,271b48cc268d5649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:54.924 INFO 21062 --- [ocp-async-1,f6722211d6775fff,a7ef90b88252b39d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:56.875 WARN 21062 --- [ocp-async-1,5f4eb9d8551bd5e9,6f538ec1b0a3ba0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:56.924 INFO 21062 --- [ocp-async-8,bc468477b14f54df,a1e06de038f38f7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:56.983 INFO 21062 --- [prometheus-minute-schedule-1,25fce3ebafec7924,9ecbb7f55ee4de49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:29:57.630 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:29:57.631 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:29:58.500 INFO 21062 --- [ocp-schedules-23,bd7c709d8f9b600a,92f353cb92e3a999] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:29:58.500 INFO 21062 --- [ocp-schedules-27,b2daa288fddc6064,caff13141a4bc7ce] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:29:58.502 INFO 21062 --- [ocp-schedules-8,78eddbfd561f0814,266394cc9493c338] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:29:58.504 INFO 21062 --- [ocp-schedules-23,bd7c709d8f9b600a,92f353cb92e3a999] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:29:58.505 INFO 21062 --- [ocp-schedules-1,b89207b1647a76e4,8eb29914000a40c2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:29:58.506 INFO 21062 --- [ocp-schedules-27,b2daa288fddc6064,caff13141a4bc7ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:29:58.509 INFO 21062 --- [ocp-schedules-26,e6a35f56928a6673,1cb3cc23aa9c1494] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:29:58.510 INFO 21062 --- [ocp-schedules-3,fb966e5ccbc17586,d190ecd00b408d0d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:29:58.510 INFO 21062 --- [ocp-schedules-3,fb966e5ccbc17586,d190ecd00b408d0d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:29:58.510 INFO 21062 --- [ocp-schedules-26,e6a35f56928a6673,1cb3cc23aa9c1494] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:29:58.510 INFO 21062 --- [ocp-schedules-12,a4dbf9fc8c4bb5cb,d1a2ee86d8837e68] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:29:58.510 INFO 21062 --- [ocp-schedules-12,a4dbf9fc8c4bb5cb,d1a2ee86d8837e68] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:29:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:29:58.589932+08:00, lastProcessTime=2024-11-04T13:29:48.593578+08:00 2024-11-04 13:29:58.600 INFO 21062 --- [ocp-async-1,63afc863eeb0e9d8,d1cd1d77f2e4fb98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:29:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:29:58.589932+08:00 2024-11-04 13:29:58.873 INFO 21062 --- [ocp-schedules-24,ea89a5d2a3e41651,7dcd5699c70f3075] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:29:58.875 WARN 21062 --- [ocp-async-6,f49986782b7e1ec2,ec3981ed38f16cb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:29:58.924 INFO 21062 --- [ocp-async-7,9eb17a28f9dc5d0a,9c71733a126741a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:29:59.506 INFO 21062 --- [ocp-schedules-12,409f5ed198dd08e0,d22d9ca9a22ddc09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:29:59.983 INFO 21062 --- [prometheus-minute-schedule-1,6a2171628000bce4,f0b2505fda084677] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698200, interval=60 2024-11-04 13:30:00.016 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0f26e990d47de8de] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:25:00.0, nextRun=2024-11-04 13:30:00.0, currentDate=Mon Nov 04 13:30:00 CST 2024 2024-11-04 13:30:00.016 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0f26e990d47de8de] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:25:00.0, nextRunTime=2024-11-04 13:30:00.0} 2024-11-04 13:30:00.042 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0f26e990d47de8de] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:30:00.046 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0f26e990d47de8de] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:30:00.046 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0f26e990d47de8de] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:30:00.631 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:00.632 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:00.875 WARN 21062 --- [ocp-async-9,624712b226e07bde,a89323a083a5f30d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:00.924 INFO 21062 --- [ocp-async-2,bfc1d4bdb0b01275,fcb5ad14e2da5c0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:01.535 INFO 21062 --- [ocp-schedules-28,36760a33993fb83e,8425b81fe1583c1e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:30:02.444 INFO 21062 --- [ocp-schedules-10,7bd6a930aa45cee0,f9fc34f446bdb629] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:30:02.444 INFO 21062 --- [ocp-schedules-10,83fd496976df0771,7ee3cd77aca0e048] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:30:02.448 INFO 21062 --- [ocp-schedules-10,83fd496976df0771,7ee3cd77aca0e048] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:02.478 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:30:02.478 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:30:02.875 WARN 21062 --- [ocp-async-1,2e10de2844760b94,4c0febc252de50e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:02.924 INFO 21062 --- [ocp-async-8,ff155a6c355e48c5,a62dedfbf0cb58e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1e975e09b0b86787,f07e77c89a2fc164] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:03.633 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:03.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:04.116 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a86077554a6a5c00] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:20:04.0, nextRun=2024-11-04 13:30:04.0, currentDate=Mon Nov 04 13:30:04 CST 2024 2024-11-04 13:30:04.116 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a86077554a6a5c00] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 13:20:04.0, nextRunTime=2024-11-04 13:30:04.0} 2024-11-04 13:30:04.123 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a86077554a6a5c00] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 13:30:04.123 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,a86077554a6a5c00] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T13:30:04.123+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 13:30:04.699 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,31777d71d85c4d1e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 74, operation:EXECUTE, state: RUNNING 2024-11-04 13:30:04.719 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=74, context=Context{parallelIdx=-1, stringMap={task_instance_id=9, task_operation=execute, latest_execution_start_time=2024-11-04T13:30:04.692+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=74}, listMap={}}, executor=10.46.128.24 2024-11-04 13:30:04.723 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:30:04.724 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:30:04.725 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.726 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:30:04.727 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.728 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.728 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.729 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:30:04.744 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.744 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.745 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.745 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:30:04.758 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.758 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.759 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.760 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:30:04.773 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.774 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.775 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.775 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:30:04.788 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.789 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.790 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.790 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:30:04.796 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.796 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.796 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.797 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:30:04.801 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.802 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.802 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.803 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:30:04.807 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.807 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.808 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.809 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:30:04.815 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.816 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.817 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.817 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:30:04.822 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.822 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.823 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.823 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:30:04.829 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.830 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.830 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.831 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:30:04.840 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.840 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.841 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.841 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:30:04.855 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.855 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.856 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.856 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:30:04.861 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.862 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.862 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.863 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:30:04.869 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.869 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.870 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.870 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:30:04.875 WARN 21062 --- [ocp-async-4,8d3c0f3ba8cd0bb1,27d6d2fd677f5177] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:04.877 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.877 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.878 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.878 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:30:04.882 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.883 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.883 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.884 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:30:04.888 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.889 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.889 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.890 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:30:04.894 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.894 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.895 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.895 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:30:04.899 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.900 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.900 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.901 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:30:04.905 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.905 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.905 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.906 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:30:04.910 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.911 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.911 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.912 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:30:04.916 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.916 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.917 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.917 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:30:04.921 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.922 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.922 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.923 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:30:04.924 INFO 21062 --- [ocp-async-10,14301609165b37ba,550b008d28cd34ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:04.928 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.929 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.929 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.930 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:30:04.934 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.934 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.935 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.935 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:30:04.940 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.940 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.940 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.941 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:30:04.945 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.945 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.946 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.946 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:30:04.951 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.951 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.951 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.952 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:30:04.956 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.956 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.957 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.957 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:30:04.961 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.962 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.963 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.963 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:30:04.968 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.968 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.969 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.969 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:30:04.974 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.974 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.975 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.975 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:30:04.980 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.980 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.981 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.981 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:30:04.989 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.989 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.990 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.990 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:30:04.995 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.996 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:04.996 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:04.997 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:30:05.003 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.003 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.004 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.004 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:30:05.010 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.010 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.011 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.011 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:30:05.015 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.015 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.016 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.016 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:30:05.020 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.021 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.021 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.022 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:30:05.025 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.026 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.026 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.027 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:30:05.030 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.031 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.031 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.031 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:30:05.041 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.042 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.042 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.042 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:30:05.064 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.064 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.065 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.065 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:30:05.071 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.072 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.072 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.072 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:30:05.078 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.079 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.079 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.079 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:30:05.085 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.085 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.086 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.086 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:30:05.090 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.091 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.091 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.091 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:30:05.095 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.095 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.096 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.097 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:30:05.101 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:30:05.101 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:30:05.102 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.103 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:30:05.104 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.104 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.105 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.105 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:30:05.112 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.113 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:30:05.114 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:05.114 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:30:05.119 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:30:05.127 INFO 21062 --- [subtask-executor7,dd193f8f67a5a423,9107a9a3ea5d3ceb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 74, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 13:30:05.487 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9, status:SUCCESS 2024-11-04 13:30:05.487 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9, status:SUCCESS 2024-11-04 13:30:05.983 INFO 21062 --- [prometheus-minute-schedule-1,b829467153d05df5,be4c4833a62386ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:06.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:06.635 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:06.875 WARN 21062 --- [ocp-async-7,bc879cdf52a9bf3f,95378fb2dc5d198c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:06.925 INFO 21062 --- [ocp-async-5,e83e7cf2e3b4e64e,ceb249ca66974b6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:06.944 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:30:06.944 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:30:06.950 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:30:06.950 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:30:06.950 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:30:06.951 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:30:06.956 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:30:06.956 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:30:06.956 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:30:06.956 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:30:06.994 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:30:07.002 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:30:07.002 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:30:07.002 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:30:07.002 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:30:07.002 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:30:07.010 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:30:07.521 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:30:08.091 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.135149246S 2024-11-04 13:30:08.091 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:30:08.192 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:30:08.192 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:30:08.220 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:30:08.281 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:30:08.281 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:30:08.281 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:30:08.281 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:30:08.281 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:30:08.281 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:30:08.292 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:30:08.471 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] o.s.boot.SpringApplication : Started application in 1.52 seconds (JVM running for 3622.15) 2024-11-04 13:30:08.471 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.521041237S 2024-11-04 13:30:08.505 INFO 21062 --- [ocp-schedules-23,bd7c709d8f9b600a,92f353cb92e3a999] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:30:08.506 INFO 21062 --- [ocp-schedules-27,b2daa288fddc6064,caff13141a4bc7ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:30:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:30:08.589525+08:00, lastProcessTime=2024-11-04T13:29:58.589932+08:00 2024-11-04 13:30:08.600 INFO 21062 --- [ocp-async-7,c7e89d6e6bbb5385,18010a3fe5c36386] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:30:08.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:30:08.589525+08:00 2024-11-04 13:30:08.876 WARN 21062 --- [ocp-async-2,0cf5a23c16564a46,4a3030118d870539] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:08.925 INFO 21062 --- [ocp-async-8,996a8b7c0458e0be,ac40d120ef960b7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:08.983 INFO 21062 --- [prometheus-minute-schedule-1,0af5786ab89823d6,8ec40b176700a086] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:08.986 INFO 21062 --- [ocp-schedules-5,1c7fa871a9ec7dc2,9b4126ceb390c3d0] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:30:09.509 INFO 21062 --- [ocp-schedules-20,f185882ef1fe9a40,28940769ed60f1f3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:30:09.635 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:09.636 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:30:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:30:10.107 INFO 21062 --- [ocp-schedules-30,5d23abde5e949ad1,142d1349a1ae603b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:30:10.112 INFO 21062 --- [ocp-schedules-30,5d23abde5e949ad1,142d1349a1ae603b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:30:10.112 INFO 21062 --- [ocp-schedules-30,5d23abde5e949ad1,142d1349a1ae603b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:30:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:30:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:30:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:30:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:30:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:30:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:30:10.876 WARN 21062 --- [ocp-async-8,cc9b40e654104477,466abd5b3cd219b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:10.925 INFO 21062 --- [ocp-async-10,09a212db720c23ba,009679f44da17d44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:11.255 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=1, affectRows=1 2024-11-04 13:30:11.256 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=1, validCount=1 2024-11-04 13:30:11.256 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=2, affectRows=2 2024-11-04 13:30:11.256 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=1, validCount=1 2024-11-04 13:30:11.256 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=20, affectRows=20 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=20, validCount=20 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=19, affectRows=19 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=37, affectRows=37 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=19, validCount=19 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=47, affectRows=47 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=54, affectRows=54 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=47, validCount=47 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=37, validCount=37 2024-11-04 13:30:11.257 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=54, validCount=54 2024-11-04 13:30:11.258 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=67, affectRows=67 2024-11-04 13:30:11.258 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=26, affectRows=26 2024-11-04 13:30:11.258 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698140, nodeCount=49, affectRows=49 2024-11-04 13:30:11.258 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=65, validCount=65 2024-11-04 13:30:11.258 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=26, validCount=26 2024-11-04 13:30:11.258 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698140, nodeCount=49, validCount=49 2024-11-04 13:30:11.577 INFO 21062 --- [ocp-schedules-29,55d971d565530397,47d27ed31591b049] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:30:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b2d7292b9ca0f3d4,cb222c1319e9dcb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:12.636 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:12.637 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:12.876 WARN 21062 --- [ocp-async-10,136c070816d72d73,d0d263cdcdef807c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:12.925 INFO 21062 --- [ocp-async-5,3314f41958035a53,333904dc7df123c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:13.984 INFO 21062 --- [prometheus-cache-monitor1,274cb1dfe654b7ef,3bf15a209d976e4d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:30:13.984 INFO 21062 --- [prometheus-cache-monitor1,274cb1dfe654b7ef,3bf15a209d976e4d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:30:14.876 WARN 21062 --- [ocp-async-5,7e0efc1357159620,ced02408fcfa9d3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:14.925 INFO 21062 --- [ocp-async-3,c22377aece08caf2,d0696e768ba5789f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9b95daa95b377e6e,28a1caa95236c7ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:15.637 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:15.638 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:16.876 WARN 21062 --- [ocp-async-3,797bd00e9eff2dbd,26d4b90a1288c5b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:16.925 INFO 21062 --- [ocp-async-6,dd41b3d47d788705,971c3c58e4736b1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7352177aecd4e729,e7eddaf086b9c7be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:30:18.589078+08:00, lastProcessTime=2024-11-04T13:30:08.589525+08:00 2024-11-04 13:30:18.600 INFO 21062 --- [ocp-async-8,d46b7cbfebc16c24,2ff7ab0c21d7a46a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:30:18.638 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:18.639 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:30:18.589078+08:00 2024-11-04 13:30:18.876 WARN 21062 --- [ocp-async-10,71291ec2c30ee0d2,a6fdd3bde6e7e095] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:18.925 INFO 21062 --- [ocp-async-5,eadfe1003cef1a54,1da9b5976ea6afe0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:19.512 INFO 21062 --- [ocp-schedules-21,1a59f75b732e84e5,0c0d1ade13aeabae] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:30:20.877 WARN 21062 --- [ocp-async-5,3202c3d82eceb78d,7e8560347d1ded1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:20.926 INFO 21062 --- [ocp-async-3,b723799457351a26,c75201204ae1eef7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:20.983 INFO 21062 --- [prometheus-minute-schedule-1,015dfe0483f00547,28ec583da3020cd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:21.618 INFO 21062 --- [ocp-schedules-1,e0a32785f1006f58,529714cd0ca4eb67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:30:21.639 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:21.641 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:22.877 WARN 21062 --- [ocp-async-3,39d569cdb207818f,1b336841dfb4e5f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:22.926 INFO 21062 --- [ocp-async-6,2df56c4ed2b09a5b,7a5c3d9e7b1c5a35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:23.983 INFO 21062 --- [prometheus-minute-schedule-1,09bd827de5306006,9a10b638e6046dc0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:24.641 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:24.642 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:24.877 WARN 21062 --- [ocp-async-6,7a7ff2b503dad6ac,4a7be039b99bcfbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:24.926 INFO 21062 --- [ocp-async-7,e0eb9afc062266e3,6070b83a7af1ce33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:26.877 WARN 21062 --- [ocp-async-7,4d737b45374af09d,4831fe980bcf2765] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:26.926 INFO 21062 --- [ocp-async-1,5b46f0abed90917b,4a8046cd9e67c41c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e6a96aa65258039a,7b2b01d35fc8edbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:26.984 INFO 21062 --- [prometheus-hour-schedule-1,fa0d29b9c382e29b,833ff251a896239b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:30:27.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:27.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:28.498 INFO 21062 --- [ocp-schedules-11,026718a2c36c36e0,157fa3db40701f34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:30:28.499 INFO 21062 --- [ocp-schedules-7,b26c753e3e6fb71e,da4394dc3bab1751] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:30:28.499 INFO 21062 --- [ocp-schedules-29,33b01a256cceca33,43eb9b4faa822d2e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:30:28.500 INFO 21062 --- [ocp-schedules-1,725a2a11ddc4041a,222b57ba800e122f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:30:28.502 INFO 21062 --- [ocp-schedules-27,8d2c3e59d45f08ec,b56f0e103a3c6ab6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:30:28.504 INFO 21062 --- [ocp-schedules-11,026718a2c36c36e0,157fa3db40701f34] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:30:28.505 INFO 21062 --- [ocp-schedules-7,b26c753e3e6fb71e,da4394dc3bab1751] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:30:28.505 INFO 21062 --- [ocp-schedules-21,e6015c5908665371,07961140f2715fc4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:30:28.505 INFO 21062 --- [ocp-schedules-1,725a2a11ddc4041a,222b57ba800e122f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:28.505 INFO 21062 --- [ocp-schedules-29,33b01a256cceca33,43eb9b4faa822d2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:28.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:30:28.510 INFO 21062 --- [ocp-schedules-25,37de0022d1e8bc25,7f284dd129f6cce2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:30:28.510 INFO 21062 --- [ocp-schedules-25,37de0022d1e8bc25,7f284dd129f6cce2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:30:28.511 INFO 21062 --- [ocp-schedules-19,dc0a45ef372fc7ee,f50c8b56b6f56502] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:30:28.511 INFO 21062 --- [ocp-schedules-19,dc0a45ef372fc7ee,f50c8b56b6f56502] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:30:28.511 INFO 21062 --- [ocp-schedules-30,8acded9e364ca96b,78f79cbc9a60ff39] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:30:28.511 INFO 21062 --- [ocp-schedules-30,8acded9e364ca96b,78f79cbc9a60ff39] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:30:28.566 INFO 21062 --- [ocp-schedules-13,20237c6e33d8e909,2de985f93c392068] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:30:28.566 INFO 21062 --- [ocp-schedules-13,20237c6e33d8e909,2de985f93c392068] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:30:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:30:28.588462+08:00, lastProcessTime=2024-11-04T13:30:18.589078+08:00 2024-11-04 13:30:28.598 INFO 21062 --- [ocp-schedules-24,6a68a16e5fadca44,a2d4974e034dac33] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:30:28.599 INFO 21062 --- [ocp-schedules-24,6a68a16e5fadca44,a2d4974e034dac33] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:30:28.600 INFO 21062 --- [ocp-async-2,c3b7a15f79469e31,ff2cc1be9b7b81ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:30:28.641 INFO 21062 --- [ocp-schedules-14,bea0bc21f69c2f6a,9a3e416d5d7ca79a] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:30:28.641 INFO 21062 --- [ocp-schedules-14,bea0bc21f69c2f6a,9a3e416d5d7ca79a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:30:28.699 INFO 21062 --- [ocp-schedules-3,08ebe3fcda31b58d,7cdfa531cbaaeb8f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:30:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:30:28.588462+08:00 2024-11-04 13:30:28.857 INFO 21062 --- [ocp-schedules-27,18ebf76ad01e58ed,5e85961ab7f25265] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:30:28.857 INFO 21062 --- [ocp-schedules-27,18ebf76ad01e58ed,5e85961ab7f25265] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:30:28.876 INFO 21062 --- [ocp-schedules-18,ee8b36475aca67c4,d4715dcbe591b268] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:30:28.877 WARN 21062 --- [ocp-async-4,5b8c198d9375f6d7,a329d82587e1236f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:28.904 INFO 21062 --- [ocp-schedules-30,8d6453a6d56075a7,c729e5089e491e76] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:30:28.910 INFO 21062 --- [ocp-schedules-30,8d6453a6d56075a7,c729e5089e491e76] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:30:28.910 INFO 21062 --- [ocp-schedules-30,8d6453a6d56075a7,c729e5089e491e76] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:30:28.926 INFO 21062 --- [ocp-async-7,38693b60db8bc7e6,a04cec5480f91064] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:29.075 INFO 21062 --- [ocp-schedules-13,5ac463847e2c44e3,0e028d0d0e382c62] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:30:29.075 INFO 21062 --- [ocp-schedules-13,5ac463847e2c44e3,0e028d0d0e382c62] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:30:29.083 INFO 21062 --- [ocp-schedules-13,5ac463847e2c44e3,0e028d0d0e382c62] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:30:29.083 INFO 21062 --- [ocp-schedules-13,5ac463847e2c44e3,0e028d0d0e382c62] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:30:29.090 INFO 21062 --- [ocp-schedules-13,5ac463847e2c44e3,0e028d0d0e382c62] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:30:29.090 INFO 21062 --- [ocp-schedules-13,5ac463847e2c44e3,0e028d0d0e382c62] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:30:29.515 INFO 21062 --- [ocp-schedules-5,29339641c7ae7957,147e2baa89529191] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:30:29.983 INFO 21062 --- [prometheus-minute-schedule-1,bc0503c63baa4684,17664524b97b3bad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:30.111 INFO 21062 --- [ocp-schedules-18,80ff0bcfce4e5293,f9aace0d0aa85264] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:30:30.111 INFO 21062 --- [ocp-schedules-18,80ff0bcfce4e5293,f9aace0d0aa85264] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:30:30.115 INFO 21062 --- [ocp-schedules-19,a75a930dd371a26d,cb9b27b3caaf6db8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:30:30.121 INFO 21062 --- [ocp-schedules-19,a75a930dd371a26d,cb9b27b3caaf6db8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:30:30.121 INFO 21062 --- [ocp-schedules-19,a75a930dd371a26d,cb9b27b3caaf6db8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:30:30.142 INFO 21062 --- [ocp-schedules-30,4ea3f35ecbccbf48,72b8c527cf6c114c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:30:30.143 INFO 21062 --- [ocp-schedules-18,80ff0bcfce4e5293,f9aace0d0aa85264] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:30:30.143 INFO 21062 --- [ocp-schedules-18,80ff0bcfce4e5293,f9aace0d0aa85264] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:30:30.272 INFO 21062 --- [ocp-schedules-22,d24aeb739b9d1d2f,b37dd817663fa344] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:30:30.272+08:00, clusters=[1] 2024-11-04 13:30:30.273 INFO 21062 --- [ocp-schedules-22,d24aeb739b9d1d2f,b37dd817663fa344] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:30:30.281 INFO 21062 --- [ocp-schedules-24,87cec34dfa667698,2134ee63eb8e7540] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:30:30.282 INFO 21062 --- [ocp-schedules-13,1b4db8284dd149bc,909526521237f348] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:30:30.282 INFO 21062 --- [ocp-schedules-13,1b4db8284dd149bc,909526521237f348] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:30:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:30:30.282 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:30:30.282 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:30:30.284 INFO 21062 --- [ocp-schedules-24,87cec34dfa667698,2134ee63eb8e7540] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:30.293 INFO 21062 --- [ocp-schedules-13,1b4db8284dd149bc,909526521237f348] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:30:30.293 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:30:30.293 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:30:30.295 WARN 21062 --- [ocp-schedules-13,1b4db8284dd149bc,909526521237f348] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:30:30.295 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:30:30.295 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:30:30.296 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:30:30.296 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:30:30.296 INFO 21062 --- [ocp-schedules-13,1b4db8284dd149bc,909526521237f348] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:30:30.297 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:30:30.297 WARN 21062 --- [ocp-schedules-13,1b4db8284dd149bc,909526521237f348] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:30:30.297 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:30:30.315 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:30:30.316 ERROR 21062 --- [ocp-schedules-13,1b4db8284dd149bc,909526521237f348] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:30:30.352 ERROR 21062 --- [ocp-schedules-24,87cec34dfa667698,2134ee63eb8e7540] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:30:30.352 INFO 21062 --- [ocp-schedules-24,87cec34dfa667698,2134ee63eb8e7540] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:30:30.402 INFO 21062 --- [ocp-schedules-9,d77fc0532c3fd427,1bdb960fff0518e7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:30:30.402+08:00, clusters=[1] 2024-11-04 13:30:30.406 INFO 21062 --- [ocp-schedules-9,d77fc0532c3fd427,1bdb960fff0518e7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:30:30.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:30.644 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:30.878 WARN 21062 --- [ocp-async-7,5a15edb39a4286bb,dcea99de155bb1bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:30.927 INFO 21062 --- [ocp-async-2,31567bec3b37e207,aa540972b6176269] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:31.665 INFO 21062 --- [ocp-schedules-8,c8f21719b0f0e4d8,4bdd90d6106fb299] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:30:32.479 INFO 21062 --- [ocp-schedules-10,83fd496976df0771,7ee3cd77aca0e048] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:30:32.479 INFO 21062 --- [ocp-schedules-10,3cf56a6b95bce0fb,7f047af99ebd3440] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:30:32.483 INFO 21062 --- [ocp-schedules-10,3cf56a6b95bce0fb,7f047af99ebd3440] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:32.510 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:30:32.510 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:30:32.878 WARN 21062 --- [ocp-async-2,860035bae99df66f,768504aae6294dc1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:32.927 INFO 21062 --- [ocp-async-8,4a76f6d8dd967883,9005c4eaa6c99a50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:32.983 INFO 21062 --- [prometheus-minute-schedule-1,c0605fcd140b874f,ade254d3d3f61739] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:33.644 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:33.645 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:33.851 INFO 21062 --- [ocp-schedules-11,93fb3bb660794570,bf5dd49c33eb1799] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:30:33.854 INFO 21062 --- [ocp-schedules-11,93fb3bb660794570,bf5dd49c33eb1799] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:33.939 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:30:33.943 ERROR 21062 --- [ocp-schedules-11,93fb3bb660794570,bf5dd49c33eb1799] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:30:33.943 INFO 21062 --- [ocp-schedules-11,93fb3bb660794570,bf5dd49c33eb1799] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:30:34.878 WARN 21062 --- [ocp-async-8,2bbf02bd08d172fb,9756d60ca2bcd36f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:34.927 INFO 21062 --- [ocp-async-10,9aab97c60d6914d0,0323cb925837d18f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:35.983 INFO 21062 --- [prometheus-minute-schedule-1,154f238838e2532e,5042b361595851fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:36.645 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:36.646 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:36.878 WARN 21062 --- [ocp-async-10,38e5f89a081011a0,71c96b5efbc950d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:36.927 INFO 21062 --- [ocp-async-9,35c8cc364d3d8a74,ee83c28cc1b2acc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:38.506 INFO 21062 --- [ocp-schedules-1,725a2a11ddc4041a,222b57ba800e122f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:30:38.506 INFO 21062 --- [ocp-schedules-29,33b01a256cceca33,43eb9b4faa822d2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:30:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:30:38.588881+08:00, lastProcessTime=2024-11-04T13:30:28.588462+08:00 2024-11-04 13:30:38.601 INFO 21062 --- [ocp-async-7,58b537283c9d7f8c,91ab036b79fbe1da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:30:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:30:38.588881+08:00 2024-11-04 13:30:38.799 INFO 21062 --- [ocp-schedules-9,db5a278facc945ae,49d0ea759ebe627f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:30:38.802 INFO 21062 --- [ocp-schedules-9,db5a278facc945ae,49d0ea759ebe627f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:38.803 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:30:38.878 WARN 21062 --- [ocp-async-1,6cdcc77c1e2adec8,c1c22c83fc74cb18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:38.927 INFO 21062 --- [ocp-async-8,3b8dcd0ecfa2faa6,85f45bbd8a1ae974] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:38.983 INFO 21062 --- [prometheus-minute-schedule-1,d96ccda50dc888dd,84e5b282e7e459ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:39.517 INFO 21062 --- [ocp-schedules-28,7d434879105141cd,af59bc517993df82] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:30:39.646 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:39.647 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:40.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:30:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:30:40.878 WARN 21062 --- [ocp-async-8,bbdedcb84071c890,6c1413df239234d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:40.927 INFO 21062 --- [ocp-async-6,5ecb930bc05d197b,b373ec1119068205] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:41.695 INFO 21062 --- [ocp-schedules-4,f07f01327deaf66e,6ba4e63ca421b83d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:30:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:30:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:30:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:30:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:30:41.853 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:30:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:30:41.983 INFO 21062 --- [prometheus-minute-schedule-1,71db22f995f93f57,2f683449492f7f1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:42.647 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:42.648 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:42.879 WARN 21062 --- [ocp-async-6,a84f68b077cf855b,8af967a3754f18e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:42.928 INFO 21062 --- [ocp-async-9,595320d55ad1c9a2,edcb25c5203d7f30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:44.879 WARN 21062 --- [ocp-async-2,688019db3bb617ec,279fa715f248b313] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:44.928 INFO 21062 --- [ocp-async-3,9c5398b0ac7cbb86,8c4ed165ba6a7da2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:44.983 INFO 21062 --- [prometheus-minute-schedule-1,1bd97c176f93631e,e5ebf26e61f40dbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:45.649 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:45.650 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:46.879 WARN 21062 --- [ocp-async-4,804c8e40d5140177,17f2bae4cd0080bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:46.928 INFO 21062 --- [ocp-async-10,f9926b2e34dddc41,1b5b3b763811fa17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:47.983 INFO 21062 --- [prometheus-minute-schedule-1,6c12852a272a85cb,8ccd74be556c8273] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:30:48.592380+08:00, lastProcessTime=2024-11-04T13:30:38.588881+08:00 2024-11-04 13:30:48.601 INFO 21062 --- [ocp-async-4,d67addf25ecca5c7,bce69931b2496392] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:30:48.650 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:48.651 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:30:48.592380+08:00 2024-11-04 13:30:48.803 INFO 21062 --- [ocp-schedules-9,db5a278facc945ae,49d0ea759ebe627f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:30:48.879 WARN 21062 --- [ocp-async-6,093808defb82e898,5624a531af59cf6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:48.928 INFO 21062 --- [ocp-async-9,2497162dd5bbb590,8b4480d41b2302f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:49.520 INFO 21062 --- [ocp-schedules-30,7cde6ed40b00e7ad,00b88670dbb71dde] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:30:50.123 INFO 21062 --- [ocp-schedules-17,2dc7a954c0ba8952,73fc66795320023b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:30:50.128 INFO 21062 --- [ocp-schedules-17,2dc7a954c0ba8952,73fc66795320023b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:30:50.128 INFO 21062 --- [ocp-schedules-17,2dc7a954c0ba8952,73fc66795320023b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:30:50.879 WARN 21062 --- [ocp-async-2,c64c0e572d8bd449,eb716d667dfc7898] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:50.928 INFO 21062 --- [ocp-async-3,290bb1f804a5b017,0c9967e670ee4897] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:50.983 INFO 21062 --- [prometheus-minute-schedule-1,5ab2b5872e1640a4,73d25d4061d42454] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:51.651 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:51.652 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:51.738 INFO 21062 --- [ocp-schedules-13,f1839f8dd39004ba,bb6d47eaf5c12b7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:30:52.880 WARN 21062 --- [ocp-async-4,51c35c971a33ac89,fe4bed873b70ff93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:52.928 INFO 21062 --- [ocp-async-10,cffa0c9266a16778,8d5d9dda60b4b19a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:53.983 INFO 21062 --- [prometheus-minute-schedule-1,38c91531119f315a,2f5936b51d31e973] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:54.652 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:54.654 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:54.880 WARN 21062 --- [ocp-async-7,9350f2d7eac1a4cc,61711bc0ffdc46d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:54.929 INFO 21062 --- [ocp-async-5,d36f1f302a8b00aa,50ab59b06e5d1dde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:56.880 WARN 21062 --- [ocp-async-9,3a3680f7c82c1c9a,ff4b07a24a296529] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:56.929 INFO 21062 --- [ocp-async-1,d5cc791a19f3e683,d54f0d437cda4007] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:56.983 INFO 21062 --- [prometheus-minute-schedule-1,a84bbfc2bd8470c3,6f153f1d0fc60283] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:30:57.654 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:30:57.655 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:30:58.500 INFO 21062 --- [ocp-schedules-19,d4ced2478da9d184,644b3a55d73e9098] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:30:58.500 INFO 21062 --- [ocp-schedules-15,07582b129910b7c1,e5bed871ffadd156] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:30:58.502 INFO 21062 --- [ocp-schedules-27,e993ce572dd76903,e505b73403acf3e9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:30:58.504 INFO 21062 --- [ocp-schedules-19,d4ced2478da9d184,644b3a55d73e9098] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:58.504 INFO 21062 --- [ocp-schedules-15,07582b129910b7c1,e5bed871ffadd156] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:30:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:30:58.505 INFO 21062 --- [ocp-schedules-9,8e97dbf4c5eac729,1aa480368f6283ac] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:30:58.510 INFO 21062 --- [ocp-schedules-5,129ebca7bc4c83b3,f6382ee9879b7f7b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:30:58.510 INFO 21062 --- [ocp-schedules-5,129ebca7bc4c83b3,f6382ee9879b7f7b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:30:58.511 INFO 21062 --- [ocp-schedules-3,7d29a93c07410935,78e5762a88eae9ce] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:30:58.511 INFO 21062 --- [ocp-schedules-3,7d29a93c07410935,78e5762a88eae9ce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:30:58.511 INFO 21062 --- [ocp-schedules-24,715a76acebdff8c2,766db0e2e3faa376] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:30:58.511 INFO 21062 --- [ocp-schedules-24,715a76acebdff8c2,766db0e2e3faa376] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:30:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:30:58.589523+08:00, lastProcessTime=2024-11-04T13:30:48.592380+08:00 2024-11-04 13:30:58.601 INFO 21062 --- [ocp-async-9,a6662fe06596596e,02db5e83d8b3e2b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:30:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:30:58.589523+08:00 2024-11-04 13:30:58.878 INFO 21062 --- [ocp-schedules-28,c18ace59adf3d8da,a878a513af2f28bf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:30:58.880 WARN 21062 --- [ocp-async-4,fb6b49910bf938fa,420f424d40af1103] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:30:58.929 INFO 21062 --- [ocp-async-10,89026e3bcb06ba33,1dff12f47d4f30e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:30:59.523 INFO 21062 --- [ocp-schedules-17,f258bbb045c779ce,ae0de04295ee4e36] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:30:59.983 INFO 21062 --- [prometheus-minute-schedule-1,cd6a7ae93950ec71,a3987e95809a6930] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698260, interval=60 2024-11-04 13:31:00.656 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:00.656 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:00.880 WARN 21062 --- [ocp-async-7,5bfab20d72059091,33357660081065b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:00.929 INFO 21062 --- [ocp-async-5,308b8df0d6938578,be17a7808ee81679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:01.767 INFO 21062 --- [ocp-schedules-22,5df584b370f091c3,2ac233931b312b6f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:31:02.511 INFO 21062 --- [ocp-schedules-10,3cf56a6b95bce0fb,7f047af99ebd3440] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:31:02.511 INFO 21062 --- [ocp-schedules-10,d8a7441fc88739d7,2882f68e6982d372] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:31:02.516 INFO 21062 --- [ocp-schedules-10,d8a7441fc88739d7,2882f68e6982d372] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:02.547 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:31:02.547 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:31:02.880 WARN 21062 --- [ocp-async-9,ddc736773bb87ad8,b717f26ef6878ed1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:02.929 INFO 21062 --- [ocp-async-1,670544d6b961e68a,8ad03be8a56887a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:02.983 INFO 21062 --- [prometheus-minute-schedule-1,283f09311703c66a,e658a25677146cee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:03.657 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:03.658 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:04.881 WARN 21062 --- [ocp-async-1,2e4a820852fb7888,7fb7f6e37d493802] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:04.929 INFO 21062 --- [ocp-async-8,1ca90c5454e5c284,4a220631eaf91b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:05.983 INFO 21062 --- [prometheus-minute-schedule-1,4a39003fda0c9dc2,86a8adf1537c25d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:06.658 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:06.660 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:06.881 WARN 21062 --- [ocp-async-8,52b4a75159b9c313,0f21f7651ca1febc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:06.930 INFO 21062 --- [ocp-async-6,0ff5b62c3b2cd60b,b4c935df0511dfcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:08.505 INFO 21062 --- [ocp-schedules-19,d4ced2478da9d184,644b3a55d73e9098] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:31:08.505 INFO 21062 --- [ocp-schedules-15,07582b129910b7c1,e5bed871ffadd156] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:31:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:31:08.590351+08:00, lastProcessTime=2024-11-04T13:30:58.589523+08:00 2024-11-04 13:31:08.601 INFO 21062 --- [ocp-async-10,9b722a235a5447c5,143ae9c54894a213] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:31:08.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:31:08.590351+08:00 2024-11-04 13:31:08.881 WARN 21062 --- [ocp-async-5,dd79de9fd88cd74e,c38e9ff4f877e83c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:08.930 INFO 21062 --- [ocp-async-1,3180f6da6400b9ac,9249acf7a578e9d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ade26e8d79eac699,45aa7f0b44c7494e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:09.525 INFO 21062 --- [ocp-schedules-14,1ae5dab3df419b4d,80b317c381500dd7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:31:09.660 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:09.661 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:10.003 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:31:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:31:10.131 INFO 21062 --- [ocp-schedules-29,be63880d6ecc4627,599dcc523de90fc3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:31:10.136 INFO 21062 --- [ocp-schedules-29,be63880d6ecc4627,599dcc523de90fc3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:31:10.136 INFO 21062 --- [ocp-schedules-29,be63880d6ecc4627,599dcc523de90fc3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:31:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:31:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:31:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:31:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:31:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:31:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:31:10.881 WARN 21062 --- [ocp-async-1,46f141345a205737,fc3805a81688b0d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:10.930 INFO 21062 --- [ocp-async-8,839943f5e7bcf9c6,cb95c8790ab26034] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:11.268 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=5, affectRows=5 2024-11-04 13:31:11.268 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=64, affectRows=64 2024-11-04 13:31:11.268 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=56, affectRows=56 2024-11-04 13:31:11.268 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=5, validCount=5 2024-11-04 13:31:11.269 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=56, validCount=56 2024-11-04 13:31:11.269 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=63, validCount=63 2024-11-04 13:31:11.271 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=34, affectRows=34 2024-11-04 13:31:11.271 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=20, affectRows=20 2024-11-04 13:31:11.271 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=34, validCount=34 2024-11-04 13:31:11.271 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=20, validCount=20 2024-11-04 13:31:11.271 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=73, affectRows=73 2024-11-04 13:31:11.272 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=72, validCount=72 2024-11-04 13:31:11.272 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=1, affectRows=1 2024-11-04 13:31:11.272 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=1, validCount=1 2024-11-04 13:31:11.273 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=14, affectRows=14 2024-11-04 13:31:11.273 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=13, validCount=13 2024-11-04 13:31:11.273 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=15, affectRows=15 2024-11-04 13:31:11.273 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=15, validCount=15 2024-11-04 13:31:11.274 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698200, nodeCount=40, affectRows=40 2024-11-04 13:31:11.274 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698200, nodeCount=40, validCount=40 2024-11-04 13:31:11.808 INFO 21062 --- [ocp-schedules-12,a2f7b795fee0646d,7f081dd307a17565] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:31:11.983 INFO 21062 --- [prometheus-minute-schedule-1,26655c92a15c2e29,181414c1f8e738bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:12.661 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:12.662 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:12.881 WARN 21062 --- [ocp-async-8,d8fdc1e2732b8d46,0e641624cb0ee3b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:12.930 INFO 21062 --- [ocp-async-6,4b5a8bf5e51b5df0,c1ea2b2eb9fbf7e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:13.984 INFO 21062 --- [prometheus-cache-monitor1,fb6a77abdc5c6709,9c1a6eba751748f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:31:13.984 INFO 21062 --- [prometheus-cache-monitor1,fb6a77abdc5c6709,9c1a6eba751748f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:31:14.572 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:27:14.553, end=2024-11-04T13:29:14.569} 2024-11-04 13:31:14.583 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:31:14.583 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:31:14.881 WARN 21062 --- [ocp-async-6,bf177a9b27980ce2,afcc30eeb106cc41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:14.930 INFO 21062 --- [ocp-async-2,90437597e07c84b9,37b7a3087889edbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d689493ef8a16484,d7f0801249ffcde5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:15.662 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:15.663 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:16.882 WARN 21062 --- [ocp-async-2,590e4bcdab7ab034,01a9bfe4711b07eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:16.930 INFO 21062 --- [ocp-async-3,f29afd3af78f4149,3d5e63489178f140] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:17.983 INFO 21062 --- [prometheus-minute-schedule-1,a0f0c0cb26faf8a9,5eb33cf2a3e55fdf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:31:18.590918+08:00, lastProcessTime=2024-11-04T13:31:08.590351+08:00 2024-11-04 13:31:18.601 INFO 21062 --- [ocp-async-2,58a2bca98f8d5240,5bf0645728621ff9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:31:18.663 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:18.664 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:31:18.590918+08:00 2024-11-04 13:31:18.882 WARN 21062 --- [ocp-async-8,28b48a6416841c43,9fc6e39c5246253c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:18.931 INFO 21062 --- [ocp-async-6,4482abc2b31f3c35,d3e975f9eb58610c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:19.529 INFO 21062 --- [ocp-schedules-29,a36039293fa10643,1d73fdce61e2aba6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:31:20.882 WARN 21062 --- [ocp-async-6,63cc59d01d76551d,b52ff562002d533a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:20.931 INFO 21062 --- [ocp-async-1,f7265edf8582af02,074100d76dc3aef5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:20.983 INFO 21062 --- [prometheus-minute-schedule-1,422fd0e2ef584951,461c9f8b651c9f23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:21.664 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:21.665 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:21.833 INFO 21062 --- [ocp-schedules-12,43e56a761aa8f0d5,8959dafc25acfe34] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:31:22.882 WARN 21062 --- [ocp-async-1,d0af151436c9d51c,49b470d323e2ea0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:22.931 INFO 21062 --- [ocp-async-3,4eaa749c7bb13bc0,aef73c14171e791c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8a27cb4fe802fc46,a269a51745252e06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:24.665 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:24.666 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:24.882 WARN 21062 --- [ocp-async-3,70d5d2d98a94e60d,e787b0d26ff3b867] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:24.931 INFO 21062 --- [ocp-async-10,b865e5eb7bf614e1,cee2c92b43ca3104] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:26.882 WARN 21062 --- [ocp-async-10,0747b86dbad325a5,4adce65a8aa9a8d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:26.931 INFO 21062 --- [ocp-async-9,e11840458efe3956,52bb34ed6f37ed88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:26.983 INFO 21062 --- [prometheus-minute-schedule-1,18d088b1e2525b09,6c32f7a9bdde009c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9e45750730d5190e,3cc73cce772ea29f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:31:27.666 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:27.667 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:28.498 INFO 21062 --- [ocp-schedules-26,e3b547ca20dd751c,c37a55572be7dd9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:31:28.499 INFO 21062 --- [ocp-schedules-25,8110e1cc60734d99,67610e5727f48a96] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:31:28.499 INFO 21062 --- [ocp-schedules-15,99c171e2a05d182c,4be05cfe2925ec83] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:31:28.500 INFO 21062 --- [ocp-schedules-14,e82c14de366ff5a6,b5bacfedfa48672c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:31:28.502 INFO 21062 --- [ocp-schedules-26,e3b547ca20dd751c,c37a55572be7dd9a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:31:28.502 INFO 21062 --- [ocp-schedules-2,11533766a1bd01e7,2cac6f7fd9c09ab8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:31:28.503 INFO 21062 --- [ocp-schedules-15,99c171e2a05d182c,4be05cfe2925ec83] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:31:28.504 INFO 21062 --- [ocp-schedules-14,e82c14de366ff5a6,b5bacfedfa48672c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:28.505 INFO 21062 --- [ocp-schedules-3,b6432bc8581f3a47,13f3ddadbe29bde1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:31:28.505 INFO 21062 --- [ocp-schedules-25,8110e1cc60734d99,67610e5727f48a96] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:31:28.510 INFO 21062 --- [ocp-schedules-21,9a83a3e4c40a5725,78523d93478429fc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:31:28.510 INFO 21062 --- [ocp-schedules-11,4397684d3c7e0914,9a6466765cccd2bb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:31:28.510 INFO 21062 --- [ocp-schedules-21,9a83a3e4c40a5725,78523d93478429fc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:31:28.510 INFO 21062 --- [ocp-schedules-16,f094941b0cac1b60,c988424e4317711e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:31:28.510 INFO 21062 --- [ocp-schedules-16,f094941b0cac1b60,c988424e4317711e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:31:28.510 INFO 21062 --- [ocp-schedules-11,4397684d3c7e0914,9a6466765cccd2bb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:31:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:31:28.589630+08:00, lastProcessTime=2024-11-04T13:31:18.590918+08:00 2024-11-04 13:31:28.602 INFO 21062 --- [ocp-async-5,e1790623ee131726,c6196a7e64f77211] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:31:28.701 INFO 21062 --- [ocp-schedules-29,81a521b48c4569e1,e92fddc3a15a4270] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:31:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:31:28.589630+08:00 2024-11-04 13:31:28.861 INFO 21062 --- [ocp-schedules-22,8327c02d037e550b,6138dc24730e660b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:31:28.861 INFO 21062 --- [ocp-schedules-22,8327c02d037e550b,6138dc24730e660b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:31:28.881 INFO 21062 --- [ocp-schedules-9,3ab08504ff9fbe51,3c2eb133f2ea51b1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:31:28.883 WARN 21062 --- [ocp-async-4,b5b9b3c95d78187f,e7420411156aac85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:28.910 INFO 21062 --- [ocp-schedules-3,263f4a4475c11d73,1e9b115a7618b410] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:31:28.915 INFO 21062 --- [ocp-schedules-3,263f4a4475c11d73,1e9b115a7618b410] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:31:28.915 INFO 21062 --- [ocp-schedules-3,263f4a4475c11d73,1e9b115a7618b410] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:31:28.932 INFO 21062 --- [ocp-async-10,0e4124e470cb0908,e723ef33d9cdf79e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:28.949 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:31:28.949 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:31:28.956 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:31:28.956 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:31:28.956 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:31:28.966 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:31:28.966 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:31:28.966 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:31:28.970 INFO 21062 --- [ocp-schedules-26,60ab441c824754d8,9478699adefeab6b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:31:29.094 INFO 21062 --- [ocp-schedules-28,bc5f3248c061c344,b8152fdb305f1943] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:31:29.094 INFO 21062 --- [ocp-schedules-28,bc5f3248c061c344,b8152fdb305f1943] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:31:29.097 INFO 21062 --- [ocp-schedules-28,bc5f3248c061c344,b8152fdb305f1943] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:31:29.097 INFO 21062 --- [ocp-schedules-28,bc5f3248c061c344,b8152fdb305f1943] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:31:29.098 INFO 21062 --- [ocp-schedules-28,bc5f3248c061c344,b8152fdb305f1943] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:31:29.098 INFO 21062 --- [ocp-schedules-28,bc5f3248c061c344,b8152fdb305f1943] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:31:29.533 INFO 21062 --- [ocp-schedules-18,9f964c4bbde14422,ae418bfe43f1a539] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:31:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6d3c530d5c262421,ed7f4be5958683be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:30.139 INFO 21062 --- [ocp-schedules-5,7fbe0268b871889e,dd9338e81f3637df] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:31:30.143 INFO 21062 --- [ocp-schedules-11,090103a3606673f1,7891949e7acf7715] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:31:30.143 INFO 21062 --- [ocp-schedules-11,090103a3606673f1,7891949e7acf7715] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:31:30.144 INFO 21062 --- [ocp-schedules-5,7fbe0268b871889e,dd9338e81f3637df] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:31:30.144 INFO 21062 --- [ocp-schedules-5,7fbe0268b871889e,dd9338e81f3637df] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:31:30.169 INFO 21062 --- [ocp-schedules-11,090103a3606673f1,7891949e7acf7715] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:31:30.169 INFO 21062 --- [ocp-schedules-11,090103a3606673f1,7891949e7acf7715] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:31:30.169 INFO 21062 --- [ocp-schedules-13,eb36cc62edfb9e70,9f35b104fe6ca57b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:31:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:31:30.299 INFO 21062 --- [ocp-schedules-20,1955c1c95faea72b,41a8af20f65b6444] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:31:30.299+08:00, clusters=[1] 2024-11-04 13:31:30.300 INFO 21062 --- [ocp-schedules-20,1955c1c95faea72b,41a8af20f65b6444] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:31:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:31:30.324 INFO 21062 --- [ocp-schedules-28,59c283e9ea7adf60,c8c9c4e6af168f67] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:31:30.324 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:31:30.324 INFO 21062 --- [ocp-schedules-28,59c283e9ea7adf60,c8c9c4e6af168f67] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:31:30.324 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:31:30.334 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:31:30.335 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:31:30.335 INFO 21062 --- [ocp-schedules-28,59c283e9ea7adf60,c8c9c4e6af168f67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:31:30.336 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:31:30.336 WARN 21062 --- [ocp-schedules-28,59c283e9ea7adf60,c8c9c4e6af168f67] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:31:30.336 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:31:30.337 INFO 21062 --- [ocp-schedules-28,59c283e9ea7adf60,c8c9c4e6af168f67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:31:30.337 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:31:30.337 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:31:30.338 WARN 21062 --- [ocp-schedules-28,59c283e9ea7adf60,c8c9c4e6af168f67] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:31:30.338 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:31:30.338 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:31:30.345 ERROR 21062 --- [ocp-schedules-28,59c283e9ea7adf60,c8c9c4e6af168f67] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:31:30.431 INFO 21062 --- [ocp-schedules-30,6a2d121fac347345,a25bc57b37bafe38] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:31:30.431+08:00, clusters=[1] 2024-11-04 13:31:30.435 INFO 21062 --- [ocp-schedules-30,6a2d121fac347345,a25bc57b37bafe38] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:31:30.667 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:30.668 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:30.883 WARN 21062 --- [ocp-async-10,767a8fe9384f6a20,74df1585f7192fd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:30.932 INFO 21062 --- [ocp-async-5,8691eadd60057fd9,d0bba197315ed538] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:31.908 INFO 21062 --- [ocp-schedules-5,4e33734238c5be57,c64dd5c60f9ca50f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:31:32.548 INFO 21062 --- [ocp-schedules-10,d8a7441fc88739d7,2882f68e6982d372] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:31:32.548 INFO 21062 --- [ocp-schedules-10,4edf387e968790a1,38c3b6ac68a71abd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:31:32.552 INFO 21062 --- [ocp-schedules-10,4edf387e968790a1,38c3b6ac68a71abd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:32.582 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:31:32.582 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:31:32.883 WARN 21062 --- [ocp-async-5,aae8e609caeaea8d,f4a62c3950bf32a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:32.932 INFO 21062 --- [ocp-async-1,36f8544d1dcfef36,ba4fa3094b45e8d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:32.983 INFO 21062 --- [prometheus-minute-schedule-1,6f635f444fdb55c3,2bc601096d05465e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:33.668 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:33.669 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:33.943 INFO 21062 --- [ocp-schedules-4,5131e4d5cb6d96b9,aed9253be736a5dd] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:31:33.947 INFO 21062 --- [ocp-schedules-4,5131e4d5cb6d96b9,aed9253be736a5dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:34.033 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:31:34.036 ERROR 21062 --- [ocp-schedules-4,5131e4d5cb6d96b9,aed9253be736a5dd] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:31:34.036 INFO 21062 --- [ocp-schedules-4,5131e4d5cb6d96b9,aed9253be736a5dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:31:34.883 WARN 21062 --- [ocp-async-1,d28fffce70c7185a,e06da4f93de9013b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:34.932 INFO 21062 --- [ocp-async-3,499526af9876876e,270a28c3673f6089] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:35.983 INFO 21062 --- [prometheus-minute-schedule-1,13269b298851c163,8d5371b9c4a80d27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:36.669 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:36.670 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:36.883 WARN 21062 --- [ocp-async-3,ee71d09adca048e8,264aa970ed081df5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:36.932 INFO 21062 --- [ocp-async-7,87d19ce998e27adb,bab6c185c8da8d20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:38.504 INFO 21062 --- [ocp-schedules-15,99c171e2a05d182c,4be05cfe2925ec83] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:31:38.504 INFO 21062 --- [ocp-schedules-14,e82c14de366ff5a6,b5bacfedfa48672c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:31:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:31:38.589467+08:00, lastProcessTime=2024-11-04T13:31:28.589630+08:00 2024-11-04 13:31:38.602 INFO 21062 --- [ocp-async-10,cef0ab6d495f3ccc,12046f78c6fdbca8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:31:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:31:38.589467+08:00 2024-11-04 13:31:38.884 WARN 21062 --- [ocp-async-9,e6c749502fbdcd91,5c5237deb672a76e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:38.933 INFO 21062 --- [ocp-async-1,a8e7fdffd508961f,a0e365dbe8551d98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:38.983 INFO 21062 --- [prometheus-minute-schedule-1,6d6d072c7b08fbfb,f024295b281546f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:39.536 INFO 21062 --- [ocp-schedules-6,62b99e23f36b4b69,5fbdf626be9b4b59] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:31:39.670 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:39.671 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:40.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:31:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:31:40.884 WARN 21062 --- [ocp-async-4,ce63045827727bb2,5f977c53ed06e007] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:40.933 INFO 21062 --- [ocp-async-3,cef0df302fe62e8e,88e336593ba01f81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:41.933 INFO 21062 --- [ocp-schedules-20,df34398d63d183cd,6e128ce50928b0ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:31:41.983 INFO 21062 --- [prometheus-minute-schedule-1,fddc2dcc90674b7b,58704de30bf873ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:42.671 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:42.672 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:42.884 WARN 21062 --- [ocp-async-10,dfb0e8d4dcc5b1d4,9c875f733bf61de4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:42.933 INFO 21062 --- [ocp-async-7,8f83cb73712a7ef7,1e125c5d1f74453a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:44.884 WARN 21062 --- [ocp-async-5,7153e39012beff72,e96c6e44398a1a0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:44.933 INFO 21062 --- [ocp-async-2,e438c684f2d775d3,94b23d66ceb57015] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5d053faa2f5c9af7,387cb4930eb86df3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:45.673 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:45.673 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:46.884 WARN 21062 --- [ocp-async-1,b2fddd9f323a443d,d1d772c7ac880227] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:46.933 INFO 21062 --- [ocp-async-8,e8fcbea549e59729,b39980910f9466f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8dd315be8ba5d1bf,f486b15e86cb23e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:31:48.590788+08:00, lastProcessTime=2024-11-04T13:31:38.589467+08:00 2024-11-04 13:31:48.602 INFO 21062 --- [ocp-async-1,257c2cce2d0f682b,3fa014fd01c2f80d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:31:48.674 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:48.674 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:48.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:31:48.590788+08:00 2024-11-04 13:31:48.803 INFO 21062 --- [ocp-schedules-7,baf9342a441e4fa6,d479536c42e89276] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:31:48.807 INFO 21062 --- [ocp-schedules-7,baf9342a441e4fa6,d479536c42e89276] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:48.808 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:31:48.885 WARN 21062 --- [ocp-async-10,b23438e401b89da1,64aac5ad2d31a120] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:48.933 INFO 21062 --- [ocp-async-7,53b8c0dab4d8b107,a92983a3c283f17a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:49.540 INFO 21062 --- [ocp-schedules-8,2325f4dbd7ac018f,310b4d6ccea000ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:31:50.147 INFO 21062 --- [ocp-schedules-24,dc35615d5a69f410,5acad564722c4741] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:31:50.154 INFO 21062 --- [ocp-schedules-24,dc35615d5a69f410,5acad564722c4741] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:31:50.154 INFO 21062 --- [ocp-schedules-24,dc35615d5a69f410,5acad564722c4741] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:31:50.885 WARN 21062 --- [ocp-async-5,4f62e88e51e0c16a,a486788f810a50f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:50.933 INFO 21062 --- [ocp-async-2,942449508eb4a257,4158a7e3e19e54c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a1001e476cc3bf85,dac7b9a9ffa3dcb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:51.675 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:51.675 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:51.978 INFO 21062 --- [ocp-schedules-26,d52c5e5cd82462dd,3a8c83b8babe3bc6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:31:52.885 WARN 21062 --- [ocp-async-1,467d8f0ae53fff2e,78c2403031dc9b42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:52.934 INFO 21062 --- [ocp-async-8,f5367183f2e11f51,9662c61dfa34e994] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1ff0b112fe2c9b5f,5fd34b1b344c1484] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:54.676 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:54.676 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:54.885 WARN 21062 --- [ocp-async-3,93427fe759537fb0,e9b47c1838b3998b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:54.934 INFO 21062 --- [ocp-async-6,688cb5e07713f0ce,e815a939cfd4b33b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:56.885 WARN 21062 --- [ocp-async-7,62fb19f8c8acf722,5d7494d883fa2586] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:56.934 INFO 21062 --- [ocp-async-9,f2a3136ed5537de2,f7e5229caeae37b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:56.983 INFO 21062 --- [prometheus-minute-schedule-1,4bd2b67676df4e26,e7c760e017c11cd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:31:57.676 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:31:57.677 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:31:58.500 INFO 21062 --- [ocp-schedules-13,0fc8f7cf73a699d6,02d8ae4e2b291a26] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:31:58.500 INFO 21062 --- [ocp-schedules-12,a7f551d60aec692f,2344bb505333df28] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:31:58.502 INFO 21062 --- [ocp-schedules-16,c56ce1ba4aa8ca1f,91989c49c32bd4ee] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:31:58.504 INFO 21062 --- [ocp-schedules-13,0fc8f7cf73a699d6,02d8ae4e2b291a26] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:58.504 INFO 21062 --- [ocp-schedules-12,a7f551d60aec692f,2344bb505333df28] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:31:58.505 INFO 21062 --- [ocp-schedules-28,b786599280118445,a63942728a5fc860] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:31:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:31:58.512 INFO 21062 --- [ocp-schedules-9,c3ddada0e3724f6f,4989106fc552c724] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:31:58.512 INFO 21062 --- [ocp-schedules-23,5fdd03328d002a9f,a59879e901725695] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:31:58.512 INFO 21062 --- [ocp-schedules-9,c3ddada0e3724f6f,4989106fc552c724] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:31:58.512 INFO 21062 --- [ocp-schedules-23,5fdd03328d002a9f,a59879e901725695] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:31:58.513 INFO 21062 --- [ocp-schedules-3,10162264cbf8e807,5e74c6fc971c93b9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:31:58.513 INFO 21062 --- [ocp-schedules-3,10162264cbf8e807,5e74c6fc971c93b9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:31:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:31:58.589665+08:00, lastProcessTime=2024-11-04T13:31:48.590788+08:00 2024-11-04 13:31:58.602 INFO 21062 --- [ocp-async-7,ff9f31e43320ea01,e2f95a4725e5355f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:31:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:31:58.589665+08:00 2024-11-04 13:31:58.808 INFO 21062 --- [ocp-schedules-7,baf9342a441e4fa6,d479536c42e89276] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:31:58.883 INFO 21062 --- [ocp-schedules-27,005d2a6fc70a7af0,0d7927ce6674d19b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:31:58.885 WARN 21062 --- [ocp-async-1,1d9dafd1610177e3,120cfe4c8e44d474] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:31:58.934 INFO 21062 --- [ocp-async-8,0ff3d40f7454dac8,6bf009b79b6632a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:31:59.543 INFO 21062 --- [ocp-schedules-24,2df180c4b1a9a805,e2cf9a4314235eed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:31:59.983 INFO 21062 --- [prometheus-minute-schedule-1,34c088331b3ae902,b401d3d34e3223a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698320, interval=60 2024-11-04 13:32:00.678 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:00.678 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:00.885 WARN 21062 --- [ocp-async-3,3f6a84904964671d,1fb7c18c7936cc06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:00.934 INFO 21062 --- [ocp-async-6,a24882b4c8756443,baeffc9723062f77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:02.004 INFO 21062 --- [ocp-schedules-28,4a3c0a95d2481532,060f848869e4a15b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:32:02.583 INFO 21062 --- [ocp-schedules-10,4edf387e968790a1,38c3b6ac68a71abd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:32:02.583 INFO 21062 --- [ocp-schedules-10,3df3d989dce6153b,31f85abe3076358d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:32:02.587 INFO 21062 --- [ocp-schedules-10,3df3d989dce6153b,31f85abe3076358d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:02.616 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:32:02.616 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:32:02.886 WARN 21062 --- [ocp-async-6,f1cdf1426333f405,a30a4d0bb287c359] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:02.934 INFO 21062 --- [ocp-async-9,65b62cefc6942e3f,5974afe19321d37c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:02.983 INFO 21062 --- [prometheus-minute-schedule-1,dbfa67a6ff99e089,1cbb6669b452560d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:03.679 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:03.680 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:04.886 WARN 21062 --- [ocp-async-9,bd61f7bf3c345e4e,0a026382c0fdb163] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:04.935 INFO 21062 --- [ocp-async-4,c7543ab602d56491,52603d9ad59a9033] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c7351325b9113baa,9da5b9144394f671] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:06.680 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:06.681 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:06.886 WARN 21062 --- [ocp-async-4,cc30301a6aeabf6a,52feeb3de9f452ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:06.935 INFO 21062 --- [ocp-async-10,e51c8ef4e9eb28d5,41236f02542a5d21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:08.505 INFO 21062 --- [ocp-schedules-12,a7f551d60aec692f,2344bb505333df28] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:32:08.506 INFO 21062 --- [ocp-schedules-13,0fc8f7cf73a699d6,02d8ae4e2b291a26] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:32:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:32:08.590348+08:00, lastProcessTime=2024-11-04T13:31:58.589665+08:00 2024-11-04 13:32:08.602 INFO 21062 --- [ocp-async-8,8288a9ecf09dd7a6,f66e1f115f0ca833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:32:08.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:32:08.590348+08:00 2024-11-04 13:32:08.886 WARN 21062 --- [ocp-async-6,65951020c8152fda,f903219786c0ec22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:08.935 INFO 21062 --- [ocp-async-9,596540f1ef9c274e,c479087f6ad4ba5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:08.983 INFO 21062 --- [prometheus-minute-schedule-1,6f7071a8107a7ab9,09bffcedaf6e4fc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:09.546 INFO 21062 --- [ocp-schedules-25,41777a314606449a,34829c5b6d3c2a7a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:32:09.681 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:09.682 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:32:10.004 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:32:10.156 INFO 21062 --- [ocp-schedules-29,673711cccc534876,8fef6a178ff71107] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:32:10.162 INFO 21062 --- [ocp-schedules-29,673711cccc534876,8fef6a178ff71107] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:32:10.162 INFO 21062 --- [ocp-schedules-29,673711cccc534876,8fef6a178ff71107] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:32:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:32:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:32:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:32:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:32:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:32:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:32:10.886 WARN 21062 --- [ocp-async-9,1f44621fda75f444,248223aaac530c43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:10.935 INFO 21062 --- [ocp-async-4,504db67d97dd46e0,0f484ceea51e83af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:11.268 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=2, affectRows=2 2024-11-04 13:32:11.268 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=4, affectRows=4 2024-11-04 13:32:11.268 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=2, validCount=2 2024-11-04 13:32:11.268 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=4, validCount=4 2024-11-04 13:32:11.268 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=48, affectRows=48 2024-11-04 13:32:11.269 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=48, validCount=48 2024-11-04 13:32:11.269 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=31, affectRows=31 2024-11-04 13:32:11.269 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=15, affectRows=15 2024-11-04 13:32:11.269 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=15, validCount=15 2024-11-04 13:32:11.269 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=31, validCount=31 2024-11-04 13:32:11.270 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=61, affectRows=61 2024-11-04 13:32:11.270 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=84, affectRows=84 2024-11-04 13:32:11.270 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=61, validCount=61 2024-11-04 13:32:11.270 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=83, validCount=83 2024-11-04 13:32:11.271 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698260, nodeCount=77, affectRows=77 2024-11-04 13:32:11.271 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698260, nodeCount=75, validCount=75 2024-11-04 13:32:11.983 INFO 21062 --- [prometheus-minute-schedule-1,3b4b0bd3d4ee1c50,498d78b5944a6642] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:12.049 INFO 21062 --- [ocp-schedules-28,f09721757c1fe166,526cdcae30db7471] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:32:12.682 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:12.683 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:12.886 WARN 21062 --- [ocp-async-4,973be398725ed98c,b99f65af4ea50449] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:12.935 INFO 21062 --- [ocp-async-10,391334dfcc62e078,b0966b7f4e7fc1e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:13.985 INFO 21062 --- [prometheus-cache-monitor1,8b98df0952070b1c,1f07170769d6ce21] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:32:13.985 INFO 21062 --- [prometheus-cache-monitor1,8b98df0952070b1c,1f07170769d6ce21] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:32:14.887 WARN 21062 --- [ocp-async-10,a9bc104607652585,3cce15a04ed540ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:14.936 INFO 21062 --- [ocp-async-7,2405f99291c52f53,f49f5198c560b941] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f320840bcb52fddd,5c0332d6dc14ff8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:15.683 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:15.684 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:16.887 WARN 21062 --- [ocp-async-7,98e7522300560da3,970635108b978b94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:16.936 INFO 21062 --- [ocp-async-2,69ebe75034783e03,3bf8be4de13d81e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:17.983 INFO 21062 --- [prometheus-minute-schedule-1,945e183f5349fd7e,b5753cab19f60768] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:32:18.590307+08:00, lastProcessTime=2024-11-04T13:32:08.590348+08:00 2024-11-04 13:32:18.602 INFO 21062 --- [ocp-async-7,f14e01cdb37d16c4,263e589e92112bee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:32:18.684 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:18.685 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:32:18.590307+08:00 2024-11-04 13:32:18.887 WARN 21062 --- [ocp-async-4,c3473c186a6e18e0,c91e79114f30baf0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:18.936 INFO 21062 --- [ocp-async-10,ea74feb8277a3701,10f9def1d62d606a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:19.549 INFO 21062 --- [ocp-schedules-16,f479d6ae4b308d99,32c0ad23071d21f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:32:20.887 WARN 21062 --- [ocp-async-10,736464cfbf3fea78,92ce92ae71afac33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:20.936 INFO 21062 --- [ocp-async-9,f4b2b344a4cfd9ab,78813f755d00cb30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:20.983 INFO 21062 --- [prometheus-minute-schedule-1,48a8dc7a7c2e44b1,54da91f8f5da521f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:21.685 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:21.686 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:22.079 INFO 21062 --- [ocp-schedules-2,ca08de8840282c34,095388800ef77371] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:32:22.887 WARN 21062 --- [ocp-async-9,808863dd3c2bad0a,e93e88f446e4ca96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:22.936 INFO 21062 --- [ocp-async-2,086a6a40020e5068,1916315e3822f9d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4b95550dd28ef485,613e6416301d1399] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:24.686 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:24.687 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:24.888 WARN 21062 --- [ocp-async-2,3401a8ba83c3f57d,8fe5eab99f8112af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:24.937 INFO 21062 --- [ocp-async-3,5fafd462921f9668,10106feb862a2c91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:26.888 WARN 21062 --- [ocp-async-3,e7b3007c0225abd2,5f68f55e4c107cc1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:26.937 INFO 21062 --- [ocp-async-5,ffe709d8ae1a1e0e,975b4e23ff4aaffb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e3a8a53fab102c7f,23274736f5f42b70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:26.984 INFO 21062 --- [prometheus-hour-schedule-1,7efe8ee5091bf34b,53f63a2710d76d8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:32:27.688 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:27.689 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:28.498 INFO 21062 --- [ocp-schedules-3,c12b3e11bb82051f,86f9ab71a3a282c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:32:28.499 INFO 21062 --- [ocp-schedules-13,5b9399e61a2f50e4,94cdeed2bef8902c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:32:28.500 INFO 21062 --- [ocp-schedules-26,17ee8ca9f3ef8ca3,974d6e4b0d1cf55e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:32:28.500 INFO 21062 --- [ocp-schedules-22,d85fbc44fe20fa32,36a2873fa5933240] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:32:28.502 INFO 21062 --- [ocp-schedules-3,c12b3e11bb82051f,86f9ab71a3a282c6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:32:28.502 INFO 21062 --- [ocp-schedules-24,37152115f8bb9ad2,30e12d7ca0a10b42] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:32:28.504 INFO 21062 --- [ocp-schedules-26,17ee8ca9f3ef8ca3,974d6e4b0d1cf55e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:28.504 INFO 21062 --- [ocp-schedules-13,5b9399e61a2f50e4,94cdeed2bef8902c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:32:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:32:28.504 INFO 21062 --- [ocp-schedules-8,bd81caec7ef67831,5ed168a410267b5f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:32:28.504 INFO 21062 --- [ocp-schedules-11,c25b5f73b640bbbe,e20f401c9a3bfcd3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:32:28.505 INFO 21062 --- [ocp-schedules-22,d85fbc44fe20fa32,36a2873fa5933240] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:28.507 INFO 21062 --- [ocp-schedules-11,c25b5f73b640bbbe,e20f401c9a3bfcd3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:28.511 INFO 21062 --- [ocp-schedules-17,081eb4f9938bf276,3b5d62c4ecf56e30] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:32:28.511 INFO 21062 --- [ocp-schedules-14,7407be748a946805,93e0d3ccd3613cc8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:32:28.511 INFO 21062 --- [ocp-schedules-29,686a504b836968c4,d399392b5fffb9f4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:32:28.511 INFO 21062 --- [ocp-schedules-29,686a504b836968c4,d399392b5fffb9f4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:32:28.511 INFO 21062 --- [ocp-schedules-17,081eb4f9938bf276,3b5d62c4ecf56e30] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:32:28.511 INFO 21062 --- [ocp-schedules-14,7407be748a946805,93e0d3ccd3613cc8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:32:28.531 INFO 21062 --- [ocp-schedules-2,23c9623bef652d88,93d12925c2f384ed] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:32:28.531 INFO 21062 --- [ocp-schedules-2,23c9623bef652d88,93d12925c2f384ed] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:32:28.566 INFO 21062 --- [ocp-schedules-12,c59ee8b37ff8d7ae,cc192a5b07a863cd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:32:28.566 INFO 21062 --- [ocp-schedules-12,c59ee8b37ff8d7ae,cc192a5b07a863cd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:32:28.573 INFO 21062 --- [pool-12-thread-3,46007de623241e1e,92a3e59e2c780627] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:32:28.573 INFO 21062 --- [pool-12-thread-1,46007de623241e1e,fdb4ace6076e6879] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:32:28.574 INFO 21062 --- [pool-12-thread-4,46007de623241e1e,62f238b80a1982d9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:32:28.574 INFO 21062 --- [pool-12-thread-4,46007de623241e1e,62f238b80a1982d9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:32:28.574 INFO 21062 --- [pool-12-thread-2,46007de623241e1e,91fd8220344b195e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:32:28.578 INFO 21062 --- [ForkJoinPool-11-worker-9,46007de623241e1e,e8d9f7abf10d63ba] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:32:28.579 INFO 21062 --- [pool-12-thread-3,46007de623241e1e,92a3e59e2c780627] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:32:28.579 INFO 21062 --- [ForkJoinPool-11-worker-2,46007de623241e1e,faf014e491f3645b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:32:28.579 INFO 21062 --- [ForkJoinPool-11-worker-2,46007de623241e1e,faf014e491f3645b] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:32:28.579 INFO 21062 --- [ForkJoinPool-11-worker-2,46007de623241e1e,faf014e491f3645b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:32:28.579 INFO 21062 --- [pool-12-thread-1,46007de623241e1e,fdb4ace6076e6879] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:32:28.579 INFO 21062 --- [pool-12-thread-4,46007de623241e1e,62f238b80a1982d9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:32:28.580 INFO 21062 --- [pool-12-thread-2,46007de623241e1e,91fd8220344b195e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:32:28.580 INFO 21062 --- [pool-12-thread-2,46007de623241e1e,91fd8220344b195e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:32:28.583 INFO 21062 --- [pool-12-thread-4,46007de623241e1e,62f238b80a1982d9] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:32:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:32:28.588788+08:00, lastProcessTime=2024-11-04T13:32:18.590307+08:00 2024-11-04 13:32:28.603 INFO 21062 --- [ocp-async-6,5ddce01ea3fc7ad7,4be37be08eaf8598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:32:28.697 INFO 21062 --- [ocp-schedules-13,99db5ecae64a13b4,51f99562901f47d2] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 13:32:28.697 INFO 21062 --- [ocp-schedules-13,99db5ecae64a13b4,51f99562901f47d2] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 13:32:28.697 INFO 21062 --- [ocp-schedules-13,99db5ecae64a13b4,51f99562901f47d2] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=322 2024-11-04 13:32:28.703 INFO 21062 --- [ocp-schedules-24,8eb8f966d48bbedd,c8f6ed4091c296f1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:32:28.711 INFO 21062 --- [ocp-schedules-13,99db5ecae64a13b4,51f99562901f47d2] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=322 2024-11-04 13:32:28.712 INFO 21062 --- [ocp-schedules-8,2484d76fb1004477,1999212e7dbea767] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:32:28.726 INFO 21062 --- [ocp-schedules-8,2484d76fb1004477,1999212e7dbea767] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:32:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:32:28.588788+08:00 2024-11-04 13:32:28.856 INFO 21062 --- [ocp-schedules-17,c06cbab7e239c72b,f6fd12b7ba58791a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 13:32:28.860 INFO 21062 --- [ocp-schedules-17,c06cbab7e239c72b,f6fd12b7ba58791a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:28.865 INFO 21062 --- [ocp-schedules-1,67e403feaf2727ab,1c281a4133ed74ff] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:32:28.865 INFO 21062 --- [ocp-schedules-1,67e403feaf2727ab,1c281a4133ed74ff] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:32:28.886 INFO 21062 --- [ocp-schedules-2,b81f2aa06de167b7,ed6a6a3ddbc658cb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:32:28.888 WARN 21062 --- [ocp-async-1,03c7817ea44ec884,961b62b3ce704a80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:28.896 ERROR 21062 --- [ocp-schedules-17,c06cbab7e239c72b,f6fd12b7ba58791a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:32:28.896 INFO 21062 --- [ocp-schedules-17,c06cbab7e239c72b,f6fd12b7ba58791a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 13:32:28.915 INFO 21062 --- [ocp-schedules-23,0732a408616c1b89,7131c92deda55d5a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:32:28.920 INFO 21062 --- [ocp-schedules-23,0732a408616c1b89,7131c92deda55d5a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:32:28.920 INFO 21062 --- [ocp-schedules-23,0732a408616c1b89,7131c92deda55d5a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:32:28.937 INFO 21062 --- [ocp-async-3,e794a3beb5bf16eb,c267ce601d552398] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:29.099 INFO 21062 --- [ocp-schedules-4,9135b5e1acee2fb4,45ffb3f6be8c398e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:32:29.099 INFO 21062 --- [ocp-schedules-4,9135b5e1acee2fb4,45ffb3f6be8c398e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:32:29.100 INFO 21062 --- [ocp-schedules-4,9135b5e1acee2fb4,45ffb3f6be8c398e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:32:29.100 INFO 21062 --- [ocp-schedules-4,9135b5e1acee2fb4,45ffb3f6be8c398e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:32:29.102 INFO 21062 --- [ocp-schedules-4,9135b5e1acee2fb4,45ffb3f6be8c398e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:32:29.102 INFO 21062 --- [ocp-schedules-4,9135b5e1acee2fb4,45ffb3f6be8c398e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:32:29.183 INFO 21062 --- [ocp-schedules-18,5f307e01f0eddf2c,f568e545f6d68cdf] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:32:29.183 INFO 21062 --- [ocp-schedules-18,5f307e01f0eddf2c,f568e545f6d68cdf] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:32:29.211 INFO 21062 --- [pool-31-thread-1,5f307e01f0eddf2c,618a1349a8a9a880] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:32:29.240 WARN 21062 --- [ocp-schedules-18,5f307e01f0eddf2c,f568e545f6d68cdf] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:32:29.240 INFO 21062 --- [ocp-schedules-18,5f307e01f0eddf2c,f568e545f6d68cdf] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:32:29.552 INFO 21062 --- [ocp-schedules-9,bdd88af606038015,50b907aa429e49e3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:32:29.983 INFO 21062 --- [prometheus-minute-schedule-1,5b258bf7094ef681,c39425cadc18b44f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:30.165 INFO 21062 --- [ocp-schedules-1,9e0a717edd2e67cf,03c134b4478d30d8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:32:30.170 INFO 21062 --- [ocp-schedules-28,f84233d57ceb83c2,60145c36cd34c2f0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:32:30.170 INFO 21062 --- [ocp-schedules-28,f84233d57ceb83c2,60145c36cd34c2f0] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:32:30.170 INFO 21062 --- [ocp-schedules-1,9e0a717edd2e67cf,03c134b4478d30d8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:32:30.170 INFO 21062 --- [ocp-schedules-1,9e0a717edd2e67cf,03c134b4478d30d8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:32:30.195 INFO 21062 --- [ocp-schedules-28,f84233d57ceb83c2,60145c36cd34c2f0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:32:30.195 INFO 21062 --- [ocp-schedules-28,f84233d57ceb83c2,60145c36cd34c2f0] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:32:30.195 INFO 21062 --- [ocp-schedules-27,9f3ff6b2bd4cdc71,56c66dd8611f8e42] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:32:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:32:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:32:30.326 INFO 21062 --- [ocp-schedules-20,30bc489c7224f7e5,ef4046569081d39f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:32:30.326+08:00, clusters=[1] 2024-11-04 13:32:30.327 INFO 21062 --- [ocp-schedules-20,30bc489c7224f7e5,ef4046569081d39f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:32:30.352 INFO 21062 --- [ocp-schedules-16,81ff0fdc4525f12f,249841d56d681a91] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:32:30.353 INFO 21062 --- [ocp-schedules-4,e40eb5cf88c6bfc5,ad7e5d5096627bec] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:32:30.353 INFO 21062 --- [ocp-schedules-4,e40eb5cf88c6bfc5,ad7e5d5096627bec] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:32:30.353 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:32:30.353 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:32:30.355 INFO 21062 --- [ocp-schedules-16,81ff0fdc4525f12f,249841d56d681a91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:30.363 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:32:30.363 INFO 21062 --- [ocp-schedules-4,e40eb5cf88c6bfc5,ad7e5d5096627bec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:32:30.363 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:32:30.365 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:32:30.365 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:32:30.365 WARN 21062 --- [ocp-schedules-4,e40eb5cf88c6bfc5,ad7e5d5096627bec] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:32:30.365 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:32:30.365 INFO 21062 --- [ocp-schedules-4,e40eb5cf88c6bfc5,ad7e5d5096627bec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:32:30.365 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:32:30.366 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:32:30.366 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:32:30.366 WARN 21062 --- [ocp-schedules-4,e40eb5cf88c6bfc5,ad7e5d5096627bec] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:32:30.374 ERROR 21062 --- [ocp-schedules-4,e40eb5cf88c6bfc5,ad7e5d5096627bec] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:32:30.410 ERROR 21062 --- [ocp-schedules-16,81ff0fdc4525f12f,249841d56d681a91] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:32:30.410 INFO 21062 --- [ocp-schedules-16,81ff0fdc4525f12f,249841d56d681a91] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:32:30.459 INFO 21062 --- [ocp-schedules-7,b770915c42ce6dac,58c767cf076bd342] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:32:30.459+08:00, clusters=[1] 2024-11-04 13:32:30.463 INFO 21062 --- [ocp-schedules-7,b770915c42ce6dac,58c767cf076bd342] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:32:30.689 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:30.690 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:30.888 WARN 21062 --- [ocp-async-3,125802a0bb765a19,ca5c53f6fc73167f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:30.937 INFO 21062 --- [ocp-async-6,5d089289d2f939ae,97bf758a9d1e929e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:32.123 INFO 21062 --- [ocp-schedules-20,6c7ab821623a7f90,24efbd56707741d5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:32:32.617 INFO 21062 --- [ocp-schedules-10,3df3d989dce6153b,31f85abe3076358d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:32:32.617 INFO 21062 --- [ocp-schedules-10,7074f567343df2b8,7f81675aae1a0043] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:32:32.621 INFO 21062 --- [ocp-schedules-10,7074f567343df2b8,7f81675aae1a0043] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:32.650 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:32:32.650 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:32:32.888 WARN 21062 --- [ocp-async-6,1a5b34508d6eed4d,a65537a52606e329] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:32.937 INFO 21062 --- [ocp-async-9,994d5754c2f84760,d026857499d29599] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:32.983 INFO 21062 --- [prometheus-minute-schedule-1,62677ded60728f4d,22df666bc305c22d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:33.690 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:33.691 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:34.036 INFO 21062 --- [ocp-schedules-17,2d5e0c74b4b37ea2,cf190c3e01f8aea9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:32:34.042 INFO 21062 --- [ocp-schedules-17,2d5e0c74b4b37ea2,cf190c3e01f8aea9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:34.126 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:32:34.129 ERROR 21062 --- [ocp-schedules-17,2d5e0c74b4b37ea2,cf190c3e01f8aea9] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:32:34.129 INFO 21062 --- [ocp-schedules-17,2d5e0c74b4b37ea2,cf190c3e01f8aea9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:32:34.889 WARN 21062 --- [ocp-async-9,e995ed8f8a21451d,e6a3f3cee54d1bdc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:34.937 INFO 21062 --- [ocp-async-2,1cc03b772f9ea79a,00f3c91b288c6588] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:35.983 INFO 21062 --- [prometheus-minute-schedule-1,2481fccba3ea10fa,f37748d3f1a6b892] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:36.691 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:36.692 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:36.889 WARN 21062 --- [ocp-async-2,e39216e7d6c716b0,f7affd2d98cce679] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:36.938 INFO 21062 --- [ocp-async-8,2cfa3f5be060afb2,2f67ecc0caf3acc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:38.505 INFO 21062 --- [ocp-schedules-26,17ee8ca9f3ef8ca3,974d6e4b0d1cf55e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:32:38.506 INFO 21062 --- [ocp-schedules-22,d85fbc44fe20fa32,36a2873fa5933240] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:32:38.508 INFO 21062 --- [ocp-schedules-11,c25b5f73b640bbbe,e20f401c9a3bfcd3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:32:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:32:38.589513+08:00, lastProcessTime=2024-11-04T13:32:28.588788+08:00 2024-11-04 13:32:38.603 INFO 21062 --- [ocp-async-3,e4d6eedd5d88a084,5d7e946502ba2e35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:32:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:32:38.589513+08:00 2024-11-04 13:32:38.889 WARN 21062 --- [ocp-async-5,d15d089e51382fe8,ff8564474fe95191] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:38.938 INFO 21062 --- [ocp-async-9,27a4c05aaa2f495d,58808ed311f966cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:38.983 INFO 21062 --- [prometheus-minute-schedule-1,b8d3a68c75e9fc24,e1c6965ba8a38f9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:39.554 INFO 21062 --- [ocp-schedules-21,40804c48deaebbd6,950c195ac81b8ae3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:32:39.692 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:39.693 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:40.002 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:32:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:32:40.889 WARN 21062 --- [ocp-async-9,c9b7deb88f39be78,4ec7e19fb66a2d76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:40.938 INFO 21062 --- [ocp-async-2,577d3e776c2947c2,666ca6fcd2448084] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:32:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:32:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:32:41.983 INFO 21062 --- [prometheus-minute-schedule-1,cf7a29b6d1c98bcc,cd2416726c94e307] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:42.149 INFO 21062 --- [ocp-schedules-22,8d809aee49e60f64,40db6c4e505033e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:32:42.693 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:42.694 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:42.889 WARN 21062 --- [ocp-async-3,fdb5a9953347e1b3,d8a4fb9786905843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:42.938 INFO 21062 --- [ocp-async-8,c705ca9e0f7b1b93,c79bcd8084ecde7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:44.889 WARN 21062 --- [ocp-async-6,97538e5374adf479,54c7c18825590948] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:44.938 INFO 21062 --- [ocp-async-7,0bedd2f289c04986,e169a9ab078d2234] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6654f48c748e966f,5a8fde9f5bba4b6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:45.695 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:45.695 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:46.890 WARN 21062 --- [ocp-async-1,083d67130fe521c8,b11e048e7e667d1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:46.938 INFO 21062 --- [ocp-async-4,0730216be41878d0,a92e9bec7dbe5304] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:47.983 INFO 21062 --- [prometheus-minute-schedule-1,90f8c0acff074aa2,19a2968829c0d6c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:32:48.590431+08:00, lastProcessTime=2024-11-04T13:32:38.589513+08:00 2024-11-04 13:32:48.603 INFO 21062 --- [ocp-async-1,4ea9c1a758629f5c,dc73c1356f17656b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:32:48.696 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:48.696 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:32:48.590431+08:00 2024-11-04 13:32:48.890 WARN 21062 --- [ocp-async-2,c30a73f89a95416b,97da43afa7e70b4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:48.939 INFO 21062 --- [ocp-async-8,941cd4eb0190ac29,b6e11eeef0d092a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:49.558 INFO 21062 --- [ocp-schedules-12,a6ff0d7eeb41dea2,dbb70f983538a738] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:32:50.173 INFO 21062 --- [ocp-schedules-29,87a33f2270ec7b39,6bf1dcd426de7efb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:32:50.180 INFO 21062 --- [ocp-schedules-29,87a33f2270ec7b39,6bf1dcd426de7efb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:32:50.180 INFO 21062 --- [ocp-schedules-29,87a33f2270ec7b39,6bf1dcd426de7efb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:32:50.890 WARN 21062 --- [ocp-async-8,b90af6769377848b,dc9262ed5e2fba35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:50.939 INFO 21062 --- [ocp-async-7,2088f67f3c48358a,6ed2b7e6a0c085d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:50.983 INFO 21062 --- [prometheus-minute-schedule-1,f45735173df8aff1,7ab594fb41de274a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:51.697 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:51.698 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:52.190 INFO 21062 --- [ocp-schedules-9,67a1e70cc551a478,711c6ec7ee847a55] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:32:52.890 WARN 21062 --- [ocp-async-7,72013c969735d4b7,a51a7e9414d04df4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:52.939 INFO 21062 --- [ocp-async-4,ac9e89d0758fc9dc,d510dd09d55bae2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9f5f247d118a0efe,fb0a910b597e650e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:54.698 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:54.698 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:54.890 WARN 21062 --- [ocp-async-4,a6dbbc54e8118d52,af42ea4b114807ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:54.939 INFO 21062 --- [ocp-async-10,9ce871f2085b2482,b4913b69276fb596] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:56.891 WARN 21062 --- [ocp-async-10,d50ee59a959d8921,649d8e599512d7e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:56.939 INFO 21062 --- [ocp-async-5,563e4e564d307de7,35dcd19841f499f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:56.983 INFO 21062 --- [prometheus-minute-schedule-1,a7682c37856a9242,5b939d001a320d52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:32:57.699 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:32:57.699 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:32:58.499 INFO 21062 --- [ocp-schedules-30,4504761c080a23f2,2e29b3cd26c33654] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:32:58.500 INFO 21062 --- [ocp-schedules-27,6a16d8290deae0c4,31ef999a61b33839] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:32:58.502 INFO 21062 --- [ocp-schedules-25,57e30b68b03ef91c,f0a1b743804e41f4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:32:58.503 INFO 21062 --- [ocp-schedules-30,4504761c080a23f2,2e29b3cd26c33654] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:32:58.504 INFO 21062 --- [ocp-schedules-27,6a16d8290deae0c4,31ef999a61b33839] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:58.505 INFO 21062 --- [ocp-schedules-13,8d032ed3b7e66e75,eb04bbfdc821edfc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:32:58.511 INFO 21062 --- [ocp-schedules-21,50f88431d46856e4,989bcda1fabbf230] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:32:58.511 INFO 21062 --- [ocp-schedules-16,fd7bf201bc0b0c5e,f5316dad995be9e1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:32:58.511 INFO 21062 --- [ocp-schedules-21,50f88431d46856e4,989bcda1fabbf230] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:32:58.511 INFO 21062 --- [ocp-schedules-16,fd7bf201bc0b0c5e,f5316dad995be9e1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:32:58.512 INFO 21062 --- [ocp-schedules-3,6575af98dea23a45,13f9d7c67739a425] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:32:58.512 INFO 21062 --- [ocp-schedules-3,6575af98dea23a45,13f9d7c67739a425] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:32:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:32:58.589945+08:00, lastProcessTime=2024-11-04T13:32:48.590431+08:00 2024-11-04 13:32:58.603 INFO 21062 --- [ocp-async-10,a53ce96cc2f628c0,f22a2adf8a24ff0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:32:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:32:58.589945+08:00 2024-11-04 13:32:58.808 INFO 21062 --- [ocp-schedules-11,566ca86e48ebf4bb,13f109fc86844f91] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:32:58.811 INFO 21062 --- [ocp-schedules-11,566ca86e48ebf4bb,13f109fc86844f91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:32:58.812 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:32:58.888 INFO 21062 --- [ocp-schedules-20,89b288bf3f606643,045ad6dfa7e8ba14] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:32:58.891 WARN 21062 --- [ocp-async-7,df4cc21f783beacd,dfd382a55e046a16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:32:58.939 INFO 21062 --- [ocp-async-4,8edf229a3b8f8d3f,552e2b3d85dc1c38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:32:59.560 INFO 21062 --- [ocp-schedules-25,a598d4f09e31ccca,48204657f318233c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:32:59.983 INFO 21062 --- [prometheus-minute-schedule-1,9df2e4fc79579dd3,ace2468479abedc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698380, interval=60 2024-11-04 13:33:00.700 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:00.700 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:00.891 WARN 21062 --- [ocp-async-4,645ba15d669d1430,682743ccddec98a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:00.940 INFO 21062 --- [ocp-async-8,bcfe4764b08aeb9c,598cab0dc7f3ad27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:02.214 INFO 21062 --- [ocp-schedules-4,02061358959b6de2,5ced6f79ffa2dc3b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:33:02.651 INFO 21062 --- [ocp-schedules-10,7074f567343df2b8,7f81675aae1a0043] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:33:02.651 INFO 21062 --- [ocp-schedules-10,ac186d7764257920,988c1384614a0bc3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:33:02.655 INFO 21062 --- [ocp-schedules-10,ac186d7764257920,988c1384614a0bc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:33:02.684 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:33:02.684 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:33:02.891 WARN 21062 --- [ocp-async-8,38fdef2a1fc5ad14,776c5f6005164808] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:02.940 INFO 21062 --- [ocp-async-5,9726f9525207cb8a,7b22e408c23917cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:02.983 INFO 21062 --- [prometheus-minute-schedule-1,30f97b346407af2d,5433ea7bffa69f4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:03.701 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:03.701 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:04.891 WARN 21062 --- [ocp-async-5,d0a44162865ec3a7,642ee229b64041a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:04.940 INFO 21062 --- [ocp-async-1,3d5fabf0c7e49e25,22ef25a9102f1e0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:05.983 INFO 21062 --- [prometheus-minute-schedule-1,025b8f653503393e,aacbb3c8ba8ec58c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:06.701 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:06.702 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:06.891 WARN 21062 --- [ocp-async-1,084d86b95d92f14a,f83948d134bd61a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:06.940 INFO 21062 --- [ocp-async-2,fdf7f0b256318154,a41a26b525be375e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:08.504 INFO 21062 --- [ocp-schedules-30,4504761c080a23f2,2e29b3cd26c33654] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:33:08.505 INFO 21062 --- [ocp-schedules-27,6a16d8290deae0c4,31ef999a61b33839] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:33:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:33:08.589262+08:00, lastProcessTime=2024-11-04T13:32:58.589945+08:00 2024-11-04 13:33:08.603 INFO 21062 --- [ocp-async-1,10728c01009c1153,cc0064823d839506] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:33:08.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:33:08.589262+08:00 2024-11-04 13:33:08.812 INFO 21062 --- [ocp-schedules-11,566ca86e48ebf4bb,13f109fc86844f91] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:33:08.892 WARN 21062 --- [ocp-async-8,df41b03cbedb57aa,03a2ca1c9a949d1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:08.940 INFO 21062 --- [ocp-async-5,d46d06667964e23a,a4651a32b4cf04b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:08.983 INFO 21062 --- [prometheus-minute-schedule-1,65bed57716282e17,62ec36897d1daccf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:08.987 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:33:08.987 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:33:08.996 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:33:08.996 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:33:08.996 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:33:08.996 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:33:09.003 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:33:09.003 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:33:09.003 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:33:09.003 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:33:09.037 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:33:09.045 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:33:09.045 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:33:09.046 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:33:09.046 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:33:09.046 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:33:09.054 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:33:09.563 INFO 21062 --- [ocp-schedules-22,dcf4434aa5f8f665,eec2a463815dc01c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:33:09.575 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:33:09.703 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:09.703 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:10.004 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:33:10.004 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:33:10.128 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.125301974S 2024-11-04 13:33:10.129 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:33:10.183 INFO 21062 --- [ocp-schedules-21,d834cf7237e9e87e,6bdfc61138c01f40] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:33:10.190 INFO 21062 --- [ocp-schedules-21,d834cf7237e9e87e,6bdfc61138c01f40] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:33:10.190 INFO 21062 --- [ocp-schedules-21,d834cf7237e9e87e,6bdfc61138c01f40] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:33:10.266 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:33:10.266 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:33:10.292 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:33:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:33:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:33:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:33:10.354 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:33:10.354 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:33:10.354 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:33:10.354 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:33:10.354 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:33:10.354 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:33:10.366 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:33:10.547 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] o.s.boot.SpringApplication : Started application in 1.551 seconds (JVM running for 3804.226) 2024-11-04 13:33:10.547 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.551609617S 2024-11-04 13:33:10.892 WARN 21062 --- [ocp-async-5,a61434227534bb95,1450f60b7dbe67ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:10.941 INFO 21062 --- [ocp-async-1,eda7121e7a747efb,3c34a7f7d1bf1816] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:10.997 INFO 21062 --- [ocp-schedules-27,fa218e1d99b6e39d,5e0dc0a301cffbbd] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:33:11.276 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698320, nodeCount=322, affectRows=322 2024-11-04 13:33:11.277 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698320, nodeCount=319, validCount=319 2024-11-04 13:33:11.983 INFO 21062 --- [prometheus-minute-schedule-1,84b8dea8e763e986,53b859ae6a5a6103] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:12.265 INFO 21062 --- [ocp-schedules-4,133ce69ba39bd0b7,10d4fedd84241fb8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:33:12.704 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:12.705 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:12.892 WARN 21062 --- [ocp-async-1,f627706a5ccd5a24,23f59870f6a76b16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:12.941 INFO 21062 --- [ocp-async-2,f1b6bb6b06d001fe,decbe4af9d6ebc79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:13.984 INFO 21062 --- [prometheus-cache-monitor1,2e7c927f6d049bf3,9a283e0c83063b5c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:33:13.984 INFO 21062 --- [prometheus-cache-monitor1,2e7c927f6d049bf3,9a283e0c83063b5c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:33:14.060 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:27:14.041, end=2024-11-04T13:32:14.057} 2024-11-04 13:33:14.060 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:27:14.041, end=2024-11-04T13:32:14.057} 2024-11-04 13:33:14.067 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:33:14.067 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:33:14.067 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:33:14.067 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:33:14.593 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:29:14.569, end=2024-11-04T13:31:14.591} 2024-11-04 13:33:14.597 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:33:14.597 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:33:14.892 WARN 21062 --- [ocp-async-2,88cf36517c9b8e85,04e6458079c41ae1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:14.941 INFO 21062 --- [ocp-async-6,346ef942e1d83572,94ea0b4adbe1431f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d805b38551e4864a,c931f45dc192fade] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:15.705 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:15.705 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:16.893 WARN 21062 --- [ocp-async-6,4e6162946272cbeb,11411f8e9fd3d796] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:16.941 INFO 21062 --- [ocp-async-7,257415354e6173e5,6c932b00e99ff8b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:17.983 INFO 21062 --- [prometheus-minute-schedule-1,0b8c2f927be260ea,9061f3a678d67513] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:33:18.590349+08:00, lastProcessTime=2024-11-04T13:33:08.589262+08:00 2024-11-04 13:33:18.604 INFO 21062 --- [ocp-async-6,4ee06626980677f8,2ced600b97777fa9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:33:18.706 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:18.706 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:18.747 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:33:18.590349+08:00 2024-11-04 13:33:18.893 WARN 21062 --- [ocp-async-1,637eaa009f8e1c6c,dec1a8242714f6dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:18.941 INFO 21062 --- [ocp-async-3,151c7e02b6aef5e0,7661422156d15f25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:19.568 INFO 21062 --- [ocp-schedules-12,c38e337f5d9e4c76,a97ff0779d78c6b1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:33:20.893 WARN 21062 --- [ocp-async-3,cbdfa426fe966ba4,bf63c5e2061429ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:20.941 INFO 21062 --- [ocp-async-10,6058088a8d556c39,91fbb6d70f683685] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e29811ab7577e38c,7d972993bdf45d39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:21.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:21.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:22.303 INFO 21062 --- [ocp-schedules-4,d0c0cbcaa73be1e3,2b24de8f43f32301] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:33:22.893 WARN 21062 --- [ocp-async-10,47de28d1d148bac3,f5baad9aae11b9f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:22.941 INFO 21062 --- [ocp-async-9,0ec07f218df72409,7318b8501ca757a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:23.983 INFO 21062 --- [prometheus-minute-schedule-1,0b73dab809ab3022,8b5f00a73b247eea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:24.708 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:24.708 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:24.893 WARN 21062 --- [ocp-async-7,c2b7feffd693d5c8,bb28d31349006d73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:24.942 INFO 21062 --- [ocp-async-4,44d4398bfe30adec,27c67e8cab3c8c71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:26.893 WARN 21062 --- [ocp-async-2,d8fb736a55dfb765,6c943a1efb2ed45e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:26.942 INFO 21062 --- [ocp-async-8,937e62370b119587,48f3d5c0dc439f06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3ec8cc40ae395f13,36310163af68b3bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:26.984 INFO 21062 --- [prometheus-hour-schedule-1,de8622c92a63d0a8,20582a9d3b0eb99a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:33:27.708 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:27.709 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:28.498 INFO 21062 --- [ocp-schedules-11,05e6e250a1938dc0,28ec9194b5d17a83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:33:28.499 INFO 21062 --- [ocp-schedules-9,d83a10528c81d3a2,bb920f128cba4e88] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:33:28.500 INFO 21062 --- [ocp-schedules-17,241d9e804d6bfda1,4c5aaeb3388fd99c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:33:28.500 INFO 21062 --- [ocp-schedules-13,e537fdbadaf5f1f1,e9ff6afab957e060] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:33:28.502 INFO 21062 --- [ocp-schedules-11,05e6e250a1938dc0,28ec9194b5d17a83] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:33:28.502 INFO 21062 --- [ocp-schedules-6,2df4a501ed6bf951,fe63cf7805326434] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:33:28.503 INFO 21062 --- [ocp-schedules-17,241d9e804d6bfda1,4c5aaeb3388fd99c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:33:28.504 INFO 21062 --- [ocp-schedules-13,e537fdbadaf5f1f1,e9ff6afab957e060] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:33:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:33:28.504 INFO 21062 --- [ocp-schedules-1,20c80112d6de40f5,8e91048c61293a48] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:33:28.504 INFO 21062 --- [ocp-schedules-9,d83a10528c81d3a2,bb920f128cba4e88] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:33:28.510 INFO 21062 --- [ocp-schedules-2,7c3da5c1ed6ec41f,427b54de0d5a1acf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:33:28.510 INFO 21062 --- [ocp-schedules-12,f5a74de097223369,c8c1d6f152edc873] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:33:28.510 INFO 21062 --- [ocp-schedules-29,db5c983ff39f4aa7,4ac46f8e9bee2554] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:33:28.510 INFO 21062 --- [ocp-schedules-12,f5a74de097223369,c8c1d6f152edc873] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:33:28.510 INFO 21062 --- [ocp-schedules-29,db5c983ff39f4aa7,4ac46f8e9bee2554] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:33:28.510 INFO 21062 --- [ocp-schedules-2,7c3da5c1ed6ec41f,427b54de0d5a1acf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:33:28.561 INFO 21062 --- [ocp-schedules-18,bdd39e5321216c48,57ca30a489a476a3] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 13:33:28.561 INFO 21062 --- [ocp-schedules-18,bdd39e5321216c48,57ca30a489a476a3] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 13:33:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:33:28.588829+08:00, lastProcessTime=2024-11-04T13:33:18.590349+08:00 2024-11-04 13:33:28.600 INFO 21062 --- [ocp-schedules-23,69e9d1efd3cfa883,9dc6b90409e592b8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:33:28.600 INFO 21062 --- [ocp-schedules-23,69e9d1efd3cfa883,9dc6b90409e592b8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:33:28.604 INFO 21062 --- [ocp-async-8,a23d01901a4657fb,63c8a80be227d3f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:33:28.706 INFO 21062 --- [ocp-schedules-30,079f1abb7ceb562f,f36ad732e6a6c019] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:33:28.740 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:33:28.588829+08:00 2024-11-04 13:33:28.869 INFO 21062 --- [ocp-schedules-5,371c450af0f8cfc2,29ba7dc199ab8409] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:33:28.869 INFO 21062 --- [ocp-schedules-5,371c450af0f8cfc2,29ba7dc199ab8409] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:33:28.891 INFO 21062 --- [ocp-schedules-6,309ac746e3898fe1,48383a4173e27b5d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:33:28.893 WARN 21062 --- [ocp-async-9,a06cb4cdcd028007,4b748600f70b29f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:28.920 INFO 21062 --- [ocp-schedules-29,51c342db05b74b70,808385834dd26b0e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:33:28.925 INFO 21062 --- [ocp-schedules-29,51c342db05b74b70,808385834dd26b0e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:33:28.925 INFO 21062 --- [ocp-schedules-29,51c342db05b74b70,808385834dd26b0e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:33:28.942 INFO 21062 --- [ocp-async-1,8d3651a4c72de29d,f54bb8dba383fa9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:29.105 INFO 21062 --- [ocp-schedules-16,a23f70a292308f9d,110b9099be176eb3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:33:29.105 INFO 21062 --- [ocp-schedules-16,a23f70a292308f9d,110b9099be176eb3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:33:29.112 INFO 21062 --- [ocp-schedules-16,a23f70a292308f9d,110b9099be176eb3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:33:29.112 INFO 21062 --- [ocp-schedules-16,a23f70a292308f9d,110b9099be176eb3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:33:29.115 INFO 21062 --- [ocp-schedules-16,a23f70a292308f9d,110b9099be176eb3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:33:29.116 INFO 21062 --- [ocp-schedules-16,a23f70a292308f9d,110b9099be176eb3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:33:29.572 INFO 21062 --- [ocp-schedules-15,b5f7eabad7c4db87,9589dc2070e47c88] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:33:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4f1959056b73b355,cf9da2bad5f0b2b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:30.193 INFO 21062 --- [ocp-schedules-6,6648405095c1050a,d9ef462a7cfd7f1c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:33:30.196 INFO 21062 --- [ocp-schedules-14,d45ad9a774b8706b,b0731942108cae9d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:33:30.196 INFO 21062 --- [ocp-schedules-14,d45ad9a774b8706b,b0731942108cae9d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:33:30.198 INFO 21062 --- [ocp-schedules-6,6648405095c1050a,d9ef462a7cfd7f1c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:33:30.198 INFO 21062 --- [ocp-schedules-6,6648405095c1050a,d9ef462a7cfd7f1c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:33:30.222 INFO 21062 --- [ocp-schedules-14,d45ad9a774b8706b,b0731942108cae9d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:33:30.222 INFO 21062 --- [ocp-schedules-14,d45ad9a774b8706b,b0731942108cae9d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:33:30.222 INFO 21062 --- [ocp-schedules-2,00ec3d76b3a9257e,49ab9b48a938534a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:33:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:33:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:33:30.354 INFO 21062 --- [ocp-schedules-21,e6211442b2d03f0e,46faebe239daec42] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:33:30.354+08:00, clusters=[1] 2024-11-04 13:33:30.355 INFO 21062 --- [ocp-schedules-21,e6211442b2d03f0e,46faebe239daec42] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:33:30.381 INFO 21062 --- [ocp-schedules-16,25e2b701b9e643b9,8ed7d2dc44d782a0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:33:30.381 INFO 21062 --- [ocp-schedules-16,25e2b701b9e643b9,8ed7d2dc44d782a0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:33:30.382 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:33:30.382 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:33:30.391 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:33:30.391 INFO 21062 --- [ocp-schedules-16,25e2b701b9e643b9,8ed7d2dc44d782a0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:33:30.391 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:33:30.392 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:33:30.393 WARN 21062 --- [ocp-schedules-16,25e2b701b9e643b9,8ed7d2dc44d782a0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:33:30.393 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:33:30.393 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:33:30.393 INFO 21062 --- [ocp-schedules-16,25e2b701b9e643b9,8ed7d2dc44d782a0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:33:30.393 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:33:30.394 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:33:30.394 WARN 21062 --- [ocp-schedules-16,25e2b701b9e643b9,8ed7d2dc44d782a0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:33:30.394 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:33:30.401 ERROR 21062 --- [ocp-schedules-16,25e2b701b9e643b9,8ed7d2dc44d782a0] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:33:30.486 INFO 21062 --- [ocp-schedules-23,9e358476c16e7dcf,9d36610157ae6e37] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:33:30.486+08:00, clusters=[1] 2024-11-04 13:33:30.489 INFO 21062 --- [ocp-schedules-23,9e358476c16e7dcf,9d36610157ae6e37] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:33:30.709 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:30.710 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:30.894 WARN 21062 --- [ocp-async-2,2bdbc85272dfcbce,5ac771afcbabd904] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:30.942 INFO 21062 --- [ocp-async-3,44068096bea2cafc,2f366a3d5b9dc25b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:32.333 INFO 21062 --- [ocp-schedules-24,28e38d87a6916623,0af86101b3fcf2e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:33:32.685 INFO 21062 --- [ocp-schedules-10,ac186d7764257920,988c1384614a0bc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:33:32.685 INFO 21062 --- [ocp-schedules-10,6864175201e564ea,7b9398a571085df6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:33:32.688 INFO 21062 --- [ocp-schedules-10,6864175201e564ea,7b9398a571085df6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:33:32.717 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:33:32.717 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:33:32.894 WARN 21062 --- [ocp-async-8,37b942b551731adc,de84ce96ae4978e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:32.942 INFO 21062 --- [ocp-async-10,0b825edd7dde8fb0,58f2e8a3c9ea7854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:32.983 INFO 21062 --- [prometheus-minute-schedule-1,04a4ef8550d2b27e,bf0610bc1a5431de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:33.710 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:33.711 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:34.129 INFO 21062 --- [ocp-schedules-6,be6ad570db985bfc,adfbcfe6cd6a2f3a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:33:34.133 INFO 21062 --- [ocp-schedules-6,be6ad570db985bfc,adfbcfe6cd6a2f3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:33:34.215 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:33:34.222 ERROR 21062 --- [ocp-schedules-6,be6ad570db985bfc,adfbcfe6cd6a2f3a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:33:34.222 INFO 21062 --- [ocp-schedules-6,be6ad570db985bfc,adfbcfe6cd6a2f3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:33:34.894 WARN 21062 --- [ocp-async-5,ece26e60e7fc9337,af4ece518ecc6868] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:34.943 INFO 21062 --- [ocp-async-7,70aa77c8411a7585,ee2c77541eef2b7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:35.983 INFO 21062 --- [prometheus-minute-schedule-1,dca8b0e4a11960c7,7ab061737cd4e521] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:36.711 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:36.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:36.894 WARN 21062 --- [ocp-async-1,a982cf7275039abd,f7467a9c77dd6abb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:36.943 INFO 21062 --- [ocp-async-4,9e8ac6e143cc17eb,87806be81870783c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:38.504 INFO 21062 --- [ocp-schedules-17,241d9e804d6bfda1,4c5aaeb3388fd99c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:33:38.504 INFO 21062 --- [ocp-schedules-13,e537fdbadaf5f1f1,e9ff6afab957e060] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:33:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:33:38.589728+08:00, lastProcessTime=2024-11-04T13:33:28.588829+08:00 2024-11-04 13:33:38.604 INFO 21062 --- [ocp-async-2,a456a30969d4b279,cbf6d0bcb2412bf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:33:38.743 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:33:38.589728+08:00 2024-11-04 13:33:38.894 WARN 21062 --- [ocp-async-6,219902e13de2cc20,dcb835b342a8df68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:38.943 INFO 21062 --- [ocp-async-5,280a51fc971f712e,5848acdb94076d38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:38.983 INFO 21062 --- [prometheus-minute-schedule-1,d5e8039724c226d9,779720eac1ec9d43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:39.575 INFO 21062 --- [ocp-schedules-6,79ee3c575efec22a,90e618d55089cf90] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:33:39.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:39.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:40.002 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:33:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:33:40.894 WARN 21062 --- [ocp-async-9,6b001341ef888689,577831680d2611c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:40.943 INFO 21062 --- [ocp-async-1,9d7a55052bed92cd,748e5af3b9beb0bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2129447ab2bf8fbd,a78712493364ef70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:42.361 INFO 21062 --- [ocp-schedules-28,03f5bf395970302d,90ce51caae0a81de] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:33:42.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:42.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:42.895 WARN 21062 --- [ocp-async-2,9ddbeefe6e9b035c,65646dbe457afe1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:42.943 INFO 21062 --- [ocp-async-3,49962bfa509d05aa,3d0562426de51930] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:44.895 WARN 21062 --- [ocp-async-8,5322bb12118c294c,5a85d5ae366ae14a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:44.944 INFO 21062 --- [ocp-async-10,b38e7498d255ff6d,acdfa52492e33541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:44.983 INFO 21062 --- [prometheus-minute-schedule-1,fea137e48c43f5c0,e456ac11428522b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:45.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:45.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:46.895 WARN 21062 --- [ocp-async-5,03f718c648efc09e,7417b7cae8b26149] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:46.944 INFO 21062 --- [ocp-async-7,4adb9528fd8eac3b,098f2645e20c7670] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:47.983 INFO 21062 --- [prometheus-minute-schedule-1,68437cb5954b7c2f,5986cac5bc38a29d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:33:48.593413+08:00, lastProcessTime=2024-11-04T13:33:38.589728+08:00 2024-11-04 13:33:48.604 INFO 21062 --- [ocp-async-5,fb35f15225d220e5,1a41cf75814b893d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:33:48.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:48.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:48.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:33:48.593413+08:00 2024-11-04 13:33:48.895 WARN 21062 --- [ocp-async-1,050a427220a266ca,446404f75deb0fc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:48.944 INFO 21062 --- [ocp-async-3,69b3873567c7478d,5ab2cbfd7a4b199b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:49.578 INFO 21062 --- [ocp-schedules-7,1c7d796b00fc277a,215d7a8d38e382b8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:33:50.200 INFO 21062 --- [ocp-schedules-29,d442da76a0cbf123,8e059ee25ce414e5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:33:50.206 INFO 21062 --- [ocp-schedules-29,d442da76a0cbf123,8e059ee25ce414e5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:33:50.206 INFO 21062 --- [ocp-schedules-29,d442da76a0cbf123,8e059ee25ce414e5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:33:50.895 WARN 21062 --- [ocp-async-3,ae04bbb55408d595,95c5125fc1e498d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:50.944 INFO 21062 --- [ocp-async-10,cf05ce30df8c1369,b18f65e1293fd3eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:50.983 INFO 21062 --- [prometheus-minute-schedule-1,457b8476b314b5e3,9bb15dd76ad81b27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:51.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:51.717 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:52.407 INFO 21062 --- [ocp-schedules-12,5ad2b467794485dc,9f7ecda440558488] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:33:52.896 WARN 21062 --- [ocp-async-10,3ba803bcda9ea046,edb1a04905fc39d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:52.944 INFO 21062 --- [ocp-async-7,b4f545a799d9c80e,a6106ac7867991fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4269126d64bb93c2,0e486d65b428d554] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:54.717 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:54.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:54.896 WARN 21062 --- [ocp-async-7,9f74e17f8eec0bba,5dc7852d3aa76370] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:54.944 INFO 21062 --- [ocp-async-4,cc949c6897aad488,8cf09f074190996e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:56.896 WARN 21062 --- [ocp-async-4,036a1d9befca8f70,e40814e490d20fc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:56.945 INFO 21062 --- [ocp-async-6,d969305baaaeab28,fd495fb685d0c7ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d3134c684de60908,fcc4b263bace6fbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:33:57.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:33:57.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:33:58.500 INFO 21062 --- [ocp-schedules-1,72dcd2e2c2fe8eb4,6c3a3eb511474709] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:33:58.500 INFO 21062 --- [ocp-schedules-19,a5f048b6401e3610,ce1328798af9be5c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:33:58.503 INFO 21062 --- [ocp-schedules-29,0b48160497b31d5f,0b855bec27f9de5d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:33:58.504 INFO 21062 --- [ocp-schedules-1,72dcd2e2c2fe8eb4,6c3a3eb511474709] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:33:58.504 INFO 21062 --- [ocp-schedules-19,a5f048b6401e3610,ce1328798af9be5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:33:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:33:58.505 INFO 21062 --- [ocp-schedules-20,8497bc4d99499b77,65ec68c20e4ebe7a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:33:58.511 INFO 21062 --- [ocp-schedules-30,c2702e985fad90e3,a52a54d92e8e5fc9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:33:58.512 INFO 21062 --- [ocp-schedules-26,cade8bc4ab17d14f,aaf7a270fce5fe20] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:33:58.512 INFO 21062 --- [ocp-schedules-30,c2702e985fad90e3,a52a54d92e8e5fc9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:33:58.512 INFO 21062 --- [ocp-schedules-26,cade8bc4ab17d14f,aaf7a270fce5fe20] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:33:58.512 INFO 21062 --- [ocp-schedules-25,9dd59657a787d277,7b6dbbabbfa66c11] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:33:58.512 INFO 21062 --- [ocp-schedules-25,9dd59657a787d277,7b6dbbabbfa66c11] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:33:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:33:58.588805+08:00, lastProcessTime=2024-11-04T13:33:48.593413+08:00 2024-11-04 13:33:58.604 INFO 21062 --- [ocp-async-4,3d0f3aaf5c17e8b6,5e6285a4588ece50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:33:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:33:58.588805+08:00 2024-11-04 13:33:58.893 INFO 21062 --- [ocp-schedules-6,e631c8ac94f225cc,e5e119dbe4e84f27] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:33:58.896 WARN 21062 --- [ocp-async-10,bdf3e76a006eeb49,e0e1c99ad908a069] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:33:58.945 INFO 21062 --- [ocp-async-7,3eed94d08467f300,f49763906d67bdcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:33:59.581 INFO 21062 --- [ocp-schedules-11,456e77796d3a15a5,dcfd96f2aff1cf5f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:33:59.983 INFO 21062 --- [prometheus-minute-schedule-1,4296e39b6a3d39d0,afd4e27af9264f65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698440, interval=60 2024-11-04 13:34:00.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:00.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:00.896 WARN 21062 --- [ocp-async-7,643e8724632c6c13,be789d46dab6db04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:00.945 INFO 21062 --- [ocp-async-3,deb9628b7c3dfdd1,3e6fdc4da39bc2cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:02.443 INFO 21062 --- [ocp-schedules-9,4bc917b5be2ff578,1e4f528c0b75046c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:34:02.718 INFO 21062 --- [ocp-schedules-10,6864175201e564ea,7b9398a571085df6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:34:02.718 INFO 21062 --- [ocp-schedules-10,e25880e9d9eda8df,35dc8ede4771c52c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:34:02.721 INFO 21062 --- [ocp-schedules-10,e25880e9d9eda8df,35dc8ede4771c52c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:02.750 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:34:02.750 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:34:02.897 WARN 21062 --- [ocp-async-3,111f17010b8aae08,6c44f9744df18555] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:02.945 INFO 21062 --- [ocp-async-6,b4f0e6d9ebb68dff,c6cec04b2e3c51b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7b76bbd744216d98,27d4c3c18b3a6514] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:03.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:03.721 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:04.897 WARN 21062 --- [ocp-async-6,2b821f32ca1bc931,fd311734f5798a86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:04.946 INFO 21062 --- [ocp-async-9,245fb4b41feaa6df,5759fc4b8681fc1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0095ffc2074881fe,882251c4e6b43490] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:06.722 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:06.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:06.897 WARN 21062 --- [ocp-async-9,0e96f06cd051afe8,7ef928be79cd4b39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:06.946 INFO 21062 --- [ocp-async-1,a0f1cdbbc8b07f3e,c1198ecd3a9d9d07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:08.505 INFO 21062 --- [ocp-schedules-19,a5f048b6401e3610,ce1328798af9be5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:34:08.505 INFO 21062 --- [ocp-schedules-1,72dcd2e2c2fe8eb4,6c3a3eb511474709] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:34:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:34:08.589912+08:00, lastProcessTime=2024-11-04T13:33:58.588805+08:00 2024-11-04 13:34:08.604 INFO 21062 --- [ocp-async-9,6a77a50bdd0b8aef,63640764ff3b4b51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:34:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:34:08.589912+08:00 2024-11-04 13:34:08.813 INFO 21062 --- [ocp-schedules-24,6d52d87060789b7c,f2522a23bd3df170] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:34:08.816 INFO 21062 --- [ocp-schedules-24,6d52d87060789b7c,f2522a23bd3df170] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:08.816 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:34:08.897 WARN 21062 --- [ocp-async-3,add7633754639dbe,f37efa0a0221cada] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:08.946 INFO 21062 --- [ocp-async-6,b54a18a47a059cd7,e1a62b93edfbb0e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:08.983 INFO 21062 --- [prometheus-minute-schedule-1,41df1d4083bcf094,0f2bcc57fbb2463f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:09.584 INFO 21062 --- [ocp-schedules-27,7d401a353aabac5c,55c2b08f11857efa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:34:09.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:09.724 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:34:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:34:10.211 INFO 21062 --- [ocp-schedules-9,cda0d20cc20b8e56,4937041fcd81671c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:34:10.216 INFO 21062 --- [ocp-schedules-9,cda0d20cc20b8e56,4937041fcd81671c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:34:10.216 INFO 21062 --- [ocp-schedules-9,cda0d20cc20b8e56,4937041fcd81671c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:34:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:34:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:34:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:34:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:34:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:34:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:34:10.897 WARN 21062 --- [ocp-async-6,0f1ad9a149723ba1,0556a9a66ad8d2cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:10.946 INFO 21062 --- [ocp-async-9,fda3419cf7a453cb,f68959f685b1b619] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698380, nodeCount=49, affectRows=49 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698380, nodeCount=79, affectRows=79 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698380, nodeCount=54, affectRows=54 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698380, nodeCount=64, affectRows=64 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698380, nodeCount=76, affectRows=76 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698380, nodeCount=48, validCount=48 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698380, nodeCount=78, validCount=78 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698380, nodeCount=53, validCount=53 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698380, nodeCount=64, validCount=64 2024-11-04 13:34:11.280 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698380, nodeCount=76, validCount=76 2024-11-04 13:34:11.983 INFO 21062 --- [prometheus-minute-schedule-1,89b558ccda7a3fea,6cc52f1106e71882] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:12.517 INFO 21062 --- [ocp-schedules-25,c3ffe84cc9a9e343,62a7e2493acfa97e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:34:12.724 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:12.724 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:12.898 WARN 21062 --- [ocp-async-9,8e32c55c9982a203,0c80f5b7c0b8eb28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:12.946 INFO 21062 --- [ocp-async-1,3fe7f5c4a4bbcc46,89ac27d154f8c23b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:13.984 INFO 21062 --- [prometheus-cache-monitor1,9749fcc05fde8ca8,41fcc4b6bdbd8009] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:34:13.984 INFO 21062 --- [prometheus-cache-monitor1,9749fcc05fde8ca8,41fcc4b6bdbd8009] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:34:14.898 WARN 21062 --- [ocp-async-1,2165e4dafc3b9ce5,f82f5d901173ead9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:14.947 INFO 21062 --- [ocp-async-8,df34b0af06488d0d,1ff480b932b10dc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9ed52e636edaa803,8f31ea7730a16d1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:15.724 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:15.726 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:16.898 WARN 21062 --- [ocp-async-8,6ab06db94417a9be,682f549e0f4e06ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:16.947 INFO 21062 --- [ocp-async-5,1a5b31749a1e54ff,adf21660adee8175] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:17.983 INFO 21062 --- [prometheus-minute-schedule-1,b9cc43ab0c62a1eb,86fafa3ada421fbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:34:18.591540+08:00, lastProcessTime=2024-11-04T13:34:08.589912+08:00 2024-11-04 13:34:18.605 INFO 21062 --- [ocp-async-8,41ddddbf1e103d4c,4ce3d2cc437385a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:34:18.726 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:18.727 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:18.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:34:18.591540+08:00 2024-11-04 13:34:18.817 INFO 21062 --- [ocp-schedules-24,6d52d87060789b7c,f2522a23bd3df170] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:34:18.898 WARN 21062 --- [ocp-async-9,e15fcae41cecdbed,57d85db74214e13b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:18.947 INFO 21062 --- [ocp-async-2,5c3f71d0a66cdf3e,79d5371f7d31837c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:19.588 INFO 21062 --- [ocp-schedules-3,c33e3f55975fd4ce,d13b07b45f9e519e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:34:20.898 WARN 21062 --- [ocp-async-2,3313aae6223b7dfe,b55419c74e073220] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:20.947 INFO 21062 --- [ocp-async-4,5bf449f5f3ee1bff,2818e87988f9c4de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:20.983 INFO 21062 --- [prometheus-minute-schedule-1,6b0fd8bd6ac9daee,9ecfe5104e616aac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:21.727 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:21.728 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:22.544 INFO 21062 --- [ocp-schedules-9,456336665c7366ec,4aefc0323ed31ba7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:34:22.899 WARN 21062 --- [ocp-async-4,7d7b755a7c8e8f78,6d6fbcd40d2087b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:22.947 INFO 21062 --- [ocp-async-10,c2426e1ae6810f0b,7f8482b666c4afcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ed8d70799724da68,0072b1cf5da39871] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:24.728 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:24.729 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:24.899 WARN 21062 --- [ocp-async-10,c4f7df3315c0b6f3,235d9d2c41903aa8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:24.947 INFO 21062 --- [ocp-async-7,de2a8edfe9418831,6ee2eee5c4de0e0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:26.899 WARN 21062 --- [ocp-async-7,0b7f8ab5033d379d,b090cb39d5f6ca6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:26.947 INFO 21062 --- [ocp-async-3,132a90c1e3de269b,5ce95ded06f0afea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:26.983 INFO 21062 --- [prometheus-minute-schedule-1,cb130619e167f996,5d7e63d8c05083e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:26.984 INFO 21062 --- [prometheus-hour-schedule-1,03d4f75e81243763,02f8b99d1fb2a1d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:34:27.730 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:27.730 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:28.498 INFO 21062 --- [ocp-schedules-20,3d72009ff5d3da8d,fbb2044f856aadeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:34:28.499 INFO 21062 --- [ocp-schedules-2,f3a77bb88d6617cf,fdc1852ac8c6b0cd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:34:28.500 INFO 21062 --- [ocp-schedules-17,2cd679cce2ff1057,7141f5e7e20834ea] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:34:28.500 INFO 21062 --- [ocp-schedules-8,ba6ed5653f8cb645,5e737c26d53c4ce3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:34:28.502 INFO 21062 --- [ocp-schedules-20,3d72009ff5d3da8d,fbb2044f856aadeb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:34:28.503 INFO 21062 --- [ocp-schedules-23,db155f98de8ae41e,66a78fbd60748a47] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:34:28.505 INFO 21062 --- [ocp-schedules-17,2cd679cce2ff1057,7141f5e7e20834ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:28.505 INFO 21062 --- [ocp-schedules-8,ba6ed5653f8cb645,5e737c26d53c4ce3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:28.505 INFO 21062 --- [ocp-schedules-22,8d694ce97dfa5a21,da995abf1718feb4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:34:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:34:28.506 INFO 21062 --- [ocp-schedules-2,f3a77bb88d6617cf,fdc1852ac8c6b0cd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:34:28.512 INFO 21062 --- [ocp-schedules-11,d481a243f64688e3,10bf54a8c540813c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:34:28.512 INFO 21062 --- [ocp-schedules-11,d481a243f64688e3,10bf54a8c540813c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:34:28.512 INFO 21062 --- [ocp-schedules-25,6505d21de2ff2681,487c1ee7434adea3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:34:28.512 INFO 21062 --- [ocp-schedules-25,6505d21de2ff2681,487c1ee7434adea3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:34:28.512 INFO 21062 --- [ocp-schedules-21,6ef6dd7995c7c930,181630fb0a5c2186] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:34:28.512 INFO 21062 --- [ocp-schedules-21,6ef6dd7995c7c930,181630fb0a5c2186] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:34:28.558 INFO 21062 --- [ocp-schedules-27,2d2c45c4ecdb3bdf,1a2f6ba00333dd4f] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:34:28.558+08:00, tenantSize=0 2024-11-04 13:34:28.567 INFO 21062 --- [ocp-schedules-28,3a75bef32d8242cb,856bbb09bb52c407] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:34:28.567 INFO 21062 --- [ocp-schedules-28,3a75bef32d8242cb,856bbb09bb52c407] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:34:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:34:28.589294+08:00, lastProcessTime=2024-11-04T13:34:18.591540+08:00 2024-11-04 13:34:28.605 INFO 21062 --- [ocp-async-3,29f342b0ef40c2a9,a17b4ffac9d7da88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:34:28.618 INFO 21062 --- [ocp-schedules-24,9f31489c30446d0a,ac8aea8dadb2fc86] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 13:34:28.618 INFO 21062 --- [ocp-schedules-24,9f31489c30446d0a,ac8aea8dadb2fc86] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 13:34:28.651 INFO 21062 --- [ocp-schedules-12,17f0fc46ef3211fd,26542d50c8260e2f] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:34:28.651 INFO 21062 --- [ocp-schedules-12,17f0fc46ef3211fd,26542d50c8260e2f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:34:28.709 INFO 21062 --- [ocp-schedules-18,64829eac3ce59a39,21efbf51513a76c4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:34:28.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:34:28.589294+08:00 2024-11-04 13:34:28.873 INFO 21062 --- [ocp-schedules-5,8d9f3e6c2deb2c65,20c493377f349d12] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:34:28.873 INFO 21062 --- [ocp-schedules-5,8d9f3e6c2deb2c65,20c493377f349d12] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:34:28.895 INFO 21062 --- [ocp-schedules-11,9f19379c9c217db6,721839eb11565d7e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:34:28.899 WARN 21062 --- [ocp-async-5,390ec58f4960b83c,9aafc4b73c89ba60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:28.925 INFO 21062 --- [ocp-schedules-20,31973ddac4bca702,ef1c36c30908c014] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:34:28.931 INFO 21062 --- [ocp-schedules-20,31973ddac4bca702,ef1c36c30908c014] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:34:28.931 INFO 21062 --- [ocp-schedules-20,31973ddac4bca702,ef1c36c30908c014] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:34:28.948 INFO 21062 --- [ocp-async-9,5e780b33bd0381b3,42ed7a11e631f909] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:28.970 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:34:28.970 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:34:28.978 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:34:28.978 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:34:28.978 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:34:28.989 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:34:28.989 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:34:28.989 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:34:28.992 INFO 21062 --- [ocp-schedules-9,698ac27ad5711bc7,5194c0597ba15e85] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:34:29.119 INFO 21062 --- [ocp-schedules-19,1de259e31378b036,ba9098b144461c63] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:34:29.119 INFO 21062 --- [ocp-schedules-19,1de259e31378b036,ba9098b144461c63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:34:29.120 INFO 21062 --- [ocp-schedules-19,1de259e31378b036,ba9098b144461c63] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:34:29.120 INFO 21062 --- [ocp-schedules-19,1de259e31378b036,ba9098b144461c63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:34:29.121 INFO 21062 --- [ocp-schedules-19,1de259e31378b036,ba9098b144461c63] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:34:29.121 INFO 21062 --- [ocp-schedules-19,1de259e31378b036,ba9098b144461c63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:34:29.592 INFO 21062 --- [ocp-schedules-30,4674d1ffe5a19d4a,f562ef486d7cd52d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:34:29.983 INFO 21062 --- [prometheus-minute-schedule-1,f98a4d4ef614ed7f,21ec6c56986bbad7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:30.219 INFO 21062 --- [ocp-schedules-5,3292106605334b50,709b82da8b9203d6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:34:30.222 INFO 21062 --- [ocp-schedules-27,72b64702bcb57728,a0d0b3bf051a8304] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:34:30.222 INFO 21062 --- [ocp-schedules-27,72b64702bcb57728,a0d0b3bf051a8304] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:34:30.226 INFO 21062 --- [ocp-schedules-5,3292106605334b50,709b82da8b9203d6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:34:30.226 INFO 21062 --- [ocp-schedules-5,3292106605334b50,709b82da8b9203d6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:34:30.249 INFO 21062 --- [ocp-schedules-29,1634bbaf58502ba0,9bc046bb3668b0c4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:34:30.249 INFO 21062 --- [ocp-schedules-27,72b64702bcb57728,a0d0b3bf051a8304] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:34:30.249 INFO 21062 --- [ocp-schedules-27,72b64702bcb57728,a0d0b3bf051a8304] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:34:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:34:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:34:30.382 INFO 21062 --- [ocp-schedules-28,d0a30a6a7b5aef75,b461b65ecc62a9dc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:34:30.382+08:00, clusters=[1] 2024-11-04 13:34:30.384 INFO 21062 --- [ocp-schedules-28,d0a30a6a7b5aef75,b461b65ecc62a9dc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:34:30.409 INFO 21062 --- [ocp-schedules-3,aeb9c6bff4c2b2b5,4f642999c949435d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:34:30.409 INFO 21062 --- [ocp-schedules-3,aeb9c6bff4c2b2b5,4f642999c949435d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:34:30.409 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:34:30.409 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:34:30.411 INFO 21062 --- [ocp-schedules-24,9413526d160c2284,d2274c5f9e8be297] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:34:30.415 INFO 21062 --- [ocp-schedules-24,9413526d160c2284,d2274c5f9e8be297] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:30.419 INFO 21062 --- [ocp-schedules-3,aeb9c6bff4c2b2b5,4f642999c949435d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:34:30.419 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:34:30.419 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:34:30.420 WARN 21062 --- [ocp-schedules-3,aeb9c6bff4c2b2b5,4f642999c949435d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:34:30.420 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:34:30.421 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:34:30.421 INFO 21062 --- [ocp-schedules-3,aeb9c6bff4c2b2b5,4f642999c949435d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:34:30.421 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:34:30.421 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:34:30.422 WARN 21062 --- [ocp-schedules-3,aeb9c6bff4c2b2b5,4f642999c949435d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:34:30.422 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:34:30.423 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:34:30.434 ERROR 21062 --- [ocp-schedules-3,aeb9c6bff4c2b2b5,4f642999c949435d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:34:30.476 ERROR 21062 --- [ocp-schedules-24,9413526d160c2284,d2274c5f9e8be297] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:34:30.476 INFO 21062 --- [ocp-schedules-24,9413526d160c2284,d2274c5f9e8be297] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:34:30.514 INFO 21062 --- [ocp-schedules-14,0e1574e42eaac595,b140b7733f285572] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:34:30.514+08:00, clusters=[1] 2024-11-04 13:34:30.518 INFO 21062 --- [ocp-schedules-14,0e1574e42eaac595,b140b7733f285572] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:34:30.731 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:30.731 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:30.899 WARN 21062 --- [ocp-async-7,013ef954bcdd57cc,45eb7931c33274ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:30.948 INFO 21062 --- [ocp-async-2,4c338067c88d179f,473ba1be6a1cd4e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:32.583 INFO 21062 --- [ocp-schedules-7,05b4ae6808adb306,d25ff26957bd9377] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:34:32.750 INFO 21062 --- [ocp-schedules-10,e25880e9d9eda8df,35dc8ede4771c52c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:34:32.751 INFO 21062 --- [ocp-schedules-10,f5a6c293212a8888,1310298e0c0f431f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:34:32.755 INFO 21062 --- [ocp-schedules-10,f5a6c293212a8888,1310298e0c0f431f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:32.784 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:34:32.784 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:34:32.900 WARN 21062 --- [ocp-async-3,ad4521e2a32ecf0d,54b1ca2b1ac542eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:32.948 INFO 21062 --- [ocp-async-4,2fb5f6c54f03dd3e,f32869275344ae84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:32.983 INFO 21062 --- [prometheus-minute-schedule-1,1a414ae9fbe72034,cd7c45480a4a8f51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:33.732 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:33.732 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:34.222 INFO 21062 --- [ocp-schedules-21,54a6091483499f17,1f81131ed8d3b946] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:34:34.227 INFO 21062 --- [ocp-schedules-21,54a6091483499f17,1f81131ed8d3b946] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:34.313 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:34:34.316 ERROR 21062 --- [ocp-schedules-21,54a6091483499f17,1f81131ed8d3b946] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:34:34.316 INFO 21062 --- [ocp-schedules-21,54a6091483499f17,1f81131ed8d3b946] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:34:34.900 WARN 21062 --- [ocp-async-6,ee4476e86d15b2af,1cb5aab05919da30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:34.948 INFO 21062 --- [ocp-async-10,119235bec855c469,51d1a95896878401] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:35.983 INFO 21062 --- [prometheus-minute-schedule-1,175087630317f496,48c654c48c704154] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:36.733 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:36.733 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:36.900 WARN 21062 --- [ocp-async-9,1634bbf095d041bb,5875602b8e8d50b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:36.948 INFO 21062 --- [ocp-async-1,f1485105c3504166,140fd7a162e33823] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:38.506 INFO 21062 --- [ocp-schedules-8,ba6ed5653f8cb645,5e737c26d53c4ce3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:34:38.506 INFO 21062 --- [ocp-schedules-17,2cd679cce2ff1057,7141f5e7e20834ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:34:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:34:38.589488+08:00, lastProcessTime=2024-11-04T13:34:28.589294+08:00 2024-11-04 13:34:38.605 INFO 21062 --- [ocp-async-7,65104e61705a8cc5,1636a7c53edb4db5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:34:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:34:38.589488+08:00 2024-11-04 13:34:38.900 WARN 21062 --- [ocp-async-8,639bcceafc08e3e3,8df11a97f701643b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:38.949 INFO 21062 --- [ocp-async-6,52f3acffef091837,4c28bba836947784] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:38.983 INFO 21062 --- [prometheus-minute-schedule-1,369ef80aa8a7b861,da83f35bd777836f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:39.595 INFO 21062 --- [ocp-schedules-30,4f1d6769415cd3a6,a88b7af66c1f8a55] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:34:39.734 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:39.734 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:34:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:34:40.536 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:34:40.536 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:34:40.538 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.539 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:34:40.540 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.540 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.541 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.541 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:34:40.557 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.558 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.558 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.559 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:34:40.571 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.572 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.572 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.573 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:34:40.586 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.587 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.587 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.588 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:34:40.600 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.601 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.601 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.602 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:34:40.608 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.608 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.609 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.609 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:34:40.613 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.613 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.614 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.614 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:34:40.619 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.620 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.621 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] ob connector is evicted from holder's cache. cause=EXPIRED, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=10.46.128.24, port=2881, database=oceanbase), size=3 2024-11-04 13:34:40.621 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.621 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-3} closing ... 2024-11-04 13:34:40.621 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:34:40.628 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.629 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.629 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-3} closed 2024-11-04 13:34:40.629 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.630 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:34:40.635 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.636 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.636 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.636 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:34:40.642 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.642 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.643 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.643 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:34:40.651 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.651 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.652 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.652 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:34:40.665 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.665 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.666 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.666 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:34:40.671 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.672 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.672 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.672 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:34:40.678 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.678 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.679 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.679 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:34:40.686 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.687 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.687 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.688 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:34:40.693 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.693 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.694 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.694 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:34:40.698 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.699 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.699 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.700 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:34:40.703 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.704 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.704 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.704 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:34:40.709 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.709 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.710 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.710 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:34:40.714 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.715 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.715 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.716 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:34:40.721 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.721 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.722 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.722 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:34:40.727 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.727 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.728 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.728 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:34:40.732 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.733 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.733 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.734 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:34:40.738 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.738 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.738 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.739 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:34:40.743 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.743 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.744 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.744 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:34:40.749 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.749 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.750 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.750 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:34:40.754 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.755 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.755 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.756 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:34:40.760 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.760 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.761 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.761 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:34:40.765 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.766 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.767 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.767 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:34:40.771 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.771 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.772 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.773 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:34:40.777 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.777 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.778 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.778 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:34:40.783 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.783 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.784 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.784 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:34:40.788 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.788 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.789 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.789 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:34:40.797 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.797 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.798 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.798 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:34:40.804 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.805 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.805 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.806 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:34:40.811 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.811 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.812 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.812 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:34:40.817 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.818 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.818 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.819 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:34:40.822 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.823 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.823 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.824 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:34:40.827 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.828 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.828 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.829 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:34:40.832 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.832 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.833 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.833 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:34:40.837 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.837 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.838 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.838 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:34:40.847 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.847 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.848 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.848 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:34:40.868 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.868 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.869 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.869 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:34:40.874 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.875 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.875 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.875 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:34:40.881 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.881 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.882 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.882 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:34:40.888 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.888 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.889 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.889 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:34:40.893 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.894 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.894 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.895 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:34:40.898 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.899 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.899 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.900 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:34:40.900 WARN 21062 --- [ocp-async-5,d54b74c448326172,5680eb41528530bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:40.904 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:34:40.904 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:34:40.905 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.906 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:34:40.907 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.908 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.908 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.909 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:34:40.915 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.915 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:34:40.916 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:40.916 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:34:40.920 INFO 21062 --- [ocp-schedules-17,54a76225e6d33468,04f3052b033e28ed] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:34:40.949 INFO 21062 --- [ocp-async-9,e4043b1f1bd0d8d0,75be09d018933770] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:34:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:34:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:34:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:34:41.853 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:34:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:34:41.983 INFO 21062 --- [prometheus-minute-schedule-1,07d5346a1d4e53aa,289efb9164a82b66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:42.611 INFO 21062 --- [ocp-schedules-25,677f60ca0415b473,5152dc55338a1ff7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:34:42.735 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:42.736 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:42.900 WARN 21062 --- [ocp-async-7,097f910305e99fc4,e0b774a1cb890999] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:42.949 INFO 21062 --- [ocp-async-2,90ade39c075df94c,161d42fcf1d7deae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:44.900 WARN 21062 --- [ocp-async-3,982fefc89ca290e7,7dbefd5dadabf34b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:44.949 INFO 21062 --- [ocp-async-4,67b3f9b7671ecacb,f72e5b5e0da8f78b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e35356159c497630,c7442d1e1e3c46fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:45.736 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:45.737 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:46.901 WARN 21062 --- [ocp-async-6,2f0ef4150bfd4542,34f377996d98df02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:46.949 INFO 21062 --- [ocp-async-10,886a3ace55feeaaf,8bb68da48639164d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:47.983 INFO 21062 --- [prometheus-minute-schedule-1,511dccd0b9849420,3bc209acddd5ff0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:34:48.590389+08:00, lastProcessTime=2024-11-04T13:34:38.589488+08:00 2024-11-04 13:34:48.605 INFO 21062 --- [ocp-async-6,fbdb206f94ae0e1f,35e064d2135041b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:34:48.737 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:48.738 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:48.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:34:48.590389+08:00 2024-11-04 13:34:48.901 WARN 21062 --- [ocp-async-7,d69429d721f3ef06,06b7ac543e59379d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:48.949 INFO 21062 --- [ocp-async-2,b1430ab89acc8146,9969fc3ba4f2cae3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:49.598 INFO 21062 --- [ocp-schedules-11,154d27467b4b3e19,4ce59cebccc3e013] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:34:50.228 INFO 21062 --- [ocp-schedules-26,1f9a21faf3689df3,eeb30a7d21f86b49] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:34:50.234 INFO 21062 --- [ocp-schedules-26,1f9a21faf3689df3,eeb30a7d21f86b49] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:34:50.234 INFO 21062 --- [ocp-schedules-26,1f9a21faf3689df3,eeb30a7d21f86b49] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:34:50.901 WARN 21062 --- [ocp-async-2,40baf4ef0f45b7b8,d5068f31a08fa220] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:50.950 INFO 21062 --- [ocp-async-4,47c604167814f38a,fbc2cac13df312bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:50.983 INFO 21062 --- [prometheus-minute-schedule-1,54acce341e61f364,8d8b044418964551] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:51.738 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:51.739 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:52.653 INFO 21062 --- [ocp-schedules-1,91e44c5e435deafd,5e40ceb7aaa265de] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:34:52.901 WARN 21062 --- [ocp-async-4,95e21a359aa288d5,97613662c7b0239d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:52.950 INFO 21062 --- [ocp-async-10,0291af8f970f5733,8f1d791526ddd499] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e8c53c9498170fd6,9231d33d8a346fb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:54.739 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:54.740 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:54.901 WARN 21062 --- [ocp-async-10,48276f7739dd76bd,9995190ce5c2c2ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:54.950 INFO 21062 --- [ocp-async-1,e12272dd822ededd,3476decd0dc5a402] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:56.901 WARN 21062 --- [ocp-async-1,1fa43c9bbf274d61,450ad05ba9e6ea3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:56.950 INFO 21062 --- [ocp-async-8,680b67fa1e7e0b32,448dbecf68719b4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:56.983 INFO 21062 --- [prometheus-minute-schedule-1,2c7543270cf6b070,6922d2974b91e16d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:34:57.740 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:34:57.741 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:34:58.500 INFO 21062 --- [ocp-schedules-17,ca57178f60068d4c,736a903bb6c2f598] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:34:58.500 INFO 21062 --- [ocp-schedules-30,bacd0f31a95db2f4,2c8b89ce731d4f32] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:34:58.502 INFO 21062 --- [ocp-schedules-23,ac138d41989ce65e,9475f8ff94e2f4bc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:34:58.504 INFO 21062 --- [ocp-schedules-17,ca57178f60068d4c,736a903bb6c2f598] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:58.504 INFO 21062 --- [ocp-schedules-30,bacd0f31a95db2f4,2c8b89ce731d4f32] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:34:58.504 INFO 21062 --- [ocp-schedules-4,37068710c643ecf5,a23619b611958991] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:34:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:34:58.510 INFO 21062 --- [ocp-schedules-24,c617e902b02fa7d0,41a152d1925273b1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:34:58.511 INFO 21062 --- [ocp-schedules-7,c4404ff4a37fb650,3ead630ca3800dc2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:34:58.511 INFO 21062 --- [ocp-schedules-24,c617e902b02fa7d0,41a152d1925273b1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:34:58.511 INFO 21062 --- [ocp-schedules-8,c455f97782ee9c1d,98950a6e3fb7def2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:34:58.511 INFO 21062 --- [ocp-schedules-7,c4404ff4a37fb650,3ead630ca3800dc2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:34:58.511 INFO 21062 --- [ocp-schedules-8,c455f97782ee9c1d,98950a6e3fb7def2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:34:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:34:58.589501+08:00, lastProcessTime=2024-11-04T13:34:48.590389+08:00 2024-11-04 13:34:58.605 INFO 21062 --- [ocp-async-1,393d452cec272512,de345a907dbc2752] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:34:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:34:58.589501+08:00 2024-11-04 13:34:58.898 INFO 21062 --- [ocp-schedules-2,7c9a6907d9ed6b29,ac660941b9610f69] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:34:58.902 WARN 21062 --- [ocp-async-4,ee3f3cd25f23e9b7,f680922c8023343c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:34:58.950 INFO 21062 --- [ocp-async-10,ea6d0d239d0a30aa,6c37e9502b2354fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:34:59.602 INFO 21062 --- [ocp-schedules-20,0df5e2089250400a,726ea746f3191209] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:34:59.983 INFO 21062 --- [prometheus-minute-schedule-1,b5f89047137c930b,ce36df7c234c7d7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698500, interval=60 2024-11-04 13:35:00.741 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:00.742 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:00.876 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,abebbcace0eef1a9] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:30:00.0, nextRun=2024-11-04 13:35:00.0, currentDate=Mon Nov 04 13:35:00 CST 2024 2024-11-04 13:35:00.876 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,abebbcace0eef1a9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:30:00.0, nextRunTime=2024-11-04 13:35:00.0} 2024-11-04 13:35:00.902 WARN 21062 --- [ocp-async-10,dcb31ac49b60f370,fae4165b8a25a5f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:00.904 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,abebbcace0eef1a9] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:35:00.910 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,abebbcace0eef1a9] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:35:00.910 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,abebbcace0eef1a9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:35:00.951 INFO 21062 --- [ocp-async-1,308a815e08efa25a,049fa0e7642fcaac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:02.679 INFO 21062 --- [ocp-schedules-29,a8ad8b4d22acd54d,649de8aa32e81d93] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:35:02.785 INFO 21062 --- [ocp-schedules-10,f5a6c293212a8888,1310298e0c0f431f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:35:02.785 INFO 21062 --- [ocp-schedules-10,3ac6abe876aaf2f4,4f36b03063b20a02] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:35:02.789 INFO 21062 --- [ocp-schedules-10,3ac6abe876aaf2f4,4f36b03063b20a02] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:02.817 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:35:02.817 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:35:02.902 WARN 21062 --- [ocp-async-1,1634ca141923024d,421c574f38e32136] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:02.951 INFO 21062 --- [ocp-async-8,363edb1d1a9caaa1,9e22e46e8590259b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:02.983 INFO 21062 --- [prometheus-minute-schedule-1,60fdaee604544afe,ab6ec851c0a77f75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:03.742 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:03.743 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:04.902 WARN 21062 --- [ocp-async-8,440cf05d1e406615,f7a1a1c6c78165ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:04.951 INFO 21062 --- [ocp-async-5,18638e9cca1684f3,4605c64bd605385f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:05.983 INFO 21062 --- [prometheus-minute-schedule-1,52a84664098ca939,6fa7e59884635bbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:06.744 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:06.744 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:06.902 WARN 21062 --- [ocp-async-5,a2c1df78dcaeba4f,b611d74876bacb09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:06.951 INFO 21062 --- [ocp-async-9,c240ce1b36887c29,6f3613b133a4f5d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:08.504 INFO 21062 --- [ocp-schedules-30,bacd0f31a95db2f4,2c8b89ce731d4f32] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:35:08.504 INFO 21062 --- [ocp-schedules-17,ca57178f60068d4c,736a903bb6c2f598] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:35:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:35:08.590349+08:00, lastProcessTime=2024-11-04T13:34:58.589501+08:00 2024-11-04 13:35:08.606 INFO 21062 --- [ocp-async-5,5612fe765a3a550c,6d66adfd04ae4bbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:35:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:35:08.590349+08:00 2024-11-04 13:35:08.903 WARN 21062 --- [ocp-async-1,a019dfc916d6a488,faeb5c92b5e2f7fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:08.951 INFO 21062 --- [ocp-async-8,f288fc3da66733ab,cbadf3d1d08ad941] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:08.983 INFO 21062 --- [prometheus-minute-schedule-1,3c199557864d7a36,7504d549f0817406] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:09.604 INFO 21062 --- [ocp-schedules-22,88ecebe34534cf54,e04eae635cd18489] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:35:09.745 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:09.745 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:35:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:35:10.237 INFO 21062 --- [ocp-schedules-1,01c5df7fc10bb89d,1f6459125184b8d5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:35:10.242 INFO 21062 --- [ocp-schedules-1,01c5df7fc10bb89d,1f6459125184b8d5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:35:10.242 INFO 21062 --- [ocp-schedules-1,01c5df7fc10bb89d,1f6459125184b8d5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:35:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:35:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:35:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:35:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:35:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:35:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:35:10.903 WARN 21062 --- [ocp-async-8,e4c81c52bc812c55,e2562af66c92972e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:10.951 INFO 21062 --- [ocp-async-5,32c2904673567921,b2db1fea61dcce33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:11.285 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=6, affectRows=6 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=6, validCount=6 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=8, affectRows=8 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=8, validCount=8 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=22, affectRows=22 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=32, affectRows=32 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=8, affectRows=8 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=28, affectRows=28 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=28, affectRows=28 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=23, affectRows=23 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=44, affectRows=44 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=8, validCount=8 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=21, validCount=21 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=28, validCount=28 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=23, validCount=23 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=32, validCount=32 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=28, validCount=28 2024-11-04 13:35:11.286 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=44, validCount=44 2024-11-04 13:35:11.287 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=64, affectRows=64 2024-11-04 13:35:11.287 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698440, nodeCount=59, affectRows=59 2024-11-04 13:35:11.287 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=58, validCount=58 2024-11-04 13:35:11.287 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698440, nodeCount=63, validCount=63 2024-11-04 13:35:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4fb858c39e871c62,5ef17a05a1fb3551] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:12.719 INFO 21062 --- [ocp-schedules-17,3a6ebb11995d93d9,3a49a9c4edcd154f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:35:12.746 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:12.746 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:12.903 WARN 21062 --- [ocp-async-5,795d5e5f6dbb1df0,be109783b2eb8d2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:12.952 INFO 21062 --- [ocp-async-9,d954aa719067e0fe,1cf7766dd9d9f2b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:13.984 INFO 21062 --- [prometheus-cache-monitor1,e1bac071a0c0277a,c9de151048f10d14] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:35:13.985 INFO 21062 --- [prometheus-cache-monitor1,e1bac071a0c0277a,c9de151048f10d14] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:35:14.608 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:31:14.591, end=2024-11-04T13:33:14.606} 2024-11-04 13:35:14.612 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:35:14.612 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:35:14.903 WARN 21062 --- [ocp-async-9,db7741a4d7101bce,a972033765c2ad83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:14.952 INFO 21062 --- [ocp-async-2,198b8e1c9d6cf31f,89eefbf8b15b8128] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:14.983 INFO 21062 --- [prometheus-minute-schedule-1,0b8bbcb571d3bbf4,07d3bc81871e2503] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:15.746 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:15.747 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:16.903 WARN 21062 --- [ocp-async-2,f494f1df23396e4a,b8529700c0dff3bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:16.952 INFO 21062 --- [ocp-async-6,761a2a379798ee25,78ca01d885ace9bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:17.983 INFO 21062 --- [prometheus-minute-schedule-1,b312b4b1a69315e0,b6018c5a52418267] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:35:18.589926+08:00, lastProcessTime=2024-11-04T13:35:08.590349+08:00 2024-11-04 13:35:18.606 INFO 21062 --- [ocp-async-2,1f759f21d729dbdc,bf568830e91523bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:35:18.747 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:18.748 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:35:18.589926+08:00 2024-11-04 13:35:18.817 INFO 21062 --- [ocp-schedules-4,e8e9e9dac42121e1,bc7b13b71abe1cf6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:35:18.821 INFO 21062 --- [ocp-schedules-4,e8e9e9dac42121e1,bc7b13b71abe1cf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:18.821 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:35:18.903 WARN 21062 --- [ocp-async-5,e1e183e1c5445f1a,681ab20be2365a56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:18.952 INFO 21062 --- [ocp-async-9,a34b8f543fd229b6,f1c791f820337350] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:19.608 INFO 21062 --- [ocp-schedules-20,809fec935ad6ef89,a0b14797e2307859] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:35:20.904 WARN 21062 --- [ocp-async-9,6b5a679aee5b6019,dd5f9f1f5b893e01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:20.952 INFO 21062 --- [ocp-async-3,36d07719f25c5a4c,fc336295f79ad917] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:20.983 INFO 21062 --- [prometheus-minute-schedule-1,016375c564d3f40c,e6e1580bed3e55a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:21.748 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:21.749 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:22.746 INFO 21062 --- [ocp-schedules-6,5fb736b6504de3c5,7218c98e0f4aa0b1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:35:22.904 WARN 21062 --- [ocp-async-3,b41059001728c525,b3a94402b5d4e480] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:22.953 INFO 21062 --- [ocp-async-4,471d87c3de01cb5b,2878211f2d31db54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b0aaa7f8cf5e948a,17ed886a9cb6cf6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:24.749 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:24.750 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:24.904 WARN 21062 --- [ocp-async-4,36914b0caf16c823,132dfad1e5639053] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:24.953 INFO 21062 --- [ocp-async-10,4d0243a89dc900ea,3bb5377db32e8e6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:26.904 WARN 21062 --- [ocp-async-10,f1db1ad9ff14ca1c,424240ef74f7fd89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:26.953 INFO 21062 --- [ocp-async-1,f1f3064399a14b3b,a70955808a137f05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:26.983 INFO 21062 --- [prometheus-minute-schedule-1,d2ca4a990dcadb39,4b05524ff9fbec06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:26.984 INFO 21062 --- [prometheus-hour-schedule-1,65c228f84a9b79db,0135dc4ed610da42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:35:27.750 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:27.751 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:28.498 INFO 21062 --- [ocp-schedules-30,0d7f8503263fbb81,4db83237498fd6b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:35:28.499 INFO 21062 --- [ocp-schedules-5,bae094fe4c3cc693,179c92a3dd3bb42b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:35:28.500 INFO 21062 --- [ocp-schedules-1,19cfe7741b0ffe31,8480894b49a4c898] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:35:28.500 INFO 21062 --- [ocp-schedules-12,a76ed7f6f28e27c0,a1ffe71ec7cdab8d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:35:28.502 INFO 21062 --- [ocp-schedules-30,0d7f8503263fbb81,4db83237498fd6b0] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:35:28.502 INFO 21062 --- [ocp-schedules-29,b43e819579a31e86,cafb2a76ab482826] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:35:28.503 INFO 21062 --- [ocp-schedules-1,19cfe7741b0ffe31,8480894b49a4c898] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:28.503 INFO 21062 --- [ocp-schedules-12,a76ed7f6f28e27c0,a1ffe71ec7cdab8d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:35:28.504 INFO 21062 --- [ocp-schedules-18,7a77ccf84773828e,e12ee89b599ab9f7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:35:28.505 INFO 21062 --- [ocp-schedules-5,bae094fe4c3cc693,179c92a3dd3bb42b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:35:28.511 INFO 21062 --- [ocp-schedules-17,8fb5ac258918fdfe,e6ab812de72c551c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:35:28.511 INFO 21062 --- [ocp-schedules-11,d2ff1649b5ff1589,ba7f45fcec51a6e5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:35:28.511 INFO 21062 --- [ocp-schedules-17,8fb5ac258918fdfe,e6ab812de72c551c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:35:28.511 INFO 21062 --- [ocp-schedules-11,d2ff1649b5ff1589,ba7f45fcec51a6e5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:35:28.511 INFO 21062 --- [ocp-schedules-21,e96687f2937ccc65,80f363dc41d07bcc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:35:28.511 INFO 21062 --- [ocp-schedules-21,e96687f2937ccc65,80f363dc41d07bcc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:35:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:35:28.589297+08:00, lastProcessTime=2024-11-04T13:35:18.589926+08:00 2024-11-04 13:35:28.606 INFO 21062 --- [ocp-async-1,21acd5024d526aed,dcdf03cd28dcfb92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:35:28.711 INFO 21062 --- [ocp-schedules-16,ae28e08893717ebe,b1c31ac538fda9af] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:35:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:35:28.589297+08:00 2024-11-04 13:35:28.822 INFO 21062 --- [ocp-schedules-4,e8e9e9dac42121e1,bc7b13b71abe1cf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:35:28.877 INFO 21062 --- [ocp-schedules-25,0290475bcfcdceb6,186d9362030fdd63] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:35:28.877 INFO 21062 --- [ocp-schedules-25,0290475bcfcdceb6,186d9362030fdd63] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:35:28.901 INFO 21062 --- [ocp-schedules-13,5695accfde1ed96f,a120eaaa704e46d5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:35:28.904 WARN 21062 --- [ocp-async-6,07b30b8172185738,33b00693a98654c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:28.931 INFO 21062 --- [ocp-schedules-17,8b23a328be98665c,61894ca31c96940a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:35:28.937 INFO 21062 --- [ocp-schedules-17,8b23a328be98665c,61894ca31c96940a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:35:28.937 INFO 21062 --- [ocp-schedules-17,8b23a328be98665c,61894ca31c96940a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:35:28.953 INFO 21062 --- [ocp-async-5,cac367308fd3e641,b222309540a09b21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:29.122 INFO 21062 --- [ocp-schedules-6,40d39161f51dccb7,26c1e88c13b390bb] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:35:29.122 INFO 21062 --- [ocp-schedules-6,40d39161f51dccb7,26c1e88c13b390bb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:35:29.125 INFO 21062 --- [ocp-schedules-6,40d39161f51dccb7,26c1e88c13b390bb] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:35:29.125 INFO 21062 --- [ocp-schedules-6,40d39161f51dccb7,26c1e88c13b390bb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:35:29.125 INFO 21062 --- [ocp-schedules-6,40d39161f51dccb7,26c1e88c13b390bb] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:35:29.125 INFO 21062 --- [ocp-schedules-6,40d39161f51dccb7,26c1e88c13b390bb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:35:29.612 INFO 21062 --- [ocp-schedules-23,4d9f8e691e57b403,99f2fdda39734781] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:35:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e15b7b3ae37cb70d,a707fe9b2ca977fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:30.245 INFO 21062 --- [ocp-schedules-9,d231f231468bf425,fbe902666e7535d5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:35:30.249 INFO 21062 --- [ocp-schedules-21,3cace01f1b71c401,9be94653a11864b5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:35:30.249 INFO 21062 --- [ocp-schedules-21,3cace01f1b71c401,9be94653a11864b5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:35:30.252 INFO 21062 --- [ocp-schedules-9,d231f231468bf425,fbe902666e7535d5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:35:30.252 INFO 21062 --- [ocp-schedules-9,d231f231468bf425,fbe902666e7535d5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:35:30.277 INFO 21062 --- [ocp-schedules-21,3cace01f1b71c401,9be94653a11864b5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:35:30.277 INFO 21062 --- [ocp-schedules-21,3cace01f1b71c401,9be94653a11864b5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:35:30.277 INFO 21062 --- [ocp-schedules-17,f627b12c977d913b,704c455e0a4cba5f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:35:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:35:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:35:30.408 INFO 21062 --- [ocp-schedules-11,4fb629d5d369f796,b4e81ec31d3bbbcb] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:35:30.408+08:00, clusters=[1] 2024-11-04 13:35:30.409 INFO 21062 --- [ocp-schedules-11,4fb629d5d369f796,b4e81ec31d3bbbcb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:35:30.442 INFO 21062 --- [ocp-schedules-6,aeeef47a5b130352,3b2b879580caeec3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:35:30.442 INFO 21062 --- [ocp-schedules-6,aeeef47a5b130352,3b2b879580caeec3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:35:30.442 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:35:30.442 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:35:30.452 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:35:30.452 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:35:30.453 INFO 21062 --- [ocp-schedules-6,aeeef47a5b130352,3b2b879580caeec3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:35:30.454 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:35:30.454 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:35:30.454 WARN 21062 --- [ocp-schedules-6,aeeef47a5b130352,3b2b879580caeec3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:35:30.455 INFO 21062 --- [ocp-schedules-6,aeeef47a5b130352,3b2b879580caeec3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:35:30.455 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:35:30.455 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:35:30.456 WARN 21062 --- [ocp-schedules-6,aeeef47a5b130352,3b2b879580caeec3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:35:30.456 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:35:30.456 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:35:30.464 ERROR 21062 --- [ocp-schedules-6,aeeef47a5b130352,3b2b879580caeec3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:35:30.544 INFO 21062 --- [ocp-schedules-24,a165abcd12c5c743,9de1baf26974aeb9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:35:30.544+08:00, clusters=[1] 2024-11-04 13:35:30.547 INFO 21062 --- [ocp-schedules-24,a165abcd12c5c743,9de1baf26974aeb9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:35:30.751 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:30.752 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:30.905 WARN 21062 --- [ocp-async-10,19dc7d6fca131d57,a87a1bc10a8cb8c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:30.953 INFO 21062 --- [ocp-async-9,28bdd4848f6cdb25,b3464d7ba55ae6a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:32.817 INFO 21062 --- [ocp-schedules-10,3ac6abe876aaf2f4,4f36b03063b20a02] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:35:32.818 INFO 21062 --- [ocp-schedules-10,6c313a873343a491,e40ddccb516a8280] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:35:32.822 INFO 21062 --- [ocp-schedules-10,6c313a873343a491,e40ddccb516a8280] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:32.828 INFO 21062 --- [ocp-schedules-30,6518f1a9f8193baa,bf0aae633b541f82] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:35:32.851 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:35:32.851 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:35:32.905 WARN 21062 --- [ocp-async-1,f1f7b4724625cee6,ea58cde65ca44dbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:32.953 INFO 21062 --- [ocp-async-3,70dc315e3dea599a,436dad25afba4d79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:32.983 INFO 21062 --- [prometheus-minute-schedule-1,35a2c4499d0a7f4a,e38c04250d90cb3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:33.752 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:33.753 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:34.316 INFO 21062 --- [ocp-schedules-6,3e08231da36cc36c,b6cc9019588ea572] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:35:34.320 INFO 21062 --- [ocp-schedules-6,3e08231da36cc36c,b6cc9019588ea572] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:34.404 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:35:34.406 ERROR 21062 --- [ocp-schedules-6,3e08231da36cc36c,b6cc9019588ea572] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:35:34.406 INFO 21062 --- [ocp-schedules-6,3e08231da36cc36c,b6cc9019588ea572] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:35:34.905 WARN 21062 --- [ocp-async-8,010ed3fc646f8c61,ad779537699df33c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:34.954 INFO 21062 --- [ocp-async-4,00491eaa5ea7e0d4,0e89219006c63bbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:35.983 INFO 21062 --- [prometheus-minute-schedule-1,a10d02aaba3ed9c1,e41d2ee86f395cc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:36.753 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:36.755 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:36.905 WARN 21062 --- [ocp-async-5,f5d00ac7a766bac8,d654a030d6e6f018] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:36.954 INFO 21062 --- [ocp-async-7,a24c90bedbad6abd,181453952690b559] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:38.504 INFO 21062 --- [ocp-schedules-12,a76ed7f6f28e27c0,a1ffe71ec7cdab8d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:35:38.504 INFO 21062 --- [ocp-schedules-1,19cfe7741b0ffe31,8480894b49a4c898] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:35:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:35:38.589673+08:00, lastProcessTime=2024-11-04T13:35:28.589297+08:00 2024-11-04 13:35:38.606 INFO 21062 --- [ocp-async-10,9121783500b58864,4fbbd92caebfd26e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:35:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:35:38.589673+08:00 2024-11-04 13:35:38.905 WARN 21062 --- [ocp-async-2,6a415a963eb0b038,aa69e927a158e3d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:38.954 INFO 21062 --- [ocp-async-8,04f6fe78bb82c177,a24fb2113614ab05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:38.983 INFO 21062 --- [prometheus-minute-schedule-1,7ba53399eb28fbff,177931c4b9d3f9fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:39.615 INFO 21062 --- [ocp-schedules-27,c81af60cf3e0f68a,cf161cb05a2ac6fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:35:39.755 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:39.758 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:35:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:35:40.905 WARN 21062 --- [ocp-async-6,2fe0acb3e7033824,636ed40f948d5b40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:40.954 INFO 21062 --- [ocp-async-5,76700922b4027a8b,4f8677d1abbca672] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:41.983 INFO 21062 --- [prometheus-minute-schedule-1,9e169c2b67dbbac7,ec2cc48b08d1ac0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:42.758 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:42.759 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:42.852 INFO 21062 --- [ocp-schedules-30,546ddcffa476b09d,3713d68c031d605e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:35:42.906 WARN 21062 --- [ocp-async-10,842748fd75aec145,3540d933d95a31b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:42.954 INFO 21062 --- [ocp-async-9,a8c1993624a7c753,6f8563df1c3c910d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:44.908 WARN 21062 --- [ocp-async-1,178881410cc3be5a,a1cdf6e9fe2a8221] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:44.954 INFO 21062 --- [ocp-async-3,733d4945e2109e42,a25a85998b7a0776] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:44.983 INFO 21062 --- [prometheus-minute-schedule-1,042e0ce68a89e766,ddcecf6dee096a16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:45.759 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:45.760 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:46.906 WARN 21062 --- [ocp-async-8,bc0cc32a9bb5855c,fa59ce019c254d9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:46.955 INFO 21062 --- [ocp-async-4,48225878a028a092,d8803ae9fda328b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e65ebb1cd7562c47,46e522092e1fd1d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:35:48.590450+08:00, lastProcessTime=2024-11-04T13:35:38.589673+08:00 2024-11-04 13:35:48.607 INFO 21062 --- [ocp-async-8,2132bbc8eb2363bd,86f59622b618b2b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:35:48.761 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:48.761 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:35:48.590450+08:00 2024-11-04 13:35:48.906 WARN 21062 --- [ocp-async-10,c1017594621becec,a5863dc316f9fb7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:48.955 INFO 21062 --- [ocp-async-9,4d95561834f34bab,b1d19fd6fc84d1d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:49.617 INFO 21062 --- [ocp-schedules-4,0a662002c5254c3f,e8c0160453dbc875] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:35:50.254 INFO 21062 --- [ocp-schedules-19,cd4e1baa2650f417,b8b59dd13a174c75] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:35:50.260 INFO 21062 --- [ocp-schedules-19,cd4e1baa2650f417,b8b59dd13a174c75] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:35:50.260 INFO 21062 --- [ocp-schedules-19,cd4e1baa2650f417,b8b59dd13a174c75] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:35:50.906 WARN 21062 --- [ocp-async-9,2f36582421e148a9,f5619a311e9f3934] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:50.955 INFO 21062 --- [ocp-async-3,a08be0eff16ff24b,d6e771a36ab3657d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:50.983 INFO 21062 --- [prometheus-minute-schedule-1,5698cbb625253ac0,43c94f1d47ae457e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:51.761 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:51.762 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:52.896 INFO 21062 --- [ocp-schedules-14,006faa2ee560df09,a6e771490a38b752] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:35:52.906 WARN 21062 --- [ocp-async-3,4e733180e018db2d,d08bb7002e6d2f5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:52.955 INFO 21062 --- [ocp-async-4,87d69042988ac27f,71a87d44ffbecdde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1149b7aeba23ef54,0e8c86b4f79cec2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:54.763 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:54.764 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:54.907 WARN 21062 --- [ocp-async-4,28037e4fa684bb33,79d45162ca8a73e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:54.955 INFO 21062 --- [ocp-async-7,157e582f8f163d5a,b39f60424f1618a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:56.907 WARN 21062 --- [ocp-async-7,82d98fa3490287b5,22d33fa116e42596] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:56.955 INFO 21062 --- [ocp-async-2,4d9f40a68e2d341b,dcf597b7a6006ef6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f6a9a50a33a0c950,00d875b207d3847a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:35:57.764 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:35:57.765 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:35:58.500 INFO 21062 --- [ocp-schedules-17,2681acc1c2968862,fba09c9ddf35b333] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:35:58.500 INFO 21062 --- [ocp-schedules-23,bd85ca3001d69de8,3ebd73d773302f56] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:35:58.502 INFO 21062 --- [ocp-schedules-6,d1ff6b5557f6a63d,05847d6c06bb585d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:35:58.504 INFO 21062 --- [ocp-schedules-23,bd85ca3001d69de8,3ebd73d773302f56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:58.504 INFO 21062 --- [ocp-schedules-17,2681acc1c2968862,fba09c9ddf35b333] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:35:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:35:58.504 INFO 21062 --- [ocp-schedules-24,e8753e8d2e049fb7,9d3b141ba10f3c90] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:35:58.511 INFO 21062 --- [ocp-schedules-12,58732a4e559af100,d42eeac328bf45d6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:35:58.511 INFO 21062 --- [ocp-schedules-27,ea0358830afe66de,1be312890a1ef8c4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:35:58.511 INFO 21062 --- [ocp-schedules-27,ea0358830afe66de,1be312890a1ef8c4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:35:58.511 INFO 21062 --- [ocp-schedules-12,58732a4e559af100,d42eeac328bf45d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:35:58.511 INFO 21062 --- [ocp-schedules-30,714828c3ca17f5ad,47933a8a53b73ce9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:35:58.511 INFO 21062 --- [ocp-schedules-30,714828c3ca17f5ad,47933a8a53b73ce9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:35:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:35:58.589925+08:00, lastProcessTime=2024-11-04T13:35:48.590450+08:00 2024-11-04 13:35:58.607 INFO 21062 --- [ocp-async-7,d11b19e201ac00f7,933ac3597d1c419a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:35:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:35:58.589925+08:00 2024-11-04 13:35:58.904 INFO 21062 --- [ocp-schedules-5,da78a986ecc656ae,03a1e9abcd16418d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:35:58.907 WARN 21062 --- [ocp-async-3,7d5d78769c78dbd4,c6d6ced196ad1fff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:35:58.956 INFO 21062 --- [ocp-async-4,c452665e498fde0c,b4d5ac1c4b866472] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:35:59.620 INFO 21062 --- [ocp-schedules-27,cc2f6ec10440146e,eb9ad38a27079881] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:35:59.983 INFO 21062 --- [prometheus-minute-schedule-1,40a471755c0d927d,29ae2b5e0edf63a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698560, interval=60 2024-11-04 13:36:00.765 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:00.767 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:00.907 WARN 21062 --- [ocp-async-4,a0611f4da615f7d0,8498cefb25dc8e50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:00.956 INFO 21062 --- [ocp-async-9,dac7f8314d997f02,e7af9598996c4b4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:02.852 INFO 21062 --- [ocp-schedules-10,6c313a873343a491,e40ddccb516a8280] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:36:02.852 INFO 21062 --- [ocp-schedules-10,6ac66b05f0dc49d7,1bbd33dedfb716d9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:36:02.856 INFO 21062 --- [ocp-schedules-10,6ac66b05f0dc49d7,1bbd33dedfb716d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:02.886 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:36:02.886 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:36:02.907 WARN 21062 --- [ocp-async-9,e272302645851628,5ac77774d6e80c4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:02.924 INFO 21062 --- [ocp-schedules-11,0e64279fbcdf5a9e,58965a0d1f3af510] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:36:02.956 INFO 21062 --- [ocp-async-2,578ff8263f2aa386,6f3dfcc4360a402b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2aeaa4482fb5e272,f8f9f488ba8903ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:03.767 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:03.768 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:04.908 WARN 21062 --- [ocp-async-2,0a063837f2a7885d,4c157245f284142f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:04.956 INFO 21062 --- [ocp-async-8,4fa3a0dcee63d4a8,7564f99f26cb635d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:05.983 INFO 21062 --- [prometheus-minute-schedule-1,89392fd7051ed6b1,f810966bdca9db67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:06.768 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:06.769 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:06.908 WARN 21062 --- [ocp-async-8,7d44b6a12f1f0e61,ba52fb287bcbd50c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:06.956 INFO 21062 --- [ocp-async-10,cefc93c502d00f5e,b022fcc860371df3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:08.504 INFO 21062 --- [ocp-schedules-23,bd85ca3001d69de8,3ebd73d773302f56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:36:08.505 INFO 21062 --- [ocp-schedules-17,2681acc1c2968862,fba09c9ddf35b333] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:36:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:36:08.588673+08:00, lastProcessTime=2024-11-04T13:35:58.589925+08:00 2024-11-04 13:36:08.607 INFO 21062 --- [ocp-async-8,76fd7508f9e19803,909c0dea145ff878] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:36:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:36:08.588673+08:00 2024-11-04 13:36:08.908 WARN 21062 --- [ocp-async-9,ee66e0e38f171062,ffcd348a8ede22b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:08.956 INFO 21062 --- [ocp-async-2,b272ea8273b3e435,ce3f1b78baef9248] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8532953de6a3f744,024929b1236c4910] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:09.623 INFO 21062 --- [ocp-schedules-1,11d9579dd8e1f3bb,41a70cb6b30c229c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:36:09.769 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:09.771 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:10.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:36:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:36:10.263 INFO 21062 --- [ocp-schedules-18,9a146eea813bb55c,04b918941e9e8140] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:36:10.268 INFO 21062 --- [ocp-schedules-18,9a146eea813bb55c,04b918941e9e8140] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:36:10.268 INFO 21062 --- [ocp-schedules-18,9a146eea813bb55c,04b918941e9e8140] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:36:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:36:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:36:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:36:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:36:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:36:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:36:10.908 WARN 21062 --- [ocp-async-2,ff3a5d9ca8ef3713,3a428190478fb325] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:10.957 INFO 21062 --- [ocp-async-8,4bc30f81534fa13a,5bca98085e086033] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:10.997 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:36:10.997 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:36:11.004 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:36:11.004 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:36:11.004 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:36:11.005 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:36:11.010 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:36:11.010 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:36:11.010 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:36:11.010 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:36:11.046 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:36:11.054 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:36:11.054 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:36:11.054 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:36:11.054 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:36:11.054 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:36:11.063 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:36:11.289 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=2, affectRows=2 2024-11-04 13:36:11.289 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=1, validCount=1 2024-11-04 13:36:11.290 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=9, affectRows=9 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=9, validCount=9 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=10, affectRows=10 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=19, affectRows=19 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=38, affectRows=38 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=27, affectRows=27 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=10, validCount=10 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=19, validCount=19 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=27, validCount=27 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=38, validCount=38 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=69, affectRows=69 2024-11-04 13:36:11.291 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=60, affectRows=60 2024-11-04 13:36:11.292 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=60, validCount=60 2024-11-04 13:36:11.292 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698500, nodeCount=88, affectRows=88 2024-11-04 13:36:11.292 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=69, validCount=69 2024-11-04 13:36:11.292 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698500, nodeCount=86, validCount=86 2024-11-04 13:36:11.603 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:36:11.983 INFO 21062 --- [prometheus-minute-schedule-1,02f67760c18c8c60,b4c5658ca1e7e00b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:12.210 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.200033204S 2024-11-04 13:36:12.211 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:36:12.309 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:36:12.309 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:36:12.337 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:36:12.394 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:36:12.394 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:36:12.394 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:36:12.394 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:36:12.394 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:36:12.394 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:36:12.405 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:36:12.581 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] o.s.boot.SpringApplication : Started application in 1.576 seconds (JVM running for 3986.26) 2024-11-04 13:36:12.581 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.576604583S 2024-11-04 13:36:12.771 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:12.772 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:12.908 WARN 21062 --- [ocp-async-8,a9b46c53b562bb7c,d3016bb718d5f3c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:12.957 INFO 21062 --- [ocp-async-10,706c8ece81e26fbd,f7e701806f625de4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:12.963 INFO 21062 --- [ocp-schedules-6,404069fedb3a737d,51379b63336c424a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:36:13.044 INFO 21062 --- [ocp-schedules-13,d43980776677c994,be42fb9ebb049bfa] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:36:13.984 INFO 21062 --- [prometheus-cache-monitor1,1414a483ff5cfde9,58e49e85f4509e98] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:36:13.985 INFO 21062 --- [prometheus-cache-monitor1,1414a483ff5cfde9,58e49e85f4509e98] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:36:14.908 WARN 21062 --- [ocp-async-10,076956394529a2f1,3738c37f86963f72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:14.957 INFO 21062 --- [ocp-async-1,0617098db5014d35,d88fa9e381b631a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3d0443aea4026d4d,6eea813a903dc4b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:15.772 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:15.774 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:16.909 WARN 21062 --- [ocp-async-1,1266099b981a926f,ead14fc187b520a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:16.957 INFO 21062 --- [ocp-async-6,69b74eb4eddb654e,9d71209cef075711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:17.983 INFO 21062 --- [prometheus-minute-schedule-1,37c59a329f8a43bf,e9cacaa67cd1778c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:36:18.589647+08:00, lastProcessTime=2024-11-04T13:36:08.588673+08:00 2024-11-04 13:36:18.607 INFO 21062 --- [ocp-async-1,a24a847f3368569a,2a570c15ff0f58df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:36:18.753 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:36:18.589647+08:00 2024-11-04 13:36:18.774 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:18.775 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:18.909 WARN 21062 --- [ocp-async-8,f12395416926da58,600613c09666be66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:18.957 INFO 21062 --- [ocp-async-10,767326c61dcaa07a,c9e497879c2cb49b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:19.626 INFO 21062 --- [ocp-schedules-29,45147e4a3069080d,9346043f8f15248f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:36:20.909 WARN 21062 --- [ocp-async-10,5038fc8dcb9ed0e5,b293307cdc188df3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:20.957 INFO 21062 --- [ocp-async-7,e3e366a874774e2f,5a64a47c9a9325e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:20.983 INFO 21062 --- [prometheus-minute-schedule-1,290cdd1f19ca5bd2,01fd6e9d6e80d1fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:21.775 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:21.776 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:22.909 WARN 21062 --- [ocp-async-7,6898680da5ea1f30,89dd35cf2b2cbac4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:22.958 INFO 21062 --- [ocp-async-3,3a6c7d39e3feaf8f,9f330325d7983bd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:22.988 INFO 21062 --- [ocp-schedules-25,faf2fbe8459eff77,b03b907d177a18f0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:36:23.983 INFO 21062 --- [prometheus-minute-schedule-1,31fdf0a62ff9c3dc,bef9916abdc1d049] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:24.776 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:24.777 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:24.909 WARN 21062 --- [ocp-async-3,bccefed89eac0f2a,5fd539da1bb5a107] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:24.958 INFO 21062 --- [ocp-async-4,4530f517e14eacc9,a37503e8bdf4bb00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:26.909 WARN 21062 --- [ocp-async-4,c6a63a8f3369934b,eca05cd09c5cb497] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:26.958 INFO 21062 --- [ocp-async-9,cb51b38c3764b3b2,d8dae978b23538f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:26.983 INFO 21062 --- [prometheus-minute-schedule-1,7c6c53b1d1460d08,d8607263dfee3149] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:26.984 INFO 21062 --- [prometheus-hour-schedule-1,d1b890546dec0a9a,4ed440fd5bf96499] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:36:27.777 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:27.778 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:28.498 INFO 21062 --- [ocp-schedules-21,56464de83fb2d2e0,625d885540456c9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:36:28.499 INFO 21062 --- [ocp-schedules-25,36d1a6c9ac47b7c7,f012cb9135743f3c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:36:28.500 INFO 21062 --- [ocp-schedules-2,0360eba52e25886b,e96385f816182ea9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:36:28.500 INFO 21062 --- [ocp-schedules-12,0921a0c5556e8305,34b1d44522206151] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:36:28.502 INFO 21062 --- [ocp-schedules-21,56464de83fb2d2e0,625d885540456c9d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:36:28.502 INFO 21062 --- [ocp-schedules-27,c0f144192b8f81cf,43ca99eca82cddbd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:36:28.503 INFO 21062 --- [ocp-schedules-2,0360eba52e25886b,e96385f816182ea9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:36:28.504 INFO 21062 --- [ocp-schedules-12,0921a0c5556e8305,34b1d44522206151] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:28.504 INFO 21062 --- [ocp-schedules-25,36d1a6c9ac47b7c7,f012cb9135743f3c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:36:28.504 INFO 21062 --- [ocp-schedules-7,1d683c9df8a6a290,3e326c76fad48242] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:36:28.512 INFO 21062 --- [ocp-schedules-28,0fd3625a884d1053,cd0aa66a55536860] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:36:28.512 INFO 21062 --- [ocp-schedules-19,3abd60164e323ed3,1da34439f9b110cb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:36:28.512 INFO 21062 --- [ocp-schedules-8,478005ec7a78fddb,385668c9e68d784e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:36:28.512 INFO 21062 --- [ocp-schedules-19,3abd60164e323ed3,1da34439f9b110cb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:36:28.512 INFO 21062 --- [ocp-schedules-8,478005ec7a78fddb,385668c9e68d784e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:36:28.512 INFO 21062 --- [ocp-schedules-28,0fd3625a884d1053,cd0aa66a55536860] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:36:28.567 INFO 21062 --- [ocp-schedules-30,6469b73f1ce5ea63,f4003c0c1ac21f27] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:36:28.567 INFO 21062 --- [ocp-schedules-30,6469b73f1ce5ea63,f4003c0c1ac21f27] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:36:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:36:28.588893+08:00, lastProcessTime=2024-11-04T13:36:18.589647+08:00 2024-11-04 13:36:28.602 INFO 21062 --- [ocp-schedules-1,49d03eef65672713,da04efb416832c0a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:36:28.602 INFO 21062 --- [ocp-schedules-1,49d03eef65672713,da04efb416832c0a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:36:28.607 INFO 21062 --- [ocp-async-1,1dd5957149035e94,b9213e182361aa78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:36:28.714 INFO 21062 --- [ocp-schedules-20,b13ba825909ad36f,89b9f8cc570586f5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:36:28.744 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:36:28.588893+08:00 2024-11-04 13:36:28.822 INFO 21062 --- [ocp-schedules-29,4a9eaccc4ffe11a3,41c401d3673a15b6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:36:28.825 INFO 21062 --- [ocp-schedules-29,4a9eaccc4ffe11a3,41c401d3673a15b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:28.826 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:36:28.881 INFO 21062 --- [ocp-schedules-25,fccdddf7f8f70b85,4730a2a3e56193c3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:36:28.881 INFO 21062 --- [ocp-schedules-25,fccdddf7f8f70b85,4730a2a3e56193c3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:36:28.906 INFO 21062 --- [ocp-schedules-27,4e12e91cba0e8637,25333830e51bca75] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:36:28.910 WARN 21062 --- [ocp-async-6,dbb164ba9274a367,476dba5de68fbff4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:28.937 INFO 21062 --- [ocp-schedules-28,70b60ba62a4f79cd,25834911a8647919] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:36:28.943 INFO 21062 --- [ocp-schedules-28,70b60ba62a4f79cd,25834911a8647919] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:36:28.943 INFO 21062 --- [ocp-schedules-28,70b60ba62a4f79cd,25834911a8647919] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:36:28.958 INFO 21062 --- [ocp-async-8,e5968a8e796c7ffa,7305685f73dc2fac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:29.126 INFO 21062 --- [ocp-schedules-4,c577961d69ce9fdc,78ff6feb247c8efd] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:36:29.126 INFO 21062 --- [ocp-schedules-4,c577961d69ce9fdc,78ff6feb247c8efd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:36:29.133 INFO 21062 --- [ocp-schedules-4,c577961d69ce9fdc,78ff6feb247c8efd] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:36:29.133 INFO 21062 --- [ocp-schedules-4,c577961d69ce9fdc,78ff6feb247c8efd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:36:29.139 INFO 21062 --- [ocp-schedules-4,c577961d69ce9fdc,78ff6feb247c8efd] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:36:29.139 INFO 21062 --- [ocp-schedules-4,c577961d69ce9fdc,78ff6feb247c8efd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:36:29.629 INFO 21062 --- [ocp-schedules-26,65834c7deee9d05a,f0e76cab8d2d8c27] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:36:29.983 INFO 21062 --- [prometheus-minute-schedule-1,bf5f0ab1628482d8,fbe015c442dcc7ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:30.271 INFO 21062 --- [ocp-schedules-3,67ab829a62d66f7a,ad37a4046fbf4551] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:36:30.277 INFO 21062 --- [ocp-schedules-3,67ab829a62d66f7a,ad37a4046fbf4551] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:36:30.277 INFO 21062 --- [ocp-schedules-3,67ab829a62d66f7a,ad37a4046fbf4551] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:36:30.277 INFO 21062 --- [ocp-schedules-3,11dba014cf008e53,dd78b487f9f9ed64] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:36:30.277 INFO 21062 --- [ocp-schedules-3,11dba014cf008e53,dd78b487f9f9ed64] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:36:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:36:30.304 INFO 21062 --- [ocp-schedules-3,11dba014cf008e53,dd78b487f9f9ed64] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:36:30.304 INFO 21062 --- [ocp-schedules-3,11dba014cf008e53,dd78b487f9f9ed64] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:36:30.305 INFO 21062 --- [ocp-schedules-28,39ed64ea0b4de695,4b910436b3d11c08] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:36:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:36:30.435 INFO 21062 --- [ocp-schedules-21,b7a55508a819be7c,aef1e20d54d24786] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:36:30.435+08:00, clusters=[1] 2024-11-04 13:36:30.437 INFO 21062 --- [ocp-schedules-21,b7a55508a819be7c,aef1e20d54d24786] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:36:30.472 INFO 21062 --- [ocp-schedules-16,21e1d382e08f3725,07178d4c46b4d4bd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:36:30.472 INFO 21062 --- [ocp-schedules-16,21e1d382e08f3725,07178d4c46b4d4bd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:36:30.472 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:36:30.472 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:36:30.476 INFO 21062 --- [ocp-schedules-11,406a43f23cdb436e,57a6235aa98c4cca] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:36:30.480 INFO 21062 --- [ocp-schedules-11,406a43f23cdb436e,57a6235aa98c4cca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:30.482 INFO 21062 --- [ocp-schedules-16,21e1d382e08f3725,07178d4c46b4d4bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:36:30.482 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:36:30.482 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:36:30.483 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:36:30.483 WARN 21062 --- [ocp-schedules-16,21e1d382e08f3725,07178d4c46b4d4bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:36:30.484 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:36:30.484 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:36:30.484 INFO 21062 --- [ocp-schedules-16,21e1d382e08f3725,07178d4c46b4d4bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:36:30.485 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:36:30.485 WARN 21062 --- [ocp-schedules-16,21e1d382e08f3725,07178d4c46b4d4bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:36:30.485 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:36:30.486 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:36:30.491 ERROR 21062 --- [ocp-schedules-16,21e1d382e08f3725,07178d4c46b4d4bd] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:36:30.538 ERROR 21062 --- [ocp-schedules-11,406a43f23cdb436e,57a6235aa98c4cca] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:36:30.538 INFO 21062 --- [ocp-schedules-11,406a43f23cdb436e,57a6235aa98c4cca] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:36:30.572 INFO 21062 --- [ocp-schedules-13,c65cecc577e51613,b6f6b90027e88ec7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:36:30.572+08:00, clusters=[1] 2024-11-04 13:36:30.575 INFO 21062 --- [ocp-schedules-13,c65cecc577e51613,b6f6b90027e88ec7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:36:30.778 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:30.779 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:30.910 WARN 21062 --- [ocp-async-4,2c21b5038b9c7494,2de7c48c0f0df925] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:30.958 INFO 21062 --- [ocp-async-10,db8e61e4cc2d8bb1,a9231c82945f8c26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:32.886 INFO 21062 --- [ocp-schedules-10,6ac66b05f0dc49d7,1bbd33dedfb716d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:36:32.886 INFO 21062 --- [ocp-schedules-10,0308606eea4170b4,5b7fc2bdc9130ef0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:36:32.890 INFO 21062 --- [ocp-schedules-10,0308606eea4170b4,5b7fc2bdc9130ef0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:32.910 WARN 21062 --- [ocp-async-1,66840cc8c8a8b8d5,8edf6c8d383ad3ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:32.919 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:36:32.919 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:36:32.958 INFO 21062 --- [ocp-async-7,f4e9ec48f96b756b,d3b2b62c6b6ceb45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:32.983 INFO 21062 --- [prometheus-minute-schedule-1,48573816d702ffa4,ce422878e17122f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:33.013 INFO 21062 --- [ocp-schedules-6,15b94ce3b5e9d627,40d9da7c6387a07a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:36:33.779 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:33.780 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:34.407 INFO 21062 --- [ocp-schedules-15,22ada37359ad4196,5dfc04d6c5157fe7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:36:34.410 INFO 21062 --- [ocp-schedules-15,22ada37359ad4196,5dfc04d6c5157fe7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:34.492 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:36:34.494 ERROR 21062 --- [ocp-schedules-15,22ada37359ad4196,5dfc04d6c5157fe7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:36:34.494 INFO 21062 --- [ocp-schedules-15,22ada37359ad4196,5dfc04d6c5157fe7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:36:34.910 WARN 21062 --- [ocp-async-2,f652aee3989fdd04,058688c83b25c652] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:34.959 INFO 21062 --- [ocp-async-3,d7da1e68a641676b,8383a6beb8c8d188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4ec528e1ed84f740,ae286597035bd3ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:36.780 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:36.781 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:36.910 WARN 21062 --- [ocp-async-8,f49f2f5a1d181720,f34c1e753b0893e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:36.959 INFO 21062 --- [ocp-async-5,2b16f5a61ff0c7b2,378b0568b2cd0774] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:38.504 INFO 21062 --- [ocp-schedules-2,0360eba52e25886b,e96385f816182ea9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:36:38.504 INFO 21062 --- [ocp-schedules-12,0921a0c5556e8305,34b1d44522206151] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:36:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:36:38.589521+08:00, lastProcessTime=2024-11-04T13:36:28.588893+08:00 2024-11-04 13:36:38.608 INFO 21062 --- [ocp-async-4,971a2d29575c9407,6627fdb311f7da80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:36:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:36:38.589521+08:00 2024-11-04 13:36:38.826 INFO 21062 --- [ocp-schedules-29,4a9eaccc4ffe11a3,41c401d3673a15b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:36:38.911 WARN 21062 --- [ocp-async-9,2753367892496263,c7ffcad2b69a8b99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:38.959 INFO 21062 --- [ocp-async-2,16421a49fdbb340a,71bc21af2f05aab0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5741e7028645867f,87ca247b8d657776] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:39.632 INFO 21062 --- [ocp-schedules-6,72a2324ac847d0e5,01655d8be0dc668f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:36:39.781 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:39.782 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:40.004 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:36:40.006 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:36:40.911 WARN 21062 --- [ocp-async-6,643cb6d7efdf6e03,fa299bdc3f58e42e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:40.959 INFO 21062 --- [ocp-async-8,fe0ecc4ce3ac3194,89f9783cecbfb720] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:36:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:36:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:36:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:36:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:36:41.854 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:36:41.854 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:36:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a89c4e56408874fe,f70831994823ab25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:42.782 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:42.783 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:42.911 WARN 21062 --- [ocp-async-4,ae03018366cc3b3f,c7ebc7692fc5d51e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:42.959 INFO 21062 --- [ocp-async-10,8e16d4b0de36a100,be888d10dd702791] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:43.038 INFO 21062 --- [ocp-schedules-12,041ed0188be2e453,384bf967f2a3b1dd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:36:44.911 WARN 21062 --- [ocp-async-1,9ac0f26f3005db5a,3d953f2dcd6fd73d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:44.960 INFO 21062 --- [ocp-async-7,4edb7e1ad56db5c7,0c70f590ba2c3b54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f61d8bbf5122796b,38e3323db2af1d01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:45.783 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:45.784 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:46.911 WARN 21062 --- [ocp-async-2,343533027c240952,e79577f898b98578] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:46.960 INFO 21062 --- [ocp-async-3,0c060e9ecd318830,124523df9cf8adf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:47.983 INFO 21062 --- [prometheus-minute-schedule-1,f2d6f88b8d1881e2,48364594c86b535b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:36:48.590103+08:00, lastProcessTime=2024-11-04T13:36:38.589521+08:00 2024-11-04 13:36:48.608 INFO 21062 --- [ocp-async-2,c2fd05b8d6e6d031,f2f550f808f5f267] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:36:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:36:48.590103+08:00 2024-11-04 13:36:48.784 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:48.785 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:48.911 WARN 21062 --- [ocp-async-4,a097d5bf29f3491d,7a059a8644122e4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:48.960 INFO 21062 --- [ocp-async-10,88e66c30cee07b62,930854e79af1e443] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:49.635 INFO 21062 --- [ocp-schedules-23,d63f85b0fb713def,566db75f35e8d63e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:36:50.281 INFO 21062 --- [ocp-schedules-8,b1de4af07b682262,23a79cc8f9a0699d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:36:50.287 INFO 21062 --- [ocp-schedules-8,b1de4af07b682262,23a79cc8f9a0699d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:36:50.287 INFO 21062 --- [ocp-schedules-8,b1de4af07b682262,23a79cc8f9a0699d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:36:50.912 WARN 21062 --- [ocp-async-1,694dcf911607be50,e82d048f5e5103ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:50.960 INFO 21062 --- [ocp-async-7,07a1fa35462a4322,d695327bd4ad2741] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:50.983 INFO 21062 --- [prometheus-minute-schedule-1,cc199ce1893ce026,cf1113416bff527e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:51.785 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:51.786 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:52.912 WARN 21062 --- [ocp-async-7,9ff6217ef45ce791,45bfbed22cc81448] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:52.960 INFO 21062 --- [ocp-async-3,98bee2a9abda55da,c74786cd001c52a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:53.082 INFO 21062 --- [ocp-schedules-19,b9631bd5c70abfa0,857d940ed16253a8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:36:53.983 INFO 21062 --- [prometheus-minute-schedule-1,76f315b27b65e185,d49ac06020ac600d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:54.786 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:54.786 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:54.912 WARN 21062 --- [ocp-async-3,59c5fb11914d234c,3f7ec903324b3bfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:54.960 INFO 21062 --- [ocp-async-5,fa0daa77fc25237f,24a47b280168ef18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:56.912 WARN 21062 --- [ocp-async-5,3c7b1eb59dc8a640,e5972988ac292a48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:56.961 INFO 21062 --- [ocp-async-9,084c9d16b8099d02,3bc29245cbc40365] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ea784bb696a12f31,24523b62acd5fd0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:36:57.787 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:36:57.787 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:36:58.500 INFO 21062 --- [ocp-schedules-11,50b53315c954382e,eb36805fd7efe29f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:36:58.500 INFO 21062 --- [ocp-schedules-30,83c8154a6a322494,57f8ace3f0a035f2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:36:58.502 INFO 21062 --- [ocp-schedules-8,9636096b9147693e,14abe2d76f9ec14c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:36:58.503 INFO 21062 --- [ocp-schedules-11,50b53315c954382e,eb36805fd7efe29f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:36:58.504 INFO 21062 --- [ocp-schedules-30,83c8154a6a322494,57f8ace3f0a035f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:36:58.505 INFO 21062 --- [ocp-schedules-2,fd66411355d68cd6,80a0758a17b86e13] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:36:58.509 INFO 21062 --- [ocp-schedules-4,789c6a835db446db,72e58b2b848be18e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:36:58.509 INFO 21062 --- [ocp-schedules-4,789c6a835db446db,72e58b2b848be18e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:36:58.510 INFO 21062 --- [ocp-schedules-28,5b60449c8586632b,3dbde74c7b7eac49] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:36:58.510 INFO 21062 --- [ocp-schedules-28,5b60449c8586632b,3dbde74c7b7eac49] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:36:58.510 INFO 21062 --- [ocp-schedules-26,019ebf659cef05f3,1b17c76c3d1c610c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:36:58.510 INFO 21062 --- [ocp-schedules-26,019ebf659cef05f3,1b17c76c3d1c610c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:36:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:36:58.588675+08:00, lastProcessTime=2024-11-04T13:36:48.590103+08:00 2024-11-04 13:36:58.608 INFO 21062 --- [ocp-async-5,f190f7999b4b30a9,f7468a41a7c1cb59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:36:58.754 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:36:58.588675+08:00 2024-11-04 13:36:58.909 INFO 21062 --- [ocp-schedules-6,e283444249146a40,48c629c8036eeab4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:36:58.912 WARN 21062 --- [ocp-async-7,d6d4adef8bd5eba2,2d3e0a1ae5ddd2cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:36:58.961 INFO 21062 --- [ocp-async-3,7ad72f83174ea264,465073c4c1eae28a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:36:59.638 INFO 21062 --- [ocp-schedules-26,6c428d7fbee6f216,74cf4c6dffe86359] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:36:59.983 INFO 21062 --- [prometheus-minute-schedule-1,03ba545acff6e7b3,93c2eaf60809c469] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698620, interval=60 2024-11-04 13:37:00.788 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:00.788 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:00.913 WARN 21062 --- [ocp-async-3,d333e426403ea0c8,cc23a96bc7a28c14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:00.961 INFO 21062 --- [ocp-async-5,845cca6c7acd0768,825ede12971ee17a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:02.913 WARN 21062 --- [ocp-async-5,938df1b7c8c98a4d,a84079eb3def14cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:02.920 INFO 21062 --- [ocp-schedules-10,0308606eea4170b4,5b7fc2bdc9130ef0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:37:02.920 INFO 21062 --- [ocp-schedules-10,6350dde9843f12d5,b3c2ec9f39615d20] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:37:02.924 INFO 21062 --- [ocp-schedules-10,6350dde9843f12d5,b3c2ec9f39615d20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:02.954 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:37:02.954 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:37:02.961 INFO 21062 --- [ocp-async-9,afde0d9aebd431c6,c3e0d38da4a1eeb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:02.983 INFO 21062 --- [prometheus-minute-schedule-1,3d0cf987f17fcbc6,588b5da3ecbb4a34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:03.108 INFO 21062 --- [ocp-schedules-15,32e951f327612bd4,a451bcae2708a659] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:37:03.788 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:03.789 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:04.913 WARN 21062 --- [ocp-async-9,88598debd5eab599,ab279adbdf08783e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:04.961 INFO 21062 --- [ocp-async-2,1f3c7492ab4979e6,2990412c527fa573] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:05.983 INFO 21062 --- [prometheus-minute-schedule-1,86cd5a9d079e31ba,f4be07c32bbfe4b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:06.789 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:06.790 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:06.913 WARN 21062 --- [ocp-async-2,1b4467aff0608d09,d0811eae2860e77c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:06.961 INFO 21062 --- [ocp-async-4,ad6961fcc1aab8b4,b06d3f580bddebad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:08.504 INFO 21062 --- [ocp-schedules-11,50b53315c954382e,eb36805fd7efe29f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:37:08.505 INFO 21062 --- [ocp-schedules-30,83c8154a6a322494,57f8ace3f0a035f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:37:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:37:08.590624+08:00, lastProcessTime=2024-11-04T13:36:58.588675+08:00 2024-11-04 13:37:08.608 INFO 21062 --- [ocp-async-2,d6e90796ffa83a0b,a95817086d1b8d35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:37:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:37:08.590624+08:00 2024-11-04 13:37:08.913 WARN 21062 --- [ocp-async-5,09003707ee9fd940,e1f1f845ba10d260] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:08.962 INFO 21062 --- [ocp-async-9,51a8e1f46e34ef78,b685b4a3a58179fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:08.983 INFO 21062 --- [prometheus-minute-schedule-1,1ce71d387b3af52a,3178e3196c48300b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:09.642 INFO 21062 --- [ocp-schedules-2,2809cd538a47e205,4dd91900efaadc17] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:37:09.790 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:09.791 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:10.002 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:37:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:37:10.290 INFO 21062 --- [ocp-schedules-15,cb6ba9cafbef5321,989f0c45067ef198] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:37:10.295 INFO 21062 --- [ocp-schedules-15,cb6ba9cafbef5321,989f0c45067ef198] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:37:10.295 INFO 21062 --- [ocp-schedules-15,cb6ba9cafbef5321,989f0c45067ef198] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:37:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:37:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:37:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:37:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:37:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:37:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:37:10.913 WARN 21062 --- [ocp-async-9,1b024c2cd617ddc9,7479984508dcc68d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:10.962 INFO 21062 --- [ocp-async-2,63cc8298944340ef,2f192eb7e4aeaa76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=4, affectRows=4 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=4, validCount=4 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=11, affectRows=11 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=22, affectRows=22 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=11, validCount=11 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=22, validCount=22 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=27, affectRows=27 2024-11-04 13:37:11.289 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=27, validCount=27 2024-11-04 13:37:11.290 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=19, affectRows=19 2024-11-04 13:37:11.290 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=37, affectRows=37 2024-11-04 13:37:11.290 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=18, validCount=18 2024-11-04 13:37:11.290 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=37, validCount=37 2024-11-04 13:37:11.290 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=54, affectRows=54 2024-11-04 13:37:11.290 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=71, affectRows=71 2024-11-04 13:37:11.290 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=54, validCount=54 2024-11-04 13:37:11.291 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=69, validCount=69 2024-11-04 13:37:11.292 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698560, nodeCount=77, affectRows=77 2024-11-04 13:37:11.292 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698560, nodeCount=77, validCount=77 2024-11-04 13:37:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c76e5afbedb1efe9,e68f039e0519ffdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:12.791 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:12.792 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:12.914 WARN 21062 --- [ocp-async-2,401b151cf084280c,77fdd238b2682fb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:12.962 INFO 21062 --- [ocp-async-4,6152a26efd6eae87,c48bc99cf8525591] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:13.149 INFO 21062 --- [ocp-schedules-20,8ae581c5005ef1eb,b5b4208714b12eba] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:37:13.984 INFO 21062 --- [prometheus-cache-monitor1,ce4ad6afb0376976,662d85fda45ec1f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:37:13.984 INFO 21062 --- [prometheus-cache-monitor1,ce4ad6afb0376976,662d85fda45ec1f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:37:14.624 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:33:14.606, end=2024-11-04T13:35:14.621} 2024-11-04 13:37:14.634 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:37:14.635 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:37:14.914 WARN 21062 --- [ocp-async-4,f0428f17d4d79e79,a23ee668ea8d02c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:14.962 INFO 21062 --- [ocp-async-10,b63b696a146f4d43,92cb068d89c5bc65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:14.983 INFO 21062 --- [prometheus-minute-schedule-1,0c5a428ed3f0738f,fb374fce549f5f3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:15.792 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:15.793 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:16.914 WARN 21062 --- [ocp-async-10,8812800d99f237e5,61250bce05d1704f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:16.962 INFO 21062 --- [ocp-async-6,58549233dd18c2db,5c857a06ad35e698] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:17.983 INFO 21062 --- [prometheus-minute-schedule-1,308433509d01492d,d4b0616a79cd6dda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:37:18.589800+08:00, lastProcessTime=2024-11-04T13:37:08.590624+08:00 2024-11-04 13:37:18.608 INFO 21062 --- [ocp-async-10,bb06ddcbc8fbda2c,c6a21b25ad5a4d93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:37:18.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:37:18.589800+08:00 2024-11-04 13:37:18.793 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:18.794 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:18.914 WARN 21062 --- [ocp-async-2,63f5f1628545c679,74733fc712dbddb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:18.962 INFO 21062 --- [ocp-async-4,ce47e8016e45da93,d1d9897a5d28d976] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:19.645 INFO 21062 --- [ocp-schedules-6,b4be0ba3c013369a,bf482759ea3e4f0a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:37:20.914 WARN 21062 --- [ocp-async-4,9dff0243a3850e44,ba09de1d03f54b3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:20.962 INFO 21062 --- [ocp-async-10,9f9b01d386e2315a,fb264028b5d0a338] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:20.983 INFO 21062 --- [prometheus-minute-schedule-1,6b6c0672801935e7,84b15e1a608c15e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:21.794 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:21.795 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:22.915 WARN 21062 --- [ocp-async-10,66962749d09f9216,714bcaab90ddb31e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:22.963 INFO 21062 --- [ocp-async-7,9e671e130e1e2297,2f8d62beef8fcba2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:23.174 INFO 21062 --- [ocp-schedules-20,326f4a0c15eea5f4,da1c133b1a1c2dca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:37:23.983 INFO 21062 --- [prometheus-minute-schedule-1,cc487be5f887bf0b,3eae94ff5443f4f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:24.795 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:24.796 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:24.915 WARN 21062 --- [ocp-async-7,d677ec1c607b6395,0615a8e182fb4abd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:24.963 INFO 21062 --- [ocp-async-3,650b5ab85a489900,9c4f50a36cdc79a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:26.915 WARN 21062 --- [ocp-async-3,c45631273afb0b10,3886e3ba3d653f76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:26.963 INFO 21062 --- [ocp-async-5,6c455c8a05ad1524,026392a0c8f6b034] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f07349871e187c83,67e33f0e315727f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5315813c60ed1d86,cdae085668d60dc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:37:27.796 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:27.797 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:28.498 INFO 21062 --- [ocp-schedules-24,478ddd087e7e93f0,1908897c456aac6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:37:28.500 INFO 21062 --- [ocp-schedules-22,216c787f2f9fd2ca,9a8bb3391c54e209] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:37:28.500 INFO 21062 --- [ocp-schedules-23,a4dd9468c1252ae1,eb550c440f38e05e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:37:28.500 INFO 21062 --- [ocp-schedules-17,e9fc0a46680d2c54,00ef0fdbcd77222d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:37:28.502 INFO 21062 --- [ocp-schedules-25,75ab62ff49a02380,fafba0187196cc56] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:37:28.503 INFO 21062 --- [ocp-schedules-24,478ddd087e7e93f0,1908897c456aac6c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:37:28.503 INFO 21062 --- [ocp-schedules-22,216c787f2f9fd2ca,9a8bb3391c54e209] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:28.504 INFO 21062 --- [ocp-schedules-17,e9fc0a46680d2c54,00ef0fdbcd77222d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:37:28.504 INFO 21062 --- [ocp-schedules-14,664687d5bc14d6a8,3eece96259ac01f7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:37:28.505 INFO 21062 --- [ocp-schedules-15,178f2a4a37e22e50,ed79fbc3e3dcc763] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:37:28.506 INFO 21062 --- [ocp-schedules-23,a4dd9468c1252ae1,eb550c440f38e05e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:37:28.508 INFO 21062 --- [ocp-schedules-15,178f2a4a37e22e50,ed79fbc3e3dcc763] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:28.511 INFO 21062 --- [ocp-schedules-18,0fc874369cf0206d,7d319a39efce5934] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:37:28.511 INFO 21062 --- [ocp-schedules-12,3e3508d0b9911502,fdb8d91da88f9e60] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:37:28.512 INFO 21062 --- [ocp-schedules-12,3e3508d0b9911502,fdb8d91da88f9e60] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:37:28.512 INFO 21062 --- [ocp-schedules-18,0fc874369cf0206d,7d319a39efce5934] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:37:28.512 INFO 21062 --- [ocp-schedules-5,9168ffc0d225bbae,3fdae96a45d0cfc3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:37:28.512 INFO 21062 --- [ocp-schedules-5,9168ffc0d225bbae,3fdae96a45d0cfc3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:37:28.532 INFO 21062 --- [ocp-schedules-28,9edf008aff7e750d,9a49e0b53a77f000] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:37:28.532 INFO 21062 --- [ocp-schedules-28,9edf008aff7e750d,9a49e0b53a77f000] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:37:28.574 INFO 21062 --- [pool-12-thread-6,42f418c55a8dbd20,00212860c6fd9f40] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:37:28.574 INFO 21062 --- [pool-12-thread-5,42f418c55a8dbd20,21c4cffb43b9c004] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:37:28.575 INFO 21062 --- [pool-12-thread-7,42f418c55a8dbd20,e5efca61ad97d262] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:37:28.575 INFO 21062 --- [pool-12-thread-8,42f418c55a8dbd20,c80ec93128394618] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:37:28.575 INFO 21062 --- [pool-12-thread-8,42f418c55a8dbd20,c80ec93128394618] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:37:28.579 INFO 21062 --- [pool-12-thread-8,42f418c55a8dbd20,c80ec93128394618] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:37:28.579 INFO 21062 --- [ForkJoinPool-11-worker-11,42f418c55a8dbd20,3c40521bfaa29eff] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:37:28.579 INFO 21062 --- [ForkJoinPool-11-worker-11,42f418c55a8dbd20,3c40521bfaa29eff] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:37:28.579 INFO 21062 --- [ForkJoinPool-11-worker-11,42f418c55a8dbd20,3c40521bfaa29eff] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:37:28.579 INFO 21062 --- [pool-12-thread-6,42f418c55a8dbd20,00212860c6fd9f40] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:37:28.580 INFO 21062 --- [ForkJoinPool-11-worker-4,42f418c55a8dbd20,6898691208eed939] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:37:28.580 INFO 21062 --- [pool-12-thread-7,42f418c55a8dbd20,e5efca61ad97d262] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:37:28.581 INFO 21062 --- [pool-12-thread-5,42f418c55a8dbd20,21c4cffb43b9c004] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:37:28.581 INFO 21062 --- [pool-12-thread-5,42f418c55a8dbd20,21c4cffb43b9c004] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:37:28.584 INFO 21062 --- [pool-12-thread-8,42f418c55a8dbd20,c80ec93128394618] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:37:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:37:28.588874+08:00, lastProcessTime=2024-11-04T13:37:18.589800+08:00 2024-11-04 13:37:28.608 INFO 21062 --- [ocp-async-1,11808c31f0f5b82e,05e0f0c488bbd563] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:37:28.716 INFO 21062 --- [ocp-schedules-21,ff9e1e4d132ab6b1,2cf382b658fe4b3f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:37:28.726 INFO 21062 --- [ocp-schedules-23,558861c0d1425f65,757f080ff6fe1d52] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:37:28.739 INFO 21062 --- [ocp-schedules-23,558861c0d1425f65,757f080ff6fe1d52] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:37:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:37:28.588874+08:00 2024-11-04 13:37:28.886 INFO 21062 --- [ocp-schedules-18,2b14a90a9bf72105,e7a4a7ab2d01ff12] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:37:28.886 INFO 21062 --- [ocp-schedules-18,2b14a90a9bf72105,e7a4a7ab2d01ff12] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:37:28.911 INFO 21062 --- [ocp-schedules-5,739b0c73a8db1783,c82656431eea6fe9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:37:28.915 WARN 21062 --- [ocp-async-6,64323f682c9b88a7,9615971824b6cf65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:28.943 INFO 21062 --- [ocp-schedules-11,e69fb6036f6fef8b,91c5bc4deffcba9d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:37:28.948 INFO 21062 --- [ocp-schedules-11,e69fb6036f6fef8b,91c5bc4deffcba9d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:37:28.948 INFO 21062 --- [ocp-schedules-11,e69fb6036f6fef8b,91c5bc4deffcba9d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:37:28.963 INFO 21062 --- [ocp-async-2,d1e72dea0a057183,8e299c0deaa3ade1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:28.992 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:37:28.992 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:37:29.000 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:37:29.000 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:37:29.000 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:37:29.011 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:37:29.011 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:37:29.011 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:37:29.015 INFO 21062 --- [ocp-schedules-8,e0915de8ca85aeac,977fb2892ef5ce1b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:37:29.142 INFO 21062 --- [ocp-schedules-6,aa88204de94e371a,f4ac7143ea6695ba] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:37:29.142 INFO 21062 --- [ocp-schedules-6,aa88204de94e371a,f4ac7143ea6695ba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:37:29.143 INFO 21062 --- [ocp-schedules-6,aa88204de94e371a,f4ac7143ea6695ba] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:37:29.143 INFO 21062 --- [ocp-schedules-6,aa88204de94e371a,f4ac7143ea6695ba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:37:29.144 INFO 21062 --- [ocp-schedules-6,aa88204de94e371a,f4ac7143ea6695ba] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:37:29.144 INFO 21062 --- [ocp-schedules-6,aa88204de94e371a,f4ac7143ea6695ba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:37:29.240 INFO 21062 --- [ocp-schedules-20,821c6c16189b6e58,867134973a45d668] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:37:29.240 INFO 21062 --- [ocp-schedules-20,821c6c16189b6e58,867134973a45d668] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:37:29.268 INFO 21062 --- [pool-32-thread-1,821c6c16189b6e58,6774e0be60eb673c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:37:29.295 WARN 21062 --- [ocp-schedules-20,821c6c16189b6e58,867134973a45d668] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:37:29.295 INFO 21062 --- [ocp-schedules-20,821c6c16189b6e58,867134973a45d668] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:37:29.648 INFO 21062 --- [ocp-schedules-7,4830b43191d1669e,b1a69db07006fa4a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:37:29.983 INFO 21062 --- [prometheus-minute-schedule-1,fbfae5d413b81b67,506b6499af9d401c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:37:30.298 INFO 21062 --- [ocp-schedules-9,46f635e6dd25c9fd,7cda45475b632782] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:37:30.304 INFO 21062 --- [ocp-schedules-13,f6e5973d00188cf1,6c96dbd6ea01eadf] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:37:30.304 INFO 21062 --- [ocp-schedules-13,f6e5973d00188cf1,6c96dbd6ea01eadf] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:37:30.304 INFO 21062 --- [ocp-schedules-9,46f635e6dd25c9fd,7cda45475b632782] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:37:30.304 INFO 21062 --- [ocp-schedules-9,46f635e6dd25c9fd,7cda45475b632782] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:37:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:37:30.331 INFO 21062 --- [ocp-schedules-13,f6e5973d00188cf1,6c96dbd6ea01eadf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:37:30.331 INFO 21062 --- [ocp-schedules-13,f6e5973d00188cf1,6c96dbd6ea01eadf] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:37:30.331 INFO 21062 --- [ocp-schedules-8,8397bd666e8c0fa2,0991f61f978955e9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:37:30.464 INFO 21062 --- [ocp-schedules-6,a665f0dd09056733,93de28a819544670] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:37:30.464+08:00, clusters=[1] 2024-11-04 13:37:30.465 INFO 21062 --- [ocp-schedules-6,a665f0dd09056733,93de28a819544670] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:37:30.499 INFO 21062 --- [ocp-schedules-1,f368e1b73fae5430,a7e29fb9f4048e66] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:37:30.499 INFO 21062 --- [ocp-schedules-1,f368e1b73fae5430,a7e29fb9f4048e66] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:37:30.499 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:37:30.499 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:37:30.508 INFO 21062 --- [ocp-schedules-1,f368e1b73fae5430,a7e29fb9f4048e66] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:37:30.509 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:37:30.509 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:37:30.510 WARN 21062 --- [ocp-schedules-1,f368e1b73fae5430,a7e29fb9f4048e66] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:37:30.511 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:37:30.511 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:37:30.511 INFO 21062 --- [ocp-schedules-1,f368e1b73fae5430,a7e29fb9f4048e66] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:37:30.511 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:37:30.512 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:37:30.512 WARN 21062 --- [ocp-schedules-1,f368e1b73fae5430,a7e29fb9f4048e66] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:37:30.513 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:37:30.513 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:37:30.518 ERROR 21062 --- [ocp-schedules-1,f368e1b73fae5430,a7e29fb9f4048e66] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:37:30.599 INFO 21062 --- [ocp-schedules-20,5264d72421518d23,421850ddb63e67a3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:37:30.599+08:00, clusters=[1] 2024-11-04 13:37:30.603 INFO 21062 --- [ocp-schedules-20,5264d72421518d23,421850ddb63e67a3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:37:30.797 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:30.798 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:30.915 WARN 21062 --- [ocp-async-3,8efd9db6b02c3495,de61e39eee8142c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:30.963 INFO 21062 --- [ocp-async-4,5b65655f5d8a476d,267b5b0f627fe23b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:32.915 WARN 21062 --- [ocp-async-1,2efce728d8e8e0b0,46814b370cbf4e82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:32.954 INFO 21062 --- [ocp-schedules-10,6350dde9843f12d5,b3c2ec9f39615d20] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:37:32.954 INFO 21062 --- [ocp-schedules-10,be560e4354734757,c583fd1718bb60bc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:37:32.959 INFO 21062 --- [ocp-schedules-10,be560e4354734757,c583fd1718bb60bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:32.964 INFO 21062 --- [ocp-async-10,52190fd4a1ee4b77,c44b50ba4fdcfc85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:32.983 INFO 21062 --- [prometheus-minute-schedule-1,526ffb33adcc7a76,896a0598a0c37609] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:32.987 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:37:32.987 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:37:33.219 INFO 21062 --- [ocp-schedules-25,678a1be2ff0eef2b,161ceb5b17e34538] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:37:33.798 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:33.799 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:34.494 INFO 21062 --- [ocp-schedules-3,55fc5e767795629f,eb8a0df49c4a4f73] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:37:34.498 INFO 21062 --- [ocp-schedules-3,55fc5e767795629f,eb8a0df49c4a4f73] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:34.584 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:37:34.586 ERROR 21062 --- [ocp-schedules-3,55fc5e767795629f,eb8a0df49c4a4f73] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:37:34.587 INFO 21062 --- [ocp-schedules-3,55fc5e767795629f,eb8a0df49c4a4f73] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:37:34.915 WARN 21062 --- [ocp-async-9,6f858db3408c32c3,c2dd93909a25f402] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:34.964 INFO 21062 --- [ocp-async-7,170d19c856c46991,e3bdfd2ceb7b0aee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4e32697dfb90ae59,40ac5b9466f4aeb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:36.799 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:36.800 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:36.916 WARN 21062 --- [ocp-async-2,8ab5c52b6a5e2374,e8bb7b6a79d0686b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:36.964 INFO 21062 --- [ocp-async-8,883853973257fac1,8953d72f68165ee3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:38.504 INFO 21062 --- [ocp-schedules-22,216c787f2f9fd2ca,9a8bb3391c54e209] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:37:38.504 INFO 21062 --- [ocp-schedules-17,e9fc0a46680d2c54,00ef0fdbcd77222d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:37:38.508 INFO 21062 --- [ocp-schedules-15,178f2a4a37e22e50,ed79fbc3e3dcc763] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:37:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:37:38.590541+08:00, lastProcessTime=2024-11-04T13:37:28.588874+08:00 2024-11-04 13:37:38.609 INFO 21062 --- [ocp-async-3,4bcc645c8dfd1f67,f75413362fea7101] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:37:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:37:38.590541+08:00 2024-11-04 13:37:38.826 INFO 21062 --- [ocp-schedules-9,fc3c0b65bbd80ac7,e2cc878af994e34b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:37:38.830 INFO 21062 --- [ocp-schedules-9,fc3c0b65bbd80ac7,e2cc878af994e34b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:38.830 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:37:38.916 WARN 21062 --- [ocp-async-5,0c50ae412a21bd62,c9f7f2c09518a12f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:38.964 INFO 21062 --- [ocp-async-9,e7a5716f0ee30e20,89891ea4186c6b3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:38.983 INFO 21062 --- [prometheus-minute-schedule-1,bfb243dd4432ccdd,a2f3664873b16d4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:39.652 INFO 21062 --- [ocp-schedules-27,02a43a036a86d090,30d6ce542904993d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:37:39.800 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:39.801 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:40.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:37:40.006 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:37:40.916 WARN 21062 --- [ocp-async-6,f0e94bf7652b0f4c,f69e5b8ac5912907] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:40.964 INFO 21062 --- [ocp-async-2,15054e07cc690633,008e49acf65c2e2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:41.983 INFO 21062 --- [prometheus-minute-schedule-1,97ba5dc3870e78c0,005bb561f15f8d23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:42.801 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:42.803 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:42.916 WARN 21062 --- [ocp-async-3,b45763032caeb68c,30bdc2bda54bee93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:42.964 INFO 21062 --- [ocp-async-4,55f6d3c1226410a8,e446e9457dbc873c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:43.246 INFO 21062 --- [ocp-schedules-7,5ddade0b6704701f,0fe9f312a0ae82e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:37:44.916 WARN 21062 --- [ocp-async-1,ee9586b4c3905a3c,5f43368cce50b175] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:44.965 INFO 21062 --- [ocp-async-10,f37cf47bdb8b5060,73e47d10f8b6067a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:44.983 INFO 21062 --- [prometheus-minute-schedule-1,8637212d9f0f956f,6db3d6a68750d045] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:45.803 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:45.804 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:46.917 WARN 21062 --- [ocp-async-9,b193467fbedcde12,118666ac3fca8196] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:46.965 INFO 21062 --- [ocp-async-7,a3f5baa13183b9f3,5a728420efd83256] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e022ed6d9256a87f,49e54bb4a94c0532] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:37:48.589860+08:00, lastProcessTime=2024-11-04T13:37:38.590541+08:00 2024-11-04 13:37:48.609 INFO 21062 --- [ocp-async-9,5911a2ec0630fa71,eb17a94a210f0f81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:37:48.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:37:48.589860+08:00 2024-11-04 13:37:48.805 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:48.805 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:48.831 INFO 21062 --- [ocp-schedules-9,fc3c0b65bbd80ac7,e2cc878af994e34b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:37:48.917 WARN 21062 --- [ocp-async-3,4924f4fc635ab3d7,f399d313866d74c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:48.965 INFO 21062 --- [ocp-async-4,919883b09ac19e46,924ef7decc1fa544] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:49.654 INFO 21062 --- [ocp-schedules-4,8b4bd74d840fa277,1469a28cc017ded8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:37:50.307 INFO 21062 --- [ocp-schedules-9,8cf20b6b4627022b,47c1550a56ccd5e2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:37:50.312 INFO 21062 --- [ocp-schedules-9,8cf20b6b4627022b,47c1550a56ccd5e2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:37:50.312 INFO 21062 --- [ocp-schedules-9,8cf20b6b4627022b,47c1550a56ccd5e2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:37:50.917 WARN 21062 --- [ocp-async-1,64a4164195271ab9,418e5d8be38ca1ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:50.965 INFO 21062 --- [ocp-async-10,b6084c06e5f49bf0,93aa370c07ce4409] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:50.983 INFO 21062 --- [prometheus-minute-schedule-1,10d4f6c24bb0a26b,feec4685b16b928d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:51.805 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:51.807 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:52.917 WARN 21062 --- [ocp-async-9,e1dbab9ad9b9f9f7,e6b5d9c3824c80ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:52.965 INFO 21062 --- [ocp-async-7,3b0b726f70da1321,7f53d6c3577173c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:53.287 INFO 21062 --- [ocp-schedules-27,1663b55b235cf047,49f36135938aff68] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:37:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4c2924e40119cd17,3450e9019ae8dcd9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:54.807 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:54.808 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:54.917 WARN 21062 --- [ocp-async-7,0ba9e3966cb33c16,360e2d945d2ae7a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:54.966 INFO 21062 --- [ocp-async-8,7d7d42090c4a72e3,8f11c010c7b4d679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:56.917 WARN 21062 --- [ocp-async-8,4e087532e9dbbf38,9351672a36691024] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:56.966 INFO 21062 --- [ocp-async-5,6031d4aa677b7a47,453f8ec6c12e65d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:56.983 INFO 21062 --- [prometheus-minute-schedule-1,eca5b0caaf7a3b89,0185ecb56285d0b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:37:57.808 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:37:57.809 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:37:58.500 INFO 21062 --- [ocp-schedules-11,6da11c192547dab5,93d2d6f267ba1faa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:37:58.500 INFO 21062 --- [ocp-schedules-2,bdf28e73bce05414,5e6378c6156f4201] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:37:58.503 INFO 21062 --- [ocp-schedules-17,63ad616e8ba84eeb,6c68c3866731d8e5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:37:58.503 INFO 21062 --- [ocp-schedules-11,6da11c192547dab5,93d2d6f267ba1faa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:37:58.504 INFO 21062 --- [ocp-schedules-2,bdf28e73bce05414,5e6378c6156f4201] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:37:58.505 INFO 21062 --- [ocp-schedules-1,3bcce5d3e43344e2,3e0a3949d8129fc6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:37:58.511 INFO 21062 --- [ocp-schedules-28,1fecbba06fe3b62b,6742f707f2688823] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:37:58.511 INFO 21062 --- [ocp-schedules-28,1fecbba06fe3b62b,6742f707f2688823] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:37:58.511 INFO 21062 --- [ocp-schedules-27,171331824146778d,f67b17aba99f99f8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:37:58.511 INFO 21062 --- [ocp-schedules-5,64e9ae16e8c32213,414b4fb13a73473a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:37:58.512 INFO 21062 --- [ocp-schedules-5,64e9ae16e8c32213,414b4fb13a73473a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:37:58.512 INFO 21062 --- [ocp-schedules-27,171331824146778d,f67b17aba99f99f8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:37:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:37:58.588893+08:00, lastProcessTime=2024-11-04T13:37:48.589860+08:00 2024-11-04 13:37:58.609 INFO 21062 --- [ocp-async-8,b3c87c49a20c2452,691f5e68e6301a72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:37:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:37:58.588893+08:00 2024-11-04 13:37:58.914 INFO 21062 --- [ocp-schedules-29,c7e3f5ef8ea2dc06,5dc1e2110847609e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:37:58.918 WARN 21062 --- [ocp-async-10,23b3ff82ccf5d858,1c1ce06b13e7d6b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:37:58.966 INFO 21062 --- [ocp-async-7,0738d665fe96c9f8,debafcec77ba7a2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:37:59.657 INFO 21062 --- [ocp-schedules-23,eaba4162dbc46569,d714abe47f9143d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:37:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f21ae98719a3735d,1f8be064fac539da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698680, interval=60 2024-11-04 13:38:00.809 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:00.810 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:00.918 WARN 21062 --- [ocp-async-7,17a10315663fb4c7,d381703cfa8c00a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:00.966 INFO 21062 --- [ocp-async-8,ea584f4276fc279a,95e96d81bb2250a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:02.918 WARN 21062 --- [ocp-async-8,1f39eaff9a7c1879,a3e311740d051c99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:02.966 INFO 21062 --- [ocp-async-5,5b0f725cbe55d519,4cc9f54166886ff6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f0a57ef5caf8eb66,4425416ff2ecb5f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:02.988 INFO 21062 --- [ocp-schedules-10,be560e4354734757,c583fd1718bb60bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:38:02.988 INFO 21062 --- [ocp-schedules-10,53cf5e5bbb3ad3bf,5085dc8215bebfc8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:38:02.991 INFO 21062 --- [ocp-schedules-10,53cf5e5bbb3ad3bf,5085dc8215bebfc8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:03.023 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:38:03.023 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:38:03.315 INFO 21062 --- [ocp-schedules-4,b225a63fec86ea92,5deab66891bfbafe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:38:03.810 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:03.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:04.918 WARN 21062 --- [ocp-async-5,a51e3718c596fadd,b9ba8bcffb8ea27b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:04.967 INFO 21062 --- [ocp-async-9,3ae8b725467a9a7d,81b63ec6a5800ba0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a0ff100b07e3e9e9,b1eb9d76b920d67a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:06.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:06.812 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:06.918 WARN 21062 --- [ocp-async-9,f0ebeaab6a1579e7,a6dac690b883b681] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:06.967 INFO 21062 --- [ocp-async-3,44fe36bec2213db0,21003ebdc520f7a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:08.504 INFO 21062 --- [ocp-schedules-11,6da11c192547dab5,93d2d6f267ba1faa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:38:08.505 INFO 21062 --- [ocp-schedules-2,bdf28e73bce05414,5e6378c6156f4201] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:38:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:38:08.590567+08:00, lastProcessTime=2024-11-04T13:37:58.588893+08:00 2024-11-04 13:38:08.609 INFO 21062 --- [ocp-async-9,fa3fa200fe52cb4b,ea46f2d937f48661] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:38:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:38:08.590567+08:00 2024-11-04 13:38:08.918 WARN 21062 --- [ocp-async-8,e26e4174c1e2cf83,02a8f5b8e459f9fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:08.967 INFO 21062 --- [ocp-async-4,de292d155f4e7b51,4cc5d92d8b56cc1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:08.983 INFO 21062 --- [prometheus-minute-schedule-1,102e366008e0d2e8,d05a2a2b5ae4e843] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:09.660 INFO 21062 --- [ocp-schedules-15,1dfb63850a2aa1d8,dd01195d76f1cbd3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:38:09.812 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:09.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:10.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:38:10.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:38:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:38:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:38:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:38:10.315 INFO 21062 --- [ocp-schedules-1,b4d3c17c7159fb99,6d0debdaed26ea4c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:38:10.320 INFO 21062 --- [ocp-schedules-1,b4d3c17c7159fb99,6d0debdaed26ea4c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:38:10.320 INFO 21062 --- [ocp-schedules-1,b4d3c17c7159fb99,6d0debdaed26ea4c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:38:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:38:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:38:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:38:10.919 WARN 21062 --- [ocp-async-4,d97787c3b7e4220d,709500e2f28b7dbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:10.967 INFO 21062 --- [ocp-async-6,351b0771feb3d0ec,62a6bc12155983a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:11.295 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698620, nodeCount=72, affectRows=72 2024-11-04 13:38:11.295 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698620, nodeCount=96, affectRows=96 2024-11-04 13:38:11.295 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698620, nodeCount=78, affectRows=78 2024-11-04 13:38:11.295 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698620, nodeCount=76, affectRows=76 2024-11-04 13:38:11.295 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698620, nodeCount=75, validCount=75 2024-11-04 13:38:11.295 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698620, nodeCount=72, validCount=72 2024-11-04 13:38:11.296 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698620, nodeCount=77, validCount=77 2024-11-04 13:38:11.296 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698620, nodeCount=95, validCount=95 2024-11-04 13:38:11.983 INFO 21062 --- [prometheus-minute-schedule-1,85ad494ea39c1695,7c94bdecafc55947] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:12.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:12.814 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:12.919 WARN 21062 --- [ocp-async-9,ca630534ad4974ac,9469efdb5c137ff4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:12.967 INFO 21062 --- [ocp-async-2,6f633247670e20a4,c3ae403f0c3d1057] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:13.392 INFO 21062 --- [ocp-schedules-18,edc8f3f56f1225d5,cce1980953fe18ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:38:13.984 INFO 21062 --- [prometheus-cache-monitor1,5976b759c414026e,b8cc8d2fd49fa57d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:38:13.984 INFO 21062 --- [prometheus-cache-monitor1,5976b759c414026e,b8cc8d2fd49fa57d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:38:14.080 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:32:14.057, end=2024-11-04T13:37:14.077} 2024-11-04 13:38:14.080 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:32:14.057, end=2024-11-04T13:37:14.077} 2024-11-04 13:38:14.086 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:38:14.086 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:38:14.086 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:38:14.086 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:38:14.919 WARN 21062 --- [ocp-async-3,d94401dfb284e108,3f6ea95378143126] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:14.967 INFO 21062 --- [ocp-async-1,3508f026bc09671f,e8aaed825f983691] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:14.983 INFO 21062 --- [prometheus-minute-schedule-1,8fc1ddb182123c86,f20bf0ca0ce0afad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:15.814 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:15.815 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:16.919 WARN 21062 --- [ocp-async-5,6afa0f9d2e86bd54,9b55bedf9bb829cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:16.968 INFO 21062 --- [ocp-async-10,8ccfeddf60ba09ea,3a570fc16ef501a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d8f1f297b671dc66,4063f49ec9c48976] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:38:18.590782+08:00, lastProcessTime=2024-11-04T13:38:08.590567+08:00 2024-11-04 13:38:18.609 INFO 21062 --- [ocp-async-5,9d4826b029e2fc09,7243ecb483b2b6c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:38:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:38:18.590782+08:00 2024-11-04 13:38:18.815 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:18.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:18.919 WARN 21062 --- [ocp-async-9,cd73e09d54e9c55c,13c8e1a14cf5e070] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:18.968 INFO 21062 --- [ocp-async-2,e03256af5c2ba75f,57bb2793ec3f15d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:19.666 INFO 21062 --- [ocp-schedules-6,8c7cc818ad499be0,ccf307334e8a9b0c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:38:20.919 WARN 21062 --- [ocp-async-3,e15b5488e669b5de,cc355b3ea7ab4bc5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:20.968 INFO 21062 --- [ocp-async-1,ad3f22b98b4fe89e,267ea068a6bfc8ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b5f041bb0181a1ac,bad10214e4f1b1ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:21.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:21.817 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:22.920 WARN 21062 --- [ocp-async-5,39faa73b8ceaff1e,3e4fa0743e43eb0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:22.968 INFO 21062 --- [ocp-async-10,5d69b12470869e63,6c753dc301f93f2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:23.416 INFO 21062 --- [ocp-schedules-29,1811eeb8090470a5,b010b0de626ce386] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:38:23.983 INFO 21062 --- [prometheus-minute-schedule-1,9ed86e28632edb87,1c1f370ca9d2cd4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:24.817 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:24.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:24.920 WARN 21062 --- [ocp-async-6,25201ea39f38b5d6,cb4a8ec1e42729ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:24.968 INFO 21062 --- [ocp-async-7,5bc0343b738a0987,1035f9853868af7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:26.920 WARN 21062 --- [ocp-async-2,0d2244bffd21cbde,b7896f2786c73101] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:26.969 INFO 21062 --- [ocp-async-8,78725da80b487dea,dad16a4d46057cf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:26.983 INFO 21062 --- [prometheus-minute-schedule-1,a1c975cf5fd80850,ed95064d42c32152] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e205ac4d3d2b8d10,ed3e0be784f9d22a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:38:27.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:27.819 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:28.498 INFO 21062 --- [ocp-schedules-24,4cac4ac52f81724f,d7a74a3e92253ca3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:38:28.499 INFO 21062 --- [ocp-schedules-12,4f4936f7db7edec9,91d44ab7241add22] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:38:28.500 INFO 21062 --- [ocp-schedules-22,7ed09fb539fab7d7,210b7dcc324bc2a4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:38:28.500 INFO 21062 --- [ocp-schedules-23,27028260e027716a,a279e16990fec1e3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:38:28.502 INFO 21062 --- [ocp-schedules-24,4cac4ac52f81724f,d7a74a3e92253ca3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:38:28.502 INFO 21062 --- [ocp-schedules-6,a58e1dfd538fe9d2,5ca49b101f76921d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:38:28.503 INFO 21062 --- [ocp-schedules-22,7ed09fb539fab7d7,210b7dcc324bc2a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:28.504 INFO 21062 --- [ocp-schedules-23,27028260e027716a,a279e16990fec1e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:38:28.504 INFO 21062 --- [ocp-schedules-13,9ea0dc7a8f004cf8,b4f10334b45b63f1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:38:28.505 INFO 21062 --- [ocp-schedules-12,4f4936f7db7edec9,91d44ab7241add22] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:38:28.510 INFO 21062 --- [ocp-schedules-21,d8919d9c0b3ad209,9a7910e62a7f1969] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:38:28.510 INFO 21062 --- [ocp-schedules-28,b2879d99444a6898,890778b2c31ec2bd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:38:28.511 INFO 21062 --- [ocp-schedules-28,b2879d99444a6898,890778b2c31ec2bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:38:28.511 INFO 21062 --- [ocp-schedules-21,d8919d9c0b3ad209,9a7910e62a7f1969] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:38:28.511 INFO 21062 --- [ocp-schedules-5,49bc4a68e3d20daa,18ef0efdf3f55a09] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:38:28.511 INFO 21062 --- [ocp-schedules-5,49bc4a68e3d20daa,18ef0efdf3f55a09] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:38:28.568 INFO 21062 --- [ocp-schedules-29,5aabe5b47faf1dc9,1b87859670c64c8a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:38:28.568 INFO 21062 --- [ocp-schedules-29,5aabe5b47faf1dc9,1b87859670c64c8a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:38:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:38:28.590331+08:00, lastProcessTime=2024-11-04T13:38:18.590782+08:00 2024-11-04 13:38:28.609 INFO 21062 --- [ocp-async-3,00c0385f58e7a720,d988cbbf6ec31497] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:38:28.662 INFO 21062 --- [ocp-schedules-15,057782fce5542b5c,63383dc26ddc7697] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:38:28.665 INFO 21062 --- [ocp-schedules-15,057782fce5542b5c,63383dc26ddc7697] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:38:28.719 INFO 21062 --- [ocp-schedules-25,baa03e99be827422,3e6116d080460d44] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:38:28.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:38:28.590331+08:00 2024-11-04 13:38:28.889 INFO 21062 --- [ocp-schedules-9,b30f4b79f9ff5eb6,74cc49cedd6bb354] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:38:28.889 INFO 21062 --- [ocp-schedules-9,b30f4b79f9ff5eb6,74cc49cedd6bb354] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:38:28.916 INFO 21062 --- [ocp-schedules-6,54dcc5726fea4c85,796ec592608db396] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:38:28.920 WARN 21062 --- [ocp-async-10,f9badc3958b4d92f,2c28650a998ce00f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:28.949 INFO 21062 --- [ocp-schedules-21,16ac4d6723a7da91,1537caf8063e2da2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:38:28.954 INFO 21062 --- [ocp-schedules-21,16ac4d6723a7da91,1537caf8063e2da2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:38:28.954 INFO 21062 --- [ocp-schedules-21,16ac4d6723a7da91,1537caf8063e2da2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:38:28.969 INFO 21062 --- [ocp-async-7,06322a6535f1cfa5,f78d2df8da0f3deb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:29.145 INFO 21062 --- [ocp-schedules-29,0f0db79dd2661673,603acc34138215de] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:38:29.145 INFO 21062 --- [ocp-schedules-29,0f0db79dd2661673,603acc34138215de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:38:29.146 INFO 21062 --- [ocp-schedules-29,0f0db79dd2661673,603acc34138215de] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:38:29.146 INFO 21062 --- [ocp-schedules-29,0f0db79dd2661673,603acc34138215de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:38:29.147 INFO 21062 --- [ocp-schedules-29,0f0db79dd2661673,603acc34138215de] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:38:29.147 INFO 21062 --- [ocp-schedules-29,0f0db79dd2661673,603acc34138215de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:38:29.669 INFO 21062 --- [ocp-schedules-17,b06b3eb1a2ac130b,e69f39dad231570b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:38:29.983 INFO 21062 --- [prometheus-minute-schedule-1,649531dfec3ca4fd,65d4a1e469c391fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:38:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:38:30.323 INFO 21062 --- [ocp-schedules-21,7a389734b8081e1f,53254968a09b87ca] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:38:30.328 INFO 21062 --- [ocp-schedules-21,7a389734b8081e1f,53254968a09b87ca] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:38:30.328 INFO 21062 --- [ocp-schedules-21,7a389734b8081e1f,53254968a09b87ca] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:38:30.331 INFO 21062 --- [ocp-schedules-24,ddc823612084623a,ce95f3fde6d4093c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:38:30.331 INFO 21062 --- [ocp-schedules-24,ddc823612084623a,ce95f3fde6d4093c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:38:30.356 INFO 21062 --- [ocp-schedules-24,ddc823612084623a,ce95f3fde6d4093c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:38:30.356 INFO 21062 --- [ocp-schedules-24,ddc823612084623a,ce95f3fde6d4093c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:38:30.356 INFO 21062 --- [ocp-schedules-3,87ef08984bf80ff5,7d3d9c9dbee71376] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:38:30.492 INFO 21062 --- [ocp-schedules-14,4b0728ce8e6378ad,d5d3aca7d96cfe1d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:38:30.492+08:00, clusters=[1] 2024-11-04 13:38:30.493 INFO 21062 --- [ocp-schedules-14,4b0728ce8e6378ad,d5d3aca7d96cfe1d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:38:30.526 INFO 21062 --- [ocp-schedules-26,50eeb011927d1ccc,13eb650e98e92348] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:38:30.526 INFO 21062 --- [ocp-schedules-26,50eeb011927d1ccc,13eb650e98e92348] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:38:30.526 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:38:30.526 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:38:30.536 INFO 21062 --- [ocp-schedules-26,50eeb011927d1ccc,13eb650e98e92348] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:38:30.536 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:38:30.536 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:38:30.537 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:38:30.537 WARN 21062 --- [ocp-schedules-26,50eeb011927d1ccc,13eb650e98e92348] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:38:30.537 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:38:30.538 INFO 21062 --- [ocp-schedules-26,50eeb011927d1ccc,13eb650e98e92348] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:38:30.538 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:38:30.538 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:38:30.538 INFO 21062 --- [ocp-schedules-19,92d1a4e60889c147,6fbd9a26ad674c4a] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:38:30.539 WARN 21062 --- [ocp-schedules-26,50eeb011927d1ccc,13eb650e98e92348] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:38:30.539 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:38:30.539 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:38:30.542 INFO 21062 --- [ocp-schedules-19,92d1a4e60889c147,6fbd9a26ad674c4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:30.546 ERROR 21062 --- [ocp-schedules-26,50eeb011927d1ccc,13eb650e98e92348] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:38:30.596 ERROR 21062 --- [ocp-schedules-19,92d1a4e60889c147,6fbd9a26ad674c4a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:38:30.597 INFO 21062 --- [ocp-schedules-19,92d1a4e60889c147,6fbd9a26ad674c4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:38:30.628 INFO 21062 --- [ocp-schedules-7,587245904948145e,8fe553b7cde6c3f6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:38:30.628+08:00, clusters=[1] 2024-11-04 13:38:30.632 INFO 21062 --- [ocp-schedules-7,587245904948145e,8fe553b7cde6c3f6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:38:30.819 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:30.820 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:30.920 WARN 21062 --- [ocp-async-9,e5604cc616181b6b,ca4c57bb102f9e69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:30.969 INFO 21062 --- [ocp-async-8,85f9fa189735f808,e6a9e364f9619ceb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:32.921 WARN 21062 --- [ocp-async-3,1cef0eed0e29a099,de57d7a702219878] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:32.969 INFO 21062 --- [ocp-async-5,962d6f1542a069bd,18adf09d60b3cd80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b5153f531c4c5030,950ad4e112f922eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:33.023 INFO 21062 --- [ocp-schedules-10,53cf5e5bbb3ad3bf,5085dc8215bebfc8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:38:33.023 INFO 21062 --- [ocp-schedules-10,86deb069d19cf804,65af6ab4974355ea] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:38:33.027 INFO 21062 --- [ocp-schedules-10,86deb069d19cf804,65af6ab4974355ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:33.055 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:38:33.055 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:38:33.458 INFO 21062 --- [ocp-schedules-7,8fe8bff47312e079,d9b926ec8b9c6c70] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:38:33.820 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:33.821 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:34.587 INFO 21062 --- [ocp-schedules-24,7677496dfd2022f3,9d13d9cc34397e41] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:38:34.591 INFO 21062 --- [ocp-schedules-24,7677496dfd2022f3,9d13d9cc34397e41] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:34.677 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:38:34.679 ERROR 21062 --- [ocp-schedules-24,7677496dfd2022f3,9d13d9cc34397e41] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:38:34.679 INFO 21062 --- [ocp-schedules-24,7677496dfd2022f3,9d13d9cc34397e41] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:38:34.921 WARN 21062 --- [ocp-async-5,2147fc37a07a4a35,e8bed02126c6461b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:34.969 INFO 21062 --- [ocp-async-6,e0743d389e4f60be,194d67b9cc41a83a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:35.983 INFO 21062 --- [prometheus-minute-schedule-1,9c0c52f0c6c66f0a,ed3ee0d2cbe68bdd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:36.821 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:36.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:36.921 WARN 21062 --- [ocp-async-6,fce001e1763fbc83,e9a7e45c630d05a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:36.969 INFO 21062 --- [ocp-async-2,2f11cc497b7a08eb,a6d4bfb041408f91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:38.504 INFO 21062 --- [ocp-schedules-23,27028260e027716a,a279e16990fec1e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:38:38.504 INFO 21062 --- [ocp-schedules-22,7ed09fb539fab7d7,210b7dcc324bc2a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:38:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:38:38.589470+08:00, lastProcessTime=2024-11-04T13:38:28.590331+08:00 2024-11-04 13:38:38.610 INFO 21062 --- [ocp-async-7,f32034658b96cff4,90114bea5069c094] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:38:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:38:38.589470+08:00 2024-11-04 13:38:38.921 WARN 21062 --- [ocp-async-3,31380379a9de1043,362e6e20625c0ba9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:38.970 INFO 21062 --- [ocp-async-4,6b229a7319584296,c87b39465f09b247] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c0d5c874c36b7089,b3aa8b4e451daffb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:39.672 INFO 21062 --- [ocp-schedules-6,d3c36a520138d504,33ba3547f8b914a4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:38:39.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:39.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:40.002 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:38:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:38:40.921 WARN 21062 --- [ocp-async-4,50f914b362ed60f8,aff3747448743658] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:40.970 INFO 21062 --- [ocp-async-7,235a54f5d5a27143,a1f6fdaedaa3af71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:38:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:38:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 13:38:41.983 INFO 21062 --- [prometheus-minute-schedule-1,f6977b5f996a7915,01abf07a0ee48863] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:42.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:42.824 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:42.922 WARN 21062 --- [ocp-async-7,5398e1b4a2802d37,426a6ba645b61d9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:42.970 INFO 21062 --- [ocp-async-8,c205bd353622e2f2,3221fba691954967] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:43.486 INFO 21062 --- [ocp-schedules-9,54fc91a831c8d7a8,650605cb4771693e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:38:44.922 WARN 21062 --- [ocp-async-8,5795d956c6e1f5f8,8cba773a9aa474f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:44.970 INFO 21062 --- [ocp-async-5,16b16cdce653b97e,7b1dce71366a8c57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:44.983 INFO 21062 --- [prometheus-minute-schedule-1,186dc7a3730e7757,c77e95298c64f302] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:45.824 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:45.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:46.922 WARN 21062 --- [ocp-async-5,db936b292002b38f,6561c6ceae5c6684] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:46.970 INFO 21062 --- [ocp-async-6,45afa82ebe0a0614,f169f4a4a56d24ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:47.983 INFO 21062 --- [prometheus-minute-schedule-1,7c3d661dfe06342d,31cd0152731ac784] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:38:48.590618+08:00, lastProcessTime=2024-11-04T13:38:38.589470+08:00 2024-11-04 13:38:48.610 INFO 21062 --- [ocp-async-5,c1f1803dcac6a9d7,8080106981c2ce03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:38:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:38:48.590618+08:00 2024-11-04 13:38:48.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:48.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:48.831 INFO 21062 --- [ocp-schedules-3,339d4a445fc9b95d,2ee028c210583f8e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:38:48.834 INFO 21062 --- [ocp-schedules-3,339d4a445fc9b95d,2ee028c210583f8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:48.835 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:38:48.922 WARN 21062 --- [ocp-async-7,914a4dcb3b65042c,d2ddd0865888d0a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:48.971 INFO 21062 --- [ocp-async-8,cb427192dcb9df4b,67cd6955cc6f9f0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:49.675 INFO 21062 --- [ocp-schedules-19,c975e6b5a52520a8,e7170be364a81b18] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:38:50.331 INFO 21062 --- [ocp-schedules-2,72b56a12fbcc2119,0940d74f93cb7bfb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:38:50.338 INFO 21062 --- [ocp-schedules-2,72b56a12fbcc2119,0940d74f93cb7bfb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:38:50.338 INFO 21062 --- [ocp-schedules-2,72b56a12fbcc2119,0940d74f93cb7bfb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:38:50.922 WARN 21062 --- [ocp-async-8,50f3b1cbf7499703,a24a605bc51672b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:50.971 INFO 21062 --- [ocp-async-5,b1b5c393375c8c96,68dbb963e7876ad6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4ebbeb57fd96fe80,1c94cb4758b9b8ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:51.826 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:51.826 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:52.923 WARN 21062 --- [ocp-async-5,30f43a40e7a8bb90,69b83e490f4345f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:52.971 INFO 21062 --- [ocp-async-6,c7213302e3f001cf,04664b74521a3e47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:53.529 INFO 21062 --- [ocp-schedules-6,8eda998e1ed416a5,d4d0a65a66ccbc12] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:38:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d897e24cc768ea3f,7c0c75740f860a66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:54.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:54.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:54.923 WARN 21062 --- [ocp-async-6,b7fceec57a37607c,362fd835562fed39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:54.971 INFO 21062 --- [ocp-async-9,ca1fdb492af8060d,6e4fbf4592b75764] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:56.923 WARN 21062 --- [ocp-async-9,2750c3c43265202e,5eb4694a49ee4274] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:56.971 INFO 21062 --- [ocp-async-1,2f40f767f32842bd,5aca32c490add2ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:56.983 INFO 21062 --- [prometheus-minute-schedule-1,aeb2ecf489249562,d755d1f2bf6cde48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:38:57.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:38:57.828 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:38:58.500 INFO 21062 --- [ocp-schedules-21,f7bd0c87e8762eb1,0e0c7a443f12ce9c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:38:58.500 INFO 21062 --- [ocp-schedules-2,b91fcf9be3d1cbf3,2ca1711b89f9916c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:38:58.502 INFO 21062 --- [ocp-schedules-25,0ec69a3cb3b715c1,8e4de793727d2dc4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:38:58.504 INFO 21062 --- [ocp-schedules-21,f7bd0c87e8762eb1,0e0c7a443f12ce9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:58.504 INFO 21062 --- [ocp-schedules-2,b91fcf9be3d1cbf3,2ca1711b89f9916c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:38:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:38:58.505 INFO 21062 --- [ocp-schedules-23,ce147d0b573475ac,6ecbc961bf18070e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:38:58.512 INFO 21062 --- [ocp-schedules-11,5546c84703cfdfe8,042b16d598308a26] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:38:58.512 INFO 21062 --- [ocp-schedules-7,215a15b9c323aa8f,ad6a70e2ed61666c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:38:58.512 INFO 21062 --- [ocp-schedules-11,5546c84703cfdfe8,042b16d598308a26] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:38:58.512 INFO 21062 --- [ocp-schedules-7,215a15b9c323aa8f,ad6a70e2ed61666c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:38:58.514 INFO 21062 --- [ocp-schedules-15,25ea51ca4aa69828,a4ed9058a29e6239] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:38:58.514 INFO 21062 --- [ocp-schedules-15,25ea51ca4aa69828,a4ed9058a29e6239] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:38:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:38:58.589944+08:00, lastProcessTime=2024-11-04T13:38:48.590618+08:00 2024-11-04 13:38:58.610 INFO 21062 --- [ocp-async-9,754720dfc0ab8161,13903ef9410cb38b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:38:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:38:58.589944+08:00 2024-11-04 13:38:58.835 INFO 21062 --- [ocp-schedules-3,339d4a445fc9b95d,2ee028c210583f8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:38:58.919 INFO 21062 --- [ocp-schedules-6,568312b34e739283,07edbd218842fee7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:38:58.923 WARN 21062 --- [ocp-async-5,efdc9d34ea3be949,3a6b8db80ae5c942] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:38:58.971 INFO 21062 --- [ocp-async-6,8e18eb4a48362e21,55eb2ff94a30f440] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:38:59.678 INFO 21062 --- [ocp-schedules-27,151db81b620810b3,7df2c2c0da8cee45] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:38:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a5d8b6d2ada5c3cf,9ab41c1863de5904] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698740, interval=60 2024-11-04 13:39:00.828 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:00.829 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:00.923 WARN 21062 --- [ocp-async-6,dafa29f45d63fdc8,d786ad494ebeb500] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:00.972 INFO 21062 --- [ocp-async-9,20f93baf666f52c4,7b314116fa7433e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:02.924 WARN 21062 --- [ocp-async-9,29c52b141bd8e65f,96574461c4ae1e35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:02.972 INFO 21062 --- [ocp-async-1,c6b8a2488f4dcecd,b1d2692afe10688e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d1df132ecaddbbb4,f89706a9a4fca3de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:03.056 INFO 21062 --- [ocp-schedules-10,86deb069d19cf804,65af6ab4974355ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:39:03.056 INFO 21062 --- [ocp-schedules-12,afa768b0c7fcde59,fcce6ec31f2cea09] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:39:03.061 INFO 21062 --- [ocp-schedules-12,afa768b0c7fcde59,fcce6ec31f2cea09] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:03.090 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:39:03.090 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:39:03.555 INFO 21062 --- [ocp-schedules-19,1d359d2315836d07,b8b58ee99905fef6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:39:03.829 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:03.830 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:04.924 WARN 21062 --- [ocp-async-1,af978431650ba61b,6799993a94d55c5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:04.972 INFO 21062 --- [ocp-async-4,b09d870ee9a4c273,8f24a5e02fc26641] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:05.983 INFO 21062 --- [prometheus-minute-schedule-1,235526e706c49df1,00ad68948fa5d488] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:06.830 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:06.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:06.924 WARN 21062 --- [ocp-async-4,5f62101fbddfa2ed,4509ef8a6a7f4514] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:06.972 INFO 21062 --- [ocp-async-7,2c18a0b06daa7aaf,f3c801efc1e71a98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:08.505 INFO 21062 --- [ocp-schedules-2,b91fcf9be3d1cbf3,2ca1711b89f9916c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:39:08.505 INFO 21062 --- [ocp-schedules-21,f7bd0c87e8762eb1,0e0c7a443f12ce9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:39:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:39:08.590134+08:00, lastProcessTime=2024-11-04T13:38:58.589944+08:00 2024-11-04 13:39:08.610 INFO 21062 --- [ocp-async-4,ab539a1df99e9aa9,2537d861abd25bf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:39:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:39:08.590134+08:00 2024-11-04 13:39:08.924 WARN 21062 --- [ocp-async-9,f6651098dddd30b5,fc25505a968d7caa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:08.972 INFO 21062 --- [ocp-async-3,fe48e67a42f3972b,87fa6aee44f8c5c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:08.983 INFO 21062 --- [prometheus-minute-schedule-1,19e79af48a362519,bf5093de3fe05bb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:09.682 INFO 21062 --- [ocp-schedules-6,3e506bef57a54767,126ce62ccde249bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:39:09.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:09.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:10.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:39:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:39:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:39:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:39:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:39:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:39:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:39:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:39:10.340 INFO 21062 --- [ocp-schedules-23,45388279e809b556,e3dade122a6e79b5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:39:10.346 INFO 21062 --- [ocp-schedules-23,45388279e809b556,e3dade122a6e79b5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:39:10.346 INFO 21062 --- [ocp-schedules-23,45388279e809b556,e3dade122a6e79b5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:39:10.924 WARN 21062 --- [ocp-async-3,26e189fc54cc455d,f37ea8b2ad4da07d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:10.972 INFO 21062 --- [ocp-async-10,c392741325ee7d03,49d2a5fcccc7db70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:11.298 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698680, nodeCount=1, affectRows=1 2024-11-04 13:39:11.298 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698680, nodeCount=1, validCount=1 2024-11-04 13:39:11.299 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698680, nodeCount=27, affectRows=27 2024-11-04 13:39:11.299 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698680, nodeCount=27, validCount=27 2024-11-04 13:39:11.300 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698680, nodeCount=40, affectRows=40 2024-11-04 13:39:11.300 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698680, nodeCount=13, affectRows=13 2024-11-04 13:39:11.300 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698680, nodeCount=13, validCount=13 2024-11-04 13:39:11.300 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698680, nodeCount=40, validCount=40 2024-11-04 13:39:11.300 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698680, nodeCount=53, affectRows=53 2024-11-04 13:39:11.300 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698680, nodeCount=52, validCount=52 2024-11-04 13:39:11.301 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698680, nodeCount=89, affectRows=89 2024-11-04 13:39:11.301 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698680, nodeCount=99, affectRows=99 2024-11-04 13:39:11.301 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698680, nodeCount=88, validCount=88 2024-11-04 13:39:11.301 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698680, nodeCount=98, validCount=98 2024-11-04 13:39:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c907ad69de85a202,6e4744e00c8c7b68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:12.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:12.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:12.924 WARN 21062 --- [ocp-async-4,a6d5bd7afd5f1e91,a0c6207db744592d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:12.972 INFO 21062 --- [ocp-async-2,cc5c9fa20f343049,3ed5677b545da1c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:13.045 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:39:13.045 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:39:13.053 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:39:13.053 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:39:13.053 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:39:13.053 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:39:13.059 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:39:13.059 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:39:13.059 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:39:13.059 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:39:13.096 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:39:13.105 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:39:13.105 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:39:13.105 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:39:13.105 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:39:13.105 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:39:13.113 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:39:13.596 INFO 21062 --- [ocp-schedules-21,984e2d9b8b344683,215a7554684a6f47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:39:13.693 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,318657b11d1c8d99,eb36f6e1de6df4f3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,318657b11d1c8d99,eb36f6e1de6df4f3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:39:14.262 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.203349155S 2024-11-04 13:39:14.263 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:39:14.366 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:39:14.366 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:39:14.396 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:39:14.452 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:39:14.452 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:39:14.452 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:39:14.452 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:39:14.452 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:39:14.452 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:39:14.463 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:39:14.643 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] o.s.boot.SpringApplication : Started application in 1.589 seconds (JVM running for 4168.322) 2024-11-04 13:39:14.643 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.590224153S 2024-11-04 13:39:14.646 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:35:14.621, end=2024-11-04T13:37:14.644} 2024-11-04 13:39:14.651 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:39:14.651 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:39:14.925 WARN 21062 --- [ocp-async-7,8c10cb483651460c,8b8e3b8dccd3da7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:14.973 INFO 21062 --- [ocp-async-8,ad89d071de229334,3285f9accdfcc7d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2b1634f033488486,ff8c85485d105f32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:15.111 INFO 21062 --- [ocp-schedules-29,5f9ab9792660359a,2339bc0d44a5dd08] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:39:15.833 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:15.834 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:16.925 WARN 21062 --- [ocp-async-1,3e48109fd2dcfb4f,027559a98a004070] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:16.973 INFO 21062 --- [ocp-async-5,690aeadc7c013976,a550995cbe5c9a7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:17.983 INFO 21062 --- [prometheus-minute-schedule-1,36f7fe00d02173cb,f77ac284d49e13b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:39:18.590346+08:00, lastProcessTime=2024-11-04T13:39:08.590134+08:00 2024-11-04 13:39:18.610 INFO 21062 --- [ocp-async-1,a839aef47154b052,e181b7b9f0f58c35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:39:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:39:18.590346+08:00 2024-11-04 13:39:18.834 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:18.834 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:18.925 WARN 21062 --- [ocp-async-4,98ef894bf4b10e1a,7227f834d8ea7d43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:18.973 INFO 21062 --- [ocp-async-2,60d8b73432b80de7,be0e423e456f00e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:19.685 INFO 21062 --- [ocp-schedules-9,26793275feb52236,898c97b1072f2d10] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:39:20.925 WARN 21062 --- [ocp-async-7,403abed533476f95,08ed3fdb03f52e90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:20.973 INFO 21062 --- [ocp-async-8,2015191d4c8480ea,56219b41a2aa46b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:20.983 INFO 21062 --- [prometheus-minute-schedule-1,7b668dba3fec1036,022282eb689bac5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:21.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:21.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:22.925 WARN 21062 --- [ocp-async-1,0fc59df74b167561,3e020f77cdfbef80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:22.973 INFO 21062 --- [ocp-async-5,ed2f5d257eefb2d1,32c1f9347f76470f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:23.623 INFO 21062 --- [ocp-schedules-11,65f073c7bc0061ea,806ecc2f20053d02] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:39:23.983 INFO 21062 --- [prometheus-minute-schedule-1,10a59138d94f3e02,911c70d5bb27c59a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:24.836 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:24.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:24.925 WARN 21062 --- [ocp-async-10,27bfa16a33837895,c1146e63336df1f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:24.973 INFO 21062 --- [ocp-async-6,f3acaca1dd9fe91f,2cce01d33a825ae7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:26.926 WARN 21062 --- [ocp-async-2,2edc5a348ca8ccc9,887205092f9cbf8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:26.974 INFO 21062 --- [ocp-async-9,410a64c3423f1888,344a9555aa4f93a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:26.983 INFO 21062 --- [prometheus-minute-schedule-1,85c3befba3900228,fd1e1f5b9602e432] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9c475bd1fb3648d7,50809c6b64226ad5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:39:27.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:27.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:28.498 INFO 21062 --- [ocp-schedules-3,0166d24423ebeb71,561c2aa13c1230c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:39:28.500 INFO 21062 --- [ocp-schedules-21,62c3d401f413e704,df9deab622aa7ec8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:39:28.500 INFO 21062 --- [ocp-schedules-4,fbfb85e1c3309fa5,a485ebe9e7f4c888] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:39:28.500 INFO 21062 --- [ocp-schedules-26,1e404bf0639a79e4,5ce3e1ce3cd2ef2a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:39:28.502 INFO 21062 --- [ocp-schedules-3,0166d24423ebeb71,561c2aa13c1230c8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:39:28.502 INFO 21062 --- [ocp-schedules-17,25a30501c2215585,610ff6eac91f93fb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:39:28.503 INFO 21062 --- [ocp-schedules-21,62c3d401f413e704,df9deab622aa7ec8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:28.504 INFO 21062 --- [ocp-schedules-26,1e404bf0639a79e4,5ce3e1ce3cd2ef2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:39:28.505 INFO 21062 --- [ocp-schedules-23,8d69bd42708dff0c,8558bc73b747ec72] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:39:28.505 INFO 21062 --- [ocp-schedules-4,fbfb85e1c3309fa5,a485ebe9e7f4c888] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:39:28.511 INFO 21062 --- [ocp-schedules-25,059fd88404385b5a,10c7e33fc681748e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:39:28.511 INFO 21062 --- [ocp-schedules-22,35b992c0d180c131,f787a1b91a328fd2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:39:28.511 INFO 21062 --- [ocp-schedules-25,059fd88404385b5a,10c7e33fc681748e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:39:28.511 INFO 21062 --- [ocp-schedules-29,67353a364ea6ce03,bac05bccd7c911a3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:39:28.511 INFO 21062 --- [ocp-schedules-22,35b992c0d180c131,f787a1b91a328fd2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:39:28.511 INFO 21062 --- [ocp-schedules-29,67353a364ea6ce03,bac05bccd7c911a3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:39:28.561 INFO 21062 --- [ocp-schedules-6,de519519a1fe7313,e0f9b4fab72af2ea] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:39:28.561+08:00, tenantSize=0 2024-11-04 13:39:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:39:28.590111+08:00, lastProcessTime=2024-11-04T13:39:18.590346+08:00 2024-11-04 13:39:28.603 INFO 21062 --- [ocp-schedules-20,09f7964a05bedc9a,c9a2febbacd8dce5] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:39:28.604 INFO 21062 --- [ocp-schedules-20,09f7964a05bedc9a,c9a2febbacd8dce5] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:39:28.610 INFO 21062 --- [ocp-async-7,31480c4a30d19210,96b739a0af350a7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:39:28.721 INFO 21062 --- [ocp-schedules-24,635c6592cddbe538,213c3bc63a764b9f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:39:28.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:39:28.590111+08:00 2024-11-04 13:39:28.894 INFO 21062 --- [ocp-schedules-15,78f4ef61095b4b15,e36ed48bd6f4cd64] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:39:28.894 INFO 21062 --- [ocp-schedules-15,78f4ef61095b4b15,e36ed48bd6f4cd64] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:39:28.921 INFO 21062 --- [ocp-schedules-17,7e42f4ff6371c8a0,edc11ff0b67e8675] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:39:28.926 WARN 21062 --- [ocp-async-5,c0706d8237fc26ae,d06fe2d4aea48785] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:28.954 INFO 21062 --- [ocp-schedules-25,fe634b9d25afec59,4d424d03cf44dcbf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:39:28.960 INFO 21062 --- [ocp-schedules-25,fe634b9d25afec59,4d424d03cf44dcbf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:39:28.960 INFO 21062 --- [ocp-schedules-25,fe634b9d25afec59,4d424d03cf44dcbf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:39:28.974 INFO 21062 --- [ocp-async-6,0e7a4edc8c8db854,69cb329c1f542763] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:29.148 INFO 21062 --- [ocp-schedules-6,d64d8b42a3c7fcda,1dcb9a5ca49a829d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:39:29.148 INFO 21062 --- [ocp-schedules-6,d64d8b42a3c7fcda,1dcb9a5ca49a829d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:39:29.194 INFO 21062 --- [ocp-schedules-6,d64d8b42a3c7fcda,1dcb9a5ca49a829d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:39:29.194 INFO 21062 --- [ocp-schedules-6,d64d8b42a3c7fcda,1dcb9a5ca49a829d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:39:29.199 INFO 21062 --- [ocp-schedules-6,d64d8b42a3c7fcda,1dcb9a5ca49a829d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:39:29.199 INFO 21062 --- [ocp-schedules-6,d64d8b42a3c7fcda,1dcb9a5ca49a829d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:39:29.689 INFO 21062 --- [ocp-schedules-19,260d8cc03bacd917,099af6fa77dc8947] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:39:29.983 INFO 21062 --- [prometheus-minute-schedule-1,52ce7b4d4e6ab588,18ae01bc4d8d71d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:39:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:39:30.348 INFO 21062 --- [ocp-schedules-1,4c7f5b9f8a2bb80c,38a9f33fe1ed2fc5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:39:30.355 INFO 21062 --- [ocp-schedules-1,4c7f5b9f8a2bb80c,38a9f33fe1ed2fc5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:39:30.355 INFO 21062 --- [ocp-schedules-1,4c7f5b9f8a2bb80c,38a9f33fe1ed2fc5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:39:30.357 INFO 21062 --- [ocp-schedules-2,f358fb33cf3f6802,4618d727b4196922] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:39:30.357 INFO 21062 --- [ocp-schedules-2,f358fb33cf3f6802,4618d727b4196922] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:39:30.384 INFO 21062 --- [ocp-schedules-2,f358fb33cf3f6802,4618d727b4196922] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:39:30.384 INFO 21062 --- [ocp-schedules-2,f358fb33cf3f6802,4618d727b4196922] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:39:30.385 INFO 21062 --- [ocp-schedules-10,d1dec2f20f743dea,9f6f5a177a24ceb2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:39:30.518 INFO 21062 --- [ocp-schedules-6,77e9626c68351dd4,ddcdfa762d43d511] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:39:30.518+08:00, clusters=[1] 2024-11-04 13:39:30.520 INFO 21062 --- [ocp-schedules-6,77e9626c68351dd4,ddcdfa762d43d511] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:39:30.554 INFO 21062 --- [ocp-schedules-28,d4d835159eee04cc,ba741b8eb096823f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:39:30.554 INFO 21062 --- [ocp-schedules-28,d4d835159eee04cc,ba741b8eb096823f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:39:30.554 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:39:30.554 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:39:30.564 INFO 21062 --- [ocp-schedules-28,d4d835159eee04cc,ba741b8eb096823f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:39:30.564 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:39:30.564 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:39:30.566 WARN 21062 --- [ocp-schedules-28,d4d835159eee04cc,ba741b8eb096823f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:39:30.566 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:39:30.566 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:39:30.567 INFO 21062 --- [ocp-schedules-28,d4d835159eee04cc,ba741b8eb096823f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:39:30.567 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:39:30.567 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:39:30.568 WARN 21062 --- [ocp-schedules-28,d4d835159eee04cc,ba741b8eb096823f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:39:30.568 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:39:30.569 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:39:30.575 ERROR 21062 --- [ocp-schedules-28,d4d835159eee04cc,ba741b8eb096823f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:39:30.657 INFO 21062 --- [ocp-schedules-11,4c1a0736a392e633,19bb78b51263ab3a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:39:30.657+08:00, clusters=[1] 2024-11-04 13:39:30.661 INFO 21062 --- [ocp-schedules-11,4c1a0736a392e633,19bb78b51263ab3a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:39:30.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:30.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:30.926 WARN 21062 --- [ocp-async-4,5c40fa64d098a22e,50ea5aec97d8c5eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:30.974 INFO 21062 --- [ocp-async-9,1f4b66cb90e0a7c3,c26638b384552a6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:32.926 WARN 21062 --- [ocp-async-7,793f654644fae7de,12995f98b5fadd72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:32.974 INFO 21062 --- [ocp-async-1,6f95845dba2958d7,614b6c56fc4db8b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f482b33a13b96f47,7c7ab15dd3d58af3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:33.090 INFO 21062 --- [ocp-schedules-12,afa768b0c7fcde59,fcce6ec31f2cea09] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:39:33.091 INFO 21062 --- [ocp-schedules-12,ff7eb7fa752e2e21,b0caba62ad489097] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:39:33.095 INFO 21062 --- [ocp-schedules-12,ff7eb7fa752e2e21,b0caba62ad489097] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:33.125 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:39:33.125 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:39:33.664 INFO 21062 --- [ocp-schedules-18,310d34f692977f93,e4c19359ff511734] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:39:33.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:33.839 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:34.679 INFO 21062 --- [ocp-schedules-22,90f2c67cbbbb36b6,bf1085bd8aed3d22] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:39:34.684 INFO 21062 --- [ocp-schedules-22,90f2c67cbbbb36b6,bf1085bd8aed3d22] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:34.774 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:39:34.776 ERROR 21062 --- [ocp-schedules-22,90f2c67cbbbb36b6,bf1085bd8aed3d22] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:39:34.776 INFO 21062 --- [ocp-schedules-22,90f2c67cbbbb36b6,bf1085bd8aed3d22] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:39:34.926 WARN 21062 --- [ocp-async-1,acfaa6c76e0bbc61,a6ea08fa640f5f90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:34.974 INFO 21062 --- [ocp-async-10,e3be8ec0dc885092,59c38e8630e97b3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:35.983 INFO 21062 --- [prometheus-minute-schedule-1,549f582937798d0b,180bf2e332eccea9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:36.839 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:36.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:36.927 WARN 21062 --- [ocp-async-10,af499cf42bcf455a,20e950211c0fc89a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:36.975 INFO 21062 --- [ocp-async-2,83223a67124b02b9,eac3c794165b4907] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:38.504 INFO 21062 --- [ocp-schedules-26,1e404bf0639a79e4,5ce3e1ce3cd2ef2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:39:38.505 INFO 21062 --- [ocp-schedules-21,62c3d401f413e704,df9deab622aa7ec8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:39:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:39:38.589708+08:00, lastProcessTime=2024-11-04T13:39:28.590111+08:00 2024-11-04 13:39:38.611 INFO 21062 --- [ocp-async-6,c747cc10291e5012,f9736a592ff4fae0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:39:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:39:38.589708+08:00 2024-11-04 13:39:38.927 WARN 21062 --- [ocp-async-7,2ab6926e90487d32,a19d4dc1f83e4666] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:38.975 INFO 21062 --- [ocp-async-3,a19f48c8329038b0,dc44023e9719f5b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:38.983 INFO 21062 --- [prometheus-minute-schedule-1,46c03d4675bd4bd8,e5d040858349ee3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:39.692 INFO 21062 --- [ocp-schedules-3,eefbcb3b2ec1430d,19e67ae20eb2f22d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:39:39.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:39.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:39:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:39:40.927 WARN 21062 --- [ocp-async-3,c1d25a0b29ad1ca6,d5f7bae3f8b57f50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:40.975 INFO 21062 --- [ocp-async-6,b513ff8332b52f53,adec75fea0da1d16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:41.983 INFO 21062 --- [prometheus-minute-schedule-1,7d82163300848a4d,10a477866e2672d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:42.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:42.842 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:42.927 WARN 21062 --- [ocp-async-6,494c53800b4a58a6,a3c6c9c6d7b4e685] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:42.975 INFO 21062 --- [ocp-async-9,329f3b2278b5a27f,483206c44015515a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:43.690 INFO 21062 --- [ocp-schedules-3,cf0823197b15181f,060c4d2d0162c6c3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:39:44.927 WARN 21062 --- [ocp-async-9,4ff7b860cb54e913,fa8ecc1326d47644] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:44.975 INFO 21062 --- [ocp-async-1,592412ef3f27bbf5,a13af34e78a24e00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:44.983 INFO 21062 --- [prometheus-minute-schedule-1,003f95839c49f3b6,cf4a4ef2ea96dc18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:45.842 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:45.843 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:46.927 WARN 21062 --- [ocp-async-1,c8076d2f5b551bb1,cb255f05cb4a8c33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:46.975 INFO 21062 --- [ocp-async-10,c0b6faa75c234f27,ed00b5aa7de35963] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fecf21a0c1c3ae05,57db1c6a62524ff9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:39:48.591600+08:00, lastProcessTime=2024-11-04T13:39:38.589708+08:00 2024-11-04 13:39:48.611 INFO 21062 --- [ocp-async-1,066710cd27f1674e,5aeb182de7b4470a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:39:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:39:48.591600+08:00 2024-11-04 13:39:48.843 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:48.844 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:48.928 WARN 21062 --- [ocp-async-6,d1800cdae117c47a,061beabc0e16f080] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:48.976 INFO 21062 --- [ocp-async-9,2ab236f04475c82a,9e8d4a7eed2bfcd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:49.695 INFO 21062 --- [ocp-schedules-13,23092b21c418a490,0360cadf680cb067] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:39:50.358 INFO 21062 --- [ocp-schedules-25,44a2ca4bb744ea4c,5c1751937ef602d7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:39:50.365 INFO 21062 --- [ocp-schedules-25,44a2ca4bb744ea4c,5c1751937ef602d7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:39:50.365 INFO 21062 --- [ocp-schedules-25,44a2ca4bb744ea4c,5c1751937ef602d7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:39:50.928 WARN 21062 --- [ocp-async-9,edf502514c72c330,d537e0fbfcbb7342] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:50.976 INFO 21062 --- [ocp-async-1,cbfc1e7f9bffb1d5,10ab9e2bed2d5019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4f82cc4d4b6faf11,ec0a91de18829127] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:51.844 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:51.845 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:52.928 WARN 21062 --- [ocp-async-1,882a96a1dfadc7ae,7d848555ffb4009e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:52.976 INFO 21062 --- [ocp-async-10,0d2030f962c03793,6ffdf11f44d176a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:53.732 INFO 21062 --- [ocp-schedules-2,1578930ce133aaa1,1e9512ffdff067ed] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:39:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4c6fdadf560c797b,eafda64a8d0e1e7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:54.845 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:54.846 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:54.928 WARN 21062 --- [ocp-async-10,133a3c35ca364ae4,d27d5b52f0575b9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:54.976 INFO 21062 --- [ocp-async-2,2b016e61415eadf6,f0305fcf6d8e9807] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:56.928 WARN 21062 --- [ocp-async-2,c7dde32a2ae122fa,cd90f2cb12989e37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:56.976 INFO 21062 --- [ocp-async-8,48dadbebaf6e6bb3,d562f8bff33551b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7a39be3f40037700,0f87da52645e878d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:39:57.846 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:39:57.847 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:39:58.500 INFO 21062 --- [ocp-schedules-2,e7990794b6fef893,6b547978f850ee08] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:39:58.500 INFO 21062 --- [ocp-schedules-14,274fb9544d67cc50,c42b446c179d3c50] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:39:58.503 INFO 21062 --- [ocp-schedules-4,35f1274011a90c46,cff84c931acfa4ae] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:39:58.504 INFO 21062 --- [ocp-schedules-2,e7990794b6fef893,6b547978f850ee08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:58.504 INFO 21062 --- [ocp-schedules-14,274fb9544d67cc50,c42b446c179d3c50] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:58.505 INFO 21062 --- [ocp-schedules-16,d57db817d0957718,fe1df616076a80f0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:39:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:39:58.512 INFO 21062 --- [ocp-schedules-27,a4e7349ffd2ab48c,31da95ca21aff2f8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:39:58.512 INFO 21062 --- [ocp-schedules-30,3099bbd50a689391,0841e18319e18984] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:39:58.512 INFO 21062 --- [ocp-schedules-27,a4e7349ffd2ab48c,31da95ca21aff2f8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:39:58.512 INFO 21062 --- [ocp-schedules-30,3099bbd50a689391,0841e18319e18984] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:39:58.512 INFO 21062 --- [ocp-schedules-15,643bcbc92f70d3ba,6b4f5a679a99de51] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:39:58.512 INFO 21062 --- [ocp-schedules-15,643bcbc92f70d3ba,6b4f5a679a99de51] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:39:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:39:58.590113+08:00, lastProcessTime=2024-11-04T13:39:48.591600+08:00 2024-11-04 13:39:58.611 INFO 21062 --- [ocp-async-2,f94d99b10d9f3db8,4d387c303a386227] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:39:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:39:58.590113+08:00 2024-11-04 13:39:58.835 INFO 21062 --- [ocp-schedules-5,e0ec11ae5a5dc455,c6edeb6e45ff3d78] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:39:58.839 INFO 21062 --- [ocp-schedules-5,e0ec11ae5a5dc455,c6edeb6e45ff3d78] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:39:58.839 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:39:58.924 INFO 21062 --- [ocp-schedules-28,c04fc5efdec32dc8,50e7c47a2daa41df] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:39:58.929 WARN 21062 --- [ocp-async-1,c3d57e8950bdfaf3,f5fd4ced55fe22f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:39:58.977 INFO 21062 --- [ocp-async-10,eb1c0d1b80025c4f,582f7acff99805c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:39:59.698 INFO 21062 --- [ocp-schedules-13,d5a096d93de42a90,699a503d34c676fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:39:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d4b24ca711b317c0,ee0f75d7f68f8826] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698800, interval=60 2024-11-04 13:40:00.716 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d7f2c60733cf1034] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:35:01.0, nextRun=2024-11-04 13:40:00.0, currentDate=Mon Nov 04 13:40:00 CST 2024 2024-11-04 13:40:00.716 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d7f2c60733cf1034] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:35:01.0, nextRunTime=2024-11-04 13:40:00.0} 2024-11-04 13:40:00.748 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d7f2c60733cf1034] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:40:00.780 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d7f2c60733cf1034] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:40:00.780 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,d7f2c60733cf1034] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:40:00.847 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:00.848 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:00.929 WARN 21062 --- [ocp-async-10,8a51eaa7f5b76487,5c0b2833086b3d76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:00.977 INFO 21062 --- [ocp-async-2,5e2aebb517cdda0c,021ae22b1c51b056] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:02.929 WARN 21062 --- [ocp-async-2,7f7a39b320fa4f8f,3da2cae50bd2727a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:02.977 INFO 21062 --- [ocp-async-7,f2cd61701e59efc3,debde7e59820d770] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e5c00a96fea79fe6,3ceeb271d9be11e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:03.125 INFO 21062 --- [ocp-schedules-12,ff7eb7fa752e2e21,b0caba62ad489097] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:40:03.125 INFO 21062 --- [ocp-schedules-12,aa9d260d536554b8,a8f01732caf1ea40] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:40:03.129 INFO 21062 --- [ocp-schedules-12,aa9d260d536554b8,a8f01732caf1ea40] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:03.158 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:40:03.158 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:40:03.757 INFO 21062 --- [ocp-schedules-4,b1f336d44a58ca03,437dbef95d7b6763] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:40:03.848 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:03.849 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:04.849 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,1cc333f94a7df856] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:30:04.0, nextRun=2024-11-04 13:40:04.0, currentDate=Mon Nov 04 13:40:04 CST 2024 2024-11-04 13:40:04.849 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,1cc333f94a7df856] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 13:30:04.0, nextRunTime=2024-11-04 13:40:04.0} 2024-11-04 13:40:04.855 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,1cc333f94a7df856] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 13:40:04.855 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,1cc333f94a7df856] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T13:40:04.855+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 13:40:04.929 WARN 21062 --- [ocp-async-7,a754f8bbdf74006d,411d508d87de20f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:04.977 INFO 21062 --- [ocp-async-3,ad783e699d0a0a29,d2c69d5655ae7368] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:05.093 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,f26fef245111a80c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 75, operation:EXECUTE, state: RUNNING 2024-11-04 13:40:05.111 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=75, context=Context{parallelIdx=-1, stringMap={task_instance_id=10, task_operation=execute, latest_execution_start_time=2024-11-04T13:40:05.086+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=75}, listMap={}}, executor=10.46.128.24 2024-11-04 13:40:05.116 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:40:05.117 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:40:05.118 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.119 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:40:05.120 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.120 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.121 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.121 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:40:05.136 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.137 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.137 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.137 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:40:05.150 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.151 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.151 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.152 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:40:05.166 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.166 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.167 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.167 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:40:05.180 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.180 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.181 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.181 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:40:05.187 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.187 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.187 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.188 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:40:05.192 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.192 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.193 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.193 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:40:05.197 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.198 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.199 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.199 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:40:05.205 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.206 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.206 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.207 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:40:05.212 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.212 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.213 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.213 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:40:05.220 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.220 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.221 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.221 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:40:05.230 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.230 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.231 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.231 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:40:05.245 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.245 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.246 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.247 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:40:05.251 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.252 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.252 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.253 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:40:05.258 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.258 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.259 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.259 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:40:05.264 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.265 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.265 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.266 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:40:05.270 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.270 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.271 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.271 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:40:05.276 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.276 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.277 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.277 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:40:05.281 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.281 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.282 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.282 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:40:05.287 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.288 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.288 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.288 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:40:05.292 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.293 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.293 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.294 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:40:05.299 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.299 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.299 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.300 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:40:05.304 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.304 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.305 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.305 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:40:05.309 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.310 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.310 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.311 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:40:05.316 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.316 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.317 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.317 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:40:05.322 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.322 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.323 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.323 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:40:05.327 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.328 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.328 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.329 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:40:05.333 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.333 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.334 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.334 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:40:05.338 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.339 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.339 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.340 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:40:05.344 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.344 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.345 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.345 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:40:05.349 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.350 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.350 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.351 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:40:05.356 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.356 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.357 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.357 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:40:05.362 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.363 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.364 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.364 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:40:05.369 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.369 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.370 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.370 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:40:05.379 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.379 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.380 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.380 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:40:05.386 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.386 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.387 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.387 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:40:05.392 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.393 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.393 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.394 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:40:05.399 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.399 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.400 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.401 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:40:05.404 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.404 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.405 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.405 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:40:05.409 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.409 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.409 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.410 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:40:05.414 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.414 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.415 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.415 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:40:05.418 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.419 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.419 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.419 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:40:05.429 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.429 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.430 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.430 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:40:05.449 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.450 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.450 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.451 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:40:05.456 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.456 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.457 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.457 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:40:05.463 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.463 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.464 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.464 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:40:05.469 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.470 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.470 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.471 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:40:05.475 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.475 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.476 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.476 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:40:05.480 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.480 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.481 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.481 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:40:05.485 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:40:05.485 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:40:05.487 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.487 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:40:05.489 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.489 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.490 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.490 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:40:05.495 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.496 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:40:05.496 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:05.497 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:40:05.501 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:40:05.509 INFO 21062 --- [subtask-executor8,dd193f8f67a5a423,941a0531d211257c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 75, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 13:40:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0e336de746696a9f,bdf28200026d79ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:06.248 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:10, status:SUCCESS 2024-11-04 13:40:06.250 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:10, status:SUCCESS 2024-11-04 13:40:06.849 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:06.849 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:06.929 WARN 21062 --- [ocp-async-3,f4bd08198fcdfc7b,e96fd52ff7943e32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:06.977 INFO 21062 --- [ocp-async-6,4a84b42ac35b4082,cdfe3fb8abc237a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:08.505 INFO 21062 --- [ocp-schedules-14,274fb9544d67cc50,c42b446c179d3c50] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:40:08.506 INFO 21062 --- [ocp-schedules-2,e7990794b6fef893,6b547978f850ee08] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:40:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:40:08.590348+08:00, lastProcessTime=2024-11-04T13:39:58.590113+08:00 2024-11-04 13:40:08.611 INFO 21062 --- [ocp-async-3,66c6722c9ad5e85c,5ffc005bd9822ad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:40:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:40:08.590348+08:00 2024-11-04 13:40:08.840 INFO 21062 --- [ocp-schedules-5,e0ec11ae5a5dc455,c6edeb6e45ff3d78] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:40:08.929 WARN 21062 --- [ocp-async-2,9ecce3bae05870a0,c7d0ee76ec1c6d9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:08.977 INFO 21062 --- [ocp-async-8,dad99cdc3ffb347c,798eb0ccc2582097] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:08.983 INFO 21062 --- [prometheus-minute-schedule-1,5e7cd94912c3bdb1,9c715f4cb25a2474] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:09.700 INFO 21062 --- [ocp-schedules-23,b99a398a2244e3ee,088e09b5dfc372f3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:40:09.850 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:09.850 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:40:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:40:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:40:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:40:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:40:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:40:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:40:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:40:10.368 INFO 21062 --- [ocp-schedules-27,f02ddb8fc1f01ab6,87804787aff4553c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:40:10.375 INFO 21062 --- [ocp-schedules-27,f02ddb8fc1f01ab6,87804787aff4553c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:40:10.375 INFO 21062 --- [ocp-schedules-27,f02ddb8fc1f01ab6,87804787aff4553c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:40:10.930 WARN 21062 --- [ocp-async-8,cbad7cf841ea5d4a,0fee83b0f6641268] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:10.978 INFO 21062 --- [ocp-async-5,e7a7c048ebc0b834,6ebf4ec335d862b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:11.313 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=5, affectRows=5 2024-11-04 13:40:11.313 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=52, affectRows=52 2024-11-04 13:40:11.313 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=16, affectRows=16 2024-11-04 13:40:11.313 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=5, validCount=5 2024-11-04 13:40:11.314 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=16, validCount=16 2024-11-04 13:40:11.314 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=1, affectRows=1 2024-11-04 13:40:11.314 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=1, validCount=1 2024-11-04 13:40:11.315 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=51, validCount=51 2024-11-04 13:40:11.315 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=15, affectRows=15 2024-11-04 13:40:11.315 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=14, validCount=14 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=12, affectRows=12 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=35, affectRows=35 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=12, validCount=12 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=35, validCount=35 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=4, affectRows=4 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=4, validCount=4 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=10, affectRows=10 2024-11-04 13:40:11.316 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=10, validCount=10 2024-11-04 13:40:11.317 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=19, affectRows=19 2024-11-04 13:40:11.317 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=19, affectRows=19 2024-11-04 13:40:11.317 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=19, validCount=19 2024-11-04 13:40:11.317 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=31, affectRows=31 2024-11-04 13:40:11.317 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=19, validCount=19 2024-11-04 13:40:11.318 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=31, validCount=31 2024-11-04 13:40:11.318 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=25, affectRows=25 2024-11-04 13:40:11.318 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=46, affectRows=46 2024-11-04 13:40:11.318 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698740, nodeCount=32, affectRows=32 2024-11-04 13:40:11.318 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=25, validCount=25 2024-11-04 13:40:11.318 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=32, validCount=32 2024-11-04 13:40:11.318 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698740, nodeCount=45, validCount=45 2024-11-04 13:40:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b6800e228c4c30b8,320a8152b843211b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:12.851 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:12.851 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:12.930 WARN 21062 --- [ocp-async-3,519c97cbcff177ca,eaefb2ebf00cac86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:12.978 INFO 21062 --- [ocp-async-4,032b22f25cb48e6e,32cffbd747eac7cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:13.800 INFO 21062 --- [ocp-schedules-10,a9547896462da242,fdafa0301710dcba] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,f51653df53aab688,76cc5ac7cf2c0a7e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,f51653df53aab688,76cc5ac7cf2c0a7e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:40:14.930 WARN 21062 --- [ocp-async-6,5ec85b05e0438a6d,77b8e267eb543ce5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:14.978 INFO 21062 --- [ocp-async-9,a41b683ced809524,bc70681caa5afbb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:14.983 INFO 21062 --- [prometheus-minute-schedule-1,92153fc393287156,d8e5528735d9af03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:15.852 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:15.853 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:16.930 WARN 21062 --- [ocp-async-7,9ed2c835c5a591b1,1457a39b2de9da44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:16.978 INFO 21062 --- [ocp-async-1,6f381d8f799eafc4,19f8401ae676f276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:17.983 INFO 21062 --- [prometheus-minute-schedule-1,dbf0d3ff2b0092b5,57e201748e2669d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:40:18.589676+08:00, lastProcessTime=2024-11-04T13:40:08.590348+08:00 2024-11-04 13:40:18.611 INFO 21062 --- [ocp-async-7,ff9ed8cd89f7567d,728603a6355cac0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:40:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:40:18.589676+08:00 2024-11-04 13:40:18.853 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:18.854 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:18.930 WARN 21062 --- [ocp-async-3,b4fc53662ddeaae5,549758e452ea11e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:18.978 INFO 21062 --- [ocp-async-4,e4c5c720d41bcc95,5cea7ebc93d126f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:19.703 INFO 21062 --- [ocp-schedules-17,a7c60a4c0b9476db,cbedc0abf77ae6bd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:40:20.931 WARN 21062 --- [ocp-async-6,f87f1d944e727799,d14d6f2c51b46990] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:20.979 INFO 21062 --- [ocp-async-9,0f9cc5b5c6ed17b7,12eaf9be07d205aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e470b824b0d979ff,6c6d67b1f003fc02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:21.854 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:21.855 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:22.931 WARN 21062 --- [ocp-async-7,2d7d7b2e7d9c1e07,58ab0d22996c3d0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:22.979 INFO 21062 --- [ocp-async-1,ccc2344d9a745a63,54f3fbf18b41c2d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:23.827 INFO 21062 --- [ocp-schedules-3,74ffa7f92692275a,37cf01be4befebe7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:40:23.983 INFO 21062 --- [prometheus-minute-schedule-1,52637bb6e55c15bd,78da2dc01ea35b13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:24.855 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:24.856 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:24.931 WARN 21062 --- [ocp-async-5,882eedbf4bfe2b48,3f39d357461ef95e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:24.979 INFO 21062 --- [ocp-async-10,41bb23f240c05d7e,79b412978eec7f5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:26.931 WARN 21062 --- [ocp-async-4,c046eb56a988f5c6,ccf1a849a4426dda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:26.979 INFO 21062 --- [ocp-async-2,8dec086353083416,34edb33f3b81e6b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:26.983 INFO 21062 --- [prometheus-minute-schedule-1,efdc8ad0d3663f8e,e1f7dd7081242f22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a41467b5395f57a7,9a82c6380267456b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:40:27.856 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:27.857 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:28.498 INFO 21062 --- [ocp-schedules-27,4c1e31bb56eba7e8,0c5da3f35e8c11a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:40:28.500 INFO 21062 --- [ocp-schedules-25,4714094f25b4d676,3d8615d59d075960] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:40:28.500 INFO 21062 --- [ocp-schedules-18,bfd6a7d99155924f,71ed745c3c68cc74] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:40:28.500 INFO 21062 --- [ocp-schedules-11,3dea148513341762,84ecbe2beab0aeda] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:40:28.502 INFO 21062 --- [ocp-schedules-27,4c1e31bb56eba7e8,0c5da3f35e8c11a3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:40:28.502 INFO 21062 --- [ocp-schedules-16,b0fa91e4a56c9d0f,042dff5b9b871f9f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:40:28.504 INFO 21062 --- [ocp-schedules-11,3dea148513341762,84ecbe2beab0aeda] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:28.504 INFO 21062 --- [ocp-schedules-19,eb739e254253a57b,e91caa62f3b372d4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:40:28.505 INFO 21062 --- [ocp-schedules-25,4714094f25b4d676,3d8615d59d075960] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:28.505 INFO 21062 --- [ocp-schedules-18,bfd6a7d99155924f,71ed745c3c68cc74] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:40:28.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:40:28.510 INFO 21062 --- [ocp-schedules-10,25fba2755ab2d595,54956396f8b24759] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:40:28.510 INFO 21062 --- [ocp-schedules-23,d93928e237d72257,03172d4c7e2c4181] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:40:28.510 INFO 21062 --- [ocp-schedules-23,d93928e237d72257,03172d4c7e2c4181] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:40:28.510 INFO 21062 --- [ocp-schedules-10,25fba2755ab2d595,54956396f8b24759] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:40:28.510 INFO 21062 --- [ocp-schedules-7,2082b42f8273b11b,e52feef03aeef058] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:40:28.510 INFO 21062 --- [ocp-schedules-7,2082b42f8273b11b,e52feef03aeef058] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:40:28.568 INFO 21062 --- [ocp-schedules-26,9950d15dcae9e2c0,54ab02786cf38f76] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1288,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":321,"seriesIdKeySize":322,"secondWriteQueue":0,"minuteWriteCacheSize":319} 2024-11-04 13:40:28.568 INFO 21062 --- [ocp-schedules-26,9950d15dcae9e2c0,54ab02786cf38f76] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=13, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=13, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:40:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:40:28.589081+08:00, lastProcessTime=2024-11-04T13:40:18.589676+08:00 2024-11-04 13:40:28.612 INFO 21062 --- [ocp-async-6,eb5941efc646aec9,b5a051de44209a66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:40:28.725 INFO 21062 --- [ocp-schedules-9,34b0d4ee81a05238,4e366e7de020e3e2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:40:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:40:28.589081+08:00 2024-11-04 13:40:28.899 INFO 21062 --- [ocp-schedules-5,17de07fad0c2b5a0,4b8bf85d55dd0952] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:40:28.899 INFO 21062 --- [ocp-schedules-5,17de07fad0c2b5a0,4b8bf85d55dd0952] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:40:28.927 INFO 21062 --- [ocp-schedules-18,eefe15173c30dce1,5abf956837df5a64] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:40:28.931 WARN 21062 --- [ocp-async-1,8fea20830a6a02e1,f6c7df621a64eb0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:28.960 INFO 21062 --- [ocp-schedules-23,e7ac64bb13532e5a,d5302a11688cf596] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:40:28.965 INFO 21062 --- [ocp-schedules-23,e7ac64bb13532e5a,d5302a11688cf596] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:40:28.965 INFO 21062 --- [ocp-schedules-23,e7ac64bb13532e5a,d5302a11688cf596] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:40:28.979 INFO 21062 --- [ocp-async-10,dd5bd8d0b78ee35c,879436f7091ebc5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:29.015 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:40:29.015 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:40:29.023 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:40:29.023 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:40:29.023 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:40:29.034 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:40:29.034 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:40:29.034 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:40:29.038 INFO 21062 --- [ocp-schedules-27,5bd1fa0e74ded608,b5a3b7cc04ca96d6] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:40:29.205 INFO 21062 --- [ocp-schedules-26,786eb02bd2039229,2b132846237f98b6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:40:29.205 INFO 21062 --- [ocp-schedules-26,786eb02bd2039229,2b132846237f98b6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:40:29.206 INFO 21062 --- [ocp-schedules-26,786eb02bd2039229,2b132846237f98b6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:40:29.206 INFO 21062 --- [ocp-schedules-26,786eb02bd2039229,2b132846237f98b6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:40:29.206 INFO 21062 --- [ocp-schedules-26,786eb02bd2039229,2b132846237f98b6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:40:29.206 INFO 21062 --- [ocp-schedules-26,786eb02bd2039229,2b132846237f98b6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:40:29.706 INFO 21062 --- [ocp-schedules-24,e1ad146d2bbfe5e8,bfcce304c52e0f40] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:40:29.983 INFO 21062 --- [prometheus-minute-schedule-1,cf7adb5b8a015f62,23b07f996e9d11a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:40:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:40:30.378 INFO 21062 --- [ocp-schedules-27,36a4ca3d0b4d7759,aba1dce2cca80153] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:40:30.383 INFO 21062 --- [ocp-schedules-27,36a4ca3d0b4d7759,aba1dce2cca80153] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:40:30.384 INFO 21062 --- [ocp-schedules-27,36a4ca3d0b4d7759,aba1dce2cca80153] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:40:30.384 INFO 21062 --- [ocp-schedules-3,759ba5e4069a7ee3,ae4c5e53878d218b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:40:30.384 INFO 21062 --- [ocp-schedules-3,759ba5e4069a7ee3,ae4c5e53878d218b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:40:30.411 INFO 21062 --- [ocp-schedules-3,759ba5e4069a7ee3,ae4c5e53878d218b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:40:30.411 INFO 21062 --- [ocp-schedules-3,759ba5e4069a7ee3,ae4c5e53878d218b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:40:30.411 INFO 21062 --- [ocp-schedules-29,534a0bead7e6c661,1a61b1f78c99abab] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:40:30.545 INFO 21062 --- [ocp-schedules-30,71903852465589d6,d513ef92dd09887d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:40:30.545+08:00, clusters=[1] 2024-11-04 13:40:30.547 INFO 21062 --- [ocp-schedules-30,71903852465589d6,d513ef92dd09887d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:40:30.582 INFO 21062 --- [ocp-schedules-20,24856a2bdf87d363,e3ba2a8c5bacb1ec] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:40:30.582 INFO 21062 --- [ocp-schedules-20,24856a2bdf87d363,e3ba2a8c5bacb1ec] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:40:30.582 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:40:30.582 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:40:30.592 INFO 21062 --- [ocp-schedules-20,24856a2bdf87d363,e3ba2a8c5bacb1ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:40:30.592 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:40:30.593 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:40:30.594 WARN 21062 --- [ocp-schedules-20,24856a2bdf87d363,e3ba2a8c5bacb1ec] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:40:30.594 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:40:30.595 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:40:30.595 INFO 21062 --- [ocp-schedules-20,24856a2bdf87d363,e3ba2a8c5bacb1ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:40:30.595 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:40:30.595 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:40:30.596 WARN 21062 --- [ocp-schedules-20,24856a2bdf87d363,e3ba2a8c5bacb1ec] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:40:30.596 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:40:30.596 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:40:30.597 INFO 21062 --- [ocp-schedules-6,7e6898549e00d909,b9f5b27c4abd9d78] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:40:30.600 INFO 21062 --- [ocp-schedules-6,7e6898549e00d909,b9f5b27c4abd9d78] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:30.605 ERROR 21062 --- [ocp-schedules-20,24856a2bdf87d363,e3ba2a8c5bacb1ec] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:40:30.655 ERROR 21062 --- [ocp-schedules-6,7e6898549e00d909,b9f5b27c4abd9d78] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:40:30.655 INFO 21062 --- [ocp-schedules-6,7e6898549e00d909,b9f5b27c4abd9d78] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:40:30.684 INFO 21062 --- [ocp-schedules-4,a6f6a4ff18a46c36,ccd4a46ec5eb8b27] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:40:30.684+08:00, clusters=[1] 2024-11-04 13:40:30.687 INFO 21062 --- [ocp-schedules-4,a6f6a4ff18a46c36,ccd4a46ec5eb8b27] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:40:30.857 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:30.858 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:30.931 WARN 21062 --- [ocp-async-3,16ce7716ad7522db,54958ee98e343344] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:30.980 INFO 21062 --- [ocp-async-6,2210e713ea9d4d79,b9cf62414474ce3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:32.932 WARN 21062 --- [ocp-async-2,6f19d65413737dc3,f6285e8f9b0b36ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:32.980 INFO 21062 --- [ocp-async-7,b8722e5c3bcc9aa1,6b9d2d1fd8f25d36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8e9129b67ff06d61,443a42351239b75c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:33.158 INFO 21062 --- [ocp-schedules-12,aa9d260d536554b8,a8f01732caf1ea40] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:40:33.158 INFO 21062 --- [ocp-schedules-12,f0d0caa71283be08,c73ac5495b37621d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:40:33.162 INFO 21062 --- [ocp-schedules-12,f0d0caa71283be08,c73ac5495b37621d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:33.191 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:40:33.191 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:40:33.858 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:33.858 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:33.864 INFO 21062 --- [ocp-schedules-5,fe91c2b2a162dcf2,1619c1ca6ab0a82f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:40:34.776 INFO 21062 --- [ocp-schedules-13,54040ffb18999055,df91751ab65b4e27] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:40:34.780 INFO 21062 --- [ocp-schedules-13,54040ffb18999055,df91751ab65b4e27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:34.865 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:40:34.867 ERROR 21062 --- [ocp-schedules-13,54040ffb18999055,df91751ab65b4e27] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:40:34.867 INFO 21062 --- [ocp-schedules-13,54040ffb18999055,df91751ab65b4e27] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:40:34.932 WARN 21062 --- [ocp-async-7,eba95a6c99c929e4,acb6d9231f4abe83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:34.980 INFO 21062 --- [ocp-async-5,7702a443f8b5f6db,bcd330678b60c030] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:35.983 INFO 21062 --- [prometheus-minute-schedule-1,2d1ddb565dc25a33,dc5084da17f705b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:36.859 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:36.859 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:36.932 WARN 21062 --- [ocp-async-5,a250f9f219088c10,c59c19a13db850a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:36.980 INFO 21062 --- [ocp-async-4,12c09bf076cbf258,a65c11d3bbec5ba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:38.505 INFO 21062 --- [ocp-schedules-11,3dea148513341762,84ecbe2beab0aeda] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:40:38.506 INFO 21062 --- [ocp-schedules-25,4714094f25b4d676,3d8615d59d075960] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:40:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:40:38.589727+08:00, lastProcessTime=2024-11-04T13:40:28.589081+08:00 2024-11-04 13:40:38.612 INFO 21062 --- [ocp-async-10,03b6dd3157f4ef05,6e67a8b2f4980f4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:40:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:40:38.589727+08:00 2024-11-04 13:40:38.932 WARN 21062 --- [ocp-async-2,df8549981a62a852,acfd90c31ea5ba4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:38.980 INFO 21062 --- [ocp-async-8,b8e3ff7335a97542,803eb0d2893234bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:38.983 INFO 21062 --- [prometheus-minute-schedule-1,6a5e9dc9a3b85d31,60c90233e4945403] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:39.709 INFO 21062 --- [ocp-schedules-14,e59781d0b7a95741,f2e2af362425c14c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:40:39.859 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:39.861 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:40:40.004 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:40:40.932 WARN 21062 --- [ocp-async-8,fa4fcd6ed09eabce,66ef3390586cd6e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:40.981 INFO 21062 --- [ocp-async-10,d182f87a52fb8cef,a3962eaffdc8e96d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:40:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:40:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:40:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:40:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:40:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:40:41.983 INFO 21062 --- [prometheus-minute-schedule-1,e7bd5921c482d6a5,621c0cfddb13b9ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:42.861 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:42.862 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:42.933 WARN 21062 --- [ocp-async-10,ae3fe26fb5d6face,3da0bb4d0ac9cf6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:42.981 INFO 21062 --- [ocp-async-6,7096621bdac3bacd,40b481919678d5e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:43.890 INFO 21062 --- [ocp-schedules-29,94bccbf057d6010c,a87cfa7933568ca7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:40:44.933 WARN 21062 --- [ocp-async-6,256b3e020f8d00f0,63f66d8d0dff3981] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:44.981 INFO 21062 --- [ocp-async-7,46f6d16025c98303,890c03c1b4a5d432] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f84ca8d9455360f1,0b2a20e9fdb44c06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:45.862 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:45.864 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:46.933 WARN 21062 --- [ocp-async-7,058d4f6a10785bc2,7aca653f3db9edcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:46.981 INFO 21062 --- [ocp-async-5,3a5973a0b8ca8e9e,e20a8efd052a1db7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:47.983 INFO 21062 --- [prometheus-minute-schedule-1,c6407137cad8512a,f250bd2647b2f08a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:40:48.589719+08:00, lastProcessTime=2024-11-04T13:40:38.589727+08:00 2024-11-04 13:40:48.612 INFO 21062 --- [ocp-async-7,303859d5b19e5bac,d7431eb04bf17c64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:40:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:40:48.589719+08:00 2024-11-04 13:40:48.864 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:48.864 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:48.933 WARN 21062 --- [ocp-async-10,9bc7f671f50c773d,45d3e73a47641963] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:48.981 INFO 21062 --- [ocp-async-6,c2ce6022b60ad66f,63dd08b91abb72f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:49.712 INFO 21062 --- [ocp-schedules-15,d8ea0091e8c86b8c,5648ef63347f6e6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:40:50.387 INFO 21062 --- [ocp-schedules-10,886a5528d2d841dd,497bbb1b73ffc449] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:40:50.394 INFO 21062 --- [ocp-schedules-10,886a5528d2d841dd,497bbb1b73ffc449] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:40:50.394 INFO 21062 --- [ocp-schedules-10,886a5528d2d841dd,497bbb1b73ffc449] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:40:50.933 WARN 21062 --- [ocp-async-6,8d0f4accd861e79b,c5c5585f96fa60d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:50.981 INFO 21062 --- [ocp-async-7,cb4559261ef16d81,fb584ac972ee5575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:50.983 INFO 21062 --- [prometheus-minute-schedule-1,519b9d3ae9eefca7,906b7498cbe8bb00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:51.864 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:51.865 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:52.933 WARN 21062 --- [ocp-async-7,669cc8c5f2f6ad2b,9151343a25c4273f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:52.982 INFO 21062 --- [ocp-async-5,038f3e6e0d2a1dba,8ff42e34891dcbb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:53.935 INFO 21062 --- [ocp-schedules-13,f32330f3daec9223,2856df184639b962] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:40:53.983 INFO 21062 --- [prometheus-minute-schedule-1,24023355fe4b6cde,b88cdf2b6c008aa7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:54.865 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:54.866 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:54.933 WARN 21062 --- [ocp-async-5,76093652ce6d1e61,26f44b02cf1f50d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:54.982 INFO 21062 --- [ocp-async-3,b7e55ae0de89eb1a,cae8e5188bc06e3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:56.934 WARN 21062 --- [ocp-async-3,641a56a0562d771f,1dffbb07b44c8fa9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:56.982 INFO 21062 --- [ocp-async-9,4cd0e39b76fad3cc,67ccb31a4e361f54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:56.983 INFO 21062 --- [prometheus-minute-schedule-1,068f48b442da64b3,1f5afba534cf7901] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:40:57.866 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:40:57.867 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:40:58.005 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f541f78a8e957aa9,a17521c679529bcd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /pre_login_loading?null, client=10.130.47.17, traceId=f541f78a8e957aa9, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.WebController.preLoginLoading(), args=] 2024-11-04 13:40:58.014 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f541f78a8e957aa9,a17521c679529bcd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /pre_login_loading client=10.130.47.17, traceId=f541f78a8e957aa9, duration=38 ms] 2024-11-04 13:40:58.500 INFO 21062 --- [ocp-schedules-10,2d8fe82bc020ac92,29f6b98385a2eb8e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:40:58.500 INFO 21062 --- [ocp-schedules-15,b5d981099c9df7bd,5df803ab6f43f444] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:40:58.502 INFO 21062 --- [ocp-schedules-8,b1b817db5f0fc131,b09045d1053b2e3c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:40:58.504 INFO 21062 --- [ocp-schedules-10,2d8fe82bc020ac92,29f6b98385a2eb8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:58.505 INFO 21062 --- [ocp-schedules-15,b5d981099c9df7bd,5df803ab6f43f444] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:40:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:40:58.506 INFO 21062 --- [ocp-schedules-13,d07cf379dd0d61a4,13234cfe0fd261b5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:40:58.510 INFO 21062 --- [ocp-schedules-6,0c2ca81c87f2463d,23e4af70a495c3c2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:40:58.510 INFO 21062 --- [ocp-schedules-28,185f3ab91563ec94,b3fd402dba089070] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:40:58.510 INFO 21062 --- [ocp-schedules-5,a1158b12893a37b1,3752db56b51cb507] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:40:58.510 INFO 21062 --- [ocp-schedules-6,0c2ca81c87f2463d,23e4af70a495c3c2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:40:58.510 INFO 21062 --- [ocp-schedules-28,185f3ab91563ec94,b3fd402dba089070] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:40:58.510 INFO 21062 --- [ocp-schedules-5,a1158b12893a37b1,3752db56b51cb507] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:40:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:40:58.590697+08:00, lastProcessTime=2024-11-04T13:40:48.589719+08:00 2024-11-04 13:40:58.612 INFO 21062 --- [ocp-async-3,18d39d2564a168e6,1d646f56595ee019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:40:58.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:40:58.590697+08:00 2024-11-04 13:40:58.930 INFO 21062 --- [ocp-schedules-25,ee2a441bf40a76b2,fde05104cad02c6d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:40:58.934 WARN 21062 --- [ocp-async-7,cc0a1a92b5357219,441a084e939adf68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:40:58.982 INFO 21062 --- [ocp-async-5,b9c718d7a9fe0f1e,9eb5e5824a35e3f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:40:59.715 INFO 21062 --- [ocp-schedules-19,403e8fba4590903f,ccf811a56dae629d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:40:59.983 INFO 21062 --- [prometheus-minute-schedule-1,75befbd4dabdca34,9db0688fe7d09f73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698860, interval=60 2024-11-04 13:41:00.867 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:00.869 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:00.934 WARN 21062 --- [ocp-async-5,2fbb5e972a76bcd2,530bc3c44ddc4274] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:00.982 INFO 21062 --- [ocp-async-3,4ceff7b2ff8d6e33,7e60a5662d388309] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:01.227 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ff5b393b9b7679f5,fa64885aec4d0fda] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /favicon.ico?null, client=10.130.47.17, traceId=ff5b393b9b7679f5, method=void com.oceanbase.ocp.server.common.controller.web.WebController.dummyFavicon(), args=] 2024-11-04 13:41:01.227 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ff5b393b9b7679f5,fa64885aec4d0fda] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /favicon.ico client=10.130.47.17, traceId=ff5b393b9b7679f5, duration=5 ms] 2024-11-04 13:41:01.252 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c9ae8ccd73c94570,621fc98dd565513f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.130.47.17, traceId=c9ae8ccd73c94570, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2024-11-04 13:41:01.297 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c9ae8ccd73c94570,621fc98dd565513f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.130.47.17, traceId=c9ae8ccd73c94570, duration=61 ms] 2024-11-04 13:41:02.061 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,0819882b940cda37,465466a012e08530] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.130.47.17, traceId=0819882b940cda37, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2024-11-04 13:41:02.065 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,0819882b940cda37,465466a012e08530] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.130.47.17, traceId=0819882b940cda37, duration=11 ms] 2024-11-04 13:41:02.329 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:41:02.332 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-2,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:41:02.333 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:41:02.934 WARN 21062 --- [ocp-async-3,a536e12bfc5ac72a,2b14bce160e7bb68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:02.983 INFO 21062 --- [ocp-async-9,75881a5dbd0a2f4d,6d8d5241d104b317] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c3b25b1d2445fde9,e3f2b41cbeff9c27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:03.192 INFO 21062 --- [ocp-schedules-12,f0d0caa71283be08,c73ac5495b37621d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:41:03.192 INFO 21062 --- [ocp-schedules-12,de8525d7ce7ee7bb,645469d2ee7268d2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:41:03.196 INFO 21062 --- [ocp-schedules-12,de8525d7ce7ee7bb,645469d2ee7268d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:03.224 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:41:03.224 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:41:03.869 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:03.870 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:03.961 INFO 21062 --- [ocp-schedules-11,77d474612563a509,17c82b1f93e4dbac] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:41:04.934 WARN 21062 --- [ocp-async-9,df40533636484c57,fbe02ff01a62648f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:04.983 INFO 21062 --- [ocp-async-8,9200f4c188a8cc90,a03109ccf0f7bdee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:05.983 INFO 21062 --- [prometheus-minute-schedule-1,7ab17de9a874b2f4,8f792b57a8f082d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:06.870 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:06.871 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:06.935 WARN 21062 --- [ocp-async-8,5038e051d5f352f0,9ca00cfbfb14fbec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:06.983 INFO 21062 --- [ocp-async-10,1f7208edf8440042,ed14075cd7fb97ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:07.971 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,3ecedee146566a62,771ceec012f5c9fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=3ecedee146566a62, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 13:41:07.971 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,3ecedee146566a62,771ceec012f5c9fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=3ecedee146566a62, duration=7 ms] 2024-11-04 13:41:08.140 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,5d897d772264181b,551ab1e8edf5907f] c.c.s.h.OcpAuthenticationSuccessListener : User[name=admin, origin=LOCAL] authenticated successes. 2024-11-04 13:41:08.160 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,5d897d772264181b,551ab1e8edf5907f] c.s.h.CustomAuthenticationSuccessHandler : Authentication successful for /api/v2/iam/login 2024-11-04 13:41:08.170 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,5d897d772264181b,551ab1e8edf5907f] c.s.h.CustomAuthenticationSuccessHandler : Redirecting to saved url: / 2024-11-04 13:41:08.506 INFO 21062 --- [ocp-schedules-15,b5d981099c9df7bd,5df803ab6f43f444] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:41:08.506 INFO 21062 --- [ocp-schedules-10,2d8fe82bc020ac92,29f6b98385a2eb8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:41:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:41:08.589296+08:00, lastProcessTime=2024-11-04T13:40:58.590697+08:00 2024-11-04 13:41:08.610 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c6816306d2e52d88,ea819e362bb05b7c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=c6816306d2e52d88, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 13:41:08.612 INFO 21062 --- [ocp-async-8,afb110a25a83d7ca,440c21e8b5b23aab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:41:08.613 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,bc9d7a408505b613,bf74da8e15375645] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=bc9d7a408505b613, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 13:41:08.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,bc9d7a408505b613,bf74da8e15375645] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=bc9d7a408505b613, duration=14 ms] 2024-11-04 13:41:08.651 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c6816306d2e52d88,ea819e362bb05b7c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=c6816306d2e52d88, duration=48 ms] 2024-11-04 13:41:08.729 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2dc82d49c1cf5cba,524ddc8f5cb5f33e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=2dc82d49c1cf5cba, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 13:41:08.739 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2dc82d49c1cf5cba,524ddc8f5cb5f33e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=2dc82d49c1cf5cba, duration=15 ms] 2024-11-04 13:41:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:41:08.589296+08:00 2024-11-04 13:41:08.840 INFO 21062 --- [ocp-schedules-28,460be8f7b0fb6d13,9b223d1d758b92a9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:41:08.843 INFO 21062 --- [ocp-schedules-28,460be8f7b0fb6d13,9b223d1d758b92a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:08.844 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:41:08.874 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,7a0f0c169d3d4069,0abb862dbff90a65] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=7a0f0c169d3d4069, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 13:41:08.935 WARN 21062 --- [ocp-async-3,ee9ae8c68462dab3,d3565f8e7b9bcf1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:08.983 INFO 21062 --- [prometheus-minute-schedule-1,77d656c934b666b1,0d91d173660d1d2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:08.983 INFO 21062 --- [ocp-async-2,fd81b348e3d2477b,d5a2b342e2fe8435] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:09.180 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,7a0f0c169d3d4069,0abb862dbff90a65] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=7a0f0c169d3d4069, duration=325 ms] 2024-11-04 13:41:09.569 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0fac1f8f85e32a2d,2f271fc5d9445912] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.130.47.17, traceId=0fac1f8f85e32a2d, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-11-04 13:41:09.570 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7af5055d0453899b,7de875fd1baa5276] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.130.47.17, traceId=7af5055d0453899b, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-11-04 13:41:09.613 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7af5055d0453899b,7de875fd1baa5276] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.130.47.17, traceId=7af5055d0453899b, duration=51 ms] 2024-11-04 13:41:09.619 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0fac1f8f85e32a2d,2f271fc5d9445912] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.130.47.17, traceId=0fac1f8f85e32a2d, duration=61 ms] 2024-11-04 13:41:09.690 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,d927335dfe3a56e7,7796b6ad9fa51338] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=d927335dfe3a56e7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:41:09.690 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,9b47e9e3152f9cfd,0b04b58760820aa9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=9b47e9e3152f9cfd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:41:09.690 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,12aa1c1ca7a49cb3,a5bee6737f8934d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=12aa1c1ca7a49cb3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:41:09.690 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,2c924a3ec63b9fc4,167df8a3a01a9ba6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=2c924a3ec63b9fc4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:41:09.703 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,203e6d597e20fb4c,9e4a5033e79418ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP, client=10.130.47.17, traceId=203e6d597e20fb4c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,] 2024-11-04 13:41:09.704 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,61cd1360a1c96e67,84a2c134c3adad93] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP, client=10.130.47.17, traceId=61cd1360a1c96e67, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,] 2024-11-04 13:41:09.719 INFO 21062 --- [ocp-schedules-10,108b999b02c0e80a,cb197308d2c4bdaf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:41:09.777 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,12aa1c1ca7a49cb3,a5bee6737f8934d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=12aa1c1ca7a49cb3, duration=146 ms] 2024-11-04 13:41:09.777 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,d927335dfe3a56e7,7796b6ad9fa51338] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d927335dfe3a56e7, duration=138 ms] 2024-11-04 13:41:09.781 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,9b47e9e3152f9cfd,0b04b58760820aa9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=9b47e9e3152f9cfd, duration=146 ms] 2024-11-04 13:41:09.782 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,203e6d597e20fb4c,9e4a5033e79418ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.130.47.17, traceId=203e6d597e20fb4c, duration=94 ms] 2024-11-04 13:41:09.784 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,61cd1360a1c96e67,84a2c134c3adad93] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.130.47.17, traceId=61cd1360a1c96e67, duration=93 ms] 2024-11-04 13:41:09.794 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,2c924a3ec63b9fc4,167df8a3a01a9ba6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=2c924a3ec63b9fc4, duration=156 ms] 2024-11-04 13:41:09.862 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b3b1a16ca8d89f36,81609a7b86a496ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP, client=10.130.47.17, traceId=b3b1a16ca8d89f36, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,] 2024-11-04 13:41:09.866 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,a5ba9ab78a0593a3,22d950cc8fb19a1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.130.47.17, traceId=a5ba9ab78a0593a3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-11-04 13:41:09.866 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,33b9f777ba5f31f6,e7026969e33d5f82] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.130.47.17, traceId=33b9f777ba5f31f6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-11-04 13:41:09.871 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:09.872 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:09.872 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,b8808d81ba7fcce4,c8ecf97b1778161c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.130.47.17, traceId=b8808d81ba7fcce4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-11-04 13:41:09.873 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,a429b6b1fcb3d7b5,00174d5bab280400] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.130.47.17, traceId=a429b6b1fcb3d7b5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-11-04 13:41:09.874 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b3b1a16ca8d89f36,81609a7b86a496ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.130.47.17, traceId=b3b1a16ca8d89f36, duration=20 ms] 2024-11-04 13:41:09.878 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,95f18acece60197d,c804f0075c1a932e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.130.47.17, traceId=95f18acece60197d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-11-04 13:41:09.887 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,a429b6b1fcb3d7b5,00174d5bab280400] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.130.47.17, traceId=a429b6b1fcb3d7b5, duration=24 ms] 2024-11-04 13:41:09.887 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,95f18acece60197d,c804f0075c1a932e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.130.47.17, traceId=95f18acece60197d, duration=15 ms] 2024-11-04 13:41:09.887 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,a5ba9ab78a0593a3,22d950cc8fb19a1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.130.47.17, traceId=a5ba9ab78a0593a3, duration=27 ms] 2024-11-04 13:41:09.887 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,33b9f777ba5f31f6,e7026969e33d5f82] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.130.47.17, traceId=33b9f777ba5f31f6, duration=28 ms] 2024-11-04 13:41:09.887 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,b8808d81ba7fcce4,c8ecf97b1778161c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.130.47.17, traceId=b8808d81ba7fcce4, duration=23 ms] 2024-11-04 13:41:09.950 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,0709e76bd1aaa1b3,a251b4bf967e8f5b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.130.47.17, traceId=0709e76bd1aaa1b3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-11-04 13:41:09.956 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,0709e76bd1aaa1b3,a251b4bf967e8f5b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.130.47.17, traceId=0709e76bd1aaa1b3, duration=13 ms] 2024-11-04 13:41:09.966 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,47630a0440f8fd84,035fc412c0d3bbe8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.130.47.17, traceId=47630a0440f8fd84, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 13:41:09.989 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,47630a0440f8fd84,035fc412c0d3bbe8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=47630a0440f8fd84, duration=31 ms] 2024-11-04 13:41:10.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:41:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:41:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:41:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:41:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:41:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:41:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:41:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:41:10.397 INFO 21062 --- [ocp-schedules-4,dde3d6cb2fee5f04,104300be45a4762f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:41:10.403 INFO 21062 --- [ocp-schedules-4,dde3d6cb2fee5f04,104300be45a4762f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:41:10.403 INFO 21062 --- [ocp-schedules-4,dde3d6cb2fee5f04,104300be45a4762f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:41:10.935 WARN 21062 --- [ocp-async-2,72826091e278766d,a6e6542af0fb9084] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:10.983 INFO 21062 --- [ocp-async-1,9e942d5f515b539c,4e05cd36746390b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:11.311 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=4, affectRows=4 2024-11-04 13:41:11.311 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=4, validCount=4 2024-11-04 13:41:11.312 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=4, affectRows=4 2024-11-04 13:41:11.312 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=14, affectRows=14 2024-11-04 13:41:11.312 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=4, validCount=4 2024-11-04 13:41:11.312 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=20, affectRows=20 2024-11-04 13:41:11.312 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=14, validCount=14 2024-11-04 13:41:11.312 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=20, validCount=20 2024-11-04 13:41:11.314 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=25, affectRows=25 2024-11-04 13:41:11.314 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=24, validCount=24 2024-11-04 13:41:11.314 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=25, affectRows=25 2024-11-04 13:41:11.315 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=25, validCount=25 2024-11-04 13:41:11.315 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=39, affectRows=39 2024-11-04 13:41:11.315 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=34, affectRows=34 2024-11-04 13:41:11.315 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=39, validCount=39 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=23, affectRows=23 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=33, validCount=33 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=47, affectRows=47 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=23, validCount=23 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=47, validCount=47 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=46, affectRows=46 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698800, nodeCount=43, affectRows=43 2024-11-04 13:41:11.316 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=45, validCount=45 2024-11-04 13:41:11.317 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698800, nodeCount=43, validCount=43 2024-11-04 13:41:11.717 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,37f670edeb0d1805,aae7d0350659f45f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=37f670edeb0d1805, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 13:41:11.717 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,37f670edeb0d1805,aae7d0350659f45f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=37f670edeb0d1805, duration=5 ms] 2024-11-04 13:41:11.717 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3e4155e25e8ecef7,105f4b98e2b0bb2a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=3e4155e25e8ecef7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 13:41:11.717 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3e4155e25e8ecef7,105f4b98e2b0bb2a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=3e4155e25e8ecef7, duration=5 ms] 2024-11-04 13:41:11.720 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,bc44c87c12ddd293,3e5db5571d4ad89d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=bc44c87c12ddd293, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 13:41:11.720 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,bc44c87c12ddd293,3e5db5571d4ad89d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=bc44c87c12ddd293, duration=5 ms] 2024-11-04 13:41:11.902 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:41:11.902 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:41:11.902 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:41:11.902 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:41:11.903 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:41:11.903 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:41:11.983 INFO 21062 --- [prometheus-minute-schedule-1,bced53e227820693,2eb6e46769af064b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:12.872 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:12.873 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:12.935 WARN 21062 --- [ocp-async-8,4e22cbd818e140a9,36f22669a6d33bed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:12.983 INFO 21062 --- [ocp-async-4,9d1069f98c325a15,162ae4b530af1209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:13.985 INFO 21062 --- [prometheus-cache-monitor1,d59cea89c0fded9b,3ee07b5e9d591e09] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1290,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":324,"secondWriteQueue":0,"minuteWriteCacheSize":321} 2024-11-04 13:41:13.985 INFO 21062 --- [prometheus-cache-monitor1,d59cea89c0fded9b,3ee07b5e9d591e09] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=16, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=16, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:41:14.003 INFO 21062 --- [ocp-schedules-19,a072c94cc3efe3d5,fe84737f7c248baf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:41:14.662 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:37:14.644, end=2024-11-04T13:39:14.66} 2024-11-04 13:41:14.670 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:41:14.670 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:41:14.935 WARN 21062 --- [ocp-async-10,69f9f14d9a8cce2d,e840b98b66cd18d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:14.983 INFO 21062 --- [prometheus-minute-schedule-1,e6680db3e3f1a7ac,cce1546f0bfb8ace] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:14.984 INFO 21062 --- [ocp-async-6,eed388af168c325c,f1b94053d157cc5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:15.873 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:15.874 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:16.936 WARN 21062 --- [ocp-async-9,6694316346cef2a2,cbdb2d864891763c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:16.984 INFO 21062 --- [ocp-async-7,4eb9907870c75166,d0afcf2da6e18022] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9e7e743e508bf212,4a3bb616047d8dab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:41:18.590457+08:00, lastProcessTime=2024-11-04T13:41:08.589296+08:00 2024-11-04 13:41:18.612 INFO 21062 --- [ocp-async-6,7244ba9aaa7e78c4,ade6414e68e6e4ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:41:18.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:41:18.590457+08:00 2024-11-04 13:41:18.844 INFO 21062 --- [ocp-schedules-28,460be8f7b0fb6d13,9b223d1d758b92a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:41:18.874 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:18.875 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:18.936 WARN 21062 --- [ocp-async-8,284440eb5766f023,0ce6cc19dccec7dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:18.984 INFO 21062 --- [ocp-async-4,e7b84b454e2c3fad,96778d80eb923c47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:19.727 INFO 21062 --- [ocp-schedules-2,c911b55a1f318983,3add9af116638aba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:41:20.345 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,82f6406832d5003c,7a504e96705dbc55] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=82f6406832d5003c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:41:20.345 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a18984322499e063,bae59dbbd3b53806] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=a18984322499e063, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:41:20.345 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,914e0449cee3d94e,d997667be644d7ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=914e0449cee3d94e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:41:20.345 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,acca0e6df290bac4,cae4f7a77189dc50] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=acca0e6df290bac4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:41:20.346 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a18984322499e063,bae59dbbd3b53806] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a18984322499e063, duration=12 ms] 2024-11-04 13:41:20.347 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,82f6406832d5003c,7a504e96705dbc55] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=82f6406832d5003c, duration=13 ms] 2024-11-04 13:41:20.347 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,914e0449cee3d94e,d997667be644d7ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=914e0449cee3d94e, duration=14 ms] 2024-11-04 13:41:20.350 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,acca0e6df290bac4,cae4f7a77189dc50] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=acca0e6df290bac4, duration=17 ms] 2024-11-04 13:41:20.936 WARN 21062 --- [ocp-async-10,28c4c543c7ee6157,8f14b7eb253502c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f9778c21503f89ef,da043d104d051125] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:20.984 INFO 21062 --- [ocp-async-9,6af76ae84415c7ca,057d2a8279690039] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:21.875 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:21.876 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:22.936 WARN 21062 --- [ocp-async-6,048555f58e839623,21b74f0e6e520987] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:22.984 INFO 21062 --- [ocp-async-7,44bf8e06d7262402,447153da2f2ecdf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a1f91ce449498d15,42b0bf3ed1174c4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:24.048 INFO 21062 --- [ocp-schedules-23,3c1c9baa96a1f704,af55550c83f63807] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:41:24.876 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:24.877 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:24.936 WARN 21062 --- [ocp-async-1,5514ad3ef9782f82,1ae44673b7673bfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:24.985 INFO 21062 --- [ocp-async-5,2a54571d830cd9d6,22a3973aec0b504f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:26.936 WARN 21062 --- [ocp-async-4,02671cc75846b45e,259206d33e2c0096] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e2585cfaf6f35bd6,098a46bbe1e28d3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:26.984 INFO 21062 --- [prometheus-hour-schedule-1,03996cd2e166c94d,4fc624b76b148198] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:41:26.985 INFO 21062 --- [ocp-async-3,7d85ef6ef44b5e2a,38e5eeb69f48ac24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:27.877 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:27.878 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:28.498 INFO 21062 --- [ocp-schedules-11,122864a249062549,f3b6371b718727ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:41:28.499 INFO 21062 --- [ocp-schedules-27,9889741f0c95b180,740ad439732a0d89] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:41:28.499 INFO 21062 --- [ocp-schedules-26,015114ecc2cf984b,9706f93b696b16d8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:41:28.500 INFO 21062 --- [ocp-schedules-2,1e45722114bb1e1d,bc8fd33f40434765] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:41:28.502 INFO 21062 --- [ocp-schedules-11,122864a249062549,f3b6371b718727ef] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:41:28.502 INFO 21062 --- [ocp-schedules-15,bea33ef1a347df5b,9def14cd6865452f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:41:28.503 INFO 21062 --- [ocp-schedules-2,1e45722114bb1e1d,bc8fd33f40434765] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:28.503 INFO 21062 --- [ocp-schedules-26,015114ecc2cf984b,9706f93b696b16d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:41:28.506 INFO 21062 --- [ocp-schedules-21,d35a4f280a12d1a0,726858783abc3698] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:41:28.506 INFO 21062 --- [ocp-schedules-27,9889741f0c95b180,740ad439732a0d89] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:41:28.511 INFO 21062 --- [ocp-schedules-20,2589fbfab1adfb89,6858b1da4343311e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:41:28.511 INFO 21062 --- [ocp-schedules-9,b17ebca4c4629cd9,b99b7663288c198d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:41:28.511 INFO 21062 --- [ocp-schedules-13,91be39cb4835f3dc,f9c162d0d739a771] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:41:28.512 INFO 21062 --- [ocp-schedules-9,b17ebca4c4629cd9,b99b7663288c198d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:41:28.512 INFO 21062 --- [ocp-schedules-20,2589fbfab1adfb89,6858b1da4343311e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:41:28.512 INFO 21062 --- [ocp-schedules-13,91be39cb4835f3dc,f9c162d0d739a771] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:41:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:41:28.589741+08:00, lastProcessTime=2024-11-04T13:41:18.590457+08:00 2024-11-04 13:41:28.613 INFO 21062 --- [ocp-async-10,2e1c4b9dbdae7068,ffd52432d295a770] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:41:28.727 INFO 21062 --- [ocp-schedules-5,f672718b4ec638dc,2a5bc3a70a8329ff] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:41:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:41:28.589741+08:00 2024-11-04 13:41:28.903 INFO 21062 --- [ocp-schedules-23,4511ae9118b5643b,9f87e7f5ec5bdfb9] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:41:28.903 INFO 21062 --- [ocp-schedules-23,4511ae9118b5643b,9f87e7f5ec5bdfb9] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:41:28.932 INFO 21062 --- [ocp-schedules-8,0f97f94ffd80013a,1338b930f5833bda] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:41:28.937 WARN 21062 --- [ocp-async-7,ec91e8fc35971464,63b1d4fddef075b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:28.965 INFO 21062 --- [ocp-schedules-9,9ef48e169fe8180e,94b1cfcede3617ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:41:28.971 INFO 21062 --- [ocp-schedules-9,9ef48e169fe8180e,94b1cfcede3617ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:41:28.971 INFO 21062 --- [ocp-schedules-9,9ef48e169fe8180e,94b1cfcede3617ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:41:28.985 INFO 21062 --- [ocp-async-5,2601fcb85e503248,912a5ce5163a4705] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:29.207 INFO 21062 --- [ocp-schedules-7,ba24fd6aea5e8e42,c872a56b040c206a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:41:29.207 INFO 21062 --- [ocp-schedules-7,ba24fd6aea5e8e42,c872a56b040c206a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:41:29.210 INFO 21062 --- [ocp-schedules-7,ba24fd6aea5e8e42,c872a56b040c206a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:41:29.210 INFO 21062 --- [ocp-schedules-7,ba24fd6aea5e8e42,c872a56b040c206a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:41:29.211 INFO 21062 --- [ocp-schedules-7,ba24fd6aea5e8e42,c872a56b040c206a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:41:29.211 INFO 21062 --- [ocp-schedules-7,ba24fd6aea5e8e42,c872a56b040c206a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908395925458944 2024-11-04 13:41:29.214 INFO 21062 --- [ocp-schedules-7,ba24fd6aea5e8e42,c872a56b040c206a] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=42, maxSyncedRecordId=709925194352054272 2024-11-04 13:41:29.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,1c47823e34ad13b1,6e3478343a59ec61] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=1c47823e34ad13b1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:41:29.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,cac8a1d1d5b5b3c9,eb9d281e30314415] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=cac8a1d1d5b5b3c9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:41:29.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,fb5f1c286b9fa971,a1d9bbbac7962e48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=fb5f1c286b9fa971, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:41:29.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8dabd0d372da383d,508cb8cd78c121fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=8dabd0d372da383d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:41:29.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,1c47823e34ad13b1,6e3478343a59ec61] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=1c47823e34ad13b1, duration=8 ms] 2024-11-04 13:41:29.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,fb5f1c286b9fa971,a1d9bbbac7962e48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=fb5f1c286b9fa971, duration=11 ms] 2024-11-04 13:41:29.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,cac8a1d1d5b5b3c9,eb9d281e30314415] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=cac8a1d1d5b5b3c9, duration=8 ms] 2024-11-04 13:41:29.624 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8dabd0d372da383d,508cb8cd78c121fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=8dabd0d372da383d, duration=14 ms] 2024-11-04 13:41:29.730 INFO 21062 --- [ocp-schedules-22,d99bebce13db9fea,ed30f9d4db97cc9c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:41:29.983 INFO 21062 --- [prometheus-minute-schedule-1,744aa636c35def12,647cc58c7e69b199] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:41:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:41:30.405 INFO 21062 --- [ocp-schedules-10,290596b7cc5d739e,07549aa80509165f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:41:30.411 INFO 21062 --- [ocp-schedules-11,f8c66bbab035197b,398512ce87f91790] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:41:30.411 INFO 21062 --- [ocp-schedules-11,f8c66bbab035197b,398512ce87f91790] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:41:30.412 INFO 21062 --- [ocp-schedules-10,290596b7cc5d739e,07549aa80509165f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:41:30.412 INFO 21062 --- [ocp-schedules-10,290596b7cc5d739e,07549aa80509165f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:41:30.438 INFO 21062 --- [ocp-schedules-24,4fb8d94c47d6de22,d13bc1cbcf4b2dae] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:41:30.438 INFO 21062 --- [ocp-schedules-11,f8c66bbab035197b,398512ce87f91790] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:41:30.438 INFO 21062 --- [ocp-schedules-11,f8c66bbab035197b,398512ce87f91790] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:41:30.572 INFO 21062 --- [ocp-schedules-25,5cf7019de7524beb,ca6a69c79c2f110e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:41:30.572+08:00, clusters=[1] 2024-11-04 13:41:30.574 INFO 21062 --- [ocp-schedules-25,5cf7019de7524beb,ca6a69c79c2f110e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:41:30.613 INFO 21062 --- [ocp-schedules-1,632d20433fd4db2b,a0b6b973919b1e5d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:41:30.613 INFO 21062 --- [ocp-schedules-1,632d20433fd4db2b,a0b6b973919b1e5d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:41:30.613 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:41:30.613 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:41:30.624 INFO 21062 --- [ocp-schedules-1,632d20433fd4db2b,a0b6b973919b1e5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:41:30.624 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:41:30.624 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:41:30.625 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:41:30.625 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:41:30.625 WARN 21062 --- [ocp-schedules-1,632d20433fd4db2b,a0b6b973919b1e5d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:41:30.626 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:41:30.626 INFO 21062 --- [ocp-schedules-1,632d20433fd4db2b,a0b6b973919b1e5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:41:30.626 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:41:30.627 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:41:30.627 WARN 21062 --- [ocp-schedules-1,632d20433fd4db2b,a0b6b973919b1e5d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:41:30.627 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:41:30.635 ERROR 21062 --- [ocp-schedules-1,632d20433fd4db2b,a0b6b973919b1e5d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:41:30.713 INFO 21062 --- [ocp-schedules-6,99dc504776b8d532,2ed5780f4abf0938] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:41:30.713+08:00, clusters=[1] 2024-11-04 13:41:30.717 INFO 21062 --- [ocp-schedules-6,99dc504776b8d532,2ed5780f4abf0938] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:41:30.878 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:30.879 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:30.937 WARN 21062 --- [ocp-async-5,cb474e7dabe79254,f2fcf4a1878e36d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:30.985 INFO 21062 --- [ocp-async-10,35988dc7e34b8feb,f065a56b4799908f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:32.937 WARN 21062 --- [ocp-async-10,f73bf28852fec1a9,99394edddfe8e559] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:32.983 INFO 21062 --- [prometheus-minute-schedule-1,6a3ef0ca3e2016f0,4c200edc6465e40a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:32.985 INFO 21062 --- [ocp-async-6,9f5a5a2ffe50df3a,bfa016f00524c18b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:33.225 INFO 21062 --- [ocp-schedules-12,de8525d7ce7ee7bb,645469d2ee7268d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:41:33.225 INFO 21062 --- [ocp-schedules-12,345ced7306df9379,3bc3868cd596b550] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:41:33.229 INFO 21062 --- [ocp-schedules-12,345ced7306df9379,3bc3868cd596b550] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:33.258 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:41:33.258 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:41:33.879 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:33.880 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:34.093 INFO 21062 --- [ocp-schedules-17,e8d860e49c22cf06,22887f6ffa1aa999] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:41:34.867 INFO 21062 --- [ocp-schedules-15,daad5ab511890282,032fc44c3306ef43] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:41:34.871 INFO 21062 --- [ocp-schedules-15,daad5ab511890282,032fc44c3306ef43] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:34.937 WARN 21062 --- [ocp-async-6,0c248fa0871d7f3e,07729f383f258c11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:34.957 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:41:34.959 ERROR 21062 --- [ocp-schedules-15,daad5ab511890282,032fc44c3306ef43] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:41:34.959 INFO 21062 --- [ocp-schedules-15,daad5ab511890282,032fc44c3306ef43] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:41:34.985 INFO 21062 --- [ocp-async-1,8e438c23994798f3,78f23c92a420b942] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1f2697e4116d11c1,b053439e0c4048fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:36.880 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:36.880 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:36.937 WARN 21062 --- [ocp-async-1,6b75d5d54994b474,59e652bf8f0c511b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:36.986 INFO 21062 --- [ocp-async-4,28dcdfa4984e4c10,001f02ce2290976f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:38.504 INFO 21062 --- [ocp-schedules-2,1e45722114bb1e1d,bc8fd33f40434765] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:41:38.505 INFO 21062 --- [ocp-schedules-26,015114ecc2cf984b,9706f93b696b16d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:41:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:41:38.589298+08:00, lastProcessTime=2024-11-04T13:41:28.589741+08:00 2024-11-04 13:41:38.613 INFO 21062 --- [ocp-async-8,0aa21d90f823336e,88d884056f8c17ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:41:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:41:38.589298+08:00 2024-11-04 13:41:38.937 WARN 21062 --- [ocp-async-3,4fae7a977a692c7f,e079480d08de807e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:38.983 INFO 21062 --- [prometheus-minute-schedule-1,4ea9c51a08551ff0,d6e27ca221df8dba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:38.986 INFO 21062 --- [ocp-async-2,4ac551a9186851c6,e88be4a213253f07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:39.592 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f6f90f847c79b840,9ad6de60e86015c7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=f6f90f847c79b840, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:41:39.592 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,743613995aa56ca2,2bf668988b166689] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=743613995aa56ca2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:41:39.593 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,d5baef961e5b3cb4,14b30815b7247e83] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=d5baef961e5b3cb4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:41:39.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,026e5e090199643f,0be03b26b3d7dc0f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=026e5e090199643f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:41:39.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f6f90f847c79b840,9ad6de60e86015c7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=f6f90f847c79b840, duration=8 ms] 2024-11-04 13:41:39.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,743613995aa56ca2,2bf668988b166689] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=743613995aa56ca2, duration=11 ms] 2024-11-04 13:41:39.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,026e5e090199643f,0be03b26b3d7dc0f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=026e5e090199643f, duration=9 ms] 2024-11-04 13:41:39.596 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,d5baef961e5b3cb4,14b30815b7247e83] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d5baef961e5b3cb4, duration=13 ms] 2024-11-04 13:41:39.735 INFO 21062 --- [ocp-schedules-30,4b25240a54dd9285,178aea54c2b673eb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:41:39.881 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:39.881 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:41:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:41:40.938 WARN 21062 --- [ocp-async-2,dd0b5482a00371cd,9f9ed38dd03f9859] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:40.986 INFO 21062 --- [ocp-async-8,74e04299305dcfd1,c5aa5042878e9d31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:41.983 INFO 21062 --- [prometheus-minute-schedule-1,931820fddfcdc5e3,b4cd3fddf20c1390] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:42.882 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:42.883 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:42.938 WARN 21062 --- [ocp-async-8,29383d9607735b28,f0cfc5fba322f387] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:42.986 INFO 21062 --- [ocp-async-10,4d255882ed130f2f,fc40133312f77266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:44.119 INFO 21062 --- [ocp-schedules-26,e0334ac29f346983,03dc700131af415f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:41:44.938 WARN 21062 --- [ocp-async-10,ce53b7cf2686e28f,00acd58a4266c91f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:44.983 INFO 21062 --- [prometheus-minute-schedule-1,017b0221aca7ce9c,adb3fcad28de1cb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:44.986 INFO 21062 --- [ocp-async-6,fdc74bd667029736,287aa81d9b57b20e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:45.883 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:45.884 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:46.938 WARN 21062 --- [ocp-async-6,48cd4b8f2f705950,836fbbf737ce3a2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:46.987 INFO 21062 --- [ocp-async-1,4f8cd7d540513469,269ff21c0b0cf3c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:47.983 INFO 21062 --- [prometheus-minute-schedule-1,d049b40f3d7269fd,af355dd43327b632] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:41:48.592119+08:00, lastProcessTime=2024-11-04T13:41:38.589298+08:00 2024-11-04 13:41:48.613 INFO 21062 --- [ocp-async-6,8f61def90673ffcd,05030fbac5cd7736] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:41:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:41:48.592119+08:00 2024-11-04 13:41:48.884 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:48.884 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:48.938 WARN 21062 --- [ocp-async-8,a5e575e8ef79ee75,7dd6001f6b56fd6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:48.987 INFO 21062 --- [ocp-async-10,0951ccecf290403c,7329498ba336713a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:49.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,9d728ff9e601963f,c2aa13d54292ef2d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=9d728ff9e601963f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:41:49.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,62c03245f64718b3,8b712617d2382a37] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=62c03245f64718b3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:41:49.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,7e2cc28ac33e8745,c4267b7a65dd5ab6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=7e2cc28ac33e8745, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:41:49.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,836a679ed118c474,b0f4946a3af3dafc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=836a679ed118c474, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:41:49.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,62c03245f64718b3,8b712617d2382a37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=62c03245f64718b3, duration=7 ms] 2024-11-04 13:41:49.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,7e2cc28ac33e8745,c4267b7a65dd5ab6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=7e2cc28ac33e8745, duration=13 ms] 2024-11-04 13:41:49.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,9d728ff9e601963f,c2aa13d54292ef2d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=9d728ff9e601963f, duration=11 ms] 2024-11-04 13:41:49.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,836a679ed118c474,b0f4946a3af3dafc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=836a679ed118c474, duration=16 ms] 2024-11-04 13:41:49.737 INFO 21062 --- [ocp-schedules-4,6f945e92462cf9ce,85054c79d3020974] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:41:50.415 INFO 21062 --- [ocp-schedules-16,beda759fd492aa2d,b502f0c8b71dc18c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:41:50.421 INFO 21062 --- [ocp-schedules-16,beda759fd492aa2d,b502f0c8b71dc18c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:41:50.421 INFO 21062 --- [ocp-schedules-16,beda759fd492aa2d,b502f0c8b71dc18c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:41:50.938 WARN 21062 --- [ocp-async-10,46a054e085f9c33b,68c4dd93a91c66df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:50.983 INFO 21062 --- [prometheus-minute-schedule-1,e2ab32bf6c49d69d,5f5347ab1d37428b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:50.987 INFO 21062 --- [ocp-async-6,4222fe20e497b238,51dfc8d71f112756] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:51.885 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:51.885 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:52.939 WARN 21062 --- [ocp-async-6,888558f11ff3ae71,553adb7263227d04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:52.987 INFO 21062 --- [ocp-async-1,5d7b078f5b6227dd,2dd2ec759f403624] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1ff8cc99165ccafc,80c4b40047a610cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:54.162 INFO 21062 --- [ocp-schedules-24,5fc1782530e49e05,3edcdaffad179b49] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:41:54.886 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:54.887 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:54.939 WARN 21062 --- [ocp-async-1,25c3321f502dbf5d,890169396be43e7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:54.987 INFO 21062 --- [ocp-async-4,37d0d64a5fc9bd7c,a8f8fe815e4de879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:56.939 WARN 21062 --- [ocp-async-4,10cfcd3651e81d39,2730349116d3ebf6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:56.983 INFO 21062 --- [prometheus-minute-schedule-1,34bae41f97aee740,bae8a5f0f3443db7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:41:56.987 INFO 21062 --- [ocp-async-9,537ca33808192cc9,8c0df9dfec565c14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:57.887 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:41:57.888 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:41:58.500 INFO 21062 --- [ocp-schedules-11,86383113e1a100dd,9db402a744a53268] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:41:58.500 INFO 21062 --- [ocp-schedules-6,e84351797f70c2f7,f68cabb0245ddda4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:41:58.502 INFO 21062 --- [ocp-schedules-30,006a7aa76168695b,55c4742887f59d33] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:41:58.504 INFO 21062 --- [ocp-schedules-6,e84351797f70c2f7,f68cabb0245ddda4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:58.504 INFO 21062 --- [ocp-schedules-11,86383113e1a100dd,9db402a744a53268] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:41:58.505 INFO 21062 --- [ocp-schedules-7,f9fcfd9b67630418,976deedd35194455] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:41:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:41:58.511 INFO 21062 --- [ocp-schedules-16,a227c53a12471983,3576331645b8fae7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:41:58.511 INFO 21062 --- [ocp-schedules-20,ac916e773ed82c73,830fc26fe7a68213] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:41:58.511 INFO 21062 --- [ocp-schedules-20,ac916e773ed82c73,830fc26fe7a68213] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:41:58.511 INFO 21062 --- [ocp-schedules-16,a227c53a12471983,3576331645b8fae7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:41:58.511 INFO 21062 --- [ocp-schedules-9,e00685ddd5fb0c6c,2e8dc2a3eb5b5720] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:41:58.511 INFO 21062 --- [ocp-schedules-9,e00685ddd5fb0c6c,2e8dc2a3eb5b5720] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:41:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:41:58.588887+08:00, lastProcessTime=2024-11-04T13:41:48.592119+08:00 2024-11-04 13:41:58.613 INFO 21062 --- [ocp-async-4,34f31b817df5e425,ee6111aa655f1873] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:41:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:41:58.588887+08:00 2024-11-04 13:41:58.934 INFO 21062 --- [ocp-schedules-5,c162c2d5f7f3c193,75e5fcdf58689c25] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:41:58.939 WARN 21062 --- [ocp-async-6,c0c52e68bf2973b4,fc9e43ae6b2e4a0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:41:58.987 INFO 21062 --- [ocp-async-7,7c1d32521d7e98f9,78af042e8d38f266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:41:59.616 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e14e9309f74de86d,f19861fd8013ca49] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=e14e9309f74de86d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:41:59.616 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e14e9309f74de86d,f19861fd8013ca49] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=e14e9309f74de86d, duration=5 ms] 2024-11-04 13:41:59.617 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,422bf5a61b7f19dc,fa6001983c9cbecc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=422bf5a61b7f19dc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:41:59.617 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b089425fac06d799,60d59408e3246eb4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=b089425fac06d799, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:41:59.618 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b089425fac06d799,60d59408e3246eb4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=b089425fac06d799, duration=9 ms] 2024-11-04 13:41:59.618 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,e47cc7760238d192,25fb6026098c70f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=e47cc7760238d192, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:41:59.619 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,e47cc7760238d192,25fb6026098c70f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=e47cc7760238d192, duration=8 ms] 2024-11-04 13:41:59.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,422bf5a61b7f19dc,fa6001983c9cbecc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=422bf5a61b7f19dc, duration=11 ms] 2024-11-04 13:41:59.739 INFO 21062 --- [ocp-schedules-10,8bd1eea85965076f,57623cfa0f307382] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:41:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1e7769ce340fe98c,c35566827a906f9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698920, interval=60 2024-11-04 13:42:00.888 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:00.889 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:00.939 WARN 21062 --- [ocp-async-7,31c76c079339adfa,b60bb2b176766c5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:00.988 INFO 21062 --- [ocp-async-5,25517fdb40cf28ec,a30a708af1a61f0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:02.939 WARN 21062 --- [ocp-async-5,3061315cf11ea2f2,6c51bc073cea1220] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:02.983 INFO 21062 --- [prometheus-minute-schedule-1,3881a977a1a383d5,9b279ce4de23b0ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:02.988 INFO 21062 --- [ocp-async-3,307e008f81a8b73a,466b8329f46cd81e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:03.259 INFO 21062 --- [ocp-schedules-12,345ced7306df9379,3bc3868cd596b550] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:42:03.259 INFO 21062 --- [ocp-schedules-12,f584c5575d99dbcf,4b15dfd40797dc37] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:42:03.263 INFO 21062 --- [ocp-schedules-12,f584c5575d99dbcf,4b15dfd40797dc37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:03.293 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:42:03.293 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:42:03.889 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:03.889 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:04.187 INFO 21062 --- [ocp-schedules-15,4ba319ff25c47588,d83fb75fab55a17a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:42:04.940 WARN 21062 --- [ocp-async-3,59858aa051843fd2,4bafd718f653b6f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:04.988 INFO 21062 --- [ocp-async-2,ce90f3a76884e129,55b1637e1701b59d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:05.983 INFO 21062 --- [prometheus-minute-schedule-1,14b91f9adf497626,44c94e06f1ade688] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:06.890 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:06.890 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:06.940 WARN 21062 --- [ocp-async-2,9bd4e57c5d9af9a6,6e4b5bec1f606107] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:06.988 INFO 21062 --- [ocp-async-8,8ea9790339098903,46eb3fb6ecd053be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:08.505 INFO 21062 --- [ocp-schedules-6,e84351797f70c2f7,f68cabb0245ddda4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:42:08.505 INFO 21062 --- [ocp-schedules-11,86383113e1a100dd,9db402a744a53268] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:42:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:42:08.591184+08:00, lastProcessTime=2024-11-04T13:41:58.588887+08:00 2024-11-04 13:42:08.613 INFO 21062 --- [ocp-async-2,c7611193a2c34b98,4bdbc02ef23ba748] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:42:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:42:08.591184+08:00 2024-11-04 13:42:08.940 WARN 21062 --- [ocp-async-5,f8b9f4531afa87c2,43521a2064d7597f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b95d824f6ff61860,c733b7b739e3087b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:08.989 INFO 21062 --- [ocp-async-9,3412cc4389d10fff,0acec3e60852d941] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:09.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,c9c8ece61d925dd7,8e84116b062b8cbf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=c9c8ece61d925dd7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:42:09.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,b5f1e3576558ffc8,8415aac37a18d789] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=b5f1e3576558ffc8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:42:09.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,e9ddb9076b801cc5,3d62a2a9800dfd4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=e9ddb9076b801cc5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:42:09.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,5d6b8a97375a2836,157556c00fad477e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=5d6b8a97375a2836, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:42:09.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,b5f1e3576558ffc8,8415aac37a18d789] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=b5f1e3576558ffc8, duration=5 ms] 2024-11-04 13:42:09.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,c9c8ece61d925dd7,8e84116b062b8cbf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=c9c8ece61d925dd7, duration=8 ms] 2024-11-04 13:42:09.623 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,5d6b8a97375a2836,157556c00fad477e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=5d6b8a97375a2836, duration=6 ms] 2024-11-04 13:42:09.625 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,e9ddb9076b801cc5,3d62a2a9800dfd4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=e9ddb9076b801cc5, duration=12 ms] 2024-11-04 13:42:09.742 INFO 21062 --- [ocp-schedules-30,f1a06e9fe74e0264,c1a4c1c00ada59fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:42:09.890 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:09.891 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:10.002 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:42:10.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:42:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:42:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:42:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:42:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:42:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:42:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:42:10.423 INFO 21062 --- [ocp-schedules-18,b1ad231dcc9975ff,db8b466c4374182b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:42:10.430 INFO 21062 --- [ocp-schedules-18,b1ad231dcc9975ff,db8b466c4374182b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:42:10.430 INFO 21062 --- [ocp-schedules-18,b1ad231dcc9975ff,db8b466c4374182b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:42:10.941 WARN 21062 --- [ocp-async-3,4e4ae3794d80d5a9,cfc75f29b996b685] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:10.989 INFO 21062 --- [ocp-async-1,78bf165afee5f49a,eee66888eb848942] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=8, affectRows=8 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=28, affectRows=28 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=8, validCount=8 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=18, affectRows=18 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=27, validCount=27 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=18, validCount=18 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=34, affectRows=34 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=61, affectRows=61 2024-11-04 13:42:11.322 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=49, affectRows=49 2024-11-04 13:42:11.323 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=34, validCount=34 2024-11-04 13:42:11.323 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=49, validCount=49 2024-11-04 13:42:11.323 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=61, validCount=61 2024-11-04 13:42:11.323 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=74, affectRows=74 2024-11-04 13:42:11.323 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=73, validCount=73 2024-11-04 13:42:11.324 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698860, nodeCount=92, affectRows=92 2024-11-04 13:42:11.324 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698860, nodeCount=91, validCount=91 2024-11-04 13:42:11.983 INFO 21062 --- [prometheus-minute-schedule-1,87f6866e533181c0,b508e2f6c4a2cb69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:12.891 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:12.892 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:12.941 WARN 21062 --- [ocp-async-2,5d9d254ad37b90f8,a00d424d40d1e963] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:12.989 INFO 21062 --- [ocp-async-4,8a1c6666018edbe1,5a0d02369debf575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:13.984 INFO 21062 --- [prometheus-cache-monitor1,025a69e44efa9a3d,df412eb9d3cc8ec6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1332,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:42:13.984 INFO 21062 --- [prometheus-cache-monitor1,025a69e44efa9a3d,df412eb9d3cc8ec6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:42:14.226 INFO 21062 --- [ocp-schedules-9,49e305f89bf73aa0,90338fd86512441f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:42:14.942 WARN 21062 --- [ocp-async-8,e0f78c3ab7f237ab,ef17d3b68695b32b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:14.983 INFO 21062 --- [prometheus-minute-schedule-1,7a6803a9a582bd34,42cd422f3037e1fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:14.989 INFO 21062 --- [ocp-async-10,82fb6cd374db61fc,57446acce69eea60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:15.111 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:42:15.111 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:42:15.120 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:42:15.120 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:42:15.120 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:42:15.120 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:42:15.126 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:42:15.126 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:42:15.126 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:42:15.126 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:42:15.160 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:42:15.168 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:42:15.168 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:42:15.168 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:42:15.168 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:42:15.168 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:42:15.214 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:42:15.733 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:42:15.892 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:15.894 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:16.289 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.162553829S 2024-11-04 13:42:16.289 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:42:16.387 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:42:16.387 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:42:16.412 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:42:16.472 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:42:16.472 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:42:16.472 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:42:16.472 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:42:16.472 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:42:16.472 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:42:16.484 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:42:16.659 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] o.s.boot.SpringApplication : Started application in 1.539 seconds (JVM running for 4350.338) 2024-11-04 13:42:16.659 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.539537072S 2024-11-04 13:42:16.942 WARN 21062 --- [ocp-async-9,7b877810a5367f58,2d2d74fec0886971] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:16.989 INFO 21062 --- [ocp-async-6,0508c1e31cf466a6,124f48eca0f5d590] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:17.115 INFO 21062 --- [ocp-schedules-28,9e336bc954965aa8,a70dfeae1ad2ebe8] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:42:17.983 INFO 21062 --- [prometheus-minute-schedule-1,915d73a3c5de5f1a,956c94d2fff69373] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:42:18.589080+08:00, lastProcessTime=2024-11-04T13:42:08.591184+08:00 2024-11-04 13:42:18.613 INFO 21062 --- [ocp-async-10,ce9f15b8df8d6302,b5251a6409a5e417] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:42:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:42:18.589080+08:00 2024-11-04 13:42:18.844 INFO 21062 --- [ocp-schedules-25,34ed88875fe1041e,22054acabe7bc2ad] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:42:18.848 INFO 21062 --- [ocp-schedules-25,34ed88875fe1041e,22054acabe7bc2ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:18.848 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:42:18.894 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:18.894 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:18.942 WARN 21062 --- [ocp-async-2,795979dd85195cb9,27cf4b4a739dafa5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:18.989 INFO 21062 --- [ocp-async-4,b91416203610f346,aa26031a458ded3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:19.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,0db1a76bbf4caee9,86af41669a17bd06] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=0db1a76bbf4caee9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:42:19.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1757276a0d29d9eb,0bc607822c147f6d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=1757276a0d29d9eb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:42:19.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,11c642d560b4924e,9c46940a7a3d0a62] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=11c642d560b4924e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:42:19.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,d811d2cfef4f941d,2743eeb007f0dc9e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=d811d2cfef4f941d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:42:19.689 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,0db1a76bbf4caee9,86af41669a17bd06] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=0db1a76bbf4caee9, duration=12 ms] 2024-11-04 13:42:19.689 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,11c642d560b4924e,9c46940a7a3d0a62] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=11c642d560b4924e, duration=12 ms] 2024-11-04 13:42:19.690 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1757276a0d29d9eb,0bc607822c147f6d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=1757276a0d29d9eb, duration=16 ms] 2024-11-04 13:42:19.692 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,d811d2cfef4f941d,2743eeb007f0dc9e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d811d2cfef4f941d, duration=18 ms] 2024-11-04 13:42:19.745 INFO 21062 --- [ocp-schedules-22,314af9a5b9736fc3,54756ec8329ccad7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:42:20.942 WARN 21062 --- [ocp-async-8,ca05d7c2315c3c1c,4f1ae9931214a420] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:20.983 INFO 21062 --- [prometheus-minute-schedule-1,2edceba3ea676c91,0bffd628239abf58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:20.989 INFO 21062 --- [ocp-async-9,4e318964e7498e7d,8ba80a00b142f532] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:21.895 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:21.896 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:22.942 WARN 21062 --- [ocp-async-10,309312d5d4efa493,290a3092b11f9657] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:22.990 INFO 21062 --- [ocp-async-6,3ac3ba7a68bf792a,2cf26d1349f6317b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:23.983 INFO 21062 --- [prometheus-minute-schedule-1,bd5329e6a4d7b83a,972a247788ef1c4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:24.253 INFO 21062 --- [ocp-schedules-4,0f91f3f99a5447d5,fffa57e721d68994] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:42:24.896 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:24.897 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:24.943 WARN 21062 --- [ocp-async-1,96ac9b43b0230f08,29370440eae596d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:24.990 INFO 21062 --- [ocp-async-7,70c84e7f962012eb,221586a596a7009c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:26.943 WARN 21062 --- [ocp-async-4,286f3e6350470a8c,455b0907497163ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:26.983 INFO 21062 --- [prometheus-minute-schedule-1,05f5d51c9951762b,35d9b17682e1c417] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8c427cfa43152e1a,fa689aacc0d27cb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:42:26.990 INFO 21062 --- [ocp-async-5,67674d2c663d1455,b18e58d928b3df35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:27.897 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:27.898 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:28.498 INFO 21062 --- [ocp-schedules-2,c9a5adab3407f8bb,e24d87b44be3bab5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:42:28.500 INFO 21062 --- [ocp-schedules-15,19f8868d6c3c5d49,55b38d80d4c00e91] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:42:28.500 INFO 21062 --- [ocp-schedules-24,2c57da1ec4c1a67f,bc32b8b5153e67a2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:42:28.500 INFO 21062 --- [ocp-schedules-13,01166cfd05088c9c,13f5e90f0351f7f0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:42:28.502 INFO 21062 --- [ocp-schedules-5,0835eebcd8adda27,78d47e1c084b852f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:42:28.503 INFO 21062 --- [ocp-schedules-2,c9a5adab3407f8bb,e24d87b44be3bab5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:42:28.503 INFO 21062 --- [ocp-schedules-15,19f8868d6c3c5d49,55b38d80d4c00e91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:28.504 INFO 21062 --- [ocp-schedules-21,3e9d198463a062f2,783219b23e5c4608] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:42:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:42:28.505 INFO 21062 --- [ocp-schedules-4,2d477c7fa937a7dd,5b1c33f8e7d8daff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:42:28.505 INFO 21062 --- [ocp-schedules-13,01166cfd05088c9c,13f5e90f0351f7f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:28.505 INFO 21062 --- [ocp-schedules-24,2c57da1ec4c1a67f,bc32b8b5153e67a2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:42:28.508 INFO 21062 --- [ocp-schedules-4,2d477c7fa937a7dd,5b1c33f8e7d8daff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:28.510 INFO 21062 --- [ocp-schedules-11,def4241e707fc887,439aa6d19858b4c7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:42:28.510 INFO 21062 --- [ocp-schedules-11,def4241e707fc887,439aa6d19858b4c7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:42:28.510 INFO 21062 --- [ocp-schedules-22,c6c265b1972a4e08,2a7a27fadb19a1d9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:42:28.510 INFO 21062 --- [ocp-schedules-10,ddde4c59531d5d07,63a4c2d0a8c53b23] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:42:28.510 INFO 21062 --- [ocp-schedules-10,ddde4c59531d5d07,63a4c2d0a8c53b23] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:42:28.510 INFO 21062 --- [ocp-schedules-22,c6c265b1972a4e08,2a7a27fadb19a1d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:42:28.532 INFO 21062 --- [ocp-schedules-26,3d9fe13f1e4e60ad,3f66371914d68931] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:42:28.532 INFO 21062 --- [ocp-schedules-26,3d9fe13f1e4e60ad,3f66371914d68931] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:42:28.568 INFO 21062 --- [ocp-schedules-20,1dbcd31ef40e75b6,13857bde628b1323] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1332,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":363,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:42:28.568 INFO 21062 --- [ocp-schedules-20,1dbcd31ef40e75b6,13857bde628b1323] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:42:28.575 INFO 21062 --- [pool-12-thread-9,0a94f3c90faed0f5,4eb4d8bd8126d2a3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:42:28.575 INFO 21062 --- [pool-12-thread-12,0a94f3c90faed0f5,beb9c20e17953e84] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:42:28.575 INFO 21062 --- [pool-12-thread-10,0a94f3c90faed0f5,d29a658fe87fee05] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:42:28.576 INFO 21062 --- [pool-12-thread-11,0a94f3c90faed0f5,744c96c4c5bfde19] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:42:28.583 INFO 21062 --- [ForkJoinPool-11-worker-13,0a94f3c90faed0f5,c35d0657a3af80b3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:42:28.583 INFO 21062 --- [ForkJoinPool-11-worker-13,0a94f3c90faed0f5,c35d0657a3af80b3] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:42:28.583 INFO 21062 --- [ForkJoinPool-11-worker-13,0a94f3c90faed0f5,c35d0657a3af80b3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:42:28.583 INFO 21062 --- [pool-12-thread-10,0a94f3c90faed0f5,d29a658fe87fee05] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:42:28.583 INFO 21062 --- [pool-12-thread-11,0a94f3c90faed0f5,744c96c4c5bfde19] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:42:28.583 INFO 21062 --- [pool-12-thread-11,0a94f3c90faed0f5,744c96c4c5bfde19] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:42:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:42:28.589261+08:00, lastProcessTime=2024-11-04T13:42:18.589080+08:00 2024-11-04 13:42:28.602 INFO 21062 --- [pool-12-thread-12,0a94f3c90faed0f5,beb9c20e17953e84] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:42:28.605 INFO 21062 --- [ocp-schedules-7,00709a61d3a77c18,59e0f0a10df7b2af] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:42:28.606 INFO 21062 --- [pool-12-thread-12,0a94f3c90faed0f5,beb9c20e17953e84] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:42:28.606 INFO 21062 --- [ocp-schedules-7,00709a61d3a77c18,59e0f0a10df7b2af] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:42:28.608 INFO 21062 --- [ForkJoinPool-11-worker-13,0a94f3c90faed0f5,2bd496a451ec5412] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:42:28.608 INFO 21062 --- [pool-12-thread-9,0a94f3c90faed0f5,4eb4d8bd8126d2a3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:42:28.611 INFO 21062 --- [pool-12-thread-12,0a94f3c90faed0f5,beb9c20e17953e84] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:42:28.614 INFO 21062 --- [ocp-async-8,171dbc0db239500f,a34c318bf60b1525] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:42:28.675 INFO 21062 --- [ocp-schedules-21,1c810700d6018c54,96e576f9be45e362] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:42:28.675 INFO 21062 --- [ocp-schedules-21,1c810700d6018c54,96e576f9be45e362] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:42:28.716 INFO 21062 --- [ocp-schedules-11,75b0438e9274f5b6,a1252e476ed78eb8] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 13:42:28.716 INFO 21062 --- [ocp-schedules-11,75b0438e9274f5b6,a1252e476ed78eb8] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 13:42:28.716 INFO 21062 --- [ocp-schedules-11,75b0438e9274f5b6,a1252e476ed78eb8] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=364 2024-11-04 13:42:28.729 INFO 21062 --- [ocp-schedules-22,226e9d5c6021d17a,d2145e0a15fadb86] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:42:28.739 INFO 21062 --- [ocp-schedules-10,bec429b2b5b8017d,1bf4730e3b95e9df] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:42:28.744 INFO 21062 --- [ocp-schedules-11,75b0438e9274f5b6,a1252e476ed78eb8] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=364 2024-11-04 13:42:28.752 INFO 21062 --- [ocp-schedules-10,bec429b2b5b8017d,1bf4730e3b95e9df] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:42:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:42:28.589261+08:00 2024-11-04 13:42:28.849 INFO 21062 --- [ocp-schedules-25,34ed88875fe1041e,22054acabe7bc2ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:42:28.896 INFO 21062 --- [ocp-schedules-19,81dfeb1700f1d2c9,fc556ed58a02c4c8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 13:42:28.900 INFO 21062 --- [ocp-schedules-19,81dfeb1700f1d2c9,fc556ed58a02c4c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:28.907 INFO 21062 --- [ocp-schedules-26,e92d10c3fbffa5d2,56fe827d8c87ddef] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:42:28.907 INFO 21062 --- [ocp-schedules-26,e92d10c3fbffa5d2,56fe827d8c87ddef] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:42:28.935 ERROR 21062 --- [ocp-schedules-19,81dfeb1700f1d2c9,fc556ed58a02c4c8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:42:28.935 INFO 21062 --- [ocp-schedules-19,81dfeb1700f1d2c9,fc556ed58a02c4c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 13:42:28.937 INFO 21062 --- [ocp-schedules-29,374fe1ad0309d3e2,9fe0bca086b06851] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:42:28.943 WARN 21062 --- [ocp-async-6,09866492831e0943,f9dcab00335bc174] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:28.971 INFO 21062 --- [ocp-schedules-30,6eb021249b3b33cd,d04f5a9779b72d47] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:42:28.975 INFO 21062 --- [ocp-schedules-30,6eb021249b3b33cd,d04f5a9779b72d47] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:42:28.975 INFO 21062 --- [ocp-schedules-30,6eb021249b3b33cd,d04f5a9779b72d47] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:42:28.990 INFO 21062 --- [ocp-async-7,e5f4f88da72e2f14,8b8517a5cbb90384] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:29.214 INFO 21062 --- [ocp-schedules-23,4e8539a53c37014e,c52cd59efb48a276] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:42:29.214 INFO 21062 --- [ocp-schedules-23,4e8539a53c37014e,c52cd59efb48a276] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:42:29.221 INFO 21062 --- [ocp-schedules-23,4e8539a53c37014e,c52cd59efb48a276] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:42:29.221 INFO 21062 --- [ocp-schedules-23,4e8539a53c37014e,c52cd59efb48a276] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:42:29.227 INFO 21062 --- [ocp-schedules-23,4e8539a53c37014e,c52cd59efb48a276] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:42:29.227 INFO 21062 --- [ocp-schedules-23,4e8539a53c37014e,c52cd59efb48a276] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709925194352054272 2024-11-04 13:42:29.296 INFO 21062 --- [ocp-schedules-24,77317c06a3665f25,99bab39ed8c0ca7c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:42:29.296 INFO 21062 --- [ocp-schedules-24,77317c06a3665f25,99bab39ed8c0ca7c] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:42:29.324 INFO 21062 --- [pool-33-thread-1,77317c06a3665f25,02d9954c0f37104e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:42:29.352 WARN 21062 --- [ocp-schedules-24,77317c06a3665f25,99bab39ed8c0ca7c] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:42:29.352 INFO 21062 --- [ocp-schedules-24,77317c06a3665f25,99bab39ed8c0ca7c] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:42:29.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,4c3528b272090214,d85eb5695d1de64e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=4c3528b272090214, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:42:29.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,efedb840a3b8e92a,5bf5ca2256017614] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=efedb840a3b8e92a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:42:29.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6f0067e8d4b1413c,85a5bad599b9bfb0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=6f0067e8d4b1413c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:42:29.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,7910c654965700fb,8b05ed82424f501b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=7910c654965700fb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:42:29.634 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,4c3528b272090214,d85eb5695d1de64e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=4c3528b272090214, duration=8 ms] 2024-11-04 13:42:29.634 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,efedb840a3b8e92a,5bf5ca2256017614] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=efedb840a3b8e92a, duration=10 ms] 2024-11-04 13:42:29.634 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6f0067e8d4b1413c,85a5bad599b9bfb0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=6f0067e8d4b1413c, duration=11 ms] 2024-11-04 13:42:29.636 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,7910c654965700fb,8b05ed82424f501b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=7910c654965700fb, duration=13 ms] 2024-11-04 13:42:29.748 INFO 21062 --- [ocp-schedules-2,8b2e31cb7766cc40,c279d3254aac841c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:42:29.983 INFO 21062 --- [prometheus-minute-schedule-1,8bdb5b9321febea2,78df735401e89370] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:42:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:42:30.433 INFO 21062 --- [ocp-schedules-1,e5ccca35c803b399,89ae578d00d93825] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:42:30.438 INFO 21062 --- [ocp-schedules-23,0daa91a6b06bf989,f45adc57df40bdf9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:42:30.438 INFO 21062 --- [ocp-schedules-23,0daa91a6b06bf989,f45adc57df40bdf9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:42:30.440 INFO 21062 --- [ocp-schedules-1,e5ccca35c803b399,89ae578d00d93825] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:42:30.440 INFO 21062 --- [ocp-schedules-1,e5ccca35c803b399,89ae578d00d93825] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:42:30.465 INFO 21062 --- [ocp-schedules-23,0daa91a6b06bf989,f45adc57df40bdf9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:42:30.465 INFO 21062 --- [ocp-schedules-23,0daa91a6b06bf989,f45adc57df40bdf9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:42:30.466 INFO 21062 --- [ocp-schedules-6,5271430f183680f6,52a19c15989ba666] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:42:30.599 INFO 21062 --- [ocp-schedules-18,8f75ffddc36e78c9,5c9261df5cc3486b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:42:30.599+08:00, clusters=[1] 2024-11-04 13:42:30.601 INFO 21062 --- [ocp-schedules-18,8f75ffddc36e78c9,5c9261df5cc3486b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:42:30.643 INFO 21062 --- [ocp-schedules-21,f82e88dda6e5eb75,3b28d1b2c08ad3f0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:42:30.643 INFO 21062 --- [ocp-schedules-21,f82e88dda6e5eb75,3b28d1b2c08ad3f0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:42:30.643 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:42:30.643 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:42:30.653 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:42:30.653 INFO 21062 --- [ocp-schedules-21,f82e88dda6e5eb75,3b28d1b2c08ad3f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:42:30.654 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:42:30.655 WARN 21062 --- [ocp-schedules-21,f82e88dda6e5eb75,3b28d1b2c08ad3f0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:42:30.655 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:42:30.655 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:42:30.655 INFO 21062 --- [ocp-schedules-21,f82e88dda6e5eb75,3b28d1b2c08ad3f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:42:30.655 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:42:30.656 INFO 21062 --- [ocp-schedules-10,b3490954ddcc999b,7bbc79e1caaf8659] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:42:30.656 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:42:30.657 WARN 21062 --- [ocp-schedules-21,f82e88dda6e5eb75,3b28d1b2c08ad3f0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:42:30.657 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:42:30.657 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:42:30.659 INFO 21062 --- [ocp-schedules-10,b3490954ddcc999b,7bbc79e1caaf8659] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:30.662 ERROR 21062 --- [ocp-schedules-21,f82e88dda6e5eb75,3b28d1b2c08ad3f0] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:42:30.713 ERROR 21062 --- [ocp-schedules-10,b3490954ddcc999b,7bbc79e1caaf8659] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:42:30.713 INFO 21062 --- [ocp-schedules-10,b3490954ddcc999b,7bbc79e1caaf8659] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:42:30.741 INFO 21062 --- [ocp-schedules-5,479fe645ea119fa4,0709ced9feb4358c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:42:30.741+08:00, clusters=[1] 2024-11-04 13:42:30.745 INFO 21062 --- [ocp-schedules-5,479fe645ea119fa4,0709ced9feb4358c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:42:30.898 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:30.899 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:30.943 WARN 21062 --- [ocp-async-7,56f571cad60da7f0,ac898722950dfa40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:30.990 INFO 21062 --- [ocp-async-8,61f99011e4beb4a2,91094bf52e72a681] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:32.943 WARN 21062 --- [ocp-async-8,229dbed872acff92,ed6ab5607ef045eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:32.983 INFO 21062 --- [prometheus-minute-schedule-1,09923b07d59dfe7c,0fe4457e7d6884d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:32.990 INFO 21062 --- [ocp-async-10,78053cfda6abd4c0,f0a6c150f97e5897] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:33.293 INFO 21062 --- [ocp-schedules-12,f584c5575d99dbcf,4b15dfd40797dc37] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:42:33.293 INFO 21062 --- [ocp-schedules-12,849f87fa11bc7586,c3ec836169455df2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:42:33.297 INFO 21062 --- [ocp-schedules-12,849f87fa11bc7586,c3ec836169455df2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:33.326 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:42:33.326 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:42:33.899 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:33.900 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:34.325 INFO 21062 --- [ocp-schedules-14,995f9b8569594b9d,ff3fa7eb76c7effd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:42:34.944 WARN 21062 --- [ocp-async-10,6617f515d2e398a8,b4c228aae8d46bc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:34.959 INFO 21062 --- [ocp-schedules-8,357b0d42c1cd71f3,fd1e91232f24d7f9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:42:34.963 INFO 21062 --- [ocp-schedules-8,357b0d42c1cd71f3,fd1e91232f24d7f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:34.990 INFO 21062 --- [ocp-async-1,ba03f0728a5645d3,9805cca55dea0d66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:35.049 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:42:35.051 ERROR 21062 --- [ocp-schedules-8,357b0d42c1cd71f3,fd1e91232f24d7f9] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:42:35.051 INFO 21062 --- [ocp-schedules-8,357b0d42c1cd71f3,fd1e91232f24d7f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:42:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c9cb402756f6eb30,3652704e597ad723] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:36.900 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:36.901 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:36.944 WARN 21062 --- [ocp-async-1,cd8c3a7c4a061dc0,a3d376c4276b0f61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:36.991 INFO 21062 --- [ocp-async-4,4da810b20a97504b,e14df699026b87c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:38.505 INFO 21062 --- [ocp-schedules-15,19f8868d6c3c5d49,55b38d80d4c00e91] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:42:38.505 INFO 21062 --- [ocp-schedules-13,01166cfd05088c9c,13f5e90f0351f7f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:42:38.508 INFO 21062 --- [ocp-schedules-4,2d477c7fa937a7dd,5b1c33f8e7d8daff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:42:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:42:38.590132+08:00, lastProcessTime=2024-11-04T13:42:28.589261+08:00 2024-11-04 13:42:38.614 INFO 21062 --- [ocp-async-2,ed5ec569f2785e55,ddb71157617e68bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:42:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:42:38.590132+08:00 2024-11-04 13:42:38.944 WARN 21062 --- [ocp-async-5,1123257a6ede5952,69cf3164e79293af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e1de984533ebdb49,99f85c9b731afe02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:38.991 INFO 21062 --- [ocp-async-3,43be0f64a998998d,c2834be5ac050874] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:39.632 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,b09247e078ffbc75,e6c40b21751ceda9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=b09247e078ffbc75, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:42:39.632 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c0519fadae4e31e0,8d9eb1571649877d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=c0519fadae4e31e0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:42:39.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c0519fadae4e31e0,8d9eb1571649877d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=c0519fadae4e31e0, duration=5 ms] 2024-11-04 13:42:39.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,b09247e078ffbc75,e6c40b21751ceda9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=b09247e078ffbc75, duration=8 ms] 2024-11-04 13:42:39.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,86dce78db4f75a4e,c878838e8e1adea6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=86dce78db4f75a4e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:42:39.633 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,fa2fd0b0d5753f81,e76b53339b3bdbe8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=fa2fd0b0d5753f81, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:42:39.634 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,86dce78db4f75a4e,c878838e8e1adea6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=86dce78db4f75a4e, duration=6 ms] 2024-11-04 13:42:39.636 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,fa2fd0b0d5753f81,e76b53339b3bdbe8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=fa2fd0b0d5753f81, duration=8 ms] 2024-11-04 13:42:39.751 INFO 21062 --- [ocp-schedules-5,e1d49c31ecbff18b,05b5b78591c0a276] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:42:39.901 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:39.902 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:40.004 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:42:40.004 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:42:40.944 WARN 21062 --- [ocp-async-3,aca834bad52a7065,d5fb46217b5b71c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:40.991 INFO 21062 --- [ocp-async-2,c9ba76e08f8908f6,cf188f3e2381b555] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:42:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:42:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:42:41.802 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:42:41.802 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:42:41.829 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:42:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:42:41.983 INFO 21062 --- [prometheus-minute-schedule-1,b8c79f241d50d1d9,1948a304f26ee9e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:42.902 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:42.904 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:42.944 WARN 21062 --- [ocp-async-2,afaa7954bbe72ff6,1b2f129799fa2258] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:42.991 INFO 21062 --- [ocp-async-8,4a3320fb0c5f9417,373aef197ac57aa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:44.349 INFO 21062 --- [ocp-schedules-22,eb437cf60199f926,81192b8459ddd446] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:42:44.945 WARN 21062 --- [ocp-async-8,b4376d57784ace7a,4918a3de72cdef60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:44.983 INFO 21062 --- [prometheus-minute-schedule-1,1d3e8468448800e0,c4046df716811200] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:44.991 INFO 21062 --- [ocp-async-9,44bb4b4b644a5d1a,19f458e9aa27b181] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:45.904 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:45.905 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:46.945 WARN 21062 --- [ocp-async-9,94deea72eec465be,b3e61872e6b4f67f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:46.991 INFO 21062 --- [ocp-async-6,81a0e1113bb7d345,30b7ccb0b2d8d142] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:47.983 INFO 21062 --- [prometheus-minute-schedule-1,691c9d52f959ebf5,341d2575efc1e9b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:42:48.590708+08:00, lastProcessTime=2024-11-04T13:42:38.590132+08:00 2024-11-04 13:42:48.614 INFO 21062 --- [ocp-async-9,c6624bcd46c6e0a9,bfb8819cbeb913d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:42:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:42:48.590708+08:00 2024-11-04 13:42:48.905 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:48.906 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:48.946 WARN 21062 --- [ocp-async-2,791ac5ee6a6841a7,c504b7cf53c5757c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:48.991 INFO 21062 --- [ocp-async-8,d03fbc400754d85c,4a2037548212694b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:49.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,1102e7e272457068,5e80293922829a8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=1102e7e272457068, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:42:49.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6888e175261f20e8,d3b972d385884c0c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=6888e175261f20e8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:42:49.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,50906d92b0597480,ee9bd2cedf86969e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=50906d92b0597480, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:42:49.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,a9cb60eb91984da2,ce8001230a52bcc7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=a9cb60eb91984da2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:42:49.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6888e175261f20e8,d3b972d385884c0c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=6888e175261f20e8, duration=8 ms] 2024-11-04 13:42:49.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,a9cb60eb91984da2,ce8001230a52bcc7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a9cb60eb91984da2, duration=9 ms] 2024-11-04 13:42:49.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,1102e7e272457068,5e80293922829a8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=1102e7e272457068, duration=11 ms] 2024-11-04 13:42:49.689 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,50906d92b0597480,ee9bd2cedf86969e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=50906d92b0597480, duration=15 ms] 2024-11-04 13:42:49.753 INFO 21062 --- [ocp-schedules-3,6c49da54b7c5429f,c6e3f1b98f4b5fbf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:42:50.443 INFO 21062 --- [ocp-schedules-28,e20d394f967c01a4,35636a5c3603ce54] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:42:50.448 INFO 21062 --- [ocp-schedules-28,e20d394f967c01a4,35636a5c3603ce54] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:42:50.448 INFO 21062 --- [ocp-schedules-28,e20d394f967c01a4,35636a5c3603ce54] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:42:50.947 WARN 21062 --- [ocp-async-8,1804162f1020b083,361a58a588b88d1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:50.983 INFO 21062 --- [prometheus-minute-schedule-1,799bc953ac8da95c,750519583f6e1787] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:50.992 INFO 21062 --- [ocp-async-9,3d84fc4c2aa6b33c,eef543becd6ed06f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:51.906 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:51.908 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:52.947 WARN 21062 --- [ocp-async-9,c85bb46deaab5198,1edebbb66e6094f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:52.992 INFO 21062 --- [ocp-async-6,bd2a452b2ff22928,5cb37cd54efdf2a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:53.983 INFO 21062 --- [prometheus-minute-schedule-1,36c0c6115bb3f119,6f2b3492bbf78a72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:54.393 INFO 21062 --- [ocp-schedules-28,3c561ffbd291d8b9,bb14fb9a380b3771] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:42:54.908 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:54.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:54.947 WARN 21062 --- [ocp-async-6,fbfbcc7675298868,fe00e6941d6aeb48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:54.992 INFO 21062 --- [ocp-async-4,acbe3ae560991316,3c377b3c4e23c5b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:56.947 WARN 21062 --- [ocp-async-4,240b419d166fbb50,74e0da57d89d6ee6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e153aca317ee0851,e915d07341b19a78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:42:56.992 INFO 21062 --- [ocp-async-10,521dc18c7a60376c,fbc9ff446f10d173] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:57.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:42:57.910 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:42:58.500 INFO 21062 --- [ocp-schedules-9,71b37e9d2d595e12,85b60f9ad7b1e49e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:42:58.500 INFO 21062 --- [ocp-schedules-30,df4b59490087c360,f13f6b97cbb93021] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:42:58.502 INFO 21062 --- [ocp-schedules-2,447594d21c2879ae,707f652f539185b9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:42:58.504 INFO 21062 --- [ocp-schedules-9,71b37e9d2d595e12,85b60f9ad7b1e49e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:58.504 INFO 21062 --- [ocp-schedules-30,df4b59490087c360,f13f6b97cbb93021] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:42:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:42:58.504 INFO 21062 --- [ocp-schedules-22,630d8ab0cddd7e16,4ce2a780fef49642] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:42:58.511 INFO 21062 --- [ocp-schedules-3,d8339177190c6b62,c9babb2bb7459241] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:42:58.511 INFO 21062 --- [ocp-schedules-15,48d366d116474275,2e9f48a50d713834] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:42:58.511 INFO 21062 --- [ocp-schedules-3,d8339177190c6b62,c9babb2bb7459241] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:42:58.511 INFO 21062 --- [ocp-schedules-15,48d366d116474275,2e9f48a50d713834] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:42:58.512 INFO 21062 --- [ocp-schedules-23,783d1389b353a813,2ab6f6fb6f01b793] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:42:58.512 INFO 21062 --- [ocp-schedules-23,783d1389b353a813,2ab6f6fb6f01b793] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:42:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:42:58.589944+08:00, lastProcessTime=2024-11-04T13:42:48.590708+08:00 2024-11-04 13:42:58.614 INFO 21062 --- [ocp-async-4,81f5b75a8f9ee666,c0e17d542b328cb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:42:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:42:58.589944+08:00 2024-11-04 13:42:58.941 INFO 21062 --- [ocp-schedules-18,c27ae8164c1e425b,517308b5824014ed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:42:58.947 WARN 21062 --- [ocp-async-9,b1a186bc2f622ed7,f75db8600ce989a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:42:58.992 INFO 21062 --- [ocp-async-6,458b719f9b0c8d16,d64a65c95c16984a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:42:59.637 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,44c45c5b90ac9f3f,283bd9e04e24c911] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=44c45c5b90ac9f3f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:42:59.637 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,4cfa7f8dd67bc22e,cc5aadd31ba9c90c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=4cfa7f8dd67bc22e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:42:59.637 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,0e1f13bcdd152d62,cf8a56fc415845ab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=0e1f13bcdd152d62, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:42:59.637 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,311dfd21e206f049,dbbd29469c214774] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=311dfd21e206f049, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:42:59.639 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,44c45c5b90ac9f3f,283bd9e04e24c911] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=44c45c5b90ac9f3f, duration=6 ms] 2024-11-04 13:42:59.639 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,0e1f13bcdd152d62,cf8a56fc415845ab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=0e1f13bcdd152d62, duration=6 ms] 2024-11-04 13:42:59.639 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,311dfd21e206f049,dbbd29469c214774] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=311dfd21e206f049, duration=10 ms] 2024-11-04 13:42:59.641 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,4cfa7f8dd67bc22e,cc5aadd31ba9c90c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=4cfa7f8dd67bc22e, duration=12 ms] 2024-11-04 13:42:59.757 INFO 21062 --- [ocp-schedules-10,a5c3c4a93d2fd349,16e25ececf91b353] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:42:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1d0ec2e0b5ac0e18,f4248fff0ff7cd16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730698980, interval=60 2024-11-04 13:43:00.910 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:00.912 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:00.947 WARN 21062 --- [ocp-async-6,66d8b8b023ef5163,5431dcbf8471cd12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:00.993 INFO 21062 --- [ocp-async-7,f17d7fc2f06eb75c,ea88280e2708872b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:02.948 WARN 21062 --- [ocp-async-7,5aadb5ebeedc58bb,9711f97a8ba5139d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f3703d77e8487e9e,c124068eccc2eb96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:02.993 INFO 21062 --- [ocp-async-5,eb00592ebc7fba6d,aaaef1291325a068] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:03.326 INFO 21062 --- [ocp-schedules-12,849f87fa11bc7586,c3ec836169455df2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:43:03.327 INFO 21062 --- [ocp-schedules-12,ad6188c6eeb24061,4ca0e823063c4025] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:43:03.331 INFO 21062 --- [ocp-schedules-12,ad6188c6eeb24061,4ca0e823063c4025] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:03.360 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:43:03.360 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:43:03.912 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:03.914 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:04.419 INFO 21062 --- [ocp-schedules-22,b52028e9911822d5,52a0ca62b16e5ba6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:43:04.948 WARN 21062 --- [ocp-async-5,3e3a4436a0b366d9,8415e4b2139c5414] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:04.993 INFO 21062 --- [ocp-async-3,60d1e1a1ca1ec3e8,832eba8ad54f7efa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a69c1d48d74e13d7,046fefdd2f743897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:06.914 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:06.915 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:06.948 WARN 21062 --- [ocp-async-3,3bb3d79d3eae3204,60f77515582bbb06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:06.993 INFO 21062 --- [ocp-async-2,ee0ad33f994103c2,c39547b12cbffb0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:08.505 INFO 21062 --- [ocp-schedules-30,df4b59490087c360,f13f6b97cbb93021] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:43:08.505 INFO 21062 --- [ocp-schedules-9,71b37e9d2d595e12,85b60f9ad7b1e49e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:43:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:43:08.590347+08:00, lastProcessTime=2024-11-04T13:42:58.589944+08:00 2024-11-04 13:43:08.614 INFO 21062 --- [ocp-async-3,7286e60c8a5fe439,9ac885669ee51586] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:43:08.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:43:08.590347+08:00 2024-11-04 13:43:08.948 WARN 21062 --- [ocp-async-7,f6326d2b92072147,01105639cfe4063e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:08.983 INFO 21062 --- [prometheus-minute-schedule-1,6acaccbd3705ccad,d56efd20e6a92fef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:08.993 INFO 21062 --- [ocp-async-10,38a8b75aa9ccd4ab,e996bb603d4e7264] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:09.760 INFO 21062 --- [ocp-schedules-6,1c0ce9a74e515214,0a1b95beb15077e0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:43:09.915 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:09.917 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:10.002 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:43:10.002 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:43:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:43:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:43:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:43:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:43:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:43:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:43:10.450 INFO 21062 --- [ocp-schedules-3,5da941605d13c045,167485059da6d8a4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:43:10.456 INFO 21062 --- [ocp-schedules-3,5da941605d13c045,167485059da6d8a4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:43:10.456 INFO 21062 --- [ocp-schedules-3,5da941605d13c045,167485059da6d8a4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:43:10.885 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,844176739842dd15,aee43e0858a36e1e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=844176739842dd15, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:43:10.885 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,0798fdb367edd6d3,0d9d0a9888dd86be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=0798fdb367edd6d3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:43:10.885 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,53b68d98036b9109,e8abf3a5a6628b2d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=53b68d98036b9109, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:43:10.885 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a26c02ae50aff9b8,d0fd26b34b11805e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=a26c02ae50aff9b8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:43:10.886 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,53b68d98036b9109,e8abf3a5a6628b2d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=53b68d98036b9109, duration=8 ms] 2024-11-04 13:43:10.886 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,0798fdb367edd6d3,0d9d0a9888dd86be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=0798fdb367edd6d3, duration=10 ms] 2024-11-04 13:43:10.886 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,844176739842dd15,aee43e0858a36e1e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=844176739842dd15, duration=8 ms] 2024-11-04 13:43:10.888 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a26c02ae50aff9b8,d0fd26b34b11805e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a26c02ae50aff9b8, duration=12 ms] 2024-11-04 13:43:10.948 WARN 21062 --- [ocp-async-5,17fa1f3474ed090f,ea5285dddfff40aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:10.994 INFO 21062 --- [ocp-async-1,a6dd62e4022fa0fe,8defb3830c5caa93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:11.323 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=2, affectRows=2 2024-11-04 13:43:11.323 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=2, validCount=2 2024-11-04 13:43:11.324 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=4, affectRows=4 2024-11-04 13:43:11.324 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=10, affectRows=10 2024-11-04 13:43:11.324 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=4, validCount=4 2024-11-04 13:43:11.324 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=10, validCount=10 2024-11-04 13:43:11.324 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=20, affectRows=20 2024-11-04 13:43:11.325 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=20, validCount=20 2024-11-04 13:43:11.325 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=30, affectRows=30 2024-11-04 13:43:11.325 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=33, affectRows=33 2024-11-04 13:43:11.326 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=30, validCount=30 2024-11-04 13:43:11.326 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=33, validCount=33 2024-11-04 13:43:11.326 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=27, affectRows=27 2024-11-04 13:43:11.326 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=27, validCount=27 2024-11-04 13:43:11.327 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=38, affectRows=38 2024-11-04 13:43:11.327 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=38, validCount=38 2024-11-04 13:43:11.327 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=53, affectRows=53 2024-11-04 13:43:11.327 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=59, affectRows=59 2024-11-04 13:43:11.328 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=51, affectRows=51 2024-11-04 13:43:11.328 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698920, nodeCount=37, affectRows=37 2024-11-04 13:43:11.328 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=51, validCount=51 2024-11-04 13:43:11.328 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=37, validCount=37 2024-11-04 13:43:11.328 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=58, validCount=58 2024-11-04 13:43:11.328 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698920, nodeCount=51, validCount=51 2024-11-04 13:43:11.983 INFO 21062 --- [prometheus-minute-schedule-1,5fd50faa8229293f,5f04aa11a688f289] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:12.918 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:12.919 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:12.948 WARN 21062 --- [ocp-async-3,6cc94e62a01fbcea,fb507435e5461840] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:12.994 INFO 21062 --- [ocp-async-2,3e93efcc8c485b84,2014ad0ffd4f356b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:13.984 INFO 21062 --- [prometheus-cache-monitor1,7e6b588d9dd1d3fe,24eda87012c5e33b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1374,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:43:13.984 INFO 21062 --- [prometheus-cache-monitor1,7e6b588d9dd1d3fe,24eda87012c5e33b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:43:14.097 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:37:14.077, end=2024-11-04T13:42:14.095} 2024-11-04 13:43:14.098 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:37:14.077, end=2024-11-04T13:42:14.096} 2024-11-04 13:43:14.106 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:43:14.106 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:43:14.106 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:43:14.106 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:43:14.460 INFO 21062 --- [ocp-schedules-7,7eae5214178b93c4,0a94d7ff2881debf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:43:14.680 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:39:14.66, end=2024-11-04T13:41:14.677} 2024-11-04 13:43:14.684 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:43:14.684 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:43:14.949 WARN 21062 --- [ocp-async-4,08e8362646a8f35e,80d89f3e1d3187fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:14.983 INFO 21062 --- [prometheus-minute-schedule-1,fb963d2af72db301,5063a59036463e5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:14.994 INFO 21062 --- [ocp-async-8,776edb20ed2ded20,35104373974f9d0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:15.919 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:15.920 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:16.949 WARN 21062 --- [ocp-async-10,eeef9b7af6a24b72,174da714f77c70ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:16.994 INFO 21062 --- [ocp-async-9,ebb3585af703c361,cb5028aad8475548] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:17.983 INFO 21062 --- [prometheus-minute-schedule-1,410c0ab69e45b774,0efb903f7cb560b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:43:18.590849+08:00, lastProcessTime=2024-11-04T13:43:08.590347+08:00 2024-11-04 13:43:18.615 INFO 21062 --- [ocp-async-8,b8d3543b3556ab07,2b0190c8ea8bff80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:43:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:43:18.590849+08:00 2024-11-04 13:43:18.920 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:18.921 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:18.949 WARN 21062 --- [ocp-async-3,08ed372f7f9b83fa,7c225cc509e6f485] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:18.994 INFO 21062 --- [ocp-async-2,d065bd9392ecb291,7dfb6efd055a98c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:19.721 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,0340f4dbde26ab9b,325fdb1f4f21b42a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=0340f4dbde26ab9b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:43:19.721 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,fa0cd67e275dc290,d0085f4e8cfd422d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=fa0cd67e275dc290, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:43:19.721 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d3648ce6f8feb002,3f3d8b706c95e932] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=d3648ce6f8feb002, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:43:19.722 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9102c2c9291b89c6,8aeee49e4bf25356] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=9102c2c9291b89c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:43:19.722 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,0340f4dbde26ab9b,325fdb1f4f21b42a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=0340f4dbde26ab9b, duration=10 ms] 2024-11-04 13:43:19.722 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,fa0cd67e275dc290,d0085f4e8cfd422d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=fa0cd67e275dc290, duration=7 ms] 2024-11-04 13:43:19.722 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9102c2c9291b89c6,8aeee49e4bf25356] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=9102c2c9291b89c6, duration=7 ms] 2024-11-04 13:43:19.724 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d3648ce6f8feb002,3f3d8b706c95e932] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d3648ce6f8feb002, duration=12 ms] 2024-11-04 13:43:19.763 INFO 21062 --- [ocp-schedules-6,97703e41d83ed989,924fc0e8595e1882] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:43:20.949 WARN 21062 --- [ocp-async-4,97726766a789164b,e4d2279667f5ed31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:20.983 INFO 21062 --- [prometheus-minute-schedule-1,d533359927d145a3,99bc98e6d67ebdee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:20.994 INFO 21062 --- [ocp-async-10,c46fd761845768b2,92535aeaf10963c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:21.921 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:21.922 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:22.949 WARN 21062 --- [ocp-async-8,fc289c1df0ab99cf,3708993c78347678] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:22.995 INFO 21062 --- [ocp-async-9,a384ad4ad9c8482b,783fe367fcd50652] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:23.983 INFO 21062 --- [prometheus-minute-schedule-1,efff100b3207f560,345001f4f908782b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:24.486 INFO 21062 --- [ocp-schedules-3,d7f63151d6d1d1df,17af79d3b79f6f0d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:43:24.922 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:24.924 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:24.949 WARN 21062 --- [ocp-async-1,37d1d96d68eb86d4,120782c0b8ee4adc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:24.995 INFO 21062 --- [ocp-async-6,0b0b8ec1b627a3e0,6af6c1e7e1840ed0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:26.950 WARN 21062 --- [ocp-async-2,96adc914409e5cad,58ee8a0262a45b29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:26.983 INFO 21062 --- [prometheus-minute-schedule-1,c3ac9eee2c959f91,2817e67890303d08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ad0a2cceaed373f3,9000ba64830f1894] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:43:26.995 INFO 21062 --- [ocp-async-7,5b30e3dab1d0c82e,f9db9cb7570159bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:27.924 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:27.925 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:28.498 INFO 21062 --- [ocp-schedules-16,96272005167b632f,420d9037512ed9a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:43:28.499 INFO 21062 --- [ocp-schedules-2,1d05fb9750adf340,401e0003c561f16d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:43:28.500 INFO 21062 --- [ocp-schedules-26,b841d9e96f72b398,ebf8be9ec1827da0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:43:28.500 INFO 21062 --- [ocp-schedules-29,a2b602516c118126,1b3fb3bb558bc1a9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:43:28.502 INFO 21062 --- [ocp-schedules-16,96272005167b632f,420d9037512ed9a1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:43:28.502 INFO 21062 --- [ocp-schedules-4,b5456a5c262ec2a6,f58a063932bb495e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:43:28.504 INFO 21062 --- [ocp-schedules-26,b841d9e96f72b398,ebf8be9ec1827da0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:28.504 INFO 21062 --- [ocp-schedules-29,a2b602516c118126,1b3fb3bb558bc1a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:28.504 INFO 21062 --- [ocp-schedules-30,14daef1f730d815b,95e0ae902d3da911] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:43:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:43:28.505 INFO 21062 --- [ocp-schedules-2,1d05fb9750adf340,401e0003c561f16d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:43:28.511 INFO 21062 --- [ocp-schedules-13,df789fe5ae35f5bf,a57aad0a05bf5774] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:43:28.511 INFO 21062 --- [ocp-schedules-13,df789fe5ae35f5bf,a57aad0a05bf5774] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:43:28.512 INFO 21062 --- [ocp-schedules-1,195cf693d63f419b,b2ed717c46aef123] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:43:28.512 INFO 21062 --- [ocp-schedules-1,195cf693d63f419b,b2ed717c46aef123] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:43:28.512 INFO 21062 --- [ocp-schedules-25,84159ceec44327f1,6085436e69bbde1f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:43:28.512 INFO 21062 --- [ocp-schedules-25,84159ceec44327f1,6085436e69bbde1f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:43:28.565 INFO 21062 --- [ocp-schedules-18,45415820a1df5d4f,2c164ed598aa8fe7] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 13:43:28.565 INFO 21062 --- [ocp-schedules-18,45415820a1df5d4f,2c164ed598aa8fe7] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 13:43:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:43:28.589296+08:00, lastProcessTime=2024-11-04T13:43:18.590849+08:00 2024-11-04 13:43:28.615 INFO 21062 --- [ocp-async-4,928e66418cf98e11,7a876d2cb2c17e1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:43:28.731 INFO 21062 --- [ocp-schedules-22,12cad53bbd9ce37f,9906766eae356420] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:43:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:43:28.589296+08:00 2024-11-04 13:43:28.849 INFO 21062 --- [ocp-schedules-14,47b87a261dded4da,055fc55cb58fe413] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:43:28.852 INFO 21062 --- [ocp-schedules-14,47b87a261dded4da,055fc55cb58fe413] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:28.853 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:43:28.912 INFO 21062 --- [ocp-schedules-2,acf2f0fabc451d38,1da7e212dd003ff0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:43:28.912 INFO 21062 --- [ocp-schedules-2,acf2f0fabc451d38,1da7e212dd003ff0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:43:28.943 INFO 21062 --- [ocp-schedules-30,2f5a2c8e8cd26895,1e1f3be76ea5d91b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:43:28.950 WARN 21062 --- [ocp-async-9,75ba64a4d37d7bcd,b0e27820b5180a72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:28.975 INFO 21062 --- [ocp-schedules-1,8a4acfb9b9711f4f,e03e7077b78ede4a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:43:28.980 INFO 21062 --- [ocp-schedules-1,8a4acfb9b9711f4f,e03e7077b78ede4a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:43:28.980 INFO 21062 --- [ocp-schedules-1,8a4acfb9b9711f4f,e03e7077b78ede4a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:43:28.995 INFO 21062 --- [ocp-async-6,6a55e7fef45a8fd8,4c026d50898ca533] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:29.039 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:43:29.039 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:43:29.046 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:43:29.046 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:43:29.046 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:43:29.057 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:43:29.057 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:43:29.057 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:43:29.060 INFO 21062 --- [ocp-schedules-3,162aa62c4da4ee00,35836572f9529a8c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:43:29.228 INFO 21062 --- [ocp-schedules-18,558d0853c6e96906,47b71794b6b9dcf4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:43:29.228 INFO 21062 --- [ocp-schedules-18,558d0853c6e96906,47b71794b6b9dcf4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:43:29.230 INFO 21062 --- [ocp-schedules-18,558d0853c6e96906,47b71794b6b9dcf4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:43:29.230 INFO 21062 --- [ocp-schedules-18,558d0853c6e96906,47b71794b6b9dcf4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:43:29.230 INFO 21062 --- [ocp-schedules-18,558d0853c6e96906,47b71794b6b9dcf4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:43:29.230 INFO 21062 --- [ocp-schedules-18,558d0853c6e96906,47b71794b6b9dcf4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709925194352054272 2024-11-04 13:43:29.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,72d11b6f52c821b4,614bbcde55a82e62] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=72d11b6f52c821b4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:43:29.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,e1e6330360102897,1b8e70dbd2416c59] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=e1e6330360102897, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:43:29.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f52b017d4175f1b0,49735f58838e599b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=f52b017d4175f1b0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:43:29.687 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,d8ba85091160e6cd,e38cb002ceaeeb5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=d8ba85091160e6cd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:43:29.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,72d11b6f52c821b4,614bbcde55a82e62] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=72d11b6f52c821b4, duration=8 ms] 2024-11-04 13:43:29.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f52b017d4175f1b0,49735f58838e599b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=f52b017d4175f1b0, duration=8 ms] 2024-11-04 13:43:29.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,e1e6330360102897,1b8e70dbd2416c59] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=e1e6330360102897, duration=11 ms] 2024-11-04 13:43:29.690 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,d8ba85091160e6cd,e38cb002ceaeeb5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d8ba85091160e6cd, duration=13 ms] 2024-11-04 13:43:29.765 INFO 21062 --- [ocp-schedules-24,340e3202de953bea,e629dfb1a1661f09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:43:29.983 INFO 21062 --- [prometheus-minute-schedule-1,040a0d1cd8eb128b,bf2131064d914bab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:43:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:43:30.459 INFO 21062 --- [ocp-schedules-15,f0423c9c3cd2a30a,e0ee369b2e59e3d1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:43:30.465 INFO 21062 --- [ocp-schedules-15,f0423c9c3cd2a30a,e0ee369b2e59e3d1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:43:30.465 INFO 21062 --- [ocp-schedules-15,f0423c9c3cd2a30a,e0ee369b2e59e3d1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:43:30.465 INFO 21062 --- [ocp-schedules-23,e4261cde9b2f960b,7fad85c0630e5538] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:43:30.465 INFO 21062 --- [ocp-schedules-23,e4261cde9b2f960b,7fad85c0630e5538] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:43:30.492 INFO 21062 --- [ocp-schedules-23,e4261cde9b2f960b,7fad85c0630e5538] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:43:30.492 INFO 21062 --- [ocp-schedules-23,e4261cde9b2f960b,7fad85c0630e5538] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:43:30.492 INFO 21062 --- [ocp-schedules-9,9098690183e093a9,2dd3e853a6325cbd] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:43:30.625 INFO 21062 --- [ocp-schedules-11,81b5cf8a3cc6f36b,e7cbd6e8d04acb03] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:43:30.625+08:00, clusters=[1] 2024-11-04 13:43:30.627 INFO 21062 --- [ocp-schedules-11,81b5cf8a3cc6f36b,e7cbd6e8d04acb03] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:43:30.670 INFO 21062 --- [ocp-schedules-22,8d9074ecadecb8f4,d7ac75249eafee4f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:43:30.671 INFO 21062 --- [ocp-schedules-22,8d9074ecadecb8f4,d7ac75249eafee4f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:43:30.671 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:43:30.671 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:43:30.681 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:43:30.681 INFO 21062 --- [ocp-schedules-22,8d9074ecadecb8f4,d7ac75249eafee4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:43:30.681 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:43:30.683 WARN 21062 --- [ocp-schedules-22,8d9074ecadecb8f4,d7ac75249eafee4f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:43:30.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:43:30.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:43:30.684 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:43:30.684 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:43:30.684 INFO 21062 --- [ocp-schedules-22,8d9074ecadecb8f4,d7ac75249eafee4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:43:30.685 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:43:30.686 WARN 21062 --- [ocp-schedules-22,8d9074ecadecb8f4,d7ac75249eafee4f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:43:30.686 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:43:30.693 ERROR 21062 --- [ocp-schedules-22,8d9074ecadecb8f4,d7ac75249eafee4f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:43:30.769 INFO 21062 --- [ocp-schedules-7,ab2de3ebaa8cd04b,0bd410e606547224] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:43:30.769+08:00, clusters=[1] 2024-11-04 13:43:30.773 INFO 21062 --- [ocp-schedules-7,ab2de3ebaa8cd04b,0bd410e606547224] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:43:30.925 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:30.926 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:30.950 WARN 21062 --- [ocp-async-3,60064432e81a45ff,71d69d3f6ac50043] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:30.995 INFO 21062 --- [ocp-async-4,6ccd6e183af68ad6,e8e18fe3e9e0055b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:32.950 WARN 21062 --- [ocp-async-4,6bd857937c794a12,567078c37c0b75b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:32.983 INFO 21062 --- [prometheus-minute-schedule-1,1fc26106957f6866,a5d3b15b9e17ce38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:32.995 INFO 21062 --- [ocp-async-8,88ff9177b0359b9d,218aaf6134dc1e07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:33.361 INFO 21062 --- [ocp-schedules-12,ad6188c6eeb24061,4ca0e823063c4025] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:43:33.361 INFO 21062 --- [ocp-schedules-12,025c9be4f9d44ea3,10206aa200fb484f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:43:33.365 INFO 21062 --- [ocp-schedules-12,025c9be4f9d44ea3,10206aa200fb484f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:33.396 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:43:33.396 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:43:33.926 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:33.927 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:34.528 INFO 21062 --- [ocp-schedules-28,07f4935036745cf1,fbfadf83a9c9371d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:43:34.950 WARN 21062 --- [ocp-async-8,c300df55beb35734,b20adbb2982bff37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:34.996 INFO 21062 --- [ocp-async-1,20e78bacb59513fa,9c1e62fd3ee0fb68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:35.051 INFO 21062 --- [ocp-schedules-9,9402dd1512cf93f5,3d7a640dde22086e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:43:35.055 INFO 21062 --- [ocp-schedules-9,9402dd1512cf93f5,3d7a640dde22086e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:35.140 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:43:35.148 ERROR 21062 --- [ocp-schedules-9,9402dd1512cf93f5,3d7a640dde22086e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:43:35.149 INFO 21062 --- [ocp-schedules-9,9402dd1512cf93f5,3d7a640dde22086e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:43:35.983 INFO 21062 --- [prometheus-minute-schedule-1,ff0ea683adb730b7,b641d2d28b969e6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:36.927 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:36.929 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:36.951 WARN 21062 --- [ocp-async-1,3ba2614a80b3c8ab,55402f7382e68e58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:36.996 INFO 21062 --- [ocp-async-2,4d2b8b9ad514612d,1d02db9dba7b38d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:38.504 INFO 21062 --- [ocp-schedules-29,a2b602516c118126,1b3fb3bb558bc1a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:43:38.505 INFO 21062 --- [ocp-schedules-26,b841d9e96f72b398,ebf8be9ec1827da0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:43:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:43:38.589478+08:00, lastProcessTime=2024-11-04T13:43:28.589296+08:00 2024-11-04 13:43:38.615 INFO 21062 --- [ocp-async-3,c658d897f5d26080,c1e92947dce830a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:43:38.822 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:43:38.589478+08:00 2024-11-04 13:43:38.853 INFO 21062 --- [ocp-schedules-14,47b87a261dded4da,055fc55cb58fe413] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:43:38.951 WARN 21062 --- [ocp-async-7,e7b381180c55bac6,c9d48695e5185730] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:38.983 INFO 21062 --- [prometheus-minute-schedule-1,8173ad31bdf75ce0,7e967aba28a1835d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:38.996 INFO 21062 --- [ocp-async-5,b04461e5c87ce8e9,ec2e2e7463c0c82f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:39.683 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,447bbbe3aba1f7ae,e1460d2233bb8fd9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=447bbbe3aba1f7ae, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:43:39.683 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,c7c275d3d832440b,eec1348c26d929d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=c7c275d3d832440b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:43:39.683 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,a674e9141c5ec665,d5076185bf7ac83d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=a674e9141c5ec665, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:43:39.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,c7c275d3d832440b,eec1348c26d929d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=c7c275d3d832440b, duration=8 ms] 2024-11-04 13:43:39.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,447bbbe3aba1f7ae,e1460d2233bb8fd9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=447bbbe3aba1f7ae, duration=10 ms] 2024-11-04 13:43:39.684 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,067ae45305cef304,d4ceef4636a19216] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=067ae45305cef304, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:43:39.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,067ae45305cef304,d4ceef4636a19216] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=067ae45305cef304, duration=8 ms] 2024-11-04 13:43:39.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,a674e9141c5ec665,d5076185bf7ac83d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a674e9141c5ec665, duration=13 ms] 2024-11-04 13:43:39.769 INFO 21062 --- [ocp-schedules-10,fd3b1dad6490c0e9,d06aa10cdc7897cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:43:39.929 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:39.931 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:40.004 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:43:40.004 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:43:40.951 WARN 21062 --- [ocp-async-5,ec3c716fadd3e8d2,1f9e5c10c06f713d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:40.996 INFO 21062 --- [ocp-async-3,b8336f7ebf60ba30,761ea1b492853738] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2b19d09d9d6b5daa,faad7ac11c5beda5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:42.931 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:42.933 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:42.951 WARN 21062 --- [ocp-async-3,6fc92de5d460a762,164012c30e23f247] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:42.996 INFO 21062 --- [ocp-async-4,c52b0b0be9056b68,159e4ea89cadef24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:44.554 INFO 21062 --- [ocp-schedules-3,afe9045addfb02b0,1898cf2ed708a3af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:43:44.951 WARN 21062 --- [ocp-async-4,668d86717ba43bdb,63b0e9bd5b11d582] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:44.983 INFO 21062 --- [prometheus-minute-schedule-1,309701ba4909ea86,58f2959fad1206ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:44.997 INFO 21062 --- [ocp-async-8,1803989fbe30be3d,4bc1404f73418a6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:45.933 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:45.934 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:46.952 WARN 21062 --- [ocp-async-8,c32abcb4b1d135f0,3ca945aed4e7d7af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:46.997 INFO 21062 --- [ocp-async-1,0fde6fa54df45b6e,5abddad6600a6527] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:47.983 INFO 21062 --- [prometheus-minute-schedule-1,c0748528de428d97,5b3925c2078ea022] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:43:48.591404+08:00, lastProcessTime=2024-11-04T13:43:38.589478+08:00 2024-11-04 13:43:48.615 INFO 21062 --- [ocp-async-8,fd2cf289ce68aae3,81047848f29d6d04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:43:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:43:48.591404+08:00 2024-11-04 13:43:48.934 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:48.935 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:48.952 WARN 21062 --- [ocp-async-3,75eb5aa6374f806b,812847d94e29c118] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:48.997 INFO 21062 --- [ocp-async-4,69d94a8616a910fe,a0bcdffb79361e64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:49.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,149791c5dfbe31a4,ff5e60a37b696b60] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=149791c5dfbe31a4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:43:49.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ace38abdc38fe71a,a11eadf665f7915a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=ace38abdc38fe71a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:43:49.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b1ed6163cde9ce4a,9434c95d127eac50] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=b1ed6163cde9ce4a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:43:49.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,a488dfd2a5a68245,68fb528525b5b78c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=a488dfd2a5a68245, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:43:49.763 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,149791c5dfbe31a4,ff5e60a37b696b60] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=149791c5dfbe31a4, duration=11 ms] 2024-11-04 13:43:49.763 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,a488dfd2a5a68245,68fb528525b5b78c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=a488dfd2a5a68245, duration=8 ms] 2024-11-04 13:43:49.764 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ace38abdc38fe71a,a11eadf665f7915a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=ace38abdc38fe71a, duration=9 ms] 2024-11-04 13:43:49.765 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b1ed6163cde9ce4a,9434c95d127eac50] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=b1ed6163cde9ce4a, duration=13 ms] 2024-11-04 13:43:49.771 INFO 21062 --- [ocp-schedules-25,8078c6ceaf5d88d0,f75d84e6967725da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:43:50.468 INFO 21062 --- [ocp-schedules-15,f30cd864d0e5a453,fb424c698b261d9c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:43:50.474 INFO 21062 --- [ocp-schedules-15,f30cd864d0e5a453,fb424c698b261d9c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:43:50.474 INFO 21062 --- [ocp-schedules-15,f30cd864d0e5a453,fb424c698b261d9c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:43:50.952 WARN 21062 --- [ocp-async-4,e77cfd9829831b76,8f38034fa46200c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:50.983 INFO 21062 --- [prometheus-minute-schedule-1,c81555c149eab94a,f83a1022bbb2d895] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:50.997 INFO 21062 --- [ocp-async-8,f1196baa4ac14f33,af56724f52b6ec32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:51.935 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:51.936 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:52.952 WARN 21062 --- [ocp-async-8,b9840a05e46e0414,1958fbb1a97bf696] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:52.997 INFO 21062 --- [ocp-async-1,755112949b248e78,6103fa6641814f04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:53.983 INFO 21062 --- [prometheus-minute-schedule-1,7aa8250450e58ca2,305fe7f8a9aa99bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:54.603 INFO 21062 --- [ocp-schedules-1,0e22178e9ecf49cf,cfbf2fe00b7a67b6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:43:54.936 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:54.937 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:54.953 WARN 21062 --- [ocp-async-1,3cf0f01ca4323c9b,92766d8a3c840ad2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:54.997 INFO 21062 --- [ocp-async-2,7a035cea0dc07c4f,a5f763e1d05edbed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:56.953 WARN 21062 --- [ocp-async-2,e712b66410014006,96772af807e5c4de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:56.983 INFO 21062 --- [prometheus-minute-schedule-1,2800a0729e557c6a,950300b6860e1630] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:43:56.998 INFO 21062 --- [ocp-async-10,b4d3dd4e3083458c,38a8012dbddbd579] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:57.937 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:43:57.938 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:43:58.500 INFO 21062 --- [ocp-schedules-2,5942756538fbd4c9,6e3eb71313c9ead6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:43:58.500 INFO 21062 --- [ocp-schedules-20,44e69dc35c320425,438b83e9cc8834f3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:43:58.502 INFO 21062 --- [ocp-schedules-17,ad8251dc6502a817,3bedbfe7eee37290] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:43:58.503 INFO 21062 --- [ocp-schedules-2,5942756538fbd4c9,6e3eb71313c9ead6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:43:58.504 INFO 21062 --- [ocp-schedules-20,44e69dc35c320425,438b83e9cc8834f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:43:58.504 INFO 21062 --- [ocp-schedules-16,0624b789c3a3d729,c9a3a587004cdc14] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:43:58.509 INFO 21062 --- [ocp-schedules-9,7b3c382aa2148135,153c743a66fa3ca6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:43:58.509 INFO 21062 --- [ocp-schedules-9,7b3c382aa2148135,153c743a66fa3ca6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:43:58.510 INFO 21062 --- [ocp-schedules-11,60ce17eb699d19af,9e25c10ae956afcc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:43:58.510 INFO 21062 --- [ocp-schedules-11,60ce17eb699d19af,9e25c10ae956afcc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:43:58.510 INFO 21062 --- [ocp-schedules-22,301f03ff0e812ed9,74e965079cd85543] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:43:58.510 INFO 21062 --- [ocp-schedules-22,301f03ff0e812ed9,74e965079cd85543] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:43:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:43:58.589516+08:00, lastProcessTime=2024-11-04T13:43:48.591404+08:00 2024-11-04 13:43:58.615 INFO 21062 --- [ocp-async-2,01ff1dda6c5d3fab,61c1ec983db33dfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:43:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:43:58.589516+08:00 2024-11-04 13:43:58.946 INFO 21062 --- [ocp-schedules-8,ee1d6bae50bb3ab0,0ba1e33e9dbd91ae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:43:58.953 WARN 21062 --- [ocp-async-8,92b74e58cf44b2f8,28a40e6c2ffb6141] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:43:58.998 INFO 21062 --- [ocp-async-9,6d092dd30061b7e9,de0113cf3d7921f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:43:59.703 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8d9f7fe3dcd40841,2a6d3cb75ff6fea4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=8d9f7fe3dcd40841, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:43:59.704 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,d6ba56a059fe1b21,21ec362277f54b21] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=d6ba56a059fe1b21, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:43:59.704 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8d9f7fe3dcd40841,2a6d3cb75ff6fea4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=8d9f7fe3dcd40841, duration=6 ms] 2024-11-04 13:43:59.705 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a9f370fa71b34af2,bdc2416688d87e25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=a9f370fa71b34af2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:43:59.705 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a9f370fa71b34af2,bdc2416688d87e25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a9f370fa71b34af2, duration=5 ms] 2024-11-04 13:43:59.706 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,d6ba56a059fe1b21,21ec362277f54b21] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d6ba56a059fe1b21, duration=9 ms] 2024-11-04 13:43:59.707 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,db44852dfa4ee1a9,741f656f9745f81c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=db44852dfa4ee1a9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:43:59.708 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,db44852dfa4ee1a9,741f656f9745f81c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=db44852dfa4ee1a9, duration=8 ms] 2024-11-04 13:43:59.774 INFO 21062 --- [ocp-schedules-29,283a561fd8c6ebe8,a2baf3219ea16629] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:43:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1ab8546b5f7ae1a6,9c7bff379bc298ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699040, interval=60 2024-11-04 13:44:00.938 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:00.939 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:00.953 WARN 21062 --- [ocp-async-9,e0dd9ece41bf17d4,fa4e3422d88cffad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:00.998 INFO 21062 --- [ocp-async-6,f51dd079879e8c02,5ef225972bae9282] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:02.953 WARN 21062 --- [ocp-async-6,8d696e9b3d9edc33,4eeca9c1193cc5af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:02.983 INFO 21062 --- [prometheus-minute-schedule-1,24cdc897438cbc94,fd515bff8d6cb1f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:02.998 INFO 21062 --- [ocp-async-7,79aaaa0389ac54ff,e9b3e24a26d15e86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:03.396 INFO 21062 --- [ocp-schedules-12,025c9be4f9d44ea3,10206aa200fb484f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:44:03.397 INFO 21062 --- [ocp-schedules-12,c5c8dbe1ccd77182,ddb73d26df3db8ea] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:44:03.401 INFO 21062 --- [ocp-schedules-12,c5c8dbe1ccd77182,ddb73d26df3db8ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:03.429 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:44:03.429 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:44:03.939 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:03.941 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:04.629 INFO 21062 --- [ocp-schedules-6,2cc7c346d3b3c38c,b5bd469cd0581e17] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:44:04.954 WARN 21062 --- [ocp-async-7,687a42ec83cc737b,4e8e0297121e0d2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:04.998 INFO 21062 --- [ocp-async-5,a6ebe0462f51d75d,8bcfc285afcdcd52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f828b89cc3a8a560,736d05a97c1c810a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:06.941 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:06.942 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:06.954 WARN 21062 --- [ocp-async-5,22e4b0778429313b,64db45d75dd107cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:06.999 INFO 21062 --- [ocp-async-3,0fadcae0faf98d0e,2ef90390029d3b32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:08.504 INFO 21062 --- [ocp-schedules-20,44e69dc35c320425,438b83e9cc8834f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:44:08.504 INFO 21062 --- [ocp-schedules-2,5942756538fbd4c9,6e3eb71313c9ead6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:44:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:44:08.590131+08:00, lastProcessTime=2024-11-04T13:43:58.589516+08:00 2024-11-04 13:44:08.615 INFO 21062 --- [ocp-async-1,210b9b55d394f17b,56bf10f193842c75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:44:08.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:44:08.590131+08:00 2024-11-04 13:44:08.954 WARN 21062 --- [ocp-async-6,b616c494a1f95669,9e1607099f3d0543] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:08.983 INFO 21062 --- [prometheus-minute-schedule-1,7a33439a0d1ffd11,8acaa140f9b6c9c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:08.999 INFO 21062 --- [ocp-async-10,551586f55ae9e339,a2d12297e2026e40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:09.741 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,61ad7a4b0eaef5d8,ec4e539e01a8ee1c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=61ad7a4b0eaef5d8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:44:09.741 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,00a3070e38a2a3db,e434e37cf1e94c68] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=00a3070e38a2a3db, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:44:09.741 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c4c976e8d667d03e,641676cd5a92a1f2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=c4c976e8d667d03e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:44:09.741 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,531edafa08b07e3e,6d39882cf360a545] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=531edafa08b07e3e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:44:09.743 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,61ad7a4b0eaef5d8,ec4e539e01a8ee1c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=61ad7a4b0eaef5d8, duration=11 ms] 2024-11-04 13:44:09.743 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c4c976e8d667d03e,641676cd5a92a1f2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=c4c976e8d667d03e, duration=11 ms] 2024-11-04 13:44:09.744 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,531edafa08b07e3e,6d39882cf360a545] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=531edafa08b07e3e, duration=14 ms] 2024-11-04 13:44:09.746 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,00a3070e38a2a3db,e434e37cf1e94c68] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=00a3070e38a2a3db, duration=16 ms] 2024-11-04 13:44:09.777 INFO 21062 --- [ocp-schedules-24,20c8871ead41b934,d8653c44a37c7fad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:44:09.942 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:09.943 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:10.002 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:44:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:44:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:44:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:44:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:44:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:44:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:44:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:44:10.477 INFO 21062 --- [ocp-schedules-3,7d2357c3a3d3f7b0,0df9b16649f71ab8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:44:10.483 INFO 21062 --- [ocp-schedules-3,7d2357c3a3d3f7b0,0df9b16649f71ab8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:44:10.483 INFO 21062 --- [ocp-schedules-3,7d2357c3a3d3f7b0,0df9b16649f71ab8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:44:10.954 WARN 21062 --- [ocp-async-7,a5dd7ac74d9808b9,51c1cf46576d184f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:10.999 INFO 21062 --- [ocp-async-5,c99c0c2a9124e9c4,1abd54bd0479a170] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:11.329 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698980, nodeCount=107, affectRows=107 2024-11-04 13:44:11.329 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698980, nodeCount=138, affectRows=138 2024-11-04 13:44:11.329 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730698980, nodeCount=119, affectRows=119 2024-11-04 13:44:11.330 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698980, nodeCount=106, validCount=106 2024-11-04 13:44:11.330 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698980, nodeCount=118, validCount=118 2024-11-04 13:44:11.330 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730698980, nodeCount=137, validCount=137 2024-11-04 13:44:11.983 INFO 21062 --- [prometheus-minute-schedule-1,693f2b3e949835ca,5dcbbb24c70c5d3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:12.943 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:12.944 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:12.954 WARN 21062 --- [ocp-async-1,ece8bc5b6a38f1f8,895d44acd824c82b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:12.999 INFO 21062 --- [ocp-async-3,95036ba6355c7505,e256726e7f14bac4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:13.985 INFO 21062 --- [prometheus-cache-monitor1,e94d5d6dbfdbfdce,d98ddff1fe700656] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1416,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:44:13.985 INFO 21062 --- [prometheus-cache-monitor1,e94d5d6dbfdbfdce,d98ddff1fe700656] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:44:14.673 INFO 21062 --- [ocp-schedules-23,0577470b4965f850,f4a17a1ff46cda28] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:44:14.955 WARN 21062 --- [ocp-async-2,cd2d6822b4b0092a,558e6c7ccbb69d37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a2b268ea68229206,291ec62b4236230d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:14.999 INFO 21062 --- [ocp-async-4,fd1580c0e2414304,9602e0268ca841ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:15.945 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:15.945 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:16.955 WARN 21062 --- [ocp-async-10,6dfd7ae199874786,8a126f2b8ff00e1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:16.999 INFO 21062 --- [ocp-async-8,ed8fe214b4d32b36,07a8e02a6b3f9dfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7d692e3f136708f9,8c15f3fb9129ca68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:44:18.591537+08:00, lastProcessTime=2024-11-04T13:44:08.590131+08:00 2024-11-04 13:44:18.616 INFO 21062 --- [ocp-async-4,3a065701eab34047,c8442cd4eafdcad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:44:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:44:18.591537+08:00 2024-11-04 13:44:18.945 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:18.946 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:18.955 WARN 21062 --- [ocp-async-1,6e443c0df8a8ab4a,c9d28842a53d47fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:19.000 INFO 21062 --- [ocp-async-3,99f9f5884c5f6df6,4ed806454e6db201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:19.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,df48723c9085d929,330b489e6fcbb3fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=df48723c9085d929, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:44:19.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3536dd44e77165e0,2e49bfdfaeffe499] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=3536dd44e77165e0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:44:19.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,49fec29e2314dd21,8ef60ee3f2b5988f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=49fec29e2314dd21, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:44:19.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f408aef0682ff0fa,85a5637ce5e3e6f2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=f408aef0682ff0fa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:44:19.758 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3536dd44e77165e0,2e49bfdfaeffe499] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=3536dd44e77165e0, duration=7 ms] 2024-11-04 13:44:19.758 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,49fec29e2314dd21,8ef60ee3f2b5988f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=49fec29e2314dd21, duration=7 ms] 2024-11-04 13:44:19.759 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,df48723c9085d929,330b489e6fcbb3fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=df48723c9085d929, duration=10 ms] 2024-11-04 13:44:19.761 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f408aef0682ff0fa,85a5637ce5e3e6f2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=f408aef0682ff0fa, duration=13 ms] 2024-11-04 13:44:19.780 INFO 21062 --- [ocp-schedules-25,efcac511799b24a7,d3e06bb80df3f882] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:44:20.955 WARN 21062 --- [ocp-async-2,497c1dcfbf1a9171,a1310436fb4ac4bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e9a3c79288a063f1,237eadd8c393ac35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:21.000 INFO 21062 --- [ocp-async-4,eb04445b3c4c807a,c8d7bec3103c5a28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:21.946 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:21.947 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:22.955 WARN 21062 --- [ocp-async-10,02a726aeb9315b74,98085e9e8001a09b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:23.000 INFO 21062 --- [ocp-async-8,6bf9bad3d97c7918,fe208e526269d2fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4dff2f7ccbca14f5,31f4fec9756d54ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:24.700 INFO 21062 --- [ocp-schedules-15,8d693034acff3043,1627ea038a495253] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:44:24.947 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:24.948 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:24.955 WARN 21062 --- [ocp-async-5,e2ddd0a747337d95,4890de0c96ba4b63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:25.000 INFO 21062 --- [ocp-async-9,55b2e3cc4168b66f,961d6c2a0d99e24b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:26.956 WARN 21062 --- [ocp-async-3,0534e34e215bf35f,1872beb2eaf550bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f599626cfc6552e4,30bd4d4ae3ccae80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8306fbeb6907719e,89f36429164098e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:44:27.000 INFO 21062 --- [ocp-async-6,45cf54333be66d9d,452b80408ae786be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:27.949 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:27.950 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:28.498 INFO 21062 --- [ocp-schedules-24,07c635b288a383c0,0969f32300c431e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:44:28.500 INFO 21062 --- [ocp-schedules-26,a2569cb750918984,da171dd4689413c6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:44:28.500 INFO 21062 --- [ocp-schedules-28,9e299136cab0fa5b,24e98d62c574f0e3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:44:28.500 INFO 21062 --- [ocp-schedules-19,33db349e7d4e7ad6,19412848c6534c59] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:44:28.502 INFO 21062 --- [ocp-schedules-24,07c635b288a383c0,0969f32300c431e5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:44:28.502 INFO 21062 --- [ocp-schedules-15,0c2fda671fad3e1d,96921071a662304e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:44:28.504 INFO 21062 --- [ocp-schedules-28,9e299136cab0fa5b,24e98d62c574f0e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:28.504 INFO 21062 --- [ocp-schedules-19,33db349e7d4e7ad6,19412848c6534c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:44:28.505 INFO 21062 --- [ocp-schedules-29,204f9abb0b4bb886,b6155bdd54f045b6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:44:28.505 INFO 21062 --- [ocp-schedules-26,a2569cb750918984,da171dd4689413c6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:44:28.511 INFO 21062 --- [ocp-schedules-2,d5c2fe8d378d51e2,360ae5511eacf8c5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:44:28.511 INFO 21062 --- [ocp-schedules-3,c2a3a0677c24d710,b1d9e4fc9f63d3d8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:44:28.511 INFO 21062 --- [ocp-schedules-3,c2a3a0677c24d710,b1d9e4fc9f63d3d8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:44:28.511 INFO 21062 --- [ocp-schedules-2,d5c2fe8d378d51e2,360ae5511eacf8c5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:44:28.511 INFO 21062 --- [ocp-schedules-11,78951e26b78ceb2e,7608a7b8d26995a3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:44:28.511 INFO 21062 --- [ocp-schedules-11,78951e26b78ceb2e,7608a7b8d26995a3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:44:28.564 INFO 21062 --- [ocp-schedules-10,41059b32b63320b0,acbb8e6d146ebae3] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:44:28.564+08:00, tenantSize=0 2024-11-04 13:44:28.569 INFO 21062 --- [ocp-schedules-7,25a98b8799baccf8,ff9989411f9311a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1416,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":363,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:44:28.569 INFO 21062 --- [ocp-schedules-7,25a98b8799baccf8,ff9989411f9311a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:44:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:44:28.590067+08:00, lastProcessTime=2024-11-04T13:44:18.591537+08:00 2024-11-04 13:44:28.616 INFO 21062 --- [ocp-async-2,0173a85b7767ef1a,2e79e8c9d7585660] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:44:28.627 INFO 21062 --- [ocp-schedules-27,08e7f2caac3ace10,4f8f611d271cde44] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 13:44:28.628 INFO 21062 --- [ocp-schedules-27,08e7f2caac3ace10,4f8f611d271cde44] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 13:44:28.734 INFO 21062 --- [ocp-schedules-22,c13073508ad0054d,23afc036c5b12837] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:44:28.782 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:44:28.590067+08:00 2024-11-04 13:44:28.916 INFO 21062 --- [ocp-schedules-15,2984d4bcaee12b35,de6b3884d772f027] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:44:28.916 INFO 21062 --- [ocp-schedules-15,2984d4bcaee12b35,de6b3884d772f027] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:44:28.949 INFO 21062 --- [ocp-schedules-29,3fcecdbc278345d6,a231494fb8edcf27] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:44:28.956 WARN 21062 --- [ocp-async-8,43164bd7aa243901,a51b243cbef95265] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:28.980 INFO 21062 --- [ocp-schedules-11,af2283c666aa658b,505b0e73672a1fd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:44:28.986 INFO 21062 --- [ocp-schedules-11,af2283c666aa658b,505b0e73672a1fd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:44:28.986 INFO 21062 --- [ocp-schedules-11,af2283c666aa658b,505b0e73672a1fd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:44:29.001 INFO 21062 --- [ocp-async-9,034e7e3b7c4da482,2eb489c2f150fe8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:29.231 INFO 21062 --- [ocp-schedules-20,a4435e5f52dd8f02,61dc35015e67c097] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:44:29.231 INFO 21062 --- [ocp-schedules-20,a4435e5f52dd8f02,61dc35015e67c097] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:44:29.232 INFO 21062 --- [ocp-schedules-20,a4435e5f52dd8f02,61dc35015e67c097] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:44:29.232 INFO 21062 --- [ocp-schedules-20,a4435e5f52dd8f02,61dc35015e67c097] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:44:29.233 INFO 21062 --- [ocp-schedules-20,a4435e5f52dd8f02,61dc35015e67c097] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:44:29.233 INFO 21062 --- [ocp-schedules-20,a4435e5f52dd8f02,61dc35015e67c097] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709925194352054272 2024-11-04 13:44:29.759 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,27a3c28a1d048b89,6afb293d9ba01b2a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=27a3c28a1d048b89, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:44:29.759 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ae150a1361a1bae1,a2e5706086d8412a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=ae150a1361a1bae1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:44:29.759 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,52083d3cc7a014ba,ca8a22634cd33c8f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=52083d3cc7a014ba, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:44:29.759 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,bbf3126c292bf828,405f390467c64a01] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=bbf3126c292bf828, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:44:29.760 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ae150a1361a1bae1,a2e5706086d8412a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=ae150a1361a1bae1, duration=10 ms] 2024-11-04 13:44:29.760 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,bbf3126c292bf828,405f390467c64a01] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=bbf3126c292bf828, duration=10 ms] 2024-11-04 13:44:29.760 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,27a3c28a1d048b89,6afb293d9ba01b2a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=27a3c28a1d048b89, duration=12 ms] 2024-11-04 13:44:29.763 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,52083d3cc7a014ba,ca8a22634cd33c8f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=52083d3cc7a014ba, duration=15 ms] 2024-11-04 13:44:29.783 INFO 21062 --- [ocp-schedules-9,acbae2c8a106186f,56d5d413c7f621e7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:44:29.983 INFO 21062 --- [prometheus-minute-schedule-1,d9178ae1dee1d1e2,bdf3501c4831e4b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:44:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:44:30.486 INFO 21062 --- [ocp-schedules-7,8610be5dfc574c71,c1524b9ede4fbfea] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:44:30.492 INFO 21062 --- [ocp-schedules-7,8610be5dfc574c71,c1524b9ede4fbfea] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:44:30.492 INFO 21062 --- [ocp-schedules-7,8610be5dfc574c71,c1524b9ede4fbfea] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:44:30.492 INFO 21062 --- [ocp-schedules-4,1a382accdda44883,596f5813e2ed547d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:44:30.492 INFO 21062 --- [ocp-schedules-4,1a382accdda44883,596f5813e2ed547d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:44:30.518 INFO 21062 --- [ocp-schedules-4,1a382accdda44883,596f5813e2ed547d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:44:30.518 INFO 21062 --- [ocp-schedules-4,1a382accdda44883,596f5813e2ed547d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:44:30.519 INFO 21062 --- [ocp-schedules-20,c034cd55f68235ba,75c9dcd22c733342] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:44:30.654 INFO 21062 --- [ocp-schedules-27,5419f27757e19566,8b55fe76aa4e45c7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:44:30.654+08:00, clusters=[1] 2024-11-04 13:44:30.656 INFO 21062 --- [ocp-schedules-27,5419f27757e19566,8b55fe76aa4e45c7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:44:30.702 INFO 21062 --- [ocp-schedules-21,0bc239a7e1a91a66,21ebdf2f11176e38] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:44:30.702 INFO 21062 --- [ocp-schedules-21,0bc239a7e1a91a66,21ebdf2f11176e38] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:44:30.702 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:44:30.702 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:44:30.712 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:44:30.712 INFO 21062 --- [ocp-schedules-21,0bc239a7e1a91a66,21ebdf2f11176e38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:44:30.712 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:44:30.713 INFO 21062 --- [ocp-schedules-5,cb60380f17301d42,a2ee5f676259f27c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:44:30.714 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:44:30.714 WARN 21062 --- [ocp-schedules-21,0bc239a7e1a91a66,21ebdf2f11176e38] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:44:30.714 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:44:30.715 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:44:30.715 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:44:30.715 INFO 21062 --- [ocp-schedules-21,0bc239a7e1a91a66,21ebdf2f11176e38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:44:30.716 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:44:30.716 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:44:30.716 WARN 21062 --- [ocp-schedules-21,0bc239a7e1a91a66,21ebdf2f11176e38] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:44:30.716 INFO 21062 --- [ocp-schedules-5,cb60380f17301d42,a2ee5f676259f27c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:30.725 ERROR 21062 --- [ocp-schedules-21,0bc239a7e1a91a66,21ebdf2f11176e38] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:44:30.773 ERROR 21062 --- [ocp-schedules-5,cb60380f17301d42,a2ee5f676259f27c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:44:30.774 INFO 21062 --- [ocp-schedules-5,cb60380f17301d42,a2ee5f676259f27c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:44:30.797 INFO 21062 --- [ocp-schedules-10,0c165f80a2d8f831,ed8d2a9c895b296b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:44:30.797+08:00, clusters=[1] 2024-11-04 13:44:30.801 INFO 21062 --- [ocp-schedules-10,0c165f80a2d8f831,ed8d2a9c895b296b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:44:30.950 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:30.951 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:30.956 WARN 21062 --- [ocp-async-1,9a18fabbcae8da37,dc80abf4ca1ae20b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:31.001 INFO 21062 --- [ocp-async-2,06ef2e80a351a1ad,16bd439c7df19a95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:32.956 WARN 21062 --- [ocp-async-2,22020151c6ce9185,0a172058f6335aef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:32.983 INFO 21062 --- [prometheus-minute-schedule-1,e00c8349b7030400,8a5f6b8bc89a8e59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:33.001 INFO 21062 --- [ocp-async-10,1e562cbddea3beb9,f9ef0b295fafdcc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:33.430 INFO 21062 --- [ocp-schedules-12,c5c8dbe1ccd77182,ddb73d26df3db8ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:44:33.430 INFO 21062 --- [ocp-schedules-12,a9742ca5a1c127f9,e902c3abc4860647] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:44:33.434 INFO 21062 --- [ocp-schedules-12,a9742ca5a1c127f9,e902c3abc4860647] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:33.463 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:44:33.463 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:44:33.951 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:33.952 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:34.742 INFO 21062 --- [ocp-schedules-7,828e9061cf6cd82e,a9d4ba6b44424e40] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:44:34.956 WARN 21062 --- [ocp-async-10,7724a97d2fa2e780,fc3ac86c4bb3bf13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:35.001 INFO 21062 --- [ocp-async-5,0124a891fc235f12,89835cec7c0b9311] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:35.149 INFO 21062 --- [ocp-schedules-30,89ef19b4b7c6e62a,ffd589ef94975adc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:44:35.152 INFO 21062 --- [ocp-schedules-30,89ef19b4b7c6e62a,ffd589ef94975adc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:35.236 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:44:35.238 ERROR 21062 --- [ocp-schedules-30,89ef19b4b7c6e62a,ffd589ef94975adc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:44:35.238 INFO 21062 --- [ocp-schedules-30,89ef19b4b7c6e62a,ffd589ef94975adc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:44:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0c99c38ea31c987c,666451db886047c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:36.953 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:36.954 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:36.957 WARN 21062 --- [ocp-async-5,05ac2665fabadb7f,a97608fc42900681] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:37.002 INFO 21062 --- [ocp-async-3,5fc4e946d7de3e84,f7bddd78a94dfe64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:38.505 INFO 21062 --- [ocp-schedules-19,33db349e7d4e7ad6,19412848c6534c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:44:38.505 INFO 21062 --- [ocp-schedules-28,9e299136cab0fa5b,24e98d62c574f0e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:44:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:44:38.589943+08:00, lastProcessTime=2024-11-04T13:44:28.590067+08:00 2024-11-04 13:44:38.616 INFO 21062 --- [ocp-async-1,519cd7a7151817c8,4d06292b7c00fdcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:44:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:44:38.589943+08:00 2024-11-04 13:44:38.853 INFO 21062 --- [ocp-schedules-3,bd98cf7d9b350c9b,5c0e7a8a4b6b7b8b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:44:38.857 INFO 21062 --- [ocp-schedules-3,bd98cf7d9b350c9b,5c0e7a8a4b6b7b8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:38.857 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:44:38.957 WARN 21062 --- [ocp-async-6,d6ebfc8299c2c80b,aa541ee57e8b7ba4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:38.983 INFO 21062 --- [prometheus-minute-schedule-1,559dcb98aeed1a8e,6726a987f068f66d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:39.002 INFO 21062 --- [ocp-async-7,91c02e2b8d926e04,2aac599b257493d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:39.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,5b1668125a3d475a,77032dab04c56ac1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=5b1668125a3d475a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:44:39.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,5e4380f78f5afe16,64efb5ebb9e92d84] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=5e4380f78f5afe16, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:44:39.758 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,4c7864b0486ae56f,a8b637698eea8fa4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=4c7864b0486ae56f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:44:39.758 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,5b1668125a3d475a,77032dab04c56ac1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=5b1668125a3d475a, duration=9 ms] 2024-11-04 13:44:39.758 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,4c7864b0486ae56f,a8b637698eea8fa4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=4c7864b0486ae56f, duration=5 ms] 2024-11-04 13:44:39.760 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,c0e65460e6374133,8934b77ba557b460] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=c0e65460e6374133, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:44:39.760 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,5e4380f78f5afe16,64efb5ebb9e92d84] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=5e4380f78f5afe16, duration=11 ms] 2024-11-04 13:44:39.761 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,c0e65460e6374133,8934b77ba557b460] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=c0e65460e6374133, duration=8 ms] 2024-11-04 13:44:39.786 INFO 21062 --- [ocp-schedules-24,eaeddc991297a0ae,1dc71ccd270a9481] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:44:39.954 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:39.955 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:44:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:44:40.928 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:44:40.928 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:44:40.930 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.930 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:44:40.932 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.932 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:40.933 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.933 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:44:40.948 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.948 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:40.949 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.949 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:44:40.957 WARN 21062 --- [ocp-async-7,9f09f92dbfb99d52,b178cdfb6d78578c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:40.962 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.962 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:40.963 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.963 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:44:40.977 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.978 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:40.979 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.979 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:44:40.993 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.994 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:40.994 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:40.995 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:44:41.000 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.000 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.001 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.001 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:44:41.002 INFO 21062 --- [ocp-async-1,925512d714fa5198,0d1ff57d20ead16b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:41.006 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.007 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.007 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.007 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:44:41.012 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.012 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.013 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.013 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:44:41.019 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.019 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.020 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.020 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:44:41.026 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.026 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.027 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.027 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:44:41.032 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.032 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.033 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.033 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:44:41.041 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.041 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.042 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.042 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:44:41.055 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.055 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.056 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.056 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:44:41.061 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.061 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.062 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.062 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:44:41.068 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.068 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.069 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.069 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:44:41.078 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.078 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.079 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.079 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:44:41.083 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.084 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.084 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.084 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:44:41.088 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.089 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.089 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.090 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:44:41.094 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.094 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.094 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.095 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:44:41.099 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.100 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.100 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.100 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:44:41.104 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.105 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.105 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.106 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:44:41.110 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.110 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.111 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.111 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:44:41.115 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.116 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.116 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.116 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:44:41.121 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.121 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.122 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.122 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:44:41.127 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.127 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.128 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.128 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:44:41.133 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.133 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.134 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.134 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:44:41.138 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.139 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.139 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.140 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:44:41.144 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.145 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.145 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.146 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:44:41.150 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.150 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.151 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.151 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:44:41.155 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.156 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.156 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.156 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:44:41.160 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.161 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.161 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.161 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:44:41.166 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.166 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.167 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.167 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:44:41.172 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.172 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.172 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.173 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:44:41.177 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.177 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.177 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.178 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:44:41.186 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.186 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.187 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.187 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:44:41.192 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.193 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.193 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.194 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:44:41.199 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.199 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.200 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.200 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:44:41.205 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.206 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.206 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.207 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:44:41.211 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.211 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.211 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.212 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:44:41.216 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.216 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.217 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.217 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:44:41.221 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.221 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.222 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.222 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:44:41.226 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.226 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.227 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.227 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:44:41.235 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.236 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.236 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.237 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:44:41.259 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.260 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.261 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.261 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:44:41.267 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.267 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.268 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.268 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:44:41.274 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.274 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.275 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.275 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:44:41.281 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.281 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.282 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.282 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:44:41.286 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.286 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.287 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.287 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:44:41.291 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.291 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.292 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.292 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:44:41.297 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:44:41.297 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:44:41.298 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.298 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:44:41.299 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.300 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.300 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.301 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:44:41.306 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.306 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:44:41.307 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:44:41.308 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:44:41.312 INFO 21062 --- [ocp-schedules-19,5cdac1cd2c48a543,be71320fa5074340] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:44:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:44:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:44:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 13:44:41.983 INFO 21062 --- [prometheus-minute-schedule-1,fe48cb14dec28ef8,bb6fe857fbecc3cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:42.955 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:42.956 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:42.957 WARN 21062 --- [ocp-async-1,bc5483d87c5d4bc6,b288d23c54834c82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:43.002 INFO 21062 --- [ocp-async-2,ab33ed354a7c7d4d,a817f5da3355d9c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:44.766 INFO 21062 --- [ocp-schedules-26,ff35c56f1b4c9c83,15ff87c23291aed3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:44:44.957 WARN 21062 --- [ocp-async-2,dd56611556a15f0c,5c9775cccf2eac68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:44.983 INFO 21062 --- [prometheus-minute-schedule-1,552393e49e28e419,90e8f5e1f2a608de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:45.002 INFO 21062 --- [ocp-async-10,f1b8f80821bd8204,752969a62380ed2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:45.956 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:45.957 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:46.958 WARN 21062 --- [ocp-async-10,80f40c72c298d222,5e9744145c4f1e4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:47.003 INFO 21062 --- [ocp-async-5,f4aad0331e281102,04e9a407b1ffffac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:47.983 INFO 21062 --- [prometheus-minute-schedule-1,7db1b14751271b7a,dd556df4705d520c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:44:48.591307+08:00, lastProcessTime=2024-11-04T13:44:38.589943+08:00 2024-11-04 13:44:48.616 INFO 21062 --- [ocp-async-10,e248a3912f1ed70c,0a1d30007bc5e0c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:44:48.801 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:44:48.591307+08:00 2024-11-04 13:44:48.857 INFO 21062 --- [ocp-schedules-3,bd98cf7d9b350c9b,5c0e7a8a4b6b7b8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:44:48.957 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:48.958 WARN 21062 --- [ocp-async-1,623a7dcbeca0d4d6,03d616cbc4b63935] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:48.958 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:49.003 INFO 21062 --- [ocp-async-2,1eb0df32f9462a20,c9a52fbd574a5023] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:49.789 INFO 21062 --- [ocp-schedules-7,2fe2c4408bf9c36f,44f6cb82d2f8d18a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:44:49.820 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,8a58c74eee30604b,13d6a2c374715130] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=8a58c74eee30604b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:44:49.820 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,60720e867e01092e,930e69c3e29f81ec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=60720e867e01092e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:44:49.820 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,55ccfb9b3bd29785,cf72a820d1c92b95] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=55ccfb9b3bd29785, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:44:49.821 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,3086cb7d4a9cbacb,9fa4044e783c489a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=3086cb7d4a9cbacb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:44:49.821 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,60720e867e01092e,930e69c3e29f81ec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=60720e867e01092e, duration=9 ms] 2024-11-04 13:44:49.821 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,8a58c74eee30604b,13d6a2c374715130] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=8a58c74eee30604b, duration=6 ms] 2024-11-04 13:44:49.821 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,3086cb7d4a9cbacb,9fa4044e783c489a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=3086cb7d4a9cbacb, duration=5 ms] 2024-11-04 13:44:49.823 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,55ccfb9b3bd29785,cf72a820d1c92b95] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=55ccfb9b3bd29785, duration=11 ms] 2024-11-04 13:44:50.494 INFO 21062 --- [ocp-schedules-30,84427979daaba3e6,d679d871f5e97c32] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:44:50.500 INFO 21062 --- [ocp-schedules-30,84427979daaba3e6,d679d871f5e97c32] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:44:50.500 INFO 21062 --- [ocp-schedules-30,84427979daaba3e6,d679d871f5e97c32] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:44:50.958 WARN 21062 --- [ocp-async-2,57481b947b90d5cd,7c09637c882c6eb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:50.983 INFO 21062 --- [prometheus-minute-schedule-1,739cc06ded8d308c,47b3ba98005ed3dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:51.003 INFO 21062 --- [ocp-async-10,27d476c83b2dc87d,aab8c0978c85d0cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:51.958 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:51.959 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:52.958 WARN 21062 --- [ocp-async-10,da4063aabe0ee978,91d88d83ecdec8d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:53.003 INFO 21062 --- [ocp-async-5,9b4466185c6fa412,7dbc9ebc6b282496] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:53.983 INFO 21062 --- [prometheus-minute-schedule-1,2dcbb2b313275615,88d1cb66dd1831f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:54.806 INFO 21062 --- [ocp-schedules-11,c103a9d895549aa9,80b5114f32930762] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:44:54.958 WARN 21062 --- [ocp-async-5,dcde3430dfe9a16e,f1c2854249ad4ada] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:54.959 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:54.960 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:55.003 INFO 21062 --- [ocp-async-3,8079d7cdc56f3fa6,9add285ae57f0b93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:56.958 WARN 21062 --- [ocp-async-3,be9ae8596a6ca19c,865ec079fbc4b5c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:56.983 INFO 21062 --- [prometheus-minute-schedule-1,35b6d46d7fbafb65,7f5e8569368cf421] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:44:57.003 INFO 21062 --- [ocp-async-4,7346e5137b05892a,95a67581333fb29a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:57.960 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:44:57.961 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:44:58.499 INFO 21062 --- [ocp-schedules-22,bafae50e484dfc30,714238539efafaea] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:44:58.500 INFO 21062 --- [ocp-schedules-25,91beac563ab730ff,c576291cf92f4e96] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:44:58.502 INFO 21062 --- [ocp-schedules-27,14ed1d6b73132298,54532a1145bf9077] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:44:58.504 INFO 21062 --- [ocp-schedules-25,91beac563ab730ff,c576291cf92f4e96] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:58.504 INFO 21062 --- [ocp-schedules-22,bafae50e484dfc30,714238539efafaea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:44:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:44:58.505 INFO 21062 --- [ocp-schedules-13,74cd1ea62c8bfdb4,f2d85185d5e00525] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:44:58.510 INFO 21062 --- [ocp-schedules-28,bb8a4297e6b13c79,3ed4e22ec18d0559] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:44:58.511 INFO 21062 --- [ocp-schedules-1,5e932a14655201f6,80d21653bdf027dc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:44:58.511 INFO 21062 --- [ocp-schedules-28,bb8a4297e6b13c79,3ed4e22ec18d0559] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:44:58.511 INFO 21062 --- [ocp-schedules-1,5e932a14655201f6,80d21653bdf027dc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:44:58.511 INFO 21062 --- [ocp-schedules-8,20cef933c52e5c0e,5883a1ed4973f301] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:44:58.511 INFO 21062 --- [ocp-schedules-8,20cef933c52e5c0e,5883a1ed4973f301] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:44:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:44:58.590568+08:00, lastProcessTime=2024-11-04T13:44:48.591307+08:00 2024-11-04 13:44:58.616 INFO 21062 --- [ocp-async-3,ede326398c7e6246,b2ee99324a0223bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:44:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:44:58.590568+08:00 2024-11-04 13:44:58.951 INFO 21062 --- [ocp-schedules-11,a87306bff8789bd7,05786650f9b9ea1a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:44:58.959 WARN 21062 --- [ocp-async-10,a918b55e211349d9,da6e1c3dc5679bb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:44:59.004 INFO 21062 --- [ocp-async-8,20c1bbba812e1ed5,7f87d8cb4fdc3714] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:44:59.774 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d7f772bd0dd81cee,7767b0aef1e92c67] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=d7f772bd0dd81cee, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:44:59.774 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,5430570617116404,5784ca50df7b779d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=5430570617116404, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:44:59.774 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,335df665984d5ab3,645c1275599cde10] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=335df665984d5ab3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:44:59.774 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,071d06465b26c43e,b21392b0c756781a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=071d06465b26c43e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:44:59.775 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,5430570617116404,5784ca50df7b779d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=5430570617116404, duration=9 ms] 2024-11-04 13:44:59.775 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,071d06465b26c43e,b21392b0c756781a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=071d06465b26c43e, duration=10 ms] 2024-11-04 13:44:59.775 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d7f772bd0dd81cee,7767b0aef1e92c67] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d7f772bd0dd81cee, duration=13 ms] 2024-11-04 13:44:59.777 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,335df665984d5ab3,645c1275599cde10] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=335df665984d5ab3, duration=15 ms] 2024-11-04 13:44:59.792 INFO 21062 --- [ocp-schedules-14,c01638b35de5e168,4d5673b9fdd51265] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:44:59.983 INFO 21062 --- [prometheus-minute-schedule-1,9a500c5a793c8ee3,b29cfcc2b4de4465] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699100, interval=60 2024-11-04 13:45:00.654 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,5af9fd4f33845e2a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:40:01.0, nextRun=2024-11-04 13:45:00.0, currentDate=Mon Nov 04 13:45:00 CST 2024 2024-11-04 13:45:00.654 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,5af9fd4f33845e2a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:40:01.0, nextRunTime=2024-11-04 13:45:00.0} 2024-11-04 13:45:00.681 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,5af9fd4f33845e2a] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:45:00.689 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,5af9fd4f33845e2a] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:45:00.689 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,5af9fd4f33845e2a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:45:00.959 WARN 21062 --- [ocp-async-8,699902e3273ecbab,295c140c33f971c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:00.961 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:00.962 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:01.004 INFO 21062 --- [ocp-async-9,38b0432fefbc5e78,0d55fb33b693bb34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:02.959 WARN 21062 --- [ocp-async-9,f0b52013566a7b3d,85db4091cc32c215] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ab1a0cacff51c964,88e4ed7de9d3e7e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:03.004 INFO 21062 --- [ocp-async-6,ef81412f04be6586,e1da69e3add65c0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:03.464 INFO 21062 --- [ocp-schedules-12,a9742ca5a1c127f9,e902c3abc4860647] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:45:03.464 INFO 21062 --- [ocp-schedules-12,2278df3a83343d83,320675e6207a257f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:45:03.469 INFO 21062 --- [ocp-schedules-12,2278df3a83343d83,320675e6207a257f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:03.500 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:45:03.500 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:45:03.962 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:03.963 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:04.831 INFO 21062 --- [ocp-schedules-30,5d28ba787c1e5075,b880881d85b60307] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:45:04.959 WARN 21062 --- [ocp-async-6,bb20458f169180d2,ad7e9ac3f04991a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:05.004 INFO 21062 --- [ocp-async-7,bcb100847681eb93,6f978d32c2656d88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:05.983 INFO 21062 --- [prometheus-minute-schedule-1,6fe587144fa34bdc,1ef4908ba2e9703e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:06.959 WARN 21062 --- [ocp-async-7,b9057267f23ee9b6,1ee5b85eab7fe70b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:06.963 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:06.964 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:07.005 INFO 21062 --- [ocp-async-1,3bb925e4855d4b9b,0c0c5e06e871b7f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:08.505 INFO 21062 --- [ocp-schedules-25,91beac563ab730ff,c576291cf92f4e96] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:45:08.505 INFO 21062 --- [ocp-schedules-22,bafae50e484dfc30,714238539efafaea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:45:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:45:08.589945+08:00, lastProcessTime=2024-11-04T13:44:58.590568+08:00 2024-11-04 13:45:08.616 INFO 21062 --- [ocp-async-5,16d765029acf4055,7f9c76e7a4b12909] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:45:08.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:45:08.589945+08:00 2024-11-04 13:45:08.960 WARN 21062 --- [ocp-async-9,a6207915706cd7ba,a94c20d854c99485] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:08.983 INFO 21062 --- [prometheus-minute-schedule-1,54273d0d09db1b84,8f90536626d02c81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:09.005 INFO 21062 --- [ocp-async-4,b613326f1ddbb432,2ed33951152211d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:09.779 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9242c9f4fa178a7a,2ab97e5c53a087d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=9242c9f4fa178a7a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:45:09.779 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,52b279a22b6cc8f0,1a8a681b0767f02d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=52b279a22b6cc8f0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:45:09.779 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,c3bd27b15f19aec3,44bb8a193eb99d6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=c3bd27b15f19aec3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:45:09.779 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6cf42de18046b610,417d8a34b6488ad7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=6cf42de18046b610, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:45:09.780 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,52b279a22b6cc8f0,1a8a681b0767f02d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=52b279a22b6cc8f0, duration=6 ms] 2024-11-04 13:45:09.780 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6cf42de18046b610,417d8a34b6488ad7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=6cf42de18046b610, duration=5 ms] 2024-11-04 13:45:09.780 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9242c9f4fa178a7a,2ab97e5c53a087d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=9242c9f4fa178a7a, duration=9 ms] 2024-11-04 13:45:09.782 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,c3bd27b15f19aec3,44bb8a193eb99d6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=c3bd27b15f19aec3, duration=11 ms] 2024-11-04 13:45:09.795 INFO 21062 --- [ocp-schedules-8,bb0f233d3e9d9dc2,6175f47eaec7e01f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:45:09.964 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:09.965 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:10.002 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:45:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:45:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:45:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:45:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:45:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:45:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:45:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:45:10.502 INFO 21062 --- [ocp-schedules-7,04f24d04ab3a2bdc,7d17072c72d95f45] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:45:10.508 INFO 21062 --- [ocp-schedules-7,04f24d04ab3a2bdc,7d17072c72d95f45] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:45:10.508 INFO 21062 --- [ocp-schedules-7,04f24d04ab3a2bdc,7d17072c72d95f45] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:45:10.960 WARN 21062 --- [ocp-async-6,06337c716953e774,415749005cdbcd0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:11.007 INFO 21062 --- [ocp-async-7,7e94c0d76c604b6f,7a3a0c7462996d45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:11.331 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=3, affectRows=3 2024-11-04 13:45:11.331 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=3, validCount=3 2024-11-04 13:45:11.332 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=5, affectRows=5 2024-11-04 13:45:11.332 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=5, validCount=5 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=53, affectRows=53 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=68, affectRows=68 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=47, affectRows=47 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=30, affectRows=30 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=32, affectRows=32 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=62, affectRows=62 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=17, affectRows=17 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=30, validCount=30 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=32, validCount=32 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699040, nodeCount=47, affectRows=47 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=52, validCount=52 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=16, validCount=16 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=47, validCount=47 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=67, validCount=67 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=62, validCount=62 2024-11-04 13:45:11.333 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699040, nodeCount=47, validCount=47 2024-11-04 13:45:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a33c40e42c361a56,97dc2c769944e93e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:12.960 WARN 21062 --- [ocp-async-5,06d1b7c92d6549da,9936c893b3d77077] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:12.965 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:12.966 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:13.005 INFO 21062 --- [ocp-async-1,84987a447d642368,8efb8d34d0789457] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:13.984 INFO 21062 --- [prometheus-cache-monitor1,11779051a4a36e25,dbfa3f5367577cc8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1456,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:45:13.985 INFO 21062 --- [prometheus-cache-monitor1,11779051a4a36e25,dbfa3f5367577cc8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:45:14.695 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:41:14.677, end=2024-11-04T13:43:14.693} 2024-11-04 13:45:14.700 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:45:14.700 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:45:14.891 INFO 21062 --- [ocp-schedules-5,a297b9dfbf593b3f,68d0efb065797ea5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:45:14.960 WARN 21062 --- [ocp-async-3,a949e9ba2d616aaf,44c1e814965260c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3021964686c60ef8,af5372c4d9646069] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:15.005 INFO 21062 --- [ocp-async-2,6398cc715a5e6c7e,ab4177f155215d54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:15.966 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:15.967 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:16.960 WARN 21062 --- [ocp-async-4,bd8b3342d95d7fca,d5778bb59321aaeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:17.005 INFO 21062 --- [ocp-async-10,b4ba37141354853b,6a1709a624ee171a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:17.115 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:45:17.116 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:45:17.125 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:45:17.125 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:45:17.125 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:45:17.125 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:45:17.130 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:45:17.130 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:45:17.130 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:45:17.131 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:45:17.167 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:45:17.176 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:45:17.176 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:45:17.176 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:45:17.176 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:45:17.177 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:45:17.185 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:45:17.725 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:45:17.983 INFO 21062 --- [prometheus-minute-schedule-1,eec4dd1c59a3ffd7,aecef1734ed6429c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:18.303 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.172062689S 2024-11-04 13:45:18.303 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:45:18.399 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:45:18.399 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:45:18.426 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:45:18.485 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:45:18.485 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:45:18.485 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:45:18.485 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:45:18.485 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:45:18.486 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:45:18.497 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:45:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:45:18.590783+08:00, lastProcessTime=2024-11-04T13:45:08.589945+08:00 2024-11-04 13:45:18.617 INFO 21062 --- [ocp-async-2,23ebbbea3e2de6da,31c3bf865dcb7894] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:45:18.713 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] o.s.boot.SpringApplication : Started application in 1.588 seconds (JVM running for 4532.392) 2024-11-04 13:45:18.713 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.588303088S 2024-11-04 13:45:18.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:45:18.590783+08:00 2024-11-04 13:45:18.960 WARN 21062 --- [ocp-async-5,4043f03ca43d14a2,9d21970572f2a324] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:18.967 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:18.968 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:19.006 INFO 21062 --- [ocp-async-1,4f8d07d7a3caab63,3331842a46633dcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:19.165 INFO 21062 --- [ocp-schedules-1,55bbcbf4ed9b0093,7524ce163dc1a0ff] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:45:19.797 INFO 21062 --- [ocp-schedules-7,ea9b942b500b4ee4,7fd651916481f699] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:45:19.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2aa6850178549304,35ccf331ecc90e7a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=2aa6850178549304, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:45:19.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e1815d9404611585,6cbbd87006c50307] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=e1815d9404611585, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:45:19.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8205aff4e6748314,ffb12289c99a8f52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=8205aff4e6748314, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:45:19.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,a0f7c9ef148934ab,31f4a0cc8bca5f61] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=a0f7c9ef148934ab, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:45:19.812 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e1815d9404611585,6cbbd87006c50307] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=e1815d9404611585, duration=9 ms] 2024-11-04 13:45:19.812 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2aa6850178549304,35ccf331ecc90e7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=2aa6850178549304, duration=11 ms] 2024-11-04 13:45:19.812 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,a0f7c9ef148934ab,31f4a0cc8bca5f61] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a0f7c9ef148934ab, duration=9 ms] 2024-11-04 13:45:19.819 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8205aff4e6748314,ffb12289c99a8f52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=8205aff4e6748314, duration=19 ms] 2024-11-04 13:45:20.961 WARN 21062 --- [ocp-async-3,0fb4d16575d999f9,aa234ec18cc035f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e15ee1c6d6292411,031632d156431304] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:21.006 INFO 21062 --- [ocp-async-2,7c6e67452da12489,6f57c8e2dd0a050b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:21.969 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:21.970 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:22.961 WARN 21062 --- [ocp-async-4,8677ce71cbf4dec8,3e66609a664ad459] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:23.006 INFO 21062 --- [ocp-async-10,b7b37c0aa3aa07c8,cda412f8df06b934] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a2ca1bf955a7818b,d53006cd990f9ffb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:24.926 INFO 21062 --- [ocp-schedules-11,18e82e8d04fc9e3c,04421e62b590088b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:45:24.961 WARN 21062 --- [ocp-async-7,d7e928d5a0b12c47,7cc1a11bd328fa99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:24.970 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:24.970 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:25.006 INFO 21062 --- [ocp-async-8,90bc063872a789f3,f0ac2c5892072a1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:26.961 WARN 21062 --- [ocp-async-1,a358b42b8ae2ad28,b62735615c107c48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ba1b40105f0bd16a,08773d86df80a6bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:26.984 INFO 21062 --- [prometheus-hour-schedule-1,83b3d2eba4711210,2659a4c664283d22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:45:27.006 INFO 21062 --- [ocp-async-9,06cbf4751d16b340,49754346146647a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:27.970 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:27.972 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:28.498 INFO 21062 --- [ocp-schedules-6,d4b51fcd4b423906,08e8d70abd56a33d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:45:28.499 INFO 21062 --- [ocp-schedules-16,4bd7c8b656f6f684,a8df9069f744a911] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:45:28.500 INFO 21062 --- [ocp-schedules-11,110cac1882a18385,3bc87c497adec889] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:45:28.500 INFO 21062 --- [ocp-schedules-25,db9dc1ce68d72f4c,d6dc1050b670f158] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:45:28.503 INFO 21062 --- [ocp-schedules-4,32e874a32e0dfc65,e470793e9cfb4c88] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:45:28.503 INFO 21062 --- [ocp-schedules-6,d4b51fcd4b423906,08e8d70abd56a33d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:45:28.503 INFO 21062 --- [ocp-schedules-11,110cac1882a18385,3bc87c497adec889] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:28.504 INFO 21062 --- [ocp-schedules-25,db9dc1ce68d72f4c,d6dc1050b670f158] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:45:28.505 INFO 21062 --- [ocp-schedules-16,4bd7c8b656f6f684,a8df9069f744a911] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:45:28.507 INFO 21062 --- [ocp-schedules-1,7b285a65faa5ee3a,585e65e63c807f2e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:45:28.513 INFO 21062 --- [ocp-schedules-17,50a15ca25023bd31,adfd28fb8496bab3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:45:28.513 INFO 21062 --- [ocp-schedules-17,50a15ca25023bd31,adfd28fb8496bab3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:45:28.513 INFO 21062 --- [ocp-schedules-8,7febaef40ee77341,4d9dcd923701abb2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:45:28.514 INFO 21062 --- [ocp-schedules-8,7febaef40ee77341,4d9dcd923701abb2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:45:28.514 INFO 21062 --- [ocp-schedules-10,63754a2a80c5a7a3,aebd86c556c1975c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:45:28.514 INFO 21062 --- [ocp-schedules-10,63754a2a80c5a7a3,aebd86c556c1975c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:45:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:45:28.589629+08:00, lastProcessTime=2024-11-04T13:45:18.590783+08:00 2024-11-04 13:45:28.607 INFO 21062 --- [ocp-schedules-20,22ed23bbbbdbe690,ac6821f74239ec14] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:45:28.608 INFO 21062 --- [ocp-schedules-20,22ed23bbbbdbe690,ac6821f74239ec14] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:45:28.617 INFO 21062 --- [ocp-async-3,eed19ea8b252a4cc,d26842a211bab4d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:45:28.736 INFO 21062 --- [ocp-schedules-14,effbc0243d25a484,7e01f4bc92f274c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:45:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:45:28.589629+08:00 2024-11-04 13:45:28.921 INFO 21062 --- [ocp-schedules-23,fc052cac18c422e2,2a5856404b62b901] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:45:28.921 INFO 21062 --- [ocp-schedules-23,fc052cac18c422e2,2a5856404b62b901] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:45:28.953 INFO 21062 --- [ocp-schedules-5,5c8c2fb2aaf61014,0461131f3e7e9370] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:45:28.961 WARN 21062 --- [ocp-async-10,3826c4cd6a8adcfe,d21a446e18f5c4de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:28.987 INFO 21062 --- [ocp-schedules-1,2e4b01cfd635e5c2,bfa87dd6548fee81] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:45:28.992 INFO 21062 --- [ocp-schedules-1,2e4b01cfd635e5c2,bfa87dd6548fee81] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:45:28.992 INFO 21062 --- [ocp-schedules-1,2e4b01cfd635e5c2,bfa87dd6548fee81] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:45:29.007 INFO 21062 --- [ocp-async-8,f2691b77d8bcc8de,1a92ec29326ac2f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:29.234 INFO 21062 --- [ocp-schedules-27,0df91f2af0d44e41,ccfe1e7edda235bb] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:45:29.234 INFO 21062 --- [ocp-schedules-27,0df91f2af0d44e41,ccfe1e7edda235bb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:45:29.241 INFO 21062 --- [ocp-schedules-27,0df91f2af0d44e41,ccfe1e7edda235bb] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:45:29.241 INFO 21062 --- [ocp-schedules-27,0df91f2af0d44e41,ccfe1e7edda235bb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:45:29.245 INFO 21062 --- [ocp-schedules-27,0df91f2af0d44e41,ccfe1e7edda235bb] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:45:29.245 INFO 21062 --- [ocp-schedules-27,0df91f2af0d44e41,ccfe1e7edda235bb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709925194352054272 2024-11-04 13:45:29.800 INFO 21062 --- [ocp-schedules-22,bca9e8219f54ad35,7f015f524aa0c04d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:45:29.858 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,275e645b8b27025f,1cd974083512d6a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=275e645b8b27025f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:45:29.858 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,7aa37de2f8e8d102,6a288b6970ae6271] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=7aa37de2f8e8d102, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:45:29.858 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9c4e9db62ddea819,31b766ca43eee771] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=9c4e9db62ddea819, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:45:29.858 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,3b7f77a9f7f3542e,d567d4a4db4fa693] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=3b7f77a9f7f3542e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:45:29.859 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,275e645b8b27025f,1cd974083512d6a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=275e645b8b27025f, duration=11 ms] 2024-11-04 13:45:29.859 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9c4e9db62ddea819,31b766ca43eee771] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=9c4e9db62ddea819, duration=12 ms] 2024-11-04 13:45:29.859 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,7aa37de2f8e8d102,6a288b6970ae6271] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=7aa37de2f8e8d102, duration=13 ms] 2024-11-04 13:45:29.861 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,3b7f77a9f7f3542e,d567d4a4db4fa693] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=3b7f77a9f7f3542e, duration=15 ms] 2024-11-04 13:45:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4774b71afcbb5c9e,2abc1bfbabdf9592] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:45:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:45:30.511 INFO 21062 --- [ocp-schedules-10,11ee4c59ded3d436,9d5b25b09e19fbe5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:45:30.518 INFO 21062 --- [ocp-schedules-10,11ee4c59ded3d436,9d5b25b09e19fbe5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:45:30.518 INFO 21062 --- [ocp-schedules-10,11ee4c59ded3d436,9d5b25b09e19fbe5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:45:30.519 INFO 21062 --- [ocp-schedules-6,b6eb2dd8b37dde76,05bec89c65045541] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:45:30.519 INFO 21062 --- [ocp-schedules-6,b6eb2dd8b37dde76,05bec89c65045541] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:45:30.545 INFO 21062 --- [ocp-schedules-6,b6eb2dd8b37dde76,05bec89c65045541] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:45:30.545 INFO 21062 --- [ocp-schedules-6,b6eb2dd8b37dde76,05bec89c65045541] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:45:30.547 INFO 21062 --- [ocp-schedules-27,2fa0db2826f98417,eb01454f8908ae76] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:45:30.681 INFO 21062 --- [ocp-schedules-29,8e18ec4a439a125e,63a81b13d29df98f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:45:30.681+08:00, clusters=[1] 2024-11-04 13:45:30.683 INFO 21062 --- [ocp-schedules-29,8e18ec4a439a125e,63a81b13d29df98f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:45:30.732 INFO 21062 --- [ocp-schedules-15,f0e7b810b9f8224d,895af798360b63db] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:45:30.732 INFO 21062 --- [ocp-schedules-15,f0e7b810b9f8224d,895af798360b63db] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:45:30.733 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:45:30.733 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:45:30.742 INFO 21062 --- [ocp-schedules-15,f0e7b810b9f8224d,895af798360b63db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:45:30.742 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:45:30.743 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:45:30.744 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:45:30.744 WARN 21062 --- [ocp-schedules-15,f0e7b810b9f8224d,895af798360b63db] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:45:30.745 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:45:30.745 INFO 21062 --- [ocp-schedules-15,f0e7b810b9f8224d,895af798360b63db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:45:30.745 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:45:30.746 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:45:30.746 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:45:30.746 WARN 21062 --- [ocp-schedules-15,f0e7b810b9f8224d,895af798360b63db] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:45:30.747 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:45:30.754 ERROR 21062 --- [ocp-schedules-15,f0e7b810b9f8224d,895af798360b63db] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:45:30.827 INFO 21062 --- [ocp-schedules-9,97c335f4447c91ef,11df8f77383f7477] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:45:30.827+08:00, clusters=[1] 2024-11-04 13:45:30.831 INFO 21062 --- [ocp-schedules-9,97c335f4447c91ef,11df8f77383f7477] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:45:30.962 WARN 21062 --- [ocp-async-8,0498e5539c12e765,60b5d89c29a3383d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:30.972 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:30.973 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:31.007 INFO 21062 --- [ocp-async-3,ca9b1b989b13fafe,b82f8341baa19102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:32.962 WARN 21062 --- [ocp-async-3,9f1f66dea4fa240d,9ffad6da524c6214] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d1f3a0d54f017031,de9d8707dca31ecd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:33.007 INFO 21062 --- [ocp-async-4,71a2bb7337f5f7f6,8e9cec14f7aa52a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:33.500 INFO 21062 --- [ocp-schedules-12,2278df3a83343d83,320675e6207a257f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:45:33.501 INFO 21062 --- [ocp-schedules-12,548875d90ad150c4,9480bccbe12b74a9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:45:33.505 INFO 21062 --- [ocp-schedules-12,548875d90ad150c4,9480bccbe12b74a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:33.533 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:45:33.533 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:45:33.973 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:33.974 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:34.962 WARN 21062 --- [ocp-async-4,f9e1a29175358a7f,ab08dfe5686b25d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:34.971 INFO 21062 --- [ocp-schedules-6,a6f3f803d8d54cb6,3354cb7987d299d6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:45:35.007 INFO 21062 --- [ocp-async-7,894884a1de58d373,8b7e14ff3597458f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:35.238 INFO 21062 --- [ocp-schedules-15,ef468ac122938c5d,cb0b2139d084e443] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:45:35.242 INFO 21062 --- [ocp-schedules-15,ef468ac122938c5d,cb0b2139d084e443] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:35.326 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:45:35.328 ERROR 21062 --- [ocp-schedules-15,ef468ac122938c5d,cb0b2139d084e443] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:45:35.328 INFO 21062 --- [ocp-schedules-15,ef468ac122938c5d,cb0b2139d084e443] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:45:35.983 INFO 21062 --- [prometheus-minute-schedule-1,ccc69629634e523e,edfa78ba38c4c6b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:36.962 WARN 21062 --- [ocp-async-7,510b79cefffeeb81,afd4d3f4200298b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:36.974 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:36.975 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:37.007 INFO 21062 --- [ocp-async-1,95a64d5d2da0ff43,120a287af68c846e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:38.504 INFO 21062 --- [ocp-schedules-25,db9dc1ce68d72f4c,d6dc1050b670f158] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:45:38.505 INFO 21062 --- [ocp-schedules-11,110cac1882a18385,3bc87c497adec889] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:45:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:45:38.589901+08:00, lastProcessTime=2024-11-04T13:45:28.589629+08:00 2024-11-04 13:45:38.617 INFO 21062 --- [ocp-async-5,08324fa4696e12e7,30d0bbb77fea5cae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:45:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:45:38.589901+08:00 2024-11-04 13:45:38.962 WARN 21062 --- [ocp-async-9,f49181d5a76e46c9,56795b78005c6516] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:38.983 INFO 21062 --- [prometheus-minute-schedule-1,41210375a3a21573,eb489e3fc6929c3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:39.007 INFO 21062 --- [ocp-async-10,5dedb8b697456321,5eb270ce7d2b3690] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:39.801 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,0707354fce8ff102,06a2c46a3d789bb3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=0707354fce8ff102, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:45:39.801 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6e8c2af5da1ea72f,ea4e99b67c87b6ab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=6e8c2af5da1ea72f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:45:39.801 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,1251cd17677b426a,83d468afd9e8a5b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=1251cd17677b426a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:45:39.801 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fff051685e0fb8b,0362d2f3958a6d77] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=8fff051685e0fb8b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:45:39.802 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,0707354fce8ff102,06a2c46a3d789bb3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=0707354fce8ff102, duration=8 ms] 2024-11-04 13:45:39.802 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,1251cd17677b426a,83d468afd9e8a5b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=1251cd17677b426a, duration=11 ms] 2024-11-04 13:45:39.802 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6e8c2af5da1ea72f,ea4e99b67c87b6ab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=6e8c2af5da1ea72f, duration=9 ms] 2024-11-04 13:45:39.803 INFO 21062 --- [ocp-schedules-4,f49f127289c0243a,8957ef52dda177f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:45:39.804 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fff051685e0fb8b,0362d2f3958a6d77] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=8fff051685e0fb8b, duration=13 ms] 2024-11-04 13:45:39.975 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:39.976 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:40.004 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:45:40.004 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:45:40.963 WARN 21062 --- [ocp-async-10,2d703b7d5ed3ec93,3f5a2d3da355f56d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:41.008 INFO 21062 --- [ocp-async-5,c40888dc0a9a1d9e,787b3ba13ef3ce2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:41.983 INFO 21062 --- [prometheus-minute-schedule-1,17641e8c4590c896,fbf101c4a171287b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:42.963 WARN 21062 --- [ocp-async-5,d6f41bed37539fdb,06e124b00306ce08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:42.976 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:42.976 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:43.008 INFO 21062 --- [ocp-async-3,8d7f52a1444e190c,587185e39fb8c95c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:44.963 WARN 21062 --- [ocp-async-3,e79866f7acb41b76,0278685f54ff6e82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:44.983 INFO 21062 --- [prometheus-minute-schedule-1,673c07274984ac9f,da323ed1f17532cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:44.995 INFO 21062 --- [ocp-schedules-23,3f2f136373621ba8,b12e24d3b8ecd2df] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:45:45.008 INFO 21062 --- [ocp-async-4,d12e887afe36ac82,996a00c74f5a0b70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:45.977 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:45.977 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:46.963 WARN 21062 --- [ocp-async-4,702211285b09c26e,6e1299c228df4fcc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:47.008 INFO 21062 --- [ocp-async-7,1e7e7a83b9e25105,bf08ef950611a324] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:47.983 INFO 21062 --- [prometheus-minute-schedule-1,afd81c86a4afab77,d06f83fda1f91845] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:45:48.591021+08:00, lastProcessTime=2024-11-04T13:45:38.589901+08:00 2024-11-04 13:45:48.617 INFO 21062 --- [ocp-async-4,dac1726421dec602,caf4770dc0a4e6a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:45:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:45:48.591021+08:00 2024-11-04 13:45:48.858 INFO 21062 --- [ocp-schedules-25,f16a0943c14ba090,12dc4adce0b17101] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:45:48.861 INFO 21062 --- [ocp-schedules-25,f16a0943c14ba090,12dc4adce0b17101] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:48.862 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:45:48.963 WARN 21062 --- [ocp-async-5,2535d02cabd8835c,6bcc064d09a4d307] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:48.977 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:48.978 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:49.008 INFO 21062 --- [ocp-async-3,c392d9de7b30d375,d4f6d71dc0aae475] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:49.804 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7e1ad2c55fb3d6e4,79cf01d683f32c35] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=7e1ad2c55fb3d6e4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:45:49.804 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,966b4b646d924360,260c15265761c5e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=966b4b646d924360, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:45:49.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7e1ad2c55fb3d6e4,79cf01d683f32c35] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=7e1ad2c55fb3d6e4, duration=8 ms] 2024-11-04 13:45:49.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,bd107adda2c58921,4becd2732fb82567] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=bd107adda2c58921, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:45:49.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b8f8e8ec754ca72a,323a1eda97a4a749] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=b8f8e8ec754ca72a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:45:49.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b8f8e8ec754ca72a,323a1eda97a4a749] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=b8f8e8ec754ca72a, duration=5 ms] 2024-11-04 13:45:49.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,bd107adda2c58921,4becd2732fb82567] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=bd107adda2c58921, duration=5 ms] 2024-11-04 13:45:49.805 INFO 21062 --- [ocp-schedules-20,c4aff0d2edc05b47,5df2e2cf26933fa9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:45:49.807 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,966b4b646d924360,260c15265761c5e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=966b4b646d924360, duration=11 ms] 2024-11-04 13:45:50.521 INFO 21062 --- [ocp-schedules-11,eb4d137adabe0769,1bf36a92f77ad36f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:45:50.528 INFO 21062 --- [ocp-schedules-11,eb4d137adabe0769,1bf36a92f77ad36f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:45:50.528 INFO 21062 --- [ocp-schedules-11,eb4d137adabe0769,1bf36a92f77ad36f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:45:50.963 WARN 21062 --- [ocp-async-3,0d3a5c326bed99c9,d821f44ab8ebde7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:50.983 INFO 21062 --- [prometheus-minute-schedule-1,c29a7433b62dac38,a4f6d88bab717b7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:51.009 INFO 21062 --- [ocp-async-4,b5a22c927d474d08,379231127aa94cb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:51.978 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:51.979 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:52.964 WARN 21062 --- [ocp-async-4,f7e7a8c2dc827e30,74b6e2c0bacb54ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:53.009 INFO 21062 --- [ocp-async-7,97dfea8e8262dd80,e37e94b94d6139d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:53.983 INFO 21062 --- [prometheus-minute-schedule-1,bbe3906273d58262,e1d18531b08b762c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:54.964 WARN 21062 --- [ocp-async-7,947713a81d1baf40,5d0d4bc378aeee93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:54.980 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:54.980 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:55.009 INFO 21062 --- [ocp-async-1,e8cbc62a81a3af09,a59574e65ae0d77d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:55.036 INFO 21062 --- [ocp-schedules-4,93d5d15f8d7fede6,c999ae86c196006f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:45:56.964 WARN 21062 --- [ocp-async-1,63cb009b3a89fcfd,e68480941db35d87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:56.983 INFO 21062 --- [prometheus-minute-schedule-1,756b4d7125d39e23,88b52c92efc3f69f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:45:57.009 INFO 21062 --- [ocp-async-2,dc98abdef00acd29,74c75864221fcf15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:57.980 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:45:57.981 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:45:58.500 INFO 21062 --- [ocp-schedules-20,8ff188b0ab64f370,79fcca8ccb6c25a5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:45:58.500 INFO 21062 --- [ocp-schedules-23,e4275bc33ccdd8a1,6ac23a11b54deb65] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:45:58.502 INFO 21062 --- [ocp-schedules-19,acf8aeafde9ce62e,1d18bf9067ed20fa] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:45:58.503 INFO 21062 --- [ocp-schedules-20,8ff188b0ab64f370,79fcca8ccb6c25a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:45:58.504 INFO 21062 --- [ocp-schedules-6,57d8f356d72c5032,c37511fd3a4a96a9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:45:58.505 INFO 21062 --- [ocp-schedules-23,e4275bc33ccdd8a1,6ac23a11b54deb65] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:45:58.511 INFO 21062 --- [ocp-schedules-27,bae3270e4c108fdb,fa279e443857bf74] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:45:58.511 INFO 21062 --- [ocp-schedules-27,bae3270e4c108fdb,fa279e443857bf74] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:45:58.511 INFO 21062 --- [ocp-schedules-13,ea7ca9544a821bd9,ead5a0d499e06751] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:45:58.511 INFO 21062 --- [ocp-schedules-13,ea7ca9544a821bd9,ead5a0d499e06751] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:45:58.512 INFO 21062 --- [ocp-schedules-17,1fe1fa0c9f500e05,cb77cbd7d098c6e3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:45:58.512 INFO 21062 --- [ocp-schedules-17,1fe1fa0c9f500e05,cb77cbd7d098c6e3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:45:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:45:58.590205+08:00, lastProcessTime=2024-11-04T13:45:48.591021+08:00 2024-11-04 13:45:58.617 INFO 21062 --- [ocp-async-1,64e6972fd1cc1169,3fb7e2d3840bbcdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:45:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:45:58.590205+08:00 2024-11-04 13:45:58.862 INFO 21062 --- [ocp-schedules-25,f16a0943c14ba090,12dc4adce0b17101] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:45:58.956 INFO 21062 --- [ocp-schedules-3,239201e1fb03bf3f,fa900fb01f728db1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:45:58.964 WARN 21062 --- [ocp-async-4,5b7a18da8a1976d4,64d481560e91afe2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:45:59.009 INFO 21062 --- [ocp-async-6,27d4a63edb13ad90,c69d2c6fe412bc02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:45:59.797 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ee7b24fba191f9ca,e1f2ca22f69c39e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=ee7b24fba191f9ca, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:45:59.797 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,57ad54f4192a5b6d,fb71bc6ee0fc2b3b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=57ad54f4192a5b6d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:45:59.797 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,6701a7c45956f92b,b2456766bb1ee735] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=6701a7c45956f92b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:45:59.798 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,6701a7c45956f92b,b2456766bb1ee735] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=6701a7c45956f92b, duration=9 ms] 2024-11-04 13:45:59.798 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,57ad54f4192a5b6d,fb71bc6ee0fc2b3b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=57ad54f4192a5b6d, duration=6 ms] 2024-11-04 13:45:59.798 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9db8f0032123dc73,5a816308ae54dffb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=9db8f0032123dc73, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:45:59.799 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9db8f0032123dc73,5a816308ae54dffb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=9db8f0032123dc73, duration=5 ms] 2024-11-04 13:45:59.801 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ee7b24fba191f9ca,e1f2ca22f69c39e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=ee7b24fba191f9ca, duration=12 ms] 2024-11-04 13:45:59.808 INFO 21062 --- [ocp-schedules-10,683dc4d4f7395f0c,581ae9278d51a4a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:45:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2cc7466f1888929b,14769f8de9579f60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699160, interval=60 2024-11-04 13:46:00.965 WARN 21062 --- [ocp-async-6,cb38f64d9b9843a8,f860be3b598f4637] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:00.981 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:00.982 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:01.009 INFO 21062 --- [ocp-async-8,5db6bd149b763f1f,5198745702702d18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:02.965 WARN 21062 --- [ocp-async-8,0f71cfb4b83bdad1,070404639f3efa57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d6513f4816f08356,3e7ea8fba5c11fb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:03.010 INFO 21062 --- [ocp-async-9,122fbd2135ba8d9b,147f902f8f02d000] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:03.533 INFO 21062 --- [ocp-schedules-12,548875d90ad150c4,9480bccbe12b74a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:46:03.534 INFO 21062 --- [ocp-schedules-12,f4e14b4deeb6761b,fdeb2a11940b38ba] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:46:03.538 INFO 21062 --- [ocp-schedules-12,f4e14b4deeb6761b,fdeb2a11940b38ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:03.567 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:46:03.567 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:46:03.982 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:03.983 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:04.965 WARN 21062 --- [ocp-async-9,82c2bb01d37812f8,647caf04d2bd035f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:05.010 INFO 21062 --- [ocp-async-10,e1670cd1ef30b8d6,ce07ddbfeb94b29f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:05.060 INFO 21062 --- [ocp-schedules-1,5a8117f13ab6af99,22c00645823f244c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:46:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a1a6d1ce8b979d4f,5c11d16a835321e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:06.965 WARN 21062 --- [ocp-async-10,ec0334b4f3ac72dc,e205f9ef1735fb89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:06.983 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:06.984 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:07.010 INFO 21062 --- [ocp-async-5,2369758697b6af78,e093b2559b36a6fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:08.504 INFO 21062 --- [ocp-schedules-20,8ff188b0ab64f370,79fcca8ccb6c25a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:46:08.506 INFO 21062 --- [ocp-schedules-23,e4275bc33ccdd8a1,6ac23a11b54deb65] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:46:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:46:08.590897+08:00, lastProcessTime=2024-11-04T13:45:58.590205+08:00 2024-11-04 13:46:08.618 INFO 21062 --- [ocp-async-7,b3523605e18cceda,a6b3b912680b8a69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:46:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:46:08.590897+08:00 2024-11-04 13:46:08.965 WARN 21062 --- [ocp-async-8,cf5625abd65f1231,42ca1fb426db43d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d094468c7d21bddc,10c1616af3f54631] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:09.010 INFO 21062 --- [ocp-async-2,0876c38e8565fdd8,5c26378f536c7aab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:09.802 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,32801bbb034c5ad9,7bd97d9b5cd7a721] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=32801bbb034c5ad9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:46:09.802 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,a95c487b9c30a420,7b2475410cea3892] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=a95c487b9c30a420, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:46:09.802 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,cab4decbda57ebc3,b6d07b756deca601] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=cab4decbda57ebc3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:46:09.804 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,cab4decbda57ebc3,b6d07b756deca601] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=cab4decbda57ebc3, duration=8 ms] 2024-11-04 13:46:09.804 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,32801bbb034c5ad9,7bd97d9b5cd7a721] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=32801bbb034c5ad9, duration=11 ms] 2024-11-04 13:46:09.804 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8feb8c3b1ebe5017,54104328949387d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=8feb8c3b1ebe5017, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:46:09.804 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8feb8c3b1ebe5017,54104328949387d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=8feb8c3b1ebe5017, duration=8 ms] 2024-11-04 13:46:09.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,a95c487b9c30a420,7b2475410cea3892] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a95c487b9c30a420, duration=12 ms] 2024-11-04 13:46:09.810 INFO 21062 --- [ocp-schedules-26,1742abb9da2d5b7b,9fd2ba4c31aa0790] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:46:09.984 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:09.986 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:10.002 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:46:10.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:46:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:46:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:46:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:46:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:46:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:46:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:46:10.531 INFO 21062 --- [ocp-schedules-6,1ddc74fd305fd166,56fd2f5dc1aceddc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:46:10.536 INFO 21062 --- [ocp-schedules-6,1ddc74fd305fd166,56fd2f5dc1aceddc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:46:10.537 INFO 21062 --- [ocp-schedules-6,1ddc74fd305fd166,56fd2f5dc1aceddc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:46:10.965 WARN 21062 --- [ocp-async-9,1959538119e9fd8b,518824b325abcee3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:11.011 INFO 21062 --- [ocp-async-10,c40e406fd3838bfa,851d9451e89bcd68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:11.336 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699100, nodeCount=34, affectRows=34 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699100, nodeCount=43, affectRows=43 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699100, nodeCount=59, affectRows=59 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699100, nodeCount=34, validCount=34 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699100, nodeCount=40, affectRows=40 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699100, nodeCount=78, affectRows=78 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699100, nodeCount=65, affectRows=65 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699100, nodeCount=45, affectRows=45 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699100, nodeCount=43, validCount=43 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699100, nodeCount=39, validCount=39 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699100, nodeCount=59, validCount=59 2024-11-04 13:46:11.337 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699100, nodeCount=45, validCount=45 2024-11-04 13:46:11.338 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699100, nodeCount=65, validCount=65 2024-11-04 13:46:11.338 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699100, nodeCount=76, validCount=76 2024-11-04 13:46:11.983 INFO 21062 --- [prometheus-minute-schedule-1,59fd24cc510cd8dd,6be5fe9ddea7bc77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:12.966 WARN 21062 --- [ocp-async-7,c27f3111337ebfc9,44dc7af919358ae4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:12.986 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:12.987 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:13.011 INFO 21062 --- [ocp-async-5,a3899df3f68055a6,d8c7577aeab359bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,e97829e123a475d5,30212e1927a27cb1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1456,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,e97829e123a475d5,30212e1927a27cb1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:46:14.966 WARN 21062 --- [ocp-async-1,63ac389068be585c,8720a9a786b9dfbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9c8d2714c8bc5b9b,404fb2971189f2c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:15.011 INFO 21062 --- [ocp-async-3,acc7449aa83093ad,ad694cefb0348f45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:15.102 INFO 21062 --- [ocp-schedules-14,23372d237dacf441,dbb59f8710a0cb88] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:46:15.987 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:15.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:16.966 WARN 21062 --- [ocp-async-2,64f0cab6402502f5,971db4ac3306b662] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:17.011 INFO 21062 --- [ocp-async-4,173f23ee42c2c420,c43aa2d3f7a0c216] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5b9c9a14997a40e8,cfbf2ca859cbe330] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:46:18.590963+08:00, lastProcessTime=2024-11-04T13:46:08.590897+08:00 2024-11-04 13:46:18.618 INFO 21062 --- [ocp-async-3,a2e92f6222fca449,266380483cb1f009] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:46:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:46:18.590963+08:00 2024-11-04 13:46:18.966 WARN 21062 --- [ocp-async-7,84f8a5ee26e1660c,ac4f2d87a538c8e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:18.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:18.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:19.011 INFO 21062 --- [ocp-async-5,904ce950dfad0b98,56a0a9145d627fa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:19.814 INFO 21062 --- [ocp-schedules-24,ce5ef957189238f8,8908d757da72b417] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:46:20.678 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5f4897f23eebb64a,151d0d421639d476] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=5f4897f23eebb64a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:46:20.678 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,312df87538173677,b52a17023b0371a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=312df87538173677, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:46:20.678 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ccf6eaf07146b27c,989250491fb4c6b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=ccf6eaf07146b27c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:46:20.678 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,a8b349db097fb915,1d5fa093e7e77f25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=a8b349db097fb915, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:46:20.679 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ccf6eaf07146b27c,989250491fb4c6b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=ccf6eaf07146b27c, duration=10 ms] 2024-11-04 13:46:20.679 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5f4897f23eebb64a,151d0d421639d476] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=5f4897f23eebb64a, duration=7 ms] 2024-11-04 13:46:20.679 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,312df87538173677,b52a17023b0371a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=312df87538173677, duration=8 ms] 2024-11-04 13:46:20.681 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,a8b349db097fb915,1d5fa093e7e77f25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=a8b349db097fb915, duration=12 ms] 2024-11-04 13:46:20.966 WARN 21062 --- [ocp-async-1,e29fd41d049ecf50,e9cc7191f0591149] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:20.983 INFO 21062 --- [prometheus-minute-schedule-1,002dddc2f689cb0e,598a046ce536303d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:21.012 INFO 21062 --- [ocp-async-3,3965ef5c143e7879,81ad5b2b6825b695] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:21.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:21.989 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:22.966 WARN 21062 --- [ocp-async-2,da3e59eecfdab73f,0681ea10009a65ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:23.012 INFO 21062 --- [ocp-async-4,a958f4d89ff115da,0093be6f97e52e4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6de3d4fa22515ace,96eac0522b0ab6a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:24.966 WARN 21062 --- [ocp-async-10,44047f4b9ea56f51,b0aa73d9ba018add] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:24.989 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:24.990 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:25.012 INFO 21062 --- [ocp-async-6,871be69edfa468d6,4a12c10a5c2e0b2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:25.127 INFO 21062 --- [ocp-schedules-16,c5c5a360cf1bd1d8,9798d00520bab597] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:46:26.967 WARN 21062 --- [ocp-async-5,06a920ec159186be,d00b1dd08b6878e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:26.983 INFO 21062 --- [prometheus-minute-schedule-1,73a9e06df96559de,2f19217d7d1d1fbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4da9d39fdcc9893d,8b70c0446bba5a0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:46:27.012 INFO 21062 --- [ocp-async-8,168978c5a5ca4e74,8c2e39b91ca04de8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:27.990 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:27.991 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:28.498 INFO 21062 --- [ocp-schedules-21,0e72ef4a29d19c2a,79a1f008b04bad91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:46:28.499 INFO 21062 --- [ocp-schedules-22,5bc941aafcb5f058,035f7c2734b9e5f0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:46:28.500 INFO 21062 --- [ocp-schedules-16,3aa3af36f57490d2,14a361e13269cab7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:46:28.500 INFO 21062 --- [ocp-schedules-17,324d540d725855aa,dcad13c70ed92014] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:46:28.502 INFO 21062 --- [ocp-schedules-21,0e72ef4a29d19c2a,79a1f008b04bad91] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:46:28.502 INFO 21062 --- [ocp-schedules-28,a44be0105f7d9dc5,8642ecc4cdfcdefb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:46:28.503 INFO 21062 --- [ocp-schedules-22,5bc941aafcb5f058,035f7c2734b9e5f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:46:28.504 INFO 21062 --- [ocp-schedules-10,cf2b70c627133898,9e95529d50dfb47f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:46:28.504 INFO 21062 --- [ocp-schedules-17,324d540d725855aa,dcad13c70ed92014] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:28.505 INFO 21062 --- [ocp-schedules-16,3aa3af36f57490d2,14a361e13269cab7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:46:28.510 INFO 21062 --- [ocp-schedules-19,12286125ca60775b,c27e1a543a510d48] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:46:28.510 INFO 21062 --- [ocp-schedules-8,ade89a56e139b3a1,d45aafb8c91d7c91] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:46:28.510 INFO 21062 --- [ocp-schedules-27,7f96217661624d20,81da6894c959a104] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:46:28.510 INFO 21062 --- [ocp-schedules-8,ade89a56e139b3a1,d45aafb8c91d7c91] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:46:28.510 INFO 21062 --- [ocp-schedules-27,7f96217661624d20,81da6894c959a104] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:46:28.510 INFO 21062 --- [ocp-schedules-19,12286125ca60775b,c27e1a543a510d48] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:46:28.569 INFO 21062 --- [ocp-schedules-15,004ace7b34f98332,c1299a433b32e037] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1456,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":363,"seriesIdKeySize":364,"secondWriteQueue":0,"minuteWriteCacheSize":361} 2024-11-04 13:46:28.569 INFO 21062 --- [ocp-schedules-15,004ace7b34f98332,c1299a433b32e037] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=37, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=37, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:46:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:46:28.589944+08:00, lastProcessTime=2024-11-04T13:46:18.590963+08:00 2024-11-04 13:46:28.618 INFO 21062 --- [ocp-async-1,b1ca91cdcd110498,b291e2c3c0ab3a24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:46:28.689 INFO 21062 --- [ocp-schedules-25,dab612ece0fca099,9bdfc1cecc8e4873] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:46:28.689 INFO 21062 --- [ocp-schedules-25,dab612ece0fca099,9bdfc1cecc8e4873] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:46:28.739 INFO 21062 --- [ocp-schedules-9,d9150afb46366e88,4362fc86f122b32e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:46:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:46:28.589944+08:00 2024-11-04 13:46:28.926 INFO 21062 --- [ocp-schedules-14,a173c8ff2facd69d,9afacbc321118133] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:46:28.926 INFO 21062 --- [ocp-schedules-14,a173c8ff2facd69d,9afacbc321118133] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:46:28.958 INFO 21062 --- [ocp-schedules-5,38167ab9f30418e6,67d6f9cacad632ba] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:46:28.967 WARN 21062 --- [ocp-async-4,26fa7b8446698aba,61ad1b2af347e854] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:28.992 INFO 21062 --- [ocp-schedules-8,144ad9f65cfea903,4204f09da0a50c94] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:46:28.997 INFO 21062 --- [ocp-schedules-8,144ad9f65cfea903,4204f09da0a50c94] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:46:28.997 INFO 21062 --- [ocp-schedules-8,144ad9f65cfea903,4204f09da0a50c94] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:46:29.012 INFO 21062 --- [ocp-async-6,d340241bcffc91ca,b076c0e479cc1283] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:29.061 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:46:29.061 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:46:29.069 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:46:29.069 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:46:29.069 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:46:29.080 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:46:29.080 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:46:29.080 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:46:29.083 INFO 21062 --- [ocp-schedules-21,e3344c77a9323d02,2c9ad2a34da96460] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:46:29.250 INFO 21062 --- [ocp-schedules-30,2b546879d6ab0378,6646960e95d1a738] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:46:29.250 INFO 21062 --- [ocp-schedules-30,2b546879d6ab0378,6646960e95d1a738] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:46:29.251 INFO 21062 --- [ocp-schedules-30,2b546879d6ab0378,6646960e95d1a738] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:46:29.251 INFO 21062 --- [ocp-schedules-30,2b546879d6ab0378,6646960e95d1a738] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:46:29.252 INFO 21062 --- [ocp-schedules-30,2b546879d6ab0378,6646960e95d1a738] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:46:29.252 INFO 21062 --- [ocp-schedules-30,2b546879d6ab0378,6646960e95d1a738] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709925194352054272 2024-11-04 13:46:29.816 INFO 21062 --- [ocp-schedules-9,99678fa4da1b62f0,c1da0b6473dbea07] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:46:29.983 INFO 21062 --- [prometheus-minute-schedule-1,50b304dca7378bf0,1307e87d3e873868] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:46:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:46:30.539 INFO 21062 --- [ocp-schedules-3,19937325ee55b441,4ca73bcb24a39954] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:46:30.545 INFO 21062 --- [ocp-schedules-3,19937325ee55b441,4ca73bcb24a39954] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:46:30.545 INFO 21062 --- [ocp-schedules-3,19937325ee55b441,4ca73bcb24a39954] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:46:30.545 INFO 21062 --- [ocp-schedules-5,1715a316ec1e09e2,4b5a1a1982867798] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:46:30.546 INFO 21062 --- [ocp-schedules-5,1715a316ec1e09e2,4b5a1a1982867798] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:46:30.572 INFO 21062 --- [ocp-schedules-5,1715a316ec1e09e2,4b5a1a1982867798] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:46:30.572 INFO 21062 --- [ocp-schedules-5,1715a316ec1e09e2,4b5a1a1982867798] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:46:30.573 INFO 21062 --- [ocp-schedules-6,dab67c73a40175bd,63d3956917542e0f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:46:30.710 INFO 21062 --- [ocp-schedules-25,7c079fc0fd30f512,92b18d7026d9f802] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:46:30.710+08:00, clusters=[1] 2024-11-04 13:46:30.714 INFO 21062 --- [ocp-schedules-25,7c079fc0fd30f512,92b18d7026d9f802] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:46:30.762 INFO 21062 --- [ocp-schedules-29,a595e75bc9717d58,dd5490e504d95cc0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:46:30.762 INFO 21062 --- [ocp-schedules-29,a595e75bc9717d58,dd5490e504d95cc0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:46:30.762 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:46:30.762 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:46:30.772 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:46:30.773 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:46:30.773 INFO 21062 --- [ocp-schedules-29,a595e75bc9717d58,dd5490e504d95cc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:46:30.774 INFO 21062 --- [ocp-schedules-20,060f0981a3f361b7,cca49b705423bbea] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:46:30.774 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:46:30.774 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:46:30.774 WARN 21062 --- [ocp-schedules-29,a595e75bc9717d58,dd5490e504d95cc0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:46:30.775 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:46:30.775 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:46:30.775 INFO 21062 --- [ocp-schedules-29,a595e75bc9717d58,dd5490e504d95cc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:46:30.776 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:46:30.776 WARN 21062 --- [ocp-schedules-29,a595e75bc9717d58,dd5490e504d95cc0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:46:30.776 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:46:30.777 INFO 21062 --- [ocp-schedules-20,060f0981a3f361b7,cca49b705423bbea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:30.783 ERROR 21062 --- [ocp-schedules-29,a595e75bc9717d58,dd5490e504d95cc0] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:46:30.834 ERROR 21062 --- [ocp-schedules-20,060f0981a3f361b7,cca49b705423bbea] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:46:30.834 INFO 21062 --- [ocp-schedules-20,060f0981a3f361b7,cca49b705423bbea] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:46:30.857 INFO 21062 --- [ocp-schedules-13,08ba3878a99da735,e1b69a5eead88135] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:46:30.857+08:00, clusters=[1] 2024-11-04 13:46:30.861 INFO 21062 --- [ocp-schedules-13,08ba3878a99da735,e1b69a5eead88135] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:46:30.967 WARN 21062 --- [ocp-async-7,810269cacf718882,f030e91bc008068f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:30.991 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:30.992 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:31.012 INFO 21062 --- [ocp-async-1,5ea146e13f29b5d0,081e82a5e2be5fec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:32.967 WARN 21062 --- [ocp-async-1,a0397bc584a8b25d,75a8991acea8c6c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3ecaf0acc19e425d,e727c49be9ed4a03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:33.013 INFO 21062 --- [ocp-async-2,a5ca38c392e845e4,d008f66adc90d7a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:33.568 INFO 21062 --- [ocp-schedules-12,f4e14b4deeb6761b,fdeb2a11940b38ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:46:33.568 INFO 21062 --- [ocp-schedules-12,5c5deb5cf60fb80b,65720ca72075b995] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:46:33.572 INFO 21062 --- [ocp-schedules-12,5c5deb5cf60fb80b,65720ca72075b995] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:33.602 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:46:33.602 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:46:33.992 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:33.993 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:34.967 WARN 21062 --- [ocp-async-2,7b1e515c2a2f030c,e1bfbfa5078cee0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:35.013 INFO 21062 --- [ocp-async-10,a10efc088c851047,22461e50b6dddee7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:35.170 INFO 21062 --- [ocp-schedules-20,a61b2b05fd2a45ae,03a34b8b225ea576] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:46:35.328 INFO 21062 --- [ocp-schedules-21,a7573efbcaa36461,65b5b18b5eeb10eb] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:46:35.332 INFO 21062 --- [ocp-schedules-21,a7573efbcaa36461,65b5b18b5eeb10eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:35.418 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:46:35.419 ERROR 21062 --- [ocp-schedules-21,a7573efbcaa36461,65b5b18b5eeb10eb] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:46:35.419 INFO 21062 --- [ocp-schedules-21,a7573efbcaa36461,65b5b18b5eeb10eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:46:35.983 INFO 21062 --- [prometheus-minute-schedule-1,a709e1052c7c52a1,871fa90226ff7b1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:36.968 WARN 21062 --- [ocp-async-10,62bc9c102cad402a,d911c2c7508a03ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:36.993 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:36.994 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:37.013 INFO 21062 --- [ocp-async-5,8d9500d49f517121,097d20ae509f3ad2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:37.905 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,217474f2deab0508,27dd65e6e00b15de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=217474f2deab0508, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:46:37.905 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d5a616f5f91e1d0d,351e365449377d9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=d5a616f5f91e1d0d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:46:37.906 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d5a616f5f91e1d0d,351e365449377d9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=d5a616f5f91e1d0d, duration=7 ms] 2024-11-04 13:46:37.906 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,6eb7cc325378cd81,6f2f47abf2bf9054] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=6eb7cc325378cd81, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:46:37.906 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,6eb7cc325378cd81,6f2f47abf2bf9054] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=6eb7cc325378cd81, duration=5 ms] 2024-11-04 13:46:37.906 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b4f736d892be9d54,af1e4a86f5fcf823] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=b4f736d892be9d54, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:46:37.907 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b4f736d892be9d54,af1e4a86f5fcf823] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=b4f736d892be9d54, duration=5 ms] 2024-11-04 13:46:37.907 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,217474f2deab0508,27dd65e6e00b15de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=217474f2deab0508, duration=10 ms] 2024-11-04 13:46:38.503 INFO 21062 --- [ocp-schedules-22,5bc941aafcb5f058,035f7c2734b9e5f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:46:38.505 INFO 21062 --- [ocp-schedules-17,324d540d725855aa,dcad13c70ed92014] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:46:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:46:38.589510+08:00, lastProcessTime=2024-11-04T13:46:28.589944+08:00 2024-11-04 13:46:38.618 INFO 21062 --- [ocp-async-7,b54fad8c4f53b806,a698f6e0061af148] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:46:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:46:38.589510+08:00 2024-11-04 13:46:38.968 WARN 21062 --- [ocp-async-8,0e8a0831e0e42f19,b375049a25172445] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:38.983 INFO 21062 --- [prometheus-minute-schedule-1,69849a24d609602c,7a1519a466735ee5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:39.013 INFO 21062 --- [ocp-async-9,2809ae452d5ee012,65baf2e7ce4c1cff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:39.820 INFO 21062 --- [ocp-schedules-9,73dc29ba5812a517,412f137ccf2fda6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:46:39.994 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:39.994 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:40.002 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:46:40.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:46:40.968 WARN 21062 --- [ocp-async-9,eb012668fee4acfe,9fe09c1df3e3e672] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:41.013 INFO 21062 --- [ocp-async-7,4f9cf0ba0f772378,a0ba51aac415dfa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:46:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:46:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:46:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:46:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:46:41.856 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:46:41.856 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:46:41.983 INFO 21062 --- [prometheus-minute-schedule-1,331317f4aaa786df,e27ca6affe2b2124] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:42.968 WARN 21062 --- [ocp-async-7,45d1b2fc23088a64,cf965e1087d36392] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:42.995 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:42.995 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:43.014 INFO 21062 --- [ocp-async-1,577eeef9b9ef2c7d,fe6d1fd12ba8a70b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:44.968 WARN 21062 --- [ocp-async-1,bb4064d9d0d72cb9,e7a62e48afc2ae65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5ffe51306683eb13,74b33d4ff2f5a4af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:45.014 INFO 21062 --- [ocp-async-3,d0f5d5334d46f8d4,2c3e95be1de9fab8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:45.196 INFO 21062 --- [ocp-schedules-28,fd614da93b586ba2,e220d21d7360ddf2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:46:45.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:45.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:46.969 WARN 21062 --- [ocp-async-3,1852db67d6f16983,0362d245d4560b98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:47.014 INFO 21062 --- [ocp-async-10,1bd4f461e72891b6,10748d7c1d28045e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:47.983 INFO 21062 --- [prometheus-minute-schedule-1,7bba7441f2fbcf81,3cca25c4d79c19b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:46:48.590160+08:00, lastProcessTime=2024-11-04T13:46:38.589510+08:00 2024-11-04 13:46:48.618 INFO 21062 --- [ocp-async-3,18aae9d3037f129f,0216520e3596057a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:46:48.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:46:48.590160+08:00 2024-11-04 13:46:48.969 WARN 21062 --- [ocp-async-7,4b143b72570e6948,23c58501fb2a2aee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:48.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:48.997 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:49.014 INFO 21062 --- [ocp-async-1,9a03c8c966be3f8d,631fa990d07a21f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:49.822 INFO 21062 --- [ocp-schedules-30,3c244d77e66faf62,857c58b19812ca3e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:46:50.548 INFO 21062 --- [ocp-schedules-26,627554f3bb088a93,e310951454b88988] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:46:50.555 INFO 21062 --- [ocp-schedules-26,627554f3bb088a93,e310951454b88988] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:46:50.555 INFO 21062 --- [ocp-schedules-26,627554f3bb088a93,e310951454b88988] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:46:50.969 WARN 21062 --- [ocp-async-1,563428f6a902e7dd,217d1e1622936cb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ffa0915c9d1fae36,585f06efc2308c04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:51.014 INFO 21062 --- [ocp-async-3,2c3cb0388aee04f3,d74215ede6b6366c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:51.997 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:51.998 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:52.969 WARN 21062 --- [ocp-async-3,d29011ab05b2fb91,41a0e0c3bcaddbd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:53.014 INFO 21062 --- [ocp-async-10,1fd2993509e4faed,077bbc1cb0eccb42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c7d3cd9842070ba6,0d0da48b25bd87f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:54.803 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b1c5b4a0642fb7d5,de8aafc1ecc1402f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.130.47.17, traceId=b1c5b4a0642fb7d5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-11-04 13:46:54.803 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,58efa381f7eddce2,0f79614f1f20e589] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.130.47.17, traceId=58efa381f7eddce2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-11-04 13:46:54.803 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6bd68ab8ee15b51c,a49f688fad69ac8e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.130.47.17, traceId=6bd68ab8ee15b51c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-11-04 13:46:54.803 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,24fcd6e62f2800c4,57e10542c663bda8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.130.47.17, traceId=24fcd6e62f2800c4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2024-11-04 13:46:54.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6bd68ab8ee15b51c,a49f688fad69ac8e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=6bd68ab8ee15b51c, duration=12 ms] 2024-11-04 13:46:54.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,24fcd6e62f2800c4,57e10542c663bda8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.130.47.17, traceId=24fcd6e62f2800c4, duration=11 ms] 2024-11-04 13:46:54.805 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,58efa381f7eddce2,0f79614f1f20e589] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=58efa381f7eddce2, duration=14 ms] 2024-11-04 13:46:54.807 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b1c5b4a0642fb7d5,de8aafc1ecc1402f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.130.47.17, traceId=b1c5b4a0642fb7d5, duration=16 ms] 2024-11-04 13:46:54.969 WARN 21062 --- [ocp-async-10,49ee6c3cdb1a26c9,a14ffb0941908bdc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:54.998 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:54.999 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:55.015 INFO 21062 --- [ocp-async-5,9ac7e7263040335b,6e5333b16fb053fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:55.237 INFO 21062 --- [ocp-schedules-4,f968f80642fcd8cb,a68d5d50a05f5997] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:46:56.970 WARN 21062 --- [ocp-async-5,7137f91a0d972fa3,a18d9a4058104257] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:56.983 INFO 21062 --- [prometheus-minute-schedule-1,0ca6ea9bf82fba2f,4d0f2a5aaff0faaa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:46:57.015 INFO 21062 --- [ocp-async-2,5ed28687c38ec1c7,c37cdb4b14497a78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:57.713 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,9f38695fa6d2f11b,4ab7a0fb5a000e29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.130.47.17, traceId=9f38695fa6d2f11b, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2024-11-04 13:46:57.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,53f9272f48932867,3a79cb3861d443bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.130.47.17, traceId=53f9272f48932867, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2024-11-04 13:46:57.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,fe5a9e9e6af26a21,f8c1f54e08f7dec7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.130.47.17, traceId=fe5a9e9e6af26a21, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-11-04 13:46:57.715 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,93e8bd3fa0ae1db1,39c0d1559b4636c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.130.47.17, traceId=93e8bd3fa0ae1db1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,] 2024-11-04 13:46:57.722 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,9f38695fa6d2f11b,4ab7a0fb5a000e29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.130.47.17, traceId=9f38695fa6d2f11b, duration=15 ms] 2024-11-04 13:46:57.724 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,93e8bd3fa0ae1db1,39c0d1559b4636c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.130.47.17, traceId=93e8bd3fa0ae1db1, duration=15 ms] 2024-11-04 13:46:57.732 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,53f9272f48932867,3a79cb3861d443bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.130.47.17, traceId=53f9272f48932867, duration=26 ms] 2024-11-04 13:46:57.744 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,fe5a9e9e6af26a21,f8c1f54e08f7dec7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.130.47.17, traceId=fe5a9e9e6af26a21, duration=34 ms] 2024-11-04 13:46:57.746 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,c67d0289089690b7,7aa5d5fee8733c29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.130.47.17, traceId=c67d0289089690b7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2024-11-04 13:46:57.781 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,c67d0289089690b7,7aa5d5fee8733c29] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 13:46:57.861 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,ff16031fd7fdcde7,3965f243559f4d87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2024-11-04T12%3A46%3A57%2B08%3A00&endTime=2024-11-04T13%3A46%3A57%2B08%3A00&maxPoints=360, client=10.130.47.17, traceId=ff16031fd7fdcde7, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-11-04T12:46:57+08:00,2024-11-04T13:46:57+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2024-11-04 13:46:57.869 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,c67d0289089690b7,7aa5d5fee8733c29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.130.47.17, traceId=c67d0289089690b7, duration=138 ms] 2024-11-04 13:46:57.921 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-10,ff16031fd7fdcde7,3965f243559f4d87] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2024-11-04 13:46:57.922 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-10,ff16031fd7fdcde7,3965f243559f4d87] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1730695620 2024-11-04 13:46:57.926 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,ff16031fd7fdcde7,3965f243559f4d87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.130.47.17, traceId=ff16031fd7fdcde7, duration=78 ms] 2024-11-04 13:46:57.999 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:46:58.000 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:46:58.499 INFO 21062 --- [ocp-schedules-18,efc7ed7b2f60d6dc,fa6029b9b3a73d1c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:46:58.500 INFO 21062 --- [ocp-schedules-14,a18ef83a0cb756c5,9abd7e9dc3cc2746] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:46:58.503 INFO 21062 --- [ocp-schedules-25,30454448c05a5b33,cd8d85823bbba612] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:46:58.503 INFO 21062 --- [ocp-schedules-18,efc7ed7b2f60d6dc,fa6029b9b3a73d1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:46:58.505 INFO 21062 --- [ocp-schedules-14,a18ef83a0cb756c5,9abd7e9dc3cc2746] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:58.505 INFO 21062 --- [ocp-schedules-20,a9a0bb7e81fab1b7,2eb8f511e2a939ef] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:46:58.510 INFO 21062 --- [ocp-schedules-17,0283de9f98092f14,7cdad715c1688747] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:46:58.510 INFO 21062 --- [ocp-schedules-17,0283de9f98092f14,7cdad715c1688747] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:46:58.510 INFO 21062 --- [ocp-schedules-11,2d325ce3087e2935,475d5f296c1d8415] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:46:58.510 INFO 21062 --- [ocp-schedules-11,2d325ce3087e2935,475d5f296c1d8415] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:46:58.510 INFO 21062 --- [ocp-schedules-6,0c3202d548f7649d,e1799e915c295c14] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:46:58.510 INFO 21062 --- [ocp-schedules-6,0c3202d548f7649d,e1799e915c295c14] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:46:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:46:58.589078+08:00, lastProcessTime=2024-11-04T13:46:48.590160+08:00 2024-11-04 13:46:58.619 INFO 21062 --- [ocp-async-5,790f5d1e042788b8,3dc506af78deb5ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:46:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:46:58.589078+08:00 2024-11-04 13:46:58.863 INFO 21062 --- [ocp-schedules-3,0a1eb3c29b880977,b99e44f25870d5bb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:46:58.866 INFO 21062 --- [ocp-schedules-3,0a1eb3c29b880977,b99e44f25870d5bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:46:58.866 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:46:58.961 INFO 21062 --- [ocp-schedules-26,e735ec3493e12f47,337eda38d06a9a67] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:46:58.970 WARN 21062 --- [ocp-async-3,ff838a9dd442ee40,e4ba9e00597e79cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:46:59.015 INFO 21062 --- [ocp-async-10,eb443e3b8d1fc5d6,943a525f97142700] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:46:59.825 INFO 21062 --- [ocp-schedules-15,5987ae64938993a0,2752b9328e26a63b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:46:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d540cbebba8190a5,2f212a46cb503f27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699220, interval=60 2024-11-04 13:47:00.970 WARN 21062 --- [ocp-async-10,0ae6ece577af9643,ca25576cb63a0817] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:01.000 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:01.001 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:01.015 INFO 21062 --- [ocp-async-6,1c0d6b9685fcb9f3,a30c7a3a0afbacf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:02.970 WARN 21062 --- [ocp-async-6,7b755767961bbfa0,62f9a657b31c073d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:02.983 INFO 21062 --- [prometheus-minute-schedule-1,fe1e4f83841f251e,f123dfc4eb04c94f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:03.015 INFO 21062 --- [ocp-async-8,777d539fafbb82c4,e066371618591626] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:03.194 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6d102154717c1127,9374b3f7576d38a7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=6d102154717c1127, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 13:47:03.197 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,265ca6a8cd2dfefa,4fcfdad943345984] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.130.47.17, traceId=265ca6a8cd2dfefa, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 13:47:03.201 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6d102154717c1127,9374b3f7576d38a7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=6d102154717c1127, duration=12 ms] 2024-11-04 13:47:03.201 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,2a2f7d092d3ebb6d,d36b5cddadaf4d94] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1&status=RUNNING%2CFAILED, client=10.130.47.17, traceId=2a2f7d092d3ebb6d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1,] 2024-11-04 13:47:03.204 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,2270f7323d120f4e,ed7434130eb18583] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=2270f7323d120f4e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 13:47:03.215 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,265ca6a8cd2dfefa,4fcfdad943345984] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=265ca6a8cd2dfefa, duration=24 ms] 2024-11-04 13:47:03.235 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,2a2f7d092d3ebb6d,d36b5cddadaf4d94] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.130.47.17, traceId=2a2f7d092d3ebb6d, duration=44 ms] 2024-11-04 13:47:03.253 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,2270f7323d120f4e,ed7434130eb18583] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 13:47:03.267 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,a48d06f1920c2c04,c8addbdf7849d6eb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=a48d06f1920c2c04, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 13:47:03.271 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,a48d06f1920c2c04,c8addbdf7849d6eb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=a48d06f1920c2c04, duration=9 ms] 2024-11-04 13:47:03.276 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,790e60c55968e66d,40884f83477fcfc7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=790e60c55968e66d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 13:47:03.279 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,41d95d9e1e3222a5,c3699af8fc3c1540] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=41d95d9e1e3222a5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 13:47:03.279 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,41d95d9e1e3222a5,c3699af8fc3c1540] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=41d95d9e1e3222a5, duration=4 ms] 2024-11-04 13:47:03.403 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,2270f7323d120f4e,ed7434130eb18583] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=2270f7323d120f4e, duration=211 ms] 2024-11-04 13:47:03.462 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,790e60c55968e66d,40884f83477fcfc7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=790e60c55968e66d, duration=193 ms] 2024-11-04 13:47:03.602 INFO 21062 --- [ocp-schedules-12,5c5deb5cf60fb80b,65720ca72075b995] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:47:03.603 INFO 21062 --- [ocp-schedules-12,59155c803d2e21a3,4b3a760594955335] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:47:03.606 INFO 21062 --- [ocp-schedules-12,59155c803d2e21a3,4b3a760594955335] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:03.635 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:47:03.635 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:47:03.747 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,79b6a70ab48f6023,b0be6c07daffe2dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/eventMeta?id=1, client=10.130.47.17, traceId=79b6a70ab48f6023, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1,] 2024-11-04 13:47:03.753 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,79b6a70ab48f6023,b0be6c07daffe2dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/eventMeta client=10.130.47.17, traceId=79b6a70ab48f6023, duration=14 ms] 2024-11-04 13:47:03.796 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,a78b27938c34d70c,d476a48e60a42438] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=a78b27938c34d70c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 13:47:03.797 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,4e4744d00a39235b,8750afd9ba50890a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topSql/switch?id=1, client=10.130.47.17, traceId=4e4744d00a39235b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1,] 2024-11-04 13:47:03.797 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c2874023e84ea9cd,277b56126003149a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/stats?id=1, client=10.130.47.17, traceId=c2874023e84ea9cd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1,] 2024-11-04 13:47:03.799 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1c9ed80b41ff31d6,8df1de29008c463d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/events?startTime=2024-10-28T13%3A47%3A03%2B08%3A00&endTime=2024-11-04T13%3A47%3A03%2B08%3A00&page=1&size=5, client=10.130.47.17, traceId=1c9ed80b41ff31d6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1,2024-10-28T13:47:03+08:00,2024-11-04T13:47:03+08:00,Page request [number: 0, size 5, sort: timestamp: DESC],] 2024-11-04 13:47:03.822 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c2874023e84ea9cd,277b56126003149a] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 13:47:03.827 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c2874023e84ea9cd,277b56126003149a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/stats client=10.130.47.17, traceId=c2874023e84ea9cd, duration=37 ms] 2024-11-04 13:47:03.831 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1c9ed80b41ff31d6,8df1de29008c463d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/events client=10.130.47.17, traceId=1c9ed80b41ff31d6, duration=39 ms] 2024-11-04 13:47:03.841 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,a78b27938c34d70c,d476a48e60a42438] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 13:47:03.891 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,4e4744d00a39235b,8750afd9ba50890a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topSql/switch client=10.130.47.17, traceId=4e4744d00a39235b, duration=103 ms] 2024-11-04 13:47:03.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,a78b27938c34d70c,d476a48e60a42438] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=a78b27938c34d70c, duration=106 ms] 2024-11-04 13:47:03.990 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,3f52935f51217637,d62ebe7ae1f111bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=3f52935f51217637, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 13:47:03.995 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,3f52935f51217637,d62ebe7ae1f111bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=3f52935f51217637, duration=10 ms] 2024-11-04 13:47:04.001 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:04.001 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:04.363 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,45f130019571556f,7a1d83bb642c2e4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/alarmSummaries?id=1, client=10.130.47.17, traceId=45f130019571556f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1,] 2024-11-04 13:47:04.363 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2b1a26c02efbeaeb,276d3d8d42373077] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/serverStats?id=1, client=10.130.47.17, traceId=2b1a26c02efbeaeb, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1,] 2024-11-04 13:47:04.363 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,128516af049627bb,14fd7fcb33830bb1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.130.47.17, traceId=128516af049627bb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 13:47:04.381 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,128516af049627bb,14fd7fcb33830bb1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=128516af049627bb, duration=28 ms] 2024-11-04 13:47:04.391 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2b1a26c02efbeaeb,276d3d8d42373077] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 13:47:04.391 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2b1a26c02efbeaeb,276d3d8d42373077] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/serverStats client=10.130.47.17, traceId=2b1a26c02efbeaeb, duration=38 ms] 2024-11-04 13:47:04.393 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,45f130019571556f,7a1d83bb642c2e4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/alarmSummaries client=10.130.47.17, traceId=45f130019571556f, duration=37 ms] 2024-11-04 13:47:04.970 WARN 21062 --- [ocp-async-8,87cdafd49b04d3d2,2d711e8179c981e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:05.016 INFO 21062 --- [ocp-async-9,b90c87f3cf99a3f2,9523340ce7b48a6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:05.026 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c15e4b8c91d28b7c,9d91d2ecddf88cb9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=c15e4b8c91d28b7c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 13:47:05.026 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c15e4b8c91d28b7c,9d91d2ecddf88cb9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=c15e4b8c91d28b7c, duration=6 ms] 2024-11-04 13:47:05.029 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,0959c541704e8f3a,a5000dbd6ee9b2f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=0959c541704e8f3a, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 13:47:05.029 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,f2ebfee2ae5dbd98,8710cc4699fbe5e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=f2ebfee2ae5dbd98, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 13:47:05.029 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,0959c541704e8f3a,a5000dbd6ee9b2f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=0959c541704e8f3a, duration=5 ms] 2024-11-04 13:47:05.029 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,f2ebfee2ae5dbd98,8710cc4699fbe5e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=f2ebfee2ae5dbd98, duration=6 ms] 2024-11-04 13:47:05.205 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:47:05.205 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-8,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:47:05.205 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:47:05.205 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:47:05.205 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:47:05.205 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:47:05.262 INFO 21062 --- [ocp-schedules-9,9403ada96e2b69cb,4db75f44b419fab6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:47:05.983 INFO 21062 --- [prometheus-minute-schedule-1,53b7a80baa212648,4d4dbebad15210fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:06.971 WARN 21062 --- [ocp-async-4,bc6b87b3918f210e,00892d81e540525d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:07.002 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:07.002 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:07.016 INFO 21062 --- [ocp-async-7,0401e6344484a341,6e8ab014f17755d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:08.504 INFO 21062 --- [ocp-schedules-18,efc7ed7b2f60d6dc,fa6029b9b3a73d1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:47:08.505 INFO 21062 --- [ocp-schedules-14,a18ef83a0cb756c5,9abd7e9dc3cc2746] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:47:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:47:08.590785+08:00, lastProcessTime=2024-11-04T13:46:58.589078+08:00 2024-11-04 13:47:08.619 INFO 21062 --- [ocp-async-9,28b7d45990ad67d8,e348a03546ed3d74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:47:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:47:08.590785+08:00 2024-11-04 13:47:08.867 INFO 21062 --- [ocp-schedules-3,0a1eb3c29b880977,b99e44f25870d5bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:47:08.971 WARN 21062 --- [ocp-async-6,63d718f2dbefccf7,2b73fcca2c6e6634] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:08.983 INFO 21062 --- [prometheus-minute-schedule-1,e9414d1eefae332a,9ce5938e6769c93d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:09.016 INFO 21062 --- [ocp-async-2,882beb78a5cda9df,f0b7062171c6a715] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:09.731 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?backUrl=/task, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:47:09.731 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/task/2, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:47:09.731 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:47:09.828 INFO 21062 --- [ocp-schedules-26,58b0cb80cdf4f0ac,4b83b2815fa48e65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:47:10.003 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:47:10.004 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:10.004 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:47:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:47:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:47:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:47:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:47:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:47:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:47:10.431 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,58acb42c56e3d2ea,dfc809330971cc80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /favicon.ico?null, client=10.130.47.17, traceId=58acb42c56e3d2ea, method=void com.oceanbase.ocp.server.common.controller.web.WebController.dummyFavicon(), args=] 2024-11-04 13:47:10.431 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,58acb42c56e3d2ea,dfc809330971cc80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /favicon.ico client=10.130.47.17, traceId=58acb42c56e3d2ea, duration=5 ms] 2024-11-04 13:47:10.557 INFO 21062 --- [ocp-schedules-7,cb599e0d03a6bb2f,c5d16be6f0f37a28] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:47:10.562 INFO 21062 --- [ocp-schedules-7,cb599e0d03a6bb2f,c5d16be6f0f37a28] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:47:10.562 INFO 21062 --- [ocp-schedules-7,cb599e0d03a6bb2f,c5d16be6f0f37a28] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:47:10.663 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,134d729d777b890b,3048d72044868bce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.130.47.17, traceId=134d729d777b890b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2024-11-04 13:47:10.665 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,134d729d777b890b,3048d72044868bce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.130.47.17, traceId=134d729d777b890b, duration=10 ms] 2024-11-04 13:47:10.824 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,b50c18ce2b09fdbc,82fe3b36072e3843] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=b50c18ce2b09fdbc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 13:47:10.824 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,da2dd5f65e1c890d,51b61816e6866c48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=da2dd5f65e1c890d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 13:47:10.824 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,b50c18ce2b09fdbc,82fe3b36072e3843] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=b50c18ce2b09fdbc, duration=6 ms] 2024-11-04 13:47:10.830 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,da2dd5f65e1c890d,51b61816e6866c48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=da2dd5f65e1c890d, duration=12 ms] 2024-11-04 13:47:10.971 WARN 21062 --- [ocp-async-8,4ecf80435f5870c2,246e08f12e859887] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:10.972 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e66b6a1ad5c4dabe,e1edb98c066b3ab0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e66b6a1ad5c4dabe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 13:47:11.016 INFO 21062 --- [ocp-async-9,1d4c438b1af24830,a834107e11087a3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:11.050 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,523a53626ef7b438,d29065e1a604ece8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=523a53626ef7b438, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 13:47:11.055 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,523a53626ef7b438,d29065e1a604ece8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=523a53626ef7b438, duration=10 ms] 2024-11-04 13:47:11.061 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e66b6a1ad5c4dabe,e1edb98c066b3ab0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e66b6a1ad5c4dabe, duration=99 ms] 2024-11-04 13:47:11.191 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,bec0511c3aa1fa61,77d14c2f860ba8fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=bec0511c3aa1fa61, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 13:47:11.341 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=1, affectRows=1 2024-11-04 13:47:11.341 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=1, validCount=1 2024-11-04 13:47:11.341 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=4, affectRows=4 2024-11-04 13:47:11.341 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=4, validCount=4 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=8, affectRows=8 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=8, validCount=8 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=14, affectRows=14 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=20, affectRows=20 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=48, affectRows=48 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=14, validCount=14 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=30, affectRows=30 2024-11-04 13:47:11.342 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=19, validCount=19 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=30, validCount=30 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=35, affectRows=35 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=46, affectRows=46 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=54, affectRows=54 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=48, validCount=48 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=35, validCount=35 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=53, affectRows=53 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=46, validCount=46 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699160, nodeCount=57, affectRows=57 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=54, validCount=54 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=51, validCount=51 2024-11-04 13:47:11.343 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699160, nodeCount=57, validCount=57 2024-11-04 13:47:11.371 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,bec0511c3aa1fa61,77d14c2f860ba8fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=bec0511c3aa1fa61, duration=189 ms] 2024-11-04 13:47:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8b01593611ef6b20,e3078775bb925ab7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:12.724 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,7c70cdbc70a943a2,f5ddd83240384446] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments?taskInstanceId=2&subtaskInstanceId=2, client=10.130.47.17, traceId=7c70cdbc70a943a2, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,2,] 2024-11-04 13:47:12.759 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,7c70cdbc70a943a2,f5ddd83240384446] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments client=10.130.47.17, traceId=7c70cdbc70a943a2, duration=43 ms] 2024-11-04 13:47:12.971 WARN 21062 --- [ocp-async-4,9b7a0a9c99e3256d,3f70c827506fd0a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:13.004 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:13.005 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:13.016 INFO 21062 --- [ocp-async-7,410720b5635d3d59,c742a906c8aae1a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:13.368 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,4eadcd9d263efbe7,2afd601d29f8eb75] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=4eadcd9d263efbe7, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 13:47:13.368 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,4eadcd9d263efbe7,2afd601d29f8eb75] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=4eadcd9d263efbe7, duration=4 ms] 2024-11-04 13:47:13.368 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,5ded568211b5556b,8e75888ade0e6a02] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=5ded568211b5556b, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 13:47:13.369 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,5ded568211b5556b,8e75888ade0e6a02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=5ded568211b5556b, duration=5 ms] 2024-11-04 13:47:13.369 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,32d6fdb6ff4e1a24,eb0111617abd94ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=32d6fdb6ff4e1a24, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 13:47:13.370 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,32d6fdb6ff4e1a24,eb0111617abd94ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=32d6fdb6ff4e1a24, duration=7 ms] 2024-11-04 13:47:13.381 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:47:13.381 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:47:13.381 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:47:13.383 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 13:47:13.383 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 13:47:13.383 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 13:47:13.984 INFO 21062 --- [prometheus-cache-monitor1,95dea66ad8459abe,7641caae577bab1e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1462,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":370,"secondWriteQueue":0,"minuteWriteCacheSize":367} 2024-11-04 13:47:13.985 INFO 21062 --- [prometheus-cache-monitor1,95dea66ad8459abe,7641caae577bab1e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=40, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=40, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:47:14.713 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:43:14.693, end=2024-11-04T13:45:14.711} 2024-11-04 13:47:14.722 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:47:14.722 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:47:14.971 WARN 21062 --- [ocp-async-5,ffe5e5f64b25fce8,25b3966d2618f11a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d88512858546712a,5f72da0ca41d4ed1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:15.016 INFO 21062 --- [ocp-async-1,6212dab78bcc48b2,ed4d46cf33881eca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:15.302 INFO 21062 --- [ocp-schedules-2,33ecc607839354c2,8327ac88a1609282] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:47:15.673 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,b32b2d259de9bb49,ad8ecaa23dbac6c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=b32b2d259de9bb49, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 13:47:15.673 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,b32b2d259de9bb49,ad8ecaa23dbac6c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=b32b2d259de9bb49, duration=5 ms] 2024-11-04 13:47:16.005 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:16.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:16.972 WARN 21062 --- [ocp-async-1,2184cd476d0d15ef,fd6af54c05c9a145] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:17.017 INFO 21062 --- [ocp-async-3,7b0638dbf87c54a8,950d7690a448edc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:17.983 INFO 21062 --- [prometheus-minute-schedule-1,cb3a2f36ff81acc7,209238124c0cf4a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:47:18.590123+08:00, lastProcessTime=2024-11-04T13:47:08.590785+08:00 2024-11-04 13:47:18.619 INFO 21062 --- [ocp-async-1,3f0f3d219d358545,e1e2381d4b41f711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:47:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:47:18.590123+08:00 2024-11-04 13:47:18.972 WARN 21062 --- [ocp-async-9,d7b09a3af4d4c4f5,208ab120a13c4983] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:19.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:19.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:19.017 INFO 21062 --- [ocp-async-7,18bc3d38e71aeb14,80defa03002657d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:19.831 INFO 21062 --- [ocp-schedules-23,62ef1a1de82ccc7b,b1f4dad193e9b7e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:47:20.972 WARN 21062 --- [ocp-async-7,852684d0c811673d,6249f8bbb892fc7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:20.983 INFO 21062 --- [prometheus-minute-schedule-1,47174c07571b3b84,145ff9aa5711eed4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:21.017 INFO 21062 --- [ocp-async-1,2a925840f71d0a55,c574e467feb0a5ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:22.007 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:22.007 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:22.972 WARN 21062 --- [ocp-async-1,94723ba0aa5c3b00,ed64c898dc090e20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:23.017 INFO 21062 --- [ocp-async-3,0862c09d36559023,1810d7706bc46507] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b2c19d8b4e859e49,62a55bcdc2fb8ca7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:24.972 WARN 21062 --- [ocp-async-3,e03d4f582093543b,046373b2e315255c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:25.007 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:25.008 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:25.017 INFO 21062 --- [ocp-async-10,0bb1e15e249cea62,ce6c6847e95849be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:25.328 INFO 21062 --- [ocp-schedules-4,48a7edba8e191c72,772a81e574722aa6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:47:26.973 WARN 21062 --- [ocp-async-10,91feb7155397a868,fffe8b3e34ac46b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:26.983 INFO 21062 --- [prometheus-minute-schedule-1,5dc1234b9aec4803,3b5890281ad967dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:26.984 INFO 21062 --- [prometheus-hour-schedule-1,00d17079d3f60cba,43ac37600e523bc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:47:27.017 INFO 21062 --- [ocp-async-5,6c48dc8deb333a28,12b305d5be6e59a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:28.009 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:28.010 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:28.498 INFO 21062 --- [ocp-schedules-27,44b16b5aeb20a8f5,ed0827f48da06996] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:47:28.499 INFO 21062 --- [ocp-schedules-9,029fd6e093898aa9,b27c0c423df88d78] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:47:28.500 INFO 21062 --- [ocp-schedules-3,ce5400aa87922ffd,535d804fc89a0cf5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:47:28.500 INFO 21062 --- [ocp-schedules-8,fb3a4159218e2d7f,870e1aaf3b9fd5f3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:47:28.502 INFO 21062 --- [ocp-schedules-20,51f206a86cf179ef,7d2a383c640a7901] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:47:28.503 INFO 21062 --- [ocp-schedules-27,44b16b5aeb20a8f5,ed0827f48da06996] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:47:28.503 INFO 21062 --- [ocp-schedules-3,ce5400aa87922ffd,535d804fc89a0cf5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:28.504 INFO 21062 --- [ocp-schedules-8,fb3a4159218e2d7f,870e1aaf3b9fd5f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:47:28.504 INFO 21062 --- [ocp-schedules-21,51e02346295fbf98,b9e6700dabf8cc8c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:47:28.505 INFO 21062 --- [ocp-schedules-9,029fd6e093898aa9,b27c0c423df88d78] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:47:28.508 INFO 21062 --- [ocp-schedules-13,79bc4446a9e9c482,682eedb8d574692e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:47:28.511 INFO 21062 --- [ocp-schedules-18,0ad5915120d52b59,83010124f78b7cb1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:47:28.511 INFO 21062 --- [ocp-schedules-2,2015408f4b4168fb,ea56a04f0cd018dd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:47:28.511 INFO 21062 --- [ocp-schedules-18,0ad5915120d52b59,83010124f78b7cb1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:47:28.511 INFO 21062 --- [ocp-schedules-2,2015408f4b4168fb,ea56a04f0cd018dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:47:28.511 INFO 21062 --- [ocp-schedules-16,f990abd8f54df852,3ae84e6d08ae1322] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:47:28.511 INFO 21062 --- [ocp-schedules-16,f990abd8f54df852,3ae84e6d08ae1322] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:47:28.511 INFO 21062 --- [ocp-schedules-13,79bc4446a9e9c482,682eedb8d574692e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:28.532 INFO 21062 --- [ocp-schedules-11,43561d2008584b3d,6e415de7e26d0923] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:47:28.532 INFO 21062 --- [ocp-schedules-11,43561d2008584b3d,6e415de7e26d0923] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:47:28.577 INFO 21062 --- [pool-12-thread-13,f864f668ef3a57c1,bafd72879bb8193d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:47:28.577 INFO 21062 --- [pool-12-thread-14,f864f668ef3a57c1,85f609e41288a265] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:47:28.578 INFO 21062 --- [pool-12-thread-15,f864f668ef3a57c1,13b6c68514a7c71b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:47:28.578 INFO 21062 --- [pool-12-thread-16,f864f668ef3a57c1,30ca31bb1ca2a21b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:47:28.578 INFO 21062 --- [pool-12-thread-16,f864f668ef3a57c1,30ca31bb1ca2a21b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:47:28.582 INFO 21062 --- [ForkJoinPool-11-worker-6,f864f668ef3a57c1,b39f3fb694890731] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:47:28.582 INFO 21062 --- [ForkJoinPool-11-worker-6,f864f668ef3a57c1,b39f3fb694890731] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:47:28.582 INFO 21062 --- [ForkJoinPool-11-worker-6,f864f668ef3a57c1,b39f3fb694890731] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:47:28.583 INFO 21062 --- [pool-12-thread-14,f864f668ef3a57c1,85f609e41288a265] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:47:28.583 INFO 21062 --- [ForkJoinPool-11-worker-15,f864f668ef3a57c1,04131817cec1849c] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:47:28.583 INFO 21062 --- [pool-12-thread-15,f864f668ef3a57c1,13b6c68514a7c71b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:47:28.583 INFO 21062 --- [pool-12-thread-13,f864f668ef3a57c1,bafd72879bb8193d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:47:28.583 INFO 21062 --- [pool-12-thread-13,f864f668ef3a57c1,bafd72879bb8193d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:47:28.583 INFO 21062 --- [pool-12-thread-16,f864f668ef3a57c1,30ca31bb1ca2a21b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:47:28.587 INFO 21062 --- [pool-12-thread-16,f864f668ef3a57c1,30ca31bb1ca2a21b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:47:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:47:28.588679+08:00, lastProcessTime=2024-11-04T13:47:18.590123+08:00 2024-11-04 13:47:28.619 INFO 21062 --- [ocp-async-6,7cb568fcd392537d,8dd885cda84e62f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:47:28.741 INFO 21062 --- [ocp-schedules-28,20233a6c75c86e61,97707bfd4cabbb33] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:47:28.753 INFO 21062 --- [ocp-schedules-9,90c9a8e289acca78,d8a85191f9ef20a8] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:47:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:47:28.588679+08:00 2024-11-04 13:47:28.767 INFO 21062 --- [ocp-schedules-9,90c9a8e289acca78,d8a85191f9ef20a8] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:47:28.930 INFO 21062 --- [ocp-schedules-2,af861d4e996116fc,80059d5524532b49] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:47:28.930 INFO 21062 --- [ocp-schedules-2,af861d4e996116fc,80059d5524532b49] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:47:28.963 INFO 21062 --- [ocp-schedules-16,1c8a6a779c1bfd84,4abffd8689ed666f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:47:28.973 WARN 21062 --- [ocp-async-2,91f1cabdab443559,5df8520add7a232f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:28.997 INFO 21062 --- [ocp-schedules-14,63d90154f387f632,3f2214d87f12e524] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:47:29.003 INFO 21062 --- [ocp-schedules-14,63d90154f387f632,3f2214d87f12e524] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:47:29.003 INFO 21062 --- [ocp-schedules-14,63d90154f387f632,3f2214d87f12e524] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:47:29.018 INFO 21062 --- [ocp-async-10,6043d4f7d7f04c3a,3b39da79c63ebc98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:29.253 INFO 21062 --- [ocp-schedules-6,b31595b08df33a44,32229d3940985fe4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:47:29.253 INFO 21062 --- [ocp-schedules-6,b31595b08df33a44,32229d3940985fe4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:47:29.254 INFO 21062 --- [ocp-schedules-6,b31595b08df33a44,32229d3940985fe4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:47:29.254 INFO 21062 --- [ocp-schedules-6,b31595b08df33a44,32229d3940985fe4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:47:29.255 INFO 21062 --- [ocp-schedules-6,b31595b08df33a44,32229d3940985fe4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:47:29.255 INFO 21062 --- [ocp-schedules-6,b31595b08df33a44,32229d3940985fe4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709925194352054272 2024-11-04 13:47:29.257 INFO 21062 --- [ocp-schedules-6,b31595b08df33a44,32229d3940985fe4] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=28, maxSyncedRecordId=709926704322465792 2024-11-04 13:47:29.354 INFO 21062 --- [ocp-schedules-30,166e687e1e566a4f,87106259c9a4b910] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:47:29.354 INFO 21062 --- [ocp-schedules-30,166e687e1e566a4f,87106259c9a4b910] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:47:29.382 INFO 21062 --- [pool-34-thread-1,166e687e1e566a4f,766c4cf37503a1cf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:47:29.410 WARN 21062 --- [ocp-schedules-30,166e687e1e566a4f,87106259c9a4b910] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:47:29.410 INFO 21062 --- [ocp-schedules-30,166e687e1e566a4f,87106259c9a4b910] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:47:29.834 INFO 21062 --- [ocp-schedules-20,c745de695b04fea8,1418f293c996aece] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:47:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3c4a90e5d552ffb1,db37525250f85224] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:47:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:47:30.566 INFO 21062 --- [ocp-schedules-17,aab46f736bfc952e,d1c8458249c1fbc2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:47:30.572 INFO 21062 --- [ocp-schedules-19,e9411dda0b18a282,fdcd9667a1ec76f4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:47:30.572 INFO 21062 --- [ocp-schedules-19,e9411dda0b18a282,fdcd9667a1ec76f4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:47:30.573 INFO 21062 --- [ocp-schedules-17,aab46f736bfc952e,d1c8458249c1fbc2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:47:30.573 INFO 21062 --- [ocp-schedules-17,aab46f736bfc952e,d1c8458249c1fbc2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:47:30.599 INFO 21062 --- [ocp-schedules-19,e9411dda0b18a282,fdcd9667a1ec76f4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:47:30.599 INFO 21062 --- [ocp-schedules-19,e9411dda0b18a282,fdcd9667a1ec76f4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:47:30.600 INFO 21062 --- [ocp-schedules-26,de699dec09448538,6aa28893483e23f2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:47:30.740 INFO 21062 --- [ocp-schedules-9,d86027aa8285fc50,229342251761609e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:47:30.740+08:00, clusters=[1] 2024-11-04 13:47:30.742 INFO 21062 --- [ocp-schedules-9,d86027aa8285fc50,229342251761609e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:47:30.791 INFO 21062 --- [ocp-schedules-10,a288409b2af3c3a4,dbb0223da7726b23] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:47:30.791 INFO 21062 --- [ocp-schedules-10,a288409b2af3c3a4,dbb0223da7726b23] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:47:30.791 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:47:30.791 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:47:30.802 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:47:30.802 INFO 21062 --- [ocp-schedules-10,a288409b2af3c3a4,dbb0223da7726b23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:47:30.802 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:47:30.804 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:47:30.804 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:47:30.804 WARN 21062 --- [ocp-schedules-10,a288409b2af3c3a4,dbb0223da7726b23] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:47:30.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:47:30.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:47:30.804 INFO 21062 --- [ocp-schedules-10,a288409b2af3c3a4,dbb0223da7726b23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:47:30.806 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:47:30.806 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:47:30.806 WARN 21062 --- [ocp-schedules-10,a288409b2af3c3a4,dbb0223da7726b23] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:47:30.812 ERROR 21062 --- [ocp-schedules-10,a288409b2af3c3a4,dbb0223da7726b23] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:47:30.886 INFO 21062 --- [ocp-schedules-25,1a2e24191d7b8cc5,87117b263a81e504] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:47:30.886+08:00, clusters=[1] 2024-11-04 13:47:30.889 INFO 21062 --- [ocp-schedules-25,1a2e24191d7b8cc5,87117b263a81e504] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:47:30.973 WARN 21062 --- [ocp-async-10,23ff1b40921efd9c,82b64fce48a7bb65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:31.010 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:31.010 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:31.018 INFO 21062 --- [ocp-async-6,e5494eb70fee8492,3d9d58c2d2b722cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:32.973 WARN 21062 --- [ocp-async-6,46d1bb1dbffa84c2,d27eb696daf618ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:32.983 INFO 21062 --- [prometheus-minute-schedule-1,476b35761e805a77,121b5f6a651cb892] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:33.018 INFO 21062 --- [ocp-async-8,52ca10402a656e23,b3c8e84d0b8b6059] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:33.635 INFO 21062 --- [ocp-schedules-12,59155c803d2e21a3,4b3a760594955335] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:47:33.635 INFO 21062 --- [ocp-schedules-12,9afa89f3d327da01,4fb18c8bfb4760cb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:47:33.640 INFO 21062 --- [ocp-schedules-12,9afa89f3d327da01,4fb18c8bfb4760cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:33.668 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:47:33.668 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:47:34.011 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:34.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:34.973 WARN 21062 --- [ocp-async-8,bdc8fbf596ea50c0,dc3daba197520e12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:35.018 INFO 21062 --- [ocp-async-9,9ca02dca768f499f,ee52fea8a2945fc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:35.374 INFO 21062 --- [ocp-schedules-1,3a8832fad6abce8a,7a19f1b88d5505d0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:47:35.419 INFO 21062 --- [ocp-schedules-23,bb9e4e7686e08a7e,d71bf7f55042b518] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:47:35.427 INFO 21062 --- [ocp-schedules-23,bb9e4e7686e08a7e,d71bf7f55042b518] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:35.512 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:47:35.514 ERROR 21062 --- [ocp-schedules-23,bb9e4e7686e08a7e,d71bf7f55042b518] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:47:35.514 INFO 21062 --- [ocp-schedules-23,bb9e4e7686e08a7e,d71bf7f55042b518] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:47:35.983 INFO 21062 --- [prometheus-minute-schedule-1,d11d774cb259734b,79a351333b64bdc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:36.973 WARN 21062 --- [ocp-async-9,e9ce75cb09cd899d,8eea851da80d8407] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:37.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:37.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:37.018 INFO 21062 --- [ocp-async-7,39ab7f2380f2ca0d,92b6b525d823f95a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:38.504 INFO 21062 --- [ocp-schedules-8,fb3a4159218e2d7f,870e1aaf3b9fd5f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:47:38.504 INFO 21062 --- [ocp-schedules-3,ce5400aa87922ffd,535d804fc89a0cf5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:47:38.512 INFO 21062 --- [ocp-schedules-13,79bc4446a9e9c482,682eedb8d574692e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:47:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:47:38.590136+08:00, lastProcessTime=2024-11-04T13:47:28.588679+08:00 2024-11-04 13:47:38.619 INFO 21062 --- [ocp-async-10,d063ebfeb15d6b21,46aa4a6be07b77e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:47:38.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:47:38.590136+08:00 2024-11-04 13:47:38.974 WARN 21062 --- [ocp-async-5,f5dc5680af478d96,e7a025c6372fc320] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:38.983 INFO 21062 --- [prometheus-minute-schedule-1,8f903e9ea3c80d5e,bc657cacc291e638] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:39.019 INFO 21062 --- [ocp-async-2,366277672ee242f4,8dad4cb1566fd3ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:39.837 INFO 21062 --- [ocp-schedules-19,579d17c8aca3887d,59bc9db6cc33c9c3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:47:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:47:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:47:40.013 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:40.015 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:40.974 WARN 21062 --- [ocp-async-2,3ce02c80d5efcf92,a2418a8c6887213b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:41.019 INFO 21062 --- [ocp-async-10,d990ca21e4680c66,76d968b125c4c818] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a98412fd59385438,23fd1082873a9579] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:42.974 WARN 21062 --- [ocp-async-10,fe1661e8087465f6,e8976970a9371b41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:43.015 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:43.016 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:43.019 INFO 21062 --- [ocp-async-6,cedfbd239ffd7c4d,b09fea8175d9d025] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:44.974 WARN 21062 --- [ocp-async-6,c9e622426e482af8,432b725ba7a81154] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d9495d0f90ea7c7b,35fa46d3fcd389db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:45.019 INFO 21062 --- [ocp-async-1,d5ed9548684c834a,665c42e7a21363ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:45.399 INFO 21062 --- [ocp-schedules-26,daba7c7f548e0e2a,7902e92442072169] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:47:46.016 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:46.017 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:46.975 WARN 21062 --- [ocp-async-1,51679cf1205c35a0,2515be28ab4ef298] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:47.019 INFO 21062 --- [ocp-async-3,4503927f9105592d,2b8e96f3339cbb3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:47.983 INFO 21062 --- [prometheus-minute-schedule-1,13012af56565daca,8178fc4765d153ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:47:48.590805+08:00, lastProcessTime=2024-11-04T13:47:38.590136+08:00 2024-11-04 13:47:48.619 INFO 21062 --- [ocp-async-8,e76297133f7d3623,42a9c995668d8f86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:47:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:47:48.590805+08:00 2024-11-04 13:47:48.975 WARN 21062 --- [ocp-async-10,536a42857b9e7d8c,742485eba64ded16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:49.017 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:49.018 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:49.019 INFO 21062 --- [ocp-async-7,aaaf576c659c651c,667f63df78d76a04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:49.840 INFO 21062 --- [ocp-schedules-21,54bc2cf1d49db8e1,7eee20489d6b6451] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:47:50.576 INFO 21062 --- [ocp-schedules-17,3d00caf7364e105b,20f0153f020059e3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:47:50.582 INFO 21062 --- [ocp-schedules-17,3d00caf7364e105b,20f0153f020059e3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:47:50.583 INFO 21062 --- [ocp-schedules-17,3d00caf7364e105b,20f0153f020059e3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:47:50.975 WARN 21062 --- [ocp-async-7,8ed9a864b16db89b,933780fc8ef1e79c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8b6c3b663dfa15e9,a1c82cda8ee31f4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:51.020 INFO 21062 --- [ocp-async-1,45b9ace7e09114e2,8d2b0e25374f5abf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:52.018 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:52.019 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:52.975 WARN 21062 --- [ocp-async-8,0e639ba3fd560790,cf3cdcf78a8158bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:53.020 INFO 21062 --- [ocp-async-9,1d4490986be96145,b32ddd4fd3818a91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:53.983 INFO 21062 --- [prometheus-minute-schedule-1,b30b5c1968cf915b,0959ac521ec19d2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:54.975 WARN 21062 --- [ocp-async-3,fb5b4a98cd471f97,c4e65d6644ec2db2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:55.019 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:55.020 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:55.023 INFO 21062 --- [ocp-async-4,ca5e01df7f9042db,fe2efce361dd2446] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:55.440 INFO 21062 --- [ocp-schedules-4,af44dbcbff47f977,4413d51edf02c0c1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:47:56.975 WARN 21062 --- [ocp-async-6,cc5d6b8e3e64d76c,6dfc0cf3fc9dab95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:56.983 INFO 21062 --- [prometheus-minute-schedule-1,dafb537e04fdbbb4,8e6d155b86830dbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:47:57.020 INFO 21062 --- [ocp-async-5,16f57384808882ef,c43a1a58f1b6b7e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:58.020 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:47:58.021 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:47:58.500 INFO 21062 --- [ocp-schedules-15,156a063d7568e418,87b3ee7bca98acb1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:47:58.500 INFO 21062 --- [ocp-schedules-11,e359cd6302d2119c,9aab2e976fa102df] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:47:58.503 INFO 21062 --- [ocp-schedules-2,2c667d712d76b523,f1e64884d0ab6682] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:47:58.504 INFO 21062 --- [ocp-schedules-15,156a063d7568e418,87b3ee7bca98acb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:58.504 INFO 21062 --- [ocp-schedules-11,e359cd6302d2119c,9aab2e976fa102df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:47:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:47:58.505 INFO 21062 --- [ocp-schedules-6,72962d438ef8256f,f962e989d0f0d276] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:47:58.509 INFO 21062 --- [ocp-schedules-13,50f37ad72745b4fa,7345ae82e9875bd5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:47:58.509 INFO 21062 --- [ocp-schedules-10,27c69fd783b55ffb,f77306ab00a0bfa7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:47:58.509 INFO 21062 --- [ocp-schedules-10,27c69fd783b55ffb,f77306ab00a0bfa7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:47:58.509 INFO 21062 --- [ocp-schedules-13,50f37ad72745b4fa,7345ae82e9875bd5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:47:58.510 INFO 21062 --- [ocp-schedules-27,df3e4a6cb340b1c5,f8be051a6affdca9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:47:58.510 INFO 21062 --- [ocp-schedules-27,df3e4a6cb340b1c5,f8be051a6affdca9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:47:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:47:58.589899+08:00, lastProcessTime=2024-11-04T13:47:48.590805+08:00 2024-11-04 13:47:58.620 INFO 21062 --- [ocp-async-4,4b9f64fc7e995c5b,2361b73ff81836de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:47:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:47:58.589899+08:00 2024-11-04 13:47:58.966 INFO 21062 --- [ocp-schedules-25,23ac9ec6e6005b3f,338cea055d45f2dc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:47:58.975 WARN 21062 --- [ocp-async-8,628844b4ced8e9a7,d651766153f438bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:47:59.020 INFO 21062 --- [ocp-async-9,952a93b28d1a6835,169cd452a28f1b2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:47:59.842 INFO 21062 --- [ocp-schedules-18,3e6b7ed366756c4c,3af43ba68d3d4140] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:47:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1e09f8f220045055,b6c4405d9d9fdb47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699280, interval=60 2024-11-04 13:48:00.976 WARN 21062 --- [ocp-async-3,98aa6b2ca0b3bb8a,71bb68a5a23cac09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:01.020 INFO 21062 --- [ocp-async-4,9e8979ce0b121847,e0c2ec1873f53cc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:01.021 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:01.022 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:02.976 WARN 21062 --- [ocp-async-6,46c545d0d14740a0,2966715477b266a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1a8428aaa96c26ba,cfa25b8ab50012b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:03.020 INFO 21062 --- [ocp-async-5,a6a65edd03f17241,590c57cb1b05dac2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:03.669 INFO 21062 --- [ocp-schedules-12,9afa89f3d327da01,4fb18c8bfb4760cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:48:03.669 INFO 21062 --- [ocp-schedules-12,3e0d321067d65209,f6d136e2190438da] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:48:03.674 INFO 21062 --- [ocp-schedules-12,3e0d321067d65209,f6d136e2190438da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:03.703 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:48:03.703 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:48:04.022 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:04.022 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:04.976 WARN 21062 --- [ocp-async-1,a88520b397967457,1846c751fe9b776c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:05.021 INFO 21062 --- [ocp-async-2,a1156081dd094b5f,1b208d54af3307d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:05.466 INFO 21062 --- [ocp-schedules-10,e2b22fd0ec2ea97a,75f2b8bd8b2511a8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:48:05.983 INFO 21062 --- [prometheus-minute-schedule-1,80ba7e4eeff7f7cb,7f9cf4b10a35c996] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:06.976 WARN 21062 --- [ocp-async-9,11a23eb72cd127be,31b5eb5a1cb2dff5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:07.021 INFO 21062 --- [ocp-async-10,5d1f67fce70159c8,049fdd47cdabf7a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:07.023 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:07.023 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:08.504 INFO 21062 --- [ocp-schedules-11,e359cd6302d2119c,9aab2e976fa102df] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:48:08.505 INFO 21062 --- [ocp-schedules-15,156a063d7568e418,87b3ee7bca98acb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:48:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:48:08.590726+08:00, lastProcessTime=2024-11-04T13:47:58.589899+08:00 2024-11-04 13:48:08.620 INFO 21062 --- [ocp-async-2,e22eb58ed86af25c,1a322b984dceb557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:48:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:48:08.590726+08:00 2024-11-04 13:48:08.868 INFO 21062 --- [ocp-schedules-9,2544c8968941f3fd,ff1762b0ee32da46] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:48:08.871 INFO 21062 --- [ocp-schedules-9,2544c8968941f3fd,ff1762b0ee32da46] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:08.872 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:48:08.976 WARN 21062 --- [ocp-async-6,568f4d5c43d8583e,dbac4c204d35c102] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a90d13b866203a35,536fce802a33e435] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:09.021 INFO 21062 --- [ocp-async-5,cf38ae412b65d385,15f35578eec35c47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:09.845 INFO 21062 --- [ocp-schedules-1,9b47b719aee61f17,6c06d6d706fdc79c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:48:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:48:10.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:48:10.024 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:10.025 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:48:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:48:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:48:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:48:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:48:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:48:10.585 INFO 21062 --- [ocp-schedules-27,fe0b704f176b8b6e,d8c0630189043db4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:48:10.590 INFO 21062 --- [ocp-schedules-27,fe0b704f176b8b6e,d8c0630189043db4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:48:10.590 INFO 21062 --- [ocp-schedules-27,fe0b704f176b8b6e,d8c0630189043db4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:48:10.976 WARN 21062 --- [ocp-async-1,e10d7410fa81dad3,b70be0750a6c7556] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:11.021 INFO 21062 --- [ocp-async-2,b9778052f0fd46dd,28d9c7de6204c2a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:11.354 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699220, nodeCount=392, affectRows=392 2024-11-04 13:48:11.356 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699220, nodeCount=389, validCount=389 2024-11-04 13:48:11.983 INFO 21062 --- [prometheus-minute-schedule-1,1ab1ae93875e80e1,41c470b5807e6d63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:12.977 WARN 21062 --- [ocp-async-9,0160185fe9a62c2d,00ecf8970551d834] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:13.021 INFO 21062 --- [ocp-async-10,29e7d03cceab9c60,78ad64107997d0ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:13.025 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:13.025 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:13.984 INFO 21062 --- [prometheus-cache-monitor1,88318a101bc2987f,5b1ebff64515ce80] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1490,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:48:13.984 INFO 21062 --- [prometheus-cache-monitor1,88318a101bc2987f,5b1ebff64515ce80] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:48:14.118 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:42:14.095, end=2024-11-04T13:47:14.115} 2024-11-04 13:48:14.118 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:42:14.096, end=2024-11-04T13:47:14.115} 2024-11-04 13:48:14.123 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:48:14.123 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:48:14.123 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:48:14.123 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:48:14.977 WARN 21062 --- [ocp-async-4,87fc4a600c4b23cb,9964f29f320e011a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f2dff4fa191a3ff0,5da75c0c59202772] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:15.022 INFO 21062 --- [ocp-async-7,00e5d0593c426878,443ec26e72d0bd27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:15.509 INFO 21062 --- [ocp-schedules-27,9a4cb37190bc8184,c0013e5ea40ce984] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:48:16.026 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:16.026 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:16.977 WARN 21062 --- [ocp-async-7,4ab62d498f80f4d2,f4e03d38feeda4db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:17.022 INFO 21062 --- [ocp-async-8,68bf22adb086b9c9,d50cb2312e45f2a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:17.983 INFO 21062 --- [prometheus-minute-schedule-1,aaf39083c3ac0064,3c01f7b781a4bc6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:48:18.591357+08:00, lastProcessTime=2024-11-04T13:48:08.590726+08:00 2024-11-04 13:48:18.620 INFO 21062 --- [ocp-async-7,d32d9f40d39db6e3,62e510391259efc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:48:18.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:48:18.591357+08:00 2024-11-04 13:48:18.872 INFO 21062 --- [ocp-schedules-9,2544c8968941f3fd,ff1762b0ee32da46] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:48:18.977 WARN 21062 --- [ocp-async-2,ee5fdab18ca2a034,8db7dfe1486999d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:19.022 INFO 21062 --- [ocp-async-10,1c2f3d63c076f53d,90c2d5da1b7f91d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:19.026 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:19.027 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:19.165 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:48:19.165 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:48:19.173 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:48:19.173 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:48:19.173 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:48:19.173 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:48:19.179 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:48:19.179 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:48:19.179 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:48:19.179 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:48:19.215 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:48:19.224 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:48:19.224 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:48:19.224 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:48:19.224 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:48:19.224 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:48:19.232 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:48:19.794 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:48:19.848 INFO 21062 --- [ocp-schedules-21,fc9d6c3b168b78a7,4221f91769028fe5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:48:20.362 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.183267875S 2024-11-04 13:48:20.362 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:48:20.466 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:48:20.466 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:48:20.493 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:48:20.558 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:48:20.558 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:48:20.558 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:48:20.558 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:48:20.558 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:48:20.559 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:48:20.569 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:48:20.746 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] o.s.boot.SpringApplication : Started application in 1.572 seconds (JVM running for 4714.425) 2024-11-04 13:48:20.746 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.572984138S 2024-11-04 13:48:20.977 WARN 21062 --- [ocp-async-10,576761ced61d178b,e5386484d0834f06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5f237b608e39e7d1,b49431c0183c4503] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:21.022 INFO 21062 --- [ocp-async-7,f3f2d0fbe0813e4f,bc14215d24c09f9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:21.218 INFO 21062 --- [ocp-schedules-2,8c0a7a678a49d1c0,26ab8b190208d0ee] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:48:22.027 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:22.028 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:22.978 WARN 21062 --- [ocp-async-7,c5b73db468824ff9,2b0a22c609080fd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:23.023 INFO 21062 --- [ocp-async-8,dcc4c388f8b15974,a2aaa246d3de00b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:23.983 INFO 21062 --- [prometheus-minute-schedule-1,abd9219324ea8e84,a84117faa334611a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:24.978 WARN 21062 --- [ocp-async-8,47f3b717982a21bc,e05336b2a8ee7e81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:25.023 INFO 21062 --- [ocp-async-9,af8b748a88a703c7,7f34b7244ca5c80e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:25.028 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:25.029 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:25.534 INFO 21062 --- [ocp-schedules-8,40ebaca4503e737b,83d2969485f32446] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:48:26.978 WARN 21062 --- [ocp-async-9,9e3c938c5db1d499,4c7acdcb2b2411cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:26.983 INFO 21062 --- [prometheus-minute-schedule-1,fc8bfff2539c599d,76b7dde3316e6e3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:26.984 INFO 21062 --- [prometheus-hour-schedule-1,23671c3f8ed71039,6faf0fcb6d02324a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:48:27.023 INFO 21062 --- [ocp-async-4,86ec3b7bdb482078,93954af2d3047358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:28.029 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:28.030 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:28.498 INFO 21062 --- [ocp-schedules-18,b3f3fb35a77f4586,d70a67b743f9e82b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:48:28.499 INFO 21062 --- [ocp-schedules-9,13fefb15a37c0dae,61f2d37dbc6ee89b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:48:28.500 INFO 21062 --- [ocp-schedules-22,9b532784c5b0f42c,6fdf3242fdcad53e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:48:28.500 INFO 21062 --- [ocp-schedules-2,864604e454cf3bcc,f933ef222b7db5d8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:48:28.502 INFO 21062 --- [ocp-schedules-27,39039cc2792551ea,42e5c55ea2299f31] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:48:28.503 INFO 21062 --- [ocp-schedules-22,9b532784c5b0f42c,6fdf3242fdcad53e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:28.503 INFO 21062 --- [ocp-schedules-18,b3f3fb35a77f4586,d70a67b743f9e82b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:48:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:48:28.504 INFO 21062 --- [ocp-schedules-2,864604e454cf3bcc,f933ef222b7db5d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:28.504 INFO 21062 --- [ocp-schedules-21,6832e015a482d06a,6380aa601919ddc6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:48:28.505 INFO 21062 --- [ocp-schedules-9,13fefb15a37c0dae,61f2d37dbc6ee89b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:48:28.512 INFO 21062 --- [ocp-schedules-4,f67c44bd2209af89,4e000d52902da591] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:48:28.512 INFO 21062 --- [ocp-schedules-24,186a82053e869253,48898da68a750582] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:48:28.512 INFO 21062 --- [ocp-schedules-24,186a82053e869253,48898da68a750582] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:48:28.512 INFO 21062 --- [ocp-schedules-4,f67c44bd2209af89,4e000d52902da591] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:48:28.513 INFO 21062 --- [ocp-schedules-30,36332620aca1982f,2242f9e05382438f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:48:28.513 INFO 21062 --- [ocp-schedules-30,36332620aca1982f,2242f9e05382438f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:48:28.569 INFO 21062 --- [ocp-schedules-19,c0c880d75e066297,d2e5292cf21e63ab] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1490,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:48:28.569 INFO 21062 --- [ocp-schedules-19,c0c880d75e066297,d2e5292cf21e63ab] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:48:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:48:28.590086+08:00, lastProcessTime=2024-11-04T13:48:18.591357+08:00 2024-11-04 13:48:28.609 INFO 21062 --- [ocp-schedules-20,145bd2d824df576c,28414ec3000781cf] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:48:28.610 INFO 21062 --- [ocp-schedules-20,145bd2d824df576c,28414ec3000781cf] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:48:28.620 INFO 21062 --- [ocp-async-6,b8979eec7c9a5c7a,8b9ac973aa97e8ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:48:28.744 INFO 21062 --- [ocp-schedules-28,8f869f1031ba09b4,de4a2ad0c0de9ccb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:48:28.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:48:28.590086+08:00 2024-11-04 13:48:28.935 INFO 21062 --- [ocp-schedules-9,61e03c19a77e17e4,110e14cb300b5c0b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:48:28.935 INFO 21062 --- [ocp-schedules-9,61e03c19a77e17e4,110e14cb300b5c0b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:48:28.968 INFO 21062 --- [ocp-schedules-27,7c943fe79ee9a1ff,2629e5f8079f4216] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:48:28.978 WARN 21062 --- [ocp-async-5,0dd460634448f9d4,3b7cea902cce24ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:29.003 INFO 21062 --- [ocp-schedules-30,b7e9995fedd8a562,c6e938cb43d7cd87] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:48:29.008 INFO 21062 --- [ocp-schedules-30,b7e9995fedd8a562,c6e938cb43d7cd87] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:48:29.008 INFO 21062 --- [ocp-schedules-30,b7e9995fedd8a562,c6e938cb43d7cd87] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:48:29.023 INFO 21062 --- [ocp-async-3,a92707581846b545,92f2e92454c2a705] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:29.257 INFO 21062 --- [ocp-schedules-14,65f7efea81de6427,c6cd79c72033aabe] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:48:29.257 INFO 21062 --- [ocp-schedules-14,65f7efea81de6427,c6cd79c72033aabe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:48:29.264 INFO 21062 --- [ocp-schedules-14,65f7efea81de6427,c6cd79c72033aabe] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:48:29.264 INFO 21062 --- [ocp-schedules-14,65f7efea81de6427,c6cd79c72033aabe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:48:29.269 INFO 21062 --- [ocp-schedules-14,65f7efea81de6427,c6cd79c72033aabe] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:48:29.269 INFO 21062 --- [ocp-schedules-14,65f7efea81de6427,c6cd79c72033aabe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:48:29.851 INFO 21062 --- [ocp-schedules-17,8c693d689f994943,e183f15fdb445bbd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:48:29.983 INFO 21062 --- [prometheus-minute-schedule-1,53ca98c04b997931,e3161ee5462feaef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:48:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:48:30.594 INFO 21062 --- [ocp-schedules-30,b7689294d0710272,87ccf666011599fc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:48:30.599 INFO 21062 --- [ocp-schedules-16,35af0cf006b2656f,f67a0adc975d9ec3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:48:30.599 INFO 21062 --- [ocp-schedules-16,35af0cf006b2656f,f67a0adc975d9ec3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:48:30.601 INFO 21062 --- [ocp-schedules-30,b7689294d0710272,87ccf666011599fc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:48:30.601 INFO 21062 --- [ocp-schedules-30,b7689294d0710272,87ccf666011599fc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:48:30.626 INFO 21062 --- [ocp-schedules-16,35af0cf006b2656f,f67a0adc975d9ec3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:48:30.626 INFO 21062 --- [ocp-schedules-16,35af0cf006b2656f,f67a0adc975d9ec3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:48:30.627 INFO 21062 --- [ocp-schedules-19,62a1932d616257f0,1c26c5d517eda3c5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:48:30.768 INFO 21062 --- [ocp-schedules-7,541a5d2cf80228a1,3d40689923a74ec6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:48:30.768+08:00, clusters=[1] 2024-11-04 13:48:30.770 INFO 21062 --- [ocp-schedules-7,541a5d2cf80228a1,3d40689923a74ec6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:48:30.820 INFO 21062 --- [ocp-schedules-1,72ce1cf62e8d4e1a,4c93bf3ed851b214] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:48:30.820 INFO 21062 --- [ocp-schedules-1,72ce1cf62e8d4e1a,4c93bf3ed851b214] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:48:30.820 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:48:30.820 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:48:30.830 INFO 21062 --- [ocp-schedules-1,72ce1cf62e8d4e1a,4c93bf3ed851b214] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:48:30.830 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:48:30.830 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:48:30.832 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:48:30.832 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:48:30.832 WARN 21062 --- [ocp-schedules-1,72ce1cf62e8d4e1a,4c93bf3ed851b214] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:48:30.833 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:48:30.833 INFO 21062 --- [ocp-schedules-1,72ce1cf62e8d4e1a,4c93bf3ed851b214] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:48:30.833 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:48:30.834 WARN 21062 --- [ocp-schedules-1,72ce1cf62e8d4e1a,4c93bf3ed851b214] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:48:30.834 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:48:30.834 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:48:30.834 INFO 21062 --- [ocp-schedules-13,d98496d29d6251aa,52308ae7b6fa65da] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:48:30.838 INFO 21062 --- [ocp-schedules-13,d98496d29d6251aa,52308ae7b6fa65da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:30.840 ERROR 21062 --- [ocp-schedules-1,72ce1cf62e8d4e1a,4c93bf3ed851b214] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:48:30.898 ERROR 21062 --- [ocp-schedules-13,d98496d29d6251aa,52308ae7b6fa65da] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:48:30.898 INFO 21062 --- [ocp-schedules-13,d98496d29d6251aa,52308ae7b6fa65da] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:48:30.921 INFO 21062 --- [ocp-schedules-23,e7a9be7edfdad1b5,02a4a350e5390f52] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:48:30.921+08:00, clusters=[1] 2024-11-04 13:48:30.924 INFO 21062 --- [ocp-schedules-23,e7a9be7edfdad1b5,02a4a350e5390f52] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:48:30.978 WARN 21062 --- [ocp-async-3,6e1c7f7c6917d0c3,ffd5b6d86d00983e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:31.023 INFO 21062 --- [ocp-async-6,f7a15d934ad977a7,4676fbb25dd0c95d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:31.030 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:31.031 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:32.978 WARN 21062 --- [ocp-async-6,8e06dd13fad8e43f,e0e643b5dd485c4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7f0a7b3575d9d00b,da004292213e061a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:33.023 INFO 21062 --- [ocp-async-1,24671eb646ef3bb2,d81991f82042508c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:33.703 INFO 21062 --- [ocp-schedules-12,3e0d321067d65209,f6d136e2190438da] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:48:33.704 INFO 21062 --- [ocp-schedules-12,5bcfef1e742d54d1,74f8c2f51d8d0f26] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:48:33.708 INFO 21062 --- [ocp-schedules-12,5bcfef1e742d54d1,74f8c2f51d8d0f26] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:33.738 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:48:33.738 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:48:34.031 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:34.032 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:34.979 WARN 21062 --- [ocp-async-1,63faa125ce6f4dbf,76f6398da8686f36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:35.023 INFO 21062 --- [ocp-async-9,3ba23dae4ea0cd68,246b84fd4f63e585] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:35.514 INFO 21062 --- [ocp-schedules-16,6a9e8f048b6b0316,b885c87f2ebe4d0d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:48:35.518 INFO 21062 --- [ocp-schedules-16,6a9e8f048b6b0316,b885c87f2ebe4d0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:35.577 INFO 21062 --- [ocp-schedules-19,0b58c39953f1084a,492532c8953f3ff8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:48:35.602 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:48:35.604 ERROR 21062 --- [ocp-schedules-16,6a9e8f048b6b0316,b885c87f2ebe4d0d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:48:35.604 INFO 21062 --- [ocp-schedules-16,6a9e8f048b6b0316,b885c87f2ebe4d0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:48:35.983 INFO 21062 --- [prometheus-minute-schedule-1,bc50a7dd78050004,2102a6cea0988d90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:36.979 WARN 21062 --- [ocp-async-9,9806d0ee3faa07d2,8b35c5f189b60b9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:37.024 INFO 21062 --- [ocp-async-10,255a170634cd9b9d,3b5c541e3963b1b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:37.032 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:37.032 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:38.504 INFO 21062 --- [ocp-schedules-22,9b532784c5b0f42c,6fdf3242fdcad53e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:48:38.504 INFO 21062 --- [ocp-schedules-2,864604e454cf3bcc,f933ef222b7db5d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:48:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:48:38.589655+08:00, lastProcessTime=2024-11-04T13:48:28.590086+08:00 2024-11-04 13:48:38.620 INFO 21062 --- [ocp-async-3,27375b1846e9d39c,71fffdd471572f1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:48:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:48:38.589655+08:00 2024-11-04 13:48:38.979 WARN 21062 --- [ocp-async-4,e0b73d92317457b6,eb391478270c04fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3206d6d28a1b6bc2,d8b6dc6f31332497] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:39.024 INFO 21062 --- [ocp-async-5,68952cd34cb119f7,5e9380634abc1706] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:39.854 INFO 21062 --- [ocp-schedules-23,c1076136371085be,7ea992203b52d0e4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:48:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:48:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:48:40.033 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:40.033 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:40.979 WARN 21062 --- [ocp-async-5,545d7b05ec5c721e,a5a3bc18e7c442fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:41.024 INFO 21062 --- [ocp-async-3,eef49e3cfcf6e5ef,28e022c01f96fd0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:48:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:48:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:48:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:48:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:48:41.853 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:48:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:48:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bdafb897f9de5804,56f896b59e9e9393] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:42.979 WARN 21062 --- [ocp-async-3,b2a68a4b573e6658,0d23586df61159b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:43.024 INFO 21062 --- [ocp-async-10,0977820e397f0164,c20910205c76c2cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:43.033 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:43.034 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:44.979 WARN 21062 --- [ocp-async-10,4ae95e7ee02566f1,8cc08421c123db6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:44.983 INFO 21062 --- [prometheus-minute-schedule-1,34544623e8da2aa2,31c3efb49e7ceda2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:45.024 INFO 21062 --- [ocp-async-7,d9409e769fda638f,61109846013ae8ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:45.605 INFO 21062 --- [ocp-schedules-6,94770960ee075944,c7e3ecd9727f3c5b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:48:46.034 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:46.035 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:46.979 WARN 21062 --- [ocp-async-7,460611c8f1441ad2,4d8bb8e1c5225d5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:47.024 INFO 21062 --- [ocp-async-8,cb7d1b7dad23816f,6ae36cb1c0ba9b88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:47.983 INFO 21062 --- [prometheus-minute-schedule-1,5a63efda209a1547,7e358394a16ec8de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:48:48.590962+08:00, lastProcessTime=2024-11-04T13:48:38.589655+08:00 2024-11-04 13:48:48.620 INFO 21062 --- [ocp-async-1,35a78d21a11429e4,e9fd2f8e9862bc6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:48:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:48:48.590962+08:00 2024-11-04 13:48:48.980 WARN 21062 --- [ocp-async-3,a3bc41174b177e3f,85008ef89e6124ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:49.025 INFO 21062 --- [ocp-async-6,75280922ce07e641,2c045edeaeea8bad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:49.035 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:49.036 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:49.858 INFO 21062 --- [ocp-schedules-2,f708968a48a8f22e,9d17a9bf3c14ec4c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:48:50.604 INFO 21062 --- [ocp-schedules-16,9b36dcee7f25ad5c,6c1db6f51dfd42c1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:48:50.609 INFO 21062 --- [ocp-schedules-16,9b36dcee7f25ad5c,6c1db6f51dfd42c1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:48:50.609 INFO 21062 --- [ocp-schedules-16,9b36dcee7f25ad5c,6c1db6f51dfd42c1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:48:50.980 WARN 21062 --- [ocp-async-6,c6e7dff4919fb338,94f64fcbdadd52e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b23892b21ae113d3,d3f4c1cb2f3be0a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:51.025 INFO 21062 --- [ocp-async-7,e591575634ba6690,638e63363277ada7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:52.036 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:52.037 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:52.980 WARN 21062 --- [ocp-async-1,63526f74ade0c569,e052045e939eb435] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:53.025 INFO 21062 --- [ocp-async-9,04dd46d5188e4132,a92cbaf5210cf3d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:53.983 INFO 21062 --- [prometheus-minute-schedule-1,837467ff6d343118,b02b1f0cfca57de3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:54.980 WARN 21062 --- [ocp-async-8,b909fc547d35f67e,111c2d88d2db8383] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:55.025 INFO 21062 --- [ocp-async-2,ca90410460daee36,76b0c3098a25a506] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:55.037 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:55.037 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:55.643 INFO 21062 --- [ocp-schedules-8,511f3748efeabbd0,b1ee920930b8db5b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:48:56.980 WARN 21062 --- [ocp-async-10,e3363ded9c8e8d86,21b938135a97f5ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:56.983 INFO 21062 --- [prometheus-minute-schedule-1,4264e5bb936e7e86,1c2fde680122188f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:48:57.026 INFO 21062 --- [ocp-async-4,3c19084b28e15c33,30264a6d405a9ba0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:58.038 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:48:58.038 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:48:58.499 INFO 21062 --- [ocp-schedules-16,a8f69e090f206ef6,8af5c05652fd678b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:48:58.500 INFO 21062 --- [ocp-schedules-7,940153b758d5348e,41d1690e8bc736ee] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:48:58.503 INFO 21062 --- [ocp-schedules-23,245776c1166c6d67,16f33a799bc504ec] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:48:58.504 INFO 21062 --- [ocp-schedules-16,a8f69e090f206ef6,8af5c05652fd678b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:58.504 INFO 21062 --- [ocp-schedules-7,940153b758d5348e,41d1690e8bc736ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:48:58.505 INFO 21062 --- [ocp-schedules-21,3e287e7fae33d58d,bfdbea89c135dad2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:48:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:48:58.512 INFO 21062 --- [ocp-schedules-27,11d14cd88ea0fdf1,b7f3fcf2e8b03968] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:48:58.513 INFO 21062 --- [ocp-schedules-14,32289ca8d5af8bee,da7b01074c8d0d0e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:48:58.513 INFO 21062 --- [ocp-schedules-14,32289ca8d5af8bee,da7b01074c8d0d0e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:48:58.513 INFO 21062 --- [ocp-schedules-27,11d14cd88ea0fdf1,b7f3fcf2e8b03968] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:48:58.513 INFO 21062 --- [ocp-schedules-19,ba04b5253729305a,7c80c94ae7b21a89] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:48:58.513 INFO 21062 --- [ocp-schedules-19,ba04b5253729305a,7c80c94ae7b21a89] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:48:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:48:58.589733+08:00, lastProcessTime=2024-11-04T13:48:48.590962+08:00 2024-11-04 13:48:58.621 INFO 21062 --- [ocp-async-2,8cef739d984cd5fb,ac015490ad2608ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:48:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:48:58.589733+08:00 2024-11-04 13:48:58.971 INFO 21062 --- [ocp-schedules-10,8d1f21196b83d241,0da9cc05d202649d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:48:58.981 WARN 21062 --- [ocp-async-1,29e18c65bd4b484c,0f193494996178f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:48:59.026 INFO 21062 --- [ocp-async-9,6a7ec4640aaa7749,1b1683d00b238583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:48:59.861 INFO 21062 --- [ocp-schedules-6,0ee4f4d4db65d00e,4b52907935e342b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:48:59.983 INFO 21062 --- [prometheus-minute-schedule-1,af8d25b7718fc950,21c0e4ceff1dd445] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699340, interval=60 2024-11-04 13:49:00.981 WARN 21062 --- [ocp-async-8,e5e5133de02d4f29,174c6a2d1255306d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:01.026 INFO 21062 --- [ocp-async-2,cc4d30dd7e9af864,1cd377491a3f8cf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:01.039 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:01.039 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:02.981 WARN 21062 --- [ocp-async-10,65457bff6ebc73fc,d5b5c99468dfed9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:02.983 INFO 21062 --- [prometheus-minute-schedule-1,14a601888767fca5,a39699ac06e8b389] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:03.026 INFO 21062 --- [ocp-async-4,0fffca5fa8534b03,dfc24d4bfcfaa878] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:03.738 INFO 21062 --- [ocp-schedules-12,5bcfef1e742d54d1,74f8c2f51d8d0f26] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:49:03.738 INFO 21062 --- [ocp-schedules-12,0651ab3ced27ccdb,90c5503cc0127bc0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:49:03.743 INFO 21062 --- [ocp-schedules-12,0651ab3ced27ccdb,90c5503cc0127bc0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:03.771 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:49:03.771 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:49:04.039 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:04.040 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:04.981 WARN 21062 --- [ocp-async-7,96073322d02afeae,47f4c67292ad7a34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:05.026 INFO 21062 --- [ocp-async-5,96109bd0201800fe,be60838471dba7e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:05.669 INFO 21062 --- [ocp-schedules-5,06a629ffa498db0e,d2f03a0034a1f39c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:49:05.983 INFO 21062 --- [prometheus-minute-schedule-1,7a2cf05a9b208718,685ecf2005d5bddc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:06.981 WARN 21062 --- [ocp-async-9,648fac54e442d5d2,d460c501239628be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:07.026 INFO 21062 --- [ocp-async-3,3fd48f0b8ae8b6e2,38c7980c443e30c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:07.040 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:07.041 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:08.505 INFO 21062 --- [ocp-schedules-7,940153b758d5348e,41d1690e8bc736ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:49:08.506 INFO 21062 --- [ocp-schedules-16,a8f69e090f206ef6,8af5c05652fd678b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:49:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:49:08.590135+08:00, lastProcessTime=2024-11-04T13:48:58.589733+08:00 2024-11-04 13:49:08.621 INFO 21062 --- [ocp-async-5,8100f0599b64bebf,e510aba62e530825] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:49:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:49:08.590135+08:00 2024-11-04 13:49:08.981 WARN 21062 --- [ocp-async-10,11846606bec6baf5,1bfb6e527e046d73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:08.983 INFO 21062 --- [prometheus-minute-schedule-1,4bc71d61dd9f5778,85e23bc87c3039ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:09.026 INFO 21062 --- [ocp-async-4,acaf0e62e0ea0a30,0b0bf84d1b2c7248] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:09.864 INFO 21062 --- [ocp-schedules-16,e107cc548fa50d0a,b86f62333471f9cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:49:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:49:10.005 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:49:10.041 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:10.042 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:49:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:49:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:49:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:49:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:49:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:49:10.611 INFO 21062 --- [ocp-schedules-7,3f1e51cde793b412,130ce1278ae7c2fb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:49:10.618 INFO 21062 --- [ocp-schedules-7,3f1e51cde793b412,130ce1278ae7c2fb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:49:10.618 INFO 21062 --- [ocp-schedules-7,3f1e51cde793b412,130ce1278ae7c2fb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:49:10.982 WARN 21062 --- [ocp-async-7,f66d4fed4dcfe6ac,419ea670e2e480b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:11.027 INFO 21062 --- [ocp-async-5,7d82567a0069b93e,dd3bece523beedb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:11.350 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699280, nodeCount=22, affectRows=22 2024-11-04 13:49:11.350 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699280, nodeCount=22, validCount=22 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699280, nodeCount=74, affectRows=74 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699280, nodeCount=43, affectRows=43 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699280, nodeCount=30, affectRows=30 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699280, nodeCount=60, affectRows=60 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699280, nodeCount=83, affectRows=83 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699280, nodeCount=30, validCount=30 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699280, nodeCount=43, validCount=43 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699280, nodeCount=73, validCount=73 2024-11-04 13:49:11.351 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699280, nodeCount=60, validCount=60 2024-11-04 13:49:11.352 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699280, nodeCount=83, validCount=83 2024-11-04 13:49:11.352 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699280, nodeCount=80, affectRows=80 2024-11-04 13:49:11.353 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699280, nodeCount=78, validCount=78 2024-11-04 13:49:11.983 INFO 21062 --- [prometheus-minute-schedule-1,7dfbbe00189aba71,b40027979073ed8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:12.982 WARN 21062 --- [ocp-async-9,9f40ff87e9e47a15,7b6c8fe43ba1ebbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:13.027 INFO 21062 --- [ocp-async-3,9fba3e3870ec30f2,21c4105e91d32953] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:13.042 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:13.043 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:13.985 INFO 21062 --- [prometheus-cache-monitor1,8d99608953cbe271,4ad28d7d6cd962e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1518,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:49:13.985 INFO 21062 --- [prometheus-cache-monitor1,8d99608953cbe271,4ad28d7d6cd962e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:49:14.733 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:45:14.711, end=2024-11-04T13:47:14.731} 2024-11-04 13:49:14.743 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:49:14.743 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:49:14.982 WARN 21062 --- [ocp-async-2,d1784329d444e197,716dda88d65545d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:14.983 INFO 21062 --- [prometheus-minute-schedule-1,e1045130d879fe56,3e604bddb8981418] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:15.027 INFO 21062 --- [ocp-async-6,76f64e4abe6caed8,a7e3e7c1ed128266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:15.714 INFO 21062 --- [ocp-schedules-18,5163a6aec830916c,bf379b602e1de01d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:49:16.043 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:16.044 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:16.982 WARN 21062 --- [ocp-async-4,a8cfc1dce81ee97f,22155f4251995a6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:17.027 INFO 21062 --- [ocp-async-1,89beb2046e68432e,75bf8a57ce40856d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:17.983 INFO 21062 --- [prometheus-minute-schedule-1,cf8b52b6af9a7e2c,8e039586d7bcfed5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:49:18.590313+08:00, lastProcessTime=2024-11-04T13:49:08.590135+08:00 2024-11-04 13:49:18.621 INFO 21062 --- [ocp-async-6,3d43d53f93e905f2,4c73e91e5b758b42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:49:18.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:49:18.590313+08:00 2024-11-04 13:49:18.872 INFO 21062 --- [ocp-schedules-23,2783138fab33119a,b62ae317ffd8b61f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:49:18.875 INFO 21062 --- [ocp-schedules-23,2783138fab33119a,b62ae317ffd8b61f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:18.876 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:49:18.982 WARN 21062 --- [ocp-async-5,46872fc1ad853c33,24992beb78a016b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:19.027 INFO 21062 --- [ocp-async-3,98d16efe38714c53,adb2bb30b387c18a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:19.045 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:19.045 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:19.868 INFO 21062 --- [ocp-schedules-16,f3cb4b41d6c65438,34873977cfb1950f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:49:20.983 WARN 21062 --- [ocp-async-3,4c518d2a7c262798,c48525d4b49c4b2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:20.983 INFO 21062 --- [prometheus-minute-schedule-1,7e5e34ea7b9d4d11,11232f0b7a0dcadd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:21.027 INFO 21062 --- [ocp-async-6,8532be1b3173c649,0be9311fcca09fc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:22.045 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:22.046 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:22.983 WARN 21062 --- [ocp-async-6,a7268d7a7173f26f,9d38f1e7cc35c961] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:23.028 INFO 21062 --- [ocp-async-1,cd86779c31e51af8,406dfcd038f96098] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:23.983 INFO 21062 --- [prometheus-minute-schedule-1,d46b41da84679b58,a7e22daa5c9c28b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:24.983 WARN 21062 --- [ocp-async-1,25111f3935abb05d,847acfe25b02be3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:25.028 INFO 21062 --- [ocp-async-8,721ac2b8fb7f3869,f0a1b63c36aca935] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:25.047 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:25.049 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:25.741 INFO 21062 --- [ocp-schedules-16,d190960f4596fa02,7a11783af0f61ede] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:49:26.983 INFO 21062 --- [prometheus-minute-schedule-1,d4000809519cb0b6,a47b9d935a898cf4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:26.983 WARN 21062 --- [ocp-async-8,4cf9cfe0a2e20471,71e1047178c5ef6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8ad4c2be6c06a2ef,5d1bd8a9dac45246] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:49:27.028 INFO 21062 --- [ocp-async-2,70096dfe14efccc6,f7a24a11446424c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:28.049 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:28.050 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:28.498 INFO 21062 --- [ocp-schedules-3,3fde7861c813f9a3,1be249e6c36cc2b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:49:28.500 INFO 21062 --- [ocp-schedules-14,ce5fb6f6ea89f1ca,44367e6509496c63] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:49:28.500 INFO 21062 --- [ocp-schedules-22,6d6d9a623dc6bc63,158ae4efbbb95546] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:49:28.500 INFO 21062 --- [ocp-schedules-15,54ed3b2ddf5f05cf,4b4794c9efceb8f1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:49:28.502 INFO 21062 --- [ocp-schedules-3,3fde7861c813f9a3,1be249e6c36cc2b1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:49:28.502 INFO 21062 --- [ocp-schedules-11,0c021a022fd0ce7b,733149e46618c6a8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:49:28.503 INFO 21062 --- [ocp-schedules-14,ce5fb6f6ea89f1ca,44367e6509496c63] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:49:28.503 INFO 21062 --- [ocp-schedules-15,54ed3b2ddf5f05cf,4b4794c9efceb8f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:28.504 INFO 21062 --- [ocp-schedules-26,f9aa760d04002274,2347edbc67faee17] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:49:28.505 INFO 21062 --- [ocp-schedules-22,6d6d9a623dc6bc63,158ae4efbbb95546] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:49:28.509 INFO 21062 --- [ocp-schedules-7,1c15b85f55a219e5,c7a90d4289661505] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:49:28.510 INFO 21062 --- [ocp-schedules-4,d1a1d0300737320e,ec80c657d99131f1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:49:28.510 INFO 21062 --- [ocp-schedules-7,1c15b85f55a219e5,c7a90d4289661505] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:49:28.510 INFO 21062 --- [ocp-schedules-4,d1a1d0300737320e,ec80c657d99131f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:49:28.510 INFO 21062 --- [ocp-schedules-10,418a45b5c4eb38a4,e850332d80e89146] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:49:28.510 INFO 21062 --- [ocp-schedules-10,418a45b5c4eb38a4,e850332d80e89146] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:49:28.570 INFO 21062 --- [ocp-schedules-5,44e66f3acd0c91ed,2b7143cc1796e682] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:49:28.570+08:00, tenantSize=0 2024-11-04 13:49:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:49:28.590133+08:00, lastProcessTime=2024-11-04T13:49:18.590313+08:00 2024-11-04 13:49:28.621 INFO 21062 --- [ocp-async-10,7ca3e4c2c77ba898,456519980127bcac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:49:28.746 INFO 21062 --- [ocp-schedules-24,2996c58ba42e19ab,acf7abf02dcc2f80] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:49:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:49:28.590133+08:00 2024-11-04 13:49:28.877 INFO 21062 --- [ocp-schedules-23,2783138fab33119a,b62ae317ffd8b61f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:49:28.939 INFO 21062 --- [ocp-schedules-13,07189de39a163bd5,c4b5f60fb47819f5] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:49:28.939 INFO 21062 --- [ocp-schedules-13,07189de39a163bd5,c4b5f60fb47819f5] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:49:28.973 INFO 21062 --- [ocp-schedules-20,3227709854a3a25e,b3a0b75c5e127b24] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:49:28.983 WARN 21062 --- [ocp-async-4,ff96ea866d2a227b,8b54ddc8d88e1c25] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:29.008 INFO 21062 --- [ocp-schedules-7,ee878c2c91ff522b,cf16f2828daef22e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:49:29.013 INFO 21062 --- [ocp-schedules-7,ee878c2c91ff522b,cf16f2828daef22e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:49:29.013 INFO 21062 --- [ocp-schedules-7,ee878c2c91ff522b,cf16f2828daef22e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:49:29.028 INFO 21062 --- [ocp-async-8,cd3155c8d6096a14,7579a395110ba0d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:29.084 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:49:29.084 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:49:29.092 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:49:29.092 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:49:29.092 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:49:29.103 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:49:29.103 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:49:29.103 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:49:29.106 INFO 21062 --- [ocp-schedules-3,9650ab81f063d372,7052e75861d743ad] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:49:29.273 INFO 21062 --- [ocp-schedules-1,d0dcc51479386591,c88e40a905cb9744] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:49:29.273 INFO 21062 --- [ocp-schedules-1,d0dcc51479386591,c88e40a905cb9744] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:49:29.274 INFO 21062 --- [ocp-schedules-1,d0dcc51479386591,c88e40a905cb9744] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:49:29.274 INFO 21062 --- [ocp-schedules-1,d0dcc51479386591,c88e40a905cb9744] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:49:29.275 INFO 21062 --- [ocp-schedules-1,d0dcc51479386591,c88e40a905cb9744] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:49:29.275 INFO 21062 --- [ocp-schedules-1,d0dcc51479386591,c88e40a905cb9744] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:49:29.870 INFO 21062 --- [ocp-schedules-24,477c7a6a570d20c9,74bc3b45e0b80e80] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:49:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6f469521361901b7,82d5330c857d918c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:49:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:49:30.621 INFO 21062 --- [ocp-schedules-19,0e9b804303977139,95c3ab5897a2b6cf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:49:30.627 INFO 21062 --- [ocp-schedules-29,21ce40ffeefd1a03,acffed897936d74a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:49:30.627 INFO 21062 --- [ocp-schedules-29,21ce40ffeefd1a03,acffed897936d74a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:49:30.628 INFO 21062 --- [ocp-schedules-19,0e9b804303977139,95c3ab5897a2b6cf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:49:30.628 INFO 21062 --- [ocp-schedules-19,0e9b804303977139,95c3ab5897a2b6cf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:49:30.654 INFO 21062 --- [ocp-schedules-29,21ce40ffeefd1a03,acffed897936d74a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:49:30.654 INFO 21062 --- [ocp-schedules-29,21ce40ffeefd1a03,acffed897936d74a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:49:30.654 INFO 21062 --- [ocp-schedules-25,eaa31ec17ec5cfb3,c238103fbdefc9ef] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:49:30.798 INFO 21062 --- [ocp-schedules-9,baa297b394f186a3,f9a7ee145e719ad6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:49:30.798+08:00, clusters=[1] 2024-11-04 13:49:30.800 INFO 21062 --- [ocp-schedules-9,baa297b394f186a3,f9a7ee145e719ad6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:49:30.848 INFO 21062 --- [ocp-schedules-6,52525323d922cf80,b3c6385aec60d55a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:49:30.848 INFO 21062 --- [ocp-schedules-6,52525323d922cf80,b3c6385aec60d55a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:49:30.848 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:49:30.848 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:49:30.858 INFO 21062 --- [ocp-schedules-6,52525323d922cf80,b3c6385aec60d55a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:49:30.859 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:49:30.859 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:49:30.860 WARN 21062 --- [ocp-schedules-6,52525323d922cf80,b3c6385aec60d55a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:49:30.861 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:49:30.861 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:49:30.861 INFO 21062 --- [ocp-schedules-6,52525323d922cf80,b3c6385aec60d55a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:49:30.861 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:49:30.862 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:49:30.862 WARN 21062 --- [ocp-schedules-6,52525323d922cf80,b3c6385aec60d55a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:49:30.863 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:49:30.864 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:49:30.869 ERROR 21062 --- [ocp-schedules-6,52525323d922cf80,b3c6385aec60d55a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:49:30.950 INFO 21062 --- [ocp-schedules-2,7f75c98abb5c5211,f5e8be571221cc95] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:49:30.950+08:00, clusters=[1] 2024-11-04 13:49:30.954 INFO 21062 --- [ocp-schedules-2,7f75c98abb5c5211,f5e8be571221cc95] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:49:30.984 WARN 21062 --- [ocp-async-8,5e916e30eabb6e9d,170bf0f148c6ef57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:31.028 INFO 21062 --- [ocp-async-10,a01fb756815063ea,b40c1010fc93627a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:31.050 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:31.051 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:32.983 INFO 21062 --- [prometheus-minute-schedule-1,040ff3ad526b1199,57b218afa36b2aec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:32.984 WARN 21062 --- [ocp-async-10,5dc0c30083efcf84,f99ef3bd0cfae5b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:33.029 INFO 21062 --- [ocp-async-7,7234b1065e2d2f6b,1bc041f141d9e7dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:33.772 INFO 21062 --- [ocp-schedules-12,0651ab3ced27ccdb,90c5503cc0127bc0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:49:33.772 INFO 21062 --- [ocp-schedules-12,59666bff8d004965,45cd09c3db29dbdd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:49:33.779 INFO 21062 --- [ocp-schedules-12,59666bff8d004965,45cd09c3db29dbdd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:33.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:49:33.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:49:34.051 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:34.051 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:34.984 WARN 21062 --- [ocp-async-7,fa003c1bcb21484f,a881e9af18036525] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:35.029 INFO 21062 --- [ocp-async-5,560fa01c8ceca09e,3ca1705503478b37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:35.604 INFO 21062 --- [ocp-schedules-9,f34811063eb7a2ff,d01db4016a27f7ed] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:49:35.608 INFO 21062 --- [ocp-schedules-9,f34811063eb7a2ff,d01db4016a27f7ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:35.695 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:49:35.702 ERROR 21062 --- [ocp-schedules-9,f34811063eb7a2ff,d01db4016a27f7ed] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:49:35.702 INFO 21062 --- [ocp-schedules-9,f34811063eb7a2ff,d01db4016a27f7ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:49:35.782 INFO 21062 --- [ocp-schedules-4,62c3f775bbb0f10b,4691321fa2d3b47e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:49:35.983 INFO 21062 --- [prometheus-minute-schedule-1,36a6aadcae10ab27,b5dcc42738e27e47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:36.984 WARN 21062 --- [ocp-async-5,ff4a2caa51dbe968,95dce5e81b3f2e98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:37.029 INFO 21062 --- [ocp-async-3,26d019d2cf95f5bf,64033013e73425ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:37.052 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:37.053 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:38.504 INFO 21062 --- [ocp-schedules-14,ce5fb6f6ea89f1ca,44367e6509496c63] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:49:38.504 INFO 21062 --- [ocp-schedules-15,54ed3b2ddf5f05cf,4b4794c9efceb8f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:49:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:49:38.590830+08:00, lastProcessTime=2024-11-04T13:49:28.590133+08:00 2024-11-04 13:49:38.621 INFO 21062 --- [ocp-async-8,2c8c49ca28f00348,8cae8a5be99f45c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:49:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:49:38.590830+08:00 2024-11-04 13:49:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0ef62c8d625d563f,486caef64c0116c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:38.984 WARN 21062 --- [ocp-async-2,1f45556ee759471e,79bb4f228d1ec135] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:39.029 INFO 21062 --- [ocp-async-4,a7c2230731491e5a,831e093b7f9f7768] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:39.874 INFO 21062 --- [ocp-schedules-18,69a9b2a2d16c8633,2a8e41a5f1c50815] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:49:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:49:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:49:40.053 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:40.055 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:40.984 WARN 21062 --- [ocp-async-4,e12cd1ea3156eab5,ad04de1e2dd94570] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:41.029 INFO 21062 --- [ocp-async-8,2b83d858557ad5e7,bb0bc55c9964e237] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:41.983 INFO 21062 --- [prometheus-minute-schedule-1,8879262c9956b7ba,0b01e4687a497dc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:42.984 WARN 21062 --- [ocp-async-8,c1d1bd24b5c621c1,77660678e0275843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:43.029 INFO 21062 --- [ocp-async-10,fbc833006bdc8fa1,33659e42aafd850d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:43.055 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:43.055 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f3758a265cdf2f98,53a9740c68dafcb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:44.985 WARN 21062 --- [ocp-async-10,344aec6b845d6b02,e5b2fd78de499b08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:45.030 INFO 21062 --- [ocp-async-6,f72333ae9e383601,73a145d9f3f31e77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:45.809 INFO 21062 --- [ocp-schedules-2,3daf0179ddd6b3b1,2dee3622890005a8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:49:46.056 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:46.056 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:46.985 WARN 21062 --- [ocp-async-6,05d65f2ca0f396cb,b388a6f486713aa5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:47.030 INFO 21062 --- [ocp-async-1,19dfd86ad6637bf8,990b8f02df54bc2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:47.983 INFO 21062 --- [prometheus-minute-schedule-1,ca5345d90cd1a4ce,185d9aead4d0592e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:49:48.589915+08:00, lastProcessTime=2024-11-04T13:49:38.590830+08:00 2024-11-04 13:49:48.621 INFO 21062 --- [ocp-async-7,11e2ff41e8f49460,44df1d608e857d9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:49:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:49:48.589915+08:00 2024-11-04 13:49:48.985 WARN 21062 --- [ocp-async-8,b4f0d7676a95f724,e3e5aba5ab22299e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:49.030 INFO 21062 --- [ocp-async-3,da429f927a9e56d8,7a3eebd5fc9c6175] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:49.056 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:49.057 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:49.877 INFO 21062 --- [ocp-schedules-6,a188b57758c9afc1,780014e7ff59e1a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:49:50.630 INFO 21062 --- [ocp-schedules-1,418290470ae525c3,cbd1f71337cba2b9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:49:50.637 INFO 21062 --- [ocp-schedules-1,418290470ae525c3,cbd1f71337cba2b9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:49:50.637 INFO 21062 --- [ocp-schedules-1,418290470ae525c3,cbd1f71337cba2b9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:49:50.983 INFO 21062 --- [prometheus-minute-schedule-1,e6734da9ac48dabc,f6be91a5483af0da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:50.985 WARN 21062 --- [ocp-async-10,8bf7eb598406d64d,f462507baeb72310] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:51.030 INFO 21062 --- [ocp-async-7,4eda20cc7cc1968a,d880770cf93d6293] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:52.057 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:52.058 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:52.985 WARN 21062 --- [ocp-async-6,9aa7088528c3a554,1b97c20298e09223] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:53.030 INFO 21062 --- [ocp-async-1,68b4711713fc084e,cbcb6bda8b35ceac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d9fb202137c8e06a,494ad4aff58ad0ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:54.986 WARN 21062 --- [ocp-async-5,7ae169d48a2ad143,8e9bf65b1497ddb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:55.030 INFO 21062 --- [ocp-async-9,08458dcd381ad97c,9def77c9108a7ccb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:55.058 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:55.059 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:55.848 INFO 21062 --- [ocp-schedules-9,4cce25efb34073d0,3bb6b26cbb870e91] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:49:56.983 INFO 21062 --- [prometheus-minute-schedule-1,71509d9c651dfe1e,e2919b721889101e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:49:56.986 WARN 21062 --- [ocp-async-3,bf91912f2d55148b,c577f3806fb8dba3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:57.031 INFO 21062 --- [ocp-async-2,3c086aae9b2efaa8,bd769a2851b618f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:58.059 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:49:58.060 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:49:58.500 INFO 21062 --- [ocp-schedules-21,babb58261f18c51e,465857a8b6696e4b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:49:58.500 INFO 21062 --- [ocp-schedules-13,07febc53e8de1bf3,1aed63d439fb1d41] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:49:58.502 INFO 21062 --- [ocp-schedules-4,adacdedbef2cbfb4,98288bf8fad6426a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:49:58.504 INFO 21062 --- [ocp-schedules-21,babb58261f18c51e,465857a8b6696e4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:49:58.505 INFO 21062 --- [ocp-schedules-13,07febc53e8de1bf3,1aed63d439fb1d41] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:49:58.505 INFO 21062 --- [ocp-schedules-20,7b052edd1f96e51d,b71e8b6e84ba1445] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:49:58.510 INFO 21062 --- [ocp-schedules-15,2be4a3de6900fa83,444eaa196df4320b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:49:58.510 INFO 21062 --- [ocp-schedules-15,2be4a3de6900fa83,444eaa196df4320b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:49:58.510 INFO 21062 --- [ocp-schedules-10,b5fdc925c81311eb,8ec06a02d3d42cde] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:49:58.510 INFO 21062 --- [ocp-schedules-10,b5fdc925c81311eb,8ec06a02d3d42cde] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:49:58.511 INFO 21062 --- [ocp-schedules-27,34c493cf18151321,5db2c73f2262a3fd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:49:58.511 INFO 21062 --- [ocp-schedules-27,34c493cf18151321,5db2c73f2262a3fd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:49:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:49:58.589344+08:00, lastProcessTime=2024-11-04T13:49:48.589915+08:00 2024-11-04 13:49:58.622 INFO 21062 --- [ocp-async-9,dd4d685b0f7dea94,4deb3c61e88cbb3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:49:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:49:58.589344+08:00 2024-11-04 13:49:58.976 INFO 21062 --- [ocp-schedules-16,d1c6e089a88167d2,39bf8e80cfc55073] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:49:58.986 WARN 21062 --- [ocp-async-6,a910e5f7ecb6d3ae,12efb3893228d6f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:49:59.031 INFO 21062 --- [ocp-async-1,c4748f0ffe69b8b0,dc54545069116dbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:49:59.880 INFO 21062 --- [ocp-schedules-2,179f040214e13f63,20673349c3d14ad0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:49:59.983 INFO 21062 --- [prometheus-minute-schedule-1,71a0064eee4a9437,aa3f38335ef3e56f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699400, interval=60 2024-11-04 13:50:00.506 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0fb971ab9e566577] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:45:01.0, nextRun=2024-11-04 13:50:00.0, currentDate=Mon Nov 04 13:50:00 CST 2024 2024-11-04 13:50:00.506 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0fb971ab9e566577] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:45:01.0, nextRunTime=2024-11-04 13:50:00.0} 2024-11-04 13:50:00.534 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0fb971ab9e566577] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:50:00.539 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0fb971ab9e566577] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:50:00.539 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0fb971ab9e566577] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:50:00.986 WARN 21062 --- [ocp-async-5,60dad2fbef36fad7,e2d76469eb6b3fcc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:01.031 INFO 21062 --- [ocp-async-9,7bd814d335a8407e,5d6f9e920b9ef267] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:01.060 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:01.061 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1417a23b7b5cf2d0,d3e2bda4d2bfc018] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:02.986 WARN 21062 --- [ocp-async-3,5053ebcd8b74b512,f1c54250312ca613] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:03.031 INFO 21062 --- [ocp-async-2,68af054bd6fab990,8cbcae2cf8c40925] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:03.812 INFO 21062 --- [ocp-schedules-12,59666bff8d004965,45cd09c3db29dbdd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:50:03.812 INFO 21062 --- [ocp-schedules-12,3f515bd2498ef46f,092d6172604a4156] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:50:03.816 INFO 21062 --- [ocp-schedules-12,3f515bd2498ef46f,092d6172604a4156] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:03.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:50:03.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:50:04.061 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:04.062 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:04.612 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,c4b61c0f576c03f3] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:40:05.0, nextRun=2024-11-04 13:50:04.0, currentDate=Mon Nov 04 13:50:04 CST 2024 2024-11-04 13:50:04.612 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,c4b61c0f576c03f3] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 13:40:05.0, nextRunTime=2024-11-04 13:50:04.0} 2024-11-04 13:50:04.618 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,c4b61c0f576c03f3] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 13:50:04.618 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,c4b61c0f576c03f3] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T13:50:04.618+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 13:50:04.986 WARN 21062 --- [ocp-async-7,44deab20bf84dc26,cc87b2f67ad4d735] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:05.031 INFO 21062 --- [ocp-async-4,220067bda4d6d14e,23e61091e19be634] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:05.493 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,58b21726722c2c07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 76, operation:EXECUTE, state: RUNNING 2024-11-04 13:50:05.513 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=76, context=Context{parallelIdx=-1, stringMap={task_instance_id=11, task_operation=execute, latest_execution_start_time=2024-11-04T13:50:05.486+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=76}, listMap={}}, executor=10.46.128.24 2024-11-04 13:50:05.517 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:50:05.518 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:50:05.519 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.520 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:50:05.521 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.521 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.522 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.523 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:50:05.539 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.539 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.540 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.540 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:50:05.553 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.554 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.554 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.554 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:50:05.568 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.568 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.569 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.569 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:50:05.582 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.583 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.583 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.584 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:50:05.589 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.590 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.590 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.591 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:50:05.595 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.595 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.596 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.596 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:50:05.601 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.601 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.602 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.602 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:50:05.608 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.608 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.609 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.610 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:50:05.615 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.615 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.616 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.616 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:50:05.622 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.622 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.623 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.623 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:50:05.632 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.633 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.633 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.633 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:50:05.647 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.648 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.648 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.648 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:50:05.653 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.653 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.654 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.654 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:50:05.660 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.660 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.661 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.661 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:50:05.670 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.670 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.671 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.672 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:50:05.676 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.676 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.677 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.677 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:50:05.681 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.682 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.682 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.683 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:50:05.687 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.687 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.688 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.688 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:50:05.693 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.694 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.694 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.695 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:50:05.699 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.699 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.700 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.700 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:50:05.705 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.705 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.706 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.707 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:50:05.710 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.711 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.712 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.712 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:50:05.716 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.717 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.717 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.718 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:50:05.722 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.722 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.723 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.723 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:50:05.727 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.728 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.728 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.729 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:50:05.733 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.733 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.734 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.734 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:50:05.739 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.739 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.740 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.740 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:50:05.744 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.745 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.745 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.745 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:50:05.750 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.750 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.751 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.751 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:50:05.755 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.756 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.756 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.757 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:50:05.761 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.761 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.762 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.762 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:50:05.767 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.767 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.768 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.768 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:50:05.772 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.773 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.773 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.774 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:50:05.782 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.783 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.783 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.784 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:50:05.790 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.791 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.791 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.792 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:50:05.798 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.798 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.799 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.799 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:50:05.805 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.805 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.806 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.806 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:50:05.810 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.810 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.811 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.811 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:50:05.815 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.815 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.816 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.816 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:50:05.820 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.821 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.821 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.822 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:50:05.826 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.826 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.827 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.827 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:50:05.836 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.836 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.837 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.837 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:50:05.857 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.858 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.858 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.859 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:50:05.864 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.865 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.865 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.866 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:50:05.871 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.872 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.872 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.873 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:50:05.874 INFO 21062 --- [ocp-schedules-2,cc47d4130154ddaa,d6ddf68b53bd7506] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:50:05.879 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.879 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.880 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.881 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:50:05.885 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.885 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.886 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.886 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:50:05.892 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.892 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.893 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.893 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:50:05.898 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:50:05.898 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:50:05.899 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.900 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:50:05.901 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.901 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.902 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.903 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:50:05.909 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.909 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:50:05.910 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:50:05.910 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:50:05.916 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:50:05.924 INFO 21062 --- [subtask-executor9,dd193f8f67a5a423,f1c43a73b882fe90] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 76, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 13:50:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ab57d20913d8ceff,bea18285351b8666] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:06.079 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:11, status:SUCCESS 2024-11-04 13:50:06.080 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:11, status:SUCCESS 2024-11-04 13:50:06.987 WARN 21062 --- [ocp-async-4,4d986b2dc3ac5950,8df850b734fe75d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:07.032 INFO 21062 --- [ocp-async-8,3ed533ea1040d27c,77dbc64ca8dcbf73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:07.062 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:07.063 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:08.505 INFO 21062 --- [ocp-schedules-21,babb58261f18c51e,465857a8b6696e4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:50:08.505 INFO 21062 --- [ocp-schedules-13,07febc53e8de1bf3,1aed63d439fb1d41] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:50:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:50:08.590134+08:00, lastProcessTime=2024-11-04T13:49:58.589344+08:00 2024-11-04 13:50:08.622 INFO 21062 --- [ocp-async-4,16fdc4b794cb2a7d,492ce446171bf644] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:50:08.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:50:08.590134+08:00 2024-11-04 13:50:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8ea9111b9dc4de10,3da960d2ed33fb78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:08.987 WARN 21062 --- [ocp-async-9,8eef51ff8ffecd98,87bc2b29a200a7dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:09.032 INFO 21062 --- [ocp-async-2,b0a6f8148e7fbb45,dcbe4edbb149dcd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:09.883 INFO 21062 --- [ocp-schedules-26,bb41b9021fc58766,0282c4fa3e807069] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:50:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:50:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:50:10.063 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:10.063 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:50:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:50:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:50:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:50:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:50:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:50:10.640 INFO 21062 --- [ocp-schedules-9,2eb674ec8164268f,70c85638cb30f2cc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:50:10.646 INFO 21062 --- [ocp-schedules-9,2eb674ec8164268f,70c85638cb30f2cc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:50:10.646 INFO 21062 --- [ocp-schedules-9,2eb674ec8164268f,70c85638cb30f2cc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:50:10.987 WARN 21062 --- [ocp-async-2,aad70b59bfb909f5,7c50216598450261] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:11.032 INFO 21062 --- [ocp-async-4,73f444219def7930,5d6993b38ad3ef73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:11.353 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=4, affectRows=4 2024-11-04 13:50:11.353 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=4, validCount=4 2024-11-04 13:50:11.353 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=18, affectRows=18 2024-11-04 13:50:11.353 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=17, validCount=17 2024-11-04 13:50:11.353 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=30, affectRows=30 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=40, affectRows=40 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=21, affectRows=21 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=30, validCount=30 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=37, affectRows=37 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=21, validCount=21 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=43, affectRows=43 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=62, affectRows=62 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=40, validCount=40 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=37, validCount=37 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=43, validCount=43 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=61, validCount=61 2024-11-04 13:50:11.354 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=59, affectRows=59 2024-11-04 13:50:11.355 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699340, nodeCount=78, affectRows=78 2024-11-04 13:50:11.355 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=59, validCount=59 2024-11-04 13:50:11.355 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699340, nodeCount=77, validCount=77 2024-11-04 13:50:11.983 INFO 21062 --- [prometheus-minute-schedule-1,fc63f001cba50619,76ff62b915b2441d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:12.987 WARN 21062 --- [ocp-async-4,cc053df26d8558bb,1ba7f9c382c68b73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:13.032 INFO 21062 --- [ocp-async-8,ad75966df3a90eb9,e57a2b833b02278d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:13.064 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:13.064 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:13.984 INFO 21062 --- [prometheus-cache-monitor1,d248fda052775f80,b0c93107d86d3565] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1546,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:50:13.985 INFO 21062 --- [prometheus-cache-monitor1,d248fda052775f80,b0c93107d86d3565] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:50:14.983 INFO 21062 --- [prometheus-minute-schedule-1,cab5d88300599971,8ea2aeae4dbdfff5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:14.987 WARN 21062 --- [ocp-async-8,28e354109e4d6d96,e5528e880e97cae0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:15.032 INFO 21062 --- [ocp-async-10,b1e4b2eaf27f2b63,ae9e0f49ca15bdc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:15.916 INFO 21062 --- [ocp-schedules-23,36ff667bd068ba73,adcc9dc11eca5ee0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:50:16.064 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:16.065 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:16.988 WARN 21062 --- [ocp-async-10,a2718080a276a3e5,e4b9ad014ac05f8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:17.032 INFO 21062 --- [ocp-async-7,de7225a1f58ba1c2,cf206fd81d2fc76e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9b470950bbf695ba,d8e0d1ade27033d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:50:18.590781+08:00, lastProcessTime=2024-11-04T13:50:08.590134+08:00 2024-11-04 13:50:18.622 INFO 21062 --- [ocp-async-10,93f80450e5f4bd4b,bd29d10e6f5a4a55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:50:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:50:18.590781+08:00 2024-11-04 13:50:18.988 WARN 21062 --- [ocp-async-4,fc40971b502e96a9,736aac2c0a4e8381] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:19.033 INFO 21062 --- [ocp-async-8,ae6f609a1b9dc87a,fbb12c48fd2b116f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:19.065 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:19.066 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:19.886 INFO 21062 --- [ocp-schedules-10,1328db7b221845ef,39a02a8809925b19] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:50:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4bc484a5bffa3237,2bc6e4f7baec0b3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:20.988 WARN 21062 --- [ocp-async-8,c8342294c80b99b4,e83c4dc67b14d411] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:21.033 INFO 21062 --- [ocp-async-10,035fd3225828dd7f,60e05001f66d087b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:22.066 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:22.067 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:22.988 WARN 21062 --- [ocp-async-10,0330910e7fa84425,1226b07d7987afb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:23.033 INFO 21062 --- [ocp-async-7,b5f86434f0d4c1cd,25638490a35a8d88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:23.983 INFO 21062 --- [prometheus-minute-schedule-1,feb5568bd0779759,0696d07f66dde4ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:24.988 WARN 21062 --- [ocp-async-7,a9f13f946388b7df,49f9c97fcadf1373] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:25.033 INFO 21062 --- [ocp-async-1,9f97db13ec69b3f1,1eb42a59313fe188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:25.067 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:25.068 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:25.941 INFO 21062 --- [ocp-schedules-2,a0be56511867f9ca,bdac784488b05291] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:50:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e819809143c0b167,7a5c44fba926b06a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8532cfc87eb544d9,c426ce854a98f4be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:50:26.988 WARN 21062 --- [ocp-async-1,6a6df4691a262213,afc3ce0ff546275f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:27.034 INFO 21062 --- [ocp-async-3,473deeedb0955728,d0a4d33754e384c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:28.068 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:28.069 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:28.498 INFO 21062 --- [ocp-schedules-11,dff77d5d8a9f0c07,311f2d46fe2a2da6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:50:28.499 INFO 21062 --- [ocp-schedules-14,c8e8d2f9134ef8b7,6e50e8462c854540] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:50:28.500 INFO 21062 --- [ocp-schedules-16,f4da7618e1165d54,7682f37a52c8e252] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:50:28.500 INFO 21062 --- [ocp-schedules-9,84c4569a558619de,0c92a946bbb90c97] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:50:28.502 INFO 21062 --- [ocp-schedules-11,dff77d5d8a9f0c07,311f2d46fe2a2da6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:50:28.502 INFO 21062 --- [ocp-schedules-13,597a7d1d7aa77eb5,e69955ff35fd13d3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:50:28.503 INFO 21062 --- [ocp-schedules-16,f4da7618e1165d54,7682f37a52c8e252] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:50:28.504 INFO 21062 --- [ocp-schedules-9,84c4569a558619de,0c92a946bbb90c97] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:28.504 INFO 21062 --- [ocp-schedules-26,dcd319defba110d7,e5458c36ff2518e0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:50:28.505 INFO 21062 --- [ocp-schedules-14,c8e8d2f9134ef8b7,6e50e8462c854540] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:50:28.510 INFO 21062 --- [ocp-schedules-22,c8afbde8d7e188d3,d19900c7dbfd2be9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:50:28.510 INFO 21062 --- [ocp-schedules-28,b962208d513cadb6,2591dd7de66a6439] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:50:28.510 INFO 21062 --- [ocp-schedules-23,4d88411617c44ca3,038b0c7d520ce6fd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:50:28.510 INFO 21062 --- [ocp-schedules-22,c8afbde8d7e188d3,d19900c7dbfd2be9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:50:28.510 INFO 21062 --- [ocp-schedules-23,4d88411617c44ca3,038b0c7d520ce6fd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:50:28.510 INFO 21062 --- [ocp-schedules-28,b962208d513cadb6,2591dd7de66a6439] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:50:28.570 INFO 21062 --- [ocp-schedules-20,3aaa078e7d7681d2,08c37571c20b0b80] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1546,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:50:28.570 INFO 21062 --- [ocp-schedules-20,3aaa078e7d7681d2,08c37571c20b0b80] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:50:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:50:28.589450+08:00, lastProcessTime=2024-11-04T13:50:18.590781+08:00 2024-11-04 13:50:28.622 INFO 21062 --- [ocp-async-8,8e2d4f6a0d1d318b,11b11348fa31c020] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:50:28.699 INFO 21062 --- [ocp-schedules-2,f3cd4054697f5add,ec891e9ffca50a2a] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:50:28.699 INFO 21062 --- [ocp-schedules-2,f3cd4054697f5add,ec891e9ffca50a2a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:50:28.748 INFO 21062 --- [ocp-schedules-15,0166d12f34fa10c2,b262fac59f38cdaf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:50:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:50:28.589450+08:00 2024-11-04 13:50:28.877 INFO 21062 --- [ocp-schedules-1,efbf37d49540a736,e9da1ce32cdce704] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:50:28.881 INFO 21062 --- [ocp-schedules-1,efbf37d49540a736,e9da1ce32cdce704] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:28.881 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:50:28.943 INFO 21062 --- [ocp-schedules-13,47a142b0042532b7,62d0a1d5f42a60cb] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:50:28.943 INFO 21062 --- [ocp-schedules-13,47a142b0042532b7,62d0a1d5f42a60cb] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:50:28.978 INFO 21062 --- [ocp-schedules-26,57572b6734bbfede,3c68ac454e794507] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:50:28.988 WARN 21062 --- [ocp-async-6,128d7097e11bf46f,a483b680842d3802] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:29.014 INFO 21062 --- [ocp-schedules-28,e5076da0992c4dc1,834ce61b75a0e813] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:50:29.019 INFO 21062 --- [ocp-schedules-28,e5076da0992c4dc1,834ce61b75a0e813] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:50:29.019 INFO 21062 --- [ocp-schedules-28,e5076da0992c4dc1,834ce61b75a0e813] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:50:29.034 INFO 21062 --- [ocp-async-4,0b1f13e5c7c36d5f,09f148c89cb1fbc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:29.276 INFO 21062 --- [ocp-schedules-8,5371ddb2e1b5f668,30453b75c040137b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:50:29.276 INFO 21062 --- [ocp-schedules-8,5371ddb2e1b5f668,30453b75c040137b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:50:29.277 INFO 21062 --- [ocp-schedules-8,5371ddb2e1b5f668,30453b75c040137b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:50:29.277 INFO 21062 --- [ocp-schedules-8,5371ddb2e1b5f668,30453b75c040137b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:50:29.278 INFO 21062 --- [ocp-schedules-8,5371ddb2e1b5f668,30453b75c040137b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:50:29.278 INFO 21062 --- [ocp-schedules-8,5371ddb2e1b5f668,30453b75c040137b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:50:29.888 INFO 21062 --- [ocp-schedules-18,f2bd3e5d0f226969,ae739615c37fc342] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:50:29.983 INFO 21062 --- [prometheus-minute-schedule-1,455d161b462d3fe4,46b6f920475f84ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:50:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:50:30.649 INFO 21062 --- [ocp-schedules-8,6603ba409b443f2a,4e34c0141e34a1fa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:50:30.654 INFO 21062 --- [ocp-schedules-8,6603ba409b443f2a,4e34c0141e34a1fa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:50:30.654 INFO 21062 --- [ocp-schedules-8,6603ba409b443f2a,4e34c0141e34a1fa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:50:30.654 INFO 21062 --- [ocp-schedules-30,15ea4fdd821145ae,eb595d3e5d238674] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:50:30.654 INFO 21062 --- [ocp-schedules-30,15ea4fdd821145ae,eb595d3e5d238674] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:50:30.681 INFO 21062 --- [ocp-schedules-30,15ea4fdd821145ae,eb595d3e5d238674] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:50:30.681 INFO 21062 --- [ocp-schedules-30,15ea4fdd821145ae,eb595d3e5d238674] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:50:30.681 INFO 21062 --- [ocp-schedules-25,6d76485f192ce16e,ab59503cd1d92b8f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:50:30.826 INFO 21062 --- [ocp-schedules-7,34d8993733113cd5,c449cfa881198020] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:50:30.826+08:00, clusters=[1] 2024-11-04 13:50:30.827 INFO 21062 --- [ocp-schedules-7,34d8993733113cd5,c449cfa881198020] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:50:30.877 INFO 21062 --- [ocp-schedules-6,a2ce0a65f6b364c9,689702b4ce6244a3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:50:30.877 INFO 21062 --- [ocp-schedules-6,a2ce0a65f6b364c9,689702b4ce6244a3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:50:30.877 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:50:30.877 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:50:30.886 INFO 21062 --- [ocp-schedules-6,a2ce0a65f6b364c9,689702b4ce6244a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:50:30.886 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:50:30.887 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:50:30.888 WARN 21062 --- [ocp-schedules-6,a2ce0a65f6b364c9,689702b4ce6244a3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:50:30.888 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:50:30.888 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:50:30.889 INFO 21062 --- [ocp-schedules-6,a2ce0a65f6b364c9,689702b4ce6244a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:50:30.889 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:50:30.889 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:50:30.890 WARN 21062 --- [ocp-schedules-6,a2ce0a65f6b364c9,689702b4ce6244a3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:50:30.890 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:50:30.890 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:50:30.906 INFO 21062 --- [ocp-schedules-2,7eed70b134799016,7162fce880d245bb] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:50:30.906 ERROR 21062 --- [ocp-schedules-6,a2ce0a65f6b364c9,689702b4ce6244a3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:50:30.909 INFO 21062 --- [ocp-schedules-2,7eed70b134799016,7162fce880d245bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:30.966 ERROR 21062 --- [ocp-schedules-2,7eed70b134799016,7162fce880d245bb] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:50:30.966 INFO 21062 --- [ocp-schedules-2,7eed70b134799016,7162fce880d245bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:50:30.979 INFO 21062 --- [ocp-schedules-24,8778d3fa9e58f2d3,7c15b95088d69d1c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:50:30.979+08:00, clusters=[1] 2024-11-04 13:50:30.983 INFO 21062 --- [ocp-schedules-24,8778d3fa9e58f2d3,7c15b95088d69d1c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:50:30.989 WARN 21062 --- [ocp-async-4,3608ff1242b30942,a79b984e50c763ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:31.034 INFO 21062 --- [ocp-async-8,4be1d829b79d96ec,d02eb6a70fcce77a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:31.069 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:31.070 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b0d7717e72647ecc,205bd526719cd58f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:32.989 WARN 21062 --- [ocp-async-8,dbea63eae71aabf1,1ad064c5f6be2aab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:33.034 INFO 21062 --- [ocp-async-2,49505e34a593e547,ad8e19c2502aadf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:33.846 INFO 21062 --- [ocp-schedules-12,3f515bd2498ef46f,092d6172604a4156] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:50:33.846 INFO 21062 --- [ocp-schedules-12,c8743f1fa08cda93,a0b005e812b7cd02] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:50:33.850 INFO 21062 --- [ocp-schedules-12,c8743f1fa08cda93,a0b005e812b7cd02] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:33.879 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:50:33.879 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:50:34.070 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:34.071 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:34.989 WARN 21062 --- [ocp-async-2,3bf2ca8bee03d558,79890fe8468cde40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:35.034 INFO 21062 --- [ocp-async-7,acb86eb222f67fb9,0747a43f7809701d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:35.703 INFO 21062 --- [ocp-schedules-15,1fc14952ae8b35a1,327e02161fabaeea] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:50:35.711 INFO 21062 --- [ocp-schedules-15,1fc14952ae8b35a1,327e02161fabaeea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:35.797 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:50:35.801 ERROR 21062 --- [ocp-schedules-15,1fc14952ae8b35a1,327e02161fabaeea] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:50:35.801 INFO 21062 --- [ocp-schedules-15,1fc14952ae8b35a1,327e02161fabaeea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:50:35.983 INFO 21062 --- [ocp-schedules-20,79948f6e1fce2ad2,6b6ccedf74c0d33e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:50:35.983 INFO 21062 --- [prometheus-minute-schedule-1,05c069c2f92a4357,70d65f961cfca35e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:36.989 WARN 21062 --- [ocp-async-7,5080b65a94cb9049,6501554a46c44eed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:37.035 INFO 21062 --- [ocp-async-1,4851e4e49ebf9660,07341638249756c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:37.071 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:37.072 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:38.504 INFO 21062 --- [ocp-schedules-16,f4da7618e1165d54,7682f37a52c8e252] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:50:38.504 INFO 21062 --- [ocp-schedules-9,84c4569a558619de,0c92a946bbb90c97] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:50:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:50:38.588896+08:00, lastProcessTime=2024-11-04T13:50:28.589450+08:00 2024-11-04 13:50:38.622 INFO 21062 --- [ocp-async-7,289a8512a8deb557,12d16f18fdde6160] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:50:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:50:38.588896+08:00 2024-11-04 13:50:38.881 INFO 21062 --- [ocp-schedules-1,efbf37d49540a736,e9da1ce32cdce704] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:50:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e16c6e050191e4d5,b50e3a6537b8f986] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:38.989 WARN 21062 --- [ocp-async-3,02d7cc33f9248444,2db0748f5e2574fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:39.035 INFO 21062 --- [ocp-async-2,684785044d73b847,4920924043b664cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:39.892 INFO 21062 --- [ocp-schedules-21,1cfdcd3ec8b08bcb,5d148c0b0ac80763] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:50:40.002 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:50:40.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:50:40.072 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:40.073 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:40.990 WARN 21062 --- [ocp-async-2,2691646a334c19e6,b1e0f6b2e59b098d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:41.035 INFO 21062 --- [ocp-async-7,5f7dc8f98e23db0c,95116f6389556559] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:50:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:50:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:50:41.983 INFO 21062 --- [prometheus-minute-schedule-1,7f7488baf92e66f2,c4501fc7187cc572] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:42.990 WARN 21062 --- [ocp-async-7,fbfaeb8c4c91dfcd,1ebc91edfc957982] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:43.035 INFO 21062 --- [ocp-async-1,e348be92f8148547,8e9b86a00492ec19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:43.073 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:43.074 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6ad9fabd7fe1a76e,fe822c83a8ff4227] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:44.990 WARN 21062 --- [ocp-async-8,2c5dc0c327738d85,d91016956b1b07d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:45.035 INFO 21062 --- [ocp-async-10,2cc435373f2b487f,e2500302c4de5920] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:46.009 INFO 21062 --- [ocp-schedules-28,c12a10e6ce6d3ee7,fc9bb5ad920c6a55] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:50:46.074 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:46.074 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:46.990 WARN 21062 --- [ocp-async-6,88e54c10cf7d0766,2c140426d66bf6cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:47.035 INFO 21062 --- [ocp-async-5,8bb26c63dcaa749d,41bd879d2124d54c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2ac153b016a7b5d1,18531a2da20fd3e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:50:48.591491+08:00, lastProcessTime=2024-11-04T13:50:38.588896+08:00 2024-11-04 13:50:48.623 INFO 21062 --- [ocp-async-10,340e7617cbe3d40e,fe500ca2a3623e94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:50:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:50:48.591491+08:00 2024-11-04 13:50:48.990 WARN 21062 --- [ocp-async-7,4a97b7fb5097685b,4f611edd9e5794c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:49.036 INFO 21062 --- [ocp-async-1,bcfcd28a32640365,fdaeb9420f465da9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:49.075 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:49.075 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:49.894 INFO 21062 --- [ocp-schedules-3,e5bc9ee86c475db3,f20d1cfba9290e8f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:50:50.656 INFO 21062 --- [ocp-schedules-11,9118267ed1ad181f,335ccb21c02f377c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:50:50.662 INFO 21062 --- [ocp-schedules-11,9118267ed1ad181f,335ccb21c02f377c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:50:50.662 INFO 21062 --- [ocp-schedules-11,9118267ed1ad181f,335ccb21c02f377c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:50:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a79469db83f3b4ca,1d66ce0e261f3f22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:50.990 WARN 21062 --- [ocp-async-8,66b3cd610d1c338f,0c9e22e88a101557] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:51.036 INFO 21062 --- [ocp-async-10,bcae75f714fdbc27,4742d50aa2d79206] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:52.075 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:52.076 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:52.991 WARN 21062 --- [ocp-async-6,deb789035a3ddcc7,4d3107f25d1edbd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:53.036 INFO 21062 --- [ocp-async-5,f4f9b17e8b8720b2,b14451a5aeae30ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:53.983 INFO 21062 --- [prometheus-minute-schedule-1,8f83e46be499614b,b2a30eca15e7595c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:54.991 WARN 21062 --- [ocp-async-4,e92b8cda3c32aa42,f6ae217c3caa5c05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:55.036 INFO 21062 --- [ocp-async-9,90e30e4042508988,83c7a808e8bd0bc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:55.076 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:55.076 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:56.051 INFO 21062 --- [ocp-schedules-11,46cca672ce00bfc4,c9f73adcb88703d5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:50:56.983 INFO 21062 --- [prometheus-minute-schedule-1,119887a5d135d3ee,aea9fcfec31bc5d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:50:56.991 WARN 21062 --- [ocp-async-1,919fbcf1e7adbc7c,7c5ea2576c743d20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:57.036 INFO 21062 --- [ocp-async-3,b287021b177e7765,f3a31092308db27e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:58.077 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:50:58.078 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:50:58.500 INFO 21062 --- [ocp-schedules-19,18029d4e99cc146b,c049c47cb8db2341] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:50:58.500 INFO 21062 --- [ocp-schedules-11,8b7da5af237204df,6b3b040decbf89c2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:50:58.502 INFO 21062 --- [ocp-schedules-18,6465b1d2aef5280a,3d73456e69c95e7e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:50:58.503 INFO 21062 --- [ocp-schedules-19,18029d4e99cc146b,c049c47cb8db2341] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:58.504 INFO 21062 --- [ocp-schedules-11,8b7da5af237204df,6b3b040decbf89c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:50:58.504 INFO 21062 --- [ocp-schedules-2,fb592656396ac79a,3dd793a432093c15] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:50:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:50:58.512 INFO 21062 --- [ocp-schedules-22,e8d5256657f7b7ef,a2a1899509554295] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:50:58.512 INFO 21062 --- [ocp-schedules-20,f1c3c743c0a8c97d,05a5e921d675293c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:50:58.512 INFO 21062 --- [ocp-schedules-10,f020b65f7102fd98,d2f06f504f9d5712] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:50:58.512 INFO 21062 --- [ocp-schedules-22,e8d5256657f7b7ef,a2a1899509554295] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:50:58.512 INFO 21062 --- [ocp-schedules-10,f020b65f7102fd98,d2f06f504f9d5712] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:50:58.512 INFO 21062 --- [ocp-schedules-20,f1c3c743c0a8c97d,05a5e921d675293c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:50:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:50:58.590690+08:00, lastProcessTime=2024-11-04T13:50:48.591491+08:00 2024-11-04 13:50:58.623 INFO 21062 --- [ocp-async-9,b443c8204276e6de,836c4e9e97f33e0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:50:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:50:58.590690+08:00 2024-11-04 13:50:58.981 INFO 21062 --- [ocp-schedules-23,64497fdfc0e92126,98669c8f25627db6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:50:58.991 WARN 21062 --- [ocp-async-6,c446161efe4b2089,b899830bbaaa6891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:50:59.036 INFO 21062 --- [ocp-async-5,881f3422765fcb30,b0304f8e1408e9a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:50:59.897 INFO 21062 --- [ocp-schedules-22,73a9ad15a4ff5376,69be5bf753a24903] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:50:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ada33ae6b9301943,ef1b230331b43521] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699460, interval=60 2024-11-04 13:51:00.991 WARN 21062 --- [ocp-async-5,c6e1f8f648cf0eeb,3a8ac0201cec20f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:01.037 INFO 21062 --- [ocp-async-9,60845fdafb10e755,eab26fcc0c4e11e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:01.078 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:01.079 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:02.983 INFO 21062 --- [prometheus-minute-schedule-1,551911b6bf2d8a48,0e4d366b0b15331e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:02.992 WARN 21062 --- [ocp-async-1,600e77788dffb14e,c9dcd0c8277efca2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:03.037 INFO 21062 --- [ocp-async-3,7136f2e8f3fc9dd5,5a853301b21c3725] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:03.880 INFO 21062 --- [ocp-schedules-12,c8743f1fa08cda93,a0b005e812b7cd02] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:51:03.880 INFO 21062 --- [ocp-schedules-12,9ca4c60b60a51194,a5ca1f53a199fe9d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:51:03.885 INFO 21062 --- [ocp-schedules-12,9ca4c60b60a51194,a5ca1f53a199fe9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:03.916 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:51:03.916 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:51:04.079 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:04.080 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:04.992 WARN 21062 --- [ocp-async-3,3e7d286659a93ec6,806d00df37e63e23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:05.037 INFO 21062 --- [ocp-async-2,25f1bf0d8e3a276a,48a9d5c66b4f0507] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:05.983 INFO 21062 --- [prometheus-minute-schedule-1,04ac2013134ee6a9,b8b2e0ae1441c8d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:06.077 INFO 21062 --- [ocp-schedules-6,9385f305c35dd5a9,a4d5d2f0d5fa043d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:51:06.992 WARN 21062 --- [ocp-async-2,6eadbebb8a95dc33,a2f497f0d631ef0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:07.037 INFO 21062 --- [ocp-async-7,ed2bb278fb1d2ce4,9d76f1d227d7e69b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:07.080 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:07.081 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:08.504 INFO 21062 --- [ocp-schedules-11,8b7da5af237204df,6b3b040decbf89c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:51:08.505 INFO 21062 --- [ocp-schedules-19,18029d4e99cc146b,c049c47cb8db2341] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:51:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:51:08.589711+08:00, lastProcessTime=2024-11-04T13:50:58.590690+08:00 2024-11-04 13:51:08.623 INFO 21062 --- [ocp-async-2,b603283e9361c31b,6822a2e39757b91b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:51:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:51:08.589711+08:00 2024-11-04 13:51:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d566be04ff0533aa,45907a467db6186c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:08.992 WARN 21062 --- [ocp-async-9,866b439d0946a979,a65fb422c7a01304] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:09.038 INFO 21062 --- [ocp-async-3,a80427d8238ef92a,ca4b7960fa4cec8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:09.901 INFO 21062 --- [ocp-schedules-20,9919dc4543560c40,0dc72f3e2acbbaf1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:51:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:51:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:51:10.082 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:10.082 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:51:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:51:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:51:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:51:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:51:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:51:10.665 INFO 21062 --- [ocp-schedules-23,991022fdba329a1f,d78d661dc5712b73] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:51:10.672 INFO 21062 --- [ocp-schedules-23,991022fdba329a1f,d78d661dc5712b73] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:51:10.672 INFO 21062 --- [ocp-schedules-23,991022fdba329a1f,d78d661dc5712b73] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:51:10.993 WARN 21062 --- [ocp-async-3,5afb2ccccc868527,97a4a6a5158c1083] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:11.038 INFO 21062 --- [ocp-async-2,ca6c9fc3f5a3f5f0,aebcade6981c8f40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:11.356 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=11, affectRows=11 2024-11-04 13:51:11.357 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=11, validCount=11 2024-11-04 13:51:11.357 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=6, affectRows=6 2024-11-04 13:51:11.357 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=6, validCount=6 2024-11-04 13:51:11.357 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=34, affectRows=34 2024-11-04 13:51:11.357 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=34, validCount=34 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=44, affectRows=44 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=59, affectRows=59 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=22, affectRows=22 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=44, validCount=44 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=22, validCount=22 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=46, affectRows=46 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=59, validCount=59 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=72, affectRows=72 2024-11-04 13:51:11.358 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=46, validCount=46 2024-11-04 13:51:11.359 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=71, validCount=71 2024-11-04 13:51:11.359 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699400, nodeCount=98, affectRows=98 2024-11-04 13:51:11.359 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699400, nodeCount=96, validCount=96 2024-11-04 13:51:11.983 INFO 21062 --- [prometheus-minute-schedule-1,cf82f74c2efd962a,1098b663b212d679] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:12.993 WARN 21062 --- [ocp-async-2,58b18d1be02d7668,54683f525958aacd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:13.038 INFO 21062 --- [ocp-async-7,03a4a9d9931870ec,497f3ed90a80aa2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:13.083 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:13.084 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:13.984 INFO 21062 --- [prometheus-cache-monitor1,ecd7c1e607020dea,0e5ee332a2e3b8f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:51:13.984 INFO 21062 --- [prometheus-cache-monitor1,ecd7c1e607020dea,0e5ee332a2e3b8f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:51:14.755 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:47:14.731, end=2024-11-04T13:49:14.752} 2024-11-04 13:51:14.760 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:51:14.760 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:51:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5ff86a08cc6dbde2,668d2dca0a445ec2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:14.993 WARN 21062 --- [ocp-async-7,a5e954f342b8c6dd,38250f13100625d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:15.038 INFO 21062 --- [ocp-async-1,ceaa9693a3bcc437,3d315bdd2341eb53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:16.084 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:16.085 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:16.118 INFO 21062 --- [ocp-schedules-15,c77c84d165547da3,fa22331c019bd4ce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:51:16.993 WARN 21062 --- [ocp-async-1,f8eeee82cce3cd2e,02a56868783fde80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:17.038 INFO 21062 --- [ocp-async-6,e8e25885ca2afa26,cfad3c09724a4a26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7b845521a3f0d66e,4ed9d7efb3e2a4e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:51:18.590350+08:00, lastProcessTime=2024-11-04T13:51:08.589711+08:00 2024-11-04 13:51:18.623 INFO 21062 --- [ocp-async-1,148b1d64110e5c0c,f51f634f59bec9c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:51:18.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:51:18.590350+08:00 2024-11-04 13:51:18.993 WARN 21062 --- [ocp-async-2,2f0131050a8729be,7fb9c0dfe307c1ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:19.039 INFO 21062 --- [ocp-async-7,e7e6e88335097e1f,9388bf488899a8d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:19.085 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:19.086 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:19.904 INFO 21062 --- [ocp-schedules-26,a0c363cd288e6acf,fd051f123032082b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:51:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ff2adf05ce23dd9d,01e3e67a2087f921] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:20.994 WARN 21062 --- [ocp-async-7,1e161e20ee6eb1f2,fdd354316b29c844] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:21.039 INFO 21062 --- [ocp-async-1,8d465c94f17c7279,cf7d2b3bfd596521] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:21.219 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:51:21.219 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:51:21.228 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:51:21.228 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:51:21.228 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:51:21.228 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:51:21.234 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:51:21.234 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:51:21.234 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:51:21.234 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:51:21.270 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:51:21.278 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:51:21.278 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:51:21.279 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:51:21.279 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:51:21.279 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:51:21.287 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:51:21.821 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:51:22.086 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:22.087 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:22.396 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.161912143S 2024-11-04 13:51:22.396 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:51:22.493 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:51:22.494 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:51:22.523 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:51:22.585 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:51:22.585 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:51:22.585 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:51:22.585 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:51:22.585 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:51:22.586 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:51:22.596 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:51:22.813 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] o.s.boot.SpringApplication : Started application in 1.585 seconds (JVM running for 4896.492) 2024-11-04 13:51:22.813 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.585439434S 2024-11-04 13:51:22.994 WARN 21062 --- [ocp-async-1,5e193c8118296f76,ee1674bd8c4c6e5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:23.039 INFO 21062 --- [ocp-async-10,2875ac53dda40284,8a27b83dede3e0ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:23.280 INFO 21062 --- [ocp-schedules-27,c809af28622af46c,5689bdb91f572852] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:51:23.983 INFO 21062 --- [prometheus-minute-schedule-1,932aae8e7f151b9b,ea21871173825d35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:24.994 WARN 21062 --- [ocp-async-6,462cbb8efb1f0a20,babb0f58ec64c460] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:25.039 INFO 21062 --- [ocp-async-5,f2078d0130f0af64,0b7e19639937fbb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:25.088 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:25.088 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:26.142 INFO 21062 --- [ocp-schedules-5,7d1f2d0ba81ee626,b13b53c1269ea443] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:51:26.983 INFO 21062 --- [prometheus-minute-schedule-1,50cf57087980b80c,6e1f485abe5789e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:26.984 INFO 21062 --- [prometheus-hour-schedule-1,27df0777d44f235e,96ae5f17cd23e3ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:51:26.994 WARN 21062 --- [ocp-async-4,7dfb581ca8cc306c,3f89fcd1c0a9e48d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:27.039 INFO 21062 --- [ocp-async-9,187b8edf9750f1b9,2416d97b5e45efc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:28.088 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:28.089 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:28.498 INFO 21062 --- [ocp-schedules-13,26e91e12ae73860e,b986701b9270eedd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:51:28.499 INFO 21062 --- [ocp-schedules-28,587f4dcfb22a0b3e,059665613e527103] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:51:28.500 INFO 21062 --- [ocp-schedules-11,e4220a7d77ba3bf7,5a2a063ab3ad523e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:51:28.500 INFO 21062 --- [ocp-schedules-30,c226e5f0da2e59af,89705f9921370a37] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:51:28.502 INFO 21062 --- [ocp-schedules-13,26e91e12ae73860e,b986701b9270eedd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:51:28.502 INFO 21062 --- [ocp-schedules-26,dab8a6f9e4f6aaeb,734e9f639966b191] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:51:28.503 INFO 21062 --- [ocp-schedules-11,e4220a7d77ba3bf7,5a2a063ab3ad523e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:51:28.504 INFO 21062 --- [ocp-schedules-30,c226e5f0da2e59af,89705f9921370a37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:28.504 INFO 21062 --- [ocp-schedules-28,587f4dcfb22a0b3e,059665613e527103] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:51:28.504 INFO 21062 --- [ocp-schedules-8,a16568465c90781e,debdc76f06a30d2f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:51:28.509 INFO 21062 --- [ocp-schedules-9,3c1946ea37aa4d6e,46b1cf35b2c0a2ce] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:51:28.509 INFO 21062 --- [ocp-schedules-9,3c1946ea37aa4d6e,46b1cf35b2c0a2ce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:51:28.510 INFO 21062 --- [ocp-schedules-5,bff2b9941cc1e3bf,346a2f12705b10ec] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:51:28.510 INFO 21062 --- [ocp-schedules-5,bff2b9941cc1e3bf,346a2f12705b10ec] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:51:28.510 INFO 21062 --- [ocp-schedules-16,eb25b7184682dfd2,bf0ad5d4808e8659] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:51:28.510 INFO 21062 --- [ocp-schedules-16,eb25b7184682dfd2,bf0ad5d4808e8659] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:51:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:51:28.589081+08:00, lastProcessTime=2024-11-04T13:51:18.590350+08:00 2024-11-04 13:51:28.610 INFO 21062 --- [ocp-schedules-29,22cf2acd616d3adc,32b8b792c1303353] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:51:28.611 INFO 21062 --- [ocp-schedules-29,22cf2acd616d3adc,32b8b792c1303353] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:51:28.623 INFO 21062 --- [ocp-async-7,51f0fb599a7e4a70,06f7908fe13a4796] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:51:28.751 INFO 21062 --- [ocp-schedules-4,45fd41839ac1447c,7f4f0fb66a981bae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:51:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:51:28.589081+08:00 2024-11-04 13:51:28.948 INFO 21062 --- [ocp-schedules-18,28ab4ecae02f1e03,f5983c66eb92fa06] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:51:28.948 INFO 21062 --- [ocp-schedules-18,28ab4ecae02f1e03,f5983c66eb92fa06] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:51:28.984 INFO 21062 --- [ocp-schedules-26,458df35dd4e0c0ae,30929eba041f3236] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:51:28.994 WARN 21062 --- [ocp-async-10,59cf6d09ef4fc2f9,d583d1dfefe74f0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:29.019 INFO 21062 --- [ocp-schedules-5,db89b1c58d9b58f4,781f0acc6e0414f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:51:29.024 INFO 21062 --- [ocp-schedules-5,db89b1c58d9b58f4,781f0acc6e0414f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:51:29.024 INFO 21062 --- [ocp-schedules-5,db89b1c58d9b58f4,781f0acc6e0414f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:51:29.040 INFO 21062 --- [ocp-async-5,8b5949d8ec731bdb,3b208d25cc2ddffd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:29.278 INFO 21062 --- [ocp-schedules-2,4116af0b97a787d0,36bfb72445745f42] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:51:29.278 INFO 21062 --- [ocp-schedules-2,4116af0b97a787d0,36bfb72445745f42] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:51:29.284 INFO 21062 --- [ocp-schedules-2,4116af0b97a787d0,36bfb72445745f42] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:51:29.284 INFO 21062 --- [ocp-schedules-2,4116af0b97a787d0,36bfb72445745f42] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:51:29.289 INFO 21062 --- [ocp-schedules-2,4116af0b97a787d0,36bfb72445745f42] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:51:29.289 INFO 21062 --- [ocp-schedules-2,4116af0b97a787d0,36bfb72445745f42] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:51:29.907 INFO 21062 --- [ocp-schedules-15,81b7acd49231d2f9,be45fddfea2ce1e9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:51:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e0fe9bf3594259e1,10982223351238ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:51:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:51:30.676 INFO 21062 --- [ocp-schedules-23,b350c454addf229a,0a4cf4184498ad0a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:51:30.681 INFO 21062 --- [ocp-schedules-2,7f774d4a1a997584,5449619aff29e117] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:51:30.681 INFO 21062 --- [ocp-schedules-2,7f774d4a1a997584,5449619aff29e117] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:51:30.683 INFO 21062 --- [ocp-schedules-23,b350c454addf229a,0a4cf4184498ad0a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:51:30.683 INFO 21062 --- [ocp-schedules-23,b350c454addf229a,0a4cf4184498ad0a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:51:30.708 INFO 21062 --- [ocp-schedules-2,7f774d4a1a997584,5449619aff29e117] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:51:30.708 INFO 21062 --- [ocp-schedules-2,7f774d4a1a997584,5449619aff29e117] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:51:30.711 INFO 21062 --- [ocp-schedules-17,3c538ef3855ff50d,17bc20fd59f36088] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:51:30.853 INFO 21062 --- [ocp-schedules-14,2e3d468a617d75cf,bdd93f129cd0475b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:51:30.853+08:00, clusters=[1] 2024-11-04 13:51:30.855 INFO 21062 --- [ocp-schedules-14,2e3d468a617d75cf,bdd93f129cd0475b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:51:30.915 INFO 21062 --- [ocp-schedules-4,4135f6098bdc6b8e,0b70862945859ee5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:51:30.916 INFO 21062 --- [ocp-schedules-4,4135f6098bdc6b8e,0b70862945859ee5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:51:30.916 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:51:30.916 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:51:30.925 INFO 21062 --- [ocp-schedules-4,4135f6098bdc6b8e,0b70862945859ee5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:51:30.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:51:30.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:51:30.927 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:51:30.927 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:51:30.927 WARN 21062 --- [ocp-schedules-4,4135f6098bdc6b8e,0b70862945859ee5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:51:30.929 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:51:30.929 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:51:30.929 INFO 21062 --- [ocp-schedules-4,4135f6098bdc6b8e,0b70862945859ee5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:51:30.930 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:51:30.930 WARN 21062 --- [ocp-schedules-4,4135f6098bdc6b8e,0b70862945859ee5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:51:30.930 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:51:30.937 ERROR 21062 --- [ocp-schedules-4,4135f6098bdc6b8e,0b70862945859ee5] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:51:30.994 WARN 21062 --- [ocp-async-2,8529514be7beef7b,dc2098bbc5ab4005] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:31.008 INFO 21062 --- [ocp-schedules-25,466f5909d0c17064,bf6071413dc9d11a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:51:31.008+08:00, clusters=[1] 2024-11-04 13:51:31.012 INFO 21062 --- [ocp-schedules-25,466f5909d0c17064,bf6071413dc9d11a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:51:31.040 INFO 21062 --- [ocp-async-7,98b871cc32c4bb86,a994ed2248ded473] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:31.089 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:31.090 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:32.983 INFO 21062 --- [prometheus-minute-schedule-1,80ef96bc90515625,d88fce0524b6c625] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:32.995 WARN 21062 --- [ocp-async-9,8478468bd8b83846,e08606bb9d40c7fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:33.040 INFO 21062 --- [ocp-async-1,dace87c1da1ffdea,8e4ace51562c8025] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:33.916 INFO 21062 --- [ocp-schedules-12,9ca4c60b60a51194,a5ca1f53a199fe9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:51:33.916 INFO 21062 --- [ocp-schedules-12,f09447e661cd4992,2804124815b473a3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:51:33.921 INFO 21062 --- [ocp-schedules-12,f09447e661cd4992,2804124815b473a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:33.950 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:51:33.950 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:51:34.090 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:34.091 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:34.995 WARN 21062 --- [ocp-async-3,ad3d4161ce38e5b6,f7bc6d37d2919524] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:35.040 INFO 21062 --- [ocp-async-6,1e3d59f02561cfa8,4d8ae64126ca0a56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:35.801 INFO 21062 --- [ocp-schedules-26,7feb86c7125521e8,3935e61e4e0fdb4d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:51:35.805 INFO 21062 --- [ocp-schedules-26,7feb86c7125521e8,3935e61e4e0fdb4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:35.894 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:51:35.896 ERROR 21062 --- [ocp-schedules-26,7feb86c7125521e8,3935e61e4e0fdb4d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:51:35.896 INFO 21062 --- [ocp-schedules-26,7feb86c7125521e8,3935e61e4e0fdb4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:51:35.983 INFO 21062 --- [prometheus-minute-schedule-1,68f1049a63ffdd12,b66884bd5bd0eeeb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:36.183 INFO 21062 --- [ocp-schedules-3,c4b3b5e3004dd701,244115ba8e44d0c8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:51:36.995 WARN 21062 --- [ocp-async-5,c0fa37f945378b37,1eb140915c2a925b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:37.040 INFO 21062 --- [ocp-async-4,958fe6b8b8c8e703,9c2026fc39343adf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:37.091 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:37.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:38.504 INFO 21062 --- [ocp-schedules-11,e4220a7d77ba3bf7,5a2a063ab3ad523e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:51:38.504 INFO 21062 --- [ocp-schedules-30,c226e5f0da2e59af,89705f9921370a37] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:51:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:51:38.589919+08:00, lastProcessTime=2024-11-04T13:51:28.589081+08:00 2024-11-04 13:51:38.623 INFO 21062 --- [ocp-async-5,3a8631c165187fd9,a4d6920b97eab4cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:51:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:51:38.589919+08:00 2024-11-04 13:51:38.881 INFO 21062 --- [ocp-schedules-5,af988117cf753394,813e04bbdfe63998] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:51:38.885 INFO 21062 --- [ocp-schedules-5,af988117cf753394,813e04bbdfe63998] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:38.885 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:51:38.983 INFO 21062 --- [prometheus-minute-schedule-1,927218706f55c15c,187d25a29721fb41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:38.995 WARN 21062 --- [ocp-async-8,fb66fc1e4e92b763,5664f88dbf3f2d27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:39.040 INFO 21062 --- [ocp-async-3,4e6229962fb15473,83745ea60da9f3fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:39.910 INFO 21062 --- [ocp-schedules-22,6a69f9c9efe608fe,e740f6a8d605fee1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:51:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:51:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:51:40.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:40.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:40.995 WARN 21062 --- [ocp-async-10,55e67e4f192750e0,7898d16cdf192eaa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:41.041 INFO 21062 --- [ocp-async-5,c5c150dd0abb4273,6ee02181d0127415] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6f621a65e2a15eb8,b1951052b6fda4ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:42.996 WARN 21062 --- [ocp-async-5,a15e3bd06cef707c,a59302003b581d79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:43.041 INFO 21062 --- [ocp-async-7,fe65b8da2e27d44c,aff4e82b5b4b9c47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:43.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:43.093 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:44.983 INFO 21062 --- [prometheus-minute-schedule-1,035042e923294ecc,a19492613b7028c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:44.996 WARN 21062 --- [ocp-async-7,51d7990a39ff2787,fe0073c9e736baf7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:45.041 INFO 21062 --- [ocp-async-1,40a78f4c08c7e129,58727b01e8be4488] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:46.093 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:46.094 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:46.208 INFO 21062 --- [ocp-schedules-14,9a4f4551ec1ebafc,b89ae7dd5d1e0885] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:51:46.996 WARN 21062 --- [ocp-async-1,9cced9e60aad5b94,7e950337df8dded1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:47.041 INFO 21062 --- [ocp-async-6,55e10d0b6c88cca0,ebbf369c646c2d1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:47.983 INFO 21062 --- [prometheus-minute-schedule-1,946969aeaa3f5e9f,8a5a9f6d9785a802] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:51:48.591407+08:00, lastProcessTime=2024-11-04T13:51:38.589919+08:00 2024-11-04 13:51:48.624 INFO 21062 --- [ocp-async-1,f1e5c59a81ff0a6f,377caa17acde645a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:51:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:51:48.591407+08:00 2024-11-04 13:51:48.885 INFO 21062 --- [ocp-schedules-5,af988117cf753394,813e04bbdfe63998] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:51:48.996 WARN 21062 --- [ocp-async-5,9fee33b02a1e76f2,ceb4ddff3f8322b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:49.042 INFO 21062 --- [ocp-async-7,b4f4d1080261afe4,9b63747da1262525] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:49.094 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:49.095 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:49.912 INFO 21062 --- [ocp-schedules-9,5e2217b0c451c1f4,f8d195012844f8d6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:51:50.686 INFO 21062 --- [ocp-schedules-5,1e24bbdc2f3d30a3,87f07366d5092b6a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:51:50.692 INFO 21062 --- [ocp-schedules-5,1e24bbdc2f3d30a3,87f07366d5092b6a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:51:50.692 INFO 21062 --- [ocp-schedules-5,1e24bbdc2f3d30a3,87f07366d5092b6a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:51:50.983 INFO 21062 --- [prometheus-minute-schedule-1,3f0c0f622d0fb516,addfe7a267c5e704] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:50.997 WARN 21062 --- [ocp-async-7,95e4c614cfe71d15,b115ddda9caf6b8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:51.042 INFO 21062 --- [ocp-async-1,2834427170ad8239,ec4b29f7d0030a6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:52.095 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:52.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:52.997 WARN 21062 --- [ocp-async-1,da84a3b037251a53,40c5d5412afe515e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:53.042 INFO 21062 --- [ocp-async-6,4d98ce4ec09e42b3,5ae0fce09515d16e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:53.983 INFO 21062 --- [prometheus-minute-schedule-1,238d3a0fcd040add,06a02b37f51e1d51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:54.997 WARN 21062 --- [ocp-async-6,6bf41f03ca783193,1be90e7b76ff29dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:55.042 INFO 21062 --- [ocp-async-4,f05473930ede6070,47f2a09eefdb4a92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:55.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:55.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:56.250 INFO 21062 --- [ocp-schedules-10,49d6ae6a4587cdf9,ec0db8eb89daf9bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:51:56.983 INFO 21062 --- [prometheus-minute-schedule-1,867976c304e48df1,3dbd8d00c9137571] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:51:56.997 WARN 21062 --- [ocp-async-4,9d56d488ced78f74,7179a8851ae6600e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:57.042 INFO 21062 --- [ocp-async-8,0c0258bee4dfab90,e0cba8419d4c3ce8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:58.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:51:58.097 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:51:58.500 INFO 21062 --- [ocp-schedules-22,b45375852a2949d3,b320c36c366e3f00] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:51:58.500 INFO 21062 --- [ocp-schedules-3,0fe5c2e8f3e2974e,9566ce633ff8a63e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:51:58.502 INFO 21062 --- [ocp-schedules-28,0dddc82967233bd8,3c2620a76edf7bf9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:51:58.504 INFO 21062 --- [ocp-schedules-22,b45375852a2949d3,b320c36c366e3f00] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:58.504 INFO 21062 --- [ocp-schedules-3,0fe5c2e8f3e2974e,9566ce633ff8a63e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:51:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:51:58.505 INFO 21062 --- [ocp-schedules-6,e30309720aedf771,82b37c9efdf1016e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:51:58.512 INFO 21062 --- [ocp-schedules-25,1b9920b2ec1d2a28,cecbf49753856c20] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:51:58.512 INFO 21062 --- [ocp-schedules-26,3585b06de008c593,abd69a8acc59e8f5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:51:58.512 INFO 21062 --- [ocp-schedules-25,1b9920b2ec1d2a28,cecbf49753856c20] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:51:58.512 INFO 21062 --- [ocp-schedules-26,3585b06de008c593,abd69a8acc59e8f5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:51:58.512 INFO 21062 --- [ocp-schedules-17,ff3ee6876fe2da7b,cc2693038867fc16] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:51:58.512 INFO 21062 --- [ocp-schedules-17,ff3ee6876fe2da7b,cc2693038867fc16] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:51:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:51:58.591060+08:00, lastProcessTime=2024-11-04T13:51:48.591407+08:00 2024-11-04 13:51:58.624 INFO 21062 --- [ocp-async-2,f7bbe75c52eae55a,86619f6e48339e09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:51:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:51:58.591060+08:00 2024-11-04 13:51:58.987 INFO 21062 --- [ocp-schedules-16,e67a9c55392b2f88,558dcb2c232a0a7f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:51:58.997 WARN 21062 --- [ocp-async-1,835fde09760ba5f1,852570e45dd7c083] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:51:59.042 INFO 21062 --- [ocp-async-6,c8425d039a32b630,a9ba11e5a0a381c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:51:59.915 INFO 21062 --- [ocp-schedules-25,4bb0e707f651c558,5882490a0ce3eee3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:51:59.983 INFO 21062 --- [prometheus-minute-schedule-1,883c320f4f3e55f6,2ee95ccd6f639897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699520, interval=60 2024-11-04 13:52:00.997 WARN 21062 --- [ocp-async-6,df5d995d0bafeef7,e12a3ae4259445fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:01.043 INFO 21062 --- [ocp-async-2,d2456e33c61fc52d,60c8c78c5c5fcf0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:01.097 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:01.098 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:02.983 INFO 21062 --- [prometheus-minute-schedule-1,35cf5a921ed69a17,fba18d19c0bbdc58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:02.998 WARN 21062 --- [ocp-async-2,70571ab0f0179904,5a41ed37ab8e2dbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:03.043 INFO 21062 --- [ocp-async-8,560ec4b66f8cb566,80b91cd99813d3ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:03.950 INFO 21062 --- [ocp-schedules-12,f09447e661cd4992,2804124815b473a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:52:03.950 INFO 21062 --- [ocp-schedules-12,04e295bb484b6761,0c9521acab4a9aed] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:52:03.954 INFO 21062 --- [ocp-schedules-12,04e295bb484b6761,0c9521acab4a9aed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:03.984 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:52:03.984 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:52:04.098 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:04.099 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:04.998 WARN 21062 --- [ocp-async-8,7c47ba882c342b9e,4eb7b5603e70f668] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:05.043 INFO 21062 --- [ocp-async-3,1bc7a90f5c3e9f50,cd687e31f49f4576] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:05.983 INFO 21062 --- [prometheus-minute-schedule-1,80645f3cbb8279bc,054aa04ec4c82919] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:06.274 INFO 21062 --- [ocp-schedules-2,90ae67507860526f,f46cacde15a3ad12] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:52:06.998 WARN 21062 --- [ocp-async-3,96e8a81151cb5e49,98b1d5479a6ebab4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:07.043 INFO 21062 --- [ocp-async-4,99a7b46330edf200,773a129060e9b7f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:07.099 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:07.100 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:08.505 INFO 21062 --- [ocp-schedules-22,b45375852a2949d3,b320c36c366e3f00] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:52:08.505 INFO 21062 --- [ocp-schedules-3,0fe5c2e8f3e2974e,9566ce633ff8a63e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:52:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:52:08.589733+08:00, lastProcessTime=2024-11-04T13:51:58.591060+08:00 2024-11-04 13:52:08.624 INFO 21062 --- [ocp-async-10,2f4db0d2583f62e0,e9ce51075ec82854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:52:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:52:08.589733+08:00 2024-11-04 13:52:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a84d8cfa60c024fc,90a53280a7a99ee2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:08.998 WARN 21062 --- [ocp-async-2,4cc4d51ab2937a27,56cfa29c3ab2cacd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:09.043 INFO 21062 --- [ocp-async-8,689e19ff7974ef5a,ab52759b342c06e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:09.918 INFO 21062 --- [ocp-schedules-26,6382c1c51655d04c,a714085c64359f6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:52:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:52:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:52:10.100 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:10.101 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:52:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:52:10.307 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:52:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:52:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:52:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:52:10.694 INFO 21062 --- [ocp-schedules-8,68034910ceff9b3b,9f20cde1455b9271] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:52:10.699 INFO 21062 --- [ocp-schedules-8,68034910ceff9b3b,9f20cde1455b9271] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:52:10.699 INFO 21062 --- [ocp-schedules-8,68034910ceff9b3b,9f20cde1455b9271] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:52:10.998 WARN 21062 --- [ocp-async-8,5971df015afa121e,798cd370e767508b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:11.043 INFO 21062 --- [ocp-async-10,bb864537bd5cd8c1,9d023e5da3acedd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:11.367 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=5, affectRows=5 2024-11-04 13:52:11.367 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=5, validCount=5 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=9, affectRows=9 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=21, affectRows=21 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=9, validCount=9 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=21, validCount=21 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=12, affectRows=12 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=28, affectRows=28 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=46, affectRows=46 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=71, affectRows=71 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=12, validCount=12 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=65, affectRows=65 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=25, affectRows=25 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=28, validCount=28 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=25, validCount=25 2024-11-04 13:52:11.368 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=45, validCount=45 2024-11-04 13:52:11.369 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=37, affectRows=37 2024-11-04 13:52:11.369 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699460, nodeCount=73, affectRows=73 2024-11-04 13:52:11.369 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=69, validCount=69 2024-11-04 13:52:11.369 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=65, validCount=65 2024-11-04 13:52:11.369 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=37, validCount=37 2024-11-04 13:52:11.369 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699460, nodeCount=73, validCount=73 2024-11-04 13:52:11.983 INFO 21062 --- [prometheus-minute-schedule-1,2774a82bdc5ee603,3ca765da654555be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:12.998 WARN 21062 --- [ocp-async-10,057c4e97504717bf,43e44c9d7c36d4b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:13.044 INFO 21062 --- [ocp-async-5,70b29432e8771975,7307457e42dc7acd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:13.101 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:13.101 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:13.984 INFO 21062 --- [prometheus-cache-monitor1,df29fd1afad72745,f9c5f39a3e6bd5b2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:52:13.984 INFO 21062 --- [prometheus-cache-monitor1,df29fd1afad72745,f9c5f39a3e6bd5b2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:52:14.983 INFO 21062 --- [prometheus-minute-schedule-1,c9719dee3bb62032,5bc0056cfbf58a6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:14.999 WARN 21062 --- [ocp-async-5,ba3ed53f909beff0,93591182fe6a3945] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:15.044 INFO 21062 --- [ocp-async-7,8908b89f464d2d3e,7911952b8dd81009] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:16.101 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:16.102 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:16.315 INFO 21062 --- [ocp-schedules-7,ee9aab4362fdf760,86e1fdbc475d62cb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:52:16.999 WARN 21062 --- [ocp-async-7,dad78f4cc2e2793d,5eac2eea9212fe9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:17.044 INFO 21062 --- [ocp-async-1,a648ce173e76f95f,ac301e53175be9f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8a406be17255b60e,a6a0aabf2362964b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:52:18.590281+08:00, lastProcessTime=2024-11-04T13:52:08.589733+08:00 2024-11-04 13:52:18.624 INFO 21062 --- [ocp-async-9,e99862d1373707bc,a9642a9d22ab76e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:52:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:52:18.590281+08:00 2024-11-04 13:52:18.999 WARN 21062 --- [ocp-async-10,12d8a45035a3fc87,02c877f7f7802707] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:19.044 INFO 21062 --- [ocp-async-4,1fd9e84163399bd1,314e8a1e208ebe96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:19.102 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:19.103 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:19.921 INFO 21062 --- [ocp-schedules-10,20d5ffdb6e061e0f,88eed8967f185350] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:52:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5901048e17d7adc8,3ac49de1d054c86e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:20.999 WARN 21062 --- [ocp-async-5,af2d82b22aedd563,0102db728880c549] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:21.044 INFO 21062 --- [ocp-async-9,0bed99251b5739f1,b8c32c9c82a0cc6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:22.103 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:22.105 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:22.999 WARN 21062 --- [ocp-async-7,82dd8d8ae6c3f3c4,38abecf6c35effb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:23.045 INFO 21062 --- [ocp-async-1,d12306c86bfb8c24,54cad52b1cffafba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:23.983 INFO 21062 --- [prometheus-minute-schedule-1,24ac414fae133570,ca213182f0065116] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:25.000 WARN 21062 --- [ocp-async-3,4d4a25d3f683aa56,16c4ffc050ddff4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:25.045 INFO 21062 --- [ocp-async-6,a51f2bf26486ed47,9e0db91692ca2cd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:25.105 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:25.106 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:26.340 INFO 21062 --- [ocp-schedules-10,64bab444d3c96cb8,20826206ba604e99] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:52:26.983 INFO 21062 --- [prometheus-minute-schedule-1,0e91ede375c17ca9,02aaea0eab4e266d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:26.984 INFO 21062 --- [prometheus-hour-schedule-1,92ce31c92a950bbb,f265475e5d6559c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:52:27.000 WARN 21062 --- [ocp-async-4,c461914c804f62ef,ce4ad89eef5fd27e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:27.045 INFO 21062 --- [ocp-async-2,8091911b7dad2a83,d727cad6aac8d6b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:28.106 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:28.107 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:28.498 INFO 21062 --- [ocp-schedules-30,e905515e4cbcb16c,ba9eff1313f08921] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:52:28.499 INFO 21062 --- [ocp-schedules-29,f9962f4956265045,bc9fe8350cf60f42] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:52:28.499 INFO 21062 --- [ocp-schedules-13,096809b5999e1178,60a3fa30691e7ce4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:52:28.500 INFO 21062 --- [ocp-schedules-8,42420a1d0e37fca7,722b53a43c6acfd9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:52:28.502 INFO 21062 --- [ocp-schedules-5,654e0e1b0cec3797,d96366f321149bfc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:52:28.503 INFO 21062 --- [ocp-schedules-30,e905515e4cbcb16c,ba9eff1313f08921] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:52:28.504 INFO 21062 --- [ocp-schedules-13,096809b5999e1178,60a3fa30691e7ce4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:28.504 INFO 21062 --- [ocp-schedules-8,42420a1d0e37fca7,722b53a43c6acfd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:52:28.504 INFO 21062 --- [ocp-schedules-23,4077f3ddf52686b4,0f06223caaa623ed] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:52:28.504 INFO 21062 --- [ocp-schedules-2,186b15753fdb88ac,6e3e43dc8c410084] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:52:28.505 INFO 21062 --- [ocp-schedules-29,f9962f4956265045,bc9fe8350cf60f42] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:52:28.507 INFO 21062 --- [ocp-schedules-2,186b15753fdb88ac,6e3e43dc8c410084] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:28.510 INFO 21062 --- [ocp-schedules-25,1af595c87935119e,b6b7584e1a2566a9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:52:28.510 INFO 21062 --- [ocp-schedules-19,d21ecd455aa55f9e,0dca4533aa9bedf8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:52:28.510 INFO 21062 --- [ocp-schedules-25,1af595c87935119e,b6b7584e1a2566a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:52:28.510 INFO 21062 --- [ocp-schedules-26,1d5ab143bcf61085,f9e6c04db81d9ef5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:52:28.510 INFO 21062 --- [ocp-schedules-19,d21ecd455aa55f9e,0dca4533aa9bedf8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:52:28.510 INFO 21062 --- [ocp-schedules-26,1d5ab143bcf61085,f9e6c04db81d9ef5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:52:28.533 INFO 21062 --- [ocp-schedules-22,cc2ef4502c01378d,71efb18cd16f14eb] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:52:28.533 INFO 21062 --- [ocp-schedules-22,cc2ef4502c01378d,71efb18cd16f14eb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:52:28.570 INFO 21062 --- [ocp-schedules-20,982fa86a1c38942a,399b985c4dab8de0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:52:28.570 INFO 21062 --- [ocp-schedules-20,982fa86a1c38942a,399b985c4dab8de0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:52:28.576 INFO 21062 --- [pool-12-thread-2,4843bc4cc6f9b686,409204d5f03a6449] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:52:28.576 INFO 21062 --- [pool-12-thread-3,4843bc4cc6f9b686,306d6e086ebc2a7e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:52:28.576 INFO 21062 --- [pool-12-thread-4,4843bc4cc6f9b686,4b69b6252c1b490d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:52:28.576 INFO 21062 --- [pool-12-thread-1,4843bc4cc6f9b686,89c2a99ec461f055] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:52:28.581 INFO 21062 --- [ForkJoinPool-11-worker-8,4843bc4cc6f9b686,c15a0268d1e1f49d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:52:28.581 INFO 21062 --- [ForkJoinPool-11-worker-8,4843bc4cc6f9b686,c15a0268d1e1f49d] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:52:28.581 INFO 21062 --- [ForkJoinPool-11-worker-8,4843bc4cc6f9b686,c15a0268d1e1f49d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:52:28.581 INFO 21062 --- [pool-12-thread-2,4843bc4cc6f9b686,409204d5f03a6449] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:52:28.585 INFO 21062 --- [pool-12-thread-1,4843bc4cc6f9b686,89c2a99ec461f055] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:52:28.585 INFO 21062 --- [pool-12-thread-1,4843bc4cc6f9b686,89c2a99ec461f055] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:52:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:52:28.589125+08:00, lastProcessTime=2024-11-04T13:52:18.590281+08:00 2024-11-04 13:52:28.604 INFO 21062 --- [pool-12-thread-3,4843bc4cc6f9b686,306d6e086ebc2a7e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:52:28.607 INFO 21062 --- [pool-12-thread-3,4843bc4cc6f9b686,306d6e086ebc2a7e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:52:28.609 INFO 21062 --- [ForkJoinPool-11-worker-8,4843bc4cc6f9b686,ec5099a00f612f0b] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:52:28.609 INFO 21062 --- [pool-12-thread-4,4843bc4cc6f9b686,4b69b6252c1b490d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:52:28.611 INFO 21062 --- [pool-12-thread-3,4843bc4cc6f9b686,306d6e086ebc2a7e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:52:28.624 INFO 21062 --- [ocp-async-5,81b3c0cac3c45de4,7824cdb2cc700db8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:52:28.748 INFO 21062 --- [ocp-schedules-29,c0bc3833b7b3a82b,67f4f8a43227ff59] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 13:52:28.749 INFO 21062 --- [ocp-schedules-29,c0bc3833b7b3a82b,67f4f8a43227ff59] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 13:52:28.749 INFO 21062 --- [ocp-schedules-29,c0bc3833b7b3a82b,67f4f8a43227ff59] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=392 2024-11-04 13:52:28.753 INFO 21062 --- [ocp-schedules-5,f2a7df23952419e0,0870d13800a9d57e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:52:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:52:28.589125+08:00 2024-11-04 13:52:28.768 INFO 21062 --- [ocp-schedules-23,375b0b2c99c8bf9d,43694133ecf921a6] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:52:28.772 INFO 21062 --- [ocp-schedules-29,c0bc3833b7b3a82b,67f4f8a43227ff59] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=392 2024-11-04 13:52:28.781 INFO 21062 --- [ocp-schedules-23,375b0b2c99c8bf9d,43694133ecf921a6] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:52:28.935 INFO 21062 --- [ocp-schedules-30,80c6cc5dc50c82b6,22fbe5e4eebc4d20] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 13:52:28.939 INFO 21062 --- [ocp-schedules-30,80c6cc5dc50c82b6,22fbe5e4eebc4d20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:28.952 INFO 21062 --- [ocp-schedules-22,44e2e1db123867d9,c42e5e00d2af58cf] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:52:28.952 INFO 21062 --- [ocp-schedules-22,44e2e1db123867d9,c42e5e00d2af58cf] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:52:28.973 ERROR 21062 --- [ocp-schedules-30,80c6cc5dc50c82b6,22fbe5e4eebc4d20] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:52:28.973 INFO 21062 --- [ocp-schedules-30,80c6cc5dc50c82b6,22fbe5e4eebc4d20] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 13:52:28.989 INFO 21062 --- [ocp-schedules-18,278edf1a1cd96585,c02c60e813d60ca3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:52:29.000 WARN 21062 --- [ocp-async-1,d75d0dba9c97e636,9da7a1e7c1bec289] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:29.024 INFO 21062 --- [ocp-schedules-27,7f190536668e277e,9d79319d7a499d15] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:52:29.029 INFO 21062 --- [ocp-schedules-27,7f190536668e277e,9d79319d7a499d15] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:52:29.029 INFO 21062 --- [ocp-schedules-27,7f190536668e277e,9d79319d7a499d15] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:52:29.045 INFO 21062 --- [ocp-async-6,79a4990bb1597726,1b01f3b57b7319de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:29.106 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:52:29.106 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:52:29.114 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:52:29.114 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:52:29.114 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:52:29.125 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:52:29.125 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:52:29.125 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:52:29.128 INFO 21062 --- [ocp-schedules-1,3dc5c8d9396fae37,adcc7d7ad280df8a] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:52:29.294 INFO 21062 --- [ocp-schedules-6,2527731ddea39157,eedd034084265a9f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:52:29.294 INFO 21062 --- [ocp-schedules-6,2527731ddea39157,eedd034084265a9f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:52:29.295 INFO 21062 --- [ocp-schedules-6,2527731ddea39157,eedd034084265a9f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:52:29.295 INFO 21062 --- [ocp-schedules-6,2527731ddea39157,eedd034084265a9f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:52:29.297 INFO 21062 --- [ocp-schedules-6,2527731ddea39157,eedd034084265a9f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:52:29.297 INFO 21062 --- [ocp-schedules-6,2527731ddea39157,eedd034084265a9f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:52:29.410 INFO 21062 --- [ocp-schedules-24,405074f95d7986a0,36d98e347b888e02] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:52:29.410 INFO 21062 --- [ocp-schedules-24,405074f95d7986a0,36d98e347b888e02] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:52:29.438 INFO 21062 --- [pool-35-thread-1,405074f95d7986a0,b5d778d578b2d20e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:52:29.466 WARN 21062 --- [ocp-schedules-24,405074f95d7986a0,36d98e347b888e02] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:52:29.466 INFO 21062 --- [ocp-schedules-24,405074f95d7986a0,36d98e347b888e02] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:52:29.924 INFO 21062 --- [ocp-schedules-25,469873635710c933,c32dc967cd4eff92] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:52:29.983 INFO 21062 --- [prometheus-minute-schedule-1,50cb38f23a68f016,0cd153ea55ffc47f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:52:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:52:30.702 INFO 21062 --- [ocp-schedules-1,275de19ba55c83ff,c09af08dd163d63c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:52:30.707 INFO 21062 --- [ocp-schedules-1,275de19ba55c83ff,c09af08dd163d63c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:52:30.707 INFO 21062 --- [ocp-schedules-1,275de19ba55c83ff,c09af08dd163d63c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:52:30.709 INFO 21062 --- [ocp-schedules-10,0b0ac466240e8dc1,ee1fe99110e8ed58] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:52:30.709 INFO 21062 --- [ocp-schedules-10,0b0ac466240e8dc1,ee1fe99110e8ed58] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:52:30.734 INFO 21062 --- [ocp-schedules-10,0b0ac466240e8dc1,ee1fe99110e8ed58] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:52:30.734 INFO 21062 --- [ocp-schedules-10,0b0ac466240e8dc1,ee1fe99110e8ed58] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:52:30.736 INFO 21062 --- [ocp-schedules-24,ddffa91fba1d137d,56910199331c0156] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:52:30.883 INFO 21062 --- [ocp-schedules-7,13ee5d7e1c3a6425,610a302740d62faf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:52:30.883+08:00, clusters=[1] 2024-11-04 13:52:30.885 INFO 21062 --- [ocp-schedules-7,13ee5d7e1c3a6425,610a302740d62faf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:52:30.946 INFO 21062 --- [ocp-schedules-26,ffb02a0c150ea05d,2c6fa4cf7276fb9e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:52:30.946 INFO 21062 --- [ocp-schedules-26,ffb02a0c150ea05d,2c6fa4cf7276fb9e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:52:30.946 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:52:30.946 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:52:30.956 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:52:30.957 INFO 21062 --- [ocp-schedules-26,ffb02a0c150ea05d,2c6fa4cf7276fb9e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:52:30.957 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:52:30.958 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:52:30.958 WARN 21062 --- [ocp-schedules-26,ffb02a0c150ea05d,2c6fa4cf7276fb9e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:52:30.959 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:52:30.959 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:52:30.959 INFO 21062 --- [ocp-schedules-26,ffb02a0c150ea05d,2c6fa4cf7276fb9e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:52:30.959 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:52:30.960 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:52:30.960 WARN 21062 --- [ocp-schedules-26,ffb02a0c150ea05d,2c6fa4cf7276fb9e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:52:30.961 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:52:30.967 INFO 21062 --- [ocp-schedules-9,5170da616935dccb,e09e9120e6cf43a6] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:52:30.969 ERROR 21062 --- [ocp-schedules-26,ffb02a0c150ea05d,2c6fa4cf7276fb9e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:52:30.972 INFO 21062 --- [ocp-schedules-9,5170da616935dccb,e09e9120e6cf43a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:31.000 WARN 21062 --- [ocp-async-10,466d3351f522b386,e097cdc4f6fd1c06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:31.030 ERROR 21062 --- [ocp-schedules-9,5170da616935dccb,e09e9120e6cf43a6] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:52:31.030 INFO 21062 --- [ocp-schedules-9,5170da616935dccb,e09e9120e6cf43a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:52:31.035 INFO 21062 --- [ocp-schedules-25,63f3379ad54d0020,78cd2758fc23a630] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:52:31.035+08:00, clusters=[1] 2024-11-04 13:52:31.039 INFO 21062 --- [ocp-schedules-25,63f3379ad54d0020,78cd2758fc23a630] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:52:31.045 INFO 21062 --- [ocp-async-5,2657b0fc817eff16,403c4cb797e4d4f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:31.107 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:31.108 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7e9294ba3142ddf9,d4dfe11947353a02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:33.000 WARN 21062 --- [ocp-async-2,f6f810eb1fdd72fc,62f2917d901d8f53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:33.046 INFO 21062 --- [ocp-async-7,dfb824d4359a0d9f,1abc41b9bb4f5090] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:33.984 INFO 21062 --- [ocp-schedules-12,04e295bb484b6761,0c9521acab4a9aed] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:52:33.984 INFO 21062 --- [ocp-schedules-12,4a6f00b115450833,0788df95f9508e80] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:52:33.988 INFO 21062 --- [ocp-schedules-12,4a6f00b115450833,0788df95f9508e80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:34.017 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:52:34.017 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:52:34.108 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:34.109 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:35.000 WARN 21062 --- [ocp-async-7,c0fb8ea315cc9a9d,1839be71444ab723] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:35.046 INFO 21062 --- [ocp-async-3,e66ecfe6f81c9c10,060609f53c692d44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:35.896 INFO 21062 --- [ocp-schedules-3,fc1a07645ed61c71,7bccff293755b9a5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:52:35.900 INFO 21062 --- [ocp-schedules-3,fc1a07645ed61c71,7bccff293755b9a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:35.983 INFO 21062 --- [prometheus-minute-schedule-1,7de7bed9e146cc84,75371cad424034d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:35.985 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:52:35.989 ERROR 21062 --- [ocp-schedules-3,fc1a07645ed61c71,7bccff293755b9a5] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:52:35.989 INFO 21062 --- [ocp-schedules-3,fc1a07645ed61c71,7bccff293755b9a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:52:36.420 INFO 21062 --- [ocp-schedules-15,7b9d9776df8039eb,6cbfce30e5633a88] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:52:37.001 WARN 21062 --- [ocp-async-3,38de46d4d8f02501,c373e3ae38fb97c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:37.046 INFO 21062 --- [ocp-async-4,9cf205ded07f81d2,ce0d384ce3bce5ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:37.109 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:37.110 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:38.504 INFO 21062 --- [ocp-schedules-13,096809b5999e1178,60a3fa30691e7ce4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:52:38.504 INFO 21062 --- [ocp-schedules-8,42420a1d0e37fca7,722b53a43c6acfd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:52:38.508 INFO 21062 --- [ocp-schedules-2,186b15753fdb88ac,6e3e43dc8c410084] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:52:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:52:38.590347+08:00, lastProcessTime=2024-11-04T13:52:28.589125+08:00 2024-11-04 13:52:38.625 INFO 21062 --- [ocp-async-6,a5b3e660b4c613fd,37d34935cc4e1616] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:52:38.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:52:38.590347+08:00 2024-11-04 13:52:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3bc8819c57490568,572c3156ed6fd885] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:39.001 WARN 21062 --- [ocp-async-2,997120a1c4edb9c1,46fb341651c37e01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:39.046 INFO 21062 --- [ocp-async-8,0a089d05caf3462e,2a23022bd510bc29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:39.929 INFO 21062 --- [ocp-schedules-18,23165c88768c593e,435698a98c732809] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:52:40.002 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:52:40.002 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:52:40.110 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:40.110 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:41.001 WARN 21062 --- [ocp-async-8,833b830f69b6c368,2ff833794a21902c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:41.046 INFO 21062 --- [ocp-async-6,0309bad2dc2aa4ff,d0387b4721cc4d71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:52:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:52:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:52:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:52:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:52:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:52:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:52:41.983 INFO 21062 --- [prometheus-minute-schedule-1,80d891e10fc5dba5,0b50d3be0fc0ff6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:43.001 WARN 21062 --- [ocp-async-6,a26d47bb17c3863e,b29cb80dc4fa1e7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:43.047 INFO 21062 --- [ocp-async-5,ac5915624a497c64,f758105cb5547201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:43.111 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:43.111 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:44.983 INFO 21062 --- [prometheus-minute-schedule-1,2b1a42b70df2d09f,fa5704827987fb0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:45.001 WARN 21062 --- [ocp-async-5,2a90954aa5cce8be,50990f1f88c33516] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:45.047 INFO 21062 --- [ocp-async-9,b5c145f52a43aed3,fb4a022e87ae172c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:46.111 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:46.112 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:46.450 INFO 21062 --- [ocp-schedules-7,ae7984c0be4c049f,2901a4691f84ceab] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:52:47.001 WARN 21062 --- [ocp-async-9,6bbb9d4b93a80cc1,e69f4f6537b5b731] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:47.047 INFO 21062 --- [ocp-async-1,7d73797764bdd21a,317d17ac3448225b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:47.983 INFO 21062 --- [prometheus-minute-schedule-1,249327787952a786,663c01a4a58f6090] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:52:48.590991+08:00, lastProcessTime=2024-11-04T13:52:38.590347+08:00 2024-11-04 13:52:48.625 INFO 21062 --- [ocp-async-9,72bcc3de75e6b6f2,a794ab8fd1b5ca55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:52:48.779 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:52:48.590991+08:00 2024-11-04 13:52:48.886 INFO 21062 --- [ocp-schedules-11,3cb729bdf27aa98a,58142ad2185c4b34] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:52:48.890 INFO 21062 --- [ocp-schedules-11,3cb729bdf27aa98a,58142ad2185c4b34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:48.890 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:52:49.002 WARN 21062 --- [ocp-async-6,0a4f0b53f54a9deb,f9041707ed16885a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:49.047 INFO 21062 --- [ocp-async-5,1ff5e39b95d06de0,b3ee0689217e9ffe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:49.112 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:49.113 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:49.933 INFO 21062 --- [ocp-schedules-25,3dee1635a94e67a1,ab67ccc8a3001809] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:52:50.710 INFO 21062 --- [ocp-schedules-8,d31328ef4b4de880,7e98fb8bc597ef3f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:52:50.715 INFO 21062 --- [ocp-schedules-8,d31328ef4b4de880,7e98fb8bc597ef3f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:52:50.715 INFO 21062 --- [ocp-schedules-8,d31328ef4b4de880,7e98fb8bc597ef3f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:52:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b4d64594fd57f228,2bba286458e9b755] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:51.002 WARN 21062 --- [ocp-async-5,6e548ed44bec3b46,bc921042a04fbbad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:51.047 INFO 21062 --- [ocp-async-9,c396f05485ce2157,2493b45568be5a38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:52.113 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:52.114 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:53.002 WARN 21062 --- [ocp-async-9,01a60763b6109be1,63e70438aeb802cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:53.047 INFO 21062 --- [ocp-async-1,56c6fa30d272db11,93579c62baecdebd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d3cdc94cf887b12e,ca0d0a663ce33767] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:55.002 WARN 21062 --- [ocp-async-1,dd7fb45e464ba9fd,32ba335f3afe2746] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:55.048 INFO 21062 --- [ocp-async-4,010edb9b9a6faac2,c913fd9872f18259] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:55.114 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:55.115 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:56.491 INFO 21062 --- [ocp-schedules-10,a45cd02ddc8a216a,acce44710181a515] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:52:56.983 INFO 21062 --- [prometheus-minute-schedule-1,aedd082c8095e9ba,7f61d91e05d86f9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:52:57.002 WARN 21062 --- [ocp-async-4,c92294aba47630fb,d59dbe520fd6d9ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:57.048 INFO 21062 --- [ocp-async-7,36ebce6bb9bfafa3,88b40bd39ad9c39a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:58.115 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:52:58.116 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:52:58.500 INFO 21062 --- [ocp-schedules-8,d13dabb0a2d1a862,7a3cb862332cd86e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:52:58.500 INFO 21062 --- [ocp-schedules-28,a4bb8647a8131adb,1c7c160718793abf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:52:58.502 INFO 21062 --- [ocp-schedules-29,ad2b99295cea2dc6,3155115414d9efbf] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:52:58.503 INFO 21062 --- [ocp-schedules-8,d13dabb0a2d1a862,7a3cb862332cd86e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:52:58.504 INFO 21062 --- [ocp-schedules-28,a4bb8647a8131adb,1c7c160718793abf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:52:58.504 INFO 21062 --- [ocp-schedules-19,4e881b96f374e431,df4c09016045da99] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:52:58.511 INFO 21062 --- [ocp-schedules-22,05e14911f313d576,97dbf9e7dd094ee2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:52:58.511 INFO 21062 --- [ocp-schedules-15,0559a59eebe3952a,9aa9da1eb6268629] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:52:58.511 INFO 21062 --- [ocp-schedules-22,05e14911f313d576,97dbf9e7dd094ee2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:52:58.511 INFO 21062 --- [ocp-schedules-15,0559a59eebe3952a,9aa9da1eb6268629] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:52:58.511 INFO 21062 --- [ocp-schedules-26,6d0bccfb1913ddcc,73d24b8eaf8e8b3e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:52:58.511 INFO 21062 --- [ocp-schedules-26,6d0bccfb1913ddcc,73d24b8eaf8e8b3e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:52:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:52:58.589617+08:00, lastProcessTime=2024-11-04T13:52:48.590991+08:00 2024-11-04 13:52:58.625 INFO 21062 --- [ocp-async-10,96376144897f8c17,538fa7304a4aad18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:52:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:52:58.589617+08:00 2024-11-04 13:52:58.890 INFO 21062 --- [ocp-schedules-11,3cb729bdf27aa98a,58142ad2185c4b34] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:52:58.992 INFO 21062 --- [ocp-schedules-4,55c4f99643098c77,960c62cfc4006d93] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:52:59.003 WARN 21062 --- [ocp-async-9,9a486c64eda5261a,3b50a0f40579c6a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:52:59.048 INFO 21062 --- [ocp-async-1,5eba013e7b9e290a,aba79b98ea375db2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:52:59.935 INFO 21062 --- [ocp-schedules-22,e093a54fbe4c8407,2ae5bd86e72ee502] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:52:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7eaf1fa4c086a4d5,7fd2a7ad6ae03c6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699580, interval=60 2024-11-04 13:53:01.003 WARN 21062 --- [ocp-async-1,268755ea5e225ae6,db1e0480c4ecfc69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:01.048 INFO 21062 --- [ocp-async-4,706542a7a381abdc,1799070232e1ff08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:01.117 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:01.117 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:02.983 INFO 21062 --- [prometheus-minute-schedule-1,08de5db5196b7126,92f786ecf68384db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:03.003 WARN 21062 --- [ocp-async-4,d39521e600975bc7,202e9b5639b611f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:03.048 INFO 21062 --- [ocp-async-2,f73367bdafea695f,580bbc425927e295] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:04.018 INFO 21062 --- [ocp-schedules-12,4a6f00b115450833,0788df95f9508e80] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:53:04.018 INFO 21062 --- [ocp-schedules-12,88ae5be99bd9f4e2,d417ea8f2e795840] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:53:04.022 INFO 21062 --- [ocp-schedules-12,88ae5be99bd9f4e2,d417ea8f2e795840] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:04.052 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:53:04.052 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:53:04.118 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:04.118 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:05.003 WARN 21062 --- [ocp-async-2,bf2cd2d1937b621f,1cbd8207051f3624] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:05.049 INFO 21062 --- [ocp-async-8,1dddc9bd54ca0eb7,7c8fcf363396403f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2e67ad8be4c56dce,7b2af9713e387d6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:06.517 INFO 21062 --- [ocp-schedules-13,31fd30e01023d4fd,d0d80af60a3f4b39] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:53:07.003 WARN 21062 --- [ocp-async-8,dfaac8bddde6104e,cc77414fc0d00a4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:07.049 INFO 21062 --- [ocp-async-6,de56d557e9253d6b,e2f2fed2a55b972e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:07.119 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:07.119 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:08.504 INFO 21062 --- [ocp-schedules-8,d13dabb0a2d1a862,7a3cb862332cd86e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:53:08.505 INFO 21062 --- [ocp-schedules-28,a4bb8647a8131adb,1c7c160718793abf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:53:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:53:08.590559+08:00, lastProcessTime=2024-11-04T13:52:58.589617+08:00 2024-11-04 13:53:08.625 INFO 21062 --- [ocp-async-3,d8fd5ff2d522944a,49e5d81c5f1dc319] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:53:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:53:08.590559+08:00 2024-11-04 13:53:08.983 INFO 21062 --- [prometheus-minute-schedule-1,94a2170aaf8b3664,9d39db4136cbb448] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:09.003 WARN 21062 --- [ocp-async-4,2a9aa443b6d24e36,e50aba9cf12869e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:09.049 INFO 21062 --- [ocp-async-7,d6cd92d3d2a8338d,1289ef00f5d28668] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:09.938 INFO 21062 --- [ocp-schedules-5,9d4193e50f0bf283,d97fd9078fe3d882] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:53:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:53:10.004 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:53:10.119 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:10.120 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:53:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:53:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:53:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:53:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:53:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:53:10.718 INFO 21062 --- [ocp-schedules-18,9e069fa6993a68d9,15035af6868b3ed1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:53:10.724 INFO 21062 --- [ocp-schedules-18,9e069fa6993a68d9,15035af6868b3ed1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:53:10.724 INFO 21062 --- [ocp-schedules-18,9e069fa6993a68d9,15035af6868b3ed1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:53:11.004 WARN 21062 --- [ocp-async-2,14964f4a358bd23b,aa58b46cbe42002d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:11.049 INFO 21062 --- [ocp-async-3,520d6d6c4818c066,9aa43476a6776141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:11.394 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=1, affectRows=1 2024-11-04 13:53:11.394 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=1, validCount=1 2024-11-04 13:53:11.395 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=7, affectRows=7 2024-11-04 13:53:11.395 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=2, affectRows=2 2024-11-04 13:53:11.395 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=2, validCount=2 2024-11-04 13:53:11.395 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=5, affectRows=5 2024-11-04 13:53:11.395 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=7, validCount=7 2024-11-04 13:53:11.395 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=5, validCount=5 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=22, affectRows=22 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=16, affectRows=16 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=35, affectRows=35 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=16, validCount=16 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=22, validCount=22 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=62, affectRows=62 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=52, affectRows=52 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=54, affectRows=54 2024-11-04 13:53:11.396 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=35, validCount=35 2024-11-04 13:53:11.397 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=59, affectRows=59 2024-11-04 13:53:11.397 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=60, validCount=60 2024-11-04 13:53:11.397 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=52, validCount=52 2024-11-04 13:53:11.397 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=54, validCount=54 2024-11-04 13:53:11.397 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=59, validCount=59 2024-11-04 13:53:11.397 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699520, nodeCount=77, affectRows=77 2024-11-04 13:53:11.397 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699520, nodeCount=76, validCount=76 2024-11-04 13:53:11.983 INFO 21062 --- [prometheus-minute-schedule-1,80c73e427bc064ba,bf4b4b95d43247b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:13.004 WARN 21062 --- [ocp-async-8,c92c2a486f593461,97a74c958f66110b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:13.049 INFO 21062 --- [ocp-async-10,874fdb427575b371,b0906edfa1be2c09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:13.120 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:13.121 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:13.984 INFO 21062 --- [prometheus-cache-monitor1,3319a6b82dbe4279,26f98c703459bbc4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:53:13.984 INFO 21062 --- [prometheus-cache-monitor1,3319a6b82dbe4279,26f98c703459bbc4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:53:14.136 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:47:14.115, end=2024-11-04T13:52:14.134} 2024-11-04 13:53:14.137 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:47:14.115, end=2024-11-04T13:52:14.134} 2024-11-04 13:53:14.145 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:53:14.145 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:53:14.145 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:53:14.145 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:53:14.772 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:49:14.752, end=2024-11-04T13:51:14.769} 2024-11-04 13:53:14.776 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:53:14.776 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:53:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2d4cc32d7c5aca92,f02f6e80d4c8bb10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:15.004 WARN 21062 --- [ocp-async-6,6fbf7fff819eec39,1bc800ec6b17b744] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:15.049 INFO 21062 --- [ocp-async-5,ba1efb6167f2ad6f,815c090a8e33ae9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:16.121 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:16.122 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:16.557 INFO 21062 --- [ocp-schedules-15,0358981decb5001e,266aac90003ced0d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:53:17.004 WARN 21062 --- [ocp-async-7,48b2da75b6c5a964,3cf2c22a726ef322] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:17.050 INFO 21062 --- [ocp-async-9,ec0cd53f434b2181,4a3e7de4d0ad7bdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:17.983 INFO 21062 --- [prometheus-minute-schedule-1,96804311a43fbe54,54b0415da53974a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:53:18.590624+08:00, lastProcessTime=2024-11-04T13:53:08.590559+08:00 2024-11-04 13:53:18.625 INFO 21062 --- [ocp-async-5,4b51dc8f603526a2,2a30c99ee1dbe22b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:53:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:53:18.590624+08:00 2024-11-04 13:53:19.004 WARN 21062 --- [ocp-async-8,8484b81f9510c7d3,c7044b7afe17afb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:19.050 INFO 21062 --- [ocp-async-10,f0dee7788588eece,95273a1293940529] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:19.122 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:19.123 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:19.941 INFO 21062 --- [ocp-schedules-29,c50566932c651ef4,0f99dfb8e6456910] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:53:20.983 INFO 21062 --- [prometheus-minute-schedule-1,6addc67e4577e8f4,0f2e35b204a71167] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:21.004 WARN 21062 --- [ocp-async-6,dbdb51300631abeb,920ab75b375bded7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:21.050 INFO 21062 --- [ocp-async-5,32194cd2378ef632,17d387c133973922] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:22.123 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:22.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:23.005 WARN 21062 --- [ocp-async-7,bc1878de9d170821,94c8331c70a8a02d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:23.050 INFO 21062 --- [ocp-async-9,bfb5d1eace2c3d33,04b63ef091990c7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6e9c09497bb98417,c958e199d8fd6aee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:25.005 WARN 21062 --- [ocp-async-3,d887de262d4991a1,a2f917331e4db7b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:25.050 INFO 21062 --- [ocp-async-1,b5ab9020feb7999f,f64f9ad3ad62c8fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:25.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:25.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:26.583 INFO 21062 --- [ocp-schedules-28,da39ea97fdaac5c3,2b35f36e6298037e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:53:26.983 INFO 21062 --- [prometheus-minute-schedule-1,5c716e2d4672a3c8,ea0e0ed71773334a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:26.984 INFO 21062 --- [prometheus-hour-schedule-1,04e041b342adeea0,ff15d494e4841efa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:53:27.005 WARN 21062 --- [ocp-async-10,dc1b28d268a29023,064d0f8a8b516559] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:27.050 INFO 21062 --- [ocp-async-4,923858f1b833edc6,09b248cdc338d387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:28.126 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:28.127 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:28.498 INFO 21062 --- [ocp-schedules-18,80d447b987aaaeb6,d4797c34f3e94d89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:53:28.499 INFO 21062 --- [ocp-schedules-20,9db962295716947d,0431a0a84f10167f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:53:28.500 INFO 21062 --- [ocp-schedules-27,bbaeb71c1e987c40,6082080ebf60efff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:53:28.500 INFO 21062 --- [ocp-schedules-13,077869a1e1bfcdb2,c85b98e43409eb57] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:53:28.502 INFO 21062 --- [ocp-schedules-17,d75badd58e28e843,3be676ff10011a56] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:53:28.503 INFO 21062 --- [ocp-schedules-18,80d447b987aaaeb6,d4797c34f3e94d89] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:53:28.503 INFO 21062 --- [ocp-schedules-27,bbaeb71c1e987c40,6082080ebf60efff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:53:28.504 INFO 21062 --- [ocp-schedules-13,077869a1e1bfcdb2,c85b98e43409eb57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:28.504 INFO 21062 --- [ocp-schedules-11,0bc1819ee1b4b568,dbecc18be5a67286] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:53:28.505 INFO 21062 --- [ocp-schedules-20,9db962295716947d,0431a0a84f10167f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:53:28.512 INFO 21062 --- [ocp-schedules-22,ebda301ae9576a4b,041e33be4d637f48] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:53:28.512 INFO 21062 --- [ocp-schedules-21,bcaf341afd9100d2,83086b7a6bac5c56] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:53:28.512 INFO 21062 --- [ocp-schedules-23,c8749d18a6150cf9,93d17ce45e8284b1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:53:28.512 INFO 21062 --- [ocp-schedules-22,ebda301ae9576a4b,041e33be4d637f48] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:53:28.512 INFO 21062 --- [ocp-schedules-21,bcaf341afd9100d2,83086b7a6bac5c56] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:53:28.512 INFO 21062 --- [ocp-schedules-23,c8749d18a6150cf9,93d17ce45e8284b1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:53:28.568 INFO 21062 --- [ocp-schedules-7,b96d1ed18f31e091,1c988ce83e2b2485] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 13:53:28.568 INFO 21062 --- [ocp-schedules-7,b96d1ed18f31e091,1c988ce83e2b2485] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 13:53:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:53:28.589515+08:00, lastProcessTime=2024-11-04T13:53:18.590624+08:00 2024-11-04 13:53:28.626 INFO 21062 --- [ocp-async-6,49446969cbccc344,4af74c3ce4edd2f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:53:28.755 INFO 21062 --- [ocp-schedules-6,938e4c71478ead83,7f4b54f41d384dc9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:53:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:53:28.589515+08:00 2024-11-04 13:53:28.956 INFO 21062 --- [ocp-schedules-5,dad43dcbbbde53cb,8853aad05bf2ddfb] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:53:28.956 INFO 21062 --- [ocp-schedules-5,dad43dcbbbde53cb,8853aad05bf2ddfb] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:53:28.995 INFO 21062 --- [ocp-schedules-20,87aafc0599888998,81627c68114dd292] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:53:29.005 WARN 21062 --- [ocp-async-9,18ac1a632bf13db0,53cb4d6cb720f6a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:29.029 INFO 21062 --- [ocp-schedules-21,b530ea00614272e6,be771a59c4eb39c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:53:29.035 INFO 21062 --- [ocp-schedules-21,b530ea00614272e6,be771a59c4eb39c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:53:29.035 INFO 21062 --- [ocp-schedules-21,b530ea00614272e6,be771a59c4eb39c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:53:29.051 INFO 21062 --- [ocp-async-1,7948ab3fdb13b23e,1057ef6b1f4e0281] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:29.298 INFO 21062 --- [ocp-schedules-30,adbdff4cad952a2a,d22513e51469dbba] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:53:29.298 INFO 21062 --- [ocp-schedules-30,adbdff4cad952a2a,d22513e51469dbba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:53:29.299 INFO 21062 --- [ocp-schedules-30,adbdff4cad952a2a,d22513e51469dbba] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:53:29.299 INFO 21062 --- [ocp-schedules-30,adbdff4cad952a2a,d22513e51469dbba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:53:29.300 INFO 21062 --- [ocp-schedules-30,adbdff4cad952a2a,d22513e51469dbba] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:53:29.300 INFO 21062 --- [ocp-schedules-30,adbdff4cad952a2a,d22513e51469dbba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:53:29.945 INFO 21062 --- [ocp-schedules-6,0c200531e51f156b,99ddb8d4c40fa1d0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:53:29.983 INFO 21062 --- [prometheus-minute-schedule-1,546849efc3affe84,e60cb6949f4d67bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:53:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:53:30.728 INFO 21062 --- [ocp-schedules-18,1ed44b62c0571953,c506ea53430c40be] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:53:30.734 INFO 21062 --- [ocp-schedules-10,1b0fb753ae803c51,a26810b1423e53ff] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:53:30.734 INFO 21062 --- [ocp-schedules-10,1b0fb753ae803c51,a26810b1423e53ff] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:53:30.735 INFO 21062 --- [ocp-schedules-18,1ed44b62c0571953,c506ea53430c40be] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:53:30.735 INFO 21062 --- [ocp-schedules-18,1ed44b62c0571953,c506ea53430c40be] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:53:30.761 INFO 21062 --- [ocp-schedules-10,1b0fb753ae803c51,a26810b1423e53ff] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:53:30.761 INFO 21062 --- [ocp-schedules-10,1b0fb753ae803c51,a26810b1423e53ff] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:53:30.762 INFO 21062 --- [ocp-schedules-26,c75be074a7d24faa,f560f266a14f16e4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:53:30.913 INFO 21062 --- [ocp-schedules-24,c6a44b7a53fb4f86,27659ee9e488becf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:53:30.913+08:00, clusters=[1] 2024-11-04 13:53:30.915 INFO 21062 --- [ocp-schedules-24,c6a44b7a53fb4f86,27659ee9e488becf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:53:30.977 INFO 21062 --- [ocp-schedules-25,443b0794518db0a1,be3f7a823fc6d03b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:53:30.977 INFO 21062 --- [ocp-schedules-25,443b0794518db0a1,be3f7a823fc6d03b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:53:30.977 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:53:30.977 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:53:30.987 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:53:30.987 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:53:30.988 INFO 21062 --- [ocp-schedules-25,443b0794518db0a1,be3f7a823fc6d03b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:53:30.989 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:53:30.989 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:53:30.989 WARN 21062 --- [ocp-schedules-25,443b0794518db0a1,be3f7a823fc6d03b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:53:30.990 INFO 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:53:30.990 INFO 21062 --- [ocp-schedules-25,443b0794518db0a1,be3f7a823fc6d03b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:53:30.990 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:53:30.991 WARN 21062 --- [ForkJoinPool.commonPool-worker-52,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:53:30.991 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:53:30.992 WARN 21062 --- [ocp-schedules-25,443b0794518db0a1,be3f7a823fc6d03b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:53:30.998 ERROR 21062 --- [ocp-schedules-25,443b0794518db0a1,be3f7a823fc6d03b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:53:31.005 WARN 21062 --- [ocp-async-8,6f231795bf383c79,4e432a7f1c3f3136] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:31.051 INFO 21062 --- [ocp-async-6,cf50d97850267356,ac2a28639511aec6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:31.064 INFO 21062 --- [ocp-schedules-1,f85e0e3158768220,c8cf129474aea3e0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:53:31.064+08:00, clusters=[1] 2024-11-04 13:53:31.068 INFO 21062 --- [ocp-schedules-1,f85e0e3158768220,c8cf129474aea3e0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:53:31.127 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:31.128 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8067a9b0cbaa0933,de4c712747c9eeda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:33.005 WARN 21062 --- [ocp-async-4,9b96965f4abb58cc,4b8876c16bb8247b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:33.051 INFO 21062 --- [ocp-async-7,682910ddab17bdb5,c5a036a02d552b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:34.053 INFO 21062 --- [ocp-schedules-12,88ae5be99bd9f4e2,d417ea8f2e795840] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:53:34.053 INFO 21062 --- [ocp-schedules-12,8888d7176254ec60,61c0873bad1f4b53] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:53:34.057 INFO 21062 --- [ocp-schedules-12,8888d7176254ec60,61c0873bad1f4b53] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:34.089 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:53:34.089 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:53:34.128 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:34.129 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:35.006 WARN 21062 --- [ocp-async-7,7e545b70251cb775,4084a03e9012a640] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:35.051 INFO 21062 --- [ocp-async-3,d9c8b877667f4c1f,5475cd5534ecc278] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:35.983 INFO 21062 --- [prometheus-minute-schedule-1,6e91e2e3fdf45d74,a3ae5cfe7b3e602a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:35.989 INFO 21062 --- [ocp-schedules-21,f1e4b31d70429cd2,8600b05ce1b39cc2] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:53:35.993 INFO 21062 --- [ocp-schedules-21,f1e4b31d70429cd2,8600b05ce1b39cc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:36.077 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:53:36.084 ERROR 21062 --- [ocp-schedules-21,f1e4b31d70429cd2,8600b05ce1b39cc2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:53:36.084 INFO 21062 --- [ocp-schedules-21,f1e4b31d70429cd2,8600b05ce1b39cc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:53:36.624 INFO 21062 --- [ocp-schedules-7,ad2300852c949468,875c4f600da532a8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:53:37.006 WARN 21062 --- [ocp-async-3,cb4821caac3769e7,58c02ca2df4ac86e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:37.051 INFO 21062 --- [ocp-async-10,cb5f8adbdd337a28,af5d714b1c7a63d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:37.129 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:37.130 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:38.504 INFO 21062 --- [ocp-schedules-27,bbaeb71c1e987c40,6082080ebf60efff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:53:38.504 INFO 21062 --- [ocp-schedules-13,077869a1e1bfcdb2,c85b98e43409eb57] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:53:38.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:53:38.592242+08:00, lastProcessTime=2024-11-04T13:53:28.589515+08:00 2024-11-04 13:53:38.626 INFO 21062 --- [ocp-async-8,3418dc7176537d3a,56cf31d65503e23b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:53:38.808 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:53:38.592242+08:00 2024-11-04 13:53:38.983 INFO 21062 --- [prometheus-minute-schedule-1,1c68bedf225e1517,3274449e1d02d635] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:39.006 WARN 21062 --- [ocp-async-4,f361a579c149a940,62c17bd06189c9d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:39.051 INFO 21062 --- [ocp-async-2,235bf131794eeca1,deb1f0202bb9b583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:39.947 INFO 21062 --- [ocp-schedules-17,c201e33a6fd1bd90,04ddd2ad8989d471] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:53:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:53:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:53:40.130 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:40.131 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:41.006 WARN 21062 --- [ocp-async-2,9c3bdf3955d83caf,5b4d8f09d08f4984] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:41.052 INFO 21062 --- [ocp-async-8,11a2b7516f31725f,e077631d69f1f340] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:41.983 INFO 21062 --- [prometheus-minute-schedule-1,e620a327ad2b11b6,125c2164a00a534f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:43.006 WARN 21062 --- [ocp-async-8,2794f65a17af4f88,12030fdd03dd6e9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:43.052 INFO 21062 --- [ocp-async-6,d435f6a538b2ecf4,6dddafe4c24a0841] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:43.131 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:43.132 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:44.983 INFO 21062 --- [prometheus-minute-schedule-1,2b6febf2d4a42168,00605ea1bf16d10d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:45.007 WARN 21062 --- [ocp-async-6,c2e89f3e944e1c01,c34d2b93eceb0c22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:45.052 INFO 21062 --- [ocp-async-5,79ec88096a3c0640,266d00fe9bd64a88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:46.132 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:46.133 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:46.652 INFO 21062 --- [ocp-schedules-3,2e25554bda698512,31f277fd41131171] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:53:47.007 WARN 21062 --- [ocp-async-5,a1ae4d7cf8e0abde,b819f15bec7d0253] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:47.052 INFO 21062 --- [ocp-async-3,afb2577cc0793e93,46fc41f9dab6f90a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b73068db02b83f8f,2dab54e31e02d705] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:53:48.591045+08:00, lastProcessTime=2024-11-04T13:53:38.592242+08:00 2024-11-04 13:53:48.626 INFO 21062 --- [ocp-async-7,90f775e1dff43708,0549b6b3ccd785ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:53:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:53:48.591045+08:00 2024-11-04 13:53:49.007 WARN 21062 --- [ocp-async-8,8e6f5e86e9577185,81a777dfc9283f26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:49.052 INFO 21062 --- [ocp-async-6,04fd79da3d34e71f,c583bdc6eeae4fde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:49.133 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:49.134 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:49.950 INFO 21062 --- [ocp-schedules-1,9ce8451c48711b1d,eee3b175a81e530c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:53:50.739 INFO 21062 --- [ocp-schedules-3,a2cb52a7d4ac6b88,2da7f10340936789] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:53:50.745 INFO 21062 --- [ocp-schedules-3,a2cb52a7d4ac6b88,2da7f10340936789] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:53:50.745 INFO 21062 --- [ocp-schedules-3,a2cb52a7d4ac6b88,2da7f10340936789] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:53:50.983 INFO 21062 --- [prometheus-minute-schedule-1,50b2eb36419d44c0,cf95a2db8e40ed9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:51.007 WARN 21062 --- [ocp-async-6,e9a1dcc192a15fd8,14468cab4ac5ecd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:51.053 INFO 21062 --- [ocp-async-7,1f7630e7a17282b1,b6057c51c356fef7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:52.134 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:52.135 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:53.007 WARN 21062 --- [ocp-async-7,e5e193203dc1a7c4,10c6fdf881029076] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:53.053 INFO 21062 --- [ocp-async-3,307525f2c129a781,40a2db839630f554] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:53.983 INFO 21062 --- [prometheus-minute-schedule-1,5f41304447a861b5,bc3ff9a40e5981bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:55.007 WARN 21062 --- [ocp-async-3,8583c3de1a631210,d2769efe620fbf56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:55.053 INFO 21062 --- [ocp-async-10,f2645d1fef2422e1,2dbeedfda067596f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:55.135 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:55.136 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:56.695 INFO 21062 --- [ocp-schedules-18,28ffece6a327a0c3,6c782871113e97c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:53:56.983 INFO 21062 --- [prometheus-minute-schedule-1,3036e01d1a1a60cb,f564928fd4b19195] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:53:57.008 WARN 21062 --- [ocp-async-10,06c24797739c8854,637c040b9f905e0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:57.053 INFO 21062 --- [ocp-async-5,6ffbd7bb06c3ee3a,122dd7f62aaa068d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:58.136 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:53:58.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:53:58.500 INFO 21062 --- [ocp-schedules-21,6d58e63f572a868b,2354cb20cae4f9cd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:53:58.500 INFO 21062 --- [ocp-schedules-4,005c24a6bddbbf8e,e57db62194c1a51f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:53:58.502 INFO 21062 --- [ocp-schedules-17,26fd37127bae7a54,7de0298398c920a6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:53:58.503 INFO 21062 --- [ocp-schedules-21,6d58e63f572a868b,2354cb20cae4f9cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:58.503 INFO 21062 --- [ocp-schedules-4,005c24a6bddbbf8e,e57db62194c1a51f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:53:58.504 INFO 21062 --- [ocp-schedules-30,ea93b6666dc14630,a57ec5f2172cf8e0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:53:58.510 INFO 21062 --- [ocp-schedules-9,f9bc507ac7aac9ca,fc7d37cb9965ffe2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:53:58.510 INFO 21062 --- [ocp-schedules-26,51b724c6b69b023c,7cae92f564ce2b25] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:53:58.510 INFO 21062 --- [ocp-schedules-26,51b724c6b69b023c,7cae92f564ce2b25] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:53:58.510 INFO 21062 --- [ocp-schedules-9,f9bc507ac7aac9ca,fc7d37cb9965ffe2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:53:58.510 INFO 21062 --- [ocp-schedules-1,1b8f9bf98a486ae8,ea2a54e22de39d01] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:53:58.510 INFO 21062 --- [ocp-schedules-1,1b8f9bf98a486ae8,ea2a54e22de39d01] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:53:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:53:58.590133+08:00, lastProcessTime=2024-11-04T13:53:48.591045+08:00 2024-11-04 13:53:58.626 INFO 21062 --- [ocp-async-1,6da690f3166ff651,7e7969796e9076cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:53:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:53:58.590133+08:00 2024-11-04 13:53:58.891 INFO 21062 --- [ocp-schedules-14,ea8bd720e83dde4e,5bbd0991e0c14255] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:53:58.894 INFO 21062 --- [ocp-schedules-14,ea8bd720e83dde4e,5bbd0991e0c14255] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:53:58.894 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:53:58.998 INFO 21062 --- [ocp-schedules-29,e5920a5ce32f289e,c47a0128359a46c5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:53:59.008 WARN 21062 --- [ocp-async-7,1250c5e337fa3081,a7ec5409758f6dcd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:53:59.053 INFO 21062 --- [ocp-async-3,9f325d8ffc587933,85d0cea99de473b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:53:59.953 INFO 21062 --- [ocp-schedules-6,36935cafb7ec0f17,eb128115c38d98a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:53:59.983 INFO 21062 --- [prometheus-minute-schedule-1,93a5fdb6249041fa,408ee53aee3af974] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699640, interval=60 2024-11-04 13:54:01.008 WARN 21062 --- [ocp-async-3,50f2caf4e4f2bf77,6c60df4f306380be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:01.054 INFO 21062 --- [ocp-async-1,5d339a437f069b84,5ad602610993252a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:01.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:01.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e9d0bef9a52a36d2,ebae8e8c1787f01c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:03.008 WARN 21062 --- [ocp-async-1,fd5476d33843a5eb,71a34fd98c74eaaa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:03.054 INFO 21062 --- [ocp-async-4,9a53c846f41190f8,2f5c255c192a8553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:04.089 INFO 21062 --- [ocp-schedules-12,8888d7176254ec60,61c0873bad1f4b53] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:54:04.090 INFO 21062 --- [ocp-schedules-12,93b8bc6714aa22d4,63db2b2cd567869f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:54:04.094 INFO 21062 --- [ocp-schedules-12,93b8bc6714aa22d4,63db2b2cd567869f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:04.126 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:54:04.126 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:54:04.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:04.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:05.008 WARN 21062 --- [ocp-async-4,a66450c1e818136e,4fdf10aa2f366e4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:05.054 INFO 21062 --- [ocp-async-2,140ce799ea5e72da,e924f728f2fe691e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0cdc90a3c12dbfa5,1fe3eea74b0503cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:06.722 INFO 21062 --- [ocp-schedules-30,b6090bd354f68f5a,b8104ff49b7297fe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:54:07.008 WARN 21062 --- [ocp-async-2,7c966e46945f0201,c35c2db4dddf6bd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:07.054 INFO 21062 --- [ocp-async-8,61c75201036395ae,ee4fde0ec38998b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:07.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:07.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:08.504 INFO 21062 --- [ocp-schedules-4,005c24a6bddbbf8e,e57db62194c1a51f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:54:08.504 INFO 21062 --- [ocp-schedules-21,6d58e63f572a868b,2354cb20cae4f9cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:54:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:54:08.590069+08:00, lastProcessTime=2024-11-04T13:53:58.590133+08:00 2024-11-04 13:54:08.626 INFO 21062 --- [ocp-async-9,ac42387739fc217e,053f59dde9533283] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:54:08.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:54:08.590069+08:00 2024-11-04 13:54:08.895 INFO 21062 --- [ocp-schedules-14,ea8bd720e83dde4e,5bbd0991e0c14255] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:54:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a2f1afb9935766ed,6941c1c6bf735ca9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:09.009 WARN 21062 --- [ocp-async-1,e928aabeb00671aa,7ddb84d21f770a69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:09.054 INFO 21062 --- [ocp-async-5,2120aa1627f5d47b,bfb055d0580f5a10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:09.957 INFO 21062 --- [ocp-schedules-4,1fd82f4902864249,35cce47492ee20df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:54:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:54:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:54:10.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:10.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:54:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:54:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:54:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:54:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:54:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:54:10.747 INFO 21062 --- [ocp-schedules-24,d0b56a768e8443e8,60448477542162fc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:54:10.754 INFO 21062 --- [ocp-schedules-24,d0b56a768e8443e8,60448477542162fc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:54:10.754 INFO 21062 --- [ocp-schedules-24,d0b56a768e8443e8,60448477542162fc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:54:11.009 WARN 21062 --- [ocp-async-4,0b6e00483be72b82,0b5b50b6c2946fe1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:11.054 INFO 21062 --- [ocp-async-9,e6db1f2a904d0a57,c14fa16eef6ba19b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:11.394 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=2, affectRows=2 2024-11-04 13:54:11.394 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=2, validCount=2 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=18, affectRows=18 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=18, validCount=18 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=15, affectRows=15 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=11, affectRows=11 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=25, affectRows=25 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=10, affectRows=10 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=11, validCount=11 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=10, validCount=10 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=29, affectRows=29 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=15, validCount=15 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=25, validCount=25 2024-11-04 13:54:11.395 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=29, validCount=29 2024-11-04 13:54:11.396 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=32, affectRows=32 2024-11-04 13:54:11.396 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=32, validCount=32 2024-11-04 13:54:11.396 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=48, affectRows=48 2024-11-04 13:54:11.396 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=56, affectRows=56 2024-11-04 13:54:11.396 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=47, validCount=47 2024-11-04 13:54:11.396 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=56, validCount=56 2024-11-04 13:54:11.396 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=67, affectRows=67 2024-11-04 13:54:11.397 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=65, validCount=65 2024-11-04 13:54:11.397 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699580, nodeCount=79, affectRows=79 2024-11-04 13:54:11.398 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699580, nodeCount=79, validCount=79 2024-11-04 13:54:11.983 INFO 21062 --- [prometheus-minute-schedule-1,10bfc29ab7202123,c81d1b56beeaea26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:13.009 WARN 21062 --- [ocp-async-2,91a06ff2015ff7a4,eb0302c6fed4ba1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:13.054 INFO 21062 --- [ocp-async-8,d52d3a54ce11d9e7,5e23ea0f4330b27f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:13.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:13.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:13.985 INFO 21062 --- [prometheus-cache-monitor1,b2acf0ab346f29e3,fd512e321872ed7b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:54:13.985 INFO 21062 --- [prometheus-cache-monitor1,b2acf0ab346f29e3,fd512e321872ed7b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:54:14.983 INFO 21062 --- [prometheus-minute-schedule-1,b3ae8ee32802bc4f,f33db0a85029b4e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:15.009 WARN 21062 --- [ocp-async-10,d09aa0e026c97ad5,6fc3b3e063bb9978] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:15.055 INFO 21062 --- [ocp-async-6,552719dfef29497f,8bd80932ff3142e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:16.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:16.143 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:16.763 INFO 21062 --- [ocp-schedules-6,ebe931a40e83a038,54bbbf1a2fd259ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:54:17.009 WARN 21062 --- [ocp-async-5,f810417f5279b99c,551de444bf0a2a5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:17.055 INFO 21062 --- [ocp-async-7,63810c09e40dda79,1b8ecf0211daf601] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4a8d710e6ef7bbe8,64fdda3460b4fd38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:54:18.590972+08:00, lastProcessTime=2024-11-04T13:54:08.590069+08:00 2024-11-04 13:54:18.627 INFO 21062 --- [ocp-async-6,5f6131f574da7eae,d607a0ef44e5dd67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:54:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:54:18.590972+08:00 2024-11-04 13:54:19.010 WARN 21062 --- [ocp-async-2,1efa266232943627,096ca87a4890edba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:19.055 INFO 21062 --- [ocp-async-8,33440ed940c1cfc7,b2f60a8ca67b7128] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:19.143 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:19.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:19.961 INFO 21062 --- [ocp-schedules-18,4f0ddf4d113e7f9c,05ad43d308430e3c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:54:20.983 INFO 21062 --- [prometheus-minute-schedule-1,847a9e84afc4a5f3,e95d9e55f5cbcb98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:21.010 WARN 21062 --- [ocp-async-10,9dba5828969a1876,6acb42f25ac17d46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:21.055 INFO 21062 --- [ocp-async-6,0a261f430533e879,80e264e668b6c203] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:22.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:22.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:23.010 WARN 21062 --- [ocp-async-5,f76b7dba32592bc3,8f5596a40f09150d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:23.056 INFO 21062 --- [ocp-async-7,ff71ae7f631a246c,39d898c778cac374] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:23.281 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:54:23.281 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:54:23.290 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:54:23.290 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:54:23.290 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:54:23.290 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:54:23.296 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:54:23.296 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:54:23.296 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:54:23.296 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:54:23.333 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:54:23.341 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:54:23.341 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:54:23.342 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:54:23.342 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:54:23.342 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:54:23.350 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:54:23.871 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:54:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c27cf3a62df59a98,9740e049a542a3ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:24.472 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.175903617S 2024-11-04 13:54:24.472 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:54:24.572 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:54:24.573 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:54:24.601 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:54:24.658 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:54:24.658 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:54:24.658 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:54:24.658 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:54:24.658 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:54:24.659 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:54:24.669 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:54:24.846 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] o.s.boot.SpringApplication : Started application in 1.555 seconds (JVM running for 5078.525) 2024-11-04 13:54:24.846 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.555808322S 2024-11-04 13:54:25.010 WARN 21062 --- [ocp-async-9,570438ed9870441f,52538b86cbf87d1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:25.056 INFO 21062 --- [ocp-async-3,ef508aac3cda6640,9a3dcfce7eaeff3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:25.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:25.148 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:25.303 INFO 21062 --- [ocp-schedules-22,5c368857f31db622,1243f8eb857b8925] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:54:26.788 INFO 21062 --- [ocp-schedules-17,69f28a5b1bf2155e,f35d39cb7748d23f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:54:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2cb395f79d3a4706,680dd2cf0266935e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5d6d76b9a2cb327e,0b4dff4ef034c5b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:54:27.010 WARN 21062 --- [ocp-async-8,c2506c5e3160b203,d89446eb44527fc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:27.056 INFO 21062 --- [ocp-async-1,8ef9ee32046b7c27,2c86633977f50734] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:28.148 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:28.149 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:28.498 INFO 21062 --- [ocp-schedules-5,52c3ba5d134788ce,a821aed3d2e26918] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:54:28.499 INFO 21062 --- [ocp-schedules-27,c260b93f95412b2f,c95e66839af8b5f7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:54:28.500 INFO 21062 --- [ocp-schedules-3,195e80acf463b92f,77529d937066cb8a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:54:28.500 INFO 21062 --- [ocp-schedules-28,2614042b63a983ff,979ed90951fb9dac] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:54:28.502 INFO 21062 --- [ocp-schedules-5,52c3ba5d134788ce,a821aed3d2e26918] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:54:28.503 INFO 21062 --- [ocp-schedules-4,8793ecf3e71edf20,3c6d60fc75a65ac8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:54:28.504 INFO 21062 --- [ocp-schedules-3,195e80acf463b92f,77529d937066cb8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:28.504 INFO 21062 --- [ocp-schedules-28,2614042b63a983ff,979ed90951fb9dac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:54:28.504 INFO 21062 --- [ocp-schedules-27,c260b93f95412b2f,c95e66839af8b5f7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:54:28.505 INFO 21062 --- [ocp-schedules-8,f2ce7bdd8b9cbe97,601a5be3078d0a34] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:54:28.511 INFO 21062 --- [ocp-schedules-15,7fb1431e769dc1bf,f6c609ee681c9802] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:54:28.511 INFO 21062 --- [ocp-schedules-15,7fb1431e769dc1bf,f6c609ee681c9802] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:54:28.511 INFO 21062 --- [ocp-schedules-21,bef8308a945593c0,854286dae712e4cb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:54:28.511 INFO 21062 --- [ocp-schedules-21,bef8308a945593c0,854286dae712e4cb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:54:28.512 INFO 21062 --- [ocp-schedules-30,9878a46dbfdddd95,4e3ce9a7f86287a8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:54:28.512 INFO 21062 --- [ocp-schedules-30,9878a46dbfdddd95,4e3ce9a7f86287a8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:54:28.571 INFO 21062 --- [ocp-schedules-25,79e33a7ef4b7e3f1,82a270fc5f16a55d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:54:28.571 INFO 21062 --- [ocp-schedules-25,79e33a7ef4b7e3f1,82a270fc5f16a55d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:54:28.573 INFO 21062 --- [ocp-schedules-24,56eef031fe233e17,c7639a5fa2230800] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:54:28.573+08:00, tenantSize=0 2024-11-04 13:54:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:54:28.589841+08:00, lastProcessTime=2024-11-04T13:54:18.590972+08:00 2024-11-04 13:54:28.612 INFO 21062 --- [ocp-schedules-29,2ee8b5dae736017a,f8fa00ea6435e03d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:54:28.613 INFO 21062 --- [ocp-schedules-29,2ee8b5dae736017a,f8fa00ea6435e03d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:54:28.627 INFO 21062 --- [ocp-async-10,d66e6ae1e397b750,9bd26c73d0a81e40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:54:28.636 INFO 21062 --- [ocp-schedules-18,b9871b68a245f40d,7b4af6af63729aa6] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 13:54:28.637 INFO 21062 --- [ocp-schedules-18,b9871b68a245f40d,7b4af6af63729aa6] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 13:54:28.708 INFO 21062 --- [ocp-schedules-6,0d195c743136f672,8883fcebf8239d78] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:54:28.709 INFO 21062 --- [ocp-schedules-6,0d195c743136f672,8883fcebf8239d78] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:54:28.749 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:54:28.589841+08:00 2024-11-04 13:54:28.758 INFO 21062 --- [ocp-schedules-16,7750f51a8369acca,677fb9e0ec6814db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:54:28.960 INFO 21062 --- [ocp-schedules-8,3cba074af252371e,a2d9b0eb0e412b51] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:54:28.960 INFO 21062 --- [ocp-schedules-8,3cba074af252371e,a2d9b0eb0e412b51] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:54:29.000 INFO 21062 --- [ocp-schedules-15,9b498ddcfe1c9787,b8fa276288d059c8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:54:29.010 WARN 21062 --- [ocp-async-7,f104bd64ec962b42,43f87ff4fe9bb6c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:29.035 INFO 21062 --- [ocp-schedules-7,28cd1b89c689d3cf,db559d0df48e3fbb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:54:29.040 INFO 21062 --- [ocp-schedules-7,28cd1b89c689d3cf,db559d0df48e3fbb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:54:29.040 INFO 21062 --- [ocp-schedules-7,28cd1b89c689d3cf,db559d0df48e3fbb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:54:29.056 INFO 21062 --- [ocp-async-3,8ef479f98962d648,28c9fc26c3a78cfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:29.302 INFO 21062 --- [ocp-schedules-1,85b1158d7a9a04df,3b521b00658736d2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:54:29.302 INFO 21062 --- [ocp-schedules-1,85b1158d7a9a04df,3b521b00658736d2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:54:29.307 INFO 21062 --- [ocp-schedules-1,85b1158d7a9a04df,3b521b00658736d2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:54:29.307 INFO 21062 --- [ocp-schedules-1,85b1158d7a9a04df,3b521b00658736d2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:54:29.310 INFO 21062 --- [ocp-schedules-1,85b1158d7a9a04df,3b521b00658736d2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:54:29.310 INFO 21062 --- [ocp-schedules-1,85b1158d7a9a04df,3b521b00658736d2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:54:29.964 INFO 21062 --- [ocp-schedules-4,c9ce8788e41e7f61,98781225f68ceb8d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:54:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b3036757714bfd32,4b76222185c5d153] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:54:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:54:30.757 INFO 21062 --- [ocp-schedules-10,0db1553f31db75e3,bb94fc7c036491fd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:54:30.761 INFO 21062 --- [ocp-schedules-19,cbe2fbbff77a344a,5107b59890cde0e5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:54:30.761 INFO 21062 --- [ocp-schedules-19,cbe2fbbff77a344a,5107b59890cde0e5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:54:30.763 INFO 21062 --- [ocp-schedules-10,0db1553f31db75e3,bb94fc7c036491fd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:54:30.763 INFO 21062 --- [ocp-schedules-10,0db1553f31db75e3,bb94fc7c036491fd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:54:30.787 INFO 21062 --- [ocp-schedules-19,cbe2fbbff77a344a,5107b59890cde0e5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:54:30.787 INFO 21062 --- [ocp-schedules-19,cbe2fbbff77a344a,5107b59890cde0e5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:54:30.789 INFO 21062 --- [ocp-schedules-14,c6fa2cfe44a2224e,80a4af61d37feaf7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:54:30.939 INFO 21062 --- [ocp-schedules-16,00b4e4a828f94bf1,fce7b24e32a25ef4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:54:30.939+08:00, clusters=[1] 2024-11-04 13:54:30.941 INFO 21062 --- [ocp-schedules-16,00b4e4a828f94bf1,fce7b24e32a25ef4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:54:31.005 INFO 21062 --- [ocp-schedules-20,a852e04d4646dcb5,2ef4cb316c02f5ef] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:54:31.005 INFO 21062 --- [ocp-schedules-20,a852e04d4646dcb5,2ef4cb316c02f5ef] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:54:31.005 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:54:31.005 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:54:31.011 WARN 21062 --- [ocp-async-9,a80fa5736984a968,316bb589a75beb5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:31.017 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:54:31.018 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:54:31.018 INFO 21062 --- [ocp-schedules-20,a852e04d4646dcb5,2ef4cb316c02f5ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:54:31.019 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:54:31.020 WARN 21062 --- [ocp-schedules-20,a852e04d4646dcb5,2ef4cb316c02f5ef] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:54:31.020 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:54:31.020 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:54:31.021 INFO 21062 --- [ocp-schedules-20,a852e04d4646dcb5,2ef4cb316c02f5ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:54:31.021 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:54:31.021 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:54:31.021 WARN 21062 --- [ocp-schedules-20,a852e04d4646dcb5,2ef4cb316c02f5ef] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:54:31.022 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:54:31.028 ERROR 21062 --- [ocp-schedules-20,a852e04d4646dcb5,2ef4cb316c02f5ef] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:54:31.031 INFO 21062 --- [ocp-schedules-23,6e8c1e4b175d23da,93bf1a312f0b5d0b] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:54:31.034 INFO 21062 --- [ocp-schedules-23,6e8c1e4b175d23da,93bf1a312f0b5d0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:31.056 INFO 21062 --- [ocp-async-10,4067311af9440344,7d8668a325b6805c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:31.088 ERROR 21062 --- [ocp-schedules-23,6e8c1e4b175d23da,93bf1a312f0b5d0b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:54:31.088 INFO 21062 --- [ocp-schedules-23,6e8c1e4b175d23da,93bf1a312f0b5d0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:54:31.090 INFO 21062 --- [ocp-schedules-4,ea163d99715e8928,ddb308a8fd5b25e1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:54:31.090+08:00, clusters=[1] 2024-11-04 13:54:31.094 INFO 21062 --- [ocp-schedules-4,ea163d99715e8928,ddb308a8fd5b25e1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:54:31.149 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:31.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:32.983 INFO 21062 --- [prometheus-minute-schedule-1,95e578952751e737,027ca596b281e44d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:33.011 WARN 21062 --- [ocp-async-1,9c4dd2309c0b43ac,56c1a6755e651050] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:33.056 INFO 21062 --- [ocp-async-5,16674ec98a9d623b,63b73697fb1f9e04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:34.127 INFO 21062 --- [ocp-schedules-12,93b8bc6714aa22d4,63db2b2cd567869f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:54:34.127 INFO 21062 --- [ocp-schedules-12,d260e815bf059cd3,04a7a19eef03e859] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:54:34.131 INFO 21062 --- [ocp-schedules-12,d260e815bf059cd3,04a7a19eef03e859] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:34.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:34.151 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:34.159 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:54:34.159 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:54:35.011 WARN 21062 --- [ocp-async-5,97cce3b8d616cf74,ad4ae52b920aec6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:35.056 INFO 21062 --- [ocp-async-2,db6fc71bdff918c0,2d464a522cc044e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5d2ed8a898dcb5c8,ab1904e7e58592b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:36.084 INFO 21062 --- [ocp-schedules-26,d0541e5f30019416,def64795f63c9e74] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:54:36.089 INFO 21062 --- [ocp-schedules-26,d0541e5f30019416,def64795f63c9e74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:36.171 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:54:36.172 ERROR 21062 --- [ocp-schedules-26,d0541e5f30019416,def64795f63c9e74] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:54:36.172 INFO 21062 --- [ocp-schedules-26,d0541e5f30019416,def64795f63c9e74] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:54:36.826 INFO 21062 --- [ocp-schedules-15,8947fd99bbe5d3dd,ac4ba52ece238631] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:54:37.011 WARN 21062 --- [ocp-async-2,5925b12ac9d3f142,64c80d7571ec61cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:37.057 INFO 21062 --- [ocp-async-8,bfa99fb9620bce8c,3fb989f23f405f75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:37.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:37.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:38.504 INFO 21062 --- [ocp-schedules-28,2614042b63a983ff,979ed90951fb9dac] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:54:38.504 INFO 21062 --- [ocp-schedules-3,195e80acf463b92f,77529d937066cb8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:54:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:54:38.589590+08:00, lastProcessTime=2024-11-04T13:54:28.589841+08:00 2024-11-04 13:54:38.627 INFO 21062 --- [ocp-async-3,2ee78ab9cb3d4f2d,44447fcd5741b57c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:54:38.748 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:54:38.589590+08:00 2024-11-04 13:54:38.983 INFO 21062 --- [prometheus-minute-schedule-1,508f5a0595813716,5c65461bab723a97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:39.011 WARN 21062 --- [ocp-async-1,15aef2465a6b94b1,fbd8b629dbd96529] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:39.057 INFO 21062 --- [ocp-async-4,1fc142ea80d1d383,88057b36a7b76711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:39.967 INFO 21062 --- [ocp-schedules-26,30f76fffd2a5cac9,bc832f804da63ac3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:54:40.002 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:54:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:54:40.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:40.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:41.011 WARN 21062 --- [ocp-async-4,7dbc88e1cc4f1ee5,5e9c364da8819888] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:41.057 INFO 21062 --- [ocp-async-3,bb164de5481358ef,8c3bd436cb6a589e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:41.319 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 13:54:41.319 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 13:54:41.321 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.321 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:54:41.323 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.323 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.324 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.324 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 13:54:41.339 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.340 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.341 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.341 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 13:54:41.354 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.354 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.355 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.355 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 13:54:41.369 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.369 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.370 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.370 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 13:54:41.383 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.384 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.384 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.385 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 13:54:41.390 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.390 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.391 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.391 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 13:54:41.395 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.395 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.396 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.396 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 13:54:41.401 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.401 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.402 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.402 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 13:54:41.408 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.408 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.409 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.409 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 13:54:41.415 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.415 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.415 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.416 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 13:54:41.421 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.421 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.422 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.423 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 13:54:41.431 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.431 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.432 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.432 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 13:54:41.446 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.446 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.446 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.447 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 13:54:41.451 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.451 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.452 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.452 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 13:54:41.458 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.459 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.459 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.460 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 13:54:41.465 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.465 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.465 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.466 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 13:54:41.470 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.470 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.471 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.471 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 13:54:41.475 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.476 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.476 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.476 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 13:54:41.480 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.480 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.481 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.481 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 13:54:41.486 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.487 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.487 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.488 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 13:54:41.491 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.492 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.492 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.493 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 13:54:41.497 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.497 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.498 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.498 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 13:54:41.502 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.503 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.503 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.503 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 13:54:41.507 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.508 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.508 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.509 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 13:54:41.514 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.514 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.515 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.515 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 13:54:41.519 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.519 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.520 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.520 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 13:54:41.525 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.525 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.526 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.526 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 13:54:41.530 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.531 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.531 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.531 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 13:54:41.536 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.536 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.537 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.537 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 13:54:41.542 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.542 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.543 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.543 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 13:54:41.548 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.548 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.549 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.549 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 13:54:41.553 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.554 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.554 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.554 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 13:54:41.560 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.560 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.561 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.561 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 13:54:41.565 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.566 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.566 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.566 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 13:54:41.573 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.574 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.574 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.575 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 13:54:41.581 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.581 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.582 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.582 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 13:54:41.588 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.588 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.589 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.589 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 13:54:41.594 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.594 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.595 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.595 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 13:54:41.599 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.600 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.600 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.601 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 13:54:41.604 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.605 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.605 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.606 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 13:54:41.609 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.610 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.610 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.610 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 13:54:41.614 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.614 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.615 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.615 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 13:54:41.624 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.624 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.624 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.625 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 13:54:41.644 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.645 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.646 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.646 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 13:54:41.652 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.652 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.653 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.654 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 13:54:41.659 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.660 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.660 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.661 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 13:54:41.667 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.667 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.668 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.668 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 13:54:41.672 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.672 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.673 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.673 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 13:54:41.677 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.677 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.678 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.678 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 13:54:41.682 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 13:54:41.682 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 13:54:41.683 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.684 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:54:41.685 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.685 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.685 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.686 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 13:54:41.691 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.691 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 13:54:41.692 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.692 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 13:54:41.696 INFO 21062 --- [ocp-schedules-2,56f88e6ef344271b,727c3c0d6603e0ac] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 13:54:41.820 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.821 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:54:41.822 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:54:41.822 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:54:41.823 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:54:41.849 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:54:41.849 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 13:54:41.983 INFO 21062 --- [prometheus-minute-schedule-1,006423dc804d671e,0b8d2022e6a36341] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:43.012 WARN 21062 --- [ocp-async-3,9bae6d89d9d4e2cf,3c2b22af598e05d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:43.057 INFO 21062 --- [ocp-async-10,a5403fc73211fb5a,42a0a0fbb7e6c26b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:43.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:43.154 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:44.983 INFO 21062 --- [prometheus-minute-schedule-1,59465434a8b5fd5b,6a3b7c7c69f81610] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:45.012 WARN 21062 --- [ocp-async-10,f3e1b758f4e84ca1,3075e754e8f6170d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:45.057 INFO 21062 --- [ocp-async-6,541a417baf4a6963,b3244a03ea8c13d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:46.154 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:46.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:46.848 INFO 21062 --- [ocp-schedules-3,7cef1d0c29b4b653,dde487ced6334848] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:54:47.012 WARN 21062 --- [ocp-async-6,a9059b1908b14cd5,c7ebf9750be42b71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:47.058 INFO 21062 --- [ocp-async-2,efe37f2b92e23cd5,75a5ee8000350b01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:47.983 INFO 21062 --- [prometheus-minute-schedule-1,3b4de018e22e1e16,adb2b19dac03e1da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:54:48.589954+08:00, lastProcessTime=2024-11-04T13:54:38.589590+08:00 2024-11-04 13:54:48.627 INFO 21062 --- [ocp-async-5,ac856e49d0d2e961,6fd01bfe7c6f7512] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:54:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:54:48.589954+08:00 2024-11-04 13:54:49.012 WARN 21062 --- [ocp-async-3,49e2670051b45d2b,6b0e1cb855c552a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:49.058 INFO 21062 --- [ocp-async-10,bfe74080d0aa0ffa,6be4ab06036ef1cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:49.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:49.156 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:49.971 INFO 21062 --- [ocp-schedules-11,7f2ae9d5220470a2,cc465442498c358b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:54:50.766 INFO 21062 --- [ocp-schedules-14,e514e709c8be5ef4,096457f7bb7765d2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:54:50.771 INFO 21062 --- [ocp-schedules-14,e514e709c8be5ef4,096457f7bb7765d2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:54:50.771 INFO 21062 --- [ocp-schedules-14,e514e709c8be5ef4,096457f7bb7765d2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:54:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1da5b50d9acb9b39,98e7cfcfb4b70ca7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:51.012 WARN 21062 --- [ocp-async-10,1b94f887e0b72fcb,6d9b031d2aa05a3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:51.058 INFO 21062 --- [ocp-async-5,37d07a0e14092cb0,159366f804a07a42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:52.156 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:52.157 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:53.012 WARN 21062 --- [ocp-async-5,9f0dd38984a95c50,0799788c46e0c339] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:53.058 INFO 21062 --- [ocp-async-2,e8d00854addd49b6,9c99cf69310e6aef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e76033fd4cb32dc8,87dcc7631e9fa173] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:55.013 WARN 21062 --- [ocp-async-2,36fb26542997cb8e,e88177e15754990c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:55.058 INFO 21062 --- [ocp-async-8,302c47e91ae23a89,a594a0f12b821094] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:55.157 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:55.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:56.873 INFO 21062 --- [ocp-schedules-9,d8f8d5f0964d93aa,952311cb2c44eb16] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:54:56.983 INFO 21062 --- [prometheus-minute-schedule-1,87d2be837f2d2439,87cd1c207843cd38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:54:57.013 WARN 21062 --- [ocp-async-8,18fe3f4642fda4ab,73286d80ef79c1e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:57.059 INFO 21062 --- [ocp-async-6,fdb76798f195b3e4,8c7acdd23df5d8fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:58.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:54:58.159 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:54:58.500 INFO 21062 --- [ocp-schedules-8,2c8ba84e7fe72872,51a6cfe81882c925] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:54:58.500 INFO 21062 --- [ocp-schedules-3,e4f0c1f1406e1978,4817f5e08d3ee99c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:54:58.502 INFO 21062 --- [ocp-schedules-2,77305dcad2c7f8cc,15827de185a887c4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:54:58.504 INFO 21062 --- [ocp-schedules-8,2c8ba84e7fe72872,51a6cfe81882c925] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:58.505 INFO 21062 --- [ocp-schedules-29,900c914036cca82f,f7d02674ee6ef2d7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:54:58.505 INFO 21062 --- [ocp-schedules-3,e4f0c1f1406e1978,4817f5e08d3ee99c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:54:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:54:58.510 INFO 21062 --- [ocp-schedules-10,ef96fdd52aeecf11,43841ba64856cfb9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:54:58.510 INFO 21062 --- [ocp-schedules-10,ef96fdd52aeecf11,43841ba64856cfb9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:54:58.510 INFO 21062 --- [ocp-schedules-14,aeb0b61f77e3b7e6,961d74d502cef3b3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:54:58.510 INFO 21062 --- [ocp-schedules-14,aeb0b61f77e3b7e6,961d74d502cef3b3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:54:58.510 INFO 21062 --- [ocp-schedules-28,746fcd87f5a54673,24bf93fd78ae4d8a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:54:58.510 INFO 21062 --- [ocp-schedules-28,746fcd87f5a54673,24bf93fd78ae4d8a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:54:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:54:58.589916+08:00, lastProcessTime=2024-11-04T13:54:48.589954+08:00 2024-11-04 13:54:58.627 INFO 21062 --- [ocp-async-9,f5aac7dd58d2716d,af22d6e32eb76279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:54:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:54:58.589916+08:00 2024-11-04 13:54:59.004 INFO 21062 --- [ocp-schedules-27,adb24b876c091742,e175a8d2e6a61f22] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:54:59.013 WARN 21062 --- [ocp-async-5,562f5b56afd940b4,b9b0118f6b8aeec9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:54:59.059 INFO 21062 --- [ocp-async-2,7edd1ee5c466f2bd,ec26fc0dc5d7fb04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:54:59.973 INFO 21062 --- [ocp-schedules-9,f67824b7bd016dba,de6c935304e27ecf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:54:59.983 INFO 21062 --- [prometheus-minute-schedule-1,b7cf819003845bf3,eb4cf673f314b846] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699700, interval=60 2024-11-04 13:55:00.357 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,70086d382f52065e] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:50:01.0, nextRun=2024-11-04 13:55:00.0, currentDate=Mon Nov 04 13:55:00 CST 2024 2024-11-04 13:55:00.358 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,70086d382f52065e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:50:01.0, nextRunTime=2024-11-04 13:55:00.0} 2024-11-04 13:55:00.385 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,70086d382f52065e] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 13:55:00.389 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,70086d382f52065e] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 13:55:00.389 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,70086d382f52065e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 13:55:01.013 WARN 21062 --- [ocp-async-2,3740e1c175b323f1,97c27eb06a9db344] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:01.059 INFO 21062 --- [ocp-async-8,66ff530575d22a49,0559c127ec9647a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:01.160 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:01.161 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e84eb4f5f8ed9e64,39d54d55dafa548d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:03.013 WARN 21062 --- [ocp-async-8,bdbcc88bab3e2228,d0c8e27acc40028a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:03.059 INFO 21062 --- [ocp-async-1,e30768f3dd83e149,65d99b7cb2ebcdb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:04.159 INFO 21062 --- [ocp-schedules-12,d260e815bf059cd3,04a7a19eef03e859] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:55:04.159 INFO 21062 --- [ocp-schedules-12,ea7a0de860c95b6e,7ff078c623dd0020] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:55:04.161 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:04.162 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:04.164 INFO 21062 --- [ocp-schedules-12,ea7a0de860c95b6e,7ff078c623dd0020] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:04.192 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:55:04.192 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:55:05.013 WARN 21062 --- [ocp-async-1,81a4ecf391af17fe,ca2d84afb7600243] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:05.059 INFO 21062 --- [ocp-async-4,3491e7914b77e804,75ea7c418ddfad8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:05.983 INFO 21062 --- [prometheus-minute-schedule-1,619727ef15a1c0ba,865086bacf3e1b6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:06.897 INFO 21062 --- [ocp-schedules-1,ab7774c2e1816caa,5874de2b0e50f579] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:55:07.014 WARN 21062 --- [ocp-async-4,c4cbf0f495948b2c,b213e7c41eed9458] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:07.059 INFO 21062 --- [ocp-async-3,47dd699b42cfc026,b31c578742c3e8d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:07.162 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:07.163 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:08.505 INFO 21062 --- [ocp-schedules-3,e4f0c1f1406e1978,4817f5e08d3ee99c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:55:08.505 INFO 21062 --- [ocp-schedules-8,2c8ba84e7fe72872,51a6cfe81882c925] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:55:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:55:08.589731+08:00, lastProcessTime=2024-11-04T13:54:58.589916+08:00 2024-11-04 13:55:08.628 INFO 21062 --- [ocp-async-7,0ade27d079796220,8d0607ee0ae0be93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:55:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:55:08.589731+08:00 2024-11-04 13:55:08.895 INFO 21062 --- [ocp-schedules-1,1851623acd8de397,4d3b032fd16a3e4a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:55:08.899 INFO 21062 --- [ocp-schedules-1,1851623acd8de397,4d3b032fd16a3e4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:08.899 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:55:08.983 INFO 21062 --- [prometheus-minute-schedule-1,845c32c4fd61b8cd,cbc2cc789b05d7a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:09.014 WARN 21062 --- [ocp-async-8,7948fffd365a675c,2ef1ee351382d306] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:09.060 INFO 21062 --- [ocp-async-6,f35190c74f21899c,b8d8dc1dbe4acc85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:09.976 INFO 21062 --- [ocp-schedules-7,749a1f354dc00899,9b5cd3daa6f726e3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:55:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:55:10.004 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:55:10.163 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:10.164 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:55:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:55:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:55:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:55:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:55:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:55:10.773 INFO 21062 --- [ocp-schedules-15,7ae8650f2af1a273,b91ea3bf862e5b5e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:55:10.779 INFO 21062 --- [ocp-schedules-15,7ae8650f2af1a273,b91ea3bf862e5b5e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:55:10.779 INFO 21062 --- [ocp-schedules-15,7ae8650f2af1a273,b91ea3bf862e5b5e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:55:11.014 WARN 21062 --- [ocp-async-6,54a65f108045e609,c9d760c30d9a4629] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:11.060 INFO 21062 --- [ocp-async-7,76f0c96ec16a9144,fa7ffd87bd7d1f2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:11.397 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=1, affectRows=1 2024-11-04 13:55:11.397 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=1, validCount=1 2024-11-04 13:55:11.398 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=10, affectRows=10 2024-11-04 13:55:11.398 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=10, validCount=10 2024-11-04 13:55:11.399 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=13, affectRows=13 2024-11-04 13:55:11.399 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=13, validCount=13 2024-11-04 13:55:11.399 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=22, affectRows=22 2024-11-04 13:55:11.399 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=25, affectRows=25 2024-11-04 13:55:11.399 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=22, validCount=22 2024-11-04 13:55:11.399 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=25, validCount=25 2024-11-04 13:55:11.399 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=43, affectRows=43 2024-11-04 13:55:11.400 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=43, affectRows=43 2024-11-04 13:55:11.400 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=42, validCount=42 2024-11-04 13:55:11.400 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=43, validCount=43 2024-11-04 13:55:11.400 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=71, affectRows=71 2024-11-04 13:55:11.400 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=85, affectRows=85 2024-11-04 13:55:11.401 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=71, validCount=71 2024-11-04 13:55:11.401 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=83, validCount=83 2024-11-04 13:55:11.401 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699640, nodeCount=79, affectRows=79 2024-11-04 13:55:11.401 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699640, nodeCount=79, validCount=79 2024-11-04 13:55:11.983 INFO 21062 --- [prometheus-minute-schedule-1,d9f9c870dee6ff46,6ae1c8e3e81a10ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:13.014 WARN 21062 --- [ocp-async-4,b5373f173e4fe6a5,1d808314c6dc6324] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:13.060 INFO 21062 --- [ocp-async-9,1bd9ba01ce90237e,eedbd62fb09eac7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:13.164 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:13.165 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:13.985 INFO 21062 --- [prometheus-cache-monitor1,96f9e768c53308be,29a0a51206b900f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:55:13.985 INFO 21062 --- [prometheus-cache-monitor1,96f9e768c53308be,29a0a51206b900f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:55:14.787 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:51:14.769, end=2024-11-04T13:53:14.784} 2024-11-04 13:55:14.797 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:55:14.797 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:55:14.983 INFO 21062 --- [prometheus-minute-schedule-1,54d77e31da5dc911,0d0f6fd4bd66df28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:15.014 WARN 21062 --- [ocp-async-3,d3d65460e59a9da8,fecaeeec71e4251d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:15.060 INFO 21062 --- [ocp-async-10,6836cd0899bde9f1,fb5c547d88559df6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:16.165 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:16.166 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:16.936 INFO 21062 --- [ocp-schedules-16,d0ceec45483af894,d41eb83217297e37] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:55:17.014 WARN 21062 --- [ocp-async-1,03c7cbbc30b8a07b,b49d0855b5a3c528] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:17.060 INFO 21062 --- [ocp-async-5,a6bd234ac88f8492,aabc9de5adbcdc71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:17.983 INFO 21062 --- [prometheus-minute-schedule-1,1a77123d5ebc504e,e5bb54e511de2b5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:55:18.590566+08:00, lastProcessTime=2024-11-04T13:55:08.589731+08:00 2024-11-04 13:55:18.628 INFO 21062 --- [ocp-async-10,9a0722a8802c330e,a5f8057881d75feb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:55:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:55:18.590566+08:00 2024-11-04 13:55:18.899 INFO 21062 --- [ocp-schedules-1,1851623acd8de397,4d3b032fd16a3e4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:55:19.015 WARN 21062 --- [ocp-async-4,822c1591165743f8,70c9fab08691c920] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:19.060 INFO 21062 --- [ocp-async-9,d8a78bee8fea222f,870c6c304f5fbc20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:19.166 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:19.167 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:19.979 INFO 21062 --- [ocp-schedules-15,ab3d1cc3d210bb4a,6d48e771779ff97d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:55:20.983 INFO 21062 --- [prometheus-minute-schedule-1,70fd6a5b286703e6,6cbe8aab838f5890] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:21.015 WARN 21062 --- [ocp-async-3,74e06ba9733c589f,a9f1575cae465b9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:21.061 INFO 21062 --- [ocp-async-10,17710c3c4f7cd9fc,b713af7ddf4bc5fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:22.167 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:22.168 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:23.015 WARN 21062 --- [ocp-async-1,516353f677d5715e,83fb0d3d63e1f084] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:23.061 INFO 21062 --- [ocp-async-5,fe2d465d6e7ab451,b2cadec42f583d13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:23.983 INFO 21062 --- [prometheus-minute-schedule-1,d549196597dfb2cd,cc4037679ad66646] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:25.015 WARN 21062 --- [ocp-async-7,2fdd732a748717e2,0a02b972e7e20360] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:25.061 INFO 21062 --- [ocp-async-2,31bd077cfb5ae754,fd1bc69caee17a9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:25.168 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:25.168 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:26.961 INFO 21062 --- [ocp-schedules-10,bb3de20c5f6ba70e,8e06487401485c71] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:55:26.983 INFO 21062 --- [prometheus-minute-schedule-1,8237953ba099e751,59597079f33c800c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:26.984 INFO 21062 --- [prometheus-hour-schedule-1,124e216f2e86bbd9,83d0d71eb7813199] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:55:27.015 WARN 21062 --- [ocp-async-9,2936b003b67a7b01,b7ece738e3595509] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:27.061 INFO 21062 --- [ocp-async-8,dacb8888b129faf2,ca17dc5117ed69cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:28.169 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:28.169 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:28.498 INFO 21062 --- [ocp-schedules-5,4b2e2066de859658,838d28d494d0f712] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:55:28.499 INFO 21062 --- [ocp-schedules-9,1315d07b15492750,f26a365c3056d830] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:55:28.500 INFO 21062 --- [ocp-schedules-13,8a1475e95a07f973,3baf5ea81177a92d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:55:28.500 INFO 21062 --- [ocp-schedules-19,c0baea9b5f4c7302,df5ee766ae669fef] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:55:28.502 INFO 21062 --- [ocp-schedules-5,4b2e2066de859658,838d28d494d0f712] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:55:28.502 INFO 21062 --- [ocp-schedules-24,afc508da11121496,53288b6a7e956c02] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:55:28.504 INFO 21062 --- [ocp-schedules-13,8a1475e95a07f973,3baf5ea81177a92d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:28.504 INFO 21062 --- [ocp-schedules-19,c0baea9b5f4c7302,df5ee766ae669fef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:55:28.504 INFO 21062 --- [ocp-schedules-21,f7bf46b9afacc79d,3cbac1a54e615e4e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:55:28.505 INFO 21062 --- [ocp-schedules-9,1315d07b15492750,f26a365c3056d830] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:55:28.510 INFO 21062 --- [ocp-schedules-18,ba78bec13ff81dd9,7a54e1f9314cd4de] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:55:28.510 INFO 21062 --- [ocp-schedules-11,b5452f9e34e08934,aba18e9d05fa66fe] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:55:28.510 INFO 21062 --- [ocp-schedules-11,b5452f9e34e08934,aba18e9d05fa66fe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:55:28.510 INFO 21062 --- [ocp-schedules-8,b9e8893da8084418,10848517b2115fcb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:55:28.510 INFO 21062 --- [ocp-schedules-18,ba78bec13ff81dd9,7a54e1f9314cd4de] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:55:28.510 INFO 21062 --- [ocp-schedules-8,b9e8893da8084418,10848517b2115fcb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:55:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:55:28.589081+08:00, lastProcessTime=2024-11-04T13:55:18.590566+08:00 2024-11-04 13:55:28.628 INFO 21062 --- [ocp-async-3,d2fefa9c354c515c,5714f1568af0780e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:55:28.754 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:55:28.589081+08:00 2024-11-04 13:55:28.761 INFO 21062 --- [ocp-schedules-7,cba317bd5f6a1067,ee5bc28bd92193bb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:55:28.964 INFO 21062 --- [ocp-schedules-16,1fe639dfd902b5de,e1217a0b06693064] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:55:28.965 INFO 21062 --- [ocp-schedules-16,1fe639dfd902b5de,e1217a0b06693064] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:55:29.006 INFO 21062 --- [ocp-schedules-29,fc60e7d68c5558d6,71e4b460da00bdf7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:55:29.015 WARN 21062 --- [ocp-async-5,56c66ce4819108f8,558271594da6403d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:29.040 INFO 21062 --- [ocp-schedules-11,b3813525b1798385,47ed1265c6e34de5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:55:29.045 INFO 21062 --- [ocp-schedules-11,b3813525b1798385,47ed1265c6e34de5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:55:29.046 INFO 21062 --- [ocp-schedules-11,b3813525b1798385,47ed1265c6e34de5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:55:29.061 INFO 21062 --- [ocp-async-2,202a4d454148a209,af99acebd6a0a72b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:29.128 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:55:29.128 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:55:29.136 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:55:29.136 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:55:29.136 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:55:29.149 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:55:29.149 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:55:29.149 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:55:29.153 INFO 21062 --- [ocp-schedules-17,e91c5779b6c0ec50,9c84b253b6b85e0d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:55:29.313 INFO 21062 --- [ocp-schedules-10,8cc90ad4e7b0a026,08317f0580ba8076] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:55:29.313 INFO 21062 --- [ocp-schedules-10,8cc90ad4e7b0a026,08317f0580ba8076] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:55:29.315 INFO 21062 --- [ocp-schedules-10,8cc90ad4e7b0a026,08317f0580ba8076] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:55:29.315 INFO 21062 --- [ocp-schedules-10,8cc90ad4e7b0a026,08317f0580ba8076] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:55:29.315 INFO 21062 --- [ocp-schedules-10,8cc90ad4e7b0a026,08317f0580ba8076] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:55:29.315 INFO 21062 --- [ocp-schedules-10,8cc90ad4e7b0a026,08317f0580ba8076] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:55:29.983 INFO 21062 --- [ocp-schedules-2,27336fbf68b0c6a5,cf23121368223979] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:55:29.983 INFO 21062 --- [prometheus-minute-schedule-1,dcc9656719b906da,cef38eea542bccc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:55:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:55:30.781 INFO 21062 --- [ocp-schedules-18,db51b334eda18f50,7209154397940b87] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:55:30.786 INFO 21062 --- [ocp-schedules-18,db51b334eda18f50,7209154397940b87] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:55:30.786 INFO 21062 --- [ocp-schedules-18,db51b334eda18f50,7209154397940b87] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:55:30.787 INFO 21062 --- [ocp-schedules-20,c6452ca54c446d93,d82f80f602c62a05] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:55:30.787 INFO 21062 --- [ocp-schedules-20,c6452ca54c446d93,d82f80f602c62a05] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:55:30.815 INFO 21062 --- [ocp-schedules-20,c6452ca54c446d93,d82f80f602c62a05] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:55:30.815 INFO 21062 --- [ocp-schedules-20,c6452ca54c446d93,d82f80f602c62a05] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:55:30.815 INFO 21062 --- [ocp-schedules-25,313cde90508c643c,abe9a0a037e74d51] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:55:30.968 INFO 21062 --- [ocp-schedules-27,fd4002fa6d44f3b3,9e680cd90108c9b8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:55:30.968+08:00, clusters=[1] 2024-11-04 13:55:30.969 INFO 21062 --- [ocp-schedules-27,fd4002fa6d44f3b3,9e680cd90108c9b8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:55:31.016 WARN 21062 --- [ocp-async-7,5c0d8dcec12d60c5,9384c5e3801dc6fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:31.036 INFO 21062 --- [ocp-schedules-30,9ed586f684a5f213,ad63ad6f5e8ee8e7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:55:31.036 INFO 21062 --- [ocp-schedules-30,9ed586f684a5f213,ad63ad6f5e8ee8e7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:55:31.036 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:55:31.036 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:55:31.047 INFO 21062 --- [ocp-schedules-30,9ed586f684a5f213,ad63ad6f5e8ee8e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:55:31.048 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:55:31.048 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:55:31.049 WARN 21062 --- [ocp-schedules-30,9ed586f684a5f213,ad63ad6f5e8ee8e7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:55:31.049 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:55:31.049 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:55:31.049 INFO 21062 --- [ocp-schedules-30,9ed586f684a5f213,ad63ad6f5e8ee8e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:55:31.050 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:55:31.050 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:55:31.051 WARN 21062 --- [ocp-schedules-30,9ed586f684a5f213,ad63ad6f5e8ee8e7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:55:31.051 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:55:31.051 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:55:31.057 ERROR 21062 --- [ocp-schedules-30,9ed586f684a5f213,ad63ad6f5e8ee8e7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:55:31.061 INFO 21062 --- [ocp-async-3,ac4acb990b43c6ba,d7327bd0e32aeb6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:31.119 INFO 21062 --- [ocp-schedules-3,5150a7b81cc86704,9ab97873626ddc95] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:55:31.119+08:00, clusters=[1] 2024-11-04 13:55:31.122 INFO 21062 --- [ocp-schedules-3,5150a7b81cc86704,9ab97873626ddc95] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:55:31.170 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:31.170 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a25f680c4a406e1c,107906bc5fa2a6d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:33.016 WARN 21062 --- [ocp-async-8,367388cd1fc8a19e,c13409f5d0403b21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:33.062 INFO 21062 --- [ocp-async-1,6e51d55bfa076365,53cf7f9f2be313d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:34.170 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:34.171 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:34.193 INFO 21062 --- [ocp-schedules-12,ea7a0de860c95b6e,7ff078c623dd0020] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:55:34.193 INFO 21062 --- [ocp-schedules-12,5786854138ecf09d,0b30d5e4db905bbe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:55:34.197 INFO 21062 --- [ocp-schedules-12,5786854138ecf09d,0b30d5e4db905bbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:34.225 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:55:34.225 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:55:35.016 WARN 21062 --- [ocp-async-6,7354746be645fb8f,11bf727aa8a73a09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:35.062 INFO 21062 --- [ocp-async-4,75764087a832b6e9,3edf75058a137a9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0928935ebdafdcd9,d36183aa9daf70df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:36.173 INFO 21062 --- [ocp-schedules-7,7f5be2a96eb75771,ebf3f7902c9e2fe1] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:55:36.176 INFO 21062 --- [ocp-schedules-7,7f5be2a96eb75771,ebf3f7902c9e2fe1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:36.261 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:55:36.265 ERROR 21062 --- [ocp-schedules-7,7f5be2a96eb75771,ebf3f7902c9e2fe1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:55:36.265 INFO 21062 --- [ocp-schedules-7,7f5be2a96eb75771,ebf3f7902c9e2fe1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:55:37.002 INFO 21062 --- [ocp-schedules-9,632e5fbfbe773422,a9babbfd26d0bf79] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:55:37.016 WARN 21062 --- [ocp-async-4,86bb78de11558c61,fb287521d9263d4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:37.062 INFO 21062 --- [ocp-async-9,520c0cbc819c3279,ba4fcf9f70d8c066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:37.172 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:37.172 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:38.504 INFO 21062 --- [ocp-schedules-19,c0baea9b5f4c7302,df5ee766ae669fef] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:55:38.505 INFO 21062 --- [ocp-schedules-13,8a1475e95a07f973,3baf5ea81177a92d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:55:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:55:38.590718+08:00, lastProcessTime=2024-11-04T13:55:28.589081+08:00 2024-11-04 13:55:38.628 INFO 21062 --- [ocp-async-2,9d2eae828ab43330,3eedfcf9b4b4be6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:55:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:55:38.590718+08:00 2024-11-04 13:55:38.983 INFO 21062 --- [prometheus-minute-schedule-1,9498b8e7ab6909d8,ed628c7be4d0f165] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:39.016 WARN 21062 --- [ocp-async-8,28089de9ba7f8e39,1aae68404191e09b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:39.062 INFO 21062 --- [ocp-async-6,7d2a66886c751810,359e7d07c7407555] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:39.986 INFO 21062 --- [ocp-schedules-24,fe607986698e28bc,506b6571911cbc78] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:55:40.002 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:55:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:55:40.172 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:40.173 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:41.016 WARN 21062 --- [ocp-async-6,77a9aecb97048fcd,dfc81051353f2c80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:41.062 INFO 21062 --- [ocp-async-2,4c662bd22c1fe087,d70a456ca318c05c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bb5d4e0c993d17a4,53c4349c0c46d642] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:43.017 WARN 21062 --- [ocp-async-2,f425e3362b14aaa1,a6b0ec2a470b046d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:43.063 INFO 21062 --- [ocp-async-3,224120a409a626b1,bde81926fb172396] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:43.173 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:43.174 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:44.983 INFO 21062 --- [prometheus-minute-schedule-1,4c5a4da31c21adfc,dde885e078897799] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:45.017 WARN 21062 --- [ocp-async-3,e94ec79cb3207f73,001fcc2b2dc8a157] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:45.063 INFO 21062 --- [ocp-async-10,42aa2c6ec9d8981c,04376f2ad8b06d9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:46.174 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:46.175 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:47.017 WARN 21062 --- [ocp-async-10,6459ac498d785fb5,bc5f8c97fc6c47a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:47.027 INFO 21062 --- [ocp-schedules-21,9d09be0eeda24a29,5296b904d786b717] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:55:47.063 INFO 21062 --- [ocp-async-5,71e0120cd18123b9,f32908f6d5c9cef8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:47.983 INFO 21062 --- [prometheus-minute-schedule-1,a23295d8b1a22985,96399d2caf4d15f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:55:48.589945+08:00, lastProcessTime=2024-11-04T13:55:38.590718+08:00 2024-11-04 13:55:48.628 INFO 21062 --- [ocp-async-1,5debad12575eaf37,b24c416a4309380b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:55:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:55:48.589945+08:00 2024-11-04 13:55:49.017 WARN 21062 --- [ocp-async-2,f39d5daeed5ac156,9aa175264d92f4c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:49.063 INFO 21062 --- [ocp-async-3,a86ec0f5b67fb3cf,6fc35a85e95d79c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:49.175 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:49.176 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:49.989 INFO 21062 --- [ocp-schedules-7,f22e80c2d86f7e20,227980f8376d62ef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:55:50.789 INFO 21062 --- [ocp-schedules-25,8e433fb1ff79c368,dafb8ba8ba91281a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:55:50.795 INFO 21062 --- [ocp-schedules-25,8e433fb1ff79c368,dafb8ba8ba91281a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:55:50.795 INFO 21062 --- [ocp-schedules-25,8e433fb1ff79c368,dafb8ba8ba91281a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:55:50.983 INFO 21062 --- [prometheus-minute-schedule-1,3aff258697eb6df0,21f502dd829c36fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:51.018 WARN 21062 --- [ocp-async-3,87a78475c4592a74,d53f1d3442bc575f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:51.063 INFO 21062 --- [ocp-async-10,e9360d4f5a8b5ff9,e4e3f5114f4feb7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:52.176 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:52.177 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:53.018 WARN 21062 --- [ocp-async-10,7cf5bba2931a6816,e2e019575e93beb8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:53.064 INFO 21062 --- [ocp-async-5,40ab36879c861fb7,bcd6acf06cd7819f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:53.983 INFO 21062 --- [prometheus-minute-schedule-1,279545ef470cc071,67bc1eb0e088785f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:55.018 WARN 21062 --- [ocp-async-5,b40209e956b35a45,6c7c992c22e21ff7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:55.064 INFO 21062 --- [ocp-async-9,6c9f2dcd8e0fb2e6,cc8e80430f0c0a38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:55.177 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:55.178 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:56.983 INFO 21062 --- [prometheus-minute-schedule-1,560b0b9167f23094,63b6b98bccdee7cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:57.018 WARN 21062 --- [ocp-async-9,a0a81497bce2b4ee,c3a60dfae479c6d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:57.064 INFO 21062 --- [ocp-async-1,26876ce489de7dc6,65dc1018a9c1bbbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:57.072 INFO 21062 --- [ocp-schedules-24,d761cf9055dad581,bfe1ba45a5fece0a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:55:58.178 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:55:58.179 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:55:58.500 INFO 21062 --- [ocp-schedules-1,6b3346f8a45a0474,424783c3c14d08a6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:55:58.500 INFO 21062 --- [ocp-schedules-14,8ec6968132adfb9c,fa0d1947f8693541] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:55:58.502 INFO 21062 --- [ocp-schedules-11,7f0e5e75fe22e334,b8a98f7b005c185a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:55:58.504 INFO 21062 --- [ocp-schedules-14,8ec6968132adfb9c,fa0d1947f8693541] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:58.504 INFO 21062 --- [ocp-schedules-1,6b3346f8a45a0474,424783c3c14d08a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:55:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:55:58.505 INFO 21062 --- [ocp-schedules-28,074279300fc0f580,65839324f8c63441] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:55:58.510 INFO 21062 --- [ocp-schedules-13,c2295a2a9537aa39,4eab4ae76d34c18d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:55:58.510 INFO 21062 --- [ocp-schedules-24,0c36c072ce95765a,ba714751f0e9514d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:55:58.511 INFO 21062 --- [ocp-schedules-24,0c36c072ce95765a,ba714751f0e9514d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:55:58.511 INFO 21062 --- [ocp-schedules-25,81003523504b0acc,cb33328a300a1545] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:55:58.511 INFO 21062 --- [ocp-schedules-13,c2295a2a9537aa39,4eab4ae76d34c18d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:55:58.511 INFO 21062 --- [ocp-schedules-25,81003523504b0acc,cb33328a300a1545] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:55:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:55:58.589684+08:00, lastProcessTime=2024-11-04T13:55:48.589945+08:00 2024-11-04 13:55:58.628 INFO 21062 --- [ocp-async-7,c1886c25c5bf130c,739dda12cfc7e579] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:55:58.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:55:58.589684+08:00 2024-11-04 13:55:59.009 INFO 21062 --- [ocp-schedules-23,3552ffdeafde499e,c6facf888d108462] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:55:59.018 WARN 21062 --- [ocp-async-10,fa33560468fbae26,e3367bafc7b72dd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:55:59.064 INFO 21062 --- [ocp-async-5,fe32c7dd26c0fc3a,490c77b4516e599f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:55:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2a40c064083544f2,00e5797e7b515bde] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699760, interval=60 2024-11-04 13:55:59.991 INFO 21062 --- [ocp-schedules-28,ade2231f54476376,dea612afa92e8823] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:56:01.019 WARN 21062 --- [ocp-async-5,76fc65df0a63a071,162a73e2a8970390] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:01.064 INFO 21062 --- [ocp-async-9,130ab22a93098942,3c440097130d528d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:01.179 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:01.180 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:02.983 INFO 21062 --- [prometheus-minute-schedule-1,dcd65a8ab979151f,305762ff920e4990] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:03.019 WARN 21062 --- [ocp-async-9,5ec663a934ab9d45,24737d7594c18625] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:03.065 INFO 21062 --- [ocp-async-8,417f3654cd8b3c60,2662ddf72e71e479] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:04.180 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:04.181 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:04.226 INFO 21062 --- [ocp-schedules-12,5786854138ecf09d,0b30d5e4db905bbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:56:04.226 INFO 21062 --- [ocp-schedules-12,4ab11b6770090616,b8fb90ffe666e951] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:56:04.231 INFO 21062 --- [ocp-schedules-12,4ab11b6770090616,b8fb90ffe666e951] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:04.261 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:56:04.261 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:56:05.019 WARN 21062 --- [ocp-async-8,83e79b2bb7ebb1c2,4f83a0b1150df72f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:05.065 INFO 21062 --- [ocp-async-6,42604c9b78d8c9ef,0615808828c07955] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a94220fcada24fba,324599c63dddc258] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:07.019 WARN 21062 --- [ocp-async-6,9331caf216f5e0fa,b075eb8f2ab82307] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:07.065 INFO 21062 --- [ocp-async-2,e343dab74684de58,a3d5c188ef7b69e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:07.098 INFO 21062 --- [ocp-schedules-9,fdc50faf15bf0d44,05c6b6962bc50099] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:56:07.181 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:07.181 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:08.504 INFO 21062 --- [ocp-schedules-14,8ec6968132adfb9c,fa0d1947f8693541] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:56:08.505 INFO 21062 --- [ocp-schedules-1,6b3346f8a45a0474,424783c3c14d08a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:56:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:56:08.589732+08:00, lastProcessTime=2024-11-04T13:55:58.589684+08:00 2024-11-04 13:56:08.629 INFO 21062 --- [ocp-async-4,480616990cccdc05,cc8d21163fed25ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:56:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:56:08.589732+08:00 2024-11-04 13:56:08.983 INFO 21062 --- [prometheus-minute-schedule-1,0b1ab41abbab1c5f,feeac6f1ffe8baf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:09.019 WARN 21062 --- [ocp-async-9,a7c3ef60ff32aa7e,de6e6caed6fd1a97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:09.065 INFO 21062 --- [ocp-async-1,eaed2c8cd4320e6c,32eee75db0daa8be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:09.994 INFO 21062 --- [ocp-schedules-5,757b105a0f8f2fa2,1d8edfa41eb35c89] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:56:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:56:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:56:10.181 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:10.182 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:56:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:56:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:56:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:56:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:56:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:56:10.798 INFO 21062 --- [ocp-schedules-24,31d3aa042ad4cce8,8c168cc676d92238] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:56:10.804 INFO 21062 --- [ocp-schedules-24,31d3aa042ad4cce8,8c168cc676d92238] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:56:10.804 INFO 21062 --- [ocp-schedules-24,31d3aa042ad4cce8,8c168cc676d92238] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:56:11.019 WARN 21062 --- [ocp-async-1,6da5e03a995d6696,797d2d0b18fc79d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:11.065 INFO 21062 --- [ocp-async-4,9226f49bdc4cefd4,7ba7cb25a0ea8d4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=2, affectRows=2 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=2, validCount=2 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=14, affectRows=14 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=14, validCount=14 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=24, affectRows=24 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=10, affectRows=10 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=10, validCount=10 2024-11-04 13:56:11.402 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=24, validCount=24 2024-11-04 13:56:11.403 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=15, affectRows=15 2024-11-04 13:56:11.403 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=38, affectRows=38 2024-11-04 13:56:11.403 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=15, validCount=15 2024-11-04 13:56:11.403 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=34, affectRows=34 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=38, validCount=38 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=34, validCount=34 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=49, affectRows=49 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=70, affectRows=70 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=63, affectRows=63 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699700, nodeCount=73, affectRows=73 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=49, validCount=49 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=68, validCount=68 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=62, validCount=62 2024-11-04 13:56:11.404 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699700, nodeCount=73, validCount=73 2024-11-04 13:56:11.983 INFO 21062 --- [prometheus-minute-schedule-1,56464b1a6b65b45b,c9c47dd41416c840] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:13.020 WARN 21062 --- [ocp-async-6,011b7e5f2b388bd0,055467e355482e9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:13.065 INFO 21062 --- [ocp-async-7,383a09e2c986ca70,b227ab65b663097a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:13.183 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:13.183 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:13.984 INFO 21062 --- [prometheus-cache-monitor1,a60c8b641c4a26d3,aff8301fafba78ed] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:56:13.984 INFO 21062 --- [prometheus-cache-monitor1,a60c8b641c4a26d3,aff8301fafba78ed] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:56:14.983 INFO 21062 --- [prometheus-minute-schedule-1,82ed36bdff5c690c,5509b4ffa0b5a8a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:15.020 WARN 21062 --- [ocp-async-2,0a44ab443504f163,3290e6d3b41f5157] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:15.065 INFO 21062 --- [ocp-async-3,c31e8423cdbe3e29,34fe1dd823af74db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:16.183 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:16.184 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:17.020 WARN 21062 --- [ocp-async-8,f7bb0c9ee5a1f4c1,6a56f7310323146b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:17.066 INFO 21062 --- [ocp-async-10,bf475d91815f7445,1078ca3a1ec82fd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:17.139 INFO 21062 --- [ocp-schedules-21,713e87731d4a02a7,fcf28e357b43c755] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:56:17.983 INFO 21062 --- [prometheus-minute-schedule-1,982758fb7e0e71d8,7e47970efcad20e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:56:18.589258+08:00, lastProcessTime=2024-11-04T13:56:08.589732+08:00 2024-11-04 13:56:18.629 INFO 21062 --- [ocp-async-3,5b39b55da7b9d241,b31f32a58dfc88db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:56:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:56:18.589258+08:00 2024-11-04 13:56:18.900 INFO 21062 --- [ocp-schedules-22,c014059a0fe215a3,b08c1929b25a20dd] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:56:18.903 INFO 21062 --- [ocp-schedules-22,c014059a0fe215a3,b08c1929b25a20dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:18.904 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:56:19.020 WARN 21062 --- [ocp-async-6,bce87c5ea8830a2a,855a6b43a18e9a55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:19.066 INFO 21062 --- [ocp-async-7,ac0b743d6d89a715,40e90a9ca5efa790] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:19.184 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:19.185 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:19.997 INFO 21062 --- [ocp-schedules-19,b6ed80f2883c4e41,213cdca018217142] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:56:20.983 INFO 21062 --- [prometheus-minute-schedule-1,0e7e44a343b305e7,491d11eea7092d1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:21.020 WARN 21062 --- [ocp-async-2,3ab692e21de0818d,d68a025dfd5edc3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:21.066 INFO 21062 --- [ocp-async-3,a464fdfaf95ae518,899b6007dfc7cc10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:22.185 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:22.186 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:23.020 WARN 21062 --- [ocp-async-8,3b947cca0e15b4cb,b0e00e24e4730975] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:23.066 INFO 21062 --- [ocp-async-10,01de184c4804f9d2,f0c9bf0d69724677] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:23.983 INFO 21062 --- [prometheus-minute-schedule-1,5ebe6b746c1fb926,96e5dca96fb247bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:25.020 WARN 21062 --- [ocp-async-4,59d0aed422db1c18,61dfe0989bacf977] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:25.066 INFO 21062 --- [ocp-async-5,f83f79a99ab8d01c,c884c2e7a8b72179] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:25.186 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:25.187 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:26.983 INFO 21062 --- [prometheus-minute-schedule-1,d4e1993e4a3cc114,f8f3649dbf6e306c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2019ddd812242961,dbd74add5d4b1954] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:56:27.021 WARN 21062 --- [ocp-async-7,5b7abdd767abd5fb,5691173dc4891014] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:27.067 INFO 21062 --- [ocp-async-9,b4f02f88a7773741,6ae0ab7424e5cfcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:27.165 INFO 21062 --- [ocp-schedules-5,04d1a17c6a9b1cab,aa0568b3551b62c6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:56:28.187 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:28.188 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:28.498 INFO 21062 --- [ocp-schedules-7,5dbada2bb34d4ec1,f81297760c7c4f9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:56:28.499 INFO 21062 --- [ocp-schedules-9,7b95867c9a8a6609,45f61785ad094e40] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:56:28.500 INFO 21062 --- [ocp-schedules-27,0a0f67b523559aa8,d4731d7dc298267d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:56:28.500 INFO 21062 --- [ocp-schedules-26,8d53e6e415a5e6e8,e85275f9283e197a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:56:28.502 INFO 21062 --- [ocp-schedules-7,5dbada2bb34d4ec1,f81297760c7c4f9c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:56:28.502 INFO 21062 --- [ocp-schedules-30,6771e7de54e5c6fe,cc2a06b74e035324] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:56:28.504 INFO 21062 --- [ocp-schedules-27,0a0f67b523559aa8,d4731d7dc298267d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:28.504 INFO 21062 --- [ocp-schedules-26,8d53e6e415a5e6e8,e85275f9283e197a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:56:28.505 INFO 21062 --- [ocp-schedules-29,aa1fd6de4e782155,a432a49e7951799b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:56:28.505 INFO 21062 --- [ocp-schedules-9,7b95867c9a8a6609,45f61785ad094e40] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:56:28.510 INFO 21062 --- [ocp-schedules-14,c26c26f826428007,f7b5e253aa849ce6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:56:28.511 INFO 21062 --- [ocp-schedules-14,c26c26f826428007,f7b5e253aa849ce6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:56:28.511 INFO 21062 --- [ocp-schedules-8,90e980dc67ce1b3e,177c06b11632b4c9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:56:28.511 INFO 21062 --- [ocp-schedules-5,c640a7178ac42eb6,1ca8c1e10cbe2592] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:56:28.511 INFO 21062 --- [ocp-schedules-5,c640a7178ac42eb6,1ca8c1e10cbe2592] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:56:28.511 INFO 21062 --- [ocp-schedules-8,90e980dc67ce1b3e,177c06b11632b4c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:56:28.571 INFO 21062 --- [ocp-schedules-28,cf7a8e0dcf60c399,8440a0c31a310a41] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:56:28.571 INFO 21062 --- [ocp-schedules-28,cf7a8e0dcf60c399,8440a0c31a310a41] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:56:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:56:28.589300+08:00, lastProcessTime=2024-11-04T13:56:18.589258+08:00 2024-11-04 13:56:28.629 INFO 21062 --- [ocp-async-2,b5d762e34e58fa6b,5011350936d6f3a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:56:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:56:28.589300+08:00 2024-11-04 13:56:28.763 INFO 21062 --- [ocp-schedules-25,cd5a0c2f5fc8d67f,fecfdfd9dc6cc061] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:56:28.904 INFO 21062 --- [ocp-schedules-22,c014059a0fe215a3,b08c1929b25a20dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:56:28.969 INFO 21062 --- [ocp-schedules-1,69a01a52c2c52fab,8871a910b7e1257a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:56:28.969 INFO 21062 --- [ocp-schedules-1,69a01a52c2c52fab,8871a910b7e1257a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:56:29.012 INFO 21062 --- [ocp-schedules-29,911b450c76f66a7f,7d6963acf4b8f47f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:56:29.021 WARN 21062 --- [ocp-async-10,00bdf28d49316303,306319af6f228eeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:29.046 INFO 21062 --- [ocp-schedules-5,34751fbeb7201abf,6437a8f7a5ae0ad2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:56:29.052 INFO 21062 --- [ocp-schedules-5,34751fbeb7201abf,6437a8f7a5ae0ad2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:56:29.052 INFO 21062 --- [ocp-schedules-5,34751fbeb7201abf,6437a8f7a5ae0ad2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:56:29.067 INFO 21062 --- [ocp-async-4,10c9695491854f5d,919895b7fa0f012b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:29.316 INFO 21062 --- [ocp-schedules-19,8587695ca7ed4791,4418dc09d2e52b85] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:56:29.316 INFO 21062 --- [ocp-schedules-19,8587695ca7ed4791,4418dc09d2e52b85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:56:29.318 INFO 21062 --- [ocp-schedules-19,8587695ca7ed4791,4418dc09d2e52b85] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:56:29.318 INFO 21062 --- [ocp-schedules-19,8587695ca7ed4791,4418dc09d2e52b85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:56:29.319 INFO 21062 --- [ocp-schedules-19,8587695ca7ed4791,4418dc09d2e52b85] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:56:29.319 INFO 21062 --- [ocp-schedules-19,8587695ca7ed4791,4418dc09d2e52b85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:56:29.983 INFO 21062 --- [prometheus-minute-schedule-1,0b1c99202aea323c,b6ba7e5b003e7537] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:30.000 INFO 21062 --- [ocp-schedules-18,babf88e23779b1eb,aa2b95e1c07c0712] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:56:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:56:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:56:30.806 INFO 21062 --- [ocp-schedules-6,bf832cf3667ac1b8,1bc5ea60df2922e6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:56:30.812 INFO 21062 --- [ocp-schedules-6,bf832cf3667ac1b8,1bc5ea60df2922e6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:56:30.812 INFO 21062 --- [ocp-schedules-6,bf832cf3667ac1b8,1bc5ea60df2922e6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:56:30.815 INFO 21062 --- [ocp-schedules-23,7488f816bf2a0261,86db13397feabce0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:56:30.815 INFO 21062 --- [ocp-schedules-23,7488f816bf2a0261,86db13397feabce0] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:56:30.843 INFO 21062 --- [ocp-schedules-17,de22f56c5a012b49,f67ea9d346b4c332] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:56:30.843 INFO 21062 --- [ocp-schedules-23,7488f816bf2a0261,86db13397feabce0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:56:30.843 INFO 21062 --- [ocp-schedules-23,7488f816bf2a0261,86db13397feabce0] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:56:30.996 INFO 21062 --- [ocp-schedules-21,79e17439d448ef53,5a42e4ea8477c264] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:56:30.996+08:00, clusters=[1] 2024-11-04 13:56:30.997 INFO 21062 --- [ocp-schedules-21,79e17439d448ef53,5a42e4ea8477c264] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:56:31.021 WARN 21062 --- [ocp-async-5,257d631ae4b0a60e,c5ce772730f8e7de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:31.066 INFO 21062 --- [ocp-schedules-10,0eecae3db2d32268,38e0c33c6756de98] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:56:31.066 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:56:31.066 INFO 21062 --- [ocp-schedules-10,0eecae3db2d32268,38e0c33c6756de98] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:56:31.066 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:56:31.067 INFO 21062 --- [ocp-async-2,c4e5664c0e8bd637,16f7d081fe598301] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:31.076 INFO 21062 --- [ocp-schedules-10,0eecae3db2d32268,38e0c33c6756de98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:56:31.077 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:56:31.077 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:56:31.078 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:56:31.078 WARN 21062 --- [ocp-schedules-10,0eecae3db2d32268,38e0c33c6756de98] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:56:31.078 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:56:31.078 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:56:31.079 INFO 21062 --- [ocp-schedules-10,0eecae3db2d32268,38e0c33c6756de98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:56:31.079 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:56:31.080 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:56:31.080 WARN 21062 --- [ocp-schedules-10,0eecae3db2d32268,38e0c33c6756de98] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:56:31.080 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:56:31.088 INFO 21062 --- [ocp-schedules-22,be34394ee9f9435f,70e71e24ef71bb25] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:56:31.088 ERROR 21062 --- [ocp-schedules-10,0eecae3db2d32268,38e0c33c6756de98] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:56:31.092 INFO 21062 --- [ocp-schedules-22,be34394ee9f9435f,70e71e24ef71bb25] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:31.148 INFO 21062 --- [ocp-schedules-3,9675c297030da46c,6b590282723ceeac] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:56:31.148+08:00, clusters=[1] 2024-11-04 13:56:31.151 INFO 21062 --- [ocp-schedules-3,9675c297030da46c,6b590282723ceeac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:56:31.153 ERROR 21062 --- [ocp-schedules-22,be34394ee9f9435f,70e71e24ef71bb25] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:56:31.153 INFO 21062 --- [ocp-schedules-22,be34394ee9f9435f,70e71e24ef71bb25] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:56:31.188 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:31.189 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:32.983 INFO 21062 --- [prometheus-minute-schedule-1,96af7f82c716a278,9e8a9af58c8aec41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:33.021 WARN 21062 --- [ocp-async-9,7c9510709866e7d4,71a6993aca39c49d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:33.067 INFO 21062 --- [ocp-async-8,0a1fb82c62688726,0606965fe98cd2e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:34.189 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:34.190 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:34.262 INFO 21062 --- [ocp-schedules-12,4ab11b6770090616,b8fb90ffe666e951] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:56:34.262 INFO 21062 --- [ocp-schedules-12,77bca1df3350e15a,1505c7793eef965a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:56:34.266 INFO 21062 --- [ocp-schedules-12,77bca1df3350e15a,1505c7793eef965a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:34.296 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:56:34.296 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:56:35.021 WARN 21062 --- [ocp-async-1,699b065e1b6f1221,6876f0167dd9f90f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:35.068 INFO 21062 --- [ocp-async-6,ab2a91104ec4572c,03a1e6cd0e422ee6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1abd4aa587ee62bd,a701aa45971fd6c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:36.265 INFO 21062 --- [ocp-schedules-21,5f340d41462145a0,e920e50b2e538b5d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:56:36.269 INFO 21062 --- [ocp-schedules-21,5f340d41462145a0,e920e50b2e538b5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:36.355 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:56:36.358 ERROR 21062 --- [ocp-schedules-21,5f340d41462145a0,e920e50b2e538b5d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:56:36.358 INFO 21062 --- [ocp-schedules-21,5f340d41462145a0,e920e50b2e538b5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:56:37.022 WARN 21062 --- [ocp-async-6,061683df203fdb34,1e94b58463f18f31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:37.068 INFO 21062 --- [ocp-async-7,31b97a966ebacb8e,ba3e3dd5f38da0e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:37.191 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:37.191 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:37.210 INFO 21062 --- [ocp-schedules-16,a96963dc7baf7429,71402ae6f62ebe78] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:56:38.504 INFO 21062 --- [ocp-schedules-26,8d53e6e415a5e6e8,e85275f9283e197a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:56:38.505 INFO 21062 --- [ocp-schedules-27,0a0f67b523559aa8,d4731d7dc298267d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:56:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:56:38.589734+08:00, lastProcessTime=2024-11-04T13:56:28.589300+08:00 2024-11-04 13:56:38.629 INFO 21062 --- [ocp-async-4,697faa7311f9273d,80bf8fad5d7e9819] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:56:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:56:38.589734+08:00 2024-11-04 13:56:38.983 INFO 21062 --- [prometheus-minute-schedule-1,49fa80184f8d7d68,2177e5f1ab22e657] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:39.022 WARN 21062 --- [ocp-async-9,4b7aae00d0e2fbb9,a9d57ce7cfa893f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:39.068 INFO 21062 --- [ocp-async-1,e625d030175cb32b,10579e76c2bb0980] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:40.002 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:56:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:56:40.003 INFO 21062 --- [ocp-schedules-10,c2f8f7fc0ebcf4b3,e753f8d2b6a46d8a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:56:40.191 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:40.192 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:41.022 WARN 21062 --- [ocp-async-1,4a7fb96dd5f710e5,3fbeda4cbcf94719] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:41.068 INFO 21062 --- [ocp-async-4,c7becbd312eb39b4,d8de86a79c66753a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:56:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:56:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 13:56:41.983 INFO 21062 --- [prometheus-minute-schedule-1,5ae1e0468356d2c7,79baa04cc6dfd8eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:43.022 WARN 21062 --- [ocp-async-4,769430fa99f6d0b3,047f7a3b4f46d991] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:43.068 INFO 21062 --- [ocp-async-2,85bb6b6ae7da1f0a,eee629b969f0d37b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:43.192 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:43.193 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:44.983 INFO 21062 --- [prometheus-minute-schedule-1,8b8a83098a295e99,0d298b26232a3aa8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:45.022 WARN 21062 --- [ocp-async-2,916b5844ea33ab85,a6a51d775b1cd195] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:45.068 INFO 21062 --- [ocp-async-8,e316ee08c14d68ac,5ffc050412811412] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:46.193 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:46.194 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:47.022 WARN 21062 --- [ocp-async-8,afde94d5814ee0be,396fd994cc844672] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:47.069 INFO 21062 --- [ocp-async-10,4234d187c2f628fa,99cba0174ba03247] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:47.238 INFO 21062 --- [ocp-schedules-11,50632d3a31d8349f,dba19c2f2deed2ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:56:47.983 INFO 21062 --- [prometheus-minute-schedule-1,ed3e098335ca891f,63193d1afd8ec269] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:56:48.590964+08:00, lastProcessTime=2024-11-04T13:56:38.589734+08:00 2024-11-04 13:56:48.629 INFO 21062 --- [ocp-async-3,8936d0c17b17a57b,9972fdc6ebe6c280] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:56:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:56:48.590964+08:00 2024-11-04 13:56:49.023 WARN 21062 --- [ocp-async-4,5cda4d65e7fb0955,7aa6419af4d3f1ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:49.069 INFO 21062 --- [ocp-async-2,8d442777e703d772,7b843ad6b6859ace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:49.194 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:49.196 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:50.006 INFO 21062 --- [ocp-schedules-30,2d5927ade4760d4e,076ff54ccc57776e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:56:50.815 INFO 21062 --- [ocp-schedules-6,662215713c553e44,6f2a7dd484cd983e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:56:50.821 INFO 21062 --- [ocp-schedules-6,662215713c553e44,6f2a7dd484cd983e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:56:50.821 INFO 21062 --- [ocp-schedules-6,662215713c553e44,6f2a7dd484cd983e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:56:50.983 INFO 21062 --- [prometheus-minute-schedule-1,862cd6a74ed85029,5b82918cb6fe9370] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:51.023 WARN 21062 --- [ocp-async-2,0a438c48328d1316,c19c1deed3bc4070] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:51.069 INFO 21062 --- [ocp-async-8,507c0b6fb758d9fb,f5d1f2f75b21ff1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:52.196 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:52.197 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:53.023 WARN 21062 --- [ocp-async-8,f146d09af0052d29,127fce332a417bc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:53.069 INFO 21062 --- [ocp-async-10,3d9a25943b48025c,d7a7f8e33980251d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1d8cb894d5d3a6db,3b36659de3ca0209] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:55.023 WARN 21062 --- [ocp-async-10,262981aadc548126,af8a1daab7844895] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:55.069 INFO 21062 --- [ocp-async-5,9e827cf8047ec290,010eb57f159c1787] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:55.197 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:55.198 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:56.983 INFO 21062 --- [prometheus-minute-schedule-1,590ee66df840b2f2,908276646607429f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:56:57.023 WARN 21062 --- [ocp-async-5,d55eb36130e235a8,8fe38bfbe5e05c8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:57.069 INFO 21062 --- [ocp-async-3,3f380b382c24dd22,7da853caae17eb0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:57.279 INFO 21062 --- [ocp-schedules-22,046fd668707d103d,23cc24bd5cc75e52] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:56:58.198 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:56:58.199 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:56:58.500 INFO 21062 --- [ocp-schedules-24,e9dd50913df5a67e,470b9dfd7b5198e0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:56:58.500 INFO 21062 --- [ocp-schedules-9,22367195549d62ca,7f68272e7c4f5a91] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:56:58.502 INFO 21062 --- [ocp-schedules-10,3424b48be77971f3,6d3ba5f24b02b458] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:56:58.504 INFO 21062 --- [ocp-schedules-9,22367195549d62ca,7f68272e7c4f5a91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:58.504 INFO 21062 --- [ocp-schedules-24,e9dd50913df5a67e,470b9dfd7b5198e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:56:58.504 INFO 21062 --- [ocp-schedules-22,b5b5a1bdba74fbc5,459a3c3e42838601] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:56:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:56:58.510 INFO 21062 --- [ocp-schedules-17,ce752ec33bf32dc7,4d5525b762e6d31c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:56:58.510 INFO 21062 --- [ocp-schedules-7,4f00907c6386c5d3,482acfef2d77988f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:56:58.510 INFO 21062 --- [ocp-schedules-7,4f00907c6386c5d3,482acfef2d77988f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:56:58.510 INFO 21062 --- [ocp-schedules-17,ce752ec33bf32dc7,4d5525b762e6d31c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:56:58.510 INFO 21062 --- [ocp-schedules-25,b85b59777887409b,fdd96ef2ad70f530] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:56:58.510 INFO 21062 --- [ocp-schedules-25,b85b59777887409b,fdd96ef2ad70f530] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:56:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:56:58.589293+08:00, lastProcessTime=2024-11-04T13:56:48.590964+08:00 2024-11-04 13:56:58.629 INFO 21062 --- [ocp-async-7,07ac8d79d1d6ede9,c7a1957fb2de3c6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:56:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:56:58.589293+08:00 2024-11-04 13:56:59.015 INFO 21062 --- [ocp-schedules-27,b4129e96867ca21f,075063329415dda2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:56:59.023 WARN 21062 --- [ocp-async-8,6a4f4ee0c515cd4e,3db9ab7903cc7cbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:56:59.070 INFO 21062 --- [ocp-async-10,33d9a12a14be5ba5,39d6738f8d8f8d81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:56:59.983 INFO 21062 --- [prometheus-minute-schedule-1,19e2ba27273b0e04,39b7bb12a967f694] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699820, interval=60 2024-11-04 13:57:00.009 INFO 21062 --- [ocp-schedules-30,1b9e241e4be6e825,19d68736abb9972e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:57:01.024 WARN 21062 --- [ocp-async-10,1acc40e77c6546ee,31bcc43a8f10e968] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:01.070 INFO 21062 --- [ocp-async-5,965ea2daca3e7fc2,cec426a2da62cfe2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:01.199 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:01.201 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:02.983 INFO 21062 --- [prometheus-minute-schedule-1,0686223d451393b0,f7415b070c393b5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:03.024 WARN 21062 --- [ocp-async-5,f2aa3c517ab33fdc,ff826a60ff111196] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:03.070 INFO 21062 --- [ocp-async-9,76aa6669e429daa2,f1ae03573ee2196d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:04.201 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:04.202 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:04.296 INFO 21062 --- [ocp-schedules-12,77bca1df3350e15a,1505c7793eef965a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:57:04.296 INFO 21062 --- [ocp-schedules-12,ebc5975c09f61bf3,811f5f917c2f9226] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:57:04.301 INFO 21062 --- [ocp-schedules-12,ebc5975c09f61bf3,811f5f917c2f9226] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:04.329 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:57:04.329 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:57:05.024 WARN 21062 --- [ocp-async-9,6157325a75bf084a,c217f58c595de9f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:05.070 INFO 21062 --- [ocp-async-1,33aa807b8c435dd6,557516e6cd314893] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:05.983 INFO 21062 --- [prometheus-minute-schedule-1,50f1f48d34540088,82c285aca32edfdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:07.024 WARN 21062 --- [ocp-async-1,79247fa83d34f772,0a10ec5c98980a5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:07.071 INFO 21062 --- [ocp-async-4,9c2002323ff4a860,f409c0a290eaaa01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:07.202 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:07.203 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:07.303 INFO 21062 --- [ocp-schedules-21,f8c09b5285da4520,572a2f6b911969ce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:57:08.505 INFO 21062 --- [ocp-schedules-9,22367195549d62ca,7f68272e7c4f5a91] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:57:08.505 INFO 21062 --- [ocp-schedules-24,e9dd50913df5a67e,470b9dfd7b5198e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:57:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:57:08.591317+08:00, lastProcessTime=2024-11-04T13:56:58.589293+08:00 2024-11-04 13:57:08.630 INFO 21062 --- [ocp-async-6,53d49fac48662c1e,a089ad3239dc2e04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:57:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:57:08.591317+08:00 2024-11-04 13:57:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a1aab4491fec90a4,80a133f1de52fbf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:09.024 WARN 21062 --- [ocp-async-5,ac033c2dbf38065d,05c5c9913a26d2b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:09.071 INFO 21062 --- [ocp-async-3,34cb1fffbadfb88c,1dd31f9bf4b1f739] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:57:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:57:10.011 INFO 21062 --- [ocp-schedules-3,bf3aa39a6f0fb1e3,c9c48e077b09dac2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:57:10.203 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:10.205 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:57:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:57:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:57:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:57:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:57:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:57:10.823 INFO 21062 --- [ocp-schedules-15,7922f086bf217c0b,62a1ec88faadc09f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:57:10.829 INFO 21062 --- [ocp-schedules-15,7922f086bf217c0b,62a1ec88faadc09f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:57:10.829 INFO 21062 --- [ocp-schedules-15,7922f086bf217c0b,62a1ec88faadc09f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:57:11.025 WARN 21062 --- [ocp-async-3,55cfbe6a020042a5,7d65580c3157d80f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:11.071 INFO 21062 --- [ocp-async-6,50c0f2b87f646640,8d4283bb474286eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:11.405 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=3, affectRows=3 2024-11-04 13:57:11.405 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=3, validCount=3 2024-11-04 13:57:11.406 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=9, affectRows=9 2024-11-04 13:57:11.406 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=2, affectRows=2 2024-11-04 13:57:11.406 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=2, validCount=2 2024-11-04 13:57:11.406 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=9, validCount=9 2024-11-04 13:57:11.406 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=24, affectRows=24 2024-11-04 13:57:11.406 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=24, validCount=24 2024-11-04 13:57:11.406 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=25, affectRows=25 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=42, affectRows=42 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=25, validCount=25 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=42, validCount=42 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=38, affectRows=38 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=26, affectRows=26 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=26, validCount=26 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=38, validCount=38 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=60, affectRows=60 2024-11-04 13:57:11.407 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=74, affectRows=74 2024-11-04 13:57:11.408 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=59, validCount=59 2024-11-04 13:57:11.408 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=74, validCount=74 2024-11-04 13:57:11.408 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699760, nodeCount=89, affectRows=89 2024-11-04 13:57:11.408 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699760, nodeCount=87, validCount=87 2024-11-04 13:57:11.983 INFO 21062 --- [prometheus-minute-schedule-1,dacf8b3c22d760c5,d5b22068f9b453ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:13.025 WARN 21062 --- [ocp-async-1,3f45f29b9e0be1c1,63f828dbd914059b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:13.071 INFO 21062 --- [ocp-async-7,8126faeaffe17d66,12747193bcfd34e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:13.205 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:13.206 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:13.985 INFO 21062 --- [prometheus-cache-monitor1,931a3467cd7e720d,3ead2975de4837d0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:57:13.985 INFO 21062 --- [prometheus-cache-monitor1,931a3467cd7e720d,3ead2975de4837d0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:57:14.810 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:53:14.784, end=2024-11-04T13:55:14.807} 2024-11-04 13:57:14.814 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:57:14.814 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:57:14.983 INFO 21062 --- [prometheus-minute-schedule-1,21e6f0f3da574da4,a659e380f8ffa005] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:15.025 WARN 21062 --- [ocp-async-4,f13438bc074d2508,f55d4f9302ed501f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:15.071 INFO 21062 --- [ocp-async-2,6641b5ea6d52951b,c6ac940cf9f0d03a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:16.206 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:16.209 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:17.025 WARN 21062 --- [ocp-async-9,cbf9fee5cd596425,ef94438e615c3154] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:17.072 INFO 21062 --- [ocp-async-8,48a23d3a75ad55fa,39c4772c9c37fec7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:17.344 INFO 21062 --- [ocp-schedules-15,2274416099a44582,1910cfad32d03898] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:57:17.983 INFO 21062 --- [prometheus-minute-schedule-1,eabf7a8ab00cfb47,be22354c862d3e71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:57:18.589942+08:00, lastProcessTime=2024-11-04T13:57:08.591317+08:00 2024-11-04 13:57:18.630 INFO 21062 --- [ocp-async-2,a7b7176b7ac9f3b4,8d7abea6c471d8b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:57:18.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:57:18.589942+08:00 2024-11-04 13:57:19.025 WARN 21062 --- [ocp-async-1,f51886222aa474ac,125680eff5291215] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:19.072 INFO 21062 --- [ocp-async-7,b02dca17d9777416,42b83694f244435d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:19.209 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:19.210 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:20.014 INFO 21062 --- [ocp-schedules-26,20a12ec3a1491cb9,13ffd848299e3076] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:57:20.983 INFO 21062 --- [prometheus-minute-schedule-1,93e806a527267c5a,1e585efdecd4d094] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:21.025 WARN 21062 --- [ocp-async-4,a43e7c1ae3d35606,d3d8cb41c4ff73c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:21.072 INFO 21062 --- [ocp-async-2,2a6d77dc84efe1ad,26579c3dbb378726] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:22.210 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:22.211 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:23.025 WARN 21062 --- [ocp-async-9,f6c1c0bcb7692f14,5cb6dee1c608c0e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:23.072 INFO 21062 --- [ocp-async-8,a92d49b68858dcd2,d764a472dc631f8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:23.983 INFO 21062 --- [prometheus-minute-schedule-1,1ed567c25161fbbf,3132ceb32abcdbb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:25.025 WARN 21062 --- [ocp-async-6,b276d5bcb53827a3,378116f139f355a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:25.072 INFO 21062 --- [ocp-async-10,1a7e8c052f7863ae,9f7820e39732900a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:25.211 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:25.212 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:25.303 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 13:57:25.303 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 13:57:25.311 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:57:25.311 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:57:25.311 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:57:25.311 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:57:25.317 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 13:57:25.317 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 13:57:25.317 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 13:57:25.317 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 13:57:25.354 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:57:25.363 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:57:25.363 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:57:25.363 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:57:25.363 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:57:25.363 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:57:25.371 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:57:25.932 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 13:57:26.487 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.169610263S 2024-11-04 13:57:26.487 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 13:57:26.571 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 13:57:26.571 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 13:57:26.601 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 13:57:26.649 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 13:57:26.649 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 13:57:26.649 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 13:57:26.649 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 13:57:26.649 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 13:57:26.649 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 13:57:26.657 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 13:57:26.784 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] o.s.boot.SpringApplication : Started application in 1.473 seconds (JVM running for 5260.463) 2024-11-04 13:57:26.784 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.473335073S 2024-11-04 13:57:26.983 INFO 21062 --- [prometheus-minute-schedule-1,322b48d08c216fcb,475bc8d9dd85795c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4a52f5971211b102,ec1f35ee41853489] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:57:27.026 WARN 21062 --- [ocp-async-7,12bf1d5edc7bb95a,7bb9d07f87a232ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:27.073 INFO 21062 --- [ocp-async-5,996795c365427a33,19ac268f70b54f5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:27.151 INFO 21062 --- [ocp-schedules-14,f3347f7d770b42e7,99f3a4a3c6b24558] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 13:57:27.367 INFO 21062 --- [ocp-schedules-29,82bd016569bb0eb8,9b206ee3749ad594] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:57:28.212 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:28.214 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:28.498 INFO 21062 --- [ocp-schedules-25,68d2cae9980f1678,e77eed18f483217e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:57:28.499 INFO 21062 --- [ocp-schedules-3,e52334b88ea4f999,1433aba3cfc93de3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:57:28.499 INFO 21062 --- [ocp-schedules-2,2c977bf85d03879d,48acc657dfa737f4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:57:28.500 INFO 21062 --- [ocp-schedules-4,0bc6c84944290483,c5e2084dff88e24f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:57:28.502 INFO 21062 --- [ocp-schedules-18,e3110747498b5796,fa617a58e8743800] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:57:28.503 INFO 21062 --- [ocp-schedules-25,68d2cae9980f1678,e77eed18f483217e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:57:28.503 INFO 21062 --- [ocp-schedules-4,0bc6c84944290483,c5e2084dff88e24f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:28.503 INFO 21062 --- [ocp-schedules-2,2c977bf85d03879d,48acc657dfa737f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:57:28.504 INFO 21062 --- [ocp-schedules-29,b88e62405073bb8d,2a35b67d20ca5244] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 13:57:28.504 INFO 21062 --- [ocp-schedules-3,e52334b88ea4f999,1433aba3cfc93de3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:57:28.505 INFO 21062 --- [ocp-schedules-27,a1d81d93794c8c34,c02f4ceb1bf256c4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:57:28.507 INFO 21062 --- [ocp-schedules-29,b88e62405073bb8d,2a35b67d20ca5244] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:28.510 INFO 21062 --- [ocp-schedules-14,7d8522c5fe26d897,940b69693af3a41f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:57:28.510 INFO 21062 --- [ocp-schedules-11,8ef430d4cf6b5b5b,d2452455e785b9bd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:57:28.510 INFO 21062 --- [ocp-schedules-14,7d8522c5fe26d897,940b69693af3a41f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:57:28.510 INFO 21062 --- [ocp-schedules-11,8ef430d4cf6b5b5b,d2452455e785b9bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:57:28.510 INFO 21062 --- [ocp-schedules-28,6dfc735eb02e4dde,48216bb079285fa8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:57:28.510 INFO 21062 --- [ocp-schedules-28,6dfc735eb02e4dde,48216bb079285fa8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:57:28.534 INFO 21062 --- [ocp-schedules-20,516c9c2818cf5499,6c341b413caa032f] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 13:57:28.534 INFO 21062 --- [ocp-schedules-20,516c9c2818cf5499,6c341b413caa032f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 13:57:28.576 INFO 21062 --- [pool-12-thread-8,ac4ab3d6fb10f0ff,009d0a9687423756] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 13:57:28.576 INFO 21062 --- [pool-12-thread-5,ac4ab3d6fb10f0ff,d7e01a95fd30115c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 13:57:28.577 INFO 21062 --- [pool-12-thread-6,ac4ab3d6fb10f0ff,446312569683b10f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 13:57:28.577 INFO 21062 --- [pool-12-thread-7,ac4ab3d6fb10f0ff,75cb6fd8ed78b1ab] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 13:57:28.581 INFO 21062 --- [ForkJoinPool-11-worker-1,ac4ab3d6fb10f0ff,095a5e5631fc7232] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 13:57:28.581 INFO 21062 --- [ForkJoinPool-11-worker-1,ac4ab3d6fb10f0ff,095a5e5631fc7232] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 13:57:28.581 INFO 21062 --- [ForkJoinPool-11-worker-1,ac4ab3d6fb10f0ff,095a5e5631fc7232] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 13:57:28.581 INFO 21062 --- [pool-12-thread-5,ac4ab3d6fb10f0ff,d7e01a95fd30115c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 13:57:28.581 INFO 21062 --- [pool-12-thread-7,ac4ab3d6fb10f0ff,75cb6fd8ed78b1ab] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 13:57:28.582 INFO 21062 --- [pool-12-thread-7,ac4ab3d6fb10f0ff,75cb6fd8ed78b1ab] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 13:57:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:57:28.589079+08:00, lastProcessTime=2024-11-04T13:57:18.589942+08:00 2024-11-04 13:57:28.599 INFO 21062 --- [pool-12-thread-6,ac4ab3d6fb10f0ff,446312569683b10f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 13:57:28.603 INFO 21062 --- [pool-12-thread-6,ac4ab3d6fb10f0ff,446312569683b10f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 13:57:28.603 INFO 21062 --- [ForkJoinPool-11-worker-1,ac4ab3d6fb10f0ff,2c5fc0c0cea14ed6] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 13:57:28.603 INFO 21062 --- [pool-12-thread-8,ac4ab3d6fb10f0ff,009d0a9687423756] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 13:57:28.607 INFO 21062 --- [pool-12-thread-6,ac4ab3d6fb10f0ff,446312569683b10f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 13:57:28.614 INFO 21062 --- [ocp-schedules-13,ec8b2d53037fb4ea,425d75989dafaf58] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:57:28.615 INFO 21062 --- [ocp-schedules-13,ec8b2d53037fb4ea,425d75989dafaf58] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 13:57:28.630 INFO 21062 --- [ocp-async-4,758ee89caf6eba2d,acd302a606e17563] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:57:28.749 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:57:28.589079+08:00 2024-11-04 13:57:28.765 INFO 21062 --- [ocp-schedules-18,15abc2c652bb16fc,c9f8dda98faf2d0c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:57:28.781 INFO 21062 --- [ocp-schedules-27,ade429b8311529db,26a7614e08f5682f] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 13:57:28.794 INFO 21062 --- [ocp-schedules-27,ade429b8311529db,26a7614e08f5682f] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 13:57:28.905 INFO 21062 --- [ocp-schedules-11,060d5c6673a57593,1e83883e5d8ee104] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:57:28.908 INFO 21062 --- [ocp-schedules-11,060d5c6673a57593,1e83883e5d8ee104] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:28.908 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:57:28.973 INFO 21062 --- [ocp-schedules-25,1a838480cfe8a0e9,2fb5f97144d091bf] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:57:28.973 INFO 21062 --- [ocp-schedules-25,1a838480cfe8a0e9,2fb5f97144d091bf] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:57:29.017 INFO 21062 --- [ocp-schedules-15,4796d8df9355d3a2,ca2a59239db79238] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:57:29.036 WARN 21062 --- [ocp-async-8,7eeb52b76b5d1eb3,38a342038acf5a9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:29.052 INFO 21062 --- [ocp-schedules-10,38ffeab7163bda2f,e31fe908a2dd32f9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:57:29.057 INFO 21062 --- [ocp-schedules-10,38ffeab7163bda2f,e31fe908a2dd32f9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:57:29.057 INFO 21062 --- [ocp-schedules-10,38ffeab7163bda2f,e31fe908a2dd32f9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:57:29.073 INFO 21062 --- [ocp-async-6,9299923be8ebe7ce,5383ad9ca6df6301] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:29.322 INFO 21062 --- [ocp-schedules-16,5b7c2da15269f7be,d0a5a229f529004c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:57:29.322 INFO 21062 --- [ocp-schedules-16,5b7c2da15269f7be,d0a5a229f529004c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:57:29.327 INFO 21062 --- [ocp-schedules-16,5b7c2da15269f7be,d0a5a229f529004c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:57:29.327 INFO 21062 --- [ocp-schedules-16,5b7c2da15269f7be,d0a5a229f529004c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:57:29.330 INFO 21062 --- [ocp-schedules-16,5b7c2da15269f7be,d0a5a229f529004c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:57:29.330 INFO 21062 --- [ocp-schedules-16,5b7c2da15269f7be,d0a5a229f529004c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:57:29.466 INFO 21062 --- [ocp-schedules-6,85c9d785f60fc91c,2bf96a41397f857d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 13:57:29.466 INFO 21062 --- [ocp-schedules-6,85c9d785f60fc91c,2bf96a41397f857d] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 13:57:29.494 INFO 21062 --- [pool-36-thread-1,85c9d785f60fc91c,4c1c645f0f8f5945] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 13:57:29.520 WARN 21062 --- [ocp-schedules-6,85c9d785f60fc91c,2bf96a41397f857d] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 13:57:29.520 INFO 21062 --- [ocp-schedules-6,85c9d785f60fc91c,2bf96a41397f857d] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 13:57:29.983 INFO 21062 --- [prometheus-minute-schedule-1,a1b3a9701e002e95,065fa06f151476fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:30.018 INFO 21062 --- [ocp-schedules-14,b7e7c71e147ad755,66579481fd842a1c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:57:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:57:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:57:30.831 INFO 21062 --- [ocp-schedules-19,9485c8e80edf3676,ef14cb092e259334] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:57:30.837 INFO 21062 --- [ocp-schedules-19,9485c8e80edf3676,ef14cb092e259334] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:57:30.837 INFO 21062 --- [ocp-schedules-19,9485c8e80edf3676,ef14cb092e259334] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:57:30.843 INFO 21062 --- [ocp-schedules-27,c251ff83e537b219,2ead1cd627233062] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:57:30.843 INFO 21062 --- [ocp-schedules-27,c251ff83e537b219,2ead1cd627233062] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:57:30.869 INFO 21062 --- [ocp-schedules-27,c251ff83e537b219,2ead1cd627233062] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:57:30.869 INFO 21062 --- [ocp-schedules-27,c251ff83e537b219,2ead1cd627233062] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:57:30.870 INFO 21062 --- [ocp-schedules-5,a7179d9266940b89,fcf92e6b802a7722] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:57:31.023 INFO 21062 --- [ocp-schedules-26,1e19b7cccd50480f,9e9ed49133af8e6d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:57:31.023+08:00, clusters=[1] 2024-11-04 13:57:31.024 INFO 21062 --- [ocp-schedules-26,1e19b7cccd50480f,9e9ed49133af8e6d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:57:31.036 WARN 21062 --- [ocp-async-7,ceebd10041c7486a,34fbf759f5d92ac1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:31.073 INFO 21062 --- [ocp-async-4,d428678b8a4577b0,a39a6ff06b5d8898] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:31.097 INFO 21062 --- [ocp-schedules-14,c8903a347982168e,8207fa6fae1e0d48] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:57:31.097 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:57:31.097 INFO 21062 --- [ocp-schedules-14,c8903a347982168e,8207fa6fae1e0d48] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:57:31.097 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:57:31.107 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:57:31.107 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:57:31.107 INFO 21062 --- [ocp-schedules-14,c8903a347982168e,8207fa6fae1e0d48] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:57:31.108 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:57:31.108 WARN 21062 --- [ocp-schedules-14,c8903a347982168e,8207fa6fae1e0d48] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:57:31.108 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:57:31.109 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:57:31.109 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:57:31.109 INFO 21062 --- [ocp-schedules-14,c8903a347982168e,8207fa6fae1e0d48] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:57:31.110 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:57:31.110 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:57:31.110 WARN 21062 --- [ocp-schedules-14,c8903a347982168e,8207fa6fae1e0d48] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:57:31.117 ERROR 21062 --- [ocp-schedules-14,c8903a347982168e,8207fa6fae1e0d48] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 13:57:31.178 INFO 21062 --- [ocp-schedules-10,e7587a17c7580081,177221470063f3e0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:57:31.178+08:00, clusters=[1] 2024-11-04 13:57:31.181 INFO 21062 --- [ocp-schedules-10,e7587a17c7580081,177221470063f3e0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:57:31.214 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:31.215 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:32.983 INFO 21062 --- [prometheus-minute-schedule-1,00a09d3936467df6,60c3af915c5fde95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:33.036 WARN 21062 --- [ocp-async-2,76b7fad790df1a33,e426aa11ead0cc8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:33.073 INFO 21062 --- [ocp-async-9,e4ded17aff70386c,d5dfdc5e05e86701] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:34.216 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:34.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:34.329 INFO 21062 --- [ocp-schedules-12,ebc5975c09f61bf3,811f5f917c2f9226] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:57:34.329 INFO 21062 --- [ocp-schedules-12,8e8791298386d373,5e0ca09b7bcc5022] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:57:34.333 INFO 21062 --- [ocp-schedules-12,8e8791298386d373,5e0ca09b7bcc5022] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:34.362 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:57:34.362 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:57:35.036 WARN 21062 --- [ocp-async-3,8e5fdfd914317f78,05b679b51c9453ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:35.073 INFO 21062 --- [ocp-async-1,3bae60cb7a3d325d,3c4edbdbbac106fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:35.983 INFO 21062 --- [prometheus-minute-schedule-1,f87eaa24604666af,801343066cb18ed1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:36.358 INFO 21062 --- [ocp-schedules-9,fbce242b83965588,c00f12fc2ba756df] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:57:36.364 INFO 21062 --- [ocp-schedules-9,fbce242b83965588,c00f12fc2ba756df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:36.449 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:57:36.451 ERROR 21062 --- [ocp-schedules-9,fbce242b83965588,c00f12fc2ba756df] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:57:36.451 INFO 21062 --- [ocp-schedules-9,fbce242b83965588,c00f12fc2ba756df] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:57:37.037 WARN 21062 --- [ocp-async-6,33322ec3e418e5a6,3b027c39c3b14eb8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:37.073 INFO 21062 --- [ocp-async-10,f2cdaf22e4aacb52,4911b62aa67e9da1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:37.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:37.218 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:37.409 INFO 21062 --- [ocp-schedules-8,f6fd8f2a088d5e26,1a978618a789f0ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:57:38.504 INFO 21062 --- [ocp-schedules-4,0bc6c84944290483,c5e2084dff88e24f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:57:38.504 INFO 21062 --- [ocp-schedules-2,2c977bf85d03879d,48acc657dfa737f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:57:38.507 INFO 21062 --- [ocp-schedules-29,b88e62405073bb8d,2a35b67d20ca5244] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 13:57:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:57:38.590130+08:00, lastProcessTime=2024-11-04T13:57:28.589079+08:00 2024-11-04 13:57:38.630 INFO 21062 --- [ocp-async-1,c9f46b5963ea2475,f8e87c06ad10e04a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:57:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:57:38.590130+08:00 2024-11-04 13:57:38.908 INFO 21062 --- [ocp-schedules-11,060d5c6673a57593,1e83883e5d8ee104] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:57:38.983 INFO 21062 --- [prometheus-minute-schedule-1,6209152b0903c8d0,f4804a7f551d6584] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:39.037 WARN 21062 --- [ocp-async-9,fa2da5cef255e0f6,8fe57c2e8b107ea4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:39.074 INFO 21062 --- [ocp-async-8,c2274380c8b5bb67,b5482c94a3a2d0c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:57:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:57:40.021 INFO 21062 --- [ocp-schedules-24,ba22980236902206,6349e30e3eabaacc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:57:40.218 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:40.219 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:41.037 WARN 21062 --- [ocp-async-3,8b71635015559d78,40c86d8a0816be8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:41.074 INFO 21062 --- [ocp-async-1,6785457ac4326c54,4a4547dcc0e1ba35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:41.983 INFO 21062 --- [prometheus-minute-schedule-1,e77c48c253240044,083800ec6ec74aff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:43.037 WARN 21062 --- [ocp-async-1,fdb586f0c7414a09,f7df195665652179] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:43.074 INFO 21062 --- [ocp-async-4,53da7521d39d6c60,c83bb0bc1272ccce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:43.219 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:43.220 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5e01bcdb3f9d14e6,d0ca88c4bcf96d25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:45.037 WARN 21062 --- [ocp-async-2,3c9226150b0cfdee,44d40b529f15a493] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:45.074 INFO 21062 --- [ocp-async-5,05879cac96106861,5c50e3aa352f52a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:46.221 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:46.221 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:47.038 WARN 21062 --- [ocp-async-5,3e8acb55170b9c4a,97c36939b7697e8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:47.074 INFO 21062 --- [ocp-async-8,9053cf8b9333e3b7,77b9469e0b72eaf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:47.434 INFO 21062 --- [ocp-schedules-5,7484ef099c5ff370,cef5c70a55ac5700] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:57:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e142a6de458ef57a,cf3fc60e261ed73c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:57:48.591380+08:00, lastProcessTime=2024-11-04T13:57:38.590130+08:00 2024-11-04 13:57:48.630 INFO 21062 --- [ocp-async-9,fb82b842cfab16ba,8fb90aed41944fcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:57:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:57:48.591380+08:00 2024-11-04 13:57:49.038 WARN 21062 --- [ocp-async-1,4c8816c6ce82d0b0,fe06c7aeea253094] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:49.074 INFO 21062 --- [ocp-async-2,9b8c5b9c0dab807e,dc8c0749a406ccdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:49.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:49.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:50.024 INFO 21062 --- [ocp-schedules-10,67703662a5cce3d4,17887255c9918a46] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:57:50.840 INFO 21062 --- [ocp-schedules-26,2c28b27809f560be,fd1ac96f7dea1763] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:57:50.847 INFO 21062 --- [ocp-schedules-26,2c28b27809f560be,fd1ac96f7dea1763] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:57:50.847 INFO 21062 --- [ocp-schedules-26,2c28b27809f560be,fd1ac96f7dea1763] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:57:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a5888277204fe498,254994b5baf4787f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:51.038 WARN 21062 --- [ocp-async-2,0682b20585c754ef,23515c5f31ea822e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:51.075 INFO 21062 --- [ocp-async-5,c56b6bbcc5cbe50d,e4d445ce749a3992] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:52.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:52.223 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:53.038 WARN 21062 --- [ocp-async-5,c3c2a6d28dd31510,cba1bdc2011f0b6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:53.075 INFO 21062 --- [ocp-async-8,1dc1acb43a8b3a76,919622f16d8691f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6ec9bfe2885bc966,544eb548c275b75e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:55.038 WARN 21062 --- [ocp-async-10,fbb1973eeae4d0d0,9794013f7a28938b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:55.075 INFO 21062 --- [ocp-async-7,a6e70d704ab110d1,feeed96a02a1f2bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:55.223 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:55.224 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:56.983 INFO 21062 --- [prometheus-minute-schedule-1,07ef19ce80582fc2,07186667b8f99856] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:57:57.038 WARN 21062 --- [ocp-async-2,b286be620a2caf3b,9851d2d12a12564d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:57.075 INFO 21062 --- [ocp-async-9,5e59760afdf94d8b,f32ee0f675c4402b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:57.475 INFO 21062 --- [ocp-schedules-19,3d4eab8d6123733e,659e96aa5aa4afaa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:57:58.224 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:57:58.225 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:57:58.500 INFO 21062 --- [ocp-schedules-16,b746fb11aabe5dba,973f6f9130aba884] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:57:58.500 INFO 21062 --- [ocp-schedules-29,2585202f19969587,430417ea339a98d9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:57:58.502 INFO 21062 --- [ocp-schedules-4,5757a583e4971f5c,ff38bc2000baf32b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:57:58.504 INFO 21062 --- [ocp-schedules-16,b746fb11aabe5dba,973f6f9130aba884] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:57:58.504 INFO 21062 --- [ocp-schedules-29,2585202f19969587,430417ea339a98d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:57:58.505 INFO 21062 --- [ocp-schedules-28,769eb29c7ad1121c,6a17be5336306c46] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:57:58.510 INFO 21062 --- [ocp-schedules-17,36adbd70953dffe5,2b5824f778e9b17d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:57:58.510 INFO 21062 --- [ocp-schedules-17,36adbd70953dffe5,2b5824f778e9b17d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:57:58.510 INFO 21062 --- [ocp-schedules-11,d28e819c7454153c,c2cdb6b1de961462] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:57:58.510 INFO 21062 --- [ocp-schedules-11,d28e819c7454153c,c2cdb6b1de961462] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:57:58.511 INFO 21062 --- [ocp-schedules-2,0f8bad06cba89ba3,29ea1c9cf7570f6c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:57:58.511 INFO 21062 --- [ocp-schedules-2,0f8bad06cba89ba3,29ea1c9cf7570f6c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:57:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:57:58.589943+08:00, lastProcessTime=2024-11-04T13:57:48.591380+08:00 2024-11-04 13:57:58.630 INFO 21062 --- [ocp-async-1,9c4fb1517750dc42,09344544a391e65d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:57:58.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:57:58.589943+08:00 2024-11-04 13:57:59.020 INFO 21062 --- [ocp-schedules-23,c2d3ca3f55fc5144,9aec972fe30cf28d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:57:59.039 WARN 21062 --- [ocp-async-5,0bd8cc7e503221aa,e9dd00eba41e3740] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:57:59.075 INFO 21062 --- [ocp-async-10,b5499c8fb558f439,181140621facbf1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:57:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a24f5fb6ba7f8a2c,d0ebc3df2f6b8afa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699880, interval=60 2024-11-04 13:58:00.027 INFO 21062 --- [ocp-schedules-8,9fbc5cadf035a58a,1e7174b1349aa489] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:58:01.039 WARN 21062 --- [ocp-async-10,8796bcbc54553d34,04121a05143413d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:01.075 INFO 21062 --- [ocp-async-2,9eee6bb667219fe5,addb61054be99239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:01.225 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:01.227 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:02.983 INFO 21062 --- [prometheus-minute-schedule-1,b963d62f178d6686,2d11263074b8b543] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:03.039 WARN 21062 --- [ocp-async-2,cd4591ac3d55ae5f,ac6302c4ff945449] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:03.076 INFO 21062 --- [ocp-async-9,cb930bb93a286a34,1e651a0886e3d010] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:04.227 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:04.228 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:04.363 INFO 21062 --- [ocp-schedules-12,8e8791298386d373,5e0ca09b7bcc5022] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:58:04.363 INFO 21062 --- [ocp-schedules-12,c0d15b1cbdcc82fb,5b39bf1edb6f70e1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:58:04.367 INFO 21062 --- [ocp-schedules-12,c0d15b1cbdcc82fb,5b39bf1edb6f70e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:04.397 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:58:04.397 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:58:05.039 WARN 21062 --- [ocp-async-9,e8e2d2d19f0b25af,44aea92a98240cf0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:05.076 INFO 21062 --- [ocp-async-3,beb2bc2a0b6f609a,38ad4bb072d64089] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:05.983 INFO 21062 --- [prometheus-minute-schedule-1,dfd29a82e30e860f,5f265d7bafc4bad7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:07.039 WARN 21062 --- [ocp-async-10,fdd556a23e201fb5,78135e7f7b1cdc0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:07.076 INFO 21062 --- [ocp-async-8,ad7967bfc6dda099,509f29c9f1f3963e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:07.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:07.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:07.503 INFO 21062 --- [ocp-schedules-9,feafd444e40b89c1,95462aba32765281] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:58:08.505 INFO 21062 --- [ocp-schedules-16,b746fb11aabe5dba,973f6f9130aba884] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:58:08.505 INFO 21062 --- [ocp-schedules-29,2585202f19969587,430417ea339a98d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:58:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:58:08.590242+08:00, lastProcessTime=2024-11-04T13:57:58.589943+08:00 2024-11-04 13:58:08.631 INFO 21062 --- [ocp-async-6,29c6a45f55577bf9,8a59166ad9388dac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:58:08.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:58:08.590242+08:00 2024-11-04 13:58:08.983 INFO 21062 --- [prometheus-minute-schedule-1,e51476ef1aac4efd,86fe61d0ef78f30d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:09.039 WARN 21062 --- [ocp-async-7,ef8fb4102f8677d6,3533a7d7a4b18176] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:09.076 INFO 21062 --- [ocp-async-4,557102e16e68b47b,6aa983897b6952d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:10.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:58:10.004 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:58:10.029 INFO 21062 --- [ocp-schedules-30,8da25ef77beb9472,5a5cd6f8751289fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:58:10.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:10.230 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:58:10.307 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:58:10.307 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:58:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:58:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:58:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:58:10.849 INFO 21062 --- [ocp-schedules-6,c8f9558b46a3c748,7a5b169054500238] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:58:10.855 INFO 21062 --- [ocp-schedules-6,c8f9558b46a3c748,7a5b169054500238] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:58:10.855 INFO 21062 --- [ocp-schedules-6,c8f9558b46a3c748,7a5b169054500238] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:58:11.040 WARN 21062 --- [ocp-async-5,e6e69cbe9b36e9de,9da43058c59ef10c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:11.076 INFO 21062 --- [ocp-async-6,5f846314f169eee8,15edfa2807cc605b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=13, affectRows=13 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=7, affectRows=7 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=8, affectRows=8 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=7, validCount=7 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=13, validCount=13 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=8, validCount=8 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=6, affectRows=6 2024-11-04 13:58:11.414 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=5, validCount=5 2024-11-04 13:58:11.416 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=25, affectRows=25 2024-11-04 13:58:11.416 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=25, validCount=25 2024-11-04 13:58:11.416 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=22, affectRows=22 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=62, affectRows=62 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=43, affectRows=43 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=54, affectRows=54 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=49, affectRows=49 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=22, validCount=22 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=38, affectRows=38 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=43, validCount=43 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=49, validCount=49 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=54, validCount=54 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=60, validCount=60 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699820, nodeCount=65, affectRows=65 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=38, validCount=38 2024-11-04 13:58:11.417 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699820, nodeCount=65, validCount=65 2024-11-04 13:58:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8c74a2f310cc7f77,f073790cae1569ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:13.040 WARN 21062 --- [ocp-async-3,a7b9fb469ca6010c,5075e43679ee7346] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:13.076 INFO 21062 --- [ocp-async-1,5c45af71689e6bfe,6574dd25972ccf0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:13.230 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:13.231 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:13.985 INFO 21062 --- [prometheus-cache-monitor1,d00aeaa61c9b4480,023f2a064a17969c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:58:13.985 INFO 21062 --- [prometheus-cache-monitor1,d00aeaa61c9b4480,023f2a064a17969c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:58:14.156 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:52:14.134, end=2024-11-04T13:57:14.153} 2024-11-04 13:58:14.157 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:52:14.134, end=2024-11-04T13:57:14.154} 2024-11-04 13:58:14.165 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 13:58:14.165 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 13:58:14.165 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:58:14.165 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 13:58:14.983 INFO 21062 --- [prometheus-minute-schedule-1,0822d2b1f3f4adc6,419bb485cee72c08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:15.040 WARN 21062 --- [ocp-async-8,d2a8ee159dac4f16,635095916c12546b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:15.077 INFO 21062 --- [ocp-async-2,90c65dad577a6fc2,65a680a11d35ae87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:16.231 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:16.233 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:17.040 WARN 21062 --- [ocp-async-9,7e97cbc964a6c524,0db1fbd01e5a5f10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:17.077 INFO 21062 --- [ocp-async-4,ff2e408d32683ad5,8b4edc561b1d9feb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:17.542 INFO 21062 --- [ocp-schedules-17,d89d7713cdbec2f4,44b70b76875984d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:58:17.983 INFO 21062 --- [prometheus-minute-schedule-1,3378f761aecb892f,ab26796101f6f149] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:58:18.589449+08:00, lastProcessTime=2024-11-04T13:58:08.590242+08:00 2024-11-04 13:58:18.631 INFO 21062 --- [ocp-async-2,e75b9c4909b5b099,54364efae9684f53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:58:18.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:58:18.589449+08:00 2024-11-04 13:58:19.040 WARN 21062 --- [ocp-async-6,f440759d817ef433,903047a057899c32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:19.077 INFO 21062 --- [ocp-async-1,03eace16e3649ce0,25b02f5e0d4005ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:19.233 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:19.234 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:20.032 INFO 21062 --- [ocp-schedules-19,0767dcd3fa9072dd,72f10c4788e210f1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:58:20.983 INFO 21062 --- [prometheus-minute-schedule-1,0a6343113e6018db,3343aa94c8a91466] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:21.041 WARN 21062 --- [ocp-async-7,f4b8f96139cc6dcc,29c215086a34adfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:21.077 INFO 21062 --- [ocp-async-2,9cb864cde194f947,0fe56d20bedc5214] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:22.234 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:22.235 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:23.041 WARN 21062 --- [ocp-async-5,e19bf161755ed596,032784028e53af4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:23.077 INFO 21062 --- [ocp-async-4,222d44aad56287f4,04a3eb311ca1ab05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a8c7d6b00eded9e7,738cbae32048b8d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:25.041 WARN 21062 --- [ocp-async-6,2a80a22d9ac40cca,84b55690faddf8ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:25.078 INFO 21062 --- [ocp-async-3,6a37e9ec7b0a7813,77c6009969397c67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:25.235 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:25.236 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ba3bf1b0db719966,e7ebd7c8d1c4ee85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c00926e2deebc86c,8c5cd60f9b3b275e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:58:27.041 WARN 21062 --- [ocp-async-7,eb0963f4cb19135a,333c08371b44bd77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:27.078 INFO 21062 --- [ocp-async-8,ce35397e78912734,1495271011f76f03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:27.568 INFO 21062 --- [ocp-schedules-4,4ae212b8fe8121c4,d869ee6a6204dd01] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:58:28.236 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:28.237 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:28.498 INFO 21062 --- [ocp-schedules-6,89167aba39ff74a2,995b19eec9faedba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:58:28.499 INFO 21062 --- [ocp-schedules-21,27ad4b179ccdd04e,127114e322795364] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:58:28.500 INFO 21062 --- [ocp-schedules-11,66d396d74f43ad22,1d375ddc38c762ef] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:58:28.500 INFO 21062 --- [ocp-schedules-13,3cfdd33278a11e7b,ae28b6ccf73a7692] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:58:28.502 INFO 21062 --- [ocp-schedules-6,89167aba39ff74a2,995b19eec9faedba] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:58:28.502 INFO 21062 --- [ocp-schedules-27,6044afc489e7b1ed,befa473534b45a88] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:58:28.503 INFO 21062 --- [ocp-schedules-11,66d396d74f43ad22,1d375ddc38c762ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:28.504 INFO 21062 --- [ocp-schedules-22,5761f71b31ed1a3e,f25b7eb2f68e5f1e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:58:28.504 INFO 21062 --- [ocp-schedules-13,3cfdd33278a11e7b,ae28b6ccf73a7692] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:58:28.506 INFO 21062 --- [ocp-schedules-21,27ad4b179ccdd04e,127114e322795364] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:58:28.512 INFO 21062 --- [ocp-schedules-26,b404b832d125190a,d010e51dc52d7d3f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:58:28.512 INFO 21062 --- [ocp-schedules-26,b404b832d125190a,d010e51dc52d7d3f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:58:28.512 INFO 21062 --- [ocp-schedules-23,5a6d8c6c87f050df,ac2bb4703ba9b06e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:58:28.512 INFO 21062 --- [ocp-schedules-23,5a6d8c6c87f050df,ac2bb4703ba9b06e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:58:28.513 INFO 21062 --- [ocp-schedules-15,b96d85d61be146a2,94e4e9bbe67e7a4b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:58:28.513 INFO 21062 --- [ocp-schedules-15,b96d85d61be146a2,94e4e9bbe67e7a4b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:58:28.571 INFO 21062 --- [ocp-schedules-4,cb992498c7c2c571,23922a91218d5b0c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:58:28.571 INFO 21062 --- [ocp-schedules-4,cb992498c7c2c571,23922a91218d5b0c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:58:28.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:58:28.590999+08:00, lastProcessTime=2024-11-04T13:58:18.589449+08:00 2024-11-04 13:58:28.631 INFO 21062 --- [ocp-async-1,e1bb5d0ae1470442,41551b3945c87a07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:58:28.719 INFO 21062 --- [ocp-schedules-10,48c2fad1bcf3c6d4,1d6ce0360d7070e7] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 13:58:28.719 INFO 21062 --- [ocp-schedules-10,48c2fad1bcf3c6d4,1d6ce0360d7070e7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 13:58:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:58:28.590999+08:00 2024-11-04 13:58:28.768 INFO 21062 --- [ocp-schedules-19,7756ac8ec63c9560,81d3e1d34c35d86d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:58:28.977 INFO 21062 --- [ocp-schedules-14,6aad2b6c0b075135,ff2c76c1beb51b63] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:58:28.977 INFO 21062 --- [ocp-schedules-14,6aad2b6c0b075135,ff2c76c1beb51b63] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:58:29.022 INFO 21062 --- [ocp-schedules-21,bd96ac82075650e2,08fd7a14d47e6f56] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:58:29.041 WARN 21062 --- [ocp-async-10,e6569c12b070afd1,01620c691eae22ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:29.057 INFO 21062 --- [ocp-schedules-15,e7745307b15cf23a,b642a451a5c22e92] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:58:29.063 INFO 21062 --- [ocp-schedules-15,e7745307b15cf23a,b642a451a5c22e92] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:58:29.063 INFO 21062 --- [ocp-schedules-15,e7745307b15cf23a,b642a451a5c22e92] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:58:29.078 INFO 21062 --- [ocp-async-6,72e646a631670f24,02ffd36cc62c2c2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:29.153 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 13:58:29.153 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 13:58:29.162 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 13:58:29.162 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 13:58:29.162 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 13:58:29.173 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 13:58:29.173 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 13:58:29.173 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 13:58:29.177 INFO 21062 --- [ocp-schedules-1,02c836a3cb12bdce,d5e7c9fee708ce87] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 13:58:29.335 INFO 21062 --- [ocp-schedules-29,b474edebf9934256,356b219959a1d860] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:58:29.335 INFO 21062 --- [ocp-schedules-29,b474edebf9934256,356b219959a1d860] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:58:29.338 INFO 21062 --- [ocp-schedules-29,b474edebf9934256,356b219959a1d860] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:58:29.338 INFO 21062 --- [ocp-schedules-29,b474edebf9934256,356b219959a1d860] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:58:29.339 INFO 21062 --- [ocp-schedules-29,b474edebf9934256,356b219959a1d860] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:58:29.339 INFO 21062 --- [ocp-schedules-29,b474edebf9934256,356b219959a1d860] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:58:29.983 INFO 21062 --- [prometheus-minute-schedule-1,a2910f91a228502d,f58189e817d0263f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:30.035 INFO 21062 --- [ocp-schedules-14,c1f02a62e2dcbbbe,1a9a0512e5e4d0ec] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:58:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:58:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:58:30.858 INFO 21062 --- [ocp-schedules-3,97e90a86630b29c8,f730e9ac2417a133] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:58:30.865 INFO 21062 --- [ocp-schedules-3,97e90a86630b29c8,f730e9ac2417a133] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:58:30.865 INFO 21062 --- [ocp-schedules-3,97e90a86630b29c8,f730e9ac2417a133] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:58:30.869 INFO 21062 --- [ocp-schedules-8,4d49f8213d693463,30d8d2871afe3cea] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:58:30.869 INFO 21062 --- [ocp-schedules-8,4d49f8213d693463,30d8d2871afe3cea] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:58:30.896 INFO 21062 --- [ocp-schedules-8,4d49f8213d693463,30d8d2871afe3cea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:58:30.896 INFO 21062 --- [ocp-schedules-8,4d49f8213d693463,30d8d2871afe3cea] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:58:30.897 INFO 21062 --- [ocp-schedules-10,7aabedbc48fc82d4,06678a99f4d32527] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:58:31.041 WARN 21062 --- [ocp-async-7,ebc3cdd6242ccf94,02d5068586fbc043] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:31.051 INFO 21062 --- [ocp-schedules-9,b9e944bdc5350c34,5f685f019a1fdb02] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:58:31.051+08:00, clusters=[1] 2024-11-04 13:58:31.052 INFO 21062 --- [ocp-schedules-9,b9e944bdc5350c34,5f685f019a1fdb02] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:58:31.078 INFO 21062 --- [ocp-async-1,39fbdfed74ba037b,49321eaf9e0bb912] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:31.125 INFO 21062 --- [ocp-schedules-18,f8b137f0cff54a74,dcd27877b623f427] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:58:31.125 INFO 21062 --- [ocp-schedules-18,f8b137f0cff54a74,dcd27877b623f427] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:58:31.125 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:58:31.125 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:58:31.136 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:58:31.137 INFO 21062 --- [ocp-schedules-18,f8b137f0cff54a74,dcd27877b623f427] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:58:31.137 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:58:31.138 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:58:31.138 WARN 21062 --- [ocp-schedules-18,f8b137f0cff54a74,dcd27877b623f427] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:58:31.139 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:58:31.139 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:58:31.139 INFO 21062 --- [ocp-schedules-18,f8b137f0cff54a74,dcd27877b623f427] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:58:31.139 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:58:31.140 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:58:31.140 WARN 21062 --- [ocp-schedules-18,f8b137f0cff54a74,dcd27877b623f427] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:58:31.140 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:58:31.146 ERROR 21062 --- [ocp-schedules-18,f8b137f0cff54a74,dcd27877b623f427] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:58:31.153 INFO 21062 --- [ocp-schedules-22,5183d024f441e5b0,87b00aed3bda089d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 13:58:31.157 INFO 21062 --- [ocp-schedules-22,5183d024f441e5b0,87b00aed3bda089d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:31.205 INFO 21062 --- [ocp-schedules-26,d0c957ed84241943,b780a6e86256f4f7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:58:31.205+08:00, clusters=[1] 2024-11-04 13:58:31.209 INFO 21062 --- [ocp-schedules-26,d0c957ed84241943,b780a6e86256f4f7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:58:31.212 ERROR 21062 --- [ocp-schedules-22,5183d024f441e5b0,87b00aed3bda089d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:58:31.212 INFO 21062 --- [ocp-schedules-22,5183d024f441e5b0,87b00aed3bda089d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 13:58:31.237 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:31.238 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:32.983 INFO 21062 --- [prometheus-minute-schedule-1,ccbddf02ce550ad6,f2f6df24ebfbe271] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:33.042 WARN 21062 --- [ocp-async-8,c1a8c745bced3fb2,4540e9460742ff59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:33.079 INFO 21062 --- [ocp-async-5,aeb88fe23eff4a0f,dccf928b54984ebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:34.238 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:34.239 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:34.397 INFO 21062 --- [ocp-schedules-12,c0d15b1cbdcc82fb,5b39bf1edb6f70e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:58:34.397 INFO 21062 --- [ocp-schedules-12,a10b1f5a8a6f76f6,517a7e0c31e3a4c2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:58:34.402 INFO 21062 --- [ocp-schedules-12,a10b1f5a8a6f76f6,517a7e0c31e3a4c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:34.432 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:58:34.432 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:58:35.042 WARN 21062 --- [ocp-async-9,e51c8777bc256115,8505b58e23f8ca86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:35.079 INFO 21062 --- [ocp-async-4,c1dccc69d6f7cd13,da4fd7bf5409b0f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e91d53f1fe68e4ab,dabb0fa71370f9ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:36.451 INFO 21062 --- [ocp-schedules-16,684911f9de64c4ba,a909e8af4e8aa4b7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:58:36.456 INFO 21062 --- [ocp-schedules-16,684911f9de64c4ba,a909e8af4e8aa4b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:36.543 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:58:36.545 ERROR 21062 --- [ocp-schedules-16,684911f9de64c4ba,a909e8af4e8aa4b7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:58:36.545 INFO 21062 --- [ocp-schedules-16,684911f9de64c4ba,a909e8af4e8aa4b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:58:37.042 WARN 21062 --- [ocp-async-6,6cfc69dcdff9a266,1896a48a0bc7a808] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:37.079 INFO 21062 --- [ocp-async-3,b091950d52b3efb4,2987e476066d400f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:37.239 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:37.239 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:37.610 INFO 21062 --- [ocp-schedules-23,688aa9955d512c3b,39b5996b49b49fc3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:58:38.504 INFO 21062 --- [ocp-schedules-13,3cfdd33278a11e7b,ae28b6ccf73a7692] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:58:38.505 INFO 21062 --- [ocp-schedules-11,66d396d74f43ad22,1d375ddc38c762ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:58:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:58:38.589711+08:00, lastProcessTime=2024-11-04T13:58:28.590999+08:00 2024-11-04 13:58:38.631 INFO 21062 --- [ocp-async-4,827ea66aa4be8d47,5642ba5b5ebde9bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:58:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:58:38.589711+08:00 2024-11-04 13:58:38.909 INFO 21062 --- [ocp-schedules-28,c04b4d25876f56fc,e84477c96921b474] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:58:38.912 INFO 21062 --- [ocp-schedules-28,c04b4d25876f56fc,e84477c96921b474] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:38.913 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:58:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c36a235b5fb54cca,1e243d90b6671028] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:39.042 WARN 21062 --- [ocp-async-2,8aa071055ffb5588,64fe304c464d9072] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:39.079 INFO 21062 --- [ocp-async-10,658e96c63e3ad98f,5a7b50be4cf4bcd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:58:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:58:40.039 INFO 21062 --- [ocp-schedules-6,295d3827929109c6,7f1f7e745b5fb6b6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:58:40.240 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:40.240 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:41.043 WARN 21062 --- [ocp-async-9,38f510e902897388,748e6cb7c078b707] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:41.079 INFO 21062 --- [ocp-async-4,2231c1ce24602238,3cbc4c12b2955b88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:58:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:58:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 13:58:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 13:58:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 13:58:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 13:58:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 13:58:41.983 INFO 21062 --- [prometheus-minute-schedule-1,83758fa830c0d8dd,5be969f210863c8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:43.043 WARN 21062 --- [ocp-async-6,52d35212ff484e02,f4d92c84dbdc6ce6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:43.080 INFO 21062 --- [ocp-async-1,f5e356bfa2a141a8,ffe5a7ab034f958b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:43.241 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:43.241 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:44.983 INFO 21062 --- [prometheus-minute-schedule-1,2b6b0721a2491f0f,ab7ec35acbcf34d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:45.043 WARN 21062 --- [ocp-async-1,0f63c96c1ce0b366,7907a0383fd7e624] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:45.080 INFO 21062 --- [ocp-async-5,b87bf35044f7691f,975ca2a068896b26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:46.241 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:46.242 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:47.043 WARN 21062 --- [ocp-async-10,1776976fc5026805,3a84415107496b1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:47.080 INFO 21062 --- [ocp-async-7,e5c2737ba51b42f7,029eb4f4be65fa89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:47.635 INFO 21062 --- [ocp-schedules-21,5374b35b0a1121c2,e542ec6f9ebc7cf2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:58:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2ef50561945fa3cb,7b306d151a54829a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:58:48.590118+08:00, lastProcessTime=2024-11-04T13:58:38.589711+08:00 2024-11-04 13:58:48.631 INFO 21062 --- [ocp-async-2,d0b8648c60f5aeed,e6721742340466a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:58:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:58:48.590118+08:00 2024-11-04 13:58:48.913 INFO 21062 --- [ocp-schedules-28,c04b4d25876f56fc,e84477c96921b474] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:58:49.043 WARN 21062 --- [ocp-async-3,19ef3780b620bea2,1792dc975e80a394] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:49.080 INFO 21062 --- [ocp-async-4,fd61ceaaa09e4909,877e0cb39fb74cad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:49.242 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:49.243 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:50.043 INFO 21062 --- [ocp-schedules-23,7dd82212789385e1,3810899b621032da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:58:50.868 INFO 21062 --- [ocp-schedules-15,823c95d12f3e2b96,c44ebbbe0830844d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:58:50.876 INFO 21062 --- [ocp-schedules-15,823c95d12f3e2b96,c44ebbbe0830844d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:58:50.876 INFO 21062 --- [ocp-schedules-15,823c95d12f3e2b96,c44ebbbe0830844d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:58:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a3800c433100a203,f83f5f6c4f3d9197] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:51.044 WARN 21062 --- [ocp-async-4,50f206dc3d074237,91a7100792b2b6e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:51.080 INFO 21062 --- [ocp-async-2,0f662f38dea9a894,03481960c96efdf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:52.243 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:52.244 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:53.044 WARN 21062 --- [ocp-async-2,59c66da4afb0750f,94a0394f6c039a51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:53.081 INFO 21062 --- [ocp-async-9,a7e4bfb4f3bfeb89,543b3c26a103102a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:53.983 INFO 21062 --- [prometheus-minute-schedule-1,2bdfebb0a054000e,736d997575fae4a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:55.044 WARN 21062 --- [ocp-async-9,90672334a5616a33,f621cf13a896aab2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:55.081 INFO 21062 --- [ocp-async-6,50a0cc9f441e975c,17e916e8a1403672] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:55.244 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:55.245 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:56.983 INFO 21062 --- [prometheus-minute-schedule-1,89c585c6c815c7f1,6ff46ea65c7ac611] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:58:57.044 WARN 21062 --- [ocp-async-8,f9a5d9b5234a5b4d,4abd69f64638a38d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:57.081 INFO 21062 --- [ocp-async-4,bbce074a2da97081,2c53ec1e81486ba1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:57.681 INFO 21062 --- [ocp-schedules-6,2fba9f8c69d098ea,f94bc892d3a8e978] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:58:58.245 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:58:58.246 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:58:58.500 INFO 21062 --- [ocp-schedules-14,136c2dd7d5f9cef4,12ebf3968b579d55] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:58:58.500 INFO 21062 --- [ocp-schedules-3,68dbc9b0fbe8ca61,750c41f2ba7e2a39] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:58:58.502 INFO 21062 --- [ocp-schedules-15,695f39b59826780b,271cc493ff573039] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:58:58.503 INFO 21062 --- [ocp-schedules-14,136c2dd7d5f9cef4,12ebf3968b579d55] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:58:58.504 INFO 21062 --- [ocp-schedules-3,68dbc9b0fbe8ca61,750c41f2ba7e2a39] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:58:58.504 INFO 21062 --- [ocp-schedules-8,423f7ddff137914f,c193f7f65b512da5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:58:58.510 INFO 21062 --- [ocp-schedules-7,ef45744e90b77af1,3787c744cb06c550] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:58:58.510 INFO 21062 --- [ocp-schedules-7,ef45744e90b77af1,3787c744cb06c550] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:58:58.510 INFO 21062 --- [ocp-schedules-16,73bf7730efcd63a6,9a1cf887b61c6e70] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:58:58.510 INFO 21062 --- [ocp-schedules-11,5320ac7277bbea80,af621495d69fafe7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:58:58.510 INFO 21062 --- [ocp-schedules-16,73bf7730efcd63a6,9a1cf887b61c6e70] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:58:58.510 INFO 21062 --- [ocp-schedules-11,5320ac7277bbea80,af621495d69fafe7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:58:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:58:58.588864+08:00, lastProcessTime=2024-11-04T13:58:48.590118+08:00 2024-11-04 13:58:58.632 INFO 21062 --- [ocp-async-1,70a078278854711c,7edd2907685b4b99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:58:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:58:58.588864+08:00 2024-11-04 13:58:59.025 INFO 21062 --- [ocp-schedules-13,ab3bf0a8e9a30829,f87cb9275a1c5cdf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:58:59.044 WARN 21062 --- [ocp-async-5,cf0febe1f469419e,390394e70ecb4130] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:58:59.081 INFO 21062 --- [ocp-async-7,b1b62ffb464933af,c047a21ab5df9d71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:58:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7f530a12bc3eea6f,b4993c7c1ad261e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730699940, interval=60 2024-11-04 13:59:00.046 INFO 21062 --- [ocp-schedules-25,a5a17da6228fac61,4e3aa8bd2bc72ebf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:59:01.044 WARN 21062 --- [ocp-async-9,ba6994302023cff8,2c94b1cb136fc523] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:01.081 INFO 21062 --- [ocp-async-1,0b6f36b8e8498521,2eb81631efc1c9be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:01.246 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:01.246 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:02.983 INFO 21062 --- [prometheus-minute-schedule-1,010d842c59cb5a88,09c4da0f448152b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:03.045 WARN 21062 --- [ocp-async-8,e2955c509cd010b8,07eb3440b46107b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:03.081 INFO 21062 --- [ocp-async-4,1f0e77744eff2956,8f59713a29555be4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:04.246 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:04.248 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:04.432 INFO 21062 --- [ocp-schedules-12,a10b1f5a8a6f76f6,517a7e0c31e3a4c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:59:04.432 INFO 21062 --- [ocp-schedules-12,eece784e3b32c6d6,4d030f83c6b418c0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:59:04.436 INFO 21062 --- [ocp-schedules-12,eece784e3b32c6d6,4d030f83c6b418c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:04.466 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:59:04.467 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:59:05.045 WARN 21062 --- [ocp-async-10,0cfd93cb7394a4f9,0e87bd9621660a9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:05.082 INFO 21062 --- [ocp-async-2,5a8748e586940668,8986180583426a89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e206fa3075b1f715,941e7656978da1ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:07.045 WARN 21062 --- [ocp-async-7,e29798f2ad953eed,83f81282a86f9780] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:07.082 INFO 21062 --- [ocp-async-3,dd673244cab30956,14f61331af68bf62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:07.248 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:07.249 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:07.708 INFO 21062 --- [ocp-schedules-30,89f26daf1d37da4d,f541e462ade874a0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:59:08.504 INFO 21062 --- [ocp-schedules-14,136c2dd7d5f9cef4,12ebf3968b579d55] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:59:08.505 INFO 21062 --- [ocp-schedules-3,68dbc9b0fbe8ca61,750c41f2ba7e2a39] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:59:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:59:08.589419+08:00, lastProcessTime=2024-11-04T13:58:58.588864+08:00 2024-11-04 13:59:08.632 INFO 21062 --- [ocp-async-2,57b11f9432bab37e,13a735c83e066f8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:59:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:59:08.589419+08:00 2024-11-04 13:59:08.983 INFO 21062 --- [prometheus-minute-schedule-1,0635b44cbd217fab,e3e0ef766a6bda55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:09.045 WARN 21062 --- [ocp-async-8,9ad4770bfd9baefb,1a8b39af4b7682a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:09.082 INFO 21062 --- [ocp-async-4,f349680981bdc702,de3641be067f2904] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:59:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:59:10.049 INFO 21062 --- [ocp-schedules-18,178a9770718adb6b,90a117adea096416] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:59:10.249 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:10.250 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:59:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:59:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 13:59:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 13:59:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 13:59:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 13:59:10.878 INFO 21062 --- [ocp-schedules-30,86b328ec26584ccd,fa28a364275c9d74] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:59:10.883 INFO 21062 --- [ocp-schedules-30,86b328ec26584ccd,fa28a364275c9d74] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:59:10.884 INFO 21062 --- [ocp-schedules-30,86b328ec26584ccd,fa28a364275c9d74] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:59:11.046 WARN 21062 --- [ocp-async-10,dab129a21af8745e,16f4d66e710d6d61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:11.082 INFO 21062 --- [ocp-async-2,34bdc5952349e527,6e6972de4d92d068] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:11.412 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=5, affectRows=5 2024-11-04 13:59:11.413 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=5, validCount=5 2024-11-04 13:59:11.414 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=16, affectRows=16 2024-11-04 13:59:11.414 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=16, validCount=16 2024-11-04 13:59:11.414 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=20, affectRows=20 2024-11-04 13:59:11.414 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=20, validCount=20 2024-11-04 13:59:11.415 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=48, affectRows=48 2024-11-04 13:59:11.415 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=40, affectRows=40 2024-11-04 13:59:11.415 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=40, affectRows=40 2024-11-04 13:59:11.415 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=47, validCount=47 2024-11-04 13:59:11.415 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=40, validCount=40 2024-11-04 13:59:11.415 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=40, validCount=40 2024-11-04 13:59:11.415 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=54, affectRows=54 2024-11-04 13:59:11.416 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=54, validCount=54 2024-11-04 13:59:11.416 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=24, affectRows=24 2024-11-04 13:59:11.416 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=72, affectRows=72 2024-11-04 13:59:11.416 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699880, nodeCount=73, affectRows=73 2024-11-04 13:59:11.416 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=24, validCount=24 2024-11-04 13:59:11.416 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=70, validCount=70 2024-11-04 13:59:11.416 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699880, nodeCount=73, validCount=73 2024-11-04 13:59:11.983 INFO 21062 --- [prometheus-minute-schedule-1,3ae1cfe2ce8b7020,21514239f3eb7c90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:13.046 WARN 21062 --- [ocp-async-7,67a51a387a02c754,d3c5e6950c61f617] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:13.083 INFO 21062 --- [ocp-async-3,6f17342b4a704f11,d30afde60a534377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:13.250 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:13.251 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:13.984 INFO 21062 --- [prometheus-cache-monitor1,ce2d8bcad4ec2fdc,82a027ea67481ae7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 13:59:13.984 INFO 21062 --- [prometheus-cache-monitor1,ce2d8bcad4ec2fdc,82a027ea67481ae7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 13:59:14.826 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:55:14.807, end=2024-11-04T13:57:14.823} 2024-11-04 13:59:14.831 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 13:59:14.831 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 13:59:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d08533509b1fc082,cb9a0a8638921042] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:15.046 WARN 21062 --- [ocp-async-8,c6e738a482e9cd19,f27eb66df83957d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:15.083 INFO 21062 --- [ocp-async-6,5a41f5fd613fc0d1,927ec9353f6600d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:16.251 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:16.252 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:17.046 WARN 21062 --- [ocp-async-10,14a6dcebb83a649f,c0dda158120dc4c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:17.083 INFO 21062 --- [ocp-async-5,1a8e9295274bc4e5,04e4113aa13d1f57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:17.749 INFO 21062 --- [ocp-schedules-18,0d26547f2128b586,8cd65e068d138b2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:59:17.983 INFO 21062 --- [prometheus-minute-schedule-1,557e16783a5b6f32,528e5a794f69cca3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:59:18.590063+08:00, lastProcessTime=2024-11-04T13:59:08.589419+08:00 2024-11-04 13:59:18.632 INFO 21062 --- [ocp-async-1,3b8f0031cf93553d,305656010c1344e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:59:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:59:18.590063+08:00 2024-11-04 13:59:19.046 WARN 21062 --- [ocp-async-9,cdd64dbaed2748f9,338e051c0a095bd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:19.083 INFO 21062 --- [ocp-async-3,6d2875223c2fc690,02fbeb0ba617e497] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:19.252 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:19.253 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:20.053 INFO 21062 --- [ocp-schedules-27,64b5c5333ee25b2e,c488ea65ad91d576] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:59:20.983 INFO 21062 --- [prometheus-minute-schedule-1,27022131dbacdb31,17d37d93c19c803a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:21.046 WARN 21062 --- [ocp-async-6,24d1f0019f12e3b6,62d3ceea270c7a99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:21.083 INFO 21062 --- [ocp-async-1,473b493959f209e1,41569a9f0c5ac6f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:22.253 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:22.254 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:23.047 WARN 21062 --- [ocp-async-4,dbd57bb3e3a4e14b,74737c96102c1625] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:23.083 INFO 21062 --- [ocp-async-5,469cdadb94acb8ce,e20ba59f25dcac8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:23.983 INFO 21062 --- [prometheus-minute-schedule-1,14d20a65a873b755,0814d4db14b362f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:25.047 WARN 21062 --- [ocp-async-2,7bc73086a24c147d,47868f9e7418cd48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:25.083 INFO 21062 --- [ocp-async-7,89cb91985063fd72,18140f0ab9fda4d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:25.254 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:25.255 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:26.983 INFO 21062 --- [prometheus-minute-schedule-1,849ecfbbd153779e,5a47d80c2296773c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ae7814540c1cbccf,8916d52d27d53831] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=3600 2024-11-04 13:59:27.047 WARN 21062 --- [ocp-async-3,1f7fcfb334d3cbc9,3f1f11556ec743ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:27.084 INFO 21062 --- [ocp-async-8,88f0989a705252d1,ec1a85b607a6f263] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:27.774 INFO 21062 --- [ocp-schedules-10,82ecc77262607d20,cdb5b3fddf8dd88d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:59:28.255 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:28.256 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:28.498 INFO 21062 --- [ocp-schedules-21,f10d569bacd504e6,9d3df984b9258991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 13:59:28.500 INFO 21062 --- [ocp-schedules-5,be7524715f50d458,922002d9477f4116] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:59:28.500 INFO 21062 --- [ocp-schedules-7,ad03cf77afcc5fe2,fe629dc53f900d96] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 13:59:28.500 INFO 21062 --- [ocp-schedules-28,abf3b787282b533e,3dd2eea54ca39bda] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:59:28.502 INFO 21062 --- [ocp-schedules-21,f10d569bacd504e6,9d3df984b9258991] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 13:59:28.502 INFO 21062 --- [ocp-schedules-1,9f476ff9fb22552f,e78ded1d37160b69] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:59:28.504 INFO 21062 --- [ocp-schedules-20,9970f47378ac8948,8d94f1239b6c7713] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:59:28.504 INFO 21062 --- [ocp-schedules-28,abf3b787282b533e,3dd2eea54ca39bda] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:28.504 INFO 21062 --- [ocp-schedules-5,be7524715f50d458,922002d9477f4116] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:59:28.506 INFO 21062 --- [ocp-schedules-7,ad03cf77afcc5fe2,fe629dc53f900d96] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 13:59:28.510 INFO 21062 --- [ocp-schedules-29,c0f59842b3f02c5c,5813984e1f2c1a77] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:59:28.510 INFO 21062 --- [ocp-schedules-18,8f2cae83c0a74201,0fa2e7a7969e9ca9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:59:28.510 INFO 21062 --- [ocp-schedules-29,c0f59842b3f02c5c,5813984e1f2c1a77] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:59:28.510 INFO 21062 --- [ocp-schedules-18,8f2cae83c0a74201,0fa2e7a7969e9ca9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:59:28.510 INFO 21062 --- [ocp-schedules-8,3c96344307036ede,007beb340f2f7497] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:59:28.510 INFO 21062 --- [ocp-schedules-8,3c96344307036ede,007beb340f2f7497] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:59:28.575 INFO 21062 --- [ocp-schedules-14,960db583620846c7,4a5d8160e4ae7811] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T13:59:28.575+08:00, tenantSize=0 2024-11-04 13:59:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:59:28.589948+08:00, lastProcessTime=2024-11-04T13:59:18.590063+08:00 2024-11-04 13:59:28.632 INFO 21062 --- [ocp-async-3,bd5e72553de477b5,ed3afc9aa56d84bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:59:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:59:28.589948+08:00 2024-11-04 13:59:28.770 INFO 21062 --- [ocp-schedules-11,121eab849171a9f0,c2cd4ff31d1ce24f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 13:59:28.982 INFO 21062 --- [ocp-schedules-22,fbcdca2d2d4641db,27cd6c1d1296351d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 13:59:28.982 INFO 21062 --- [ocp-schedules-22,fbcdca2d2d4641db,27cd6c1d1296351d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 13:59:29.028 INFO 21062 --- [ocp-schedules-1,f4ba17c5a14f5b9b,221dab7362848241] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:59:29.047 WARN 21062 --- [ocp-async-10,9bdfa74ad100f91d,8b5fc30160e19011] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:29.063 INFO 21062 --- [ocp-schedules-8,11da1c6b783ec69a,b3c8b71d54c4c10a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 13:59:29.068 INFO 21062 --- [ocp-schedules-8,11da1c6b783ec69a,b3c8b71d54c4c10a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 13:59:29.068 INFO 21062 --- [ocp-schedules-8,11da1c6b783ec69a,b3c8b71d54c4c10a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 13:59:29.084 INFO 21062 --- [ocp-async-2,42206a1fddb1ed9f,e174f3f359ce5fd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:29.342 INFO 21062 --- [ocp-schedules-9,28d7da3b510c7293,b46259206a21e7ea] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 13:59:29.342 INFO 21062 --- [ocp-schedules-9,28d7da3b510c7293,b46259206a21e7ea] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 13:59:29.343 INFO 21062 --- [ocp-schedules-9,28d7da3b510c7293,b46259206a21e7ea] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 13:59:29.343 INFO 21062 --- [ocp-schedules-9,28d7da3b510c7293,b46259206a21e7ea] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 13:59:29.344 INFO 21062 --- [ocp-schedules-9,28d7da3b510c7293,b46259206a21e7ea] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 13:59:29.344 INFO 21062 --- [ocp-schedules-9,28d7da3b510c7293,b46259206a21e7ea] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 13:59:29.983 INFO 21062 --- [prometheus-minute-schedule-1,d1020822031ad005,f5cc9bc4abab8080] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:30.055 INFO 21062 --- [ocp-schedules-24,b382182ecf06a746,5a6603d9737c3699] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:59:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:59:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 13:59:30.886 INFO 21062 --- [ocp-schedules-25,28f5a6ad816d8f60,36ed567f61174fba] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:59:30.891 INFO 21062 --- [ocp-schedules-25,28f5a6ad816d8f60,36ed567f61174fba] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:59:30.891 INFO 21062 --- [ocp-schedules-25,28f5a6ad816d8f60,36ed567f61174fba] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:59:30.897 INFO 21062 --- [ocp-schedules-23,f287e70b691aeb19,01aae6e0b652fbbc] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 13:59:30.897 INFO 21062 --- [ocp-schedules-23,f287e70b691aeb19,01aae6e0b652fbbc] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 13:59:30.924 INFO 21062 --- [ocp-schedules-30,022aeffbc4aac3cf,f58a1594908293ba] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 13:59:30.924 INFO 21062 --- [ocp-schedules-23,f287e70b691aeb19,01aae6e0b652fbbc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 13:59:30.924 INFO 21062 --- [ocp-schedules-23,f287e70b691aeb19,01aae6e0b652fbbc] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 13:59:31.047 WARN 21062 --- [ocp-async-2,2771bcb59e195c0c,936281a1c1d2b03c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:31.078 INFO 21062 --- [ocp-schedules-19,e0c63a83ec8a039c,40ca8ef098341b31] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T13:59:31.078+08:00, clusters=[1] 2024-11-04 13:59:31.079 INFO 21062 --- [ocp-schedules-19,e0c63a83ec8a039c,40ca8ef098341b31] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 13:59:31.084 INFO 21062 --- [ocp-async-6,b03ea3b6e95fe229,bc8c3ea9a84f031c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:31.154 INFO 21062 --- [ocp-schedules-16,66e5a2fdd820478c,e2fbf85f70daf08b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 13:59:31.154 INFO 21062 --- [ocp-schedules-16,66e5a2fdd820478c,e2fbf85f70daf08b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 13:59:31.154 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 13:59:31.154 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 13:59:31.164 INFO 21062 --- [ocp-schedules-16,66e5a2fdd820478c,e2fbf85f70daf08b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:59:31.165 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:59:31.165 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:59:31.166 WARN 21062 --- [ocp-schedules-16,66e5a2fdd820478c,e2fbf85f70daf08b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:59:31.166 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:59:31.166 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:59:31.166 INFO 21062 --- [ocp-schedules-16,66e5a2fdd820478c,e2fbf85f70daf08b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 13:59:31.167 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 13:59:31.167 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 13:59:31.167 WARN 21062 --- [ocp-schedules-16,66e5a2fdd820478c,e2fbf85f70daf08b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 13:59:31.168 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 13:59:31.168 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 13:59:31.174 ERROR 21062 --- [ocp-schedules-16,66e5a2fdd820478c,e2fbf85f70daf08b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 13:59:31.233 INFO 21062 --- [ocp-schedules-24,4b392c543e0d62fe,b5057500eb5cd434] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T13:59:31.233+08:00, clusters=[1] 2024-11-04 13:59:31.237 INFO 21062 --- [ocp-schedules-24,4b392c543e0d62fe,b5057500eb5cd434] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 13:59:31.256 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:31.257 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f7e5eb411d90c2df,f05dee9cf58fad4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:33.047 WARN 21062 --- [ocp-async-6,44862d8494f6173e,80a4776a39e36955] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:33.084 INFO 21062 --- [ocp-async-4,7b58cf952b207adc,7697c760b2a653ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:34.257 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:34.258 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:34.467 INFO 21062 --- [ocp-schedules-12,eece784e3b32c6d6,4d030f83c6b418c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:59:34.467 INFO 21062 --- [ocp-schedules-12,37160d3aa7a42f57,5c9b7ec9f0e65c6d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 13:59:34.472 INFO 21062 --- [ocp-schedules-12,37160d3aa7a42f57,5c9b7ec9f0e65c6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:34.502 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 13:59:34.502 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:59:35.048 WARN 21062 --- [ocp-async-4,d6a47dd1b168578b,8c7e39c08c838071] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:35.084 INFO 21062 --- [ocp-async-9,372762e3fef04913,a068475cad9d4b4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:35.983 INFO 21062 --- [prometheus-minute-schedule-1,81101cba0545e87b,8faa7026e088c2ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:36.545 INFO 21062 --- [ocp-schedules-10,cf906ffb61513fc6,89a6834467cf25cf] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 13:59:36.550 INFO 21062 --- [ocp-schedules-10,cf906ffb61513fc6,89a6834467cf25cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:36.637 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 13:59:36.640 ERROR 21062 --- [ocp-schedules-10,cf906ffb61513fc6,89a6834467cf25cf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 13:59:36.640 INFO 21062 --- [ocp-schedules-10,cf906ffb61513fc6,89a6834467cf25cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 13:59:37.048 WARN 21062 --- [ocp-async-9,825856903b417bc7,577bc3677d6f9b59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:37.084 INFO 21062 --- [ocp-async-7,a2c898b08601832f,90cbd22f98360c02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:37.259 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:37.260 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:37.817 INFO 21062 --- [ocp-schedules-11,e0779533f2460cbe,206f9f505d3139ed] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:59:38.505 INFO 21062 --- [ocp-schedules-28,abf3b787282b533e,3dd2eea54ca39bda] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 13:59:38.505 INFO 21062 --- [ocp-schedules-5,be7524715f50d458,922002d9477f4116] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:59:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:59:38.589704+08:00, lastProcessTime=2024-11-04T13:59:28.589948+08:00 2024-11-04 13:59:38.632 INFO 21062 --- [ocp-async-9,964fc9f6c2f3b074,6a50d9b9d116bca9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:59:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:59:38.589704+08:00 2024-11-04 13:59:38.983 INFO 21062 --- [prometheus-minute-schedule-1,495f0d98178d0581,8b5f708f58ab9064] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:39.048 WARN 21062 --- [ocp-async-8,7bf3f59101c8185a,d24f4c4f98097665] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:39.085 INFO 21062 --- [ocp-async-10,7b22324dae2b0e00,8230b7778dd584dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 13:59:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 13:59:40.058 INFO 21062 --- [ocp-schedules-3,2d7b888f792a1aa5,7d9e7017cdde0038] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:59:40.260 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:40.261 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:41.048 WARN 21062 --- [ocp-async-10,1a0e97a69b877777,4fd1fe057cbe3c65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:41.085 INFO 21062 --- [ocp-async-2,2b10a793505c4744,0d78e7a4fa76094a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a8f719442adf9e46,a9f142c3148cb914] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:43.048 WARN 21062 --- [ocp-async-2,cb3f44a3df48ce8e,846f68b9db5d4617] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:43.085 INFO 21062 --- [ocp-async-6,5d7ae6d40ff291d6,2c59678381069342] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:43.261 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:43.261 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5258851a070bea8c,0384d3c6691da38a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:45.048 WARN 21062 --- [ocp-async-6,5187cdf9aa7b79be,f881d144ef29d15c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:45.085 INFO 21062 --- [ocp-async-4,2a4c4c0b98c16896,06f075ea498eeb7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:46.262 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:46.262 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:47.049 WARN 21062 --- [ocp-async-4,4014f21b974024b1,31366374004f90bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:47.085 INFO 21062 --- [ocp-async-9,b9315636fdd864bc,d64b8606526f003d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:47.849 INFO 21062 --- [ocp-schedules-15,1ff3507b79041873,5fa4fe32e7a5a41a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:59:47.983 INFO 21062 --- [prometheus-minute-schedule-1,62cb87e1c9062c32,a53b86f0f0886fe6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:59:48.591200+08:00, lastProcessTime=2024-11-04T13:59:38.589704+08:00 2024-11-04 13:59:48.632 INFO 21062 --- [ocp-async-1,33bc2f3c88a65934,e8752709347badff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:59:48.787 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:59:48.591200+08:00 2024-11-04 13:59:48.913 INFO 21062 --- [ocp-schedules-28,d73619e04e5f5e19,353f053a7fe45bce] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 13:59:48.917 INFO 21062 --- [ocp-schedules-28,d73619e04e5f5e19,353f053a7fe45bce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:48.917 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 13:59:49.049 WARN 21062 --- [ocp-async-2,8db41278d72662db,6086d98f06045398] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:49.085 INFO 21062 --- [ocp-async-6,7fa1be4a440e6721,0afbe690fbb6dd80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:49.262 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:49.263 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:50.060 INFO 21062 --- [ocp-schedules-1,69975bf40f3b3872,8d8c7f7180b4ef2e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 13:59:50.894 INFO 21062 --- [ocp-schedules-10,ad604a6e262b113f,0ed03cf90de12e74] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 13:59:50.900 INFO 21062 --- [ocp-schedules-10,ad604a6e262b113f,0ed03cf90de12e74] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:59:50.900 INFO 21062 --- [ocp-schedules-10,ad604a6e262b113f,0ed03cf90de12e74] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 13:59:50.983 INFO 21062 --- [prometheus-minute-schedule-1,6aaf796af624c355,ea9f60320a8200e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:51.049 WARN 21062 --- [ocp-async-6,b3d82f7b145ebbd2,89779b37149ed15c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:51.086 INFO 21062 --- [ocp-async-1,9a5123e4149d64c1,0539168cd9424045] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:52.263 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:52.264 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:53.049 WARN 21062 --- [ocp-async-1,bb71132ecee7e478,03de8eb9ebb631e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:53.086 INFO 21062 --- [ocp-async-5,53c8768c06afbf67,f83f2bbe16f2267e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:53.983 INFO 21062 --- [prometheus-minute-schedule-1,51e60db73f2f4829,ccacaf94f6f03a78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:55.049 WARN 21062 --- [ocp-async-5,0aa190850c918c69,6a5fbd559f769158] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:55.086 INFO 21062 --- [ocp-async-3,814ca7bea5db76d4,c1a9dc8949c1fd3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:55.264 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:55.265 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e1ffadb98142bf59,4a6c8057f649cb7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 13:59:57.049 WARN 21062 --- [ocp-async-3,5203b055636f4c3c,d09d2e23a242d32d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:57.086 INFO 21062 --- [ocp-async-8,93c763cf32839311,51960c4f9053d663] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:57.918 INFO 21062 --- [ocp-schedules-25,472a0f65b5aea01b,0f42e45f54bd1201] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 13:59:58.265 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 13:59:58.266 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 13:59:58.500 INFO 21062 --- [ocp-schedules-16,104940451e1bf96f,f97a539cbb9fcd7c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 13:59:58.500 INFO 21062 --- [ocp-schedules-17,f9113a013b16423b,8ff10370bfb47fa5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 13:59:58.503 INFO 21062 --- [ocp-schedules-20,eeb35d1b00816ca5,fad31ad453516bc5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 13:59:58.503 INFO 21062 --- [ocp-schedules-16,104940451e1bf96f,f97a539cbb9fcd7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 13:59:58.504 INFO 21062 --- [ocp-schedules-17,f9113a013b16423b,8ff10370bfb47fa5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 13:59:58.505 INFO 21062 --- [ocp-schedules-2,e7e22377e93355f6,ac33c4138f7e305f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 13:59:58.509 INFO 21062 --- [ocp-schedules-14,1c0e0617b748b668,ba619aa3c1623176] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 13:59:58.509 INFO 21062 --- [ocp-schedules-14,1c0e0617b748b668,ba619aa3c1623176] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 13:59:58.510 INFO 21062 --- [ocp-schedules-30,738d5550fd973932,ce06f504acae9a0f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 13:59:58.510 INFO 21062 --- [ocp-schedules-30,738d5550fd973932,ce06f504acae9a0f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 13:59:58.510 INFO 21062 --- [ocp-schedules-19,97b409c5ad37d456,124e141bc33263da] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 13:59:58.510 INFO 21062 --- [ocp-schedules-19,97b409c5ad37d456,124e141bc33263da] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 13:59:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T13:59:58.590330+08:00, lastProcessTime=2024-11-04T13:59:48.591200+08:00 2024-11-04 13:59:58.633 INFO 21062 --- [ocp-async-2,d441f53ddedb0c92,d59f6ae85db05caa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 13:59:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T13:59:58.590330+08:00 2024-11-04 13:59:58.918 INFO 21062 --- [ocp-schedules-28,d73619e04e5f5e19,353f053a7fe45bce] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 13:59:59.030 INFO 21062 --- [ocp-schedules-24,6a6f7f715a9892d9,941bad1ab5d5e161] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 13:59:59.050 WARN 21062 --- [ocp-async-8,30e2aee1d85a2e2d,f78914aeb7c6e259] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 13:59:59.086 INFO 21062 --- [ocp-async-10,856293093275bfd5,5e0c11e8beff360a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 13:59:59.983 INFO 21062 --- [prometheus-minute-schedule-1,4f9ba6805f38a204,e7407cbafb05ed5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700000, interval=60 2024-11-04 14:00:00.063 INFO 21062 --- [ocp-schedules-23,d47d2c76a620c2a3,0ed9d102e7447de8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:00:00.178 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2024-11-04 14:00:00.0, currentDate=Mon Nov 04 14:00:00 CST 2024 2024-11-04 14:00:00.178 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999974, name=Detect log piece, templateName=BackupDetectLogPieceSchedule, scheduleRule=0 0 */2 * * ?, lastRunTime=null, nextRunTime=2024-11-04 14:00:00.0} 2024-11-04 14:00:00.186 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Detect log piece, nodeCount=1, taskCount=1 2024-11-04 14:00:00.186 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Detect log piece', taskDefinitionId=999974, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T14:00:00.186+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Detect log piece, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.schedule.DetectLogPieceTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 14:00:00.202 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:55:00.0, nextRun=2024-11-04 14:00:00.0, currentDate=Mon Nov 04 14:00:00 CST 2024 2024-11-04 14:00:00.202 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 13:55:00.0, nextRunTime=2024-11-04 14:00:00.0} 2024-11-04 14:00:00.226 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:00:00.230 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:00:00.230 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3753b1bfb4f13fa8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:00:01.050 WARN 21062 --- [ocp-async-9,c0e0d46586955898,88ad7922e5489d4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:01.087 INFO 21062 --- [ocp-async-2,ea2a35c146345160,c72b6c44f5ea492a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:01.121 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,9e14742db39389c9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 77, operation:EXECUTE, state: RUNNING 2024-11-04 14:00:01.141 INFO 21062 --- [subtask-executor10,dd193f8f67a5a423,209fbcd2129813be] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=77, context=Context{parallelIdx=-1, stringMap={task_instance_id=12, task_operation=execute, latest_execution_start_time=2024-11-04T14:00:01.114+08:00, sub_task_instance_name=Detect log piece, sub_task_instance_id=77}, listMap={}}, executor=10.46.128.24 2024-11-04 14:00:01.142 INFO 21062 --- [subtask-executor10,dd193f8f67a5a423,209fbcd2129813be] c.o.o.b.i.t.schedule.DetectLogPieceTask : start detect log piece task. 2024-11-04 14:00:01.148 INFO 21062 --- [subtask-executor10,dd193f8f67a5a423,209fbcd2129813be] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 77, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:00:01.266 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:01.267 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:01.842 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:12, status:SUCCESS 2024-11-04 14:00:01.843 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:12, status:SUCCESS 2024-11-04 14:00:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e17c95751152d6b2,84af01a3434a553d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:03.050 WARN 21062 --- [ocp-async-7,17e56849adecb5e4,6946d853434f716e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:03.087 INFO 21062 --- [ocp-async-6,cbdae5c7519a75e1,cc95dcfa3b7936e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:04.267 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:04.267 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:04.301 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8dbab66be279d4e7] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:50:05.0, nextRun=2024-11-04 14:00:04.0, currentDate=Mon Nov 04 14:00:04 CST 2024 2024-11-04 14:00:04.301 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8dbab66be279d4e7] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 13:50:05.0, nextRunTime=2024-11-04 14:00:04.0} 2024-11-04 14:00:04.307 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8dbab66be279d4e7] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 14:00:04.307 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8dbab66be279d4e7] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T14:00:04.307+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 14:00:04.503 INFO 21062 --- [ocp-schedules-12,37160d3aa7a42f57,5c9b7ec9f0e65c6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:00:04.503 INFO 21062 --- [ocp-schedules-12,1a5e6d8d5c8e751f,b1ce7692c10295f4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:00:04.506 INFO 21062 --- [ocp-schedules-12,1a5e6d8d5c8e751f,b1ce7692c10295f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:04.536 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:00:04.536 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:00:05.050 WARN 21062 --- [ocp-async-4,03ae81c969cd3f09,2b3dc17735ceabf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:05.087 INFO 21062 --- [ocp-async-1,52915ae897c80c63,6f664c3f630e3e04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:05.142 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,bbc7f812489f2496] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 78, operation:EXECUTE, state: RUNNING 2024-11-04 14:00:05.160 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=78, context=Context{parallelIdx=-1, stringMap={task_instance_id=13, task_operation=execute, latest_execution_start_time=2024-11-04T14:00:05.140+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=78}, listMap={}}, executor=10.46.128.24 2024-11-04 14:00:05.167 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:00:05.167 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:00:05.168 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.169 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:00:05.170 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.170 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.171 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.171 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:00:05.185 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.186 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.187 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.187 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:00:05.200 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.200 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.201 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.201 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:00:05.214 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.215 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.216 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.216 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:00:05.229 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.230 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.230 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.231 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:00:05.236 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.236 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.237 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.237 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:00:05.241 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.242 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.242 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.243 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:00:05.247 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.247 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.248 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.248 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:00:05.254 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.255 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.255 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.256 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:00:05.261 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.261 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.262 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.262 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:00:05.267 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.268 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.269 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.269 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:00:05.277 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.277 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.278 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.278 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:00:05.291 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.291 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.292 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.292 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:00:05.297 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.297 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.298 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.298 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:00:05.304 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.304 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.305 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.305 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:00:05.311 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.311 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.312 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.312 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:00:05.317 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.317 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.318 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.318 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:00:05.322 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.322 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.323 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.323 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:00:05.327 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.328 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.328 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.329 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:00:05.333 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.334 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.334 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.335 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:00:05.338 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.339 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.339 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.340 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:00:05.344 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.344 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.345 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.345 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:00:05.349 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.350 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.350 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.351 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:00:05.355 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.355 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.356 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.356 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:00:05.361 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.361 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.362 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.362 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:00:05.366 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.367 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.367 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.367 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:00:05.372 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.373 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.373 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.374 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:00:05.378 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.378 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.379 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.380 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:00:05.384 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.384 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.385 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.385 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:00:05.390 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.390 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.391 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.391 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:00:05.395 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.396 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.396 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.397 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:00:05.401 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.401 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.401 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.402 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:00:05.407 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.407 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.408 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.408 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:00:05.412 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.412 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.413 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.413 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:00:05.420 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.420 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.421 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.421 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:00:05.426 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.427 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.427 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.428 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:00:05.433 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.433 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.434 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.434 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:00:05.440 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.440 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.440 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.441 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:00:05.444 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.445 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.445 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.446 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:00:05.449 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.450 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.451 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.451 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:00:05.454 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.455 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.455 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.456 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:00:05.459 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.460 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.460 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.461 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:00:05.470 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.471 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.471 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.472 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:00:05.492 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.492 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.493 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.493 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:00:05.499 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.499 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.500 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.501 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:00:05.506 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.507 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.508 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.509 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:00:05.514 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.515 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.515 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.516 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:00:05.521 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.521 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.522 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.522 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:00:05.526 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.526 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.527 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.528 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:00:05.532 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:00:05.532 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:00:05.533 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.534 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:00:05.535 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.536 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.536 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.537 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:00:05.542 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.543 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:00:05.543 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:05.544 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:00:05.548 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:00:05.555 INFO 21062 --- [subtask-executor11,dd193f8f67a5a423,c9fcb0ec48cd46fb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 78, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:00:05.861 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:13, status:SUCCESS 2024-11-04 14:00:05.862 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:13, status:SUCCESS 2024-11-04 14:00:05.983 INFO 21062 --- [prometheus-minute-schedule-1,51eaea63a071e087,0c604170e13554e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:07.050 WARN 21062 --- [ocp-async-10,09a1c03c43075963,bde99cc90fbebc39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:07.087 INFO 21062 --- [ocp-async-5,ca0d1fb6a7fdb967,143ff03ad60f3461] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:07.268 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:07.268 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:08.109 INFO 21062 --- [ocp-schedules-29,34f86943fcbd27a0,5408f0840bf2a816] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:00:08.504 INFO 21062 --- [ocp-schedules-16,104940451e1bf96f,f97a539cbb9fcd7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:00:08.505 INFO 21062 --- [ocp-schedules-17,f9113a013b16423b,8ff10370bfb47fa5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:00:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:00:08.589931+08:00, lastProcessTime=2024-11-04T13:59:58.590330+08:00 2024-11-04 14:00:08.633 INFO 21062 --- [ocp-async-1,116775199e584338,3f5cf542580f301e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:00:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:00:08.589931+08:00 2024-11-04 14:00:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f7c2fa62a0ce4fcc,969cbf77a6bada20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:09.051 WARN 21062 --- [ocp-async-7,093ac594a6bebfd0,ad604613c1e8f843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:09.088 INFO 21062 --- [ocp-async-6,403e7b2f76781758,44d6175d531b5ab0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:10.002 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:00:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:00:10.066 INFO 21062 --- [ocp-schedules-3,1cebe4a592465648,c1beba18046d6339] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:00:10.269 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:10.269 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:00:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:00:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:00:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:00:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:00:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:00:10.903 INFO 21062 --- [ocp-schedules-23,bfe7640876b528a0,c60084ffadc0b8ac] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:00:10.909 INFO 21062 --- [ocp-schedules-23,bfe7640876b528a0,c60084ffadc0b8ac] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:00:10.909 INFO 21062 --- [ocp-schedules-23,bfe7640876b528a0,c60084ffadc0b8ac] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:00:11.051 WARN 21062 --- [ocp-async-4,f09d8a889effcd79,b23a38bd0a547af9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:11.088 INFO 21062 --- [ocp-async-1,c30233dbe8b33e11,2c389ccfe1394ab6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:11.424 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699940, nodeCount=209, affectRows=209 2024-11-04 14:00:11.424 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730699940, nodeCount=183, affectRows=183 2024-11-04 14:00:11.425 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699940, nodeCount=183, validCount=183 2024-11-04 14:00:11.425 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730699940, nodeCount=206, validCount=206 2024-11-04 14:00:11.983 INFO 21062 --- [prometheus-minute-schedule-1,75e914e5c1b70723,e0d025f8531f1c50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:13.051 WARN 21062 --- [ocp-async-10,e34a5add485f4001,05ff95065a702f29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:13.088 INFO 21062 --- [ocp-async-5,ef98894509d9d923,ad141b6ddb29ce99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:13.269 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:13.270 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:13.984 INFO 21062 --- [prometheus-cache-monitor1,31c3620d00ee3bd8,33908f75fe88f73c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:00:13.984 INFO 21062 --- [prometheus-cache-monitor1,31c3620d00ee3bd8,33908f75fe88f73c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:00:14.983 INFO 21062 --- [prometheus-minute-schedule-1,e8ed1f8bff90b77c,f65b6d4985049fb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:15.051 WARN 21062 --- [ocp-async-2,0d5c07e54513d5a6,d49c40b27e102dc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:15.088 INFO 21062 --- [ocp-async-3,98c7ea5594352be6,4aa4b87b28ea40b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:16.270 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:16.271 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:17.051 WARN 21062 --- [ocp-async-6,734aab05a3785f9c,3c0a6b623b26070f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:17.088 INFO 21062 --- [ocp-async-8,bab572942efd65ec,6df51a98ed4e676e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:17.983 INFO 21062 --- [prometheus-minute-schedule-1,63432f930b1ae6bf,9cf9cdab5f2b1e28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:18.157 INFO 21062 --- [ocp-schedules-21,b64e8edf6cb085b3,261ffc7094c49264] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:00:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:00:18.589295+08:00, lastProcessTime=2024-11-04T14:00:08.589931+08:00 2024-11-04 14:00:18.633 INFO 21062 --- [ocp-async-7,dd9e3190b6d365da,5c1d2787111a156d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:00:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:00:18.589295+08:00 2024-11-04 14:00:19.051 WARN 21062 --- [ocp-async-10,8f9c7d972116592c,2e77f75e2675cc45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:19.088 INFO 21062 --- [ocp-async-5,11fd307726943392,16c7c9929e145ff1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:19.271 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:19.272 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:20.068 INFO 21062 --- [ocp-schedules-25,8491ac8063156b4e,12f95720cfaefdb4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:00:20.983 INFO 21062 --- [prometheus-minute-schedule-1,c9eb13da44cd6500,141afb4f58a7a8c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:21.052 WARN 21062 --- [ocp-async-2,0c51032289eca817,cb3e8a7ac9a4dc23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:21.089 INFO 21062 --- [ocp-async-7,3259380234ba33bc,323cc316e7d4799b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:22.272 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:22.273 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:23.052 WARN 21062 --- [ocp-async-6,ca5a7d93a5f52010,fddbff7f37896c3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:23.089 INFO 21062 --- [ocp-async-8,ef68d76891ca4ef8,31610c9a4fb0cb50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8096d6e6c2314154,aadaad1720154cad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:25.052 WARN 21062 --- [ocp-async-1,0534fa817fd36f0f,8aedebb7751e78da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:25.089 INFO 21062 --- [ocp-async-9,fe0c3c9d574c6d09,6be0a7f7fbddf618] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:25.273 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:25.274 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:26.983 INFO 21062 --- [prometheus-minute-schedule-1,786d79e213e08b62,b522039ce905abae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:26.984 INFO 21062 --- [prometheus-hour-schedule-1,74149008c3bc0f79,7cd8d94a676ee04e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:00:27.052 WARN 21062 --- [ocp-async-5,3c4bab15dd87f9ce,839a490a2dbd7448] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:27.089 INFO 21062 --- [ocp-async-3,1e0833c3f12bff0e,eb2099e229b7797e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:27.152 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:00:27.152 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:00:27.161 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:00:27.161 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:00:27.161 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:00:27.162 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:00:27.168 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:00:27.168 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:00:27.168 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:00:27.168 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:00:27.211 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:00:27.219 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:00:27.219 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:00:27.220 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:00:27.220 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:00:27.220 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:00:27.228 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:00:27.770 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:00:28.196 INFO 21062 --- [ocp-schedules-25,bb51982455c2a389,93c7e36a7b8a3b8d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:00:28.274 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:28.275 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:28.373 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.204529297S 2024-11-04 14:00:28.373 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:00:28.498 INFO 21062 --- [ocp-schedules-30,ac72ef25e1822a34,a653dd22809a438a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:00:28.499 INFO 21062 --- [ocp-schedules-24,c6bbd85cd22acbca,8f0cfde13d46797b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:00:28.500 INFO 21062 --- [ocp-schedules-10,0b34c8102401bff9,c4527826689da70d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:00:28.500 INFO 21062 --- [ocp-schedules-16,7530414a3f2da694,7b2a001d47f68cdb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:00:28.502 INFO 21062 --- [ocp-schedules-30,ac72ef25e1822a34,a653dd22809a438a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:00:28.503 INFO 21062 --- [ocp-schedules-26,1abfd2b041a89ef2,d0785929da29139a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:00:28.503 INFO 21062 --- [ocp-schedules-10,0b34c8102401bff9,c4527826689da70d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:00:28.504 INFO 21062 --- [ocp-schedules-16,7530414a3f2da694,7b2a001d47f68cdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:28.504 INFO 21062 --- [ocp-schedules-24,c6bbd85cd22acbca,8f0cfde13d46797b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:00:28.505 INFO 21062 --- [ocp-schedules-20,7eed6fa8f97730f4,0c0e8ba596cf2b45] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:00:28.510 INFO 21062 --- [ocp-schedules-6,bf2f3f0fd3d2e85f,74886feb34034228] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:00:28.510 INFO 21062 --- [ocp-schedules-6,bf2f3f0fd3d2e85f,74886feb34034228] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:00:28.511 INFO 21062 --- [ocp-schedules-22,fcf09172dbde49c5,5dd9a48c5e817241] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:00:28.511 INFO 21062 --- [ocp-schedules-21,087e4671ba284798,14e4289e011ab184] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:00:28.511 INFO 21062 --- [ocp-schedules-21,087e4671ba284798,14e4289e011ab184] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:00:28.511 INFO 21062 --- [ocp-schedules-22,fcf09172dbde49c5,5dd9a48c5e817241] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:00:28.511 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:00:28.511 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:00:28.533 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:00:28.589 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:00:28.589 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:00:28.589 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:00:28.589 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:00:28.589 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:00:28.589 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:00:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:00:28.590937+08:00, lastProcessTime=2024-11-04T14:00:18.589295+08:00 2024-11-04 14:00:28.600 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:00:28.615 INFO 21062 --- [ocp-schedules-27,8387c016801f1a32,27fdcb9c66f4d113] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:00:28.616 INFO 21062 --- [ocp-schedules-27,8387c016801f1a32,27fdcb9c66f4d113] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:00:28.633 INFO 21062 --- [ocp-async-5,eccc1dd80c554c63,ef970b0eadf4fd78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:00:28.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:00:28.590937+08:00 2024-11-04 14:00:28.772 INFO 21062 --- [ocp-schedules-1,9ff0fa831320a768,4fc3d0f1c0a67d1a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:00:28.781 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] o.s.boot.SpringApplication : Started application in 1.619 seconds (JVM running for 5442.46) 2024-11-04 14:00:28.781 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.619891048S 2024-11-04 14:00:28.987 INFO 21062 --- [ocp-schedules-20,4cf081fcfa4e1443,511d5a60402d5c1d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:00:28.987 INFO 21062 --- [ocp-schedules-20,4cf081fcfa4e1443,511d5a60402d5c1d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:00:29.033 INFO 21062 --- [ocp-schedules-6,b56ee200730ea6c1,9fa132bf5796022f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:00:29.052 WARN 21062 --- [ocp-async-4,d96c21a0200c03c5,aeb81dcdb29a8734] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:29.068 INFO 21062 --- [ocp-schedules-11,88a8bf24a5db6707,9ee7d235c5664619] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:00:29.073 INFO 21062 --- [ocp-schedules-11,88a8bf24a5db6707,9ee7d235c5664619] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:00:29.073 INFO 21062 --- [ocp-schedules-11,88a8bf24a5db6707,9ee7d235c5664619] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:00:29.089 INFO 21062 --- [ocp-async-1,9ac1100b550a403c,1f652cc877c90ef1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:29.184 INFO 21062 --- [ocp-schedules-8,5416ce09bfa35a6e,abde1908d0de9d57] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:00:29.345 INFO 21062 --- [ocp-schedules-14,ab5d19f0b789b1cf,836e772078e2ed90] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:00:29.345 INFO 21062 --- [ocp-schedules-14,ab5d19f0b789b1cf,836e772078e2ed90] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:00:29.351 INFO 21062 --- [ocp-schedules-14,ab5d19f0b789b1cf,836e772078e2ed90] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:00:29.351 INFO 21062 --- [ocp-schedules-14,ab5d19f0b789b1cf,836e772078e2ed90] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:00:29.355 INFO 21062 --- [ocp-schedules-14,ab5d19f0b789b1cf,836e772078e2ed90] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:00:29.355 INFO 21062 --- [ocp-schedules-14,ab5d19f0b789b1cf,836e772078e2ed90] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:00:29.983 INFO 21062 --- [prometheus-minute-schedule-1,2c60a266828922ff,498e7bfc0ae2bc94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:30.071 INFO 21062 --- [ocp-schedules-1,031de4d43cc1d042,f085d7c046c96a0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:00:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:00:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:00:30.911 INFO 21062 --- [ocp-schedules-3,c0d49f5db87d81d5,e21eeb8079d87c89] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:00:30.916 INFO 21062 --- [ocp-schedules-3,c0d49f5db87d81d5,e21eeb8079d87c89] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:00:30.916 INFO 21062 --- [ocp-schedules-3,c0d49f5db87d81d5,e21eeb8079d87c89] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:00:30.924 INFO 21062 --- [ocp-schedules-26,cf450f81a8e3eee4,8c9a9a8f65102e08] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:00:30.924 INFO 21062 --- [ocp-schedules-26,cf450f81a8e3eee4,8c9a9a8f65102e08] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:00:30.950 INFO 21062 --- [ocp-schedules-26,cf450f81a8e3eee4,8c9a9a8f65102e08] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:00:30.950 INFO 21062 --- [ocp-schedules-26,cf450f81a8e3eee4,8c9a9a8f65102e08] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:00:30.951 INFO 21062 --- [ocp-schedules-17,5d4c85c6e6d80f4f,1af7fcfbcb4ddca1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:00:31.052 WARN 21062 --- [ocp-async-1,245f9c2193194045,a549e0b699b71c95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:31.090 INFO 21062 --- [ocp-async-2,bd428f2d513aac8f,271f821b6dd641ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:31.104 INFO 21062 --- [ocp-schedules-23,73a3839703923113,324115e651b7be8f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:00:31.104+08:00, clusters=[1] 2024-11-04 14:00:31.105 INFO 21062 --- [ocp-schedules-23,73a3839703923113,324115e651b7be8f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:00:31.182 INFO 21062 --- [ocp-schedules-7,253a5b09a5be0e8a,d05d324cf9122eed] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:00:31.183 INFO 21062 --- [ocp-schedules-7,253a5b09a5be0e8a,d05d324cf9122eed] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:00:31.183 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:00:31.183 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:00:31.191 INFO 21062 --- [ocp-schedules-7,253a5b09a5be0e8a,d05d324cf9122eed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:00:31.192 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:00:31.193 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:00:31.193 WARN 21062 --- [ocp-schedules-7,253a5b09a5be0e8a,d05d324cf9122eed] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:00:31.193 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:00:31.194 INFO 21062 --- [ocp-schedules-7,253a5b09a5be0e8a,d05d324cf9122eed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:00:31.194 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:00:31.195 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:00:31.195 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:00:31.195 WARN 21062 --- [ocp-schedules-7,253a5b09a5be0e8a,d05d324cf9122eed] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:00:31.196 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:00:31.196 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:00:31.201 ERROR 21062 --- [ocp-schedules-7,253a5b09a5be0e8a,d05d324cf9122eed] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 14:00:31.212 INFO 21062 --- [ocp-schedules-1,b02c780ad8f6bfb2,c3386dcf4cddc5aa] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:00:31.215 INFO 21062 --- [ocp-schedules-1,b02c780ad8f6bfb2,c3386dcf4cddc5aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:31.260 INFO 21062 --- [ocp-schedules-20,4ea116f2001c432b,487b0537caae4e7b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:00:31.260+08:00, clusters=[1] 2024-11-04 14:00:31.264 INFO 21062 --- [ocp-schedules-20,4ea116f2001c432b,487b0537caae4e7b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:00:31.270 ERROR 21062 --- [ocp-schedules-1,b02c780ad8f6bfb2,c3386dcf4cddc5aa] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:00:31.270 INFO 21062 --- [ocp-schedules-1,b02c780ad8f6bfb2,c3386dcf4cddc5aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:00:31.275 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:31.276 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:32.983 INFO 21062 --- [prometheus-minute-schedule-1,5b6f64f57cd16c1e,32a18e86455c13d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:33.053 WARN 21062 --- [ocp-async-2,ab50254d8b2d5901,188cf9399e73eeba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:33.090 INFO 21062 --- [ocp-async-6,895bf1a8efd2b0bf,83628bc8eb2974a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:34.276 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:34.276 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:34.536 INFO 21062 --- [ocp-schedules-12,1a5e6d8d5c8e751f,b1ce7692c10295f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:00:34.536 INFO 21062 --- [ocp-schedules-12,a46a397fc2536adf,26a785c7a6dd07b8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:00:34.540 INFO 21062 --- [ocp-schedules-12,a46a397fc2536adf,26a785c7a6dd07b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:34.568 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:00:34.568 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:00:35.053 WARN 21062 --- [ocp-async-6,5b8bee4a3bd45ad6,10c3074eca87b227] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:35.090 INFO 21062 --- [ocp-async-8,008272018c8ed826,1543d3dfd2f4a7ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:35.983 INFO 21062 --- [prometheus-minute-schedule-1,21caa05a550753bc,7f36b76edeed7675] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:36.640 INFO 21062 --- [ocp-schedules-21,f276bb2e17d50fab,fb0f14097247a891] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:00:36.644 INFO 21062 --- [ocp-schedules-21,f276bb2e17d50fab,fb0f14097247a891] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:36.727 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:00:36.729 ERROR 21062 --- [ocp-schedules-21,f276bb2e17d50fab,fb0f14097247a891] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:00:36.729 INFO 21062 --- [ocp-schedules-21,f276bb2e17d50fab,fb0f14097247a891] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:00:37.053 WARN 21062 --- [ocp-async-8,5afd3ad5a40f6695,5c49da7fd0751ef4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:37.090 INFO 21062 --- [ocp-async-9,14ed288c794711ca,94dead0037fbe99b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:37.277 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:37.278 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:38.245 INFO 21062 --- [ocp-schedules-3,537ad2f4d8381f25,8f6a33a82fb59be1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:00:38.504 INFO 21062 --- [ocp-schedules-10,0b34c8102401bff9,c4527826689da70d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:00:38.504 INFO 21062 --- [ocp-schedules-16,7530414a3f2da694,7b2a001d47f68cdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:00:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:00:38.589725+08:00, lastProcessTime=2024-11-04T14:00:28.590937+08:00 2024-11-04 14:00:38.633 INFO 21062 --- [ocp-async-8,ba1c7286776e2566,4eb2ef007db8e2ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:00:38.786 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:00:38.589725+08:00 2024-11-04 14:00:38.983 INFO 21062 --- [prometheus-minute-schedule-1,7ed6663e6a1d9570,45d0180b8a18af40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:39.053 WARN 21062 --- [ocp-async-3,2f2fafb06af0485d,4011b772ac07190a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:39.090 INFO 21062 --- [ocp-async-4,41e52d8168bb9df0,08b49c02d9eff85d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:00:40.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:00:40.073 INFO 21062 --- [ocp-schedules-24,4069e28053304d7c,7a3c896de800f9dd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:00:40.278 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:40.278 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:41.054 WARN 21062 --- [ocp-async-4,cfad7e7b6f6711ca,b3a1b923eca08587] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:41.091 INFO 21062 --- [ocp-async-1,5019b889da35b404,fc727f8319bdca3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:00:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:00:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:00:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:00:41.859 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:00:41.859 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=32ms, period=120000ms 2024-11-04 14:00:41.983 INFO 21062 --- [prometheus-minute-schedule-1,facf57412cca29c0,983cffa39cc72bca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:43.054 WARN 21062 --- [ocp-async-1,764ad9607c44d1d2,b5a19ccd13b23736] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:43.091 INFO 21062 --- [ocp-async-2,ec8f6601a9baba8d,faa5b5fcab49978f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:43.279 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:43.280 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:44.983 INFO 21062 --- [prometheus-minute-schedule-1,822db56fcf4e9868,dcdd35e0286f8447] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:45.054 WARN 21062 --- [ocp-async-2,5deeaebf5f2d6208,dd7cf9a69045fb27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:45.091 INFO 21062 --- [ocp-async-6,a573941fb1c20673,7891e125afc4fc81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:46.280 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:46.281 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:47.054 WARN 21062 --- [ocp-async-6,a365971e35b10084,63462d2ecac27bb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:47.091 INFO 21062 --- [ocp-async-8,41b3495cbe96525a,f8b34fe024eca72c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:47.983 INFO 21062 --- [prometheus-minute-schedule-1,52a81f8beebdea8c,89c0fef5147b6046] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:48.276 INFO 21062 --- [ocp-schedules-4,41a3bf5323ec9691,d2fc6421b1f3beea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:00:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:00:48.591408+08:00, lastProcessTime=2024-11-04T14:00:38.589725+08:00 2024-11-04 14:00:48.633 INFO 21062 --- [ocp-async-7,48b6adb8df7f5470,3e3f0486d4636f0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:00:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:00:48.591408+08:00 2024-11-04 14:00:49.054 WARN 21062 --- [ocp-async-1,64868deeee0e5025,73fffc637119d3d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:49.091 INFO 21062 --- [ocp-async-9,c68ed2e5d55d24a0,fc2f7ab781335132] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:49.281 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:49.282 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:50.076 INFO 21062 --- [ocp-schedules-10,7ce6a1925442e5a1,5596e0dd18b0469a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:00:50.918 INFO 21062 --- [ocp-schedules-7,c1eab1506c933ae8,1f0e6c7c5d80cf4f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:00:50.924 INFO 21062 --- [ocp-schedules-7,c1eab1506c933ae8,1f0e6c7c5d80cf4f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:00:50.924 INFO 21062 --- [ocp-schedules-7,c1eab1506c933ae8,1f0e6c7c5d80cf4f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:00:50.983 INFO 21062 --- [prometheus-minute-schedule-1,43ce602a48e9965b,29d7bc798cdee14a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:51.055 WARN 21062 --- [ocp-async-9,faac5f0ee0e4064e,b249fe31e91ba1db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:51.091 INFO 21062 --- [ocp-async-7,777421db58821b76,cdb7d0f9ba51a9e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:52.282 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:52.283 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:53.055 WARN 21062 --- [ocp-async-7,5c9bb31c7717e67f,2e89a7a683cbc23d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:53.091 INFO 21062 --- [ocp-async-10,6b954995cbed46ab,15013513ac5ba11c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:53.983 INFO 21062 --- [prometheus-minute-schedule-1,99c63553414014ef,dd08cd0ad8fe5960] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:55.055 WARN 21062 --- [ocp-async-10,3db87e4ebc1204e8,a421580c404df62f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:55.092 INFO 21062 --- [ocp-async-5,2bf60bcecbe9895b,8e093d981c6d30b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:55.284 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:55.285 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:56.983 INFO 21062 --- [prometheus-minute-schedule-1,81b12770ce5ce116,ab98f44d5b519f9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:00:57.055 WARN 21062 --- [ocp-async-5,ef26bcf6813b84ee,cfda5e7a0849d2d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:57.092 INFO 21062 --- [ocp-async-3,d1a68b7f68408292,2e18a80b3da0c345] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:58.285 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:00:58.285 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:00:58.318 INFO 21062 --- [ocp-schedules-25,cd400b5554b43c6e,b6205822be0c2e98] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:00:58.500 INFO 21062 --- [ocp-schedules-22,81646c4210b1f227,32c0230f3bc2f030] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:00:58.500 INFO 21062 --- [ocp-schedules-29,d7a08cda140e150d,33ce383cfc41b693] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:00:58.502 INFO 21062 --- [ocp-schedules-26,b0b2e5151af6169f,606051356028ad48] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:00:58.503 INFO 21062 --- [ocp-schedules-22,81646c4210b1f227,32c0230f3bc2f030] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:58.503 INFO 21062 --- [ocp-schedules-29,d7a08cda140e150d,33ce383cfc41b693] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:00:58.504 INFO 21062 --- [ocp-schedules-28,dbffed2de03b5a7e,d679f5b8fbe50db9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:00:58.510 INFO 21062 --- [ocp-schedules-15,57c83c77bb56c37d,61910c1982fccc2b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:00:58.510 INFO 21062 --- [ocp-schedules-7,8c8d72f4cf2ab03c,20634e55b2109af6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:00:58.510 INFO 21062 --- [ocp-schedules-15,57c83c77bb56c37d,61910c1982fccc2b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:00:58.510 INFO 21062 --- [ocp-schedules-7,8c8d72f4cf2ab03c,20634e55b2109af6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:00:58.510 INFO 21062 --- [ocp-schedules-6,18ffa922232b8fb3,ea0934afa144df71] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:00:58.510 INFO 21062 --- [ocp-schedules-6,18ffa922232b8fb3,ea0934afa144df71] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:00:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:00:58.588847+08:00, lastProcessTime=2024-11-04T14:00:48.591408+08:00 2024-11-04 14:00:58.634 INFO 21062 --- [ocp-async-2,4a3eccbfc4c48805,f941a55c92556557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:00:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:00:58.588847+08:00 2024-11-04 14:00:58.918 INFO 21062 --- [ocp-schedules-2,13e071cb114b821c,2ce7222384f2999d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:00:58.921 INFO 21062 --- [ocp-schedules-2,13e071cb114b821c,2ce7222384f2999d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:00:58.922 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:00:59.036 INFO 21062 --- [ocp-schedules-18,3592195f91019b96,e5e881fd78cef552] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:00:59.055 WARN 21062 --- [ocp-async-7,140c6875803489ab,c2cbbaad70a5325a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:00:59.092 INFO 21062 --- [ocp-async-8,db3b292cef016fba,d1e3fecef2f4c198] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:00:59.983 INFO 21062 --- [prometheus-minute-schedule-1,086781e4c75ad8cb,5ebd91331d66f106] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700060, interval=60 2024-11-04 14:01:00.079 INFO 21062 --- [ocp-schedules-19,8626d9cad3ef1496,696d6b1bebb10d4a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:01:00.205 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8e15167b1e2d8282] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 13:01:01.0, nextRun=2024-11-04 14:01:00.0, currentDate=Mon Nov 04 14:01:00 CST 2024 2024-11-04 14:01:00.205 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8e15167b1e2d8282] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999995, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=2024-11-04 13:01:01.0, nextRunTime=2024-11-04 14:01:00.0} 2024-11-04 14:01:00.207 WARN 21062 --- [task-coordinator2,dd193f8f67a5a423,8e15167b1e2d8282] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-11-04 14:01:00.207 WARN 21062 --- [task-coordinator2,dd193f8f67a5a423,8e15167b1e2d8282] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-11-04 14:01:00.208 WARN 21062 --- [task-coordinator2,dd193f8f67a5a423,8e15167b1e2d8282] c.o.o.c.t.e.c.worker.ReadyTaskWorker : failed to get template for class, action=com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule java.lang.NullPointerException: null at com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule.ready(CollectClusterSnapshotSchedule.java:74) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.getTaskMeta(ReadyTaskWorker.java:366) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.checkAndScheduleCronTask(ReadyTaskWorker.java:173) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.lambda$loopScheduleTask$1(ReadyTaskWorker.java:134) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.task.engine.coordinator.worker.AbstractTaskWorker.transactionWithoutResult(AbstractTaskWorker.java:61) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.loopScheduleTask(ReadyTaskWorker.java:133) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.work(ReadyTaskWorker.java:102) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lockAndExecute(TaskCoordinator.java:137) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$startup$0(TaskCoordinator.java:75) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:01:00.212 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8e15167b1e2d8282] c.o.o.c.t.e.c.worker.SubtaskGenerator : Split list is empty, name=Collect cluster hour metric, splitKey=cluster_ids, context=Argument{parallelIdx=-1, stringMap={}, listMap={}} 2024-11-04 14:01:00.221 WARN 21062 --- [task-coordinator2,dd193f8f67a5a423,8e15167b1e2d8282] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Subtask is empty, mark task failed, name=Collect cluster hour metric, taskId=14 2024-11-04 14:01:01.055 WARN 21062 --- [ocp-async-10,bde24e8d007ffa0e,0222cec256be7884] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:01.092 INFO 21062 --- [ocp-async-2,233a8152bdc1c999,295fc85ec39a6441] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:01.285 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:01.286 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7b81ab6a0b297d1e,74df25f53b94ccb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:03.056 WARN 21062 --- [ocp-async-5,a414b61a56940db4,c8884612a343b8f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:03.093 INFO 21062 --- [ocp-async-3,3d9f4ccdd194b9c1,b78f5f294ec5a833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:04.286 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:04.287 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:04.568 INFO 21062 --- [ocp-schedules-12,a46a397fc2536adf,26a785c7a6dd07b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:01:04.569 INFO 21062 --- [ocp-schedules-12,4acf5886efac93c4,e8beb84fd0c9ab88] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:01:04.573 INFO 21062 --- [ocp-schedules-12,4acf5886efac93c4,e8beb84fd0c9ab88] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:01:04.603 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:01:04.603 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:01:05.056 WARN 21062 --- [ocp-async-6,5f7282bf52d2dcc7,01e151b1c8c0b479] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:05.093 INFO 21062 --- [ocp-async-4,f8a46e05bec54e1b,5293f3dd2d0f46b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:05.983 INFO 21062 --- [prometheus-minute-schedule-1,343eab37375a34e0,971fc8103a909b03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:07.056 WARN 21062 --- [ocp-async-8,e606e003312ceb55,af2ddb7bed1fbc0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:07.093 INFO 21062 --- [ocp-async-1,36417f802bc7d30a,f61474b8c487c936] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:07.287 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:07.288 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:08.341 INFO 21062 --- [ocp-schedules-1,98dd2dbfc500f44b,31b02ede0e6dcb27] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:01:08.504 INFO 21062 --- [ocp-schedules-29,d7a08cda140e150d,33ce383cfc41b693] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:01:08.504 INFO 21062 --- [ocp-schedules-22,81646c4210b1f227,32c0230f3bc2f030] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:01:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:01:08.589295+08:00, lastProcessTime=2024-11-04T14:00:58.588847+08:00 2024-11-04 14:01:08.634 INFO 21062 --- [ocp-async-4,8217a2f533b90c7d,06dfad78116901bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:01:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:01:08.589295+08:00 2024-11-04 14:01:08.922 INFO 21062 --- [ocp-schedules-2,13e071cb114b821c,2ce7222384f2999d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:01:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a939af900c2351df,f93a354b8e93566d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:09.056 WARN 21062 --- [ocp-async-5,b65157e7aa9d954d,97f934319e2f167c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:09.093 INFO 21062 --- [ocp-async-3,bec98aefd9519c7a,92930f8f3b3adac6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:01:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:01:10.081 INFO 21062 --- [ocp-schedules-9,0067edcdf8ff6f20,24682dd696c96597] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:01:10.288 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:10.289 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:01:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:01:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:01:10.320 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:01:10.320 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:01:10.320 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:01:10.926 INFO 21062 --- [ocp-schedules-15,a2d847e24928ca74,5bd8076d58bf7343] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:01:10.931 INFO 21062 --- [ocp-schedules-15,a2d847e24928ca74,5bd8076d58bf7343] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:01:10.931 INFO 21062 --- [ocp-schedules-15,a2d847e24928ca74,5bd8076d58bf7343] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:01:11.056 WARN 21062 --- [ocp-async-6,20038b988472ed5f,352f80d617a4d050] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:11.093 INFO 21062 --- [ocp-async-4,00112ab8643a8719,31f8510623eccd51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:11.423 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=12, affectRows=12 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=12, validCount=12 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=20, affectRows=20 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=22, affectRows=22 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=17, affectRows=17 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=20, validCount=20 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=17, validCount=17 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=21, validCount=21 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=36, affectRows=36 2024-11-04 14:01:11.424 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=49, affectRows=49 2024-11-04 14:01:11.425 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=36, validCount=36 2024-11-04 14:01:11.425 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=49, validCount=49 2024-11-04 14:01:11.425 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=61, affectRows=61 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=59, affectRows=59 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=49, affectRows=49 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=23, affectRows=23 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700000, nodeCount=44, affectRows=44 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=60, validCount=60 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=23, validCount=23 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=58, validCount=58 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=49, validCount=49 2024-11-04 14:01:11.426 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700000, nodeCount=44, validCount=44 2024-11-04 14:01:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0d1fd85af68a176b,1630a2848c41d20f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:13.056 WARN 21062 --- [ocp-async-8,40b05a72b4b8d1e7,e11dd6784f50a05f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:13.094 INFO 21062 --- [ocp-async-1,ed83d583ad9e72db,07504f994342cade] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:13.289 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:13.290 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:13.984 INFO 21062 --- [prometheus-cache-monitor1,c5c9d51eccf762fc,5d409f26a7d15e0a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":778} 2024-11-04 14:01:13.985 INFO 21062 --- [prometheus-cache-monitor1,c5c9d51eccf762fc,5d409f26a7d15e0a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:01:14.845 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:57:14.823, end=2024-11-04T13:59:14.842} 2024-11-04 14:01:14.854 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:01:14.854 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:01:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2a82f32e41ef9fd6,3923bb2a153e29c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:15.057 WARN 21062 --- [ocp-async-2,d558cd1590753f36,967a9bac35cb7964] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:15.094 INFO 21062 --- [ocp-async-9,bc6e01ac49c11bc7,d455fc5380d1f6b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:16.290 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:16.291 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:17.057 WARN 21062 --- [ocp-async-3,f754da4ba9124f07,0f5fa1f629363114] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:17.094 INFO 21062 --- [ocp-async-7,7d1377ee0746cfe5,28086bf830f8bcb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8c4c325d98f25ff6,430e057e2c4c67a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:18.382 INFO 21062 --- [ocp-schedules-10,742ede7f5f0afb8b,fa42462962fe6fea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:01:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:01:18.589444+08:00, lastProcessTime=2024-11-04T14:01:08.589295+08:00 2024-11-04 14:01:18.634 INFO 21062 --- [ocp-async-5,8b804658d0c24a37,4f7d479c035cdb0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:01:18.784 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:01:18.589444+08:00 2024-11-04 14:01:19.057 WARN 21062 --- [ocp-async-8,3807e4729a2e41af,eea9664768816921] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:19.094 INFO 21062 --- [ocp-async-1,8d57be6c5ac7e274,566042fc7fd186e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:19.291 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:19.292 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:20.084 INFO 21062 --- [ocp-schedules-28,330802bd479e5f69,6089006453e2907a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:01:20.983 INFO 21062 --- [prometheus-minute-schedule-1,cb25f8fafa189f08,437376aa5a6066e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:21.057 WARN 21062 --- [ocp-async-2,c38b231fe0a9c943,08b884c0f60f20aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:21.094 INFO 21062 --- [ocp-async-5,fdc1cc700a2b5e3a,3193309e9a6599a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:22.292 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:22.293 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:23.057 WARN 21062 --- [ocp-async-3,0782b27f20631e6c,5a89447ec7ac1007] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:23.094 INFO 21062 --- [ocp-async-7,918dd26f0cb9021c,3600b629ce7e4de3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:23.983 INFO 21062 --- [prometheus-minute-schedule-1,9d257106f765f4c4,e9299789c7fecb17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:25.058 WARN 21062 --- [ocp-async-4,123ee8c0555018fe,45ae8d819d5061b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:25.095 INFO 21062 --- [ocp-async-10,6468cd9b62cd33bd,a14f514c526557ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:25.293 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:25.294 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ee496c32ad978bf3,e91b96fbd6cd7460] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:26.984 INFO 21062 --- [prometheus-hour-schedule-1,95bce83ba56cfa11,715ac7844c0f5b3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:01:27.058 WARN 21062 --- [ocp-async-10,4931ce55e01ea42e,8599605c69f93ae2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:27.095 INFO 21062 --- [ocp-async-9,ca61b34e64e092be,3c4ae4e10ee74c42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:28.294 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:28.295 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:28.446 INFO 21062 --- [ocp-schedules-25,b443319d9db869de,e3140291da12778e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:01:28.498 INFO 21062 --- [ocp-schedules-11,0b7008f7a36d10b5,c61b4e26f26e073d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:01:28.499 INFO 21062 --- [ocp-schedules-18,c3679bba821e16a4,e231ef869c560208] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:01:28.499 INFO 21062 --- [ocp-schedules-19,5ea4bce1bdac3738,669f9df2e0a638fb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:01:28.500 INFO 21062 --- [ocp-schedules-3,ab5aa3e04b783dbd,0e59753282e080ef] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:01:28.502 INFO 21062 --- [ocp-schedules-11,0b7008f7a36d10b5,c61b4e26f26e073d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:01:28.502 INFO 21062 --- [ocp-schedules-26,60ac5c3a3f74e823,70678a207c0741d8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:01:28.503 INFO 21062 --- [ocp-schedules-19,5ea4bce1bdac3738,669f9df2e0a638fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:01:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:01:28.504 INFO 21062 --- [ocp-schedules-8,af8313191f0a8dfc,29d82b9bec5c389b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:01:28.504 INFO 21062 --- [ocp-schedules-18,c3679bba821e16a4,e231ef869c560208] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:01:28.504 INFO 21062 --- [ocp-schedules-3,ab5aa3e04b783dbd,0e59753282e080ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:01:28.511 INFO 21062 --- [ocp-schedules-14,d13cddc97fdb546a,a8e1ad96f06809d4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:01:28.511 INFO 21062 --- [ocp-schedules-14,d13cddc97fdb546a,a8e1ad96f06809d4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:01:28.511 INFO 21062 --- [ocp-schedules-17,08a7dd189fd6759c,8381547da43a60fd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:01:28.511 INFO 21062 --- [ocp-schedules-16,f0234a86722029a6,6f5ff04fb8b447fb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:01:28.511 INFO 21062 --- [ocp-schedules-17,08a7dd189fd6759c,8381547da43a60fd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:01:28.511 INFO 21062 --- [ocp-schedules-16,f0234a86722029a6,6f5ff04fb8b447fb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:01:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:01:28.589490+08:00, lastProcessTime=2024-11-04T14:01:18.589444+08:00 2024-11-04 14:01:28.634 INFO 21062 --- [ocp-async-1,f3ff66466f2a4a20,540e2d8011b1a943] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:01:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:01:28.589490+08:00 2024-11-04 14:01:28.774 INFO 21062 --- [ocp-schedules-2,91c7b598d4ae941b,0f55d7562067e2f7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:01:28.991 INFO 21062 --- [ocp-schedules-1,e63553a36c8d0d63,57d0a5483ace27ce] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:01:28.991 INFO 21062 --- [ocp-schedules-1,e63553a36c8d0d63,57d0a5483ace27ce] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:01:29.038 INFO 21062 --- [ocp-schedules-18,44fb49992d6cd3be,5194b99ac4275d20] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:01:29.058 WARN 21062 --- [ocp-async-6,0f38aaf8b11e348f,b02bf437f8249a01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:29.073 INFO 21062 --- [ocp-schedules-17,786cddfaf54c79c7,250dfdafd185399a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:01:29.078 INFO 21062 --- [ocp-schedules-17,786cddfaf54c79c7,250dfdafd185399a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:01:29.078 INFO 21062 --- [ocp-schedules-17,786cddfaf54c79c7,250dfdafd185399a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:01:29.095 INFO 21062 --- [ocp-async-4,506d967d81d6fe30,35659c843ed045bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:29.177 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:01:29.178 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:01:29.186 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:01:29.186 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:01:29.186 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:01:29.197 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:01:29.197 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:01:29.197 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:01:29.201 INFO 21062 --- [ocp-schedules-5,216848e928eab3ac,15f231a0ab161a5e] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:01:29.360 INFO 21062 --- [ocp-schedules-7,157a281a42ff3cdf,8dbf011aa63fd237] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:01:29.360 INFO 21062 --- [ocp-schedules-7,157a281a42ff3cdf,8dbf011aa63fd237] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:01:29.363 INFO 21062 --- [ocp-schedules-7,157a281a42ff3cdf,8dbf011aa63fd237] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:01:29.363 INFO 21062 --- [ocp-schedules-7,157a281a42ff3cdf,8dbf011aa63fd237] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:01:29.363 INFO 21062 --- [ocp-schedules-7,157a281a42ff3cdf,8dbf011aa63fd237] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:01:29.364 INFO 21062 --- [ocp-schedules-7,157a281a42ff3cdf,8dbf011aa63fd237] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:01:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4efcef966ff70277,54db655826cc1543] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:30.088 INFO 21062 --- [ocp-schedules-9,b189357c4f393d52,e5ac984d0deb75a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:01:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:01:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:01:30.934 INFO 21062 --- [ocp-schedules-27,818473593b36a9ca,e08715958e126cce] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:01:30.940 INFO 21062 --- [ocp-schedules-27,818473593b36a9ca,e08715958e126cce] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:01:30.940 INFO 21062 --- [ocp-schedules-27,818473593b36a9ca,e08715958e126cce] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:01:30.950 INFO 21062 --- [ocp-schedules-25,d9649f6a322fce20,fb575ef0c8dc0f4f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:01:30.950 INFO 21062 --- [ocp-schedules-25,d9649f6a322fce20,fb575ef0c8dc0f4f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:01:30.976 INFO 21062 --- [ocp-schedules-23,15f919c770ad32d4,aa134f8bf91b7a4a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:01:30.977 INFO 21062 --- [ocp-schedules-25,d9649f6a322fce20,fb575ef0c8dc0f4f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:01:30.977 INFO 21062 --- [ocp-schedules-25,d9649f6a322fce20,fb575ef0c8dc0f4f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:01:31.058 WARN 21062 --- [ocp-async-4,61c4cb816df05dbb,8a95d0a20b86704c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:31.095 INFO 21062 --- [ocp-async-2,af1b3de1cbf2dfb2,84910ce54105bbeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:31.132 INFO 21062 --- [ocp-schedules-22,5ade088c78a11634,a4836b989f4d2f0d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:01:31.132+08:00, clusters=[1] 2024-11-04 14:01:31.134 INFO 21062 --- [ocp-schedules-22,5ade088c78a11634,a4836b989f4d2f0d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:01:31.209 INFO 21062 --- [ocp-schedules-9,f0277d46060f2205,5731d6b6f6a3ec06] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:01:31.209 INFO 21062 --- [ocp-schedules-9,f0277d46060f2205,5731d6b6f6a3ec06] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:01:31.209 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:01:31.209 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:01:31.219 INFO 21062 --- [ocp-schedules-9,f0277d46060f2205,5731d6b6f6a3ec06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:01:31.220 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:01:31.220 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:01:31.221 WARN 21062 --- [ocp-schedules-9,f0277d46060f2205,5731d6b6f6a3ec06] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:01:31.221 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:01:31.221 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:01:31.222 INFO 21062 --- [ocp-schedules-9,f0277d46060f2205,5731d6b6f6a3ec06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:01:31.222 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:01:31.222 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:01:31.223 WARN 21062 --- [ocp-schedules-9,f0277d46060f2205,5731d6b6f6a3ec06] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:01:31.223 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:01:31.223 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:01:31.232 ERROR 21062 --- [ocp-schedules-9,f0277d46060f2205,5731d6b6f6a3ec06] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 14:01:31.289 INFO 21062 --- [ocp-schedules-1,151b51814a3e4fda,36bc38024aacb306] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:01:31.289+08:00, clusters=[1] 2024-11-04 14:01:31.293 INFO 21062 --- [ocp-schedules-1,151b51814a3e4fda,36bc38024aacb306] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:01:31.295 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:31.296 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f8427adf09dd68ed,5ba05a640e5835e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:33.058 WARN 21062 --- [ocp-async-2,106c5c6a286866a8,4378157cd1f4dc10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:33.096 INFO 21062 --- [ocp-async-3,76aa91b9da07c77d,667a5c18393ac169] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:34.296 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:34.297 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:34.603 INFO 21062 --- [ocp-schedules-12,4acf5886efac93c4,e8beb84fd0c9ab88] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:01:34.603 INFO 21062 --- [ocp-schedules-12,15d8bb50467cc919,fe84a6bbccfaafa3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:01:34.607 INFO 21062 --- [ocp-schedules-12,15d8bb50467cc919,fe84a6bbccfaafa3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:01:34.637 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:01:34.637 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:01:35.059 WARN 21062 --- [ocp-async-3,9baca89d2a2cd863,a37ff901c5e434cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:35.096 INFO 21062 --- [ocp-async-7,85159a707363e4b7,ba1ded4636dbb316] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:35.983 INFO 21062 --- [prometheus-minute-schedule-1,71dfcc4445d08edc,ceaa28f43ecd0196] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:36.729 INFO 21062 --- [ocp-schedules-14,addbc3e92c18383c,336f0d16d91d0962] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:01:36.733 INFO 21062 --- [ocp-schedules-14,addbc3e92c18383c,336f0d16d91d0962] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:01:36.819 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:01:36.821 ERROR 21062 --- [ocp-schedules-14,addbc3e92c18383c,336f0d16d91d0962] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:01:36.821 INFO 21062 --- [ocp-schedules-14,addbc3e92c18383c,336f0d16d91d0962] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:01:37.059 WARN 21062 --- [ocp-async-7,68b108230dfa2351,ceaf3e6c3d35de31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:37.096 INFO 21062 --- [ocp-async-1,177081b0e09cbd2c,934448307555f310] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:37.297 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:37.298 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:38.489 INFO 21062 --- [ocp-schedules-16,53f23e8d4647ee31,c1bfd815d73d5224] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:01:38.504 INFO 21062 --- [ocp-schedules-19,5ea4bce1bdac3738,669f9df2e0a638fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:01:38.505 INFO 21062 --- [ocp-schedules-3,ab5aa3e04b783dbd,0e59753282e080ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:01:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:01:38.589080+08:00, lastProcessTime=2024-11-04T14:01:28.589490+08:00 2024-11-04 14:01:38.635 INFO 21062 --- [ocp-async-7,01e6c022ae700e9f,fba596c8e9905d7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:01:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:01:38.589080+08:00 2024-11-04 14:01:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a5d7b9116d5d01c2,a2592d9acac57ce9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:39.059 WARN 21062 --- [ocp-async-9,13cd80017a7ab689,503184b3a5a19cae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:39.096 INFO 21062 --- [ocp-async-6,845cd0054ea2ea13,958c30a051231fa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:01:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:01:40.091 INFO 21062 --- [ocp-schedules-21,62c06fe76c6eb7ef,e413d71b3e286f0e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:01:40.298 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:40.299 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:41.059 WARN 21062 --- [ocp-async-6,930a1e0925703521,28df364dd4932268] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:41.096 INFO 21062 --- [ocp-async-4,34d7f37d74a3d661,248a0cd84c648cd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:41.983 INFO 21062 --- [prometheus-minute-schedule-1,0dbfe3e71ea026ed,24e943542864cc73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:43.059 WARN 21062 --- [ocp-async-4,824c0ab34a3c85fc,93837446be2a46ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:43.096 INFO 21062 --- [ocp-async-2,e1beca1779b73d4c,2288e7a87cdc616b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:43.299 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:43.300 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:44.983 INFO 21062 --- [prometheus-minute-schedule-1,50bbf7710aa7d05a,9577ad569d8e037d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:45.060 WARN 21062 --- [ocp-async-2,413d248d62d3b35b,256ac165b1594714] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:45.097 INFO 21062 --- [ocp-async-3,381355728c14922b,4f0f6992b4a20209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:46.300 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:46.301 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:47.060 WARN 21062 --- [ocp-async-3,6895863effb92fd4,4018ac8a89bf8216] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:47.097 INFO 21062 --- [ocp-async-7,7c8fcae8207a5d14,ebe210eedecfc42a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:47.983 INFO 21062 --- [prometheus-minute-schedule-1,75f534b913f7c29f,8103eb8ddc136651] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:48.516 INFO 21062 --- [ocp-schedules-25,e500384ad2d06bef,8493f318783cc766] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:01:48.601 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:01:48.601560+08:00, lastProcessTime=2024-11-04T14:01:38.589080+08:00 2024-11-04 14:01:48.635 INFO 21062 --- [ocp-async-5,045587868e3bb0a5,6106ca229240f264] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:01:48.780 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:01:48.601560+08:00 2024-11-04 14:01:49.060 WARN 21062 --- [ocp-async-4,c60238c2f37f95ee,d29c506837e65eb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:49.097 INFO 21062 --- [ocp-async-1,a92b80e8f66672d7,22d347c4e0e110c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:49.301 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:49.302 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:50.093 INFO 21062 --- [ocp-schedules-15,7d36ff2dbcbe9554,0aefab1a6af60525] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:01:50.942 INFO 21062 --- [ocp-schedules-5,d34db5254de5a5b2,eb6bccec89e72500] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:01:50.947 INFO 21062 --- [ocp-schedules-5,d34db5254de5a5b2,eb6bccec89e72500] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:01:50.948 INFO 21062 --- [ocp-schedules-5,d34db5254de5a5b2,eb6bccec89e72500] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:01:50.983 INFO 21062 --- [prometheus-minute-schedule-1,591a60492199ec31,116516e690612cd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:51.060 WARN 21062 --- [ocp-async-1,9d2a5bd622224d49,3472c78e79e33abd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:51.097 INFO 21062 --- [ocp-async-5,d6efee3d67216b47,b254152cf22f5182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:52.302 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:52.303 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:53.060 WARN 21062 --- [ocp-async-5,caf586017bdbd7fd,a7cc5f68480ed215] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:53.097 INFO 21062 --- [ocp-async-8,a5bccfae6d2f4944,fd426332f005a40f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:53.983 INFO 21062 --- [prometheus-minute-schedule-1,291d8236489d3246,addb6059faddf9e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:55.060 WARN 21062 --- [ocp-async-8,80f4df85bd821cec,3ea9a23f428a6ed5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:55.097 INFO 21062 --- [ocp-async-10,522add5d9cebd521,f51d976002fce774] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:55.303 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:55.304 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:56.983 INFO 21062 --- [prometheus-minute-schedule-1,4b2574679ece0a19,cfb12b1a79a67c59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:01:57.061 WARN 21062 --- [ocp-async-10,5c2a9ce2eabd8770,d43717fe6bceae8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:57.098 INFO 21062 --- [ocp-async-9,668580a564f4cda5,c09c9644b3dedd62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:58.304 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:01:58.304 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:01:58.499 INFO 21062 --- [ocp-schedules-28,9b9702ecfdd4fb7d,1b42f0ec6d6987d4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:01:58.500 INFO 21062 --- [ocp-schedules-9,dcea0077285f7ad3,5266462b00f19ce4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:01:58.502 INFO 21062 --- [ocp-schedules-27,59dfb57f05adb24a,55502809b7e57e30] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:01:58.504 INFO 21062 --- [ocp-schedules-28,9b9702ecfdd4fb7d,1b42f0ec6d6987d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:01:58.504 INFO 21062 --- [ocp-schedules-9,dcea0077285f7ad3,5266462b00f19ce4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:01:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:01:58.505 INFO 21062 --- [ocp-schedules-21,2db069e7411bd746,59e454daccdd83d7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:01:58.509 INFO 21062 --- [ocp-schedules-1,0797c70993f4a24f,34580facf3f41a81] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:01:58.509 INFO 21062 --- [ocp-schedules-1,0797c70993f4a24f,34580facf3f41a81] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:01:58.510 INFO 21062 --- [ocp-schedules-30,c5860ffc3dce7527,2fc4784eda7d310c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:01:58.510 INFO 21062 --- [ocp-schedules-30,c5860ffc3dce7527,2fc4784eda7d310c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:01:58.510 INFO 21062 --- [ocp-schedules-5,67508436af50d388,41a17860e3c2d42d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:01:58.510 INFO 21062 --- [ocp-schedules-5,67508436af50d388,41a17860e3c2d42d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:01:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:01:58.589929+08:00, lastProcessTime=2024-11-04T14:01:48.601560+08:00 2024-11-04 14:01:58.602 INFO 21062 --- [ocp-schedules-19,f21cfc92276e1700,2ba2137b011261a0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:01:58.635 INFO 21062 --- [ocp-async-2,89526b12d40fb2f0,2c0f8e043e0161a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:01:58.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:01:58.589929+08:00 2024-11-04 14:01:59.041 INFO 21062 --- [ocp-schedules-13,7549aac344eb4c04,aa660acc5cc5c897] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:01:59.061 WARN 21062 --- [ocp-async-5,d367b80260411b65,1a75e73daeac9843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:01:59.098 INFO 21062 --- [ocp-async-7,924f1bb34b56b0d5,d1626f8e9dfebeb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:01:59.983 INFO 21062 --- [prometheus-minute-schedule-1,fd7a0620cf301e65,1e6663ce42faf8fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700120, interval=60 2024-11-04 14:02:00.097 INFO 21062 --- [ocp-schedules-10,f54409bc51878bc7,a3567650bc3a418f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:02:01.061 WARN 21062 --- [ocp-async-8,b6949c87bfcafcdf,3b45bef88ddf2e94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:01.098 INFO 21062 --- [ocp-async-2,b8f824f177143a6c,53aa33351a421c56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:01.305 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:01.305 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:02.983 INFO 21062 --- [prometheus-minute-schedule-1,48a4913dc0e489de,cd496bae49b8768d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:03.061 WARN 21062 --- [ocp-async-10,511b3c6bc370264d,812a1283a406d9af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:03.098 INFO 21062 --- [ocp-async-9,eeb234d7971c2751,4c87994347333735] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:04.306 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:04.306 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:04.638 INFO 21062 --- [ocp-schedules-12,15d8bb50467cc919,fe84a6bbccfaafa3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:02:04.638 INFO 21062 --- [ocp-schedules-12,7c12955f1fe4eb85,5d52328b95476496] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:02:04.642 INFO 21062 --- [ocp-schedules-12,7c12955f1fe4eb85,5d52328b95476496] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:04.673 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:02:04.673 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:02:05.061 WARN 21062 --- [ocp-async-3,c10ac2bcffa8a0f0,71b8a0708c552f85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:05.098 INFO 21062 --- [ocp-async-6,e71a3c63cbaf65ae,1b9b36deca8c4ccc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c9e526fd2ab1f09e,1ea04838dbae254f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:07.062 WARN 21062 --- [ocp-async-7,bdced0ae908a3238,c1a20d3d9fade92f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:07.098 INFO 21062 --- [ocp-async-4,2fafdad4610d2daf,89bea850f94c3563] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:07.307 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:07.307 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:08.504 INFO 21062 --- [ocp-schedules-9,dcea0077285f7ad3,5266462b00f19ce4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:02:08.505 INFO 21062 --- [ocp-schedules-28,9b9702ecfdd4fb7d,1b42f0ec6d6987d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:02:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:02:08.590350+08:00, lastProcessTime=2024-11-04T14:01:58.589929+08:00 2024-11-04 14:02:08.629 INFO 21062 --- [ocp-schedules-17,5f46fb9d9d3255ea,9ae669d82471ed83] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:02:08.635 INFO 21062 --- [ocp-async-6,0bf5bfddb89f96e6,3e2b3c80d1633772] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:02:08.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:02:08.590350+08:00 2024-11-04 14:02:08.923 INFO 21062 --- [ocp-schedules-1,48309e22b16381af,ad249ade3b7817a9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:02:08.926 INFO 21062 --- [ocp-schedules-1,48309e22b16381af,ad249ade3b7817a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:08.927 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:02:08.983 INFO 21062 --- [prometheus-minute-schedule-1,60f7e72cbb2bfe09,ef754e009b0e193b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:09.062 WARN 21062 --- [ocp-async-10,73a2abc266b33afe,917dc167cde552a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:09.099 INFO 21062 --- [ocp-async-9,e7e9dd7e717d20d0,837a56e55c3cf76d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:10.004 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:02:10.004 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:02:10.100 INFO 21062 --- [ocp-schedules-17,8f66de5ac314e325,75d2d69249ca1d88] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:02:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:02:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:02:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:02:10.307 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:10.308 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:02:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:02:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:02:10.951 INFO 21062 --- [ocp-schedules-18,a8ae29a0401a5b8b,ff85d1edae37f04a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:02:10.956 INFO 21062 --- [ocp-schedules-18,a8ae29a0401a5b8b,ff85d1edae37f04a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:02:10.956 INFO 21062 --- [ocp-schedules-18,a8ae29a0401a5b8b,ff85d1edae37f04a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:02:11.062 WARN 21062 --- [ocp-async-3,92a43485e3189ee5,b416fb627154dfdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:11.099 INFO 21062 --- [ocp-async-6,ad9fdc8edc04d4f6,884bddbce2cb4a46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:11.426 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=22, affectRows=22 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=22, validCount=22 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=35, affectRows=35 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=53, affectRows=53 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=31, affectRows=31 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=39, affectRows=39 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=54, affectRows=54 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=35, validCount=35 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=31, validCount=31 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=39, validCount=39 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=52, validCount=52 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=54, validCount=54 2024-11-04 14:02:11.427 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=96, affectRows=96 2024-11-04 14:02:11.428 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700060, nodeCount=62, affectRows=62 2024-11-04 14:02:11.428 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=94, validCount=94 2024-11-04 14:02:11.428 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700060, nodeCount=62, validCount=62 2024-11-04 14:02:11.983 INFO 21062 --- [prometheus-minute-schedule-1,708ff208af333c5b,5ea2eb1fc498ede0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:13.062 WARN 21062 --- [ocp-async-7,c254984033602695,43dfaaa41a298d14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:13.099 INFO 21062 --- [ocp-async-4,436a0dcd8edc5173,e5fabf9c236b629a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:13.308 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:13.309 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:13.984 INFO 21062 --- [prometheus-cache-monitor1,fa99c6569dfb67e1,0066236b80ae6556] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":778} 2024-11-04 14:02:13.984 INFO 21062 --- [prometheus-cache-monitor1,fa99c6569dfb67e1,0066236b80ae6556] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:02:14.983 INFO 21062 --- [prometheus-minute-schedule-1,943bdceb86acaa91,a004c9edf29c6613] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:15.062 WARN 21062 --- [ocp-async-2,d36ad4d77182f05e,a0009c861673468c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:15.099 INFO 21062 --- [ocp-async-1,707293a575154ad1,62541c2049e8ed93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:16.309 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:16.310 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:17.062 WARN 21062 --- [ocp-async-9,d3131699df65e52e,5e1166713fba40a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:17.099 INFO 21062 --- [ocp-async-5,44dc50a4fe9a9350,0bad9dc05457b45a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:17.983 INFO 21062 --- [prometheus-minute-schedule-1,cc220fe029b9341f,32e2b498e5db0f4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:18.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:02:18.591840+08:00, lastProcessTime=2024-11-04T14:02:08.590350+08:00 2024-11-04 14:02:18.635 INFO 21062 --- [ocp-async-10,eb32977b5d3b47c2,d86d961478511d5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:02:18.669 INFO 21062 --- [ocp-schedules-29,2a66e3f2c4d7b7ca,63b33c96cd441e0e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:02:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:02:18.591840+08:00 2024-11-04 14:02:18.927 INFO 21062 --- [ocp-schedules-1,48309e22b16381af,ad249ade3b7817a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:02:19.063 WARN 21062 --- [ocp-async-7,fa0c57a877402bb0,838a8fc87508bf08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:19.100 INFO 21062 --- [ocp-async-4,330a2d018fbf6a58,9192e3b7cfb86cc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:19.310 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:19.311 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:20.103 INFO 21062 --- [ocp-schedules-25,012a6a12026ec4d7,de1ca9cf431012ff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:02:20.983 INFO 21062 --- [prometheus-minute-schedule-1,6431f321fac6d9e2,6e840432bdfac83f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:21.063 WARN 21062 --- [ocp-async-2,a40162145081fb23,e8ee86863e81c206] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:21.100 INFO 21062 --- [ocp-async-1,2371ec8213a0df90,1aa1e62e648ba62a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:22.311 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:22.312 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:23.063 WARN 21062 --- [ocp-async-9,65464df2bcab7217,f42897a1f6c706dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:23.100 INFO 21062 --- [ocp-async-5,98c5c85e42e373d9,cdeba368256357f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a4add82e9fd17269,4b493b64bee5b694] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:25.063 WARN 21062 --- [ocp-async-6,c51d15b650b9abbe,c3b3b2e0ce904f54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:25.100 INFO 21062 --- [ocp-async-8,103a9cada6def517,716118e0f93a1767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:25.312 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:25.313 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f7c647f5d980b3c4,abddd9ba0176c771] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:26.984 INFO 21062 --- [prometheus-hour-schedule-1,bcd6f54cf36a0cac,4eb4933eb5913af4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:02:27.064 WARN 21062 --- [ocp-async-8,0a85605b26d5b1b6,1e7fcae65d6cb10a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:27.100 INFO 21062 --- [ocp-async-10,6b447e21f7bb7b72,871afc75643a98d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:28.313 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:28.314 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:28.498 INFO 21062 --- [ocp-schedules-9,480e97e98937b292,18f85481aa0b3a87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:02:28.499 INFO 21062 --- [ocp-schedules-3,6b30ca550f306ade,1f3ae4d5386397ea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:02:28.500 INFO 21062 --- [ocp-schedules-13,5381d4a9698719bc,856dfb5d28f1507c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:02:28.500 INFO 21062 --- [ocp-schedules-21,678c7d6cefd51f2e,0d28c8bebb067b63] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:02:28.502 INFO 21062 --- [ocp-schedules-9,480e97e98937b292,18f85481aa0b3a87] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:02:28.502 INFO 21062 --- [ocp-schedules-1,45151ff6b339ecbe,c419c1b02e877e91] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:02:28.503 INFO 21062 --- [ocp-schedules-13,5381d4a9698719bc,856dfb5d28f1507c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:02:28.504 INFO 21062 --- [ocp-schedules-21,678c7d6cefd51f2e,0d28c8bebb067b63] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:28.504 INFO 21062 --- [ocp-schedules-7,69eaf2439e17e1cf,c26499409154fb36] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:02:28.504 INFO 21062 --- [ocp-schedules-14,f14f2ceac687bc40,381d2f9dce26aaf3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:02:28.505 INFO 21062 --- [ocp-schedules-3,6b30ca550f306ade,1f3ae4d5386397ea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:02:28.507 INFO 21062 --- [ocp-schedules-14,f14f2ceac687bc40,381d2f9dce26aaf3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:28.510 INFO 21062 --- [ocp-schedules-19,7761bc64a3f77cd3,dc30e395927f6bb1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:02:28.510 INFO 21062 --- [ocp-schedules-6,69d64f0c422dde10,3200a226abd65eb2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:02:28.510 INFO 21062 --- [ocp-schedules-6,69d64f0c422dde10,3200a226abd65eb2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:02:28.510 INFO 21062 --- [ocp-schedules-19,7761bc64a3f77cd3,dc30e395927f6bb1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:02:28.510 INFO 21062 --- [ocp-schedules-23,fe52b594202375ba,52e0dfc5db1a0a2b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:02:28.510 INFO 21062 --- [ocp-schedules-23,fe52b594202375ba,52e0dfc5db1a0a2b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:02:28.535 INFO 21062 --- [ocp-schedules-30,10c3569dc403a2b0,780fd12446f2bb2d] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:02:28.535 INFO 21062 --- [ocp-schedules-30,10c3569dc403a2b0,780fd12446f2bb2d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:02:28.572 INFO 21062 --- [ocp-schedules-11,7f58daadad97b934,75ef24d5ef109394] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":319,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":778} 2024-11-04 14:02:28.572 INFO 21062 --- [ocp-schedules-11,7f58daadad97b934,75ef24d5ef109394] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:02:28.578 INFO 21062 --- [pool-12-thread-10,c0a7ef3a66c9904f,71ec189e77889913] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:02:28.578 INFO 21062 --- [pool-12-thread-11,c0a7ef3a66c9904f,fe8022bec7a409ba] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:02:28.578 INFO 21062 --- [pool-12-thread-10,c0a7ef3a66c9904f,71ec189e77889913] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:02:28.578 INFO 21062 --- [pool-12-thread-12,c0a7ef3a66c9904f,a6cdc4ff13dbd06b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:02:28.578 INFO 21062 --- [pool-12-thread-9,c0a7ef3a66c9904f,be790c06ad3b01e4] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:02:28.583 INFO 21062 --- [ForkJoinPool-11-worker-10,c0a7ef3a66c9904f,a82a7414e877b36e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:02:28.583 INFO 21062 --- [ForkJoinPool-11-worker-3,c0a7ef3a66c9904f,42fae5423ff45336] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:02:28.583 INFO 21062 --- [ForkJoinPool-11-worker-10,c0a7ef3a66c9904f,a82a7414e877b36e] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:02:28.584 INFO 21062 --- [ForkJoinPool-11-worker-10,c0a7ef3a66c9904f,a82a7414e877b36e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:02:28.584 INFO 21062 --- [pool-12-thread-12,c0a7ef3a66c9904f,a6cdc4ff13dbd06b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:02:28.584 INFO 21062 --- [pool-12-thread-9,c0a7ef3a66c9904f,be790c06ad3b01e4] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:02:28.585 INFO 21062 --- [pool-12-thread-10,c0a7ef3a66c9904f,71ec189e77889913] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:02:28.586 INFO 21062 --- [pool-12-thread-11,c0a7ef3a66c9904f,fe8022bec7a409ba] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:02:28.586 INFO 21062 --- [pool-12-thread-11,c0a7ef3a66c9904f,fe8022bec7a409ba] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:02:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:02:28.590354+08:00, lastProcessTime=2024-11-04T14:02:18.591840+08:00 2024-11-04 14:02:28.601 INFO 21062 --- [pool-12-thread-10,c0a7ef3a66c9904f,71ec189e77889913] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:02:28.636 INFO 21062 --- [ocp-async-2,2c6a2b858b3302b3,cbf0958c153101bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:02:28.696 INFO 21062 --- [ocp-schedules-3,ccf486bebea9a673,e39dbcb5bf4eaae3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:02:28.736 INFO 21062 --- [ocp-schedules-1,99fbe746e1c07836,65ea64b46013c458] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:02:28.737 INFO 21062 --- [ocp-schedules-1,99fbe746e1c07836,65ea64b46013c458] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:02:28.776 INFO 21062 --- [ocp-schedules-19,80b190c4380ff659,dfa5daca42a64257] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 14:02:28.777 INFO 21062 --- [ocp-schedules-19,80b190c4380ff659,dfa5daca42a64257] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 14:02:28.777 INFO 21062 --- [ocp-schedules-19,80b190c4380ff659,dfa5daca42a64257] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=392 2024-11-04 14:02:28.777 INFO 21062 --- [ocp-schedules-6,5571e3d63fdd0443,a97aa99a1820a558] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:02:28.779 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:02:28.590354+08:00 2024-11-04 14:02:28.794 INFO 21062 --- [ocp-schedules-19,80b190c4380ff659,dfa5daca42a64257] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=392 2024-11-04 14:02:28.794 INFO 21062 --- [ocp-schedules-23,6cd5f2b9c392b335,514667ba91483ed2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:02:28.806 INFO 21062 --- [ocp-schedules-23,6cd5f2b9c392b335,514667ba91483ed2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 14:02:28.973 INFO 21062 --- [ocp-schedules-30,e9bace89a470b232,4eb27fd228a32d36] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 14:02:28.977 INFO 21062 --- [ocp-schedules-30,e9bace89a470b232,4eb27fd228a32d36] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:28.996 INFO 21062 --- [ocp-schedules-2,df2cf8b71239a1eb,7fc3f1841012ce81] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:02:28.996 INFO 21062 --- [ocp-schedules-2,df2cf8b71239a1eb,7fc3f1841012ce81] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:02:29.017 ERROR 21062 --- [ocp-schedules-30,e9bace89a470b232,4eb27fd228a32d36] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:02:29.017 INFO 21062 --- [ocp-schedules-30,e9bace89a470b232,4eb27fd228a32d36] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 14:02:29.043 INFO 21062 --- [ocp-schedules-20,badb435852e3abfc,27a088f66cb60a37] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:02:29.064 WARN 21062 --- [ocp-async-3,1eb18b54047704fd,e8e2d7a79039f63f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:29.079 INFO 21062 --- [ocp-schedules-11,b9109763103a829e,d6dfe30c57cc342f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:02:29.084 INFO 21062 --- [ocp-schedules-11,b9109763103a829e,d6dfe30c57cc342f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:02:29.084 INFO 21062 --- [ocp-schedules-11,b9109763103a829e,d6dfe30c57cc342f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:02:29.100 INFO 21062 --- [ocp-async-6,47e0de31b0858d91,061c70a55e800bb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:29.364 INFO 21062 --- [ocp-schedules-16,98500f05d7bb9501,85f00b0de2088ab4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:02:29.364 INFO 21062 --- [ocp-schedules-16,98500f05d7bb9501,85f00b0de2088ab4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:02:29.365 INFO 21062 --- [ocp-schedules-16,98500f05d7bb9501,85f00b0de2088ab4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:02:29.365 INFO 21062 --- [ocp-schedules-16,98500f05d7bb9501,85f00b0de2088ab4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:02:29.368 INFO 21062 --- [ocp-schedules-16,98500f05d7bb9501,85f00b0de2088ab4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:02:29.368 INFO 21062 --- [ocp-schedules-16,98500f05d7bb9501,85f00b0de2088ab4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:02:29.520 INFO 21062 --- [ocp-schedules-28,1586a8e695e94ade,5d2fa6319f847d5e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:02:29.520 INFO 21062 --- [ocp-schedules-28,1586a8e695e94ade,5d2fa6319f847d5e] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:02:29.548 INFO 21062 --- [pool-37-thread-1,1586a8e695e94ade,e044eaf348748808] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:02:29.576 WARN 21062 --- [ocp-schedules-28,1586a8e695e94ade,5d2fa6319f847d5e] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:02:29.576 INFO 21062 --- [ocp-schedules-28,1586a8e695e94ade,5d2fa6319f847d5e] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:02:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6aebfb6111a5686b,9e5244606c750776] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:30.107 INFO 21062 --- [ocp-schedules-5,ecc3ca5296fc9612,1021aaa65fa13708] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:02:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:02:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:02:30.959 INFO 21062 --- [ocp-schedules-16,31edcb610864527e,29f4842917decbca] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:02:30.965 INFO 21062 --- [ocp-schedules-16,31edcb610864527e,29f4842917decbca] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:02:30.965 INFO 21062 --- [ocp-schedules-16,31edcb610864527e,29f4842917decbca] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:02:30.977 INFO 21062 --- [ocp-schedules-23,df590fbc647cd7b3,328f3f0a6aa85f58] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:02:30.977 INFO 21062 --- [ocp-schedules-23,df590fbc647cd7b3,328f3f0a6aa85f58] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:02:31.003 INFO 21062 --- [ocp-schedules-23,df590fbc647cd7b3,328f3f0a6aa85f58] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:02:31.003 INFO 21062 --- [ocp-schedules-23,df590fbc647cd7b3,328f3f0a6aa85f58] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:02:31.003 INFO 21062 --- [ocp-schedules-19,cc1483da18bde756,bdf0b0206caaa8f2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:02:31.064 WARN 21062 --- [ocp-async-6,cc88248692f7fd40,be5b05442f2fcdda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:31.100 INFO 21062 --- [ocp-async-4,0a8008679569c215,15c4da3a763712b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:31.160 INFO 21062 --- [ocp-schedules-9,6a1485a2dd8a485b,af8cbf71896117c1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:02:31.160+08:00, clusters=[1] 2024-11-04 14:02:31.161 INFO 21062 --- [ocp-schedules-9,6a1485a2dd8a485b,af8cbf71896117c1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:02:31.240 INFO 21062 --- [ocp-schedules-30,8bf6af22795caf66,f9c094c74d67c3a7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:02:31.240 INFO 21062 --- [ocp-schedules-30,8bf6af22795caf66,f9c094c74d67c3a7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:02:31.240 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:02:31.240 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:02:31.251 INFO 21062 --- [ocp-schedules-30,8bf6af22795caf66,f9c094c74d67c3a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:02:31.252 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:02:31.252 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:02:31.253 WARN 21062 --- [ocp-schedules-30,8bf6af22795caf66,f9c094c74d67c3a7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:02:31.253 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:02:31.253 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:02:31.254 INFO 21062 --- [ocp-schedules-30,8bf6af22795caf66,f9c094c74d67c3a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:02:31.254 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:02:31.254 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:02:31.255 WARN 21062 --- [ocp-schedules-30,8bf6af22795caf66,f9c094c74d67c3a7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:02:31.255 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:02:31.255 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:02:31.262 ERROR 21062 --- [ocp-schedules-30,8bf6af22795caf66,f9c094c74d67c3a7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 14:02:31.270 INFO 21062 --- [ocp-schedules-5,f5313d552d669e68,b2eb65a47836c689] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:02:31.274 INFO 21062 --- [ocp-schedules-5,f5313d552d669e68,b2eb65a47836c689] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:31.314 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:31.315 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:31.317 INFO 21062 --- [ocp-schedules-11,4c97cb3745bbb961,6a795f7e8e46d8f0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:02:31.316+08:00, clusters=[1] 2024-11-04 14:02:31.320 INFO 21062 --- [ocp-schedules-11,4c97cb3745bbb961,6a795f7e8e46d8f0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:02:31.328 ERROR 21062 --- [ocp-schedules-5,f5313d552d669e68,b2eb65a47836c689] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:02:31.328 INFO 21062 --- [ocp-schedules-5,f5313d552d669e68,b2eb65a47836c689] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:02:32.983 INFO 21062 --- [prometheus-minute-schedule-1,288e304e6c9621b5,83b0901dd4843768] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:33.064 WARN 21062 --- [ocp-async-4,ddd2e65bb6d6e1dc,3be91a337bdc3db9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:33.101 INFO 21062 --- [ocp-async-9,e83ec8a4fc5ece66,5f01366c33d62a8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:34.315 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:34.316 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:34.674 INFO 21062 --- [ocp-schedules-12,7c12955f1fe4eb85,5d52328b95476496] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:02:34.674 INFO 21062 --- [ocp-schedules-12,135f3130855d0603,dea070ff25f04cd2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:02:34.678 INFO 21062 --- [ocp-schedules-12,135f3130855d0603,dea070ff25f04cd2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:34.711 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:02:34.711 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:02:35.064 WARN 21062 --- [ocp-async-9,f4d1665123284511,3b8a3b2b222c8625] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:35.101 INFO 21062 --- [ocp-async-5,57372fe371f34196,dd66e24fee53dc48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b9c70887dffb97ef,0af42f446628b7fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:36.821 INFO 21062 --- [ocp-schedules-28,6a666888c9ef6bac,e2e279e5798f4910] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:02:36.825 INFO 21062 --- [ocp-schedules-28,6a666888c9ef6bac,e2e279e5798f4910] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:36.911 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:02:36.913 ERROR 21062 --- [ocp-schedules-28,6a666888c9ef6bac,e2e279e5798f4910] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:02:36.913 INFO 21062 --- [ocp-schedules-28,6a666888c9ef6bac,e2e279e5798f4910] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:02:37.064 WARN 21062 --- [ocp-async-5,2d0f913efbdb11d4,69f97bde22c928ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:37.101 INFO 21062 --- [ocp-async-2,eb2695b0be016139,614f5eee58810f39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:37.316 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:37.317 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:38.505 INFO 21062 --- [ocp-schedules-13,5381d4a9698719bc,856dfb5d28f1507c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:02:38.505 INFO 21062 --- [ocp-schedules-21,678c7d6cefd51f2e,0d28c8bebb067b63] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:02:38.508 INFO 21062 --- [ocp-schedules-14,f14f2ceac687bc40,381d2f9dce26aaf3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:02:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:02:38.590343+08:00, lastProcessTime=2024-11-04T14:02:28.590354+08:00 2024-11-04 14:02:38.636 INFO 21062 --- [ocp-async-5,4c51d9ba75582354,b06af3f729c77cf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:02:38.741 INFO 21062 --- [ocp-schedules-11,46173af3aca30db6,593b31574e2069e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:02:38.781 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:02:38.590343+08:00 2024-11-04 14:02:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c3a1479a88bc27e8,d22f6c054a8c861d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:39.064 WARN 21062 --- [ocp-async-10,8310fb76e1397962,ecd04d3dfb4517d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:39.101 INFO 21062 --- [ocp-async-3,6eadc72df476f714,79a2f82242109dc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:02:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:02:40.109 INFO 21062 --- [ocp-schedules-10,7536e10f03383408,0d74394cbf18cedf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:02:40.317 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:40.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:41.065 WARN 21062 --- [ocp-async-3,07e75169f9508377,92544ca163c429ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:41.101 INFO 21062 --- [ocp-async-6,e8a16350a3496b10,f7ddb14b8beee305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:02:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:02:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 14:02:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c003f9c13d4bb06d,e9a1462113b29ee1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:43.065 WARN 21062 --- [ocp-async-6,f506ccef6219a428,50b5dc42954701b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:43.102 INFO 21062 --- [ocp-async-4,235bdef76cdf0005,64f9c9d810a84b23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:43.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:43.319 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9d7030fa8c1ceeb6,35402347856fd9be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:45.065 WARN 21062 --- [ocp-async-4,4048beabcebb6f10,e51c25896951d535] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:45.102 INFO 21062 --- [ocp-async-9,c5a3a86d718e43a5,c24b46d7d8a9b27f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:46.319 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:46.320 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:47.065 WARN 21062 --- [ocp-async-9,2d1c7aaf6655bc2a,03cf8f1f1d7ed93d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:47.102 INFO 21062 --- [ocp-async-5,0c6cc8d2d0dbec88,d89a23c3e4198512] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:47.983 INFO 21062 --- [prometheus-minute-schedule-1,9419b5812980649a,bee01eb8e164af7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:02:48.591314+08:00, lastProcessTime=2024-11-04T14:02:38.590343+08:00 2024-11-04 14:02:48.636 INFO 21062 --- [ocp-async-1,627ec61a29f76da0,e98e3fdab5817c87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:02:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:02:48.591314+08:00 2024-11-04 14:02:48.778 INFO 21062 --- [ocp-schedules-29,658992165056c4ce,5bb3542c7f088c89] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:02:49.065 WARN 21062 --- [ocp-async-6,7894316b2dd1631e,0147dea178a57b2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:49.102 INFO 21062 --- [ocp-async-2,e1d5bb9bb4388b51,eff9d2980f1d4c30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:49.320 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:49.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:50.112 INFO 21062 --- [ocp-schedules-11,23bdbb5954f709c1,8046ed5a06584075] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:02:50.968 INFO 21062 --- [ocp-schedules-27,571a2caf2f63b53d,a6ca64bdba02689a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:02:50.975 INFO 21062 --- [ocp-schedules-27,571a2caf2f63b53d,a6ca64bdba02689a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:02:50.975 INFO 21062 --- [ocp-schedules-27,571a2caf2f63b53d,a6ca64bdba02689a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:02:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1a1e171623e41a4b,44e0f9873268f805] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:51.066 WARN 21062 --- [ocp-async-2,859d59900034c052,e66b477631990b49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:51.102 INFO 21062 --- [ocp-async-1,308e8a0952bbfa8d,958ae710abc1f6ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:52.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:52.322 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:53.066 WARN 21062 --- [ocp-async-1,bec88deea05f0cdc,32bac11573162861] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:53.102 INFO 21062 --- [ocp-async-7,41dc7e88c869615f,dc7b150a4b43e2c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:53.983 INFO 21062 --- [prometheus-minute-schedule-1,751c250d343f8a86,a7325d4e694be2d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:55.066 WARN 21062 --- [ocp-async-7,94300cfe5c82235c,fd32a85e742b6427] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:55.102 INFO 21062 --- [ocp-async-8,b1acb729fc668c79,bcea020c8b4578ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:55.322 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:55.323 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f2f995493a1fdad4,6bf88de70819d6d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:02:57.066 WARN 21062 --- [ocp-async-4,b82c0be60536d76f,f27195e4d8a6571b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:57.103 INFO 21062 --- [ocp-async-10,a4563735fc63e628,816f6ae505b58aa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:58.323 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:02:58.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:02:58.500 INFO 21062 --- [ocp-schedules-15,dc872e946bfcc8a9,48173fab9d75deea] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:02:58.500 INFO 21062 --- [ocp-schedules-4,4857650ece31d1b7,357275eb2eee6c49] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:02:58.503 INFO 21062 --- [ocp-schedules-8,b2ae786296cccc3e,553c260dc814cb86] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:02:58.504 INFO 21062 --- [ocp-schedules-15,dc872e946bfcc8a9,48173fab9d75deea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:58.504 INFO 21062 --- [ocp-schedules-4,4857650ece31d1b7,357275eb2eee6c49] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:02:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:02:58.505 INFO 21062 --- [ocp-schedules-28,de4b79b9626ab9a0,ae0b68e732e6b9b8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:02:58.511 INFO 21062 --- [ocp-schedules-24,ca4196b3c01a6ba7,3a8c07f520913652] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:02:58.511 INFO 21062 --- [ocp-schedules-6,ecb268207d19cae7,867c28b12cd9b7f1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:02:58.511 INFO 21062 --- [ocp-schedules-24,ca4196b3c01a6ba7,3a8c07f520913652] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:02:58.511 INFO 21062 --- [ocp-schedules-6,ecb268207d19cae7,867c28b12cd9b7f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:02:58.512 INFO 21062 --- [ocp-schedules-16,f3cf40bfc11d0663,940b66a9bb322537] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:02:58.512 INFO 21062 --- [ocp-schedules-16,f3cf40bfc11d0663,940b66a9bb322537] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:02:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:02:58.591812+08:00, lastProcessTime=2024-11-04T14:02:48.591314+08:00 2024-11-04 14:02:58.636 INFO 21062 --- [ocp-async-8,0193036354049801,22abaf621674767b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:02:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:02:58.591812+08:00 2024-11-04 14:02:58.826 INFO 21062 --- [ocp-schedules-3,e4469da4d94f9174,d2784aeb04c41b60] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:02:59.046 INFO 21062 --- [ocp-schedules-14,9703e1de5c77dd49,39d970af0af11c41] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:02:59.066 WARN 21062 --- [ocp-async-1,3926d10bfe623541,57dffb8c7c1f184d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:02:59.103 INFO 21062 --- [ocp-async-5,84590e30c24f6659,5e071f349e00d71b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:02:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f358dc91f2fd8dbe,5cb2830b1d0a9bab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700180, interval=60 2024-11-04 14:03:00.115 INFO 21062 --- [ocp-schedules-2,885341c51a37f80b,e0d3385f96c56bdc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:03:01.066 WARN 21062 --- [ocp-async-7,efa3b67c78e21143,cb6007200197970b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:01.103 INFO 21062 --- [ocp-async-8,8e5c22f8318a7912,7d72d8d05c643af0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:01.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:01.325 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2d7d1daddb7ad62c,82a45747d62fb0c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:03.067 WARN 21062 --- [ocp-async-4,2825e06d977a89a7,23d5143d35cb5f8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:03.103 INFO 21062 --- [ocp-async-10,ed694718c485eaee,263bfe6eb3b9e572] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:04.189 INFO 21062 --- [metric-archive-minute-29,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=31, affectRows=31 2024-11-04 14:03:04.189 INFO 21062 --- [metric-archive-minute-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=31, validCount=31 2024-11-04 14:03:04.194 INFO 21062 --- [metric-archive-minute-26,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=4, affectRows=4 2024-11-04 14:03:04.194 INFO 21062 --- [metric-archive-minute-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=4, validCount=4 2024-11-04 14:03:04.204 INFO 21062 --- [metric-archive-minute-20,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=23, affectRows=23 2024-11-04 14:03:04.205 INFO 21062 --- [metric-archive-minute-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=23, validCount=23 2024-11-04 14:03:04.205 INFO 21062 --- [metric-archive-minute-14,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=30, affectRows=30 2024-11-04 14:03:04.205 INFO 21062 --- [metric-archive-minute-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=30, validCount=30 2024-11-04 14:03:04.209 INFO 21062 --- [metric-archive-minute-52,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=27, affectRows=27 2024-11-04 14:03:04.209 INFO 21062 --- [metric-archive-minute-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=27, validCount=27 2024-11-04 14:03:04.214 INFO 21062 --- [metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=17, affectRows=17 2024-11-04 14:03:04.214 INFO 21062 --- [metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=17, validCount=17 2024-11-04 14:03:04.218 INFO 21062 --- [metric-archive-minute-27,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=7, affectRows=7 2024-11-04 14:03:04.218 INFO 21062 --- [metric-archive-minute-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=7, validCount=7 2024-11-04 14:03:04.218 INFO 21062 --- [metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=11, affectRows=11 2024-11-04 14:03:04.218 INFO 21062 --- [metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=11, validCount=11 2024-11-04 14:03:04.219 INFO 21062 --- [metric-archive-minute-35,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=23, affectRows=23 2024-11-04 14:03:04.219 INFO 21062 --- [metric-archive-minute-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=23, validCount=23 2024-11-04 14:03:04.221 INFO 21062 --- [metric-archive-minute-10,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=27, affectRows=27 2024-11-04 14:03:04.221 INFO 21062 --- [metric-archive-minute-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=27, validCount=27 2024-11-04 14:03:04.226 INFO 21062 --- [metric-archive-minute-37,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=27, affectRows=27 2024-11-04 14:03:04.226 INFO 21062 --- [metric-archive-minute-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=27, validCount=27 2024-11-04 14:03:04.228 INFO 21062 --- [metric-archive-minute-36,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=27, affectRows=27 2024-11-04 14:03:04.228 INFO 21062 --- [metric-archive-minute-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=27, validCount=27 2024-11-04 14:03:04.228 INFO 21062 --- [metric-archive-minute-43,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=10, affectRows=10 2024-11-04 14:03:04.228 INFO 21062 --- [metric-archive-minute-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=10, validCount=10 2024-11-04 14:03:04.230 INFO 21062 --- [metric-archive-minute-54,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=23, affectRows=23 2024-11-04 14:03:04.230 INFO 21062 --- [metric-archive-minute-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=23, validCount=23 2024-11-04 14:03:04.234 INFO 21062 --- [metric-archive-minute-15,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=20, affectRows=20 2024-11-04 14:03:04.234 INFO 21062 --- [metric-archive-minute-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=20, validCount=20 2024-11-04 14:03:04.235 INFO 21062 --- [metric-archive-minute-46,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=6, affectRows=6 2024-11-04 14:03:04.235 INFO 21062 --- [metric-archive-minute-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=6, validCount=6 2024-11-04 14:03:04.235 INFO 21062 --- [metric-archive-minute-21,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=2, affectRows=2 2024-11-04 14:03:04.235 INFO 21062 --- [metric-archive-minute-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=2, validCount=2 2024-11-04 14:03:04.236 INFO 21062 --- [metric-archive-minute-49,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=21, affectRows=21 2024-11-04 14:03:04.236 INFO 21062 --- [metric-archive-minute-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=21, validCount=21 2024-11-04 14:03:04.240 INFO 21062 --- [metric-archive-minute-44,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=27, affectRows=27 2024-11-04 14:03:04.240 INFO 21062 --- [metric-archive-minute-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=27, validCount=27 2024-11-04 14:03:04.242 INFO 21062 --- [metric-archive-minute-42,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=26, affectRows=26 2024-11-04 14:03:04.242 INFO 21062 --- [metric-archive-minute-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730696400, nodeCount=26, validCount=26 2024-11-04 14:03:04.325 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:04.325 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:04.712 INFO 21062 --- [ocp-schedules-12,135f3130855d0603,dea070ff25f04cd2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:03:04.712 INFO 21062 --- [ocp-schedules-12,bf2f8272b518e7ac,00d73da333b5c68a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:03:04.716 INFO 21062 --- [ocp-schedules-12,bf2f8272b518e7ac,00d73da333b5c68a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:04.747 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:03:04.747 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:03:05.067 WARN 21062 --- [ocp-async-9,417717e470aefb6d,2bf075c6ead8a21d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:05.103 INFO 21062 --- [ocp-async-3,97a3225eb3e1e0f7,403d3ab5ec8e4dcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:05.983 INFO 21062 --- [prometheus-minute-schedule-1,55f3e6559034f104,c93b19a4a8351886] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:07.067 WARN 21062 --- [ocp-async-5,0c9eab06d70a6357,75142836df095461] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:07.104 INFO 21062 --- [ocp-async-6,879cc6354a9b6506,edae221896611936] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:07.326 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:07.326 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:08.505 INFO 21062 --- [ocp-schedules-4,4857650ece31d1b7,357275eb2eee6c49] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:03:08.505 INFO 21062 --- [ocp-schedules-15,dc872e946bfcc8a9,48173fab9d75deea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:03:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:03:08.589945+08:00, lastProcessTime=2024-11-04T14:02:58.591812+08:00 2024-11-04 14:03:08.636 INFO 21062 --- [ocp-async-1,c554041ab30a3523,8b3c0b557735fa6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:03:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:03:08.589945+08:00 2024-11-04 14:03:08.858 INFO 21062 --- [ocp-schedules-29,4de6661ef4998bad,565f1b14036a20f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:03:08.983 INFO 21062 --- [prometheus-minute-schedule-1,dfe2c84e087d3e6f,07585fc13bcea1aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:09.067 WARN 21062 --- [ocp-async-4,acc7ef912aa9321f,e823373b64c0fd1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:09.104 INFO 21062 --- [ocp-async-10,6f1f10c64eb86e57,2e54b8a9f5b7ccb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:03:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:03:10.117 INFO 21062 --- [ocp-schedules-23,4bb8c4b85a42027a,ff3191abdcb35b87] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:03:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:03:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:03:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:03:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:03:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:03:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:03:10.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:10.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:10.978 INFO 21062 --- [ocp-schedules-6,7c1691a9697a7d69,282c9ae2fd72f3cc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:03:10.983 INFO 21062 --- [ocp-schedules-6,7c1691a9697a7d69,282c9ae2fd72f3cc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:03:10.983 INFO 21062 --- [ocp-schedules-6,7c1691a9697a7d69,282c9ae2fd72f3cc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:03:11.067 WARN 21062 --- [ocp-async-9,94d2b8db4786d742,b5fadc9c14c9f784] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:11.104 INFO 21062 --- [ocp-async-1,ca3169e63cc96bfc,0e1beb0f6f2896f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:11.436 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=1, affectRows=1 2024-11-04 14:03:11.436 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=1, validCount=1 2024-11-04 14:03:11.437 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=12, affectRows=12 2024-11-04 14:03:11.437 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=12, validCount=12 2024-11-04 14:03:11.437 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=9, affectRows=9 2024-11-04 14:03:11.437 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=9, validCount=9 2024-11-04 14:03:11.437 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=45, affectRows=45 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=45, validCount=45 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=38, affectRows=38 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=52, affectRows=52 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=27, affectRows=27 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=38, validCount=38 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=27, validCount=27 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=65, affectRows=65 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=52, validCount=52 2024-11-04 14:03:11.438 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=64, validCount=64 2024-11-04 14:03:11.439 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=64, affectRows=64 2024-11-04 14:03:11.439 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=56, affectRows=56 2024-11-04 14:03:11.439 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700120, nodeCount=23, affectRows=23 2024-11-04 14:03:11.439 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=63, validCount=63 2024-11-04 14:03:11.439 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=23, validCount=23 2024-11-04 14:03:11.439 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700120, nodeCount=55, validCount=55 2024-11-04 14:03:11.983 INFO 21062 --- [prometheus-minute-schedule-1,bfab87adfad0b6d1,f0dc8b97facfdfaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:13.067 WARN 21062 --- [ocp-async-5,cc0825907f4486fc,030fa962621e66ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:13.104 INFO 21062 --- [ocp-async-6,0e2ac69accf4b828,62dc3d03cf831c57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:13.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:13.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:13.984 INFO 21062 --- [prometheus-cache-monitor1,727639fa7c013143,fc7920126adeeaa7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:03:13.984 INFO 21062 --- [prometheus-cache-monitor1,727639fa7c013143,fc7920126adeeaa7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:03:14.177 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T13:57:14.154, end=2024-11-04T14:02:14.174} 2024-11-04 14:03:14.177 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T13:57:14.153, end=2024-11-04T14:02:14.174} 2024-11-04 14:03:14.182 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:03:14.182 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:03:14.182 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:03:14.182 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:03:14.865 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T13:59:14.842, end=2024-11-04T14:01:14.862} 2024-11-04 14:03:14.869 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:03:14.869 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:03:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2e8f0884b6568932,61d237e79b8edb6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:15.068 WARN 21062 --- [ocp-async-8,3287b19a32ea88f4,01043c9c59588725] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:15.104 INFO 21062 --- [ocp-async-2,f0e6864be851e067,b1834cce6c82a31a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:16.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:16.330 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:17.068 WARN 21062 --- [ocp-async-10,cf12c1dbf47a12cc,f16daf30aadba60e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:17.104 INFO 21062 --- [ocp-async-3,6deaef7f6f94ff82,5faa9c771ff35ab1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:17.983 INFO 21062 --- [prometheus-minute-schedule-1,28793d0fc0f6e1aa,e7fa61886cfcf29d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:03:18.589888+08:00, lastProcessTime=2024-11-04T14:03:08.589945+08:00 2024-11-04 14:03:18.636 INFO 21062 --- [ocp-async-4,a0bd0faf90c26a06,53e90f037a4c516e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:03:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:03:18.589888+08:00 2024-11-04 14:03:18.901 INFO 21062 --- [ocp-schedules-30,3789346861e6ab4d,cf9f45e058a1a56d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:03:18.927 INFO 21062 --- [ocp-schedules-9,feb03763cf8d1309,b2aa6c6ee8c2a7fb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:03:18.931 INFO 21062 --- [ocp-schedules-9,feb03763cf8d1309,b2aa6c6ee8c2a7fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:18.931 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:03:19.068 WARN 21062 --- [ocp-async-5,4d4194523ce9db67,b7d3fae3a1e90235] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:19.104 INFO 21062 --- [ocp-async-6,56ae58926f104f07,18215aea0908a13d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:19.330 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:19.330 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:20.120 INFO 21062 --- [ocp-schedules-17,a3070493cd4c3caa,0656c0aab1c4b357] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:03:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a8c9ae0bd73819f2,5376af7e6e4f0884] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:21.068 WARN 21062 --- [ocp-async-8,f55df0124eab6c8e,96e080eb56072242] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:21.105 INFO 21062 --- [ocp-async-2,77f918202e4e1b32,b0b9ba17a9169881] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:22.331 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:22.331 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:23.068 WARN 21062 --- [ocp-async-10,8af6755d0e5f66d4,d4271a4a36424f55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:23.105 INFO 21062 --- [ocp-async-3,75eac94d93b42a42,877ee6620d02a563] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4a2d1ccc61d4f992,1c08d721918da59e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:25.069 WARN 21062 --- [ocp-async-1,4b66fc36700afb98,b2b3c10b9efb574f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:25.105 INFO 21062 --- [ocp-async-7,82733dc6e60bdddd,4259921d6333b4ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:25.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:25.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3ee0ec8ed763edcd,980889985dd0ae42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:26.984 INFO 21062 --- [prometheus-hour-schedule-1,17f9b95d20722925,2f5cb30890853f75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:03:27.069 WARN 21062 --- [ocp-async-7,46c316046fb01a0c,d9f3bbbe26485a26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:27.105 INFO 21062 --- [ocp-async-4,2ad27b8d9ef100b9,79bb36e66f089c2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:28.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:28.333 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:28.498 INFO 21062 --- [ocp-schedules-4,eeaf9a3d88612765,43f3192b99cf10a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:03:28.499 INFO 21062 --- [ocp-schedules-21,ea52e98c9b9b8258,33f23f9ba7cf0941] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:03:28.500 INFO 21062 --- [ocp-schedules-14,2ca676bc15e1d92c,54fa32fb4757404d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:03:28.500 INFO 21062 --- [ocp-schedules-2,59208ee0c0bb7eef,212cf3e70981b7c9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:03:28.503 INFO 21062 --- [ocp-schedules-22,3d80bb820ecf2bff,d3bcc72857e8be19] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:03:28.504 INFO 21062 --- [ocp-schedules-4,eeaf9a3d88612765,43f3192b99cf10a2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:03:28.504 INFO 21062 --- [ocp-schedules-14,2ca676bc15e1d92c,54fa32fb4757404d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:03:28.505 INFO 21062 --- [ocp-schedules-1,a751a1a5bd19059a,59aed9cb6202db3f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:03:28.506 INFO 21062 --- [ocp-schedules-21,ea52e98c9b9b8258,33f23f9ba7cf0941] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:03:28.507 INFO 21062 --- [ocp-schedules-2,59208ee0c0bb7eef,212cf3e70981b7c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:28.511 INFO 21062 --- [ocp-schedules-18,d3b7038e1c7395a4,78435a48aa26fa4c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:03:28.511 INFO 21062 --- [ocp-schedules-15,3d061a54f3f83f14,4934855c9be09fa9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:03:28.511 INFO 21062 --- [ocp-schedules-18,d3b7038e1c7395a4,78435a48aa26fa4c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:03:28.511 INFO 21062 --- [ocp-schedules-15,3d061a54f3f83f14,4934855c9be09fa9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:03:28.511 INFO 21062 --- [ocp-schedules-24,7596e0928a66688a,a3baf601321f6c1a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:03:28.511 INFO 21062 --- [ocp-schedules-24,7596e0928a66688a,a3baf601321f6c1a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:03:28.571 INFO 21062 --- [ocp-schedules-16,2c8b41033f87eb9c,69a9fa8f2e7e909c] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 14:03:28.571 INFO 21062 --- [ocp-schedules-16,2c8b41033f87eb9c,69a9fa8f2e7e909c] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 14:03:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:03:28.589952+08:00, lastProcessTime=2024-11-04T14:03:18.589888+08:00 2024-11-04 14:03:28.617 INFO 21062 --- [ocp-schedules-27,b97682ca80f72e46,8dc7c242cbbed77c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:03:28.618 INFO 21062 --- [ocp-schedules-27,b97682ca80f72e46,8dc7c242cbbed77c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:03:28.637 INFO 21062 --- [ocp-async-8,2104cc5ef6f3e662,a009c62e2ac9d1df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:03:28.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:03:28.589952+08:00 2024-11-04 14:03:28.779 INFO 21062 --- [ocp-schedules-6,a1976e27d60a0020,5dbc3fcf315cd47f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:03:28.931 INFO 21062 --- [ocp-schedules-9,feb03763cf8d1309,b2aa6c6ee8c2a7fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:03:28.932 INFO 21062 --- [ocp-schedules-9,98d8dd1c079aefbd,0933d77131ef2507] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:03:29.000 INFO 21062 --- [ocp-schedules-18,8b7730b2cbaa9aa8,9a2bd4600f775b95] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:03:29.000 INFO 21062 --- [ocp-schedules-18,8b7730b2cbaa9aa8,9a2bd4600f775b95] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:03:29.048 INFO 21062 --- [ocp-schedules-15,3528b2e04de62937,a3ca3d7d243ed674] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:03:29.069 WARN 21062 --- [ocp-async-9,6bc361141cc7e3fe,8e5ae737ccba24bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:29.084 INFO 21062 --- [ocp-schedules-3,abac7df3bd67be53,d892cab655ade930] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:03:29.090 INFO 21062 --- [ocp-schedules-3,abac7df3bd67be53,d892cab655ade930] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:03:29.090 INFO 21062 --- [ocp-schedules-3,abac7df3bd67be53,d892cab655ade930] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:03:29.106 INFO 21062 --- [ocp-async-1,4afc78f9c76b7ac9,c05d065392c0352b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:29.184 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:03:29.184 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:03:29.193 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:03:29.193 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:03:29.193 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:03:29.193 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:03:29.200 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:03:29.200 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:03:29.200 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:03:29.200 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:03:29.237 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:03:29.246 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:03:29.246 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:03:29.246 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:03:29.246 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:03:29.246 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:03:29.255 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:03:29.368 INFO 21062 --- [ocp-schedules-7,2953325092f498ed,db4d776f6163f8b1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:03:29.368 INFO 21062 --- [ocp-schedules-7,2953325092f498ed,db4d776f6163f8b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:03:29.370 INFO 21062 --- [ocp-schedules-7,2953325092f498ed,db4d776f6163f8b1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:03:29.370 INFO 21062 --- [ocp-schedules-7,2953325092f498ed,db4d776f6163f8b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:03:29.370 INFO 21062 --- [ocp-schedules-7,2953325092f498ed,db4d776f6163f8b1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:03:29.370 INFO 21062 --- [ocp-schedules-7,2953325092f498ed,db4d776f6163f8b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:03:29.812 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:03:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e85359aef9bcd90b,5d62eb24a0206d9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:30.122 INFO 21062 --- [ocp-schedules-18,a85776ecf4d8b7a8,c951c7d102b4ab51] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:03:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:03:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:03:30.364 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.163933427S 2024-11-04 14:03:30.364 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:03:30.464 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:03:30.465 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:03:30.494 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:03:30.561 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:03:30.561 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:03:30.561 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:03:30.561 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:03:30.561 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:03:30.562 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:03:30.573 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:03:30.748 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] o.s.boot.SpringApplication : Started application in 1.555 seconds (JVM running for 5624.427) 2024-11-04 14:03:30.748 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.555620645S 2024-11-04 14:03:30.985 INFO 21062 --- [ocp-schedules-20,2188396685d628c5,535d5b908b7c2c0c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:03:30.991 INFO 21062 --- [ocp-schedules-20,2188396685d628c5,535d5b908b7c2c0c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:03:30.991 INFO 21062 --- [ocp-schedules-20,2188396685d628c5,535d5b908b7c2c0c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:03:31.003 INFO 21062 --- [ocp-schedules-13,bfc1c945c6bd04c5,c5534c552bedeaa5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:03:31.003 INFO 21062 --- [ocp-schedules-13,bfc1c945c6bd04c5,c5534c552bedeaa5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:03:31.031 INFO 21062 --- [ocp-schedules-28,61bd492f070fdf9e,3ed624f84d354445] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:03:31.032 INFO 21062 --- [ocp-schedules-13,bfc1c945c6bd04c5,c5534c552bedeaa5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:03:31.032 INFO 21062 --- [ocp-schedules-13,bfc1c945c6bd04c5,c5534c552bedeaa5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:03:31.069 WARN 21062 --- [ocp-async-1,a211efaa19a0d025,c195d2c49f9778bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:31.106 INFO 21062 --- [ocp-async-6,db8784cd3108998d,a5922d2d4f7d9d66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:31.188 INFO 21062 --- [ocp-schedules-8,a685b402d3b50b63,5f03839ad800b3fc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:03:31.188+08:00, clusters=[1] 2024-11-04 14:03:31.198 INFO 21062 --- [ocp-schedules-5,5a73a9a9bae88fc4,899413981641ccb1] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:03:31.204 INFO 21062 --- [ocp-schedules-8,a685b402d3b50b63,5f03839ad800b3fc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:03:31.271 INFO 21062 --- [ocp-schedules-6,5007645b7dbf6aa5,8afb0fbdad8fdcb1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:03:31.272 INFO 21062 --- [ocp-schedules-6,5007645b7dbf6aa5,8afb0fbdad8fdcb1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:03:31.272 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:03:31.272 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:03:31.282 INFO 21062 --- [ocp-schedules-6,5007645b7dbf6aa5,8afb0fbdad8fdcb1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:03:31.282 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:03:31.283 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:03:31.284 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:03:31.284 WARN 21062 --- [ocp-schedules-6,5007645b7dbf6aa5,8afb0fbdad8fdcb1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:03:31.285 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:03:31.285 INFO 21062 --- [ocp-schedules-6,5007645b7dbf6aa5,8afb0fbdad8fdcb1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:03:31.285 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:03:31.286 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:03:31.286 WARN 21062 --- [ocp-schedules-6,5007645b7dbf6aa5,8afb0fbdad8fdcb1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:03:31.286 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:03:31.287 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:03:31.292 ERROR 21062 --- [ocp-schedules-6,5007645b7dbf6aa5,8afb0fbdad8fdcb1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 14:03:31.333 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:31.334 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:31.349 INFO 21062 --- [ocp-schedules-3,26101fdbb6b8a39c,40a542b71ebc2fbd] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:03:31.349+08:00, clusters=[1] 2024-11-04 14:03:31.352 INFO 21062 --- [ocp-schedules-3,26101fdbb6b8a39c,40a542b71ebc2fbd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:03:32.983 INFO 21062 --- [prometheus-minute-schedule-1,bbc651d145ad9bd8,cff5f1111c13e3eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:33.069 WARN 21062 --- [ocp-async-6,737e8fd687f59497,03f96e17c072893e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:33.106 INFO 21062 --- [ocp-async-10,3c54d4c33a868968,c7e68c821fe5c43a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:34.334 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:34.335 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:34.747 INFO 21062 --- [ocp-schedules-12,bf2f8272b518e7ac,00d73da333b5c68a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:03:34.747 INFO 21062 --- [ocp-schedules-12,58197de8c16c2d47,032df50d80296c24] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:03:34.751 INFO 21062 --- [ocp-schedules-12,58197de8c16c2d47,032df50d80296c24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:34.781 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:03:34.781 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:03:35.070 WARN 21062 --- [ocp-async-10,1ab27ced64848c77,17f179e36056e151] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:35.106 INFO 21062 --- [ocp-async-3,4067679f70ed331a,19f5b8f6564fd1d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:35.983 INFO 21062 --- [prometheus-minute-schedule-1,2b780ac4e86131fe,394e312951165bea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:36.913 INFO 21062 --- [ocp-schedules-4,d1d825ea1d35e181,e6b1c5452a2cfe3e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:03:36.917 INFO 21062 --- [ocp-schedules-4,d1d825ea1d35e181,e6b1c5452a2cfe3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:37.006 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:03:37.011 ERROR 21062 --- [ocp-schedules-4,d1d825ea1d35e181,e6b1c5452a2cfe3e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:03:37.011 INFO 21062 --- [ocp-schedules-4,d1d825ea1d35e181,e6b1c5452a2cfe3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:03:37.070 WARN 21062 --- [ocp-async-3,0c70894dc388433c,b19c86d90919dddf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:37.106 INFO 21062 --- [ocp-async-8,8c286e3b374de30f,177ee4a1bb6bca99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:37.335 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:37.336 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:38.505 INFO 21062 --- [ocp-schedules-14,2ca676bc15e1d92c,54fa32fb4757404d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:03:38.508 INFO 21062 --- [ocp-schedules-2,59208ee0c0bb7eef,212cf3e70981b7c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:03:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:03:38.591389+08:00, lastProcessTime=2024-11-04T14:03:28.589952+08:00 2024-11-04 14:03:38.637 INFO 21062 --- [ocp-async-3,ece828d9e529be37,b84774f2c3dc0f97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:03:38.752 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:03:38.591389+08:00 2024-11-04 14:03:38.973 INFO 21062 --- [ocp-schedules-17,4165e44cc8a73237,be464e4a0076225f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:03:38.983 INFO 21062 --- [prometheus-minute-schedule-1,fd434fc165378172,df71ecb21a03fd44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:39.070 WARN 21062 --- [ocp-async-2,4e82d2f3aeee3278,6ccff2575391a99f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:39.106 INFO 21062 --- [ocp-async-9,82c41c4a32367364,b8bddb58b83945aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:03:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:03:40.125 INFO 21062 --- [ocp-schedules-15,20d9f00f0d2aaf4a,c2a382de88a00df8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:03:40.336 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:40.337 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:41.070 WARN 21062 --- [ocp-async-9,063ce86359e3009c,7243e4d946f6fc7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:41.107 INFO 21062 --- [ocp-async-1,74afb5c8642d10a1,c3fa2101a02dbc97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:41.983 INFO 21062 --- [prometheus-minute-schedule-1,844b27eadbae90a4,55cb1ce5f12298a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:43.070 WARN 21062 --- [ocp-async-1,03b86adf0fb1e422,e1ceb31314bca28b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:43.107 INFO 21062 --- [ocp-async-6,544d1affaa64e1c1,80a0f317c4a88761] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:43.337 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:43.339 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:44.983 INFO 21062 --- [prometheus-minute-schedule-1,77ca85c578648faa,c1f81d69bd736415] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:45.071 WARN 21062 --- [ocp-async-6,e2662b3d92de33f0,b2db46893acaa066] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:45.107 INFO 21062 --- [ocp-async-10,b4b7e705ecd86d27,30020f486c9ad19a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:46.339 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:46.340 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:47.071 WARN 21062 --- [ocp-async-10,4131ab87c9586592,c454697ec02c4f94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:47.107 INFO 21062 --- [ocp-async-5,384da78cf056f6e4,796b314ec52feb2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fe5a59fc9eb8710d,7c4edc6472095dc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:03:48.590558+08:00, lastProcessTime=2024-11-04T14:03:38.591389+08:00 2024-11-04 14:03:48.637 INFO 21062 --- [ocp-async-4,211a5cf96b7065df,b05e72f6998adea0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:03:48.799 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:03:48.590558+08:00 2024-11-04 14:03:48.999 INFO 21062 --- [ocp-schedules-4,ba4372ec47859126,36ad93d63b480571] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:03:49.071 WARN 21062 --- [ocp-async-1,9e9c3f5b5dba9906,05a8301356c16fa4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:49.107 INFO 21062 --- [ocp-async-8,33dc0e9802764d6d,f6bd8fc4a0c513d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:49.340 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:49.340 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:50.128 INFO 21062 --- [ocp-schedules-8,6b3af10387822dfa,bc7ecd568d7161cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:03:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8f4521cd2c38a5de,3b13a94bec5a9fd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:50.994 INFO 21062 --- [ocp-schedules-4,9fb68403f88dfb67,b8cd91a9e5b02c71] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:03:51.000 INFO 21062 --- [ocp-schedules-4,9fb68403f88dfb67,b8cd91a9e5b02c71] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:03:51.000 INFO 21062 --- [ocp-schedules-4,9fb68403f88dfb67,b8cd91a9e5b02c71] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:03:51.071 WARN 21062 --- [ocp-async-6,086a39cc52292270,b7c63d635c0e7d60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:51.107 INFO 21062 --- [ocp-async-4,0bcdcda791c81213,75cad5f7e338db75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:52.341 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:52.341 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:53.071 WARN 21062 --- [ocp-async-10,8de92802e7a2e7b7,40f4e4556083237f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:53.108 INFO 21062 --- [ocp-async-3,4f7c393ecacdd400,80f2275eaeb95b80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:53.983 INFO 21062 --- [prometheus-minute-schedule-1,62f78faedfcb3400,3229a88197df182f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:55.071 WARN 21062 --- [ocp-async-5,0be3ceb54523be64,33102d802fc3d64c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:55.108 INFO 21062 --- [ocp-async-7,213927bd8dddd1af,71ed2ec8c0ab0da7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:55.342 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:55.342 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d1a4692be43545af,ca2fcbe6967e876b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:03:57.072 WARN 21062 --- [ocp-async-8,d68867457ff63f98,12af48a70b3c85dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:57.108 INFO 21062 --- [ocp-async-2,9f7169c9f5d7d8e4,7cf7c0e3fb2d7fd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:58.342 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:03:58.343 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:03:58.499 INFO 21062 --- [ocp-schedules-18,dc9e8aefd953eeb0,60c8b283041618b8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:03:58.500 INFO 21062 --- [ocp-schedules-20,e26dac38b5cd521d,47e5a1744f45178d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:03:58.503 INFO 21062 --- [ocp-schedules-5,1a4c5966a2cfafcb,e04c89e787548ad6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:03:58.504 INFO 21062 --- [ocp-schedules-18,dc9e8aefd953eeb0,60c8b283041618b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:58.504 INFO 21062 --- [ocp-schedules-20,e26dac38b5cd521d,47e5a1744f45178d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:03:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:03:58.505 INFO 21062 --- [ocp-schedules-1,9cbbe322bcf61f70,63909d432c219d8a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:03:58.510 INFO 21062 --- [ocp-schedules-10,f7197da967fba547,46e02efa5e19f39a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:03:58.510 INFO 21062 --- [ocp-schedules-10,f7197da967fba547,46e02efa5e19f39a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:03:58.510 INFO 21062 --- [ocp-schedules-8,a986e373043b24c8,10bc090d47fb694f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:03:58.511 INFO 21062 --- [ocp-schedules-8,a986e373043b24c8,10bc090d47fb694f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:03:58.512 INFO 21062 --- [ocp-schedules-19,51752f15fc7e862e,b8557a970b24a22d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:03:58.512 INFO 21062 --- [ocp-schedules-19,51752f15fc7e862e,b8557a970b24a22d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:03:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:03:58.590559+08:00, lastProcessTime=2024-11-04T14:03:48.590558+08:00 2024-11-04 14:03:58.637 INFO 21062 --- [ocp-async-1,4685a484b315b64b,350acc68dab6fba7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:03:58.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:03:58.590559+08:00 2024-11-04 14:03:59.042 INFO 21062 --- [ocp-schedules-6,6d20cfb5de832343,c36ace3dbe512c68] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:03:59.057 INFO 21062 --- [ocp-schedules-24,77e89e5b7077537e,1a4c7375b2511828] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:03:59.072 WARN 21062 --- [ocp-async-10,748556d7b23b230f,c9ea4eea08586989] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:03:59.108 INFO 21062 --- [ocp-async-3,79f282d6fffab033,25416438c0feecbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:03:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ccecb96ff916d8cb,5046d61a51b46981] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700240, interval=60 2024-11-04 14:04:00.130 INFO 21062 --- [ocp-schedules-17,bd5208d69f51506b,94e9cca1f3aa242f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:04:01.072 WARN 21062 --- [ocp-async-5,eb204e691d0972b8,630227d88186643a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:01.108 INFO 21062 --- [ocp-async-7,35454a87642f124c,fa1952746919040b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:01.344 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:01.344 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ef8be8a5ca96540f,bdb975b722418019] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:03.072 WARN 21062 --- [ocp-async-8,c813c44d1d7d01a5,14e1a2c3faeb378c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:03.108 INFO 21062 --- [ocp-async-2,e54251b79129cdaa,b2fca931dfe2a725] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:04.344 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:04.345 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:04.782 INFO 21062 --- [ocp-schedules-12,58197de8c16c2d47,032df50d80296c24] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:04:04.782 INFO 21062 --- [ocp-schedules-12,c624cc7989afbcdf,d93b1fa14e7911ca] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:04:04.786 INFO 21062 --- [ocp-schedules-12,c624cc7989afbcdf,d93b1fa14e7911ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:04.816 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:04:04.816 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:04:05.072 WARN 21062 --- [ocp-async-4,d511e299016ef6d1,320cef27e9073f22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:05.109 INFO 21062 --- [ocp-async-9,a5e47ce4161cdc3d,cbe6bd8ad155f6d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2912d8898f3cc402,0a1f9c8ce2ecd87c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:07.072 WARN 21062 --- [ocp-async-3,8d305a1cc320a3f0,8dd393d84c2b81c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:07.109 INFO 21062 --- [ocp-async-1,b768cb00c1c1d4ce,2d9e7537c51ce859] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:07.345 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:07.346 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:08.505 INFO 21062 --- [ocp-schedules-20,e26dac38b5cd521d,47e5a1744f45178d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:04:08.505 INFO 21062 --- [ocp-schedules-18,dc9e8aefd953eeb0,60c8b283041618b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:04:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:04:08.591171+08:00, lastProcessTime=2024-11-04T14:03:58.590559+08:00 2024-11-04 14:04:08.637 INFO 21062 --- [ocp-async-10,f38aab92c6e278eb,9b7ce3147f6ab0a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:04:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:04:08.591171+08:00 2024-11-04 14:04:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f67438811a2417c3,a7d42d18b4569ede] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:09.073 WARN 21062 --- [ocp-async-7,b22d68d872832de3,bff295283f120f1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:09.076 INFO 21062 --- [ocp-schedules-18,00485a86b411e5d8,9e8117a8348afd91] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:04:09.109 INFO 21062 --- [ocp-async-2,c40eb4226be524c6,e2251dbfe93c85ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:04:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:04:10.133 INFO 21062 --- [ocp-schedules-2,22b8eda6c8d60745,ecb486c882dc360b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:04:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:04:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:04:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:04:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:04:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:04:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:04:10.346 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:10.347 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:11.003 INFO 21062 --- [ocp-schedules-16,03c899a2cfd211e3,b10ac305fa08f6e4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:04:11.009 INFO 21062 --- [ocp-schedules-16,03c899a2cfd211e3,b10ac305fa08f6e4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:04:11.009 INFO 21062 --- [ocp-schedules-16,03c899a2cfd211e3,b10ac305fa08f6e4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:04:11.073 WARN 21062 --- [ocp-async-2,32e230efe20003ba,c1f56ca79e58dde4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:11.109 INFO 21062 --- [ocp-async-9,034682b1d5061e0d,7f274a3c9e42be8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=8, affectRows=8 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=18, affectRows=18 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=8, validCount=8 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=10, affectRows=10 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=36, affectRows=36 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=29, affectRows=29 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=10, validCount=10 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=18, validCount=18 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=29, validCount=29 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=36, validCount=36 2024-11-04 14:04:11.434 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=55, affectRows=55 2024-11-04 14:04:11.435 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=68, affectRows=68 2024-11-04 14:04:11.435 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=55, validCount=55 2024-11-04 14:04:11.435 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=12, affectRows=12 2024-11-04 14:04:11.435 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=12, validCount=12 2024-11-04 14:04:11.435 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=68, validCount=68 2024-11-04 14:04:11.435 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=75, affectRows=75 2024-11-04 14:04:11.435 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=73, validCount=73 2024-11-04 14:04:11.436 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700180, nodeCount=81, affectRows=81 2024-11-04 14:04:11.436 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700180, nodeCount=80, validCount=80 2024-11-04 14:04:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a5ded39bcf29ae33,93355a73f5f20c83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:13.073 WARN 21062 --- [ocp-async-9,000797856f20e28e,6f7a5fef82b6ebfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:13.109 INFO 21062 --- [ocp-async-1,87f05afa52c90b45,d7cac1ffda3269f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:13.347 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:13.348 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:13.984 INFO 21062 --- [prometheus-cache-monitor1,d59db315dbb635fb,463d5b60cd7b6cfb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:04:13.984 INFO 21062 --- [prometheus-cache-monitor1,d59db315dbb635fb,463d5b60cd7b6cfb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:04:14.983 INFO 21062 --- [prometheus-minute-schedule-1,adfe3871a16d5e19,ede91e9377132a7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:15.073 WARN 21062 --- [ocp-async-1,706718e38a9ea878,a82fa5e61f96c75c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:15.109 INFO 21062 --- [ocp-async-8,569f2e465b5d60ce,08135d0e364df2a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:16.348 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:16.350 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:17.073 WARN 21062 --- [ocp-async-8,a35df1e30a62c264,d105a563069989df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:17.110 INFO 21062 --- [ocp-async-4,515f4d43b77eab40,9f73461c32c3227f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:17.983 INFO 21062 --- [prometheus-minute-schedule-1,337c432245e0188b,99147074a7fb67f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:04:18.590568+08:00, lastProcessTime=2024-11-04T14:04:08.591171+08:00 2024-11-04 14:04:18.638 INFO 21062 --- [ocp-async-6,ee8566dd4861dcdf,839fecf682d7d0cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:04:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:04:18.590568+08:00 2024-11-04 14:04:19.074 WARN 21062 --- [ocp-async-9,0d364d412d12da74,2e926ad2909eed82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:19.110 INFO 21062 --- [ocp-async-1,f719108eb2358d58,e99dec77952d7353] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:19.118 INFO 21062 --- [ocp-schedules-22,ba6cc27c7c506515,09123299b220f5e1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:04:19.350 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:19.351 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:20.136 INFO 21062 --- [ocp-schedules-2,42dd7f8d9d732ce7,70c2f3dd1f39ed21] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:04:20.983 INFO 21062 --- [prometheus-minute-schedule-1,380b0f35caebdc0c,693b8fd809ea8975] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:21.074 WARN 21062 --- [ocp-async-1,109ee5df27797631,3e41c067f76e335d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:21.110 INFO 21062 --- [ocp-async-8,6973ccc408327b2f,5903ddbddb73435f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:22.351 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:22.352 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:23.074 WARN 21062 --- [ocp-async-8,b6825cd344647ce4,84398c49d685f2a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:23.110 INFO 21062 --- [ocp-async-4,72202fe259f0518c,7af8077fd18aadd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:23.983 INFO 21062 --- [prometheus-minute-schedule-1,614473ea76758ba3,267bc18a2b82bbb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:25.074 WARN 21062 --- [ocp-async-4,d9acf152dc2778b9,7cb0ec321087ec7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:25.110 INFO 21062 --- [ocp-async-5,fe9fec5c4564509e,cb3fbfcc1b2a2377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:25.352 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:25.353 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:26.983 INFO 21062 --- [prometheus-minute-schedule-1,de36115dd1258ee2,e6b29686a7a7416d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2f3dd46cdc2d1fec,de81ce3261a90945] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:04:27.074 WARN 21062 --- [ocp-async-5,18368729eaa74f52,58ec02c2d2077c33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:27.111 INFO 21062 --- [ocp-async-6,5c3b1279ef7530e7,dd6aa4019fa1d940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:28.353 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:28.354 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:28.498 INFO 21062 --- [ocp-schedules-24,39803ff44984625f,623233500ddb776c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:04:28.499 INFO 21062 --- [ocp-schedules-2,5538c4bc9332a838,15b4bf780bfc39de] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:04:28.500 INFO 21062 --- [ocp-schedules-4,e72dcb340b761add,ea3c2e08e12ea51a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:04:28.500 INFO 21062 --- [ocp-schedules-17,19e75af07670d077,6b6c84ceeec1bfa6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:04:28.502 INFO 21062 --- [ocp-schedules-24,39803ff44984625f,623233500ddb776c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:04:28.502 INFO 21062 --- [ocp-schedules-27,b4fe07f315db6688,5ce750dd6c4434f5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:04:28.503 INFO 21062 --- [ocp-schedules-4,e72dcb340b761add,ea3c2e08e12ea51a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:28.504 INFO 21062 --- [ocp-schedules-17,19e75af07670d077,6b6c84ceeec1bfa6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:04:28.505 INFO 21062 --- [ocp-schedules-13,26ff9972b074bfdf,54ebb3d609d7671d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:04:28.505 INFO 21062 --- [ocp-schedules-2,5538c4bc9332a838,15b4bf780bfc39de] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:04:28.512 INFO 21062 --- [ocp-schedules-3,f02649d3980bae2e,5068313d3d28ad72] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:04:28.512 INFO 21062 --- [ocp-schedules-5,7352ceb5db6f4226,8a74f961aff3f7eb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:04:28.512 INFO 21062 --- [ocp-schedules-15,2d498c476e6b906f,860ad9260267dc79] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:04:28.512 INFO 21062 --- [ocp-schedules-3,f02649d3980bae2e,5068313d3d28ad72] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:04:28.512 INFO 21062 --- [ocp-schedules-5,7352ceb5db6f4226,8a74f961aff3f7eb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:04:28.512 INFO 21062 --- [ocp-schedules-15,2d498c476e6b906f,860ad9260267dc79] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:04:28.572 INFO 21062 --- [ocp-schedules-29,f3d5c7b6ea06c9ba,971b6b56c1763afb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:04:28.572 INFO 21062 --- [ocp-schedules-29,f3d5c7b6ea06c9ba,971b6b56c1763afb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:04:28.578 INFO 21062 --- [ocp-schedules-14,8f810ad690585aa4,73613e1a26b3ac07] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:04:28.578+08:00, tenantSize=0 2024-11-04 14:04:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:04:28.589080+08:00, lastProcessTime=2024-11-04T14:04:18.590568+08:00 2024-11-04 14:04:28.638 INFO 21062 --- [ocp-async-1,6f6ea06a34182f87,a3f2288c46cb115b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:04:28.646 INFO 21062 --- [ocp-schedules-19,bb09d304316e2465,2681c6cc6c228bfa] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 14:04:28.646 INFO 21062 --- [ocp-schedules-19,bb09d304316e2465,2681c6cc6c228bfa] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 14:04:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:04:28.589080+08:00 2024-11-04 14:04:28.782 INFO 21062 --- [ocp-schedules-22,4360bc69e4de3332,0d49ab19a1998d01] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:04:28.932 INFO 21062 --- [ocp-schedules-13,4fdb3e03f8a5c966,2b23b604858be697] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:04:28.935 INFO 21062 --- [ocp-schedules-13,4fdb3e03f8a5c966,2b23b604858be697] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:28.936 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:04:29.004 INFO 21062 --- [ocp-schedules-1,953aeebea2ba2ac9,fef60ea292941dbf] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:04:29.004 INFO 21062 --- [ocp-schedules-1,953aeebea2ba2ac9,fef60ea292941dbf] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:04:29.059 INFO 21062 --- [ocp-schedules-3,1aead711d60f2ad4,79f0faac67400543] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:04:29.075 WARN 21062 --- [ocp-async-10,528b2bebf45baace,0dba62d5256e865e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:29.090 INFO 21062 --- [ocp-schedules-24,32278bedd839ecdf,afac1f5225797246] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:04:29.096 INFO 21062 --- [ocp-schedules-24,32278bedd839ecdf,afac1f5225797246] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:04:29.096 INFO 21062 --- [ocp-schedules-24,32278bedd839ecdf,afac1f5225797246] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:04:29.111 INFO 21062 --- [ocp-async-3,bc9a2b54ea1cfd13,2535225e71069ff9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:29.145 INFO 21062 --- [ocp-schedules-16,731b82bbbc3d9a4f,d39c5455ef973a3b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:04:29.201 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:04:29.201 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:04:29.209 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:04:29.209 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:04:29.209 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:04:29.220 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:04:29.220 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:04:29.220 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:04:29.224 INFO 21062 --- [ocp-schedules-28,177504cf985da98e,7ac8b4f27286acde] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:04:29.371 INFO 21062 --- [ocp-schedules-8,30efb6d0351506f4,80738bfaa8e85a50] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:04:29.371 INFO 21062 --- [ocp-schedules-8,30efb6d0351506f4,80738bfaa8e85a50] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:04:29.378 INFO 21062 --- [ocp-schedules-8,30efb6d0351506f4,80738bfaa8e85a50] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:04:29.378 INFO 21062 --- [ocp-schedules-8,30efb6d0351506f4,80738bfaa8e85a50] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:04:29.383 INFO 21062 --- [ocp-schedules-8,30efb6d0351506f4,80738bfaa8e85a50] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:04:29.383 INFO 21062 --- [ocp-schedules-8,30efb6d0351506f4,80738bfaa8e85a50] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:04:29.983 INFO 21062 --- [prometheus-minute-schedule-1,1548867242a5f8be,7e4b02c23fe81fd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:30.139 INFO 21062 --- [ocp-schedules-22,c0ec1099bdd04179,58c451040d334cba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:04:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:04:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:04:31.011 INFO 21062 --- [ocp-schedules-23,7cac59ee26ef4681,23ea196dd2ce9cf2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:04:31.017 INFO 21062 --- [ocp-schedules-23,7cac59ee26ef4681,23ea196dd2ce9cf2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:04:31.017 INFO 21062 --- [ocp-schedules-23,7cac59ee26ef4681,23ea196dd2ce9cf2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:04:31.032 INFO 21062 --- [ocp-schedules-15,e30cecff23f1473b,f5e1c9edea00e89b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:04:31.032 INFO 21062 --- [ocp-schedules-15,e30cecff23f1473b,f5e1c9edea00e89b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:04:31.058 INFO 21062 --- [ocp-schedules-15,e30cecff23f1473b,f5e1c9edea00e89b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:04:31.058 INFO 21062 --- [ocp-schedules-15,e30cecff23f1473b,f5e1c9edea00e89b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:04:31.060 INFO 21062 --- [ocp-schedules-1,f345fdc98f863156,ed14d3b63f82ac1a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:04:31.075 WARN 21062 --- [ocp-async-3,c75803cead6f5187,8cdbdbbb8a7aff17] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:31.111 INFO 21062 --- [ocp-async-1,33dc5e0adc1cd418,20918fb580314af7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:31.233 INFO 21062 --- [ocp-schedules-21,3368edfe5c1c4f64,eac10cefc106175c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:04:31.233+08:00, clusters=[1] 2024-11-04 14:04:31.234 INFO 21062 --- [ocp-schedules-21,3368edfe5c1c4f64,eac10cefc106175c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:04:31.300 INFO 21062 --- [ocp-schedules-22,9417e794d386ae3d,08b354a32e77ea06] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:04:31.301 INFO 21062 --- [ocp-schedules-22,9417e794d386ae3d,08b354a32e77ea06] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:04:31.301 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:04:31.301 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:04:31.312 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:04:31.312 INFO 21062 --- [ocp-schedules-22,9417e794d386ae3d,08b354a32e77ea06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:04:31.312 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:04:31.314 WARN 21062 --- [ocp-schedules-22,9417e794d386ae3d,08b354a32e77ea06] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:04:31.314 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:04:31.314 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:04:31.314 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:04:31.314 INFO 21062 --- [ocp-schedules-22,9417e794d386ae3d,08b354a32e77ea06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:04:31.315 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:04:31.316 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:04:31.316 WARN 21062 --- [ocp-schedules-22,9417e794d386ae3d,08b354a32e77ea06] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:04:31.316 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:04:31.325 ERROR 21062 --- [ocp-schedules-22,9417e794d386ae3d,08b354a32e77ea06] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 14:04:31.328 INFO 21062 --- [ocp-schedules-14,9f8b30aa9521f875,77d2119672adf967] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:04:31.332 INFO 21062 --- [ocp-schedules-14,9f8b30aa9521f875,77d2119672adf967] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:31.354 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:31.355 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:31.376 INFO 21062 --- [ocp-schedules-16,1824005920a50353,2e7c05281b2bab4a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:04:31.376+08:00, clusters=[1] 2024-11-04 14:04:31.380 INFO 21062 --- [ocp-schedules-16,1824005920a50353,2e7c05281b2bab4a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:04:31.387 ERROR 21062 --- [ocp-schedules-14,9f8b30aa9521f875,77d2119672adf967] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:04:31.387 INFO 21062 --- [ocp-schedules-14,9f8b30aa9521f875,77d2119672adf967] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:04:32.983 INFO 21062 --- [prometheus-minute-schedule-1,1da62c724c024fe4,3af361960bb117ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:33.075 WARN 21062 --- [ocp-async-1,786e56e2bf412b83,678e784396efba8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:33.111 INFO 21062 --- [ocp-async-8,2c4106aca6c42568,8461d60ae4f908a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:34.355 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:34.356 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:34.817 INFO 21062 --- [ocp-schedules-12,c624cc7989afbcdf,d93b1fa14e7911ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:04:34.817 INFO 21062 --- [ocp-schedules-12,7174b10885a62858,9ef9c98db17934fb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:04:34.821 INFO 21062 --- [ocp-schedules-12,7174b10885a62858,9ef9c98db17934fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:34.850 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:04:34.850 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:04:35.075 WARN 21062 --- [ocp-async-8,9ecb89691c0eb963,46ed64d71983e61c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:35.111 INFO 21062 --- [ocp-async-9,672f99c3767cb66b,0716933541c6e6be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:35.983 INFO 21062 --- [prometheus-minute-schedule-1,dcdc053aaea1f79b,0c7bd05caa50637f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:37.012 INFO 21062 --- [ocp-schedules-8,2e7856c09f5efa57,57a7997623f9857b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:04:37.015 INFO 21062 --- [ocp-schedules-8,2e7856c09f5efa57,57a7997623f9857b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:37.075 WARN 21062 --- [ocp-async-9,ed301b181d668366,07bff57d5a44fd19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:37.102 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:04:37.104 ERROR 21062 --- [ocp-schedules-8,2e7856c09f5efa57,57a7997623f9857b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:04:37.104 INFO 21062 --- [ocp-schedules-8,2e7856c09f5efa57,57a7997623f9857b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:04:37.111 INFO 21062 --- [ocp-async-5,89c31c3790fc7281,975782832bdcf9de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:37.356 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:37.357 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:38.504 INFO 21062 --- [ocp-schedules-17,19e75af07670d077,6b6c84ceeec1bfa6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:04:38.505 INFO 21062 --- [ocp-schedules-4,e72dcb340b761add,ea3c2e08e12ea51a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:04:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:04:38.589940+08:00, lastProcessTime=2024-11-04T14:04:28.589080+08:00 2024-11-04 14:04:38.638 INFO 21062 --- [ocp-async-9,6bb9eaf1cf1ddb99,1acbaae54661e3cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:04:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:04:38.589940+08:00 2024-11-04 14:04:38.936 INFO 21062 --- [ocp-schedules-13,4fdb3e03f8a5c966,2b23b604858be697] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:04:38.983 INFO 21062 --- [prometheus-minute-schedule-1,6ecb55871975781d,421ff105af95b388] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:39.075 WARN 21062 --- [ocp-async-6,e4d0b075a2fc0192,6cf211dca2914916] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:39.112 INFO 21062 --- [ocp-async-8,0868610480771a13,01ba9b31925bf201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:39.187 INFO 21062 --- [ocp-schedules-21,a9c194498672f66a,9064cddb5217e239] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:04:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:04:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:04:40.143 INFO 21062 --- [ocp-schedules-10,144d8ca54872e0ff,1e759414dc067365] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:04:40.357 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:40.358 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:41.076 WARN 21062 --- [ocp-async-10,a16b072c1b07383a,b00fced21fed5247] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:41.112 INFO 21062 --- [ocp-async-9,69be226d99242b07,b5d82435208b077c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:41.704 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:04:41.704 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:04:41.707 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.708 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:04:41.709 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.710 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.710 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.711 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:04:41.726 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.726 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.727 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.727 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:04:41.742 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.742 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.743 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.744 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:04:41.758 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.758 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.759 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.759 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:04:41.772 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.772 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.773 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.773 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:04:41.779 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.780 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.780 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.781 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:04:41.785 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.786 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.786 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.787 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:04:41.791 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.791 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.792 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.792 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:04:41.798 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.799 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.799 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.800 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:04:41.805 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.805 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.806 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.806 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:04:41.811 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.811 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.812 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.812 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:04:41.821 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.821 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.822 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.822 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:04:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:04:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:04:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:04:41.839 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.840 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.840 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.841 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:04:41.847 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.848 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.848 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.849 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:04:41.855 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:04:41.856 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:04:41.857 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.858 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.858 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.858 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:04:41.863 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.864 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.864 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.864 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:04:41.869 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.869 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.870 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.870 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:04:41.874 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.875 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.875 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.876 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:04:41.881 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.881 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.881 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.882 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:04:41.888 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.888 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.889 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.889 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:04:41.893 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.894 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.894 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.895 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:04:41.900 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.900 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.901 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.901 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:04:41.905 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.905 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.906 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.906 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:04:41.911 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.911 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.911 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.912 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:04:41.916 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.916 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.917 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.917 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:04:41.921 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.922 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.922 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.923 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:04:41.927 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.927 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.928 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.928 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:04:41.932 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.932 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.933 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.934 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:04:41.938 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.938 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.939 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.939 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:04:41.944 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.944 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.945 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.945 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:04:41.949 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.950 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.950 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.950 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:04:41.955 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.955 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.956 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.956 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:04:41.960 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.961 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.961 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.962 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:04:41.966 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.966 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.967 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.967 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:04:41.977 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.977 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.978 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.978 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:04:41.983 INFO 21062 --- [prometheus-minute-schedule-1,732e1ced789dec0e,0ebfc5f3bcf0e886] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:41.984 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.985 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.985 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.986 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:04:41.991 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.992 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.992 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.993 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:04:41.998 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.998 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:41.999 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:41.999 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:04:42.002 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.003 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.003 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.004 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:04:42.007 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.008 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.009 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.009 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:04:42.013 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.013 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.014 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.014 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:04:42.018 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.018 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.019 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.019 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:04:42.028 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.028 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.029 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.029 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:04:42.049 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.050 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.051 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.051 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:04:42.057 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.058 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.059 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.059 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:04:42.065 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.065 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.065 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.066 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:04:42.071 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.072 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.073 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.073 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:04:42.077 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.077 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.078 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.078 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:04:42.082 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.082 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.083 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.083 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:04:42.087 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:04:42.087 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:04:42.088 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.088 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:04:42.089 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.090 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.090 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.091 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:04:42.096 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.097 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:04:42.097 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:04:42.097 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:04:42.102 INFO 21062 --- [ocp-schedules-9,6ec5407751304440,4f67863de3c676c6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:04:43.076 WARN 21062 --- [ocp-async-3,b676ca75c91aeeb8,b1434b8e48ba5282] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:43.112 INFO 21062 --- [ocp-async-7,c43c255bade70de8,0834bb7d0bdf50df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:43.358 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:43.359 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:44.983 INFO 21062 --- [prometheus-minute-schedule-1,ad25bf79d8b4962c,0b5868151bfe684c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:45.076 WARN 21062 --- [ocp-async-1,a6414b3d518d108c,c8dd89a6a80eb45f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:45.113 INFO 21062 --- [ocp-async-2,3ea30276ab248217,94212a39ad50c556] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:46.359 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:46.361 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:47.076 WARN 21062 --- [ocp-async-8,0413e8de30926ce4,9b299e54beb29aee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:47.113 INFO 21062 --- [ocp-async-4,5a631634b9995fd7,a8f0817c4cba51ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e07feb91bdcb26c1,233c2d0875ff55bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:04:48.591699+08:00, lastProcessTime=2024-11-04T14:04:38.589940+08:00 2024-11-04 14:04:48.638 INFO 21062 --- [ocp-async-6,cb74e8f3e0b3fde8,6ba140f115641ac2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:04:48.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:04:48.591699+08:00 2024-11-04 14:04:49.076 WARN 21062 --- [ocp-async-3,a701eb729325d8a8,22726358a04105da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:49.113 INFO 21062 --- [ocp-async-7,8188577b7772f955,c8753af87e64e190] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:49.213 INFO 21062 --- [ocp-schedules-6,41921affac2a4713,895df4f6614634d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:04:49.361 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:49.362 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:50.146 INFO 21062 --- [ocp-schedules-5,dd59e7afc1a74651,dcc8e3f27e7a47dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:04:50.983 INFO 21062 --- [prometheus-minute-schedule-1,015c4d996519ff55,9609bfd1fd742ed4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:51.019 INFO 21062 --- [ocp-schedules-9,faa4fd6edebfae04,bd59c205a1a1765d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:04:51.025 INFO 21062 --- [ocp-schedules-9,faa4fd6edebfae04,bd59c205a1a1765d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:04:51.025 INFO 21062 --- [ocp-schedules-9,faa4fd6edebfae04,bd59c205a1a1765d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:04:51.077 WARN 21062 --- [ocp-async-1,86752b7e6800d416,2fe9334e0c12d1b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:51.113 INFO 21062 --- [ocp-async-6,120147a9f0005a72,5454dfab4a76afa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:52.362 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:52.363 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:53.077 WARN 21062 --- [ocp-async-8,17351d208c3d497f,5d6b579688ec3729] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:53.113 INFO 21062 --- [ocp-async-4,c00c4d5ea50be187,09fe6afd8174dc0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e0443aa844f6bd97,8fc156bd4b58f04b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:55.077 WARN 21062 --- [ocp-async-9,737012cae68b47b6,00b99606c70cdfe1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:55.113 INFO 21062 --- [ocp-async-5,c37ade96caab7ecf,2f1cde701855ccff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:55.364 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:55.364 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:56.983 INFO 21062 --- [prometheus-minute-schedule-1,54f313caab411870,d602d48377e6efb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:04:57.077 WARN 21062 --- [ocp-async-7,ed7813f9433bec48,df926727e6904f7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:57.114 INFO 21062 --- [ocp-async-2,31d782e4b0486d39,a7f002d29a4535f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:58.365 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:04:58.366 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:04:58.500 INFO 21062 --- [ocp-schedules-6,87edfa27196a6ea6,3b3247e4a34223c0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:04:58.500 INFO 21062 --- [ocp-schedules-16,89652801bc629578,6e01967bc461e5c9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:04:58.503 INFO 21062 --- [ocp-schedules-17,dacfe482094548de,90a6666c98e96eed] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:04:58.504 INFO 21062 --- [ocp-schedules-6,87edfa27196a6ea6,3b3247e4a34223c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:58.504 INFO 21062 --- [ocp-schedules-16,89652801bc629578,6e01967bc461e5c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:04:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:04:58.505 INFO 21062 --- [ocp-schedules-26,4c43c512668d03f4,89953ff7412bd178] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:04:58.510 INFO 21062 --- [ocp-schedules-7,e5f5c156a958070f,13424270dd4675a2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:04:58.510 INFO 21062 --- [ocp-schedules-7,e5f5c156a958070f,13424270dd4675a2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:04:58.510 INFO 21062 --- [ocp-schedules-10,11ca21d20d5996bd,30bdcbb86b387b59] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:04:58.510 INFO 21062 --- [ocp-schedules-10,11ca21d20d5996bd,30bdcbb86b387b59] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:04:58.511 INFO 21062 --- [ocp-schedules-23,a8a463f4f0af9de4,8401587ec1c96476] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:04:58.511 INFO 21062 --- [ocp-schedules-23,a8a463f4f0af9de4,8401587ec1c96476] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:04:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:04:58.589729+08:00, lastProcessTime=2024-11-04T14:04:48.591699+08:00 2024-11-04 14:04:58.638 INFO 21062 --- [ocp-async-3,14e6a2bf6fc96236,ca8c6ad4b6218649] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:04:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:04:58.589729+08:00 2024-11-04 14:04:59.062 INFO 21062 --- [ocp-schedules-24,1baa9e9e15386291,26076e6422ec8197] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:04:59.077 WARN 21062 --- [ocp-async-8,9e26f180cf114a26,4f5fc9172281c299] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:04:59.114 INFO 21062 --- [ocp-async-4,af5f3e97b9acb26b,1bd4fcf5b07ba9ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:04:59.256 INFO 21062 --- [ocp-schedules-19,57b5d9456c54435e,c298b7d3a1cc5d7b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:04:59.983 INFO 21062 --- [prometheus-minute-schedule-1,3e22f44c7bf13728,8d91f4a5b3c0b939] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700300, interval=60 2024-11-04 14:05:00.034 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,5ea8a684a52bc062] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:00:00.0, nextRun=2024-11-04 14:05:00.0, currentDate=Mon Nov 04 14:05:00 CST 2024 2024-11-04 14:05:00.035 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,5ea8a684a52bc062] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:00:00.0, nextRunTime=2024-11-04 14:05:00.0} 2024-11-04 14:05:00.061 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,5ea8a684a52bc062] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:05:00.065 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,5ea8a684a52bc062] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:05:00.065 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,5ea8a684a52bc062] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:05:00.149 INFO 21062 --- [ocp-schedules-1,83f119981639ae40,6e10b33f53f0bf1a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:05:01.077 WARN 21062 --- [ocp-async-9,47a1166acd7cb5ad,2497b5606e1df296] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:01.114 INFO 21062 --- [ocp-async-5,6c0e60862dfcbd11,fa07fe93588b7e4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:01.366 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:01.366 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:02.983 INFO 21062 --- [prometheus-minute-schedule-1,0e50883334558f8c,10c4440e21d8c8ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:03.078 WARN 21062 --- [ocp-async-7,7cb4b9a783f1cfa6,1eba907a89d813da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:03.114 INFO 21062 --- [ocp-async-2,7ab030f24e26ac3b,982e4cd9d3495978] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:04.367 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:04.368 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:04.851 INFO 21062 --- [ocp-schedules-12,7174b10885a62858,9ef9c98db17934fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:05:04.851 INFO 21062 --- [ocp-schedules-12,c9cac4cda7cc8a68,e16cd9e4f784deff] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:05:04.855 INFO 21062 --- [ocp-schedules-12,c9cac4cda7cc8a68,e16cd9e4f784deff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:04.884 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:05:04.884 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:05:05.078 WARN 21062 --- [ocp-async-6,fa3e5513a9be7c62,b4b0cfc2a42f682b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:05.114 INFO 21062 --- [ocp-async-10,dff3ac9fe6c4d7df,8240552dbf02ea1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8e48228b9e4653a2,f8528a3bb4d7c5b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:07.078 WARN 21062 --- [ocp-async-10,a70117ffa6550118,3898b429f0bee1d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:07.114 INFO 21062 --- [ocp-async-3,d37aa5a407596731,dc49bd35361d5122] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:07.368 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:07.370 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:08.504 INFO 21062 --- [ocp-schedules-16,89652801bc629578,6e01967bc461e5c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:05:08.505 INFO 21062 --- [ocp-schedules-6,87edfa27196a6ea6,3b3247e4a34223c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:05:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:05:08.590135+08:00, lastProcessTime=2024-11-04T14:04:58.589729+08:00 2024-11-04 14:05:08.639 INFO 21062 --- [ocp-async-8,cd9776707a64b16c,1829a6a4bd55c820] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:05:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:05:08.590135+08:00 2024-11-04 14:05:08.983 INFO 21062 --- [prometheus-minute-schedule-1,1e50a58504682f0b,7882671b3b0fdac4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:09.078 WARN 21062 --- [ocp-async-5,ec2e93b074f7b0aa,c5187d379d25e419] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:09.115 INFO 21062 --- [ocp-async-2,de614e6a799fd6a3,c059711d313a42b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:09.282 INFO 21062 --- [ocp-schedules-19,58a6266dd813a0a8,05f7776750e11cdf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:05:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:05:10.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:05:10.153 INFO 21062 --- [ocp-schedules-25,ac216bc295df5453,26264555a178c723] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:05:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:05:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:05:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:05:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:05:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:05:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:05:10.370 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:10.371 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:11.028 INFO 21062 --- [ocp-schedules-15,61c48b9fd00088c2,df328916b09da017] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:05:11.034 INFO 21062 --- [ocp-schedules-15,61c48b9fd00088c2,df328916b09da017] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:05:11.034 INFO 21062 --- [ocp-schedules-15,61c48b9fd00088c2,df328916b09da017] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:05:11.079 WARN 21062 --- [ocp-async-2,ebbefcadc830bdfe,de5c925b728151a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:11.115 INFO 21062 --- [ocp-async-10,fc0f2d8c7f6a881c,ac8c5abff9416a3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:11.439 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700240, nodeCount=55, affectRows=55 2024-11-04 14:05:11.439 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700240, nodeCount=55, validCount=55 2024-11-04 14:05:11.439 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700240, nodeCount=47, affectRows=47 2024-11-04 14:05:11.439 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700240, nodeCount=46, affectRows=46 2024-11-04 14:05:11.439 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700240, nodeCount=46, validCount=46 2024-11-04 14:05:11.439 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700240, nodeCount=46, validCount=46 2024-11-04 14:05:11.440 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700240, nodeCount=63, affectRows=63 2024-11-04 14:05:11.440 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700240, nodeCount=80, affectRows=80 2024-11-04 14:05:11.440 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700240, nodeCount=62, validCount=62 2024-11-04 14:05:11.440 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700240, nodeCount=79, validCount=79 2024-11-04 14:05:11.440 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700240, nodeCount=101, affectRows=101 2024-11-04 14:05:11.441 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700240, nodeCount=101, validCount=101 2024-11-04 14:05:11.983 INFO 21062 --- [prometheus-minute-schedule-1,ae4f637907ea3e9e,f880c2f3e67c6cfc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:13.079 WARN 21062 --- [ocp-async-10,82ceb2943dfb6b43,845809dd6bb46583] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:13.115 INFO 21062 --- [ocp-async-3,0689a4a26b69e98a,350e0dc9f8b82b1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:13.371 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:13.373 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:13.984 INFO 21062 --- [prometheus-cache-monitor1,9f5e3221781aa2ca,ac2aad66362e1a3d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:05:13.984 INFO 21062 --- [prometheus-cache-monitor1,9f5e3221781aa2ca,ac2aad66362e1a3d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:05:14.880 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:01:14.862, end=2024-11-04T14:03:14.877} 2024-11-04 14:05:14.891 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:05:14.891 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:05:14.983 INFO 21062 --- [prometheus-minute-schedule-1,35742df38a30cdfc,db57e496f45dc9ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:15.079 WARN 21062 --- [ocp-async-3,7c430dd992d7e437,e5373e77a253ce6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:15.115 INFO 21062 --- [ocp-async-7,c97953034bfcab55,335a9f0ce0368e81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:16.373 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:16.374 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:17.079 WARN 21062 --- [ocp-async-7,ef3af55cc071b6dc,2923bc356a2c7f8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:17.115 INFO 21062 --- [ocp-async-8,1e4744935615cda0,bb50da4c41efdfe1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4cd2657c0e65faa3,5380b3e4aab1c06b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:05:18.590786+08:00, lastProcessTime=2024-11-04T14:05:08.590135+08:00 2024-11-04 14:05:18.639 INFO 21062 --- [ocp-async-1,0470fc3d8036dbd5,620b3351a7fd840c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:05:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:05:18.590786+08:00 2024-11-04 14:05:19.079 WARN 21062 --- [ocp-async-10,84b529ef08dd93ce,7383abfd6ecb6df2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:19.116 INFO 21062 --- [ocp-async-3,19dfe256e9625e90,8307d5a06e4ee7a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:19.323 INFO 21062 --- [ocp-schedules-25,15e108a0d2911a2c,91a88047a1086669] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:05:19.374 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:19.375 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:20.156 INFO 21062 --- [ocp-schedules-9,773fdf40b53c8258,44027b520a0dfa32] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:05:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8781b33d60e7f33f,f8211bf798672929] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:21.079 WARN 21062 --- [ocp-async-3,d3e6dc530849ce56,5eb16bffb279efec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:21.116 INFO 21062 --- [ocp-async-7,aeafcf0d3df3d71e,2c9cbcf4c2a978fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:22.375 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:22.376 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:23.080 WARN 21062 --- [ocp-async-7,d90313eb51089236,8a8904a1717b8069] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:23.116 INFO 21062 --- [ocp-async-8,89d3e4a36c80df8a,b8f72eee764e44b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:23.983 INFO 21062 --- [prometheus-minute-schedule-1,1d632106b813c466,68bcfc5df70af404] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:25.080 WARN 21062 --- [ocp-async-8,d80bf5659ceb55b5,272f98eb9d00ab76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:25.116 INFO 21062 --- [ocp-async-4,b33af10f85317d01,e1032c0f890a4a6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:25.376 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:25.377 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e7cf62e38725c5db,8cc602f4daa26f62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f539e996523191dc,bc665e09a51295f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:05:27.080 WARN 21062 --- [ocp-async-4,71e1fb90a31362d4,382333a882ac0c81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:27.116 INFO 21062 --- [ocp-async-5,7b0aebf6d629a1cb,f8da91b9f2ac5f2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:28.377 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:28.378 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:28.498 INFO 21062 --- [ocp-schedules-27,efa53e8b128d031f,b37585d24faf3da5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:05:28.499 INFO 21062 --- [ocp-schedules-23,e3a0daeae921d204,6acfeae922877e80] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:05:28.500 INFO 21062 --- [ocp-schedules-2,78363dd97e596ab4,e916a12052e653c4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:05:28.500 INFO 21062 --- [ocp-schedules-4,81472aedf28a73c1,99e55ca39fda7859] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:05:28.503 INFO 21062 --- [ocp-schedules-26,58d83b21d46f3c23,04695f9f96894013] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:05:28.503 INFO 21062 --- [ocp-schedules-27,efa53e8b128d031f,b37585d24faf3da5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:05:28.503 INFO 21062 --- [ocp-schedules-2,78363dd97e596ab4,e916a12052e653c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:28.504 INFO 21062 --- [ocp-schedules-4,81472aedf28a73c1,99e55ca39fda7859] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:28.504 INFO 21062 --- [ocp-schedules-9,06e7b8eda0bdc72b,a173e208ffb82b52] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:05:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:05:28.507 INFO 21062 --- [ocp-schedules-23,e3a0daeae921d204,6acfeae922877e80] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:05:28.511 INFO 21062 --- [ocp-schedules-21,b5c84985cad409fb,74d6464bae8a1467] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:05:28.511 INFO 21062 --- [ocp-schedules-21,b5c84985cad409fb,74d6464bae8a1467] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:05:28.511 INFO 21062 --- [ocp-schedules-28,473f85d895f5176a,e00f4a979bf91bad] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:05:28.511 INFO 21062 --- [ocp-schedules-20,328815e6ab9295c9,65bd2b6ec1f4b215] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:05:28.511 INFO 21062 --- [ocp-schedules-20,328815e6ab9295c9,65bd2b6ec1f4b215] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:05:28.511 INFO 21062 --- [ocp-schedules-28,473f85d895f5176a,e00f4a979bf91bad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:05:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:05:28.590043+08:00, lastProcessTime=2024-11-04T14:05:18.590786+08:00 2024-11-04 14:05:28.639 INFO 21062 --- [ocp-async-3,38c4955bd509fa43,fe9cd01a7764524a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:05:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:05:28.590043+08:00 2024-11-04 14:05:28.784 INFO 21062 --- [ocp-schedules-25,54302f360ab1d444,84d6920a2d118efc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:05:29.011 INFO 21062 --- [ocp-schedules-5,65bf1e907df487c1,c3c175e3d2c4e32e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:05:29.011 INFO 21062 --- [ocp-schedules-5,65bf1e907df487c1,c3c175e3d2c4e32e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:05:29.065 INFO 21062 --- [ocp-schedules-15,f3a0e3a1fb1b8a25,acf62c6f8e5e38ac] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:05:29.080 WARN 21062 --- [ocp-async-6,2a38df78df4b1d64,508c2c6eae513fd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:29.096 INFO 21062 --- [ocp-schedules-26,415dfdb01ea26965,bb92dd0e521cf1a2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:05:29.102 INFO 21062 --- [ocp-schedules-26,415dfdb01ea26965,bb92dd0e521cf1a2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:05:29.102 INFO 21062 --- [ocp-schedules-26,415dfdb01ea26965,bb92dd0e521cf1a2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:05:29.117 INFO 21062 --- [ocp-async-8,a43e902b2fe0316a,8e4c373f07eb8f27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:29.348 INFO 21062 --- [ocp-schedules-30,4ffcf444794b3f80,a19db195d31fe1bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:05:29.388 INFO 21062 --- [ocp-schedules-19,28bf84d993a126bc,27d4959b5fcc17e5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:05:29.388 INFO 21062 --- [ocp-schedules-19,28bf84d993a126bc,27d4959b5fcc17e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:05:29.389 INFO 21062 --- [ocp-schedules-19,28bf84d993a126bc,27d4959b5fcc17e5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:05:29.389 INFO 21062 --- [ocp-schedules-19,28bf84d993a126bc,27d4959b5fcc17e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:05:29.389 INFO 21062 --- [ocp-schedules-19,28bf84d993a126bc,27d4959b5fcc17e5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:05:29.389 INFO 21062 --- [ocp-schedules-19,28bf84d993a126bc,27d4959b5fcc17e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:05:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4bb9cb4a28b76968,f018e16a58015cea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:30.160 INFO 21062 --- [ocp-schedules-8,5879364ad9bbdddd,239795b51412115c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:05:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:05:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:05:31.037 INFO 21062 --- [ocp-schedules-19,e019575d8fbeaba5,0b25b955ed544b67] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:05:31.046 INFO 21062 --- [ocp-schedules-19,e019575d8fbeaba5,0b25b955ed544b67] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:05:31.046 INFO 21062 --- [ocp-schedules-19,e019575d8fbeaba5,0b25b955ed544b67] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:05:31.059 INFO 21062 --- [ocp-schedules-13,ca29d3f9aebcff39,04c1eb4148931ff5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:05:31.059 INFO 21062 --- [ocp-schedules-13,ca29d3f9aebcff39,04c1eb4148931ff5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:05:31.080 WARN 21062 --- [ocp-async-8,d2e69a03f0724904,4a9d93064c7f3789] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:31.087 INFO 21062 --- [ocp-schedules-13,ca29d3f9aebcff39,04c1eb4148931ff5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:05:31.087 INFO 21062 --- [ocp-schedules-13,ca29d3f9aebcff39,04c1eb4148931ff5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:05:31.087 INFO 21062 --- [ocp-schedules-17,4dbc7724641e01ed,930bdbb60ff1f0b8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:05:31.117 INFO 21062 --- [ocp-async-3,3a749eae0b903ab5,a3e94398e0aba881] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:31.259 INFO 21062 --- [ocp-schedules-8,07865e6bca89a251,84ae0153c2cfa87e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:05:31.259+08:00, clusters=[1] 2024-11-04 14:05:31.261 INFO 21062 --- [ocp-schedules-8,07865e6bca89a251,84ae0153c2cfa87e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:05:31.335 INFO 21062 --- [ocp-schedules-26,a3ccc353ac3f7139,bd7763c58c56144e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:05:31.335 INFO 21062 --- [ocp-schedules-26,a3ccc353ac3f7139,bd7763c58c56144e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:05:31.335 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:05:31.335 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:05:31.345 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:05:31.345 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:05:31.345 INFO 21062 --- [ocp-schedules-26,a3ccc353ac3f7139,bd7763c58c56144e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:05:31.348 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:05:31.348 WARN 21062 --- [ocp-schedules-26,a3ccc353ac3f7139,bd7763c58c56144e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:05:31.348 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:05:31.349 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:05:31.349 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:05:31.349 INFO 21062 --- [ocp-schedules-26,a3ccc353ac3f7139,bd7763c58c56144e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:05:31.350 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:05:31.350 WARN 21062 --- [ocp-schedules-26,a3ccc353ac3f7139,bd7763c58c56144e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:05:31.350 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:05:31.359 ERROR 21062 --- [ocp-schedules-26,a3ccc353ac3f7139,bd7763c58c56144e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 14:05:31.378 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:31.379 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:31.405 INFO 21062 --- [ocp-schedules-9,0345524939899da1,9dc9a1d18ba6bd8e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:05:31.405+08:00, clusters=[1] 2024-11-04 14:05:31.409 INFO 21062 --- [ocp-schedules-9,0345524939899da1,9dc9a1d18ba6bd8e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:05:32.983 INFO 21062 --- [prometheus-minute-schedule-1,6639a8d09b9c74f7,1994fde293d9fc34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:33.080 WARN 21062 --- [ocp-async-3,c6c2ad5b0579c0fa,47c22ea99e900250] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:33.117 INFO 21062 --- [ocp-async-7,32e833e4d871aa90,3f657f0ec2c8f901] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:34.379 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:34.379 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:34.885 INFO 21062 --- [ocp-schedules-12,c9cac4cda7cc8a68,e16cd9e4f784deff] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:05:34.885 INFO 21062 --- [ocp-schedules-12,ec257c48ca9f992f,b98f342e6967f4bd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:05:34.888 INFO 21062 --- [ocp-schedules-12,ec257c48ca9f992f,b98f342e6967f4bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:34.917 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:05:34.917 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:05:35.081 WARN 21062 --- [ocp-async-7,e657705ad1ca2e0d,9a3f3e79585a469d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:35.117 INFO 21062 --- [ocp-async-10,4059ba06f03c2570,f6460e32ba541985] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c4bdd3a89a10f203,e3c5888a5959a68a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:37.081 WARN 21062 --- [ocp-async-9,6a6ad22fee458b0e,ef493095f3955ba6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:37.104 INFO 21062 --- [ocp-schedules-23,22cf9520304e90ba,0140e05ff3e051bc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:05:37.108 INFO 21062 --- [ocp-schedules-23,22cf9520304e90ba,0140e05ff3e051bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:37.117 INFO 21062 --- [ocp-async-4,6d4f1daf35b80ae3,df821a9d706b1ebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:37.193 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:05:37.195 ERROR 21062 --- [ocp-schedules-23,22cf9520304e90ba,0140e05ff3e051bc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:05:37.196 INFO 21062 --- [ocp-schedules-23,22cf9520304e90ba,0140e05ff3e051bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:05:37.380 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:37.380 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:38.504 INFO 21062 --- [ocp-schedules-4,81472aedf28a73c1,99e55ca39fda7859] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:05:38.504 INFO 21062 --- [ocp-schedules-2,78363dd97e596ab4,e916a12052e653c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:05:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:05:38.590993+08:00, lastProcessTime=2024-11-04T14:05:28.590043+08:00 2024-11-04 14:05:38.639 INFO 21062 --- [ocp-async-9,7c723117ccb214d1,6b0f21da042e6cdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:05:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:05:38.590993+08:00 2024-11-04 14:05:38.936 INFO 21062 --- [ocp-schedules-1,d46c38e121788a46,d2d72c4494c25027] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:05:38.940 INFO 21062 --- [ocp-schedules-1,d46c38e121788a46,d2d72c4494c25027] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:38.940 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:05:38.983 INFO 21062 --- [prometheus-minute-schedule-1,25bbcf84807cd9f9,9a097b82493bf7ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:39.081 WARN 21062 --- [ocp-async-5,1649acf8cdba4bee,9bd8313e4a56136f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:39.117 INFO 21062 --- [ocp-async-6,e21b3723c010225c,f5aed5aefc267d1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:39.393 INFO 21062 --- [ocp-schedules-27,d44f6c5a2d1a1285,d2b3ce91fe3b688c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:05:40.002 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:05:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:05:40.162 INFO 21062 --- [ocp-schedules-11,64cde52c1cb74918,91ecf2074ddb428e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:05:40.380 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:40.381 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:41.081 WARN 21062 --- [ocp-async-7,f6496491ef40f7c2,2b72ac82436e6fe8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:41.118 INFO 21062 --- [ocp-async-9,956e5bfd6915ea26,6fd51aeed843cd34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:41.983 INFO 21062 --- [prometheus-minute-schedule-1,85be8bcebbb06d7d,9199293d7a320b6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:43.081 WARN 21062 --- [ocp-async-8,9e5fa3dfcb0d0d41,b746175a0c8d6f72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:43.118 INFO 21062 --- [ocp-async-1,b7b9f41fe0dc7c59,e4e8e79131586644] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:43.381 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:43.382 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f3bc135a09821876,9eb7a67b21ef6d1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:45.081 WARN 21062 --- [ocp-async-3,ee38ab833ae76943,46572fcbce009400] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:45.118 INFO 21062 --- [ocp-async-2,b37f12b6ff15c2a4,6332ca8c5cad0d4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:46.382 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:46.383 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:47.082 WARN 21062 --- [ocp-async-6,e85acb4b327b367e,7c81a21fea7a89a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:47.118 INFO 21062 --- [ocp-async-10,ef3c6318e1dec997,ceeec7581a9b8805] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:47.983 INFO 21062 --- [prometheus-minute-schedule-1,13f7780c76fe48b9,1e955b557c4d03a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:05:48.591331+08:00, lastProcessTime=2024-11-04T14:05:38.590993+08:00 2024-11-04 14:05:48.639 INFO 21062 --- [ocp-async-5,97451a635c1be6b8,aa55b389bef23058] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:05:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:05:48.591331+08:00 2024-11-04 14:05:48.941 INFO 21062 --- [ocp-schedules-1,d46c38e121788a46,d2d72c4494c25027] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:05:49.082 WARN 21062 --- [ocp-async-8,184146ddbe55fa5d,73cb801bb40944b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:49.118 INFO 21062 --- [ocp-async-1,2b659e71bc6483dd,e8860c1f8f2bce89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:49.383 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:49.383 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:49.418 INFO 21062 --- [ocp-schedules-21,abaee28ca2bb09d4,6da79df4e04fc257] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:05:50.165 INFO 21062 --- [ocp-schedules-16,931b28cb70d50dd3,a38747b2e2e3a149] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:05:50.983 INFO 21062 --- [prometheus-minute-schedule-1,94e53dbc69cbcee6,c97c7446c2a78c7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:51.048 INFO 21062 --- [ocp-schedules-15,d2fc8905cd2e13ba,2a6f642721c5af1c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:05:51.054 INFO 21062 --- [ocp-schedules-15,d2fc8905cd2e13ba,2a6f642721c5af1c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:05:51.054 INFO 21062 --- [ocp-schedules-15,d2fc8905cd2e13ba,2a6f642721c5af1c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:05:51.082 WARN 21062 --- [ocp-async-3,8a1055bcd9df2f6c,a64628c2c5740821] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:51.119 INFO 21062 --- [ocp-async-2,a31e924baef1cddc,b1899c6b3122b3f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:52.384 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:52.384 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:53.082 WARN 21062 --- [ocp-async-6,d4925fbf13411dd0,15305eea7e89505c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:53.119 INFO 21062 --- [ocp-async-10,c2c8cc7badd33585,5a5a5422a754bce4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:53.983 INFO 21062 --- [prometheus-minute-schedule-1,47146a727a66574c,1ae0b9d12dff768e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:55.082 WARN 21062 --- [ocp-async-9,82b9138f69e0bc34,9457be3a64a7d2f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:55.119 INFO 21062 --- [ocp-async-4,7b7035595ec9a3cf,d4062c5ef709b10c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:55.385 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:55.385 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ff60a14c7a04d975,1be91b0c6bd5a1d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:05:57.082 WARN 21062 --- [ocp-async-1,ec93443284931499,9d3ed77d570002b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:57.119 INFO 21062 --- [ocp-async-5,1a303f104a65a877,86a3f67efe007934] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:58.385 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:05:58.386 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:05:58.500 INFO 21062 --- [ocp-schedules-2,3205f498e40f1122,18f5a196a0a91156] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:05:58.500 INFO 21062 --- [ocp-schedules-8,c9c8059c536c3e9f,f0ba5b270ea9b984] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:05:58.502 INFO 21062 --- [ocp-schedules-14,91c42732b5e3baea,7e1f0b4bf83a0f2d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:05:58.504 INFO 21062 --- [ocp-schedules-8,c9c8059c536c3e9f,f0ba5b270ea9b984] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:58.504 INFO 21062 --- [ocp-schedules-2,3205f498e40f1122,18f5a196a0a91156] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:05:58.504 INFO 21062 --- [ocp-schedules-19,398fe0d291ec0565,b09040daaca732fc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:05:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:05:58.511 INFO 21062 --- [ocp-schedules-23,556a3971aa851a9f,94a9b3f9099384f7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:05:58.511 INFO 21062 --- [ocp-schedules-15,d6a76eae8b4bf254,8db1d4de0e59e504] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:05:58.511 INFO 21062 --- [ocp-schedules-23,556a3971aa851a9f,94a9b3f9099384f7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:05:58.511 INFO 21062 --- [ocp-schedules-15,d6a76eae8b4bf254,8db1d4de0e59e504] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:05:58.512 INFO 21062 --- [ocp-schedules-30,3f7fd1d23f36640f,49f5a1be5cadc997] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:05:58.512 INFO 21062 --- [ocp-schedules-30,3f7fd1d23f36640f,49f5a1be5cadc997] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:05:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:05:58.589731+08:00, lastProcessTime=2024-11-04T14:05:48.591331+08:00 2024-11-04 14:05:58.640 INFO 21062 --- [ocp-async-8,1ba95b9cbc111a73,21b18a73ba0d4e84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:05:58.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:05:58.589731+08:00 2024-11-04 14:05:59.067 INFO 21062 --- [ocp-schedules-5,fce48c7d17338dff,4cf8089dc822bb7f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:05:59.082 WARN 21062 --- [ocp-async-6,9086165b6acc9dc4,1badf8f2bc0c34c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:05:59.119 INFO 21062 --- [ocp-async-10,bda0b062475c928b,4bcd3c188717d17a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:05:59.464 INFO 21062 --- [ocp-schedules-14,89b5875f51331864,54778cbbab39f2ba] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:05:59.983 INFO 21062 --- [prometheus-minute-schedule-1,97da73f7284faf75,a8a7deeba4ede451] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700360, interval=60 2024-11-04 14:06:00.168 INFO 21062 --- [ocp-schedules-28,02b4350561f008bd,28c08244f7f9a957] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:06:01.083 WARN 21062 --- [ocp-async-9,c05cf312f25400fc,22c6a54b34863dca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:01.120 INFO 21062 --- [ocp-async-4,dd5eb429f92b8758,a338c5deba056f6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:01.387 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:01.387 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5904e125f981c1f3,d2fa4606d737776c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:03.083 WARN 21062 --- [ocp-async-1,d350e7c9d8911665,d9c07126f0793da2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:03.120 INFO 21062 --- [ocp-async-5,51f194ded2a5ffe6,110dc045540e6ecf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:04.388 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:04.388 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:04.918 INFO 21062 --- [ocp-schedules-12,ec257c48ca9f992f,b98f342e6967f4bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:06:04.918 INFO 21062 --- [ocp-schedules-12,ee07d01c549dc401,f9599789f1fb4707] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:06:04.923 INFO 21062 --- [ocp-schedules-12,ee07d01c549dc401,f9599789f1fb4707] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:04.956 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:06:04.956 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:06:05.083 WARN 21062 --- [ocp-async-5,7fe6af1cfe47b43a,dd35271b938b43e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:05.120 INFO 21062 --- [ocp-async-7,d50de57fda0da4c8,c31cb37f48552c79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:05.983 INFO 21062 --- [prometheus-minute-schedule-1,093a7778e54c4eda,7595b44ddd7151c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:07.083 WARN 21062 --- [ocp-async-7,eecbeeac33d8de87,b03ede76445d13f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:07.120 INFO 21062 --- [ocp-async-8,04dde135ff070226,a38d542f5621dab8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:07.389 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:07.389 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:08.505 INFO 21062 --- [ocp-schedules-8,c9c8059c536c3e9f,f0ba5b270ea9b984] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:06:08.505 INFO 21062 --- [ocp-schedules-2,3205f498e40f1122,18f5a196a0a91156] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:06:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:06:08.590136+08:00, lastProcessTime=2024-11-04T14:05:58.589731+08:00 2024-11-04 14:06:08.640 INFO 21062 --- [ocp-async-6,fb22b4662b3f2bf9,a6eec6deb835674a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:06:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:06:08.590136+08:00 2024-11-04 14:06:08.983 INFO 21062 --- [prometheus-minute-schedule-1,4df44fd4516b5ad0,27d7a042b889494b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:09.083 WARN 21062 --- [ocp-async-4,750cb018649a8c67,2c53f28d35daaa01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:09.120 INFO 21062 --- [ocp-async-5,2f92603210d5b27c,4363ddd5cab1ab80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:09.489 INFO 21062 --- [ocp-schedules-11,a8ec87ecc6db55b4,4415eb74a221fd9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:06:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:06:10.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:06:10.171 INFO 21062 --- [ocp-schedules-29,2583859991746d65,b4eecfc2a0447598] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:06:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:06:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:06:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:06:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:06:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:06:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:06:10.389 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:10.390 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:11.056 INFO 21062 --- [ocp-schedules-25,c7c1b7d4e2bbfbc7,68cca0e946acee0d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:06:11.062 INFO 21062 --- [ocp-schedules-25,c7c1b7d4e2bbfbc7,68cca0e946acee0d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:06:11.062 INFO 21062 --- [ocp-schedules-25,c7c1b7d4e2bbfbc7,68cca0e946acee0d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:06:11.084 WARN 21062 --- [ocp-async-5,45430acd48d41b48,bf2a52ac2d14403d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:11.120 INFO 21062 --- [ocp-async-7,625bb0bcc1ac9522,3f8112c04c790533] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=19, affectRows=19 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=49, affectRows=49 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=53, affectRows=53 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=46, affectRows=46 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=15, affectRows=15 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=27, affectRows=27 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=19, validCount=19 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=15, validCount=15 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=54, affectRows=54 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=49, validCount=49 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=46, validCount=46 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=52, validCount=52 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=56, affectRows=56 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=27, validCount=27 2024-11-04 14:06:11.454 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700300, nodeCount=73, affectRows=73 2024-11-04 14:06:11.455 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=54, validCount=54 2024-11-04 14:06:11.455 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=54, validCount=54 2024-11-04 14:06:11.455 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700300, nodeCount=73, validCount=73 2024-11-04 14:06:11.983 INFO 21062 --- [prometheus-minute-schedule-1,165a691e63f4b44a,505b0df66de07969] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:13.084 WARN 21062 --- [ocp-async-7,4ecbb78785bf1e7e,84149f8e42d30dc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:13.121 INFO 21062 --- [ocp-async-8,3328576dd1b42a17,7dc99a09abb11376] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:13.390 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:13.391 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:13.984 INFO 21062 --- [prometheus-cache-monitor1,b17fe5f4fc2bf2c0,a2c5c3039a6c9d17] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:06:13.984 INFO 21062 --- [prometheus-cache-monitor1,b17fe5f4fc2bf2c0,a2c5c3039a6c9d17] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:06:14.983 INFO 21062 --- [prometheus-minute-schedule-1,0163a25a80193889,abe339c6eaefd231] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:15.084 WARN 21062 --- [ocp-async-8,2e3124221242fa91,76b6909ec916d3ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:15.121 INFO 21062 --- [ocp-async-1,e225aa6a01c13792,42957400984a9d88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:16.391 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:16.392 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:17.084 WARN 21062 --- [ocp-async-1,f1022c9f739d1ea9,87894e0fcb25d32d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:17.121 INFO 21062 --- [ocp-async-6,04bcf3fcf99bd505,ef80f486fb62b6f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:17.983 INFO 21062 --- [prometheus-minute-schedule-1,003196b296484361,6b62743b1181bfef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:06:18.590346+08:00, lastProcessTime=2024-11-04T14:06:08.590136+08:00 2024-11-04 14:06:18.640 INFO 21062 --- [ocp-async-3,1f303f2a5ef1faaa,2446368dafe290a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:06:18.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:06:18.590346+08:00 2024-11-04 14:06:19.084 WARN 21062 --- [ocp-async-7,cd63f7827042c2b6,533cdc5f1b590a09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:19.121 INFO 21062 --- [ocp-async-8,2fa9132bfc399e03,2f3b38fe343e7aea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:19.392 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:19.393 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:19.533 INFO 21062 --- [ocp-schedules-1,51bdc10211354863,b44eef3c3aedc352] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:06:20.173 INFO 21062 --- [ocp-schedules-10,b7db65c6b8981fd1,ee5cabaf78f3fc47] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:06:20.983 INFO 21062 --- [prometheus-minute-schedule-1,33d9a860a957a5a4,0236235a1a4f4821] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:21.084 WARN 21062 --- [ocp-async-8,f518ca3bdf83cdc1,f2ded856d16aba77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:21.121 INFO 21062 --- [ocp-async-1,6972395a35e38a0c,db3688a0c8d2b71f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:22.393 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:22.394 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:23.085 WARN 21062 --- [ocp-async-1,69104aae4aee1add,0190abe5d5048491] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:23.121 INFO 21062 --- [ocp-async-6,ddc6888a2b55b8d2,b62bd26065e0a912] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:23.983 INFO 21062 --- [prometheus-minute-schedule-1,2f9dcaeaf1f6df4e,7928c601cbd299f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:25.085 WARN 21062 --- [ocp-async-6,d4944a9e2b461c1b,51d95bfdc2485dbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:25.122 INFO 21062 --- [ocp-async-10,7e6bb7218b68f6a6,2aa4cdbd55a76992] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:25.394 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:25.395 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f3d03efbaeb8785f,15b3229c05078e58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:26.984 INFO 21062 --- [prometheus-hour-schedule-1,17b850d9bc01e119,cab0a01a360df2b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:06:27.085 WARN 21062 --- [ocp-async-10,433860da8f3ed129,b6b6674b44773845] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:27.122 INFO 21062 --- [ocp-async-4,41c23d21a36da029,606c5c302f898e28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:28.395 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:28.397 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:28.498 INFO 21062 --- [ocp-schedules-28,514e3999023e4c5e,f9350835b9b2c8a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:06:28.499 INFO 21062 --- [ocp-schedules-15,509618f8139ca534,fa73c1c9196637c6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:06:28.500 INFO 21062 --- [ocp-schedules-3,2f4c604628a8d9e0,4b0e744ff42fa94d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:06:28.500 INFO 21062 --- [ocp-schedules-21,1d1a7f2788ca1121,c866e092ab8ebb8b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:06:28.502 INFO 21062 --- [ocp-schedules-28,514e3999023e4c5e,f9350835b9b2c8a3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:06:28.502 INFO 21062 --- [ocp-schedules-2,84719cb3ac343534,b7f6a1d2b28da15a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:06:28.504 INFO 21062 --- [ocp-schedules-21,1d1a7f2788ca1121,c866e092ab8ebb8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:28.504 INFO 21062 --- [ocp-schedules-3,2f4c604628a8d9e0,4b0e744ff42fa94d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:28.504 INFO 21062 --- [ocp-schedules-25,71200215db703671,81dffd1df8d5270b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:06:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:06:28.506 INFO 21062 --- [ocp-schedules-15,509618f8139ca534,fa73c1c9196637c6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:06:28.510 INFO 21062 --- [ocp-schedules-19,7cecc56b677f707e,e015e8a3ed0ce0aa] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:06:28.511 INFO 21062 --- [ocp-schedules-19,7cecc56b677f707e,e015e8a3ed0ce0aa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:06:28.511 INFO 21062 --- [ocp-schedules-5,295e176b671ac63d,c472ed1e45429fb4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:06:28.511 INFO 21062 --- [ocp-schedules-5,295e176b671ac63d,c472ed1e45429fb4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:06:28.511 INFO 21062 --- [ocp-schedules-23,268ecfa919e2da0c,bceaf8ddf5d9f96a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:06:28.511 INFO 21062 --- [ocp-schedules-23,268ecfa919e2da0c,bceaf8ddf5d9f96a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:06:28.573 INFO 21062 --- [ocp-schedules-20,f5ad571c17af4656,996ad5a692bef4ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:06:28.573 INFO 21062 --- [ocp-schedules-20,f5ad571c17af4656,996ad5a692bef4ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:06:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:06:28.589673+08:00, lastProcessTime=2024-11-04T14:06:18.590346+08:00 2024-11-04 14:06:28.619 INFO 21062 --- [ocp-schedules-13,dc757234c717be8f,a8b962aba41f4e77] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:06:28.620 INFO 21062 --- [ocp-schedules-13,dc757234c717be8f,a8b962aba41f4e77] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:06:28.640 INFO 21062 --- [ocp-async-8,b7d5753810124764,053355e7dfa447c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:06:28.748 INFO 21062 --- [ocp-schedules-1,8e86d8b26f472c49,60987a9d478d5413] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:06:28.748 INFO 21062 --- [ocp-schedules-1,8e86d8b26f472c49,60987a9d478d5413] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:06:28.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:06:28.589673+08:00 2024-11-04 14:06:28.787 INFO 21062 --- [ocp-schedules-17,3d6e556088669487,db40e4c75b59850f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:06:29.015 INFO 21062 --- [ocp-schedules-25,0e542975cc3abd2d,60bfdb374d968e66] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:06:29.015 INFO 21062 --- [ocp-schedules-25,0e542975cc3abd2d,60bfdb374d968e66] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:06:29.069 INFO 21062 --- [ocp-schedules-5,4f7c32abe61dcadb,37f9e1fd09d73056] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:06:29.085 WARN 21062 --- [ocp-async-2,43ebafd5e04b7cf5,51680f0f324a8e4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:29.102 INFO 21062 --- [ocp-schedules-11,3145bf3df0e383d6,0f933ea0f50ce6d9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:06:29.108 INFO 21062 --- [ocp-schedules-11,3145bf3df0e383d6,0f933ea0f50ce6d9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:06:29.108 INFO 21062 --- [ocp-schedules-11,3145bf3df0e383d6,0f933ea0f50ce6d9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:06:29.122 INFO 21062 --- [ocp-async-6,3fefaae643294888,783fe703dba64cdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:29.390 INFO 21062 --- [ocp-schedules-4,c61741dd98e9e77f,e18bbb610c31a4df] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:06:29.390 INFO 21062 --- [ocp-schedules-4,c61741dd98e9e77f,e18bbb610c31a4df] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:06:29.392 INFO 21062 --- [ocp-schedules-4,c61741dd98e9e77f,e18bbb610c31a4df] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:06:29.392 INFO 21062 --- [ocp-schedules-4,c61741dd98e9e77f,e18bbb610c31a4df] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:06:29.394 INFO 21062 --- [ocp-schedules-4,c61741dd98e9e77f,e18bbb610c31a4df] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:06:29.394 INFO 21062 --- [ocp-schedules-4,c61741dd98e9e77f,e18bbb610c31a4df] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:06:29.559 INFO 21062 --- [ocp-schedules-29,9a78dc75f0275118,adf6b47a6d185f10] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:06:29.983 INFO 21062 --- [prometheus-minute-schedule-1,ddec627b2dfcd11e,2f1429e4ecab3d97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:30.176 INFO 21062 --- [ocp-schedules-26,3a7fd297d5bf38ae,543db3dd9b2a90a9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:06:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:06:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:06:31.064 INFO 21062 --- [ocp-schedules-1,0c3cf96c8aad31a6,5d9b2ac6ec60ca32] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:06:31.070 INFO 21062 --- [ocp-schedules-1,0c3cf96c8aad31a6,5d9b2ac6ec60ca32] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:06:31.070 INFO 21062 --- [ocp-schedules-1,0c3cf96c8aad31a6,5d9b2ac6ec60ca32] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:06:31.085 WARN 21062 --- [ocp-async-6,37fb660fe7776f42,c392236afb8f837f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:31.087 INFO 21062 --- [ocp-schedules-2,fa4acb86b2817d1f,f12446f1c160779a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:06:31.087 INFO 21062 --- [ocp-schedules-2,fa4acb86b2817d1f,f12446f1c160779a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:06:31.115 INFO 21062 --- [ocp-schedules-2,fa4acb86b2817d1f,f12446f1c160779a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:06:31.115 INFO 21062 --- [ocp-schedules-2,fa4acb86b2817d1f,f12446f1c160779a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:06:31.115 INFO 21062 --- [ocp-schedules-15,73a9f85a2f8af411,0be3b5a35f2b8808] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:06:31.122 INFO 21062 --- [ocp-async-8,21c5bf97ca9c9897,041151bfed35f1f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:31.198 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:06:31.198 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:06:31.206 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:06:31.206 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:06:31.206 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:06:31.207 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:06:31.211 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:06:31.211 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:06:31.211 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:06:31.211 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:06:31.252 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:06:31.261 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:06:31.261 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:06:31.261 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:06:31.262 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:06:31.262 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:06:31.271 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:06:31.287 INFO 21062 --- [ocp-schedules-23,8943437678a5a7b4,982242f97c9cdc83] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:06:31.287+08:00, clusters=[1] 2024-11-04 14:06:31.289 INFO 21062 --- [ocp-schedules-23,8943437678a5a7b4,982242f97c9cdc83] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:06:31.367 INFO 21062 --- [ocp-schedules-7,3bf7853fa98d34b1,8074f8f87e6aba05] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:06:31.367 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:06:31.367 INFO 21062 --- [ocp-schedules-7,3bf7853fa98d34b1,8074f8f87e6aba05] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:06:31.367 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:06:31.377 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:06:31.377 INFO 21062 --- [ocp-schedules-7,3bf7853fa98d34b1,8074f8f87e6aba05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:06:31.377 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:06:31.379 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:06:31.379 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:06:31.379 WARN 21062 --- [ocp-schedules-7,3bf7853fa98d34b1,8074f8f87e6aba05] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:06:31.379 INFO 21062 --- [ocp-schedules-7,3bf7853fa98d34b1,8074f8f87e6aba05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:06:31.379 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:06:31.379 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:06:31.381 WARN 21062 --- [ocp-schedules-7,3bf7853fa98d34b1,8074f8f87e6aba05] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:06:31.381 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:06:31.381 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:06:31.388 INFO 21062 --- [ocp-schedules-13,98a67fa3f3e65a9e,54c1012d1788940b] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:06:31.389 ERROR 21062 --- [ocp-schedules-7,3bf7853fa98d34b1,8074f8f87e6aba05] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 14:06:31.392 INFO 21062 --- [ocp-schedules-13,98a67fa3f3e65a9e,54c1012d1788940b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:31.397 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:31.398 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:31.432 INFO 21062 --- [ocp-schedules-4,a4e238e86e3ff49e,7e4033933971d451] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:06:31.432+08:00, clusters=[1] 2024-11-04 14:06:31.435 INFO 21062 --- [ocp-schedules-4,a4e238e86e3ff49e,7e4033933971d451] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:06:31.446 ERROR 21062 --- [ocp-schedules-13,98a67fa3f3e65a9e,54c1012d1788940b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:06:31.446 INFO 21062 --- [ocp-schedules-13,98a67fa3f3e65a9e,54c1012d1788940b] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:06:31.805 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:06:32.368 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.156291618S 2024-11-04 14:06:32.368 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:06:32.473 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:06:32.473 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:06:32.500 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:06:32.564 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:06:32.564 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:06:32.564 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:06:32.564 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:06:32.564 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:06:32.565 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:06:32.577 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:06:32.759 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] o.s.boot.SpringApplication : Started application in 1.552 seconds (JVM running for 5806.438) 2024-11-04 14:06:32.759 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.553032404S 2024-11-04 14:06:32.983 INFO 21062 --- [prometheus-minute-schedule-1,ed4b5a44889a2508,ec25155e26c013f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:33.086 WARN 21062 --- [ocp-async-8,78ccc9205b686c73,c649c64746c6e4f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:33.122 INFO 21062 --- [ocp-async-1,1ce4f285de0c914e,aa27d8b1aa6a7b13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:33.222 INFO 21062 --- [ocp-schedules-14,e4e9a6fc61aebf79,7beb8cc2c7cadfed] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:06:34.398 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:34.400 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:34.957 INFO 21062 --- [ocp-schedules-12,ee07d01c549dc401,f9599789f1fb4707] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:06:34.957 INFO 21062 --- [ocp-schedules-12,c656241274dbfc1c,e0143478a388da17] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:06:34.961 INFO 21062 --- [ocp-schedules-12,c656241274dbfc1c,e0143478a388da17] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:34.990 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:06:34.990 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:06:35.086 WARN 21062 --- [ocp-async-1,081557cd886d6530,164dcfd786963437] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:35.122 INFO 21062 --- [ocp-async-7,5ab18f79080f5ce0,340641b0ea6eed23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:35.983 INFO 21062 --- [prometheus-minute-schedule-1,790ef7e6b8464137,524dd0ee2bbf9559] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:37.086 WARN 21062 --- [ocp-async-9,fc485eb5877427a6,be02454e8da36926] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:37.123 INFO 21062 --- [ocp-async-10,44cc0c3045d023e2,a0a92d10c838ca62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:37.196 INFO 21062 --- [ocp-schedules-29,adf9d107ab9a1389,d5a02e4f5bb7a979] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:06:37.200 INFO 21062 --- [ocp-schedules-29,adf9d107ab9a1389,d5a02e4f5bb7a979] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:37.285 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:06:37.287 ERROR 21062 --- [ocp-schedules-29,adf9d107ab9a1389,d5a02e4f5bb7a979] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:06:37.287 INFO 21062 --- [ocp-schedules-29,adf9d107ab9a1389,d5a02e4f5bb7a979] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:06:37.400 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:37.400 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:38.505 INFO 21062 --- [ocp-schedules-21,1d1a7f2788ca1121,c866e092ab8ebb8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:06:38.505 INFO 21062 --- [ocp-schedules-3,2f4c604628a8d9e0,4b0e744ff42fa94d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:06:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:06:38.590338+08:00, lastProcessTime=2024-11-04T14:06:28.589673+08:00 2024-11-04 14:06:38.640 INFO 21062 --- [ocp-async-7,a16ff27f52120d92,02de0fb19f572f65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:06:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:06:38.590338+08:00 2024-11-04 14:06:38.983 INFO 21062 --- [prometheus-minute-schedule-1,b8e4600ec848b7b8,8d16b19a279ef18a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:39.086 WARN 21062 --- [ocp-async-4,e51892a73142cbae,f6bb6898f87348b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:39.123 INFO 21062 --- [ocp-async-2,7cd9e7938bb47608,9ff61aa9ed407189] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:39.600 INFO 21062 --- [ocp-schedules-17,8d9c622bbf960c98,258027849d87a02b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:06:40.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:06:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:06:40.180 INFO 21062 --- [ocp-schedules-11,e6282c0d270ec2aa,1344f88e1fff2bc4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:06:40.400 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:40.402 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:41.086 WARN 21062 --- [ocp-async-1,90eaa6e78e54aab4,19708c8e18cd3651] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:41.123 INFO 21062 --- [ocp-async-7,ad0b3e9162d64a01,3d85ee47d1660e8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:06:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:06:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:06:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:06:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:06:41.853 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:06:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:06:41.983 INFO 21062 --- [prometheus-minute-schedule-1,8359ffff1a25ddcb,2a41ccc4de085850] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:43.087 WARN 21062 --- [ocp-async-6,8996950272f5de64,828ccb66784009fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:43.123 INFO 21062 --- [ocp-async-3,deca7dccd5962848,a72e95f448fdd8eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:43.402 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:43.403 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:44.983 INFO 21062 --- [prometheus-minute-schedule-1,eec50d8a2e6ddc86,aa13e52692d0b06f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:45.087 WARN 21062 --- [ocp-async-8,bc5708233b845c56,f5928a2ca1e83da3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:45.124 INFO 21062 --- [ocp-async-5,a85a7c0cf71e66cb,796e226cac466f85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:46.403 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:46.404 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:47.087 WARN 21062 --- [ocp-async-2,fff1b4bf3a9ccc0d,ead9a38ccacdc284] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:47.124 INFO 21062 --- [ocp-async-9,0eee750a6221d94e,a4a1eb92f5de7e4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:47.983 INFO 21062 --- [prometheus-minute-schedule-1,93ef9c67e27bb419,f4246bef2bd29c27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:06:48.589906+08:00, lastProcessTime=2024-11-04T14:06:38.590338+08:00 2024-11-04 14:06:48.640 INFO 21062 --- [ocp-async-4,a9a146bbbb043f0a,f452b8387dd66360] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:06:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:06:48.589906+08:00 2024-11-04 14:06:48.941 INFO 21062 --- [ocp-schedules-17,9db2c4404dece02e,b8e64a627d4e1e96] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:06:48.944 INFO 21062 --- [ocp-schedules-17,9db2c4404dece02e,b8e64a627d4e1e96] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:48.945 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:06:49.087 WARN 21062 --- [ocp-async-6,68a97c5f4d6745af,2aad5d0e88d438c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:49.124 INFO 21062 --- [ocp-async-3,09ef34ad84f40914,82242f45458b78fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:49.405 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:49.405 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:49.627 INFO 21062 --- [ocp-schedules-18,2c17a6f3ff5dde8c,623ef3ce9cf1e157] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:06:50.184 INFO 21062 --- [ocp-schedules-15,43d51047062ffa9a,58a4062b9eb0aad1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:06:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d3fcfc133ce40192,5f96300759b1f1fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:51.073 INFO 21062 --- [ocp-schedules-2,e8bc01bb6c0b890b,114162d4c2d74f44] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:06:51.080 INFO 21062 --- [ocp-schedules-2,e8bc01bb6c0b890b,114162d4c2d74f44] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:06:51.080 INFO 21062 --- [ocp-schedules-2,e8bc01bb6c0b890b,114162d4c2d74f44] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:06:51.087 WARN 21062 --- [ocp-async-8,c896c6f268a68fd1,f03f2d4974308a7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:51.124 INFO 21062 --- [ocp-async-5,ff03a0f66d095d12,761ad21f0a9e796a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:52.406 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:52.407 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:53.087 WARN 21062 --- [ocp-async-2,e386da36e7fb0314,1526d2bb2345e9c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:53.124 INFO 21062 --- [ocp-async-9,c8c436b8e6d155f2,e337999db53be952] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:53.983 INFO 21062 --- [prometheus-minute-schedule-1,995b718794e5d419,7d2a639f4edaa865] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:55.088 WARN 21062 --- [ocp-async-7,11ec010e8ff77ad9,8a2419a547a0f5da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:55.124 INFO 21062 --- [ocp-async-10,3507d1c9dc6c1474,c1d9935978148608] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:55.407 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:55.408 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:56.983 INFO 21062 --- [prometheus-minute-schedule-1,1e3e82af8316198a,1ac4cee73a8f6d23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:06:57.088 WARN 21062 --- [ocp-async-3,08c1a50d74d7c838,147a7b77850b8787] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:57.125 INFO 21062 --- [ocp-async-4,6fe9e5f4d5c98104,10c9511950367bf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:58.408 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:06:58.410 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:06:58.499 INFO 21062 --- [ocp-schedules-9,25b0c9891b1d5815,7e712995c894f849] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:06:58.500 INFO 21062 --- [ocp-schedules-24,d2836dcac839c532,cc2ff9a7c2f90dcc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:06:58.503 INFO 21062 --- [ocp-schedules-18,930ac6b288ad0607,495e4721ac080de7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:06:58.504 INFO 21062 --- [ocp-schedules-9,25b0c9891b1d5815,7e712995c894f849] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:58.504 INFO 21062 --- [ocp-schedules-24,d2836dcac839c532,cc2ff9a7c2f90dcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:06:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:06:58.505 INFO 21062 --- [ocp-schedules-8,a7b72ba3b6ef87f4,a1084cbedf548299] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:06:58.510 INFO 21062 --- [ocp-schedules-16,0abdd063f478c161,2a568a4e2c1bdf45] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:06:58.510 INFO 21062 --- [ocp-schedules-30,444b107e07b049ec,35882032db50fcdb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:06:58.510 INFO 21062 --- [ocp-schedules-16,0abdd063f478c161,2a568a4e2c1bdf45] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:06:58.510 INFO 21062 --- [ocp-schedules-30,444b107e07b049ec,35882032db50fcdb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:06:58.510 INFO 21062 --- [ocp-schedules-26,898d7a916ae443f4,50315198b50b293b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:06:58.510 INFO 21062 --- [ocp-schedules-26,898d7a916ae443f4,50315198b50b293b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:06:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:06:58.588904+08:00, lastProcessTime=2024-11-04T14:06:48.589906+08:00 2024-11-04 14:06:58.641 INFO 21062 --- [ocp-async-6,72ce1fb3d1edc992,c157b73648c9ace5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:06:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:06:58.588904+08:00 2024-11-04 14:06:58.945 INFO 21062 --- [ocp-schedules-17,9db2c4404dece02e,b8e64a627d4e1e96] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:06:59.072 INFO 21062 --- [ocp-schedules-14,a5e6eddb0b6f29b9,123f328c3351bf60] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:06:59.088 WARN 21062 --- [ocp-async-2,4d64f9bbc3d17a4e,d6f66b2e1a7aab9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:06:59.125 INFO 21062 --- [ocp-async-9,66975c0cdb894200,92fb16917697b784] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:06:59.670 INFO 21062 --- [ocp-schedules-18,c3acdc11f9320891,d23e7199b3e0d8d2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:06:59.983 INFO 21062 --- [prometheus-minute-schedule-1,cb0eaaeb922069d9,289b8f30dc649012] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700420, interval=60 2024-11-04 14:07:00.187 INFO 21062 --- [ocp-schedules-6,546390b1c09c5632,8324eeb8eb4fb4ef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:07:01.088 WARN 21062 --- [ocp-async-7,34c43461bbde3a54,d581b21b70f5d2b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:01.125 INFO 21062 --- [ocp-async-10,2e6bdfcbec3c81e9,c3a1f916e813aa78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:01.410 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:01.411 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e3ab6d8e3f6e273d,eecdffc0ad7d6bc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:03.088 WARN 21062 --- [ocp-async-3,4fa5f418866cde30,b18c0d6f9239a3e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:03.125 INFO 21062 --- [ocp-async-4,01ff1de8b04344bd,c099ab978aa0fb4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:04.411 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:04.412 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:04.990 INFO 21062 --- [ocp-schedules-12,c656241274dbfc1c,e0143478a388da17] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:07:04.991 INFO 21062 --- [ocp-schedules-12,016f7f22e4edc4f9,75f3a247127e391f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:07:04.995 INFO 21062 --- [ocp-schedules-12,016f7f22e4edc4f9,75f3a247127e391f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:05.026 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:07:05.026 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:07:05.088 WARN 21062 --- [ocp-async-4,7cd03f87edd93f51,6902719bc4fca5a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:05.126 INFO 21062 --- [ocp-async-1,673763c4a31e3aa3,69a239be87be9374] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:05.983 INFO 21062 --- [prometheus-minute-schedule-1,33219b3e033e292f,a367598137d2945d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:07.089 WARN 21062 --- [ocp-async-1,4a9194a97da0356a,0fbc182e70647b55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:07.126 INFO 21062 --- [ocp-async-6,d096385e6abc2397,4a053f27a3c12bf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:07.412 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:07.414 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:08.505 INFO 21062 --- [ocp-schedules-24,d2836dcac839c532,cc2ff9a7c2f90dcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:07:08.505 INFO 21062 --- [ocp-schedules-9,25b0c9891b1d5815,7e712995c894f849] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:07:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:07:08.590528+08:00, lastProcessTime=2024-11-04T14:06:58.588904+08:00 2024-11-04 14:07:08.641 INFO 21062 --- [ocp-async-2,2fd16aac3bfc66eb,358925d536ad830a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:07:08.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:07:08.590528+08:00 2024-11-04 14:07:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d8dde15c3d1a9c16,9c72de4635fa5897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:09.089 WARN 21062 --- [ocp-async-10,ff389088a0ee7247,fef0ced3b73b536c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:09.126 INFO 21062 --- [ocp-async-4,dfaa04d4e087df98,063093b0fbd5d24a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:09.694 INFO 21062 --- [ocp-schedules-14,a3f083c3636206fd,3ab82be48c5ba86e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:07:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:07:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:07:10.190 INFO 21062 --- [ocp-schedules-11,46916b1d9a7ff62c,3a5b6b19d189cc44] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:07:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:07:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:07:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:07:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:07:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:07:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:07:10.414 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:10.415 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:11.083 INFO 21062 --- [ocp-schedules-28,ca7e10be66274e85,63c05d213aff551f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:07:11.089 INFO 21062 --- [ocp-schedules-28,ca7e10be66274e85,63c05d213aff551f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:07:11.089 INFO 21062 --- [ocp-schedules-28,ca7e10be66274e85,63c05d213aff551f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:07:11.089 WARN 21062 --- [ocp-async-4,539877e42ce73eac,0023ecd3e923de40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:11.126 INFO 21062 --- [ocp-async-1,190fc8ca5720a75b,9200c35b82f588d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:11.461 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=1, affectRows=1 2024-11-04 14:07:11.461 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=1, validCount=1 2024-11-04 14:07:11.461 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=3, affectRows=3 2024-11-04 14:07:11.461 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=3, validCount=3 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=17, affectRows=17 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=17, validCount=17 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=14, affectRows=14 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=13, affectRows=13 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=14, validCount=14 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=30, affectRows=30 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=13, validCount=13 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=27, affectRows=27 2024-11-04 14:07:11.463 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=30, validCount=30 2024-11-04 14:07:11.464 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=27, validCount=27 2024-11-04 14:07:11.464 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=50, affectRows=50 2024-11-04 14:07:11.464 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=36, affectRows=36 2024-11-04 14:07:11.464 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=55, affectRows=55 2024-11-04 14:07:11.465 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=36, validCount=36 2024-11-04 14:07:11.465 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=49, validCount=49 2024-11-04 14:07:11.465 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=55, validCount=55 2024-11-04 14:07:11.465 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=69, affectRows=69 2024-11-04 14:07:11.465 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700360, nodeCount=77, affectRows=77 2024-11-04 14:07:11.465 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=69, validCount=69 2024-11-04 14:07:11.465 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700360, nodeCount=75, validCount=75 2024-11-04 14:07:11.983 INFO 21062 --- [prometheus-minute-schedule-1,180406a8a77d34e2,1385df392c1027a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:13.089 WARN 21062 --- [ocp-async-1,11bd4118e18edc4c,fa250dc9f89215a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:13.126 INFO 21062 --- [ocp-async-6,d3a1ba2f1570858d,252e7b794c029eb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:13.415 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:13.416 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:13.984 INFO 21062 --- [prometheus-cache-monitor1,ffdcf21436fdaafd,2d5532064899b1cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:07:13.984 INFO 21062 --- [prometheus-cache-monitor1,ffdcf21436fdaafd,2d5532064899b1cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:07:14.902 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:03:14.877, end=2024-11-04T14:05:14.9} 2024-11-04 14:07:14.911 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:07:14.912 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:07:14.983 INFO 21062 --- [prometheus-minute-schedule-1,6f2067bc007b783e,39ad18a0543c7ef0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:15.089 WARN 21062 --- [ocp-async-6,400489094aa68f43,f59fc7bd8f86c9ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:15.126 INFO 21062 --- [ocp-async-8,833a7242b7fb349a,601886fb538842dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:16.416 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:16.417 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:17.089 WARN 21062 --- [ocp-async-8,7f071fa66036c08b,347968024a0f1c8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:17.126 INFO 21062 --- [ocp-async-2,c3fa4881ec4c5973,454ba1fec649fca7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9b0eaee9de853435,8bbc89b8646cb173] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:07:18.591406+08:00, lastProcessTime=2024-11-04T14:07:08.590528+08:00 2024-11-04 14:07:18.641 INFO 21062 --- [ocp-async-3,e587ec98502e841c,0cb5be83df23f564] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:07:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:07:18.591406+08:00 2024-11-04 14:07:19.089 WARN 21062 --- [ocp-async-1,8c292a5a865bcb00,10a8b83a5a633cf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:19.127 INFO 21062 --- [ocp-async-6,48b434d4d0e4ab50,a21f0748efa89bcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:19.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:19.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:19.736 INFO 21062 --- [ocp-schedules-20,2410319c8071a293,53ba2a1d76d463a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:07:20.192 INFO 21062 --- [ocp-schedules-10,ab02ae639c60a693,f9de3b665a9059b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:07:20.983 INFO 21062 --- [prometheus-minute-schedule-1,1fae624315d4746c,a2297ddc802c741f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:21.090 WARN 21062 --- [ocp-async-6,120de55954f3273f,494e12c454f8160a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:21.127 INFO 21062 --- [ocp-async-8,12ce37dcf6e500dd,0e2c0c665c1e5e6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:22.418 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:22.420 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:23.090 WARN 21062 --- [ocp-async-8,0a61508db673659d,6030e2a9e69befc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:23.127 INFO 21062 --- [ocp-async-2,07e108fa8caab7a2,0ff63915e0800689] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:23.983 INFO 21062 --- [prometheus-minute-schedule-1,598d39027df44620,0a51ce23cda453f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:25.090 WARN 21062 --- [ocp-async-2,7a81f3b1246c8066,ec4ee558f4fd8645] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:25.127 INFO 21062 --- [ocp-async-9,d24b7fb74c7d095a,662112b03cc39423] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:25.420 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:25.421 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:26.983 INFO 21062 --- [prometheus-minute-schedule-1,17053b7e4bb816a7,7079e23f1731a39f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:26.984 INFO 21062 --- [prometheus-hour-schedule-1,3c8fe306b2b746ef,b5a4ed1f4527afae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:07:27.090 WARN 21062 --- [ocp-async-9,274ffba0f54d7110,01370c1c350e27c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:27.128 INFO 21062 --- [ocp-async-10,fc0660764b5716ca,894033de1a8b6712] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:28.421 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:28.423 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:28.498 INFO 21062 --- [ocp-schedules-15,169ee20c470fc4a0,90ae44c203f663cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:07:28.499 INFO 21062 --- [ocp-schedules-23,6316fcfab695c6f2,ab463ab1427f33b2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:07:28.500 INFO 21062 --- [ocp-schedules-21,2c6a7e536fa62390,5cb29c5bbe05430d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:07:28.500 INFO 21062 --- [ocp-schedules-20,f5774d7a9a8a415f,31b373d327096bac] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:07:28.502 INFO 21062 --- [ocp-schedules-27,05f4717ea82fb9f2,584edeb6c4a2b550] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:07:28.502 INFO 21062 --- [ocp-schedules-15,169ee20c470fc4a0,90ae44c203f663cd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:07:28.504 INFO 21062 --- [ocp-schedules-21,2c6a7e536fa62390,5cb29c5bbe05430d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:28.504 INFO 21062 --- [ocp-schedules-20,f5774d7a9a8a415f,31b373d327096bac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:28.504 INFO 21062 --- [ocp-schedules-28,2019e54312a7f459,bab53668d0089e30] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:07:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:07:28.505 INFO 21062 --- [ocp-schedules-1,f1b7d0e170374109,1125c1c4a4c9030c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:07:28.505 INFO 21062 --- [ocp-schedules-23,6316fcfab695c6f2,ab463ab1427f33b2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:07:28.507 INFO 21062 --- [ocp-schedules-28,2019e54312a7f459,bab53668d0089e30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:28.512 INFO 21062 --- [ocp-schedules-24,658a9a55dfadeb2b,005c1ff66db0f82d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:07:28.512 INFO 21062 --- [ocp-schedules-29,bcf7be47bdb34fd3,10e9d563bf552fa6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:07:28.512 INFO 21062 --- [ocp-schedules-24,658a9a55dfadeb2b,005c1ff66db0f82d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:07:28.512 INFO 21062 --- [ocp-schedules-29,bcf7be47bdb34fd3,10e9d563bf552fa6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:07:28.512 INFO 21062 --- [ocp-schedules-25,356d1605b9b441c8,634342ec84eb67ff] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:07:28.512 INFO 21062 --- [ocp-schedules-25,356d1605b9b441c8,634342ec84eb67ff] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:07:28.535 INFO 21062 --- [ocp-schedules-17,52a121b2e50e1bc2,8347e545e21aec6b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:07:28.535 INFO 21062 --- [ocp-schedules-17,52a121b2e50e1bc2,8347e545e21aec6b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:07:28.578 INFO 21062 --- [pool-12-thread-13,3809b11c74cca753,4b45d81e81365c9d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:07:28.578 INFO 21062 --- [pool-12-thread-14,3809b11c74cca753,f49e24899375cd0b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:07:28.578 INFO 21062 --- [pool-12-thread-14,3809b11c74cca753,f49e24899375cd0b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:07:28.578 INFO 21062 --- [pool-12-thread-15,3809b11c74cca753,0eb4b4a8cde4cd87] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:07:28.578 INFO 21062 --- [pool-12-thread-16,3809b11c74cca753,1de9cee0f75e6c9c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:07:28.580 INFO 21062 --- [pool-12-thread-14,3809b11c74cca753,f49e24899375cd0b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:07:28.583 INFO 21062 --- [ForkJoinPool-11-worker-12,3809b11c74cca753,7423f1b2c29540a3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:07:28.583 INFO 21062 --- [ForkJoinPool-11-worker-12,3809b11c74cca753,7423f1b2c29540a3] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:07:28.583 INFO 21062 --- [ForkJoinPool-11-worker-12,3809b11c74cca753,7423f1b2c29540a3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:07:28.583 INFO 21062 --- [ForkJoinPool-11-worker-5,3809b11c74cca753,9e2c61ecb4636dc5] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:07:28.583 INFO 21062 --- [pool-12-thread-13,3809b11c74cca753,4b45d81e81365c9d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:07:28.583 INFO 21062 --- [pool-12-thread-16,3809b11c74cca753,1de9cee0f75e6c9c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:07:28.586 INFO 21062 --- [pool-12-thread-15,3809b11c74cca753,0eb4b4a8cde4cd87] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:07:28.586 INFO 21062 --- [pool-12-thread-15,3809b11c74cca753,0eb4b4a8cde4cd87] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:07:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:07:28.588922+08:00, lastProcessTime=2024-11-04T14:07:18.591406+08:00 2024-11-04 14:07:28.591 INFO 21062 --- [pool-12-thread-14,3809b11c74cca753,f49e24899375cd0b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:07:28.641 INFO 21062 --- [ocp-async-6,0e47ba87cae31e40,9638dc1fd58fab90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:07:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:07:28.588922+08:00 2024-11-04 14:07:28.790 INFO 21062 --- [ocp-schedules-23,6c5733d826e21f4b,6ec597bcbae2a1fc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:07:28.806 INFO 21062 --- [ocp-schedules-27,ccf23325232c3e3a,76138904b7ab7050] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:07:28.819 INFO 21062 --- [ocp-schedules-27,ccf23325232c3e3a,76138904b7ab7050] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=0, cachedSize=0 2024-11-04 14:07:29.021 INFO 21062 --- [ocp-schedules-1,50a1a7b015bb7b3c,9fbf3e807071ebe7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:07:29.021 INFO 21062 --- [ocp-schedules-1,50a1a7b015bb7b3c,9fbf3e807071ebe7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:07:29.075 INFO 21062 --- [ocp-schedules-14,2978c0e12f803570,986e0f39527fe7b0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:07:29.090 WARN 21062 --- [ocp-async-5,f555e05a08268673,a1b836191a0045b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:29.108 INFO 21062 --- [ocp-schedules-2,5c670fc035948ee7,8275e294aff6690c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:07:29.113 INFO 21062 --- [ocp-schedules-2,5c670fc035948ee7,8275e294aff6690c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:07:29.113 INFO 21062 --- [ocp-schedules-2,5c670fc035948ee7,8275e294aff6690c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:07:29.128 INFO 21062 --- [ocp-async-2,fc4c77c9575da71b,7e02792d3ed366d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:29.224 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:07:29.224 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:07:29.233 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:07:29.233 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:07:29.233 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:07:29.244 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:07:29.244 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:07:29.244 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:07:29.248 INFO 21062 --- [ocp-schedules-30,cbd9b4b910a6ffb7,7e44904b6cb1a1c2] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:07:29.395 INFO 21062 --- [ocp-schedules-8,a2f27a0700f91d92,69a58d0ee286ca45] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:07:29.395 INFO 21062 --- [ocp-schedules-8,a2f27a0700f91d92,69a58d0ee286ca45] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:07:29.403 INFO 21062 --- [ocp-schedules-8,a2f27a0700f91d92,69a58d0ee286ca45] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:07:29.403 INFO 21062 --- [ocp-schedules-8,a2f27a0700f91d92,69a58d0ee286ca45] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:07:29.407 INFO 21062 --- [ocp-schedules-8,a2f27a0700f91d92,69a58d0ee286ca45] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:07:29.407 INFO 21062 --- [ocp-schedules-8,a2f27a0700f91d92,69a58d0ee286ca45] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:07:29.576 INFO 21062 --- [ocp-schedules-18,d80a3ff0938e3e8b,5be5973dab30d254] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:07:29.576 INFO 21062 --- [ocp-schedules-18,d80a3ff0938e3e8b,5be5973dab30d254] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:07:29.604 INFO 21062 --- [pool-38-thread-1,d80a3ff0938e3e8b,2acc7a9dd59a4e9d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:07:29.632 WARN 21062 --- [ocp-schedules-18,d80a3ff0938e3e8b,5be5973dab30d254] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:07:29.632 INFO 21062 --- [ocp-schedules-18,d80a3ff0938e3e8b,5be5973dab30d254] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:07:29.760 INFO 21062 --- [ocp-schedules-23,02a84af12620c253,fdab856b32f8187e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:07:29.983 INFO 21062 --- [prometheus-minute-schedule-1,2abfe42e341624e9,bd8de237291ae8a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:30.195 INFO 21062 --- [ocp-schedules-15,7ab472db1d86865b,d02e182a00192b38] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:07:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:07:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:07:31.091 WARN 21062 --- [ocp-async-2,5b0ab80015703288,b1d68ac38e9ad917] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:31.092 INFO 21062 --- [ocp-schedules-8,15613c32b0b7287e,e85148bc084f6713] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:07:31.098 INFO 21062 --- [ocp-schedules-8,15613c32b0b7287e,e85148bc084f6713] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:07:31.098 INFO 21062 --- [ocp-schedules-8,15613c32b0b7287e,e85148bc084f6713] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:07:31.116 INFO 21062 --- [ocp-schedules-1,088f54dab21f90dd,4867fde95de359f2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:07:31.116 INFO 21062 --- [ocp-schedules-1,088f54dab21f90dd,4867fde95de359f2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:07:31.128 INFO 21062 --- [ocp-async-6,919079f2818d140d,1c7bad01eadebfcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:31.142 INFO 21062 --- [ocp-schedules-1,088f54dab21f90dd,4867fde95de359f2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:07:31.142 INFO 21062 --- [ocp-schedules-1,088f54dab21f90dd,4867fde95de359f2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:07:31.142 INFO 21062 --- [ocp-schedules-14,23f4fe62b8e3654b,8392d85e827eaa5a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:07:31.316 INFO 21062 --- [ocp-schedules-17,7b28dfe6bfc30ace,8c231ada08e73c54] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:07:31.316+08:00, clusters=[1] 2024-11-04 14:07:31.317 INFO 21062 --- [ocp-schedules-17,7b28dfe6bfc30ace,8c231ada08e73c54] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:07:31.398 INFO 21062 --- [ocp-schedules-11,0a4381976847981b,c517c395629e2319] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:07:31.398 INFO 21062 --- [ocp-schedules-11,0a4381976847981b,c517c395629e2319] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:07:31.398 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:07:31.398 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:07:31.407 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:07:31.408 INFO 21062 --- [ocp-schedules-11,0a4381976847981b,c517c395629e2319] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:07:31.408 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:07:31.409 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:07:31.409 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:07:31.409 WARN 21062 --- [ocp-schedules-11,0a4381976847981b,c517c395629e2319] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:07:31.410 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:07:31.410 INFO 21062 --- [ocp-schedules-11,0a4381976847981b,c517c395629e2319] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:07:31.410 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:07:31.411 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:07:31.411 WARN 21062 --- [ocp-schedules-11,0a4381976847981b,c517c395629e2319] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:07:31.411 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:07:31.418 ERROR 21062 --- [ocp-schedules-11,0a4381976847981b,c517c395629e2319] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 14:07:31.423 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:31.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:31.459 INFO 21062 --- [ocp-schedules-30,c09617cc6ab30f73,659790fc34a819ef] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:07:31.459+08:00, clusters=[1] 2024-11-04 14:07:31.463 INFO 21062 --- [ocp-schedules-30,c09617cc6ab30f73,659790fc34a819ef] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:07:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7ed55ebf2ab6503e,3fe8abaa17c8f376] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:33.091 WARN 21062 --- [ocp-async-6,1ec80a9a8950eb54,9a609b15f6cabae5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:33.128 INFO 21062 --- [ocp-async-8,7568056f8f5ea82f,15b0d3bbc255c00f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:34.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:34.425 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:35.026 INFO 21062 --- [ocp-schedules-12,016f7f22e4edc4f9,75f3a247127e391f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:07:35.029 INFO 21062 --- [ocp-schedules-12,7f88f39db32c8057,87d2dddc56200dc7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:07:35.033 INFO 21062 --- [ocp-schedules-12,7f88f39db32c8057,87d2dddc56200dc7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:35.064 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:07:35.064 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:07:35.091 WARN 21062 --- [ocp-async-4,93242040a6bf9729,6dd6ec60e1a09203] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:35.128 INFO 21062 --- [ocp-async-1,1dff59785344b868,c953fc0ce0eeabee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:35.983 INFO 21062 --- [prometheus-minute-schedule-1,740289b20cf77739,298c68ee0820b476] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:37.091 WARN 21062 --- [ocp-async-7,b1c634bf9c9ef1be,b58bb987eb6b2869] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:37.129 INFO 21062 --- [ocp-async-9,06ae85609bd70726,48ea17063944459b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:37.287 INFO 21062 --- [ocp-schedules-26,af2b5de22aabd227,f8598423690b4263] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:07:37.291 INFO 21062 --- [ocp-schedules-26,af2b5de22aabd227,f8598423690b4263] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:37.373 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:07:37.376 ERROR 21062 --- [ocp-schedules-26,af2b5de22aabd227,f8598423690b4263] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:07:37.376 INFO 21062 --- [ocp-schedules-26,af2b5de22aabd227,f8598423690b4263] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:07:37.425 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:37.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:38.505 INFO 21062 --- [ocp-schedules-20,f5774d7a9a8a415f,31b373d327096bac] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:07:38.505 INFO 21062 --- [ocp-schedules-21,2c6a7e536fa62390,5cb29c5bbe05430d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:07:38.508 INFO 21062 --- [ocp-schedules-28,2019e54312a7f459,bab53668d0089e30] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:07:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:07:38.590308+08:00, lastProcessTime=2024-11-04T14:07:28.588922+08:00 2024-11-04 14:07:38.641 INFO 21062 --- [ocp-async-1,ce99a79ee536fa7d,44f9f05331f8cc1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:07:38.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:07:38.590308+08:00 2024-11-04 14:07:38.983 INFO 21062 --- [prometheus-minute-schedule-1,1286ee482a5b1dde,d419f3581df71250] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:39.091 WARN 21062 --- [ocp-async-10,86170c231daf3d43,fbf13fa092e49964] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:39.129 INFO 21062 --- [ocp-async-4,cd4df5b179850a57,068ead3bab47439f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:39.804 INFO 21062 --- [ocp-schedules-16,a0fb48dbd18e1bab,b669722521ef8550] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:07:40.002 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:07:40.002 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:07:40.198 INFO 21062 --- [ocp-schedules-25,95218ddd8bf12e07,0cbe53a53dc6ef6b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:07:40.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:40.428 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:41.092 WARN 21062 --- [ocp-async-5,b16e2e2dea40b454,38ddca1d43270010] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:41.129 INFO 21062 --- [ocp-async-7,def2114aeebe442f,ce8227272c745a12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a5aaa4bcf1b7300d,cb9b1daf846cf384] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:43.092 WARN 21062 --- [ocp-async-2,59c36a5198fc1754,342967f6b091b7b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:43.129 INFO 21062 --- [ocp-async-3,4768e252a1d37934,df39f58d71549c73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:43.428 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:43.429 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:44.983 INFO 21062 --- [prometheus-minute-schedule-1,1d7f581cce23bf13,771c7b6c3194a2db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:45.092 WARN 21062 --- [ocp-async-6,1801cdde766c668a,0b2bed3a1517e4a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:45.129 INFO 21062 --- [ocp-async-8,69d6b18f184a35ac,bf585e1cdea91fca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:45.501 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,433d8dd7430acf08,487ca7526381259a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/21/logSegments?taskInstanceId=2&subtaskInstanceId=21, client=10.130.47.17, traceId=433d8dd7430acf08, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,21,] 2024-11-04 14:07:45.519 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,433d8dd7430acf08,487ca7526381259a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/21/logSegments client=10.130.47.17, traceId=433d8dd7430acf08, duration=35 ms] 2024-11-04 14:07:46.429 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:46.431 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:46.529 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,433b0bcef36322b6,e3172f6f0315dda9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/35/logSegments?taskInstanceId=2&subtaskInstanceId=35, client=10.130.47.17, traceId=433b0bcef36322b6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,35,] 2024-11-04 14:07:46.535 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,433b0bcef36322b6,e3172f6f0315dda9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/35/logSegments client=10.130.47.17, traceId=433b0bcef36322b6, duration=13 ms] 2024-11-04 14:07:47.092 WARN 21062 --- [ocp-async-4,976cbd2e35a0a231,4d1e952c0bdc5790] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:47.130 INFO 21062 --- [ocp-async-1,fd6632450e1a6479,b59736d6852dd412] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:47.655 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,7b82866082bda4e9,ca53b633f7461f35] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/21/logSegments?taskInstanceId=2&subtaskInstanceId=21, client=10.130.47.17, traceId=7b82866082bda4e9, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,21,] 2024-11-04 14:07:47.662 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,7b82866082bda4e9,ca53b633f7461f35] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/21/logSegments client=10.130.47.17, traceId=7b82866082bda4e9, duration=15 ms] 2024-11-04 14:07:47.983 INFO 21062 --- [prometheus-minute-schedule-1,d09c5933dc636ea3,9484dbe0e89af570] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:07:48.592554+08:00, lastProcessTime=2024-11-04T14:07:38.590308+08:00 2024-11-04 14:07:48.642 INFO 21062 --- [ocp-async-10,48dd75b87b71244b,07ffd9b80ec42b2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:07:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:07:48.592554+08:00 2024-11-04 14:07:49.092 WARN 21062 --- [ocp-async-2,c504f0e0d9e333cd,01d845c5df8e4807] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:49.130 INFO 21062 --- [ocp-async-3,cb934e01bd578a38,f59bbf2a79cd7540] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:49.431 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:49.432 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:49.769 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f2fec9fece8d4456,d9ad982687ccf75b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/54/logSegments?taskInstanceId=2&subtaskInstanceId=54, client=10.130.47.17, traceId=f2fec9fece8d4456, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,54,] 2024-11-04 14:07:49.770 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f2fec9fece8d4456,d9ad982687ccf75b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/54/logSegments client=10.130.47.17, traceId=f2fec9fece8d4456, duration=9 ms] 2024-11-04 14:07:49.829 INFO 21062 --- [ocp-schedules-19,37d6a7077693ad2e,ce32e18a2a8bafa1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:07:50.201 INFO 21062 --- [ocp-schedules-6,6585fdba317a6243,fba022ef2372d11f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:07:50.983 INFO 21062 --- [prometheus-minute-schedule-1,c5d516f37947a83a,11c4d8d52f7cfe7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:51.092 WARN 21062 --- [ocp-async-6,80bcdea086e23bac,8ef616ea930029b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:51.100 INFO 21062 --- [ocp-schedules-7,7921e68db47164ef,27d8d1e9fc5be58e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:07:51.106 INFO 21062 --- [ocp-schedules-7,7921e68db47164ef,27d8d1e9fc5be58e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:07:51.106 INFO 21062 --- [ocp-schedules-7,7921e68db47164ef,27d8d1e9fc5be58e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:07:51.130 INFO 21062 --- [ocp-async-8,c84a8968bd074248,b4cd37acd62c3593] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:52.432 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:52.434 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:53.093 WARN 21062 --- [ocp-async-4,1c88185be4bc6496,77d5d5bef6e85ce8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:53.130 INFO 21062 --- [ocp-async-1,4da6160e84a80ea3,1a9304f5296a975e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:53.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,4f1e9baf800db5ae,e6f47528e2fcabb5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments?taskInstanceId=2&subtaskInstanceId=2, client=10.130.47.17, traceId=4f1e9baf800db5ae, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,2,] 2024-11-04 14:07:53.820 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,4f1e9baf800db5ae,e6f47528e2fcabb5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments client=10.130.47.17, traceId=4f1e9baf800db5ae, duration=16 ms] 2024-11-04 14:07:53.983 INFO 21062 --- [prometheus-minute-schedule-1,027c50bbb1a317d6,7ee75eea692d9eef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:55.093 WARN 21062 --- [ocp-async-7,7a0973b21791622e,631546e0ce80fdfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:55.130 INFO 21062 --- [ocp-async-9,34c043841ad83103,48b8a95f525f9499] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:55.434 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:55.435 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:56.983 INFO 21062 --- [prometheus-minute-schedule-1,3c16ad100f78fad3,20bc710c2110d80f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:07:57.093 WARN 21062 --- [ocp-async-3,9611c0b3bc9b928e,a584e8b31b7e1f53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:57.131 INFO 21062 --- [ocp-async-10,bd74113f656449c7,d3f28f90f15b4937] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:58.435 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:07:58.436 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:07:58.500 INFO 21062 --- [ocp-schedules-6,7a8c779f6fb92cca,b3dd2edb4abdb4a6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:07:58.500 INFO 21062 --- [ocp-schedules-2,bf58ae25baced8fb,9093a6121f1284ad] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:07:58.503 INFO 21062 --- [ocp-schedules-22,2686b18d81f1169f,9a657195c8b4f09d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:07:58.504 INFO 21062 --- [ocp-schedules-6,7a8c779f6fb92cca,b3dd2edb4abdb4a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:07:58.504 INFO 21062 --- [ocp-schedules-16,fc46465fa6145133,1fb7212bab5502d5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:07:58.507 INFO 21062 --- [ocp-schedules-2,bf58ae25baced8fb,9093a6121f1284ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:58.511 INFO 21062 --- [ocp-schedules-23,5639b5324f8cd324,4393743ea21c6324] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:07:58.511 INFO 21062 --- [ocp-schedules-25,462d1c8eb5944c80,df1825d961b4f439] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:07:58.511 INFO 21062 --- [ocp-schedules-23,5639b5324f8cd324,4393743ea21c6324] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:07:58.511 INFO 21062 --- [ocp-schedules-15,e0cbeed881a96022,ef0caf8361820f35] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:07:58.511 INFO 21062 --- [ocp-schedules-25,462d1c8eb5944c80,df1825d961b4f439] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:07:58.511 INFO 21062 --- [ocp-schedules-15,e0cbeed881a96022,ef0caf8361820f35] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:07:58.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:07:58.593079+08:00, lastProcessTime=2024-11-04T14:07:48.592554+08:00 2024-11-04 14:07:58.642 INFO 21062 --- [ocp-async-2,0f52d4c2bb43be8c,30fccf7051a6e523] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:07:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:07:58.593079+08:00 2024-11-04 14:07:58.946 INFO 21062 --- [ocp-schedules-4,e745aea63760c458,ffb92681e4835f5f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:07:58.949 INFO 21062 --- [ocp-schedules-4,e745aea63760c458,ffb92681e4835f5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:07:58.950 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:07:59.077 INFO 21062 --- [ocp-schedules-10,edc106f47bc78f82,f9fb3a9b8138672a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:07:59.093 WARN 21062 --- [ocp-async-4,bdbe5b59135f1aaf,9c9a6cdd87caa6c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:07:59.131 INFO 21062 --- [ocp-async-1,c215d56b1ed02024,ce90f2ba7c595c74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:07:59.872 INFO 21062 --- [ocp-schedules-1,835e9f0e307dafa1,e039d7b78484a634] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:07:59.983 INFO 21062 --- [prometheus-minute-schedule-1,9d0abadb72df3285,45397d71e11de4f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700480, interval=60 2024-11-04 14:08:00.204 INFO 21062 --- [ocp-schedules-30,8d2857dbca16d3ba,acb4ca4296870474] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:08:01.094 WARN 21062 --- [ocp-async-7,f33619d33700daf3,3cc0d8db6a0eb80f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:01.131 INFO 21062 --- [ocp-async-9,6cd7cc2d392d80a4,d9cbaff55728a45e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:01.436 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:01.438 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:02.983 INFO 21062 --- [prometheus-minute-schedule-1,fe53421d8ea90d8b,97dfb62428af0d5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:03.094 WARN 21062 --- [ocp-async-9,a3fdd6c348ccf3ba,a95417b5c2ecc365] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:03.131 INFO 21062 --- [ocp-async-10,c89ad207df8bba09,b9f4654800727852] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:04.438 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:04.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:05.064 INFO 21062 --- [ocp-schedules-12,7f88f39db32c8057,87d2dddc56200dc7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:08:05.065 INFO 21062 --- [ocp-schedules-12,8b7b96b1a28737b1,303e1d3a5b1d1d49] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:08:05.070 INFO 21062 --- [ocp-schedules-12,8b7b96b1a28737b1,303e1d3a5b1d1d49] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:05.094 WARN 21062 --- [ocp-async-10,ebcc5bb1565b3231,89ffd9326a529998] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:05.101 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:08:05.101 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:08:05.131 INFO 21062 --- [ocp-async-5,87cd7c4ce1eccd41,e7f133112ea4bee8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:05.983 INFO 21062 --- [prometheus-minute-schedule-1,41df1b1b0ef84816,3ccf712eeb333271] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:07.094 WARN 21062 --- [ocp-async-5,b2f9e62e84ce890c,668c872657fab0ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:07.131 INFO 21062 --- [ocp-async-2,3408b3bbd7c82090,022523746ce31007] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:07.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:07.440 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:08.505 INFO 21062 --- [ocp-schedules-6,7a8c779f6fb92cca,b3dd2edb4abdb4a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:08:08.507 INFO 21062 --- [ocp-schedules-2,bf58ae25baced8fb,9093a6121f1284ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:08:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:08:08.590124+08:00, lastProcessTime=2024-11-04T14:07:58.593079+08:00 2024-11-04 14:08:08.642 INFO 21062 --- [ocp-async-8,4eefec8f8dc73ed6,2108cce3df0ebe91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:08:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:08:08.590124+08:00 2024-11-04 14:08:08.950 INFO 21062 --- [ocp-schedules-4,e745aea63760c458,ffb92681e4835f5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:08:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d6a541a4d4bd7351,54707ee396133167] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:09.094 WARN 21062 --- [ocp-async-9,3798e6fe0be72d44,4d55ade790624f9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:09.132 INFO 21062 --- [ocp-async-10,dd792237677e31f8,70f9af651fdcf4e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:09.900 INFO 21062 --- [ocp-schedules-19,01d8417f5749a53a,9830b533ed15cfc0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:08:10.004 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:08:10.004 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:08:10.207 INFO 21062 --- [ocp-schedules-9,2997a6380def4eb8,0c266b39e8ffcbd5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:08:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:08:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:08:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:08:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:08:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:08:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:08:10.440 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:10.441 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:11.094 WARN 21062 --- [ocp-async-10,1400e786fb2e4cca,eec4fb79c60ac939] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:11.109 INFO 21062 --- [ocp-schedules-13,6283e6a8b371ed98,cf771beb1a189a27] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:08:11.115 INFO 21062 --- [ocp-schedules-13,6283e6a8b371ed98,cf771beb1a189a27] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:08:11.115 INFO 21062 --- [ocp-schedules-13,6283e6a8b371ed98,cf771beb1a189a27] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:08:11.132 INFO 21062 --- [ocp-async-5,1935a5d74b0d971b,c4826ba610f15416] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:11.474 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=19, affectRows=19 2024-11-04 14:08:11.474 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=29, affectRows=29 2024-11-04 14:08:11.474 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=2, affectRows=2 2024-11-04 14:08:11.465 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=1, affectRows=1 2024-11-04 14:08:11.474 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=2, validCount=2 2024-11-04 14:08:11.474 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=1, validCount=1 2024-11-04 14:08:11.474 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=19, validCount=19 2024-11-04 14:08:11.474 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=29, validCount=29 2024-11-04 14:08:11.476 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=10, affectRows=10 2024-11-04 14:08:11.476 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=10, affectRows=10 2024-11-04 14:08:11.476 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=10, validCount=10 2024-11-04 14:08:11.476 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=36, affectRows=36 2024-11-04 14:08:11.476 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=22, affectRows=22 2024-11-04 14:08:11.476 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=9, validCount=9 2024-11-04 14:08:11.477 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=22, validCount=22 2024-11-04 14:08:11.477 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=36, validCount=36 2024-11-04 14:08:11.477 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=37, affectRows=37 2024-11-04 14:08:11.477 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=37, validCount=37 2024-11-04 14:08:11.477 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=35, affectRows=35 2024-11-04 14:08:11.477 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=51, affectRows=51 2024-11-04 14:08:11.478 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=35, validCount=35 2024-11-04 14:08:11.478 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=51, validCount=51 2024-11-04 14:08:11.478 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=60, affectRows=60 2024-11-04 14:08:11.478 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=60, validCount=60 2024-11-04 14:08:11.478 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700420, nodeCount=80, affectRows=80 2024-11-04 14:08:11.479 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700420, nodeCount=78, validCount=78 2024-11-04 14:08:11.983 INFO 21062 --- [prometheus-minute-schedule-1,dbc13fd88dd04dd4,cd64f901f67b6e80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:13.094 WARN 21062 --- [ocp-async-5,7d05975b1d69dc5b,7f6f969c9963da38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:13.132 INFO 21062 --- [ocp-async-2,6729e87dfcd64ada,1b96ebd8d597fb4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:13.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:13.443 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:13.984 INFO 21062 --- [prometheus-cache-monitor1,a8a2e6c387314a73,7f693ac8746b7114] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:08:13.985 INFO 21062 --- [prometheus-cache-monitor1,a8a2e6c387314a73,7f693ac8746b7114] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:08:14.195 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:02:14.174, end=2024-11-04T14:07:14.192} 2024-11-04 14:08:14.195 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:02:14.174, end=2024-11-04T14:07:14.193} 2024-11-04 14:08:14.199 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:08:14.199 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:08:14.199 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:08:14.200 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:08:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3429cbdbea3e99a1,89be6a5df164acdf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:15.095 WARN 21062 --- [ocp-async-2,3643facb38d1ab89,78c9ecdc863ef3cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:15.132 INFO 21062 --- [ocp-async-3,fbb41158dd8f7d34,b21909169e85c652] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:16.443 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:16.444 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:17.095 WARN 21062 --- [ocp-async-3,f383416d1ec51214,ed47e4fe81738a56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:17.132 INFO 21062 --- [ocp-async-8,18f69c4533f95cd6,40ebbb9043ee78dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:17.983 INFO 21062 --- [prometheus-minute-schedule-1,2dd144e2d7881fea,64abe14c814cd55b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:18.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:08:18.591345+08:00, lastProcessTime=2024-11-04T14:08:08.590124+08:00 2024-11-04 14:08:18.642 INFO 21062 --- [ocp-async-6,bf42743ce584b705,34a004fb68e4ff50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:08:18.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:08:18.591345+08:00 2024-11-04 14:08:19.095 WARN 21062 --- [ocp-async-5,ffd25bee7ea1791b,b00dd39b20065118] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:19.133 INFO 21062 --- [ocp-async-2,29f6cd20667af301,782f66ca13f4b186] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:19.445 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:19.446 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:19.944 INFO 21062 --- [ocp-schedules-16,796e86fc5f1e1475,3f97af822abb1189] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:08:20.209 INFO 21062 --- [ocp-schedules-10,62a5d7c751aafda0,dda25c26e56cbc88] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:08:20.983 INFO 21062 --- [prometheus-minute-schedule-1,632a53891e801ca2,d5c95457ed515c0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:21.095 WARN 21062 --- [ocp-async-2,12444e36280625ce,d346627961100887] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:21.133 INFO 21062 --- [ocp-async-3,8d30a772af10affe,e82c5d62afbb4e1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:22.446 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:22.447 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:23.095 WARN 21062 --- [ocp-async-3,4079d0e9ce0b1481,194ecea46900ff54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:23.133 INFO 21062 --- [ocp-async-8,466842eca0127d56,80fa9cc425b3f57e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a407cd200111ee6b,5f18cb2a5a0065f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:25.096 WARN 21062 --- [ocp-async-8,827dc491727feb08,80b8748e59fc8e8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:25.133 INFO 21062 --- [ocp-async-1,cdd3908b91a05e4e,45063382d4cf2c72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:25.448 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:25.448 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:26.983 INFO 21062 --- [prometheus-minute-schedule-1,039cddd8d7778ae0,dcebd78f742d5660] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:26.984 INFO 21062 --- [prometheus-hour-schedule-1,aea06bd7b2482818,3a77c445c649c852] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:08:27.096 WARN 21062 --- [ocp-async-1,e4822e075574d3a9,8d3b4bb6d1594069] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:27.133 INFO 21062 --- [ocp-async-9,aef0e9315c249c9a,5869c1f7d547aa84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:28.449 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:28.450 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:28.498 INFO 21062 --- [ocp-schedules-10,56a168328e05ebb8,5cc3bce4ae2baadd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:08:28.499 INFO 21062 --- [ocp-schedules-25,f8b333b01db66900,7167ace67accdc51] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:08:28.500 INFO 21062 --- [ocp-schedules-9,95ee929777d13228,a9645944ef2a4f51] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:08:28.500 INFO 21062 --- [ocp-schedules-16,0d2aee6dfaba5993,a38829aeb29c4454] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:08:28.502 INFO 21062 --- [ocp-schedules-10,56a168328e05ebb8,5cc3bce4ae2baadd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:08:28.503 INFO 21062 --- [ocp-schedules-19,e682c2702b0a2930,d65fb0009e7ad263] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:08:28.504 INFO 21062 --- [ocp-schedules-9,95ee929777d13228,a9645944ef2a4f51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:28.504 INFO 21062 --- [ocp-schedules-16,0d2aee6dfaba5993,a38829aeb29c4454] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:08:28.505 INFO 21062 --- [ocp-schedules-27,ea82080cda0b05c4,6c3a2d7450524b1d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:08:28.506 INFO 21062 --- [ocp-schedules-25,f8b333b01db66900,7167ace67accdc51] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:08:28.511 INFO 21062 --- [ocp-schedules-1,43782b72362e4047,4642169162cd870e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:08:28.511 INFO 21062 --- [ocp-schedules-1,43782b72362e4047,4642169162cd870e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:08:28.511 INFO 21062 --- [ocp-schedules-28,cf8975a8919eefcc,01ba73f6cb4e2303] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:08:28.511 INFO 21062 --- [ocp-schedules-22,8f491fe756921fe2,68ebb14697498a29] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:08:28.511 INFO 21062 --- [ocp-schedules-22,8f491fe756921fe2,68ebb14697498a29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:08:28.511 INFO 21062 --- [ocp-schedules-28,cf8975a8919eefcc,01ba73f6cb4e2303] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:08:28.573 INFO 21062 --- [ocp-schedules-29,ff20a7bb51da72e6,72189ae1d1b194cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:08:28.573 INFO 21062 --- [ocp-schedules-29,ff20a7bb51da72e6,72189ae1d1b194cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:08:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:08:28.589949+08:00, lastProcessTime=2024-11-04T14:08:18.591345+08:00 2024-11-04 14:08:28.642 INFO 21062 --- [ocp-async-2,1ac45a00e8029082,ad289ac6b363d354] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:08:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:08:28.589949+08:00 2024-11-04 14:08:28.792 INFO 21062 --- [ocp-schedules-18,38ca34faf71bb560,eac316ae392381f5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:08:29.025 INFO 21062 --- [ocp-schedules-25,e7eb9018259ff8ca,aa51824a57c40033] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:08:29.025 INFO 21062 --- [ocp-schedules-25,e7eb9018259ff8ca,aa51824a57c40033] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:08:29.080 INFO 21062 --- [ocp-schedules-27,c0801a00310484fa,fccfb9eaccf422a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:08:29.096 WARN 21062 --- [ocp-async-4,168e2d7b5de25e69,29088628e938da9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:29.113 INFO 21062 --- [ocp-schedules-28,e5c6b5a3ea20399c,679978a1a57f92b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:08:29.119 INFO 21062 --- [ocp-schedules-28,e5c6b5a3ea20399c,679978a1a57f92b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:08:29.119 INFO 21062 --- [ocp-schedules-28,e5c6b5a3ea20399c,679978a1a57f92b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:08:29.133 INFO 21062 --- [ocp-async-8,6a5ab19c565bcd52,6a1d24b0d5e92cfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:29.412 INFO 21062 --- [ocp-schedules-29,2ce59dda4e64ce67,2b9bbc69c797e2e3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:08:29.412 INFO 21062 --- [ocp-schedules-29,2ce59dda4e64ce67,2b9bbc69c797e2e3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:08:29.414 INFO 21062 --- [ocp-schedules-29,2ce59dda4e64ce67,2b9bbc69c797e2e3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:08:29.414 INFO 21062 --- [ocp-schedules-29,2ce59dda4e64ce67,2b9bbc69c797e2e3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:08:29.415 INFO 21062 --- [ocp-schedules-29,2ce59dda4e64ce67,2b9bbc69c797e2e3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:08:29.415 INFO 21062 --- [ocp-schedules-29,2ce59dda4e64ce67,2b9bbc69c797e2e3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:08:29.969 INFO 21062 --- [ocp-schedules-7,4f8bbff748a7a47c,b5058da0de1d97fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:08:29.983 INFO 21062 --- [prometheus-minute-schedule-1,0b9e8e2544768bed,1a56c32956c0bdb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:30.212 INFO 21062 --- [ocp-schedules-6,49bcdf963cb1b72a,afe2fc09a96b0353] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:08:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:08:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:08:31.096 WARN 21062 --- [ocp-async-8,67d8da1e5b76e94d,c713194fb1bd748d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:31.118 INFO 21062 --- [ocp-schedules-3,ca4b63b0eeed6a69,60b9adc13d335119] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:08:31.124 INFO 21062 --- [ocp-schedules-3,ca4b63b0eeed6a69,60b9adc13d335119] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:08:31.124 INFO 21062 --- [ocp-schedules-3,ca4b63b0eeed6a69,60b9adc13d335119] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:08:31.134 INFO 21062 --- [ocp-async-2,ea008a7b8b69ab27,accfbbf09abe13eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:31.142 INFO 21062 --- [ocp-schedules-2,e051e2edad968534,03bc4a362be5769e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:08:31.142 INFO 21062 --- [ocp-schedules-2,e051e2edad968534,03bc4a362be5769e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:08:31.168 INFO 21062 --- [ocp-schedules-2,e051e2edad968534,03bc4a362be5769e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:08:31.168 INFO 21062 --- [ocp-schedules-2,e051e2edad968534,03bc4a362be5769e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:08:31.169 INFO 21062 --- [ocp-schedules-7,6423c4740f83c5fb,ac5d39778b32fed5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:08:31.344 INFO 21062 --- [ocp-schedules-25,e18d6a21015efd4c,f4d63b18fe2e9970] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:08:31.344+08:00, clusters=[1] 2024-11-04 14:08:31.346 INFO 21062 --- [ocp-schedules-25,e18d6a21015efd4c,f4d63b18fe2e9970] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:08:31.427 INFO 21062 --- [ocp-schedules-22,b4cef1832bc3ee08,1b2e7e6390434bb6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:08:31.427 INFO 21062 --- [ocp-schedules-22,b4cef1832bc3ee08,1b2e7e6390434bb6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:08:31.427 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:08:31.427 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:08:31.437 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:08:31.438 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:08:31.438 INFO 21062 --- [ocp-schedules-22,b4cef1832bc3ee08,1b2e7e6390434bb6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:08:31.440 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:08:31.440 WARN 21062 --- [ocp-schedules-22,b4cef1832bc3ee08,1b2e7e6390434bb6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:08:31.440 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:08:31.440 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:08:31.440 INFO 21062 --- [ocp-schedules-22,b4cef1832bc3ee08,1b2e7e6390434bb6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:08:31.440 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:08:31.442 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:08:31.442 WARN 21062 --- [ocp-schedules-22,b4cef1832bc3ee08,1b2e7e6390434bb6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:08:31.442 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:08:31.446 INFO 21062 --- [ocp-schedules-13,dd42c857aca0663f,ae6065ecabd85edf] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:08:31.450 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:31.450 INFO 21062 --- [ocp-schedules-13,dd42c857aca0663f,ae6065ecabd85edf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:31.450 ERROR 21062 --- [ocp-schedules-22,b4cef1832bc3ee08,1b2e7e6390434bb6] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 14:08:31.450 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:31.486 INFO 21062 --- [ocp-schedules-4,47c0ac4fc980e24f,2a6772e7aa25bd0b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:08:31.486+08:00, clusters=[1] 2024-11-04 14:08:31.489 INFO 21062 --- [ocp-schedules-4,47c0ac4fc980e24f,2a6772e7aa25bd0b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:08:31.504 ERROR 21062 --- [ocp-schedules-13,dd42c857aca0663f,ae6065ecabd85edf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:08:31.504 INFO 21062 --- [ocp-schedules-13,dd42c857aca0663f,ae6065ecabd85edf] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:08:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f61814fa98767b4d,dd38a48fca561578] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:33.096 WARN 21062 --- [ocp-async-2,2789e27816e12269,3c213e330ac0376b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:33.134 INFO 21062 --- [ocp-async-3,82550402ad672164,2192ee24bb975e51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:34.451 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:34.452 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:35.096 WARN 21062 --- [ocp-async-10,116bf7b15d2b2b78,a4a5e8396f4fb468] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:35.101 INFO 21062 --- [ocp-schedules-12,8b7b96b1a28737b1,303e1d3a5b1d1d49] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:08:35.101 INFO 21062 --- [ocp-schedules-12,80846019521fa0a6,fb697c9aeb0527eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:08:35.105 INFO 21062 --- [ocp-schedules-12,80846019521fa0a6,fb697c9aeb0527eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:35.134 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:08:35.134 INFO 21062 --- [ocp-async-5,d4c3af0581fac86f,45092fd253ca4c0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:35.134 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:08:35.983 INFO 21062 --- [prometheus-minute-schedule-1,80fc1a7363e9f77e,e612207aed0a87c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:37.097 WARN 21062 --- [ocp-async-7,94562396a60e60d8,bf1fae02ccecea92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:37.134 INFO 21062 --- [ocp-async-1,53fb8690625a7428,af19bb74a8efa32c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:37.376 INFO 21062 --- [ocp-schedules-25,f395ad7447215f95,6a647131b61239ff] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:08:37.381 INFO 21062 --- [ocp-schedules-25,f395ad7447215f95,6a647131b61239ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:37.452 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:37.453 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:37.464 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:08:37.467 ERROR 21062 --- [ocp-schedules-25,f395ad7447215f95,6a647131b61239ff] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:08:37.467 INFO 21062 --- [ocp-schedules-25,f395ad7447215f95,6a647131b61239ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:08:38.505 INFO 21062 --- [ocp-schedules-16,0d2aee6dfaba5993,a38829aeb29c4454] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:08:38.505 INFO 21062 --- [ocp-schedules-9,95ee929777d13228,a9645944ef2a4f51] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:08:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:08:38.590350+08:00, lastProcessTime=2024-11-04T14:08:28.589949+08:00 2024-11-04 14:08:38.642 INFO 21062 --- [ocp-async-5,0fa33cd75bf6ceff,9a705789482efd53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:08:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:08:38.590350+08:00 2024-11-04 14:08:38.983 INFO 21062 --- [prometheus-minute-schedule-1,9fd76f34a4108c5e,b578875823ec4b0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:39.097 WARN 21062 --- [ocp-async-9,8a04780ed6a81c51,2adf21c53b1ef614] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:39.134 INFO 21062 --- [ocp-async-10,aa56701759b06a54,c432049cd8b28ac3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:08:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:08:40.011 INFO 21062 --- [ocp-schedules-5,8674a5ead466a082,d790c8cfa73806af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:08:40.216 INFO 21062 --- [ocp-schedules-24,d845e768fade9b84,b1adead9404998d6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:08:40.453 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:40.454 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:41.097 WARN 21062 --- [ocp-async-4,d0c279a2267183f2,eb6897c7890c8067] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:41.135 INFO 21062 --- [ocp-async-7,fd186b55e7c89a64,30d6e0424c16e831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:08:41.822 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:08:41.823 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=25ms, period=120000ms 2024-11-04 14:08:41.983 INFO 21062 --- [prometheus-minute-schedule-1,06304b4fb322419e,c40ec7a929723b90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:43.097 WARN 21062 --- [ocp-async-8,03f19f0d25bfc2f6,7bbae0ebd6aea965] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:43.135 INFO 21062 --- [ocp-async-6,f6bcfb29af300c0e,32f33806969583bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:43.454 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:43.455 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:44.983 INFO 21062 --- [prometheus-minute-schedule-1,0bf1f03533297632,d2b2792cb2856005] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:45.098 WARN 21062 --- [ocp-async-2,37c935a8f4e73aac,600cf60e42c5d545] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:45.135 INFO 21062 --- [ocp-async-3,2adc7d332518bcd0,2808cd3ad744a8f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:46.455 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:46.457 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:47.098 WARN 21062 --- [ocp-async-10,db16eeb24ae8883c,b1e623aa7d2df672] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:47.135 INFO 21062 --- [ocp-async-5,49ebc55b4d21b3fa,a67228639c084f77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:47.983 INFO 21062 --- [prometheus-minute-schedule-1,55d61366caa3f30e,6788a821007af28b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:08:48.591646+08:00, lastProcessTime=2024-11-04T14:08:38.590350+08:00 2024-11-04 14:08:48.642 INFO 21062 --- [ocp-async-9,80646b62d4b20a67,7ce356f8a6cd7230] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:08:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:08:48.591646+08:00 2024-11-04 14:08:49.098 WARN 21062 --- [ocp-async-8,8c4a51cf68a2259b,e10dcda2eb304179] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:49.135 INFO 21062 --- [ocp-async-6,0b40e22771f21b3f,63e9e8e7978293fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:49.457 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:49.458 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:50.038 INFO 21062 --- [ocp-schedules-3,e966b162fba260d0,eea156e7847d4f7e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:08:50.219 INFO 21062 --- [ocp-schedules-29,c7098f5c6d19dc96,c7706ec63c6b84d0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:08:50.983 INFO 21062 --- [prometheus-minute-schedule-1,44f87f86699eb580,34cddb5e559dc40c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:51.098 WARN 21062 --- [ocp-async-2,0fa742a9c754c6bd,8d2ed29e0d2ca752] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:51.127 INFO 21062 --- [ocp-schedules-23,bbe4d1799d43aeb6,96601133a082d7af] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:08:51.134 INFO 21062 --- [ocp-schedules-23,bbe4d1799d43aeb6,96601133a082d7af] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:08:51.134 INFO 21062 --- [ocp-schedules-23,bbe4d1799d43aeb6,96601133a082d7af] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:08:51.135 INFO 21062 --- [ocp-async-3,f7622aab6fcee986,dc785bd6f383bba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:52.459 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:52.460 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:53.098 WARN 21062 --- [ocp-async-10,fa033784f397a382,57738a2b743f619b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:53.136 INFO 21062 --- [ocp-async-5,5df7fe22efebab20,7936ee42ed89fe63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:53.983 INFO 21062 --- [prometheus-minute-schedule-1,352cf25a0cde7800,146446654dc0f7ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:55.098 WARN 21062 --- [ocp-async-7,4b3762800646958f,0a03676f331fff8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:55.136 INFO 21062 --- [ocp-async-1,472ac14d7ccc8dc7,114d8f9d9e464eed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:55.460 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:55.461 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c31f77f716d32627,e04cba7760e5cec1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:08:57.098 WARN 21062 --- [ocp-async-6,2bcfe53808a69c32,834e53f673bdc974] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:57.136 INFO 21062 --- [ocp-async-9,eed1bc624a802d93,69104cfacda827b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:58.461 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:08:58.463 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:08:58.500 INFO 21062 --- [ocp-schedules-20,69ad03a514b6c0d3,3d5cd601aab66a2b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:08:58.500 INFO 21062 --- [ocp-schedules-14,9135ceadac185bf4,2290ef1b50306478] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:08:58.503 INFO 21062 --- [ocp-schedules-4,6b748ad9a2a206b4,d43fbd91a969a620] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:08:58.504 INFO 21062 --- [ocp-schedules-20,69ad03a514b6c0d3,3d5cd601aab66a2b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:58.504 INFO 21062 --- [ocp-schedules-14,9135ceadac185bf4,2290ef1b50306478] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:08:58.505 INFO 21062 --- [ocp-schedules-18,7e858a3e4301e2df,86fa273379c14972] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:08:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:08:58.512 INFO 21062 --- [ocp-schedules-6,661a4f9d5acf49fa,e10c5eaca5e5589a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:08:58.512 INFO 21062 --- [ocp-schedules-6,661a4f9d5acf49fa,e10c5eaca5e5589a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:08:58.512 INFO 21062 --- [ocp-schedules-21,2057c1deaf6888f3,9737e9159aaeeebd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:08:58.512 INFO 21062 --- [ocp-schedules-21,2057c1deaf6888f3,9737e9159aaeeebd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:08:58.512 INFO 21062 --- [ocp-schedules-3,63a043e3791f814e,69597f5cb6c90460] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:08:58.512 INFO 21062 --- [ocp-schedules-3,63a043e3791f814e,69597f5cb6c90460] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:08:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:08:58.590572+08:00, lastProcessTime=2024-11-04T14:08:48.591646+08:00 2024-11-04 14:08:58.643 INFO 21062 --- [ocp-async-8,5ccb5a20452e5a40,6f16dc21dfbbe721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:08:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:08:58.590572+08:00 2024-11-04 14:08:59.083 INFO 21062 --- [ocp-schedules-10,cc2b613eeb7cb41e,dc99d65f95e5c52c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:08:59.099 WARN 21062 --- [ocp-async-10,0aff2be043c9fa78,dae025b9eb616119] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:08:59.136 INFO 21062 --- [ocp-async-5,bd5e1aa5293e398e,e714277f6f4f26b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:08:59.983 INFO 21062 --- [prometheus-minute-schedule-1,31a4bd0beb901a2c,fb31817558ac065d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700540, interval=60 2024-11-04 14:09:00.081 INFO 21062 --- [ocp-schedules-30,b8db6c4d927d98b3,8333049b7fd52b2f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:09:00.222 INFO 21062 --- [ocp-schedules-15,707ccaf3c17910d4,814a3ebca47ef276] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:09:01.099 WARN 21062 --- [ocp-async-7,cc1e58cf2543ce44,8ca571e58179ed27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:01.136 INFO 21062 --- [ocp-async-1,f4a305fe31f90d92,ba660c581b7dd016] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:01.463 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:01.464 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f6d890fe78698ef5,6cf491a897cc9d48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:03.099 WARN 21062 --- [ocp-async-1,c2036faa93bd3dd8,5e031a33d7c64689] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:03.137 INFO 21062 --- [ocp-async-9,2d42c02c9bd14e8d,3e83b1c19cb2340a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:04.464 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:04.465 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:05.099 WARN 21062 --- [ocp-async-9,9329b5f034f7a39c,13b8ebf1e51edc8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:05.135 INFO 21062 --- [ocp-schedules-12,80846019521fa0a6,fb697c9aeb0527eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:09:05.135 INFO 21062 --- [ocp-schedules-12,ad7a3bb55db19dc1,7b46ea79c9d6ce8f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:09:05.137 INFO 21062 --- [ocp-async-4,0e3a2ce58fe27094,e49c902c2849a885] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:05.139 INFO 21062 --- [ocp-schedules-12,ad7a3bb55db19dc1,7b46ea79c9d6ce8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:05.169 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:09:05.169 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:09:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0d51dcd73b762e34,b4914d644366938f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:07.099 WARN 21062 --- [ocp-async-4,94b29d8528962616,e189cf3956082bac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:07.137 INFO 21062 --- [ocp-async-8,bee49315b277b477,ed60083124ce9a97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:07.465 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:07.467 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:08.505 INFO 21062 --- [ocp-schedules-14,9135ceadac185bf4,2290ef1b50306478] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:09:08.506 INFO 21062 --- [ocp-schedules-20,69ad03a514b6c0d3,3d5cd601aab66a2b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:09:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:09:08.590937+08:00, lastProcessTime=2024-11-04T14:08:58.590572+08:00 2024-11-04 14:09:08.643 INFO 21062 --- [ocp-async-3,bc19ca49ccfc7be4,97f0c95e10d28a0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:09:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:09:08.590937+08:00 2024-11-04 14:09:08.950 INFO 21062 --- [ocp-schedules-22,94eeb4e686aa4a8a,8fb75a64ab6f065d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:09:08.954 INFO 21062 --- [ocp-schedules-22,94eeb4e686aa4a8a,8fb75a64ab6f065d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:08.954 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:09:08.983 INFO 21062 --- [prometheus-minute-schedule-1,27729089e11350e1,f89fb417190f0133] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:09.100 WARN 21062 --- [ocp-async-1,e2bb7bed6b92742f,5a49627d349ec22f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:09.137 INFO 21062 --- [ocp-async-9,74dede53ff57d073,6a9b3571ff76ac9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:09:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:09:10.105 INFO 21062 --- [ocp-schedules-15,e5ce8eca4bb9a2a4,a626e3b3ece66a7c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:09:10.225 INFO 21062 --- [ocp-schedules-25,21f21725536cf8c0,b2ab5f9307880a49] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:09:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:09:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:09:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:09:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:09:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:09:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:09:10.467 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:10.467 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:11.100 WARN 21062 --- [ocp-async-9,cb73477804e0d499,9925d27647a887fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:11.137 INFO 21062 --- [ocp-schedules-26,5eae749d3cf2b8fd,60ebceb09cb0ce32] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:09:11.137 INFO 21062 --- [ocp-async-4,c5360c2b3da48561,8295d33f61fce5a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:11.143 INFO 21062 --- [ocp-schedules-26,5eae749d3cf2b8fd,60ebceb09cb0ce32] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:09:11.143 INFO 21062 --- [ocp-schedules-26,5eae749d3cf2b8fd,60ebceb09cb0ce32] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:09:11.492 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=1, affectRows=1 2024-11-04 14:09:11.492 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=1, validCount=1 2024-11-04 14:09:11.492 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=3, affectRows=3 2024-11-04 14:09:11.492 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=3, validCount=3 2024-11-04 14:09:11.493 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=21, affectRows=21 2024-11-04 14:09:11.493 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=21, validCount=21 2024-11-04 14:09:11.493 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=14, affectRows=14 2024-11-04 14:09:11.493 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=13, validCount=13 2024-11-04 14:09:11.493 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=48, affectRows=48 2024-11-04 14:09:11.493 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=33, affectRows=33 2024-11-04 14:09:11.493 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=12, affectRows=12 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=20, affectRows=20 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=12, validCount=12 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=20, validCount=20 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=33, validCount=33 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=48, validCount=48 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=56, affectRows=56 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=73, affectRows=73 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=44, affectRows=44 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=56, validCount=56 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=44, validCount=44 2024-11-04 14:09:11.494 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=72, validCount=72 2024-11-04 14:09:11.495 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700480, nodeCount=67, affectRows=67 2024-11-04 14:09:11.495 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700480, nodeCount=66, validCount=66 2024-11-04 14:09:11.983 INFO 21062 --- [prometheus-minute-schedule-1,74249664d14a4484,9319150b46b1f3a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:13.100 WARN 21062 --- [ocp-async-4,17fe1b3efcd34115,2ca6bec1f25a1a4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:13.137 INFO 21062 --- [ocp-async-8,98e23ffbcd1db0c3,234162caa7c63e21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:13.468 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:13.468 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:13.984 INFO 21062 --- [prometheus-cache-monitor1,95d1cb5a3bcdc7b9,cb293264937c0690] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:09:13.984 INFO 21062 --- [prometheus-cache-monitor1,95d1cb5a3bcdc7b9,cb293264937c0690] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:09:14.923 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:05:14.9, end=2024-11-04T14:07:14.92} 2024-11-04 14:09:14.927 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:09:14.927 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:09:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9be06aaab9a2e252,0e162bed9c502a81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:15.100 WARN 21062 --- [ocp-async-8,72a448084b0ab1b7,e9f3ff1858d19bb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:15.138 INFO 21062 --- [ocp-async-6,f400ebf46970b769,72805dd16212ff3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:16.469 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:16.470 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:17.100 WARN 21062 --- [ocp-async-6,be42863090d1db41,c65fc93f69da899c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:17.138 INFO 21062 --- [ocp-async-3,92ad560dd097460c,84f62d639fc9652c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d8a57f908cdcf7a2,a96f82814e7e9ee7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:09:18.591186+08:00, lastProcessTime=2024-11-04T14:09:08.590937+08:00 2024-11-04 14:09:18.643 INFO 21062 --- [ocp-async-2,6580297aca37c87a,f66ae4cf337fef08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:09:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:09:18.591186+08:00 2024-11-04 14:09:18.954 INFO 21062 --- [ocp-schedules-22,94eeb4e686aa4a8a,8fb75a64ab6f065d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:09:19.100 WARN 21062 --- [ocp-async-4,44bab7cacb60a283,7a7fb00a0b5769f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:19.138 INFO 21062 --- [ocp-async-8,36e72d8aba71caec,d4503f8dab1965b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:19.470 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:19.471 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:20.146 INFO 21062 --- [ocp-schedules-30,e80e85e0f3d4e044,20e9f37416bff7a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:09:20.227 INFO 21062 --- [ocp-schedules-10,96ba0a474e7cccd7,65605e15792bfb8e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:09:20.983 INFO 21062 --- [prometheus-minute-schedule-1,fb86ffcdfe112af3,bcc6f33c8325e2f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:21.101 WARN 21062 --- [ocp-async-8,cd4e32a4698de5ca,8cb72d4cba4d7e94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:21.138 INFO 21062 --- [ocp-async-6,e73eebf63339cd75,0c321d2c62ea8ccf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:22.471 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:22.472 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:23.101 WARN 21062 --- [ocp-async-6,0a022e05abcc5ced,812fb91f3924558c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:23.138 INFO 21062 --- [ocp-async-3,1294262d740f81ab,4714b521a498fec3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:23.983 INFO 21062 --- [prometheus-minute-schedule-1,46783e666834a32d,07288ca9116bd897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:25.101 WARN 21062 --- [ocp-async-3,1e68b5cc29c7bddf,b15257a17cbc67d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:25.138 INFO 21062 --- [ocp-async-5,918233c2d416b42d,78074b2b622834da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:25.473 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:25.474 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:26.983 INFO 21062 --- [prometheus-minute-schedule-1,fe8c12758ebbac30,9288f9867214ec11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4a8d129063f01b44,0f4ed4727a58405e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:09:27.101 WARN 21062 --- [ocp-async-5,f77513892401c4ea,905ada70920515d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:27.138 INFO 21062 --- [ocp-async-1,9fd717709171d960,ce8609001c8f6102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:28.474 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:28.475 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:28.498 INFO 21062 --- [ocp-schedules-9,8a01b1ccdf1f7518,489fe3ecccdca4dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:09:28.500 INFO 21062 --- [ocp-schedules-4,6436d47957cbdd6d,0de8877ffef236a5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:09:28.500 INFO 21062 --- [ocp-schedules-2,8cc006ce1ac0e180,a1d0c2e124da917a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:09:28.500 INFO 21062 --- [ocp-schedules-8,9aeee19db66d64d7,2ae67093362c909b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:09:28.502 INFO 21062 --- [ocp-schedules-19,b176b3ef9ddf8f19,5c50157cfcee031c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:09:28.503 INFO 21062 --- [ocp-schedules-9,8a01b1ccdf1f7518,489fe3ecccdca4dc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:09:28.504 INFO 21062 --- [ocp-schedules-8,9aeee19db66d64d7,2ae67093362c909b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:28.504 INFO 21062 --- [ocp-schedules-4,6436d47957cbdd6d,0de8877ffef236a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:09:28.504 INFO 21062 --- [ocp-schedules-3,013af70500fd4ea5,4ddfbe011ea0fcc3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:09:28.506 INFO 21062 --- [ocp-schedules-2,8cc006ce1ac0e180,a1d0c2e124da917a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:09:28.512 INFO 21062 --- [ocp-schedules-15,ebb6ee90de861588,1ed33adf4415adbe] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:09:28.512 INFO 21062 --- [ocp-schedules-13,d6583a7b35bfaa28,7937394fcdb37776] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:09:28.512 INFO 21062 --- [ocp-schedules-15,ebb6ee90de861588,1ed33adf4415adbe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:09:28.512 INFO 21062 --- [ocp-schedules-6,f9321edcd4462c6b,31509ef5018be69a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:09:28.512 INFO 21062 --- [ocp-schedules-13,d6583a7b35bfaa28,7937394fcdb37776] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:09:28.512 INFO 21062 --- [ocp-schedules-6,f9321edcd4462c6b,31509ef5018be69a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:09:28.581 INFO 21062 --- [ocp-schedules-18,89d9665eca073eca,632aa33c18ef1173] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:09:28.581+08:00, tenantSize=0 2024-11-04 14:09:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:09:28.590132+08:00, lastProcessTime=2024-11-04T14:09:18.591186+08:00 2024-11-04 14:09:28.621 INFO 21062 --- [ocp-schedules-11,d41c8aec157efefc,248c799005b42d03] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:09:28.621 INFO 21062 --- [ocp-schedules-11,d41c8aec157efefc,248c799005b42d03] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:09:28.643 INFO 21062 --- [ocp-async-5,5a29d96cbf02baad,4f0f8a240ff8ccbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:09:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:09:28.590132+08:00 2024-11-04 14:09:28.794 INFO 21062 --- [ocp-schedules-30,9a7ec0671abf7521,63d1e5db697d8bf8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:09:29.029 INFO 21062 --- [ocp-schedules-3,690e54db2030d84a,37c98ef2a3c741ef] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:09:29.029 INFO 21062 --- [ocp-schedules-3,690e54db2030d84a,37c98ef2a3c741ef] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:09:29.086 INFO 21062 --- [ocp-schedules-15,666f67365c25ff31,84c75fb25be1f438] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:09:29.101 WARN 21062 --- [ocp-async-10,ccc1f9283e8d0b99,52a1357dbd032b73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:29.119 INFO 21062 --- [ocp-schedules-9,f77845eeeac7527f,a4cec43e79b9cc64] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:09:29.124 INFO 21062 --- [ocp-schedules-9,f77845eeeac7527f,a4cec43e79b9cc64] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:09:29.124 INFO 21062 --- [ocp-schedules-9,f77845eeeac7527f,a4cec43e79b9cc64] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:09:29.139 INFO 21062 --- [ocp-async-3,38eb7990306d2281,e08cd469827b0327] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:29.416 INFO 21062 --- [ocp-schedules-11,4e7eec58dee9ae49,3f9eb2fcd7ac5ffa] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:09:29.416 INFO 21062 --- [ocp-schedules-11,4e7eec58dee9ae49,3f9eb2fcd7ac5ffa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:09:29.417 INFO 21062 --- [ocp-schedules-11,4e7eec58dee9ae49,3f9eb2fcd7ac5ffa] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:09:29.417 INFO 21062 --- [ocp-schedules-11,4e7eec58dee9ae49,3f9eb2fcd7ac5ffa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:09:29.418 INFO 21062 --- [ocp-schedules-11,4e7eec58dee9ae49,3f9eb2fcd7ac5ffa] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:09:29.418 INFO 21062 --- [ocp-schedules-11,4e7eec58dee9ae49,3f9eb2fcd7ac5ffa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:09:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7ab4d776758ac898,7555ca020a6a22f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:30.173 INFO 21062 --- [ocp-schedules-25,8b1a39ba3d71da34,9bf828f0dcbf5dd9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:09:30.231 INFO 21062 --- [ocp-schedules-15,2b707815d28d617e,88b5f4cc11407f7d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:09:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:09:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:09:31.102 WARN 21062 --- [ocp-async-3,7ee086e9d26c50c2,609744efb4069ca3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:31.139 INFO 21062 --- [ocp-async-5,d5b18a90108474d6,02c384598e4765b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:31.145 INFO 21062 --- [ocp-schedules-2,9707f513632ea834,4f2127d77b01690f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:09:31.152 INFO 21062 --- [ocp-schedules-2,9707f513632ea834,4f2127d77b01690f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:09:31.152 INFO 21062 --- [ocp-schedules-2,9707f513632ea834,4f2127d77b01690f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:09:31.168 INFO 21062 --- [ocp-schedules-19,6db4654a4ecacd8a,369d9d993af073b5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:09:31.168 INFO 21062 --- [ocp-schedules-19,6db4654a4ecacd8a,369d9d993af073b5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:09:31.196 INFO 21062 --- [ocp-schedules-19,6db4654a4ecacd8a,369d9d993af073b5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:09:31.196 INFO 21062 --- [ocp-schedules-19,6db4654a4ecacd8a,369d9d993af073b5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:09:31.196 INFO 21062 --- [ocp-schedules-17,94058a801a6ad93e,a41d509fcf5d0d94] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:09:31.372 INFO 21062 --- [ocp-schedules-25,ca826b521f50de45,9b15bc1beb39bcaa] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:09:31.372+08:00, clusters=[1] 2024-11-04 14:09:31.373 INFO 21062 --- [ocp-schedules-25,ca826b521f50de45,9b15bc1beb39bcaa] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:09:31.460 INFO 21062 --- [ocp-schedules-15,9c127dc3bd7196ad,3d31e4081cce0fb9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:09:31.460 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:09:31.460 INFO 21062 --- [ocp-schedules-15,9c127dc3bd7196ad,3d31e4081cce0fb9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:09:31.460 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:09:31.470 INFO 21062 --- [ocp-schedules-15,9c127dc3bd7196ad,3d31e4081cce0fb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:09:31.470 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:09:31.471 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:09:31.472 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:09:31.472 WARN 21062 --- [ocp-schedules-15,9c127dc3bd7196ad,3d31e4081cce0fb9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:09:31.472 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:09:31.472 INFO 21062 --- [ocp-schedules-15,9c127dc3bd7196ad,3d31e4081cce0fb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:09:31.472 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:09:31.472 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:09:31.473 WARN 21062 --- [ocp-schedules-15,9c127dc3bd7196ad,3d31e4081cce0fb9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:09:31.473 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:09:31.474 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:09:31.475 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:31.476 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:31.483 ERROR 21062 --- [ocp-schedules-15,9c127dc3bd7196ad,3d31e4081cce0fb9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=3 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2024-11-04 14:09:31.517 INFO 21062 --- [ocp-schedules-9,64bf52a31353e382,0ca28aee484230a8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:09:31.517+08:00, clusters=[1] 2024-11-04 14:09:31.521 INFO 21062 --- [ocp-schedules-9,64bf52a31353e382,0ca28aee484230a8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:09:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f9f5f5f78fabf0b2,a47052c439f8876d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:33.102 WARN 21062 --- [ocp-async-5,310995854768bb4c,a9a243d633477d50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:33.139 INFO 21062 --- [ocp-async-6,665ef6cab3b7fab3,d86457531956cf3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:33.223 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:09:33.223 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:09:33.231 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:09:33.231 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:09:33.231 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:09:33.231 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:09:33.237 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:09:33.237 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:09:33.237 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:09:33.237 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:09:33.272 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:09:33.280 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:09:33.280 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:09:33.280 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:09:33.280 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:09:33.280 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:09:33.289 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:09:33.799 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:09:34.385 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.148480991S 2024-11-04 14:09:34.386 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:09:34.476 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:34.477 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:34.488 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:09:34.488 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:09:34.515 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:09:34.577 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:09:34.577 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:09:34.577 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:09:34.577 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:09:34.577 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:09:34.577 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:09:34.590 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:09:34.762 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] o.s.boot.SpringApplication : Started application in 1.531 seconds (JVM running for 5988.442) 2024-11-04 14:09:34.763 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.531696947S 2024-11-04 14:09:35.102 WARN 21062 --- [ocp-async-9,518f82b581641eb3,271a54393331d394] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:35.139 INFO 21062 --- [ocp-async-4,fe1d77e77422ddc3,7b5644224cf40df6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:35.170 INFO 21062 --- [ocp-schedules-12,ad7a3bb55db19dc1,7b46ea79c9d6ce8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:09:35.170 INFO 21062 --- [ocp-schedules-12,1c0dac54ee6fdf84,c9841b23a2b9fbe5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:09:35.174 INFO 21062 --- [ocp-schedules-12,1c0dac54ee6fdf84,c9841b23a2b9fbe5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:35.204 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:09:35.204 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:09:35.228 INFO 21062 --- [ocp-schedules-21,742a3fd14449430d,cfab48e277d64d1f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:09:35.983 INFO 21062 --- [prometheus-minute-schedule-1,60cac1914918a284,40d5da62d2f9e17c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:37.102 WARN 21062 --- [ocp-async-7,3f35cf814652cd8f,a6595df01e131c9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:37.139 INFO 21062 --- [ocp-async-8,ac10f6756895d9ff,eab33c35028a5468] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:37.467 INFO 21062 --- [ocp-schedules-13,469f8eb9ca61022e,e835fe5cafde709e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:09:37.471 INFO 21062 --- [ocp-schedules-13,469f8eb9ca61022e,e835fe5cafde709e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:37.477 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:37.478 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:37.556 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:09:37.559 ERROR 21062 --- [ocp-schedules-13,469f8eb9ca61022e,e835fe5cafde709e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:09:37.559 INFO 21062 --- [ocp-schedules-13,469f8eb9ca61022e,e835fe5cafde709e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:09:38.504 INFO 21062 --- [ocp-schedules-8,9aeee19db66d64d7,2ae67093362c909b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:09:38.505 INFO 21062 --- [ocp-schedules-4,6436d47957cbdd6d,0de8877ffef236a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:09:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:09:38.591184+08:00, lastProcessTime=2024-11-04T14:09:28.590132+08:00 2024-11-04 14:09:38.643 INFO 21062 --- [ocp-async-4,1de7d69e0620d8f5,8c26de29a8903605] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:09:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:09:38.591184+08:00 2024-11-04 14:09:38.983 INFO 21062 --- [prometheus-minute-schedule-1,65ac60bb3ab3e1b6,7b5ec807ce422140] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:39.102 WARN 21062 --- [ocp-async-1,d54a5f70bfd58c4d,c6004cbf0c3e72b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:39.140 INFO 21062 --- [ocp-async-9,2b1775ecb357e913,77629b3deeab31f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:09:40.004 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:09:40.214 INFO 21062 --- [ocp-schedules-9,1d2523ad412327d3,418083a3c8e75e43] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:09:40.234 INFO 21062 --- [ocp-schedules-3,093f2307298de25e,720cef2a2d17917a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:09:40.478 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:40.479 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:41.102 WARN 21062 --- [ocp-async-10,01abe9f1bd9dc7b9,43f7828c00d4efa8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:41.140 INFO 21062 --- [ocp-async-7,7f08ffa1fce00bbc,3988960ce98cc066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:41.983 INFO 21062 --- [prometheus-minute-schedule-1,36198bc1d1fe6f7e,32d7dfe84f2f65cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:43.103 WARN 21062 --- [ocp-async-3,5d0ff99694911207,123ef8bf6df90ec0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:43.140 INFO 21062 --- [ocp-async-2,498aaa7d835e840b,417dc8bd324e0f57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:43.479 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:43.480 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:44.983 INFO 21062 --- [prometheus-minute-schedule-1,cf24881b0d8aebf5,81c2f0dae3c30d63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:45.103 WARN 21062 --- [ocp-async-5,65662a6a4dbef3c2,716819a7a6645462] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:45.140 INFO 21062 --- [ocp-async-6,b1d556e735963381,e044a8047b5e9978] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:46.480 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:46.482 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:47.103 WARN 21062 --- [ocp-async-9,34c0f11a42ddcebe,0cd026e7a037809c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:47.140 INFO 21062 --- [ocp-async-4,4a31474f585c687c,79e2a0a68bc38c71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:47.983 INFO 21062 --- [prometheus-minute-schedule-1,cfb7f8056471fab9,3853089fee2bf028] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:09:48.590854+08:00, lastProcessTime=2024-11-04T14:09:38.591184+08:00 2024-11-04 14:09:48.644 INFO 21062 --- [ocp-async-1,4ce3bee68ef6ca8a,19bd2b0be8a35983] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:09:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:09:48.590854+08:00 2024-11-04 14:09:49.103 WARN 21062 --- [ocp-async-3,359cb1e366454397,aeb02dc50667a6b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:49.140 INFO 21062 --- [ocp-async-2,41ee4db4e3774bd0,99523e2fe869fe01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:49.482 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:49.482 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:50.236 INFO 21062 --- [ocp-schedules-1,637edf4ddccb4d96,41f32b9eea0a313b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:09:50.238 INFO 21062 --- [ocp-schedules-14,a6fed2ee932e945b,09133cd8f09db0c0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:09:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1838895c9aa88b4a,fb12ead190c4496f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:51.103 WARN 21062 --- [ocp-async-5,72b33c581e82c2b6,3e635e6fbd956969] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:51.140 INFO 21062 --- [ocp-async-6,8d6ef0c175f2580c,6ed9111ed824878f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:51.153 INFO 21062 --- [ocp-schedules-7,e254a9f3681932ec,7d00d3e2e05278fa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:09:51.158 INFO 21062 --- [ocp-schedules-7,e254a9f3681932ec,7d00d3e2e05278fa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:09:51.158 INFO 21062 --- [ocp-schedules-7,e254a9f3681932ec,7d00d3e2e05278fa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:09:52.483 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:52.484 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:53.104 WARN 21062 --- [ocp-async-9,be80027df36ea3a6,40e97b2da4ccac7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:53.141 INFO 21062 --- [ocp-async-4,102cdd0c9cf955a1,6ccc7abd136eb661] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:53.983 INFO 21062 --- [prometheus-minute-schedule-1,2c47f24a8a01dc41,5d2a7ea60fe2c5b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:55.104 WARN 21062 --- [ocp-async-4,3b5d7d1305d24466,3cac6fda022425ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:55.141 INFO 21062 --- [ocp-async-8,55c2fcf207fd962e,37679bb1c095effc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:55.484 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:55.485 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:56.983 INFO 21062 --- [prometheus-minute-schedule-1,79de242af61d7e5d,f6515618b7ee0706] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:09:57.104 WARN 21062 --- [ocp-async-8,471f9e04bb6ed30a,c8a53bdb66d5a699] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:57.141 INFO 21062 --- [ocp-async-1,6483eb1ae6701dd9,e4a27f778c959bd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:58.485 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:09:58.486 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:09:58.499 INFO 21062 --- [ocp-schedules-16,b94e4baee6fa7798,8d70eb1d8cd9efc6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:09:58.500 INFO 21062 --- [ocp-schedules-13,c8e610e30309e0fc,f7963090f1eff59d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:09:58.502 INFO 21062 --- [ocp-schedules-26,332b0571c8a68f62,823ce6de360eb54b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:09:58.504 INFO 21062 --- [ocp-schedules-16,b94e4baee6fa7798,8d70eb1d8cd9efc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:58.504 INFO 21062 --- [ocp-schedules-13,c8e610e30309e0fc,f7963090f1eff59d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:09:58.505 INFO 21062 --- [ocp-schedules-19,f3b6c647b1d3dd41,a58aafb2129e66fe] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:09:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:09:58.510 INFO 21062 --- [ocp-schedules-28,e270177ee5a1489f,1d35d000a957cb23] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:09:58.510 INFO 21062 --- [ocp-schedules-28,e270177ee5a1489f,1d35d000a957cb23] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:09:58.511 INFO 21062 --- [ocp-schedules-7,88428fb44f8901e2,34d28d2ec2ee5af3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:09:58.511 INFO 21062 --- [ocp-schedules-7,88428fb44f8901e2,34d28d2ec2ee5af3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:09:58.512 INFO 21062 --- [ocp-schedules-24,d5506306f437984a,3b6dadcb96461812] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:09:58.512 INFO 21062 --- [ocp-schedules-24,d5506306f437984a,3b6dadcb96461812] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:09:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:09:58.589953+08:00, lastProcessTime=2024-11-04T14:09:48.590854+08:00 2024-11-04 14:09:58.644 INFO 21062 --- [ocp-async-3,4a132bd1dab2c3e9,7ca19fc1d53c369f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:09:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:09:58.589953+08:00 2024-11-04 14:09:59.089 INFO 21062 --- [ocp-schedules-2,b641071a1cd3e9cb,4e0deafb7375508f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:09:59.104 WARN 21062 --- [ocp-async-6,59454c5d92d9e35c,240b1afce530e10a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:09:59.141 INFO 21062 --- [ocp-async-4,5fcfb8f4a824f6c6,a63cacd62c1cb8d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:09:59.983 INFO 21062 --- [prometheus-minute-schedule-1,064ad605633d2651,c5023eb5b9320fa0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700600, interval=60 2024-11-04 14:10:00.239 INFO 21062 --- [ocp-schedules-27,73ced402ca8127f9,76670a963e516cdf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:10:00.280 INFO 21062 --- [ocp-schedules-20,72a38654c68c316a,3e015a83dbbcc084] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:10:00.898 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0632f877b65b27b7] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:05:00.0, nextRun=2024-11-04 14:10:00.0, currentDate=Mon Nov 04 14:10:00 CST 2024 2024-11-04 14:10:00.898 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0632f877b65b27b7] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:05:00.0, nextRunTime=2024-11-04 14:10:00.0} 2024-11-04 14:10:00.925 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0632f877b65b27b7] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:10:00.957 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0632f877b65b27b7] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:10:00.957 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,0632f877b65b27b7] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:10:01.104 WARN 21062 --- [ocp-async-4,ef90cf917ccf9915,82a1a76920d4d76d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:01.141 INFO 21062 --- [ocp-async-8,7a08c955c0a70f4c,994131014cabd224] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:01.486 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:01.487 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:02.983 INFO 21062 --- [prometheus-minute-schedule-1,0a8496ec401f3d11,21f86f40919e3bfd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:03.105 WARN 21062 --- [ocp-async-8,7eb5e54179a3e2c1,946d42b203155de7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:03.142 INFO 21062 --- [ocp-async-1,32219257ebaf1229,2a61c39050fcf202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:04.010 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,06bb8ce20a5774c1] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:00:04.0, nextRun=2024-11-04 14:10:04.0, currentDate=Mon Nov 04 14:10:04 CST 2024 2024-11-04 14:10:04.011 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,06bb8ce20a5774c1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 14:00:04.0, nextRunTime=2024-11-04 14:10:04.0} 2024-11-04 14:10:04.016 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,06bb8ce20a5774c1] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 14:10:04.016 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,06bb8ce20a5774c1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T14:10:04.016+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 14:10:04.487 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:04.488 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:04.551 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,76cf8d638edc6f03] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 79, operation:EXECUTE, state: RUNNING 2024-11-04 14:10:04.570 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=79, context=Context{parallelIdx=-1, stringMap={task_instance_id=15, task_operation=execute, latest_execution_start_time=2024-11-04T14:10:04.543+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=79}, listMap={}}, executor=10.46.128.24 2024-11-04 14:10:04.575 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:10:04.575 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:10:04.577 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.578 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:10:04.579 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.580 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.580 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.581 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:10:04.597 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.597 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.598 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.598 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:10:04.611 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.612 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.612 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.613 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:10:04.627 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.627 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.628 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.628 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:10:04.642 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.642 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.643 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.643 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:10:04.648 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.649 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.649 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.649 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:10:04.654 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.654 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.655 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.655 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:10:04.659 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.660 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.660 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.660 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:10:04.667 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.667 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.668 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.668 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:10:04.673 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.674 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.674 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.675 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:10:04.680 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.681 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.681 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.682 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:10:04.690 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.690 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.691 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.691 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:10:04.704 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.704 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.705 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.706 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:10:04.710 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.711 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.711 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.712 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:10:04.717 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.718 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.718 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.719 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:10:04.724 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.724 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.725 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.725 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:10:04.730 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.730 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.731 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.731 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:10:04.735 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.736 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.736 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.737 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:10:04.741 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.741 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.742 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.742 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:10:04.747 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.747 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.748 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.748 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:10:04.752 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.753 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.753 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.753 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:10:04.758 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.758 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.759 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.759 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:10:04.764 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.765 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.765 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.766 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:10:04.770 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.770 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.771 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.771 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:10:04.777 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.777 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.778 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.778 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:10:04.782 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.783 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.783 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.784 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:10:04.788 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.789 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.789 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.790 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:10:04.794 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.795 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.795 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.796 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:10:04.800 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.800 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.801 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.801 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:10:04.806 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.806 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.807 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.807 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:10:04.811 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.811 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.812 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.812 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:10:04.817 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.817 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.818 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.818 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:10:04.822 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.823 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.823 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.824 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:10:04.828 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.828 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.829 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.829 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:10:04.837 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.837 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.838 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.838 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:10:04.844 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.845 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.845 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.846 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:10:04.851 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.852 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.852 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.853 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:10:04.858 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.858 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.859 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.859 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:10:04.862 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.863 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.864 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.864 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:10:04.867 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.868 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.868 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.868 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:10:04.872 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.872 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.873 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.873 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:10:04.877 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.877 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.878 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.878 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:10:04.887 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.887 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.888 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.888 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:10:04.908 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.908 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.909 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.909 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:10:04.915 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.915 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.916 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.916 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:10:04.921 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.922 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.923 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.923 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:10:04.929 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.929 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.930 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.930 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:10:04.934 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.935 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.935 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.936 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:10:04.939 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.939 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.940 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.940 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:10:04.945 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:10:04.945 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:10:04.946 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.946 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:10:04.947 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.948 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.948 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.949 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:10:04.954 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.955 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:10:04.955 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:04.956 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:10:04.960 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:10:04.968 INFO 21062 --- [subtask-executor12,dd193f8f67a5a423,1e55321c81469b4d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 79, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:10:05.105 WARN 21062 --- [ocp-async-1,cc230808ba939917,ac9eb5770a3ff8ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:05.142 INFO 21062 --- [ocp-async-10,16e573573b004230,b5c982dfaba396f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:05.204 INFO 21062 --- [ocp-schedules-12,1c0dac54ee6fdf84,c9841b23a2b9fbe5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:10:05.205 INFO 21062 --- [ocp-schedules-12,ceae2185f84651b2,77feb1358724a5f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:10:05.208 INFO 21062 --- [ocp-schedules-12,ceae2185f84651b2,77feb1358724a5f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:05.237 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:10:05.237 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:10:05.702 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:15, status:SUCCESS 2024-11-04 14:10:05.703 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:15, status:SUCCESS 2024-11-04 14:10:05.983 INFO 21062 --- [prometheus-minute-schedule-1,81737e927e610365,6fed554a05239c03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:07.105 WARN 21062 --- [ocp-async-10,68056d09d43eb322,4a1e4c40aca8f0a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:07.142 INFO 21062 --- [ocp-async-3,841967ec8b9cb1d7,69d3242293f6ced0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:07.488 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:07.489 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:08.505 INFO 21062 --- [ocp-schedules-13,c8e610e30309e0fc,f7963090f1eff59d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:10:08.506 INFO 21062 --- [ocp-schedules-16,b94e4baee6fa7798,8d70eb1d8cd9efc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:10:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:10:08.591406+08:00, lastProcessTime=2024-11-04T14:09:58.589953+08:00 2024-11-04 14:10:08.644 INFO 21062 --- [ocp-async-9,de7d1fb6ad5be82c,cf60c4b995305e37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:10:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:10:08.591406+08:00 2024-11-04 14:10:08.983 INFO 21062 --- [prometheus-minute-schedule-1,18acf0a7b724489d,d8a35a8c63a0a975] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:09.105 WARN 21062 --- [ocp-async-8,f8aaa162c91f5bb8,41f6eafe992e9fc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:09.142 INFO 21062 --- [ocp-async-1,315fa30e046bf8ff,0a67c90efa9900f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:10:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:10:10.242 INFO 21062 --- [ocp-schedules-14,6fefa1d976f5ac46,2a98e34be6be6c58] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:10:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:10:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:10:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:10:10.308 INFO 21062 --- [ocp-schedules-30,0a4c5bb7d4009f51,6334814c4aa17a08] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:10:10.329 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:10:10.329 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:10:10.330 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:10:10.489 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:10.490 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:11.105 WARN 21062 --- [ocp-async-1,3063aabdc814559e,e2de02ee7edfa278] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:11.142 INFO 21062 --- [ocp-async-10,0eb2d074314cfc65,30dbf412ac129a1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:11.161 INFO 21062 --- [ocp-schedules-7,599fefa464db8389,ad880b31fb7ebaed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:10:11.166 INFO 21062 --- [ocp-schedules-7,599fefa464db8389,ad880b31fb7ebaed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:10:11.166 INFO 21062 --- [ocp-schedules-7,599fefa464db8389,ad880b31fb7ebaed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:10:11.495 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=5, affectRows=5 2024-11-04 14:10:11.495 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=5, validCount=5 2024-11-04 14:10:11.496 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=3, affectRows=3 2024-11-04 14:10:11.496 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=3, validCount=3 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=86, affectRows=86 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=18, affectRows=18 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=52, affectRows=52 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=28, affectRows=28 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=41, affectRows=41 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=18, validCount=18 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=28, validCount=28 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=52, validCount=52 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=40, validCount=40 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=85, validCount=85 2024-11-04 14:10:11.497 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=72, affectRows=72 2024-11-04 14:10:11.498 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=72, validCount=72 2024-11-04 14:10:11.498 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700540, nodeCount=87, affectRows=87 2024-11-04 14:10:11.498 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700540, nodeCount=86, validCount=86 2024-11-04 14:10:11.983 INFO 21062 --- [prometheus-minute-schedule-1,1faf09e0ba9fa104,4c9b7bccbcaea73a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:13.105 WARN 21062 --- [ocp-async-10,c9233f2a1e6e37e5,e0ec2481c52eea80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:13.143 INFO 21062 --- [ocp-async-3,5cb14d048b0cf9a2,7dd044d54b9e6b8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:13.490 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:13.491 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:13.984 INFO 21062 --- [prometheus-cache-monitor1,8e14cee73e1e9c5f,4cea713d6f71eb5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:10:13.984 INFO 21062 --- [prometheus-cache-monitor1,8e14cee73e1e9c5f,4cea713d6f71eb5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:10:14.983 INFO 21062 --- [prometheus-minute-schedule-1,40f0abb540e6504f,73a31eeaa73b183b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:15.105 WARN 21062 --- [ocp-async-3,e39d71c1d719e45e,b3ae0fef9733ad6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:15.143 INFO 21062 --- [ocp-async-2,ad56e9431d67d7b3,043b0a18adac4b5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:16.491 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:16.492 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:17.106 WARN 21062 --- [ocp-async-2,1696584968885d0c,e592f965c0d7be8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:17.143 INFO 21062 --- [ocp-async-6,0744f18b2b171eb5,ece46f87fda7b6eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5c2f75acc1e6ba6e,cae72c13b3856939] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:10:18.590933+08:00, lastProcessTime=2024-11-04T14:10:08.591406+08:00 2024-11-04 14:10:18.644 INFO 21062 --- [ocp-async-8,30d3f69e995c64c4,c9fbc8998f1e9069] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:10:18.810 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:10:18.590933+08:00 2024-11-04 14:10:18.955 INFO 21062 --- [ocp-schedules-24,541cb0002a7208d5,0b5a617fd1a5b810] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:10:18.960 INFO 21062 --- [ocp-schedules-24,541cb0002a7208d5,0b5a617fd1a5b810] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:18.961 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:10:19.106 WARN 21062 --- [ocp-async-10,3d8ce8be091ec28a,8a46c8b7442d3e51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:19.143 INFO 21062 --- [ocp-async-7,50df846898b8a42e,70fe4b91dbb9d5e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:19.492 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:19.493 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:20.245 INFO 21062 --- [ocp-schedules-8,43a94126f8e9e852,95b3959f2f1cb933] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:10:20.349 INFO 21062 --- [ocp-schedules-26,fe726c814a3e122a,dfae0f05ae343355] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:10:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9d02430bfead9e1c,7ccda697c451e634] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:21.106 WARN 21062 --- [ocp-async-7,d1a8371f7dbdbda6,1896afa37d15b7cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:21.143 INFO 21062 --- [ocp-async-8,69095d587f9dfb95,07bf3adb79e905c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:22.493 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:22.495 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:23.106 WARN 21062 --- [ocp-async-8,945eb014d252abaf,cd58d7c612b32117] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:23.143 INFO 21062 --- [ocp-async-9,28c58e21346e9825,9a2d719932b32e23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b060295595302dca,2c004bc151b5464a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:25.107 WARN 21062 --- [ocp-async-6,893e50114fdfb4b1,e4e9a4f8815a03d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:25.144 INFO 21062 --- [ocp-async-4,cff00342e82bf605,38dbd4db047adae4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:25.495 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:25.496 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:26.983 INFO 21062 --- [prometheus-minute-schedule-1,643469f9f8348b2e,2d25aeaf7fb7d278] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9c06d4b537ecb420,7b5c0a7190c412dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:10:27.107 WARN 21062 --- [ocp-async-3,ddd77a17ab1cdc3a,48527dff8313298c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:27.144 INFO 21062 --- [ocp-async-5,500abb56041a5fc6,70366bc824017caa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:28.496 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:28.497 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:28.498 INFO 21062 --- [ocp-schedules-19,f9edf74b78fa7006,6f9468c5c682a132] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:10:28.499 INFO 21062 --- [ocp-schedules-28,8dc8ae0ab397e587,e809d22de9b7d97b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:10:28.500 INFO 21062 --- [ocp-schedules-17,20c1a1adc5424b9d,af0f4e440884f88d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:10:28.500 INFO 21062 --- [ocp-schedules-9,7f42139a61fab2e7,60bafd0f9e0c2773] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:10:28.502 INFO 21062 --- [ocp-schedules-19,f9edf74b78fa7006,6f9468c5c682a132] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:10:28.502 INFO 21062 --- [ocp-schedules-27,f799c2ff3bb6c038,b1e162163c7cef78] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:10:28.503 INFO 21062 --- [ocp-schedules-17,20c1a1adc5424b9d,af0f4e440884f88d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:28.504 INFO 21062 --- [ocp-schedules-9,7f42139a61fab2e7,60bafd0f9e0c2773] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:10:28.504 INFO 21062 --- [ocp-schedules-26,725e425aaf8f62a6,4d8844c917432114] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:10:28.505 INFO 21062 --- [ocp-schedules-28,8dc8ae0ab397e587,e809d22de9b7d97b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:10:28.509 INFO 21062 --- [ocp-schedules-4,bb73270c30e73a4b,714929c5feb07baa] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:10:28.509 INFO 21062 --- [ocp-schedules-25,100566208c51bf87,553b57b0dd143438] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:10:28.509 INFO 21062 --- [ocp-schedules-2,4b5423570d121952,e14a3be8c90ed202] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:10:28.509 INFO 21062 --- [ocp-schedules-2,4b5423570d121952,e14a3be8c90ed202] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:10:28.509 INFO 21062 --- [ocp-schedules-25,100566208c51bf87,553b57b0dd143438] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:10:28.509 INFO 21062 --- [ocp-schedules-4,bb73270c30e73a4b,714929c5feb07baa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:10:28.573 INFO 21062 --- [ocp-schedules-8,1db09bb810f295c4,ba98e43680d78b5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":391,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:10:28.573 INFO 21062 --- [ocp-schedules-8,1db09bb810f295c4,ba98e43680d78b5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:10:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:10:28.590137+08:00, lastProcessTime=2024-11-04T14:10:18.590933+08:00 2024-11-04 14:10:28.645 INFO 21062 --- [ocp-async-3,23bf5f5aa11768f9,6eb4202a52131129] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:10:28.760 INFO 21062 --- [ocp-schedules-10,ab48a862999ea93e,bba2e7b5dcffd4bf] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:10:28.760 INFO 21062 --- [ocp-schedules-10,ab48a862999ea93e,bba2e7b5dcffd4bf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:10:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:10:28.590137+08:00 2024-11-04 14:10:28.796 INFO 21062 --- [ocp-schedules-13,f68fe0b8df06cf35,7e6362d943b014a4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:10:28.961 INFO 21062 --- [ocp-schedules-24,541cb0002a7208d5,0b5a617fd1a5b810] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:10:29.034 INFO 21062 --- [ocp-schedules-30,4deb0ef9fe8d2362,22b2a7a111e4d85f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:10:29.034 INFO 21062 --- [ocp-schedules-30,4deb0ef9fe8d2362,22b2a7a111e4d85f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:10:29.091 INFO 21062 --- [ocp-schedules-25,4132ad33ca4555cb,453491a2dc2a80bd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:10:29.107 WARN 21062 --- [ocp-async-9,9aaed85aedec07e2,76eab19b9eb47c0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:29.124 INFO 21062 --- [ocp-schedules-14,ad9b18b7546d4b0f,74aa5f7d19b2c91e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:10:29.130 INFO 21062 --- [ocp-schedules-14,ad9b18b7546d4b0f,74aa5f7d19b2c91e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:10:29.130 INFO 21062 --- [ocp-schedules-14,ad9b18b7546d4b0f,74aa5f7d19b2c91e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:10:29.144 INFO 21062 --- [ocp-async-6,b2226fe6524bba91,4e71cdc6ddca2250] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:29.249 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:10:29.249 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:10:29.257 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:10:29.257 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:10:29.257 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:10:29.268 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:10:29.268 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:10:29.268 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:10:29.271 INFO 21062 --- [ocp-schedules-3,e138008d6f6ed599,60e145912a595852] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:10:29.419 INFO 21062 --- [ocp-schedules-6,85f78880a2639ca6,ab1486e9a769a562] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:10:29.419 INFO 21062 --- [ocp-schedules-6,85f78880a2639ca6,ab1486e9a769a562] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:10:29.427 INFO 21062 --- [ocp-schedules-6,85f78880a2639ca6,ab1486e9a769a562] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:10:29.427 INFO 21062 --- [ocp-schedules-6,85f78880a2639ca6,ab1486e9a769a562] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:10:29.433 INFO 21062 --- [ocp-schedules-6,85f78880a2639ca6,ab1486e9a769a562] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:10:29.433 INFO 21062 --- [ocp-schedules-6,85f78880a2639ca6,ab1486e9a769a562] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:10:29.983 INFO 21062 --- [prometheus-minute-schedule-1,ceab1f36cc64b217,ff73bf1a34598e20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:30.249 INFO 21062 --- [ocp-schedules-25,1257194d9812463b,57427c7f319107c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:10:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:10:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:10:30.375 INFO 21062 --- [ocp-schedules-4,98d4a2eda4368f53,b9e99a84eb27b0ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:10:31.107 WARN 21062 --- [ocp-async-4,bbf5f39a5563678e,d914a132af0247b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:31.144 INFO 21062 --- [ocp-async-7,8660dc889ef97d5a,c974d3ad18dddecf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:31.168 INFO 21062 --- [ocp-schedules-7,4031b5b7f3c7a994,e746cde44dd92865] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:10:31.174 INFO 21062 --- [ocp-schedules-7,4031b5b7f3c7a994,e746cde44dd92865] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:10:31.174 INFO 21062 --- [ocp-schedules-7,4031b5b7f3c7a994,e746cde44dd92865] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:10:31.196 INFO 21062 --- [ocp-schedules-28,0a0aac15fc2d4868,01e0d92c4cc1db58] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:10:31.196 INFO 21062 --- [ocp-schedules-28,0a0aac15fc2d4868,01e0d92c4cc1db58] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:10:31.225 INFO 21062 --- [ocp-schedules-28,0a0aac15fc2d4868,01e0d92c4cc1db58] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:10:31.225 INFO 21062 --- [ocp-schedules-28,0a0aac15fc2d4868,01e0d92c4cc1db58] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:10:31.226 INFO 21062 --- [ocp-schedules-18,340b1a67d6b4011d,89277e1cc30ee7c7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:10:31.399 INFO 21062 --- [ocp-schedules-2,29ac0e40527a7a57,0e454f8d5efdacad] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:10:31.399+08:00, clusters=[1] 2024-11-04 14:10:31.401 INFO 21062 --- [ocp-schedules-2,29ac0e40527a7a57,0e454f8d5efdacad] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:10:31.491 INFO 21062 --- [ocp-schedules-19,9173f2dba110e0c7,f60363f1f836344b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:10:31.491 INFO 21062 --- [ocp-schedules-19,9173f2dba110e0c7,f60363f1f836344b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:10:31.491 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:10:31.491 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:10:31.497 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:31.498 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:31.501 INFO 21062 --- [ocp-schedules-19,9173f2dba110e0c7,f60363f1f836344b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:10:31.502 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:10:31.502 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:10:31.503 WARN 21062 --- [ocp-schedules-19,9173f2dba110e0c7,f60363f1f836344b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:10:31.503 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:10:31.503 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:10:31.503 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:10:31.503 INFO 21062 --- [ocp-schedules-19,9173f2dba110e0c7,f60363f1f836344b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:10:31.504 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:10:31.504 INFO 21062 --- [ocp-schedules-24,9f0948ea2d08bca1,9d184dfdbdc8f564] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:10:31.505 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:10:31.505 WARN 21062 --- [ocp-schedules-19,9173f2dba110e0c7,f60363f1f836344b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:10:31.505 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:10:31.508 INFO 21062 --- [ocp-schedules-24,9f0948ea2d08bca1,9d184dfdbdc8f564] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:31.520 ERROR 21062 --- [ocp-schedules-19,9173f2dba110e0c7,f60363f1f836344b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1706) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1587) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2024-11-04 14:10:31.545 INFO 21062 --- [ocp-schedules-4,f71649388b6dc722,a7554b05e9a76deb] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:10:31.545+08:00, clusters=[1] 2024-11-04 14:10:31.549 INFO 21062 --- [ocp-schedules-4,f71649388b6dc722,a7554b05e9a76deb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:10:31.572 ERROR 21062 --- [ocp-schedules-24,9f0948ea2d08bca1,9d184dfdbdc8f564] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:10:31.573 INFO 21062 --- [ocp-schedules-24,9f0948ea2d08bca1,9d184dfdbdc8f564] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:10:32.983 INFO 21062 --- [prometheus-minute-schedule-1,c33a0c9445904a81,17238d253e916202] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:33.107 WARN 21062 --- [ocp-async-5,edbc07ba81f04274,12ab413181e6baae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:33.144 INFO 21062 --- [ocp-async-8,217e1078557d0310,922520f0686070c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:34.498 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:34.499 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:35.107 WARN 21062 --- [ocp-async-1,a056e07fc1e20a28,ad8d60c9457819bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:35.144 INFO 21062 --- [ocp-async-10,a1fda236168c318b,492da0362b1ab012] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:35.238 INFO 21062 --- [ocp-schedules-12,ceae2185f84651b2,77feb1358724a5f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:10:35.238 INFO 21062 --- [ocp-schedules-12,dfce808f7645a137,8430fad8b91f60a7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:10:35.242 INFO 21062 --- [ocp-schedules-12,dfce808f7645a137,8430fad8b91f60a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:35.269 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:10:35.269 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:10:35.983 INFO 21062 --- [prometheus-minute-schedule-1,fff990c6785903f2,7f2782ebdd1b7f56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:37.108 WARN 21062 --- [ocp-async-6,0a591b3c0cb7f362,cb1a53c1abf17f2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:37.145 INFO 21062 --- [ocp-async-3,1d6f5d8b7470c976,b0b4ec759f383425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:37.499 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:37.500 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:37.559 INFO 21062 --- [ocp-schedules-19,7c21edf2c058dda1,bc6833c8eec6586a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:10:37.563 INFO 21062 --- [ocp-schedules-19,7c21edf2c058dda1,bc6833c8eec6586a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:37.646 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:10:37.648 ERROR 21062 --- [ocp-schedules-19,7c21edf2c058dda1,bc6833c8eec6586a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:10:37.648 INFO 21062 --- [ocp-schedules-19,7c21edf2c058dda1,bc6833c8eec6586a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:10:38.504 INFO 21062 --- [ocp-schedules-9,7f42139a61fab2e7,60bafd0f9e0c2773] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:10:38.504 INFO 21062 --- [ocp-schedules-17,20c1a1adc5424b9d,af0f4e440884f88d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:10:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:10:38.590897+08:00, lastProcessTime=2024-11-04T14:10:28.590137+08:00 2024-11-04 14:10:38.645 INFO 21062 --- [ocp-async-10,5de7f117de836830,30bb2cd209d7fd6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:10:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:10:38.590897+08:00 2024-11-04 14:10:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0d35746e63111d6d,d003fe28793aa17a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:39.108 WARN 21062 --- [ocp-async-2,8b50fd0e5ec71c54,2dcd3d81e885bc3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:39.145 INFO 21062 --- [ocp-async-1,58bd7102afb261dd,619dce7e69d4e914] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:10:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:10:40.252 INFO 21062 --- [ocp-schedules-13,0511b6422b57331e,2177a92ca69769d0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:10:40.413 INFO 21062 --- [ocp-schedules-10,66bdeda28f7262e4,f81cb33be719f2d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:10:40.500 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:40.501 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:41.108 WARN 21062 --- [ocp-async-9,f6041b2d767d78c5,3bd54a8263ec8305] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:41.145 INFO 21062 --- [ocp-async-6,46fb3db6ecaeca51,d175e94e3ed67285] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:10:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:10:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:10:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:10:41.826 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:10:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=25ms, period=120000ms 2024-11-04 14:10:41.983 INFO 21062 --- [prometheus-minute-schedule-1,dd57d7e281c7cc8c,fa0492d973a1aeb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:43.108 WARN 21062 --- [ocp-async-4,a8445e48305ae1bb,c7c8fafb6d8cfd65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:43.145 INFO 21062 --- [ocp-async-7,f4d82b31ee359867,9b56dc7313aac935] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:43.501 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:43.502 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:44.983 INFO 21062 --- [prometheus-minute-schedule-1,ffaee17d555e9333,6d10571dcaccc3dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:45.108 WARN 21062 --- [ocp-async-5,ddf3a2e421dfbdb0,6bc66ce3e92ca147] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:45.145 INFO 21062 --- [ocp-async-8,45ed56a100787147,464201da556fdfd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:46.502 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:46.504 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:47.108 WARN 21062 --- [ocp-async-1,f838c17f97203d4f,d433313946219ae7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:47.145 INFO 21062 --- [ocp-async-10,12ceb20222771f09,d77e75efe71734bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8eb9b6126a337ffc,38227662834bf8a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:10:48.592832+08:00, lastProcessTime=2024-11-04T14:10:38.590897+08:00 2024-11-04 14:10:48.645 INFO 21062 --- [ocp-async-2,c3a8f95cb291639e,2e89abd0f0fba198] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:10:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:10:48.592832+08:00 2024-11-04 14:10:49.109 WARN 21062 --- [ocp-async-4,6574476a61b2b6c8,c4f1aba5abccd437] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:49.146 INFO 21062 --- [ocp-async-7,0321d540381bf7b6,b9cffdb37f6a5813] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:49.504 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:49.505 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:50.255 INFO 21062 --- [ocp-schedules-10,6ceb4578ce7d7b72,a0df893020ae5498] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:10:50.437 INFO 21062 --- [ocp-schedules-19,2b80fb0c880d939b,67b70e1e0b31adb2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:10:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b12f71b240e16499,88a50837d6cfeb46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:51.109 WARN 21062 --- [ocp-async-5,0d1019c8332c7b9a,d030101cc8998972] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:51.146 INFO 21062 --- [ocp-async-8,6c96078b1bd372ee,3efa0275f74665e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:51.176 INFO 21062 --- [ocp-schedules-29,c57dd0d9d2b5f518,bb22b32e28799b79] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:10:51.182 INFO 21062 --- [ocp-schedules-29,c57dd0d9d2b5f518,bb22b32e28799b79] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:10:51.182 INFO 21062 --- [ocp-schedules-29,c57dd0d9d2b5f518,bb22b32e28799b79] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:10:52.505 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:52.506 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:53.109 WARN 21062 --- [ocp-async-1,4cd79d3b87df133c,1e7d39203a75fd67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:53.146 INFO 21062 --- [ocp-async-10,6a292dae08847ee7,63a9ad13911bc81e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:53.983 INFO 21062 --- [prometheus-minute-schedule-1,5288bae34898884e,780cd39d97b128ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:55.109 WARN 21062 --- [ocp-async-10,967d15f30556648d,92460bd26bad2194] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:55.146 INFO 21062 --- [ocp-async-3,27498550482f82d5,9190a197cd5d7e17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:55.506 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:55.507 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d8ba981c37b1b251,5ff950298124479b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:10:57.109 WARN 21062 --- [ocp-async-3,f84c27965019fc1b,80d05d9922e361c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:57.146 INFO 21062 --- [ocp-async-2,f7c418a6ee638346,5ad84d2df02103bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:58.500 INFO 21062 --- [ocp-schedules-18,11d42c9d5c905a6f,7e359336e07601a5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:10:58.500 INFO 21062 --- [ocp-schedules-4,e05499b3061f3fda,1e69c35fafc145a7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:10:58.502 INFO 21062 --- [ocp-schedules-2,a11748c4d4649cfe,614695fff9b58462] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:10:58.503 INFO 21062 --- [ocp-schedules-18,11d42c9d5c905a6f,7e359336e07601a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:10:58.504 INFO 21062 --- [ocp-schedules-4,e05499b3061f3fda,1e69c35fafc145a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:10:58.504 INFO 21062 --- [ocp-schedules-7,96a2e6cf0e1e2603,563a42c238b1e5a3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:10:58.507 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:10:58.507 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:10:58.511 INFO 21062 --- [ocp-schedules-24,6028f075f505d947,f6227d969131f3cc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:10:58.511 INFO 21062 --- [ocp-schedules-25,1ff3c4cbb473c32e,b4b8cbd2c25ceb8b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:10:58.511 INFO 21062 --- [ocp-schedules-25,1ff3c4cbb473c32e,b4b8cbd2c25ceb8b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:10:58.511 INFO 21062 --- [ocp-schedules-24,6028f075f505d947,f6227d969131f3cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:10:58.512 INFO 21062 --- [ocp-schedules-29,845cc31c664c3849,285947183b2d7c4b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:10:58.512 INFO 21062 --- [ocp-schedules-29,845cc31c664c3849,285947183b2d7c4b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:10:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:10:58.589695+08:00, lastProcessTime=2024-11-04T14:10:48.592832+08:00 2024-11-04 14:10:58.645 INFO 21062 --- [ocp-async-4,b1880ecc417746b7,2bd330eae4497db7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:10:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:10:58.589695+08:00 2024-11-04 14:10:59.095 INFO 21062 --- [ocp-schedules-15,d549828e9da40465,988dcaab91311999] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:10:59.110 WARN 21062 --- [ocp-async-8,37fae2c6288da5ac,a63e9558591bd61d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:10:59.146 INFO 21062 --- [ocp-async-10,d063102baa8815c2,a8061e6f05b8abc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:10:59.983 INFO 21062 --- [prometheus-minute-schedule-1,fdd15d509899c561,51fbf0cafe2e6573] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700660, interval=60 2024-11-04 14:11:00.257 INFO 21062 --- [ocp-schedules-21,680e7a30127374c8,88d5d589ae580964] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:11:00.478 INFO 21062 --- [ocp-schedules-17,de5e23f852279815,884547f586038742] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:11:01.110 WARN 21062 --- [ocp-async-10,22aa542cb3a5a1ea,797cff32f6ccbe15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:01.147 INFO 21062 --- [ocp-async-3,77dd4480f251ad24,640fe212d2679f22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:01.507 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:01.508 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:02.983 INFO 21062 --- [prometheus-minute-schedule-1,9ee81e4b528a4e6c,59f55a97a3326cd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:03.110 WARN 21062 --- [ocp-async-3,dc55b6b33c1f0667,605fb4c2b4226008] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:03.147 INFO 21062 --- [ocp-async-2,8f12406345454f50,000aeb02f7be9d92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:04.508 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:04.509 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:05.110 WARN 21062 --- [ocp-async-2,264111541e17112d,b9a91b31df38cf0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:05.147 INFO 21062 --- [ocp-async-9,e3b13aeb0404c76d,5aaa4d148058aafe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:05.270 INFO 21062 --- [ocp-schedules-12,dfce808f7645a137,8430fad8b91f60a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:11:05.270 INFO 21062 --- [ocp-schedules-12,49ef4ba267b216b6,61632db2d0438def] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:11:05.273 INFO 21062 --- [ocp-schedules-12,49ef4ba267b216b6,61632db2d0438def] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:05.302 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:11:05.302 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:11:05.983 INFO 21062 --- [prometheus-minute-schedule-1,3c0c087b87ebf7ab,90cf3796b53af0e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:06.814 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,ac3bed942540a81f,782bf8c9e38f0c3e] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.130.47.17, traceId ac3bed942540a81f] 2024-11-04 14:11:06.826 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,ac3bed942540a81f,782bf8c9e38f0c3e] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.130.47.17, traceId=ac3bed942540a81f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2024-11-04 14:11:07.015 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,ac3bed942540a81f,782bf8c9e38f0c3e] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 2, current state: FAILED, new state: READY 2024-11-04 14:11:07.016 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,ac3bed942540a81f,782bf8c9e38f0c3e] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 35, current state: FAILED, new state: READY 2024-11-04 14:11:07.016 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,ac3bed942540a81f,782bf8c9e38f0c3e] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 21, current state: FAILED, new state: READY 2024-11-04 14:11:07.051 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,ac3bed942540a81f,782bf8c9e38f0c3e] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.130.47.17, serverIp=null, startTime=2024-11-04T14:11:06.839+08:00, endTime=2024-11-04T14:11:07.050+08:00, role=null, aliUid=null) 2024-11-04 14:11:07.066 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,ac3bed942540a81f,782bf8c9e38f0c3e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.130.47.17, traceId=ac3bed942540a81f, duration=261 ms] 2024-11-04 14:11:07.111 WARN 21062 --- [ocp-async-9,6723379b6afc953b,c5b2a4cb117045ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:07.147 INFO 21062 --- [ocp-async-4,b5667289d37ed477,05666dd54b2ed361] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:07.480 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,2f32b14aa9152f04,d2a71299d7109e8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=2f32b14aa9152f04, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:07.509 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:07.514 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:07.515 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,2f32b14aa9152f04,d2a71299d7109e8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=2f32b14aa9152f04, duration=42 ms] 2024-11-04 14:11:07.539 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,65868e65fb9748fe,8c8f8b57c736f1b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=65868e65fb9748fe, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 14:11:07.540 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,65868e65fb9748fe,8c8f8b57c736f1b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=65868e65fb9748fe, duration=7 ms] 2024-11-04 14:11:07.720 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,f0e9288a22758dc3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 2, operation:RETRY, state: RUNNING 2024-11-04 14:11:07.726 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,f0e9288a22758dc3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:RETRY, state: RUNNING 2024-11-04 14:11:07.733 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,f0e9288a22758dc3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 35, operation:RETRY, state: RUNNING 2024-11-04 14:11:07.741 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=2, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:07.717+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=2}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:07.747 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2024-11-04 14:11:07.747 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=21, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:07.724+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=21}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:07.752 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2024-11-04 14:11:07.755 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2024-11-04 14:11:07.749 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2024-11-04 14:11:07.752 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=35, context=Context{parallelIdx=1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:07.730+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=35}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:07.763 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2024-11-04 14:11:07.766 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2024-11-04 14:11:07.776 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.11, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null) 2024-11-04 14:11:07.777 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 14:11:07.780 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - BY_APPLICATION 2024-11-04 14:11:07.778 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 14:11:07.788 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.12, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null) 2024-11-04 14:11:07.788 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - BY_APPLICATION 2024-11-04 14:11:07.789 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.24, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null) 2024-11-04 14:11:07.789 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - BY_APPLICATION 2024-11-04 14:11:07.786 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 14:11:07.792 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 14:11:07.793 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 14:11:07.799 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 14:11:07.803 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3 2024-11-04 14:11:07.805 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3 2024-11-04 14:11:07.810 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_7.4 2024-11-04 14:11:07.819 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.12:1046 success 2024-11-04 14:11:07.821 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.11:1046 success 2024-11-04 14:11:07.827 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.24:1046 success 2024-11-04 14:11:07.877 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.12 2024-11-04 14:11:07.879 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.24 2024-11-04 14:11:07.880 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.12, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:1 2024-11-04 14:11:07.883 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.24, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:2 2024-11-04 14:11:07.889 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.11 2024-11-04 14:11:07.892 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.11, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:3 2024-11-04 14:11:07.928 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:07.928 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:07.936 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:07.937 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:07.942 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:07.948 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:07.975 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:07.983 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:07.990 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.009 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.012 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.018 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.021 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.024 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.027 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.051 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.056 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.059 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.065 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.066 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.075 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.103 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.105 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:08.118 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.129 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.132 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.150 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.151 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.154 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.157 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.175 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.181 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.190 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.194 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.198 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.200 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.223 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.238 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:08.242 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.248 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.251 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.274 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.278 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.282 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.285 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.295 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.300 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.321 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.322 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.328 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.330 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.343 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.367 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.368 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:08.370 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.374 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.410 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.413 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.414 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.415 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.419 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.424 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.453 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.454 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.460 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.462 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.467 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.492 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.497 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.500 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:08.504 INFO 21062 --- [ocp-schedules-18,11d42c9d5c905a6f,7e359336e07601a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:11:08.505 INFO 21062 --- [ocp-schedules-4,e05499b3061f3fda,1e69c35fafc145a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:11:08.510 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.539 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.545 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.546 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.547 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.548 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.554 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.585 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:11:08.590405+08:00, lastProcessTime=2024-11-04T14:10:58.589695+08:00 2024-11-04 14:11:08.587 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.592 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.594 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.600 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.614 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.617 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.632 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:08.645 INFO 21062 --- [ocp-async-1,8fb9595278e8f0cf,ae7d5a18c0411dff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:11:08.646 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.655 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.660 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.676 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.679 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.679 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.684 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.707 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.715 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.721 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.726 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.732 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.733 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.737 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.745 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,7e2892d2d08a0aa8,6342a08eb9c79b53] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments?taskInstanceId=2&subtaskInstanceId=2, client=10.130.47.17, traceId=7e2892d2d08a0aa8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,2,] 2024-11-04 14:11:08.750 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,7e2892d2d08a0aa8,6342a08eb9c79b53] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments client=10.130.47.17, traceId=7e2892d2d08a0aa8, duration=12 ms] 2024-11-04 14:11:08.761 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:11:08.590405+08:00 2024-11-04 14:11:08.775 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.778 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.780 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.805 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.808 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.810 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.813 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.827 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.844 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.851 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.853 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.854 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.856 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.861 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.890 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:08.899 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:08.902 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:08.904 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.934 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.937 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.946 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.949 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.951 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:08.977 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.980 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.983 INFO 21062 --- [prometheus-minute-schedule-1,5d1e385d0e285706,358ce0675cfea312] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:08.982 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:08.986 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:08.987 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:08.993 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.023 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:09.029 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.030 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:09.032 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:09.064 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.067 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.075 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 14:11:09.076 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.080 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.106 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:09.111 WARN 21062 --- [ocp-async-3,2ebfdab76a056696,a1f14d81c75435d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:09.111 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.112 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.114 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.116 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:09.120 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:11:09.126 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:09.128 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.147 INFO 21062 --- [ocp-async-2,60ac490c6b4d8a19,cc8cb33b8f26c20e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:09.154 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 14:11:09.155 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:09.162 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.167 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24 2024-11-04 14:11:09.191 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.194 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.196 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:11:09.198 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:09.207 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:09.232 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.234 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:09.235 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.240 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.279 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:09.284 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.286 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:09.320 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.324 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.331 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:09.356 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.359 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.366 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:09.371 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.403 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:09.409 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.418 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:09.450 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.454 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.455 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:11:09.481 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.484 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.487 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:09.498 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:09.504 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.527 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:09.532 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.549 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b4a13be1162ded5c,8a639e216f7de06b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=b4a13be1162ded5c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:09.550 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:11:09.575 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 14:11:09.580 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b4a13be1162ded5c,8a639e216f7de06b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=b4a13be1162ded5c, duration=39 ms] 2024-11-04 14:11:09.584 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.586 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.589 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:09.611 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.614 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.616 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:11:09.619 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:09.621 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:11:09.630 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:09.636 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.655 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:09.661 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.682 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 14:11:09.707 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 14:11:09.720 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.723 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.726 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:11:09.728 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:09.731 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:11:09.745 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.748 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.750 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:11:09.753 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:11:09.766 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:09.772 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.791 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:09.796 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.818 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 14:11:09.843 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.12 2024-11-04 14:11:09.854 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.857 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.859 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:11:09.861 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:11:09.857 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.866 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.868 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2024-11-04 14:11:09.878 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,67f3c63cd94ae68b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:RETRY, state: SUCCESSFUL 2024-11-04 14:11:09.898 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:09.904 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.950 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.11 2024-11-04 14:11:09.965 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.968 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:09.970 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2024-11-04 14:11:09.979 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,fc31cad60ae173d2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 2, operation:RETRY, state: SUCCESSFUL 2024-11-04 14:11:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:11:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:11:10.261 INFO 21062 --- [ocp-schedules-27,49d10c1e10b8b249,6bcf46b3a6456edb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:11:10.276 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.280 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.282 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:11:10.284 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:11:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:11:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:11:10.316 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:11:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:11:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:11:10.323 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.362 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:10.405 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.408 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.432 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,adb65adac7ae53d6,150e07c5bc868abb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments?taskInstanceId=2&subtaskInstanceId=2, client=10.130.47.17, traceId=adb65adac7ae53d6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,2,] 2024-11-04 14:11:10.437 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,adb65adac7ae53d6,150e07c5bc868abb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments client=10.130.47.17, traceId=adb65adac7ae53d6, duration=12 ms] 2024-11-04 14:11:10.445 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:10.451 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.491 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:10.502 INFO 21062 --- [ocp-schedules-24,ab481431660f300a,d9dfa95a5609727d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:11:10.514 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:10.516 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:10.536 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.539 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.576 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:10.583 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.623 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:10.667 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.670 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.673 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:10.707 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:10.714 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:10.753 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24 2024-11-04 14:11:11.111 WARN 21062 --- [ocp-async-2,12c33797d4691aa2,0743c75551e5f644] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:11.148 INFO 21062 --- [ocp-async-9,e557376237d01fac,ec80b24a670dcd97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:11.185 INFO 21062 --- [ocp-schedules-1,33df0b52e3ceb7f0,b21feec0a2b36f28] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:11:11.190 INFO 21062 --- [ocp-schedules-1,33df0b52e3ceb7f0,b21feec0a2b36f28] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:11:11.190 INFO 21062 --- [ocp-schedules-1,33df0b52e3ceb7f0,b21feec0a2b36f28] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:11:11.497 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=12, affectRows=12 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=12, validCount=12 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=8, affectRows=8 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=30, affectRows=30 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=8, validCount=8 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=14, affectRows=14 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=30, validCount=30 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=14, validCount=14 2024-11-04 14:11:11.498 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=40, affectRows=40 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=39, affectRows=39 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=40, validCount=40 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=24, affectRows=24 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=72, affectRows=72 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=39, validCount=39 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=58, affectRows=58 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=43, affectRows=43 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700600, nodeCount=52, affectRows=52 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=24, validCount=24 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=43, validCount=43 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=70, validCount=70 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=58, validCount=58 2024-11-04 14:11:11.499 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700600, nodeCount=51, validCount=51 2024-11-04 14:11:11.529 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d17d87c819e0406c,ba9883d396faee86] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=d17d87c819e0406c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:11.556 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d17d87c819e0406c,ba9883d396faee86] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=d17d87c819e0406c, duration=35 ms] 2024-11-04 14:11:11.881 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:11.884 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:11.887 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:11:11.889 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:11:11.891 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:11:11.921 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:11.928 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:11.968 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.24 2024-11-04 14:11:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0d7503e9e42645dd,1cfa5499b2e11809] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:12.132 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,818d7c8023f3c17f,7b5beb236e273df9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments?taskInstanceId=2&subtaskInstanceId=2, client=10.130.47.17, traceId=818d7c8023f3c17f, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,2,] 2024-11-04 14:11:12.136 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,818d7c8023f3c17f,7b5beb236e273df9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/2/logSegments client=10.130.47.17, traceId=818d7c8023f3c17f, duration=11 ms] 2024-11-04 14:11:13.082 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:13.085 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:13.088 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:11:13.090 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:11:13.111 WARN 21062 --- [ocp-async-9,04401aa749a2f766,bc6ce4f47e856783] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:13.121 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:13.128 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:13.148 INFO 21062 --- [ocp-async-4,942803fbf8a0226e,975bc49ce976c0f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:13.168 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.24 2024-11-04 14:11:13.191 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:13.193 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:13.196 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2024-11-04 14:11:13.204 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,7c19094526c157eb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 35, operation:RETRY, state: SUCCESSFUL 2024-11-04 14:11:13.270 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e854402202ec8a74,2dbcf7ff29f8ae25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e854402202ec8a74, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:13.297 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e854402202ec8a74,2dbcf7ff29f8ae25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e854402202ec8a74, duration=34 ms] 2024-11-04 14:11:13.516 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:13.517 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:13.750 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[18] 2024-11-04 14:11:13.984 INFO 21062 --- [prometheus-cache-monitor1,666908a75e94d6fb,dfc4922677323f08] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1568,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":392,"secondWriteQueue":0,"minuteWriteCacheSize":389} 2024-11-04 14:11:13.984 INFO 21062 --- [prometheus-cache-monitor1,666908a75e94d6fb,dfc4922677323f08] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=51, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=51, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:11:14.761 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,9c093e484aba3a29] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 18, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:14.782 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,0caf2513ea99b8dd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=18, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:14.759+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=18}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:14.790 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,0caf2513ea99b8dd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 18, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:11:14.938 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:07:14.92, end=2024-11-04T14:09:14.936} 2024-11-04 14:11:14.948 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:11:14.948 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:11:14.950 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,44d5ea658009a65f,ca89206fcd6acfaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=44d5ea658009a65f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:14.983 INFO 21062 --- [prometheus-minute-schedule-1,3b455a9650f9a04e,8662a14dbb0d5298] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:14.983 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,44d5ea658009a65f,ca89206fcd6acfaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=44d5ea658009a65f, duration=40 ms] 2024-11-04 14:11:15.111 WARN 21062 --- [ocp-async-4,56c02738910a084e,9b95d145ae872a35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:15.148 INFO 21062 --- [ocp-async-7,bd60baf0571c8540,fb497fd9b68c61ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:16.080 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[68, 54, 28] 2024-11-04 14:11:16.517 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:16.518 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:16.564 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d446a6f32fb0e991,b1d04aa46b310428] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=d446a6f32fb0e991, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:16.590 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d446a6f32fb0e991,b1d04aa46b310428] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=d446a6f32fb0e991, duration=33 ms] 2024-11-04 14:11:16.776 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,705cc112b6c6f9e7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 28, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:16.784 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,705cc112b6c6f9e7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 54, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:16.791 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,705cc112b6c6f9e7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 68, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:16.794 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,e5e0a627913760c9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=28, context=Context{parallelIdx=1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:16.774+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=28}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:16.800 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,cdfcb997bc60976a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=54, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:16.782+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=54}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:16.801 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,e5e0a627913760c9] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=ae6dc583-f590-407d-a0c8-0982411632f3, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=2, authType=OCP-HMACSHA1)) 2024-11-04 14:11:16.803 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,cdfcb997bc60976a] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=329b99ac-4483-476c-9268-cf63127a67d3, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1, authType=OCP-HMACSHA1)) 2024-11-04 14:11:16.808 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,05113bf40e9643c7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=68, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:16.789+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=68}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:16.811 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,05113bf40e9643c7] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=f3d76e6d-eafc-4e85-94cb-924e1d9e9e60, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=3, authType=OCP-HMACSHA1)) 2024-11-04 14:11:16.840 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,cdfcb997bc60976a] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=329b99ac-4483-476c-9268-cf63127a67d3, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1, authType=OCP-HMACSHA1)) 2024-11-04 14:11:16.840 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,e5e0a627913760c9] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=ae6dc583-f590-407d-a0c8-0982411632f3, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=2, authType=OCP-HMACSHA1)) 2024-11-04 14:11:16.842 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,05113bf40e9643c7] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=f3d76e6d-eafc-4e85-94cb-924e1d9e9e60, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=3, authType=OCP-HMACSHA1)) 2024-11-04 14:11:16.844 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,cdfcb997bc60976a] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:4 2024-11-04 14:11:16.846 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,e5e0a627913760c9] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:5 2024-11-04 14:11:16.848 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,05113bf40e9643c7] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:6 2024-11-04 14:11:16.854 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,cdfcb997bc60976a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 54, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:11:16.856 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,e5e0a627913760c9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 28, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:11:16.858 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,05113bf40e9643c7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 68, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:11:17.111 WARN 21062 --- [ocp-async-7,d0b151d2be613032,dc5b4487a3ee2d13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:17.148 INFO 21062 --- [ocp-async-8,a43ec5e2d592cb48,5348aaacae45a6d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:17.256 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[67] 2024-11-04 14:11:17.805 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,dbbf757c30330d31] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 67, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:17.824 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,d9a2072f49cbd0e8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=67, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:17.803+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=67}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:17.832 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,d9a2072f49cbd0e8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 67, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:11:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ab6241885ea523f7,7fc43148302d836e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:18.165 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,3944986802fa8df8,440317280aa9b404] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=3944986802fa8df8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:18.192 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,3944986802fa8df8,440317280aa9b404] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=3944986802fa8df8, duration=33 ms] 2024-11-04 14:11:18.420 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[51, 42, 13] 2024-11-04 14:11:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:11:18.591621+08:00, lastProcessTime=2024-11-04T14:11:08.590405+08:00 2024-11-04 14:11:18.645 INFO 21062 --- [ocp-async-3,33f31835c67d6142,a3dddf0bd9e3a55e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:11:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:11:18.591621+08:00 2024-11-04 14:11:18.818 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,ab7e7b81f492e20d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:18.826 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,ab7e7b81f492e20d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:18.833 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,ab7e7b81f492e20d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 51, operation:EXECUTE, state: RUNNING 2024-11-04 14:11:18.835 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=13, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:18.816+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=13}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:18.838 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:11:18.841 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:11:18.842 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=42, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:18.823+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=42}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:18.846 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:11:18.846 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:11:18.849 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:11:18.850 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=51, context=Context{parallelIdx=1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:11:18.831+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=51}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:11:18.852 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:11:18.854 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:11:18.856 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:11:18.860 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:11:18.862 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:11:18.864 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:11:18.867 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:11:18.870 WARN 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:11:18.872 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:11:18.876 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:11:18.876 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:11:18.878 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:11:18.880 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:11:18.886 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:11:18.889 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:11:18.894 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:11:18.899 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:18.903 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:18.905 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:18.909 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:18.911 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:18.917 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:18.950 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:18.955 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:18.958 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:18.956 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:18.961 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:18.965 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.002 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:11:19.005 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.24 2024-11-04 14:11:19.007 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:11:19.027 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.030 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.031 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.035 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.070 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:19.075 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.077 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.079 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:19.081 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.085 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.111 WARN 21062 --- [ocp-async-9,c2fb2df88589b7cb,525ec65359ce441c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:19.117 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:19.118 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:11:19.124 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.127 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:11:19.144 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.148 INFO 21062 --- [ocp-async-6,103b3501f933123c,26d5485d52c5dbb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:19.147 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.150 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.153 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.164 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.24 2024-11-04 14:11:19.207 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:11:19.207 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:11:19.227 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:19.227 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:19.232 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.235 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.242 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.245 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.263 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:11:19.275 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12 2024-11-04 14:11:19.282 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11 2024-11-04 14:11:19.283 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:19.291 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.299 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,495dabe17d958b86,3110e4be479d17a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.12, traceId=495dabe17d958b86, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:11:19.303 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,87f6be08c1551841,9d9d4d3858bfe51d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.11, traceId=87f6be08c1551841, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:11:19.309 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,87f6be08c1551841,9d9d4d3858bfe51d] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:11:19.309 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,495dabe17d958b86,3110e4be479d17a0] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:11:19.311 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,87f6be08c1551841,9d9d4d3858bfe51d] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:11:19.311 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,495dabe17d958b86,3110e4be479d17a0] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:11:19.328 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,495dabe17d958b86,3110e4be479d17a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.12, traceId=495dabe17d958b86, duration=34 ms] 2024-11-04 14:11:19.328 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,87f6be08c1551841,9d9d4d3858bfe51d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.11, traceId=87f6be08c1551841, duration=26 ms] 2024-11-04 14:11:19.330 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.24 2024-11-04 14:11:19.358 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,fe19d2971f61a979,e15bde5f4c71edeb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.24, traceId=fe19d2971f61a979, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:11:19.362 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,fe19d2971f61a979,e15bde5f4c71edeb] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:11:19.362 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,fe19d2971f61a979,e15bde5f4c71edeb] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:11:19.362 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,fe19d2971f61a979,e15bde5f4c71edeb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.24, traceId=fe19d2971f61a979, duration=6 ms] 2024-11-04 14:11:19.425 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.429 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.433 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.436 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.466 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:19.466 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.471 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.472 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.475 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:19.481 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.506 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:19.513 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.518 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:19.519 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:19.518 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11 2024-11-04 14:11:19.523 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12 2024-11-04 14:11:19.553 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.24 2024-11-04 14:11:19.671 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.674 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.676 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.677 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:11:19.679 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.680 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.684 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:11:19.687 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.691 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:11:19.710 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:19.717 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.718 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:19.723 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:19.726 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.728 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.758 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d2d04f9bf15745e7 on 10.46.128.24 2024-11-04 14:11:19.766 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d2d04f9bf15745e7 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=mkdir -p /tmp/d2d04f9bf15745e7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.769 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d2d04f9bf15745e7 on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=mkdir -p /tmp/d2d04f9bf15745e7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.770 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/a73a415925994a44 on 10.46.128.11 2024-11-04 14:11:19.775 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/27fd0d0805fb44e5 on 10.46.128.12 2024-11-04 14:11:19.777 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/a73a415925994a44 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a73a415925994a44, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.782 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,3d8e6b0d4fb263e5,f82a8676fd010768] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=3d8e6b0d4fb263e5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:19.780 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/a73a415925994a44 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a73a415925994a44, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.781 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/27fd0d0805fb44e5 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/27fd0d0805fb44e5, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.784 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/27fd0d0805fb44e5 on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/27fd0d0805fb44e5, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.807 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,3d8e6b0d4fb263e5,f82a8676fd010768] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=3d8e6b0d4fb263e5, duration=32 ms] 2024-11-04 14:11:19.805 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:19.812 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.822 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:19.823 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:19.827 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.829 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.852 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d2d04f9bf15745e7/cpio && chmod +x /tmp/d2d04f9bf15745e7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d2d04f9bf15745e7/cpio on 10.46.128.24 2024-11-04 14:11:19.861 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d2d04f9bf15745e7/cpio && chmod +x /tmp/d2d04f9bf15745e7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d2d04f9bf15745e7/cpio on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=touch /tmp/d2d04f9bf15745e7/cpio && chmod +x /tmp/d2d04f9bf15745e7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d2d04f9bf15745e7/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.863 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d2d04f9bf15745e7/cpio && chmod +x /tmp/d2d04f9bf15745e7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d2d04f9bf15745e7/cpio on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=touch /tmp/d2d04f9bf15745e7/cpio && chmod +x /tmp/d2d04f9bf15745e7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d2d04f9bf15745e7/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.870 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/a73a415925994a44/cpio && chmod +x /tmp/a73a415925994a44/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a73a415925994a44/cpio on 10.46.128.11 2024-11-04 14:11:19.871 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/27fd0d0805fb44e5/cpio && chmod +x /tmp/27fd0d0805fb44e5/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/27fd0d0805fb44e5/cpio on 10.46.128.12 2024-11-04 14:11:19.877 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/a73a415925994a44/cpio && chmod +x /tmp/a73a415925994a44/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a73a415925994a44/cpio on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a73a415925994a44/cpio && chmod +x /tmp/a73a415925994a44/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a73a415925994a44/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.879 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/27fd0d0805fb44e5/cpio && chmod +x /tmp/27fd0d0805fb44e5/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/27fd0d0805fb44e5/cpio on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/27fd0d0805fb44e5/cpio && chmod +x /tmp/27fd0d0805fb44e5/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/27fd0d0805fb44e5/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.880 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/a73a415925994a44/cpio && chmod +x /tmp/a73a415925994a44/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a73a415925994a44/cpio on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a73a415925994a44/cpio && chmod +x /tmp/a73a415925994a44/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a73a415925994a44/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.882 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/27fd0d0805fb44e5/cpio && chmod +x /tmp/27fd0d0805fb44e5/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/27fd0d0805fb44e5/cpio on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/27fd0d0805fb44e5/cpio && chmod +x /tmp/27fd0d0805fb44e5/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/27fd0d0805fb44e5/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.901 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:19.907 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.918 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:19.923 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:19.923 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.928 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.947 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d2d04f9bf15745e7 on 10.46.128.24 2024-11-04 14:11:19.954 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d2d04f9bf15745e7 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=mkdir -p /tmp/d2d04f9bf15745e7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.957 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d2d04f9bf15745e7 on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=mkdir -p /tmp/d2d04f9bf15745e7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.970 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/a73a415925994a44 on 10.46.128.11 2024-11-04 14:11:19.975 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/27fd0d0805fb44e5 on 10.46.128.12 2024-11-04 14:11:19.976 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/a73a415925994a44 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a73a415925994a44, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.980 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/a73a415925994a44 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a73a415925994a44, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.981 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/27fd0d0805fb44e5 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/27fd0d0805fb44e5, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.986 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/27fd0d0805fb44e5 on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/27fd0d0805fb44e5, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:19.994 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:20.000 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.018 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:20.023 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:20.023 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.028 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.039 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d2d04f9bf15745e7/install_package.sh && chmod +x /tmp/d2d04f9bf15745e7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d2d04f9bf15745e7/install_package.sh on 10.46.128.24 2024-11-04 14:11:20.048 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d2d04f9bf15745e7/install_package.sh && chmod +x /tmp/d2d04f9bf15745e7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d2d04f9bf15745e7/install_package.sh on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=touch /tmp/d2d04f9bf15745e7/install_package.sh && chmod +x /tmp/d2d04f9bf15745e7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d2d04f9bf15745e7/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.050 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d2d04f9bf15745e7/install_package.sh && chmod +x /tmp/d2d04f9bf15745e7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d2d04f9bf15745e7/install_package.sh on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=touch /tmp/d2d04f9bf15745e7/install_package.sh && chmod +x /tmp/d2d04f9bf15745e7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d2d04f9bf15745e7/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.066 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/a73a415925994a44/install_package.sh && chmod +x /tmp/a73a415925994a44/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a73a415925994a44/install_package.sh on 10.46.128.11 2024-11-04 14:11:20.073 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/a73a415925994a44/install_package.sh && chmod +x /tmp/a73a415925994a44/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a73a415925994a44/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a73a415925994a44/install_package.sh && chmod +x /tmp/a73a415925994a44/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a73a415925994a44/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.075 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/27fd0d0805fb44e5/install_package.sh && chmod +x /tmp/27fd0d0805fb44e5/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/27fd0d0805fb44e5/install_package.sh on 10.46.128.12 2024-11-04 14:11:20.076 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/a73a415925994a44/install_package.sh && chmod +x /tmp/a73a415925994a44/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a73a415925994a44/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a73a415925994a44/install_package.sh && chmod +x /tmp/a73a415925994a44/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a73a415925994a44/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.083 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/27fd0d0805fb44e5/install_package.sh && chmod +x /tmp/27fd0d0805fb44e5/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/27fd0d0805fb44e5/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/27fd0d0805fb44e5/install_package.sh && chmod +x /tmp/27fd0d0805fb44e5/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/27fd0d0805fb44e5/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.085 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/27fd0d0805fb44e5/install_package.sh && chmod +x /tmp/27fd0d0805fb44e5/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/27fd0d0805fb44e5/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/27fd0d0805fb44e5/install_package.sh && chmod +x /tmp/27fd0d0805fb44e5/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/27fd0d0805fb44e5/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.087 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:20.096 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.118 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:11:20.123 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:11:20.124 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.128 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:20.135 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/d2d04f9bf15745e7/install_package.sh on 10.46.128.24 2024-11-04 14:11:20.166 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/a73a415925994a44/install_package.sh on 10.46.128.11 2024-11-04 14:11:20.175 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/27fd0d0805fb44e5/install_package.sh on 10.46.128.12 2024-11-04 14:11:20.201 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/27fd0d0805fb44e5/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/27fd0d0805fb44e5/install_package.sh, out=, err=/tmp/27fd0d0805fb44e5/install_package.sh: line 35: rpm2cpio: command not found cpio: premature end of archive , extOut=null, exitStatus=2) 2024-11-04 14:11:20.203 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/a73a415925994a44/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/a73a415925994a44/install_package.sh, out=, err=/tmp/a73a415925994a44/install_package.sh: line 35: rpm2cpio: command not found cpio: premature end of archive , extOut=null, exitStatus=2) 2024-11-04 14:11:20.204 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/27fd0d0805fb44e5/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/27fd0d0805fb44e5/install_package.sh, out=, err=/tmp/27fd0d0805fb44e5/install_package.sh: line 35: rpm2cpio: command not found cpio: premature end of archive , extOut=null, exitStatus=2) 2024-11-04 14:11:20.207 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/a73a415925994a44/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/a73a415925994a44/install_package.sh, out=, err=/tmp/a73a415925994a44/install_package.sh: line 35: rpm2cpio: command not found cpio: premature end of archive , extOut=null, exitStatus=2) 2024-11-04 14:11:20.210 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/27fd0d0805fb44e5/install_package.sh, return code:2, output:, err:/tmp/27fd0d0805fb44e5/install_package.sh: line 35: rpm2cpio: command not found cpio: premature end of archive 2024-11-04 14:11:20.212 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/a73a415925994a44/install_package.sh, return code:2, output:, err:/tmp/a73a415925994a44/install_package.sh: line 35: rpm2cpio: command not found cpio: premature end of archive 2024-11-04 14:11:20.215 ERROR 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/27fd0d0805fb44e5/install_package.sh, exit code: 2, cause:null 2024-11-04 14:11:20.217 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/a73a415925994a44/install_package.sh, exit code: 2, cause:null 2024-11-04 14:11:20.220 ERROR 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/27fd0d0805fb44e5/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/27fd0d0805fb44e5/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:11:20.223 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/a73a415925994a44/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/a73a415925994a44/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:11:20.232 INFO 21062 --- [manual-subtask-executor15,e46f513af4706458,a6165ae28ba7d01d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:EXECUTE, state: FAILED 2024-11-04 14:11:20.236 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b463e4647cf4042e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:EXECUTE, state: FAILED 2024-11-04 14:11:20.264 INFO 21062 --- [ocp-schedules-16,44dc9ec1d13b0a96,e16f698a2b7a3d73] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:11:20.553 INFO 21062 --- [ocp-schedules-20,42429b3e42d3d817,cb37071bff3fbdba] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:11:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b2b363545deea17b,163e45c3f6e75fba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:21.112 WARN 21062 --- [ocp-async-6,76548735ec28d349,708395e375d1f96e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:21.148 INFO 21062 --- [ocp-async-3,108fbe190ed2d6c4,0565ce5a289b9101] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:21.428 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,cd3ef73c08c4e3e3,9e492fefef18dc22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=cd3ef73c08c4e3e3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:21.453 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,cd3ef73c08c4e3e3,9e492fefef18dc22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=cd3ef73c08c4e3e3, duration=31 ms] 2024-11-04 14:11:22.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:22.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:23.029 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a52cc3d3e0aea8e4,2f2be6c9c52ad427] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=a52cc3d3e0aea8e4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:23.054 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,a52cc3d3e0aea8e4,2f2be6c9c52ad427] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=a52cc3d3e0aea8e4, duration=32 ms] 2024-11-04 14:11:23.112 WARN 21062 --- [ocp-async-3,06e5bf56b3e8706b,e90dd8db1fd1bbf5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:23.149 INFO 21062 --- [ocp-async-1,d5dac17884d83226,dae0b6daec7f7350] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:23.944 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,265b4e368fcfc7d1,c124a5c76d214a1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=265b4e368fcfc7d1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:11:23.950 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,265b4e368fcfc7d1,c124a5c76d214a1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=265b4e368fcfc7d1, duration=13 ms] 2024-11-04 14:11:23.983 INFO 21062 --- [prometheus-minute-schedule-1,149f160bdb93dd4f,d0a481f3c7b8ce54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:24.799 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,fb6896de192315e3,93d252ad9ba8f20b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=fb6896de192315e3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:24.827 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,fb6896de192315e3,93d252ad9ba8f20b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=fb6896de192315e3, duration=35 ms] 2024-11-04 14:11:25.112 WARN 21062 --- [ocp-async-8,7b76a602e5b0bec8,f604d0f05013f37c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:25.149 INFO 21062 --- [ocp-async-10,4660964f6e06161d,f48cd88d5a2079d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:25.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:25.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:26.362 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,5157804fbbcac845,6ad95a01442c7f29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=5157804fbbcac845, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:26.391 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,5157804fbbcac845,6ad95a01442c7f29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=5157804fbbcac845, duration=37 ms] 2024-11-04 14:11:26.931 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/d2d04f9bf15745e7/install_package.sh on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=/tmp/d2d04f9bf15745e7/install_package.sh, out=Preparing... ######################################## Updating / installing... ocp-agent-ce-4.3.2-20241012145836.el7 ########################################, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:26.935 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/d2d04f9bf15745e7/install_package.sh on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=/tmp/d2d04f9bf15745e7/install_package.sh, out=Preparing... ######################################## Updating / installing... ocp-agent-ce-4.3.2-20241012145836.el7 ########################################, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:26.971 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:26.979 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:26.983 INFO 21062 --- [prometheus-minute-schedule-1,21fbe551e0768f35,5f82e5c40342b4e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:26.984 INFO 21062 --- [prometheus-hour-schedule-1,aa039abd392397eb,dd131ade3bd18c44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:11:27.018 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: rm -rf /tmp/d2d04f9bf15745e7 on 10.46.128.24 2024-11-04 14:11:27.026 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: rm -rf /tmp/d2d04f9bf15745e7 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=rm -rf /tmp/d2d04f9bf15745e7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.028 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: rm -rf /tmp/d2d04f9bf15745e7 on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=rm -rf /tmp/d2d04f9bf15745e7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.031 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.ocp.executor.executor.SshExecutor : install complete, rpmName=/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, result=Preparing... ######################################## Updating / installing... ocp-agent-ce-4.3.2-20241012145836.el7 ######################################## 2024-11-04 14:11:27.065 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:27.072 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.112 WARN 21062 --- [ocp-async-4,7d037881a73417ea,60f98b7a45e8d82e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:27.111 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'' on 10.46.128.24 2024-11-04 14:11:27.149 INFO 21062 --- [ocp-async-5,4a92faee62739100,564678f36985c37c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:27.185 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.188 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.191 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent success 2024-11-04 14:11:27.225 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:27.232 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.272 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx on 10.46.128.24 2024-11-04 14:11:27.449 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx, out={"successful":true,"message":"success","error":""}, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.452 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx, out={"successful":true,"message":"success","error":""}, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.457 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.c.AgentInstallationTaskService : start ocp-agent... 2024-11-04 14:11:27.489 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:27.496 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:27.536 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start' on 10.46.128.24 2024-11-04 14:11:27.955 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,869033fe57512068,e54853809723ec39] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=869033fe57512068, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:27.989 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,869033fe57512068,e54853809723ec39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=869033fe57512068, duration=41 ms] 2024-11-04 14:11:28.498 INFO 21062 --- [ocp-schedules-25,fc65fcf74267d7e3,f483bd473807b317] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:11:28.499 INFO 21062 --- [ocp-schedules-20,521cc1ae039401e0,129aac123904474b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:11:28.500 INFO 21062 --- [ocp-schedules-27,be63f5c8f690b381,b64473ec32e68611] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:11:28.500 INFO 21062 --- [ocp-schedules-29,a0ae0abaf08d38a4,164309f01fbb1d44] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:11:28.502 INFO 21062 --- [ocp-schedules-25,fc65fcf74267d7e3,f483bd473807b317] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:11:28.502 INFO 21062 --- [ocp-schedules-15,c6849f5fe15adfad,20da326688e22c94] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:11:28.503 INFO 21062 --- [ocp-schedules-27,be63f5c8f690b381,b64473ec32e68611] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:11:28.504 INFO 21062 --- [ocp-schedules-18,343a660b1a663b01,b73f247b07d030e9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:11:28.504 INFO 21062 --- [ocp-schedules-29,a0ae0abaf08d38a4,164309f01fbb1d44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:28.505 INFO 21062 --- [ocp-schedules-20,521cc1ae039401e0,129aac123904474b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:11:28.509 INFO 21062 --- [ocp-schedules-28,936d1d61408ae50b,67aaf8037bce271a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:11:28.509 INFO 21062 --- [ocp-schedules-28,936d1d61408ae50b,67aaf8037bce271a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:11:28.509 INFO 21062 --- [ocp-schedules-23,462ad2a53b020d94,18667b0a4d09f5f9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:11:28.509 INFO 21062 --- [ocp-schedules-23,462ad2a53b020d94,18667b0a4d09f5f9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:11:28.510 INFO 21062 --- [ocp-schedules-14,e6d1e87d0ccdeb2c,1491e611552f1443] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:11:28.510 INFO 21062 --- [ocp-schedules-14,e6d1e87d0ccdeb2c,1491e611552f1443] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:11:28.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:28.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:11:28.589578+08:00, lastProcessTime=2024-11-04T14:11:18.591621+08:00 2024-11-04 14:11:28.646 INFO 21062 --- [ocp-async-4,1dc81d8a640618d2,3213a10516431a99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:11:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:11:28.589578+08:00 2024-11-04 14:11:28.799 INFO 21062 --- [ocp-schedules-24,3abc77c49a797a08,9de19a027782bc09] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:11:28.961 INFO 21062 --- [ocp-schedules-6,a404674c893a65bf,db0855f2986fbdea] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:11:28.965 INFO 21062 --- [ocp-schedules-6,a404674c893a65bf,db0855f2986fbdea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:28.965 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:11:29.038 INFO 21062 --- [ocp-schedules-11,83f0039295c2af27,9f7aad0d7226d989] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:11:29.038 INFO 21062 --- [ocp-schedules-11,83f0039295c2af27,9f7aad0d7226d989] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:11:29.097 INFO 21062 --- [ocp-schedules-15,87060b62584fd46b,0742b37f98bcc0a2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:11:29.112 WARN 21062 --- [ocp-async-1,0c9398370dd1bcdc,80fdfe084efc4ea4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:29.130 INFO 21062 --- [ocp-schedules-28,c6e7969da3111a10,944e26c19a207126] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:11:29.136 INFO 21062 --- [ocp-schedules-28,c6e7969da3111a10,944e26c19a207126] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:11:29.136 INFO 21062 --- [ocp-schedules-28,c6e7969da3111a10,944e26c19a207126] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:11:29.149 INFO 21062 --- [ocp-async-8,14d58f87b7a4b565,ce7dee14cf03f71e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:29.438 INFO 21062 --- [ocp-schedules-10,aadcb4f5f5bf495f,d489feea7181649e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:11:29.438 INFO 21062 --- [ocp-schedules-10,aadcb4f5f5bf495f,d489feea7181649e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:11:29.439 INFO 21062 --- [ocp-schedules-10,aadcb4f5f5bf495f,d489feea7181649e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:11:29.439 INFO 21062 --- [ocp-schedules-10,aadcb4f5f5bf495f,d489feea7181649e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:11:29.440 INFO 21062 --- [ocp-schedules-10,aadcb4f5f5bf495f,d489feea7181649e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:11:29.440 INFO 21062 --- [ocp-schedules-10,aadcb4f5f5bf495f,d489feea7181649e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709926704322465792 2024-11-04 14:11:29.441 INFO 21062 --- [ocp-schedules-10,aadcb4f5f5bf495f,d489feea7181649e] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=4, maxSyncedRecordId=709932806753767424 2024-11-04 14:11:29.619 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,860cfefc6c26436e,ac9ea3f63e0df32e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=860cfefc6c26436e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:29.646 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,860cfefc6c26436e,ac9ea3f63e0df32e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=860cfefc6c26436e, duration=37 ms] 2024-11-04 14:11:29.983 INFO 21062 --- [prometheus-minute-schedule-1,eb2cfaabfd6dde51,e8ccc88d4e21a71d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:30.266 INFO 21062 --- [ocp-schedules-20,766cf2bf7f962403,cc4f830e2e609369] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:11:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:11:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:11:30.580 INFO 21062 --- [ocp-schedules-14,e7f7b095ccc62674,1bdd8a09e1919b43] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:11:30.660 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start' on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start', out={"successful":true,"message":"ok","error":""}, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.664 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start' on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start', out={"successful":true,"message":"ok","error":""}, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.700 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:30.708 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.747 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:30.790 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.793 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.796 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.c.AgentInstallationTaskService : ocp_agentd running 2024-11-04 14:11:30.831 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:30.838 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.878 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:30.919 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.923 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:30.926 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.c.AgentInstallationTaskService : ocp_mgragent running 2024-11-04 14:11:30.959 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.24 2024-11-04 14:11:30.966 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:31.005 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24 2024-11-04 14:11:31.048 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24, result: SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:31.051 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.24,result:SshResult(host=10.46.128.24, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:11:31.053 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.c.AgentInstallationTaskService : ocp_monagent running 2024-11-04 14:11:31.086 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.connector.Connectors : init global executor service. 2024-11-04 14:11:31.112 WARN 21062 --- [ocp-async-10,2892c7cff1c9351a,c78950884b01c16e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:31.149 INFO 21062 --- [ocp-async-6,7f7c91d4b7d66b7d,a12733cfa8828af1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:31.193 INFO 21062 --- [ocp-schedules-16,afef2104b682c101,a28306d4002ca550] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:11:31.196 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,5fc63dfa71717c4f,9a87691e4f2ff59b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=5fc63dfa71717c4f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:31.198 INFO 21062 --- [ocp-schedules-16,afef2104b682c101,a28306d4002ca550] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:11:31.198 INFO 21062 --- [ocp-schedules-16,afef2104b682c101,a28306d4002ca550] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:11:31.222 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,5fc63dfa71717c4f,9a87691e4f2ff59b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=5fc63dfa71717c4f, duration=35 ms] 2024-11-04 14:11:31.225 INFO 21062 --- [ocp-schedules-24,32ff2264394cfa79,df6dcdc848294b6f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:11:31.225 INFO 21062 --- [ocp-schedules-24,32ff2264394cfa79,df6dcdc848294b6f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:11:31.249 INFO 21062 --- [ocp-schedules-24,32ff2264394cfa79,df6dcdc848294b6f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:11:31.249 INFO 21062 --- [ocp-schedules-24,32ff2264394cfa79,df6dcdc848294b6f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:11:31.251 INFO 21062 --- [ocp-schedules-7,df5be6ec96cb1c20,2a8fada8c58f8853] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:11:31.289 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=10.46.128.24, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:4 2024-11-04 14:11:31.293 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62888/api/v1/time, params:null 2024-11-04 14:11:31.426 INFO 21062 --- [ocp-schedules-15,b2749f5bd05721cd,1864d26acf0f7814] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:11:31.426+08:00, clusters=[1] 2024-11-04 14:11:31.428 INFO 21062 --- [ocp-schedules-15,b2749f5bd05721cd,1864d26acf0f7814] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:11:31.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:31.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:31.530 INFO 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:11:31.530 INFO 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:11:31.530 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:11:31.530 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:11:31.539 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:11:31.539 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:11:31.539 INFO 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:11:31.540 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:11:31.540 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:11:31.541 WARN 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:11:31.541 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:11:31.541 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:11:31.541 INFO 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:11:31.542 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:11:31.542 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:11:31.542 WARN 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:11:31.561 INFO 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=10.46.128.24, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:5 2024-11-04 14:11:31.561 INFO 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:11:31.561 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=10.46.128.12, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:6 2024-11-04 14:11:31.562 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:11:31.562 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=10.46.128.11, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:7 2024-11-04 14:11:31.562 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:11:31.573 INFO 21062 --- [ocp-schedules-2,1a10108472e60c87,6121afddc588fc4e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:11:31.573+08:00, clusters=[1] 2024-11-04 14:11:31.577 INFO 21062 --- [ocp-schedules-2,1a10108472e60c87,6121afddc588fc4e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:11:32.389 ERROR 21062 --- [ocp-schedules-25,f75f4b8364e0f14a,dc9c5bb750621716] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:11:32.386 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] completed 2024-11-04 14:11:32.396 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e2eed9c1e0389aab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 51, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:11:32.747 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,1ef6377f95549bf1,7ad02fffb695fae2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=1ef6377f95549bf1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:32.774 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,1ef6377f95549bf1,7ad02fffb695fae2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=1ef6377f95549bf1, duration=37 ms] 2024-11-04 14:11:32.983 INFO 21062 --- [prometheus-minute-schedule-1,e3268c25b5659a8c,77b016d00936d360] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:33.113 WARN 21062 --- [ocp-async-5,e985b44a7d3d9116,dbc30bac614b5e71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:33.149 INFO 21062 --- [ocp-async-3,a18db0c2f27b5583,e0713df3f34db5f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:33.589 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=2024-11-04T12:33:00.221+08:00, subtasks=[SubtaskInstanceEntity{id=68, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=33, startTime=2024-11-04T14:11:16.789+08:00, endTime=2024-11-04T14:11:16.854+08:00}, SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busin 2024-11-04 14:11:33.599 WARN 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:11:33.599 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 14:11:33.600 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:2, status:FAILED 2024-11-04 14:11:34.328 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e7cca1daad7611f0,21954dd0b24cc6bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e7cca1daad7611f0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:11:34.354 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e7cca1daad7611f0,21954dd0b24cc6bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e7cca1daad7611f0, duration=34 ms] 2024-11-04 14:11:34.522 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:34.523 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:35.113 WARN 21062 --- [ocp-async-2,132ec9c889180337,91086cd3610850f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:35.150 INFO 21062 --- [ocp-async-9,1f40cb20273b63e0,7ae2f2d402f8dcfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:35.303 INFO 21062 --- [ocp-schedules-12,49ef4ba267b216b6,61632db2d0438def] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:11:35.303 INFO 21062 --- [ocp-schedules-12,0ba5e0cf8d7c35ba,2feddd5e6324b314] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:11:35.307 INFO 21062 --- [ocp-schedules-12,0ba5e0cf8d7c35ba,2feddd5e6324b314] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:35.335 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:11:35.335 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:11:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1e47e9f8b790ceef,743c8ca1e62b59cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:37.113 WARN 21062 --- [ocp-async-8,4766c8a35ed50321,b1d19eca0326dbd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:37.150 INFO 21062 --- [ocp-async-4,e8b592df4f861205,44fd10e8e5a7048b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:37.523 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:37.524 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:37.648 INFO 21062 --- [ocp-schedules-23,865a885d896f6ae4,a1cb8d2e3eb2e8e6] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:11:37.654 INFO 21062 --- [ocp-schedules-23,865a885d896f6ae4,a1cb8d2e3eb2e8e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:37.741 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:11:37.744 ERROR 21062 --- [ocp-schedules-23,865a885d896f6ae4,a1cb8d2e3eb2e8e6] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:11:37.744 INFO 21062 --- [ocp-schedules-23,865a885d896f6ae4,a1cb8d2e3eb2e8e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:11:38.504 INFO 21062 --- [ocp-schedules-27,be63f5c8f690b381,b64473ec32e68611] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:11:38.505 INFO 21062 --- [ocp-schedules-29,a0ae0abaf08d38a4,164309f01fbb1d44] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:11:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:11:38.589957+08:00, lastProcessTime=2024-11-04T14:11:28.589578+08:00 2024-11-04 14:11:38.646 INFO 21062 --- [ocp-async-9,d7e45db4253d776c,26cd8288d6fd8d63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:11:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:11:38.589957+08:00 2024-11-04 14:11:38.966 INFO 21062 --- [ocp-schedules-6,a404674c893a65bf,db0855f2986fbdea] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:11:38.983 INFO 21062 --- [prometheus-minute-schedule-1,26311dc95c89b412,b28387605f5099f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:39.113 WARN 21062 --- [ocp-async-7,6b7fa93b5489f3b2,4741c21ef34b45f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:39.150 INFO 21062 --- [ocp-async-2,8cf2153e8a6ca4d9,97072a7eca106c99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:11:40.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:11:40.269 INFO 21062 --- [ocp-schedules-5,66950c0c5007a334,5725291d7ef403f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:11:40.524 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:40.525 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:40.623 INFO 21062 --- [ocp-schedules-13,331df0c2af2cbdda,36e0ba5b3733f715] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:11:41.113 WARN 21062 --- [ocp-async-1,5339609784374265,619550a0316aceec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:41.150 INFO 21062 --- [ocp-async-8,025abf88f2bb528a,29f6334d67e2ea6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:41.983 INFO 21062 --- [prometheus-minute-schedule-1,1c6547669e42ef5b,4097d3eec0dc594b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:43.113 WARN 21062 --- [ocp-async-10,7f809f4a6cd4e7b9,d295526b31285c08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:43.150 INFO 21062 --- [ocp-async-6,da377f05dfdfb2b9,69207bf396b9872d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:43.525 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:43.526 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:44.983 INFO 21062 --- [prometheus-minute-schedule-1,da2feddca53cb597,f590270e9d55f389] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:45.114 WARN 21062 --- [ocp-async-5,ef2a5f6127e0e945,9196982b8ec238dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:45.150 INFO 21062 --- [ocp-async-3,c68f4c3ae208bccd,b85a8cc1e7f94d0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:46.526 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:46.527 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:47.114 WARN 21062 --- [ocp-async-2,f7c5bc2639683592,b1a448a01351a5fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:47.151 INFO 21062 --- [ocp-async-9,7600f8957b871b42,c11caa0e4e20c2ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e56ff220ea9856fa,c21b689c7b3770a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:11:48.591102+08:00, lastProcessTime=2024-11-04T14:11:38.589957+08:00 2024-11-04 14:11:48.646 INFO 21062 --- [ocp-async-7,2b7ab372e390adbf,92f160cb99a94e86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:11:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:11:48.591102+08:00 2024-11-04 14:11:49.114 WARN 21062 --- [ocp-async-8,151aacce1f20fc2e,8fd4cd9d962e164b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:49.151 INFO 21062 --- [ocp-async-6,b08f447253f3b369,3b13dd5c5335c0cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:49.527 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:49.528 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:50.272 INFO 21062 --- [ocp-schedules-11,f65ccf17d2eb08ae,c7b54bf334c5996c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:11:50.648 INFO 21062 --- [ocp-schedules-26,8c7cba0cb29dce34,8bf960f8151ee531] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:11:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d1b4d1ece228bd65,ea6d02418893c68b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:51.114 WARN 21062 --- [ocp-async-6,f71623da3b0fb2e2,6f38757997bf7c44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:51.151 INFO 21062 --- [ocp-async-3,bae7bfa40c9d8f6f,c997f70c443037e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:51.200 INFO 21062 --- [ocp-schedules-21,85a76dd0226e6ad5,856df4367f259595] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:11:51.206 INFO 21062 --- [ocp-schedules-21,85a76dd0226e6ad5,856df4367f259595] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:11:51.206 INFO 21062 --- [ocp-schedules-21,85a76dd0226e6ad5,856df4367f259595] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:11:52.528 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:52.529 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:53.114 WARN 21062 --- [ocp-async-3,227ca7bc742257e6,99ed7f0c5725ab34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:53.151 INFO 21062 --- [ocp-async-9,183204c88b4d6c36,ea814c03247caf60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:53.983 INFO 21062 --- [prometheus-minute-schedule-1,53863acc2ed532a6,7ef9394058d92323] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:55.114 WARN 21062 --- [ocp-async-9,676db6b6c1ee3fa6,1db235b046ff1908] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:55.151 INFO 21062 --- [ocp-async-4,634a357b4f3393d5,7a2c466b74ec16a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:55.529 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:55.530 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:56.983 INFO 21062 --- [prometheus-minute-schedule-1,02ceb54e598985c0,90377a966642f923] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:11:57.115 WARN 21062 --- [ocp-async-4,f30e9ee9870815d6,15f9b20b1b543f1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:57.151 INFO 21062 --- [ocp-async-5,fb579392f4723264,f35c1a728397c4cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:58.500 INFO 21062 --- [ocp-schedules-8,8c162347f9ff0acd,c305bfb7c6b6ab55] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:11:58.500 INFO 21062 --- [ocp-schedules-1,ae327e550a3a6185,19e27607963cb700] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:11:58.502 INFO 21062 --- [ocp-schedules-5,9f0de984aece2ae6,d33bc9b05df5a222] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:11:58.504 INFO 21062 --- [ocp-schedules-1,ae327e550a3a6185,19e27607963cb700] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:58.505 INFO 21062 --- [ocp-schedules-8,8c162347f9ff0acd,c305bfb7c6b6ab55] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:11:58.505 INFO 21062 --- [ocp-schedules-28,400a11ce0f6b8024,1bd71ceb722dd51e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:11:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:11:58.510 INFO 21062 --- [ocp-schedules-15,7df5192b4dcb6e56,73abca5f7cd71762] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:11:58.510 INFO 21062 --- [ocp-schedules-11,542615592ae65a57,07c6ecaf73c7dccf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:11:58.510 INFO 21062 --- [ocp-schedules-15,7df5192b4dcb6e56,73abca5f7cd71762] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:11:58.510 INFO 21062 --- [ocp-schedules-11,542615592ae65a57,07c6ecaf73c7dccf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:11:58.510 INFO 21062 --- [ocp-schedules-30,7e11b5ef2ab9146a,58b596f9e9079b1c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:11:58.510 INFO 21062 --- [ocp-schedules-30,7e11b5ef2ab9146a,58b596f9e9079b1c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:11:58.530 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:11:58.530 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:11:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:11:58.589295+08:00, lastProcessTime=2024-11-04T14:11:48.591102+08:00 2024-11-04 14:11:58.646 INFO 21062 --- [ocp-async-8,e48ee5067e4979ea,08434629e547eb0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:11:58.789 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:11:58.589295+08:00 2024-11-04 14:11:59.100 INFO 21062 --- [ocp-schedules-20,3f806a987b6bdee3,f39b6dc352a8fc9b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:11:59.115 WARN 21062 --- [ocp-async-3,bdece1c6093600a7,ab0c99670c16f367] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:11:59.152 INFO 21062 --- [ocp-async-9,ecf558ee12b0e86a,8896fffc90403ece] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:11:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a7ead2857bebe41e,abd088f7b8cd21ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700720, interval=60 2024-11-04 14:12:00.274 INFO 21062 --- [ocp-schedules-23,0a3f8d74ffe7137d,ca22056f0d9bfca8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:12:00.688 INFO 21062 --- [ocp-schedules-25,e628242bb4e6aa3f,f2d338ab83251636] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:12:01.115 WARN 21062 --- [ocp-async-9,dac4e274f26b7dee,e1fcee65c556886c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:01.152 INFO 21062 --- [ocp-async-4,e7db7a5560451316,381e445dc62bd0fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:01.530 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:01.531 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:02.983 INFO 21062 --- [prometheus-minute-schedule-1,03e12601d89dc673,4a7e9cad82ee7172] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:03.115 WARN 21062 --- [ocp-async-4,b8e609accde40643,9bb4f83d8a606441] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:03.152 INFO 21062 --- [ocp-async-5,44933f234ad85336,b945c108f7763446] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:04.531 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:04.532 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:05.115 WARN 21062 --- [ocp-async-5,2f9efa5abf9bbf07,6a8e1fd198e95692] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:05.152 INFO 21062 --- [ocp-async-2,e042d2ca57e14d90,803ebffdc0e75472] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:05.335 INFO 21062 --- [ocp-schedules-12,0ba5e0cf8d7c35ba,2feddd5e6324b314] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:12:05.335 INFO 21062 --- [ocp-schedules-12,0e045d5c8e7e3073,68a2c4ca77d3fc6e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:12:05.339 INFO 21062 --- [ocp-schedules-12,0e045d5c8e7e3073,68a2c4ca77d3fc6e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:05.369 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:12:05.369 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:12:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8cc28bbbbc8e68f6,fa0fa43031c0256c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:07.116 WARN 21062 --- [ocp-async-2,5007ada6ae5c5986,5e24f32deb555b95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:07.153 INFO 21062 --- [ocp-async-8,39215d21c02ee9ed,f05d9d08bd82e34a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:07.532 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:07.533 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:08.505 INFO 21062 --- [ocp-schedules-1,ae327e550a3a6185,19e27607963cb700] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:12:08.506 INFO 21062 --- [ocp-schedules-8,8c162347f9ff0acd,c305bfb7c6b6ab55] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:12:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:12:08.591401+08:00, lastProcessTime=2024-11-04T14:11:58.589295+08:00 2024-11-04 14:12:08.646 INFO 21062 --- [ocp-async-3,c82ddeca2c0219d5,29723f51ce240922] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:12:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:12:08.591401+08:00 2024-11-04 14:12:08.983 INFO 21062 --- [prometheus-minute-schedule-1,29658262d5624ed0,2d33aae83cd135aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:09.116 WARN 21062 --- [ocp-async-4,0600d3b50237a9a8,56da282e058cd590] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:09.153 INFO 21062 --- [ocp-async-5,7af3226449d9ffad,f73be16e361e1d50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:12:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:12:10.277 INFO 21062 --- [ocp-schedules-4,37cb6013a8e6a353,c18d1419e30d4466] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:12:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:12:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:12:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:12:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:12:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:12:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:12:10.533 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:10.534 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:10.713 INFO 21062 --- [ocp-schedules-3,292e0f08b696a5e0,dcdab20e21cbcd03] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:12:11.116 WARN 21062 --- [ocp-async-5,27adde564cb10bc7,7ce343c4750e79f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:11.153 INFO 21062 --- [ocp-async-3,d3796c1700a53063,d0571b3896e159ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:11.208 INFO 21062 --- [ocp-schedules-2,9837fc3119c59d61,c15fbd3417244530] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:12:11.214 INFO 21062 --- [ocp-schedules-2,9837fc3119c59d61,c15fbd3417244530] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:12:11.214 INFO 21062 --- [ocp-schedules-2,9837fc3119c59d61,c15fbd3417244530] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:12:11.501 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=3, affectRows=3 2024-11-04 14:12:11.501 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=3, validCount=3 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=26, affectRows=26 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=26, validCount=26 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=25, affectRows=25 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=14, affectRows=14 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=43, affectRows=43 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=14, validCount=14 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=25, validCount=25 2024-11-04 14:12:11.502 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=43, validCount=43 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=42, affectRows=42 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=47, affectRows=47 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=69, affectRows=69 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=52, affectRows=52 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700660, nodeCount=75, affectRows=75 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=42, validCount=42 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=47, validCount=47 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=69, validCount=69 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=52, validCount=52 2024-11-04 14:12:11.503 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700660, nodeCount=72, validCount=72 2024-11-04 14:12:11.983 INFO 21062 --- [prometheus-minute-schedule-1,70845ef7c473e760,599152296591ef83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:13.116 WARN 21062 --- [ocp-async-3,57d52c8b9294a3e6,ae13ce5cf58b4d58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:13.153 INFO 21062 --- [ocp-async-10,872d6cb9b8035ee1,ddab32082b1614ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:13.534 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:13.535 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:13.984 INFO 21062 --- [prometheus-cache-monitor1,bc8541acce1d542f,2b091007b44f4ae8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1572,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:12:13.985 INFO 21062 --- [prometheus-cache-monitor1,bc8541acce1d542f,2b091007b44f4ae8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:12:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5463f7be5f1a22a0,4e6c72323fd1e74f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:15.116 WARN 21062 --- [ocp-async-10,004eeb29f410ca3e,738085b7190f66ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:15.153 INFO 21062 --- [ocp-async-6,0b791e49e7ec586f,1da2c98aa1cb6b63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:16.535 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:16.536 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:17.116 WARN 21062 --- [ocp-async-6,c6c4fb0e52cc8c09,67263a11f9918001] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:17.154 INFO 21062 --- [ocp-async-1,30b727b3ae30dc44,53462db26aefc2c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:17.983 INFO 21062 --- [prometheus-minute-schedule-1,132566c73c6443a1,35fb26754830e8c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:12:18.590140+08:00, lastProcessTime=2024-11-04T14:12:08.591401+08:00 2024-11-04 14:12:18.646 INFO 21062 --- [ocp-async-4,8d60f58d2d94c12f,ede7ec047f5c4e6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:12:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:12:18.590140+08:00 2024-11-04 14:12:19.116 WARN 21062 --- [ocp-async-3,9d1cdc4618ec952b,2e447011b646a950] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:19.154 INFO 21062 --- [ocp-async-8,25f73cc374b45cd9,28e0c1e44f49ff96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:19.536 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:19.537 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:20.279 INFO 21062 --- [ocp-schedules-8,1503b1938b5044e8,c9c793f0b80734f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:12:20.755 INFO 21062 --- [ocp-schedules-14,3f4945881975236e,8afcc481cef00a21] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:12:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a6fc3bf2b9779dac,7e739a46dca7acdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:21.117 WARN 21062 --- [ocp-async-10,f0225411e693de0b,b7e8f6bfeb6e047c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:21.154 INFO 21062 --- [ocp-async-7,7acebe57b3569ae8,53159c3def71ba33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:22.537 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:22.538 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:23.117 WARN 21062 --- [ocp-async-6,281e3ffb3d89114f,905397422e42f02a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:23.154 INFO 21062 --- [ocp-async-2,abe687bb72422ce4,993bdc256f39e298] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:23.983 INFO 21062 --- [prometheus-minute-schedule-1,3ca343d456de8bac,bca8202f709c2f85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:25.117 WARN 21062 --- [ocp-async-1,00c2464e4ae60563,724bfd9ff238f265] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:25.154 INFO 21062 --- [ocp-async-9,feea6259f8f3a645,fc0eacead8c53d4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:25.538 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:25.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:26.983 INFO 21062 --- [prometheus-minute-schedule-1,c782c4b3840d4d3d,45bc389c0853e23c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2b03f43a8359fab3,eb4ec0963ca0f7f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:12:27.117 WARN 21062 --- [ocp-async-8,6944e854b23f497f,b28251c42ae24355] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:27.154 INFO 21062 --- [ocp-async-4,0f60ac807b0c2a6b,65f963e9947dbdf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:28.498 INFO 21062 --- [ocp-schedules-3,8431250b00ed4721,404704c28541d7f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:12:28.499 INFO 21062 --- [ocp-schedules-8,33173c1f4f2f6838,d00e372ca132fa7a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:12:28.500 INFO 21062 --- [ocp-schedules-19,ac0aaa170d28143e,c8b214db04c96f05] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:12:28.500 INFO 21062 --- [ocp-schedules-25,bb4b878865cd3ad9,8f61cda37c09239f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:12:28.502 INFO 21062 --- [ocp-schedules-3,8431250b00ed4721,404704c28541d7f0] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:12:28.502 INFO 21062 --- [ocp-schedules-6,df2252f8b7393d2b,1ef7f4321d746ac4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:12:28.505 INFO 21062 --- [ocp-schedules-25,bb4b878865cd3ad9,8f61cda37c09239f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:28.506 INFO 21062 --- [ocp-schedules-16,f3980012694aa40b,1ec2f5137539b5f8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:12:28.506 INFO 21062 --- [ocp-schedules-20,7649e5ded144a69d,986b8be7f7269700] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:12:28.506 INFO 21062 --- [ocp-schedules-19,ac0aaa170d28143e,c8b214db04c96f05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:28.506 INFO 21062 --- [ocp-schedules-8,33173c1f4f2f6838,d00e372ca132fa7a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:12:28.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:12:28.509 INFO 21062 --- [ocp-schedules-20,7649e5ded144a69d,986b8be7f7269700] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:28.512 INFO 21062 --- [ocp-schedules-11,67626796c5e7bfbb,115f31061e7e496a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:12:28.512 INFO 21062 --- [ocp-schedules-7,b92b0f393128099a,79424c4a30e62035] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:12:28.512 INFO 21062 --- [ocp-schedules-7,b92b0f393128099a,79424c4a30e62035] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:12:28.512 INFO 21062 --- [ocp-schedules-11,67626796c5e7bfbb,115f31061e7e496a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:12:28.512 INFO 21062 --- [ocp-schedules-13,7b8031d110195ab8,e3c56751f7801f8c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:12:28.512 INFO 21062 --- [ocp-schedules-13,7b8031d110195ab8,e3c56751f7801f8c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:12:28.536 INFO 21062 --- [ocp-schedules-2,7268b0ae01ec640d,5e07c7bd52c8dd9d] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:12:28.536 INFO 21062 --- [ocp-schedules-2,7268b0ae01ec640d,5e07c7bd52c8dd9d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:12:28.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:28.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:28.574 INFO 21062 --- [ocp-schedules-18,282c66b6a55c8ef4,ae3ebae68970ec61] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1572,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:12:28.574 INFO 21062 --- [ocp-schedules-18,282c66b6a55c8ef4,ae3ebae68970ec61] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:12:28.578 INFO 21062 --- [pool-12-thread-2,1e933920f8fe9ab5,57b4b7b2919db0f5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:12:28.579 INFO 21062 --- [pool-12-thread-3,1e933920f8fe9ab5,05add30f14562a58] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:12:28.579 INFO 21062 --- [pool-12-thread-2,1e933920f8fe9ab5,57b4b7b2919db0f5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:12:28.579 INFO 21062 --- [pool-12-thread-1,1e933920f8fe9ab5,7b8704129f95497e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:12:28.580 INFO 21062 --- [pool-12-thread-4,1e933920f8fe9ab5,f6094d69856c0269] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:12:28.589 INFO 21062 --- [pool-12-thread-2,1e933920f8fe9ab5,57b4b7b2919db0f5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:12:28.590 INFO 21062 --- [pool-12-thread-2,1e933920f8fe9ab5,57b4b7b2919db0f5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:12:28.593 INFO 21062 --- [ForkJoinPool-11-worker-14,1e933920f8fe9ab5,06840085bf834083] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:12:28.593 INFO 21062 --- [pool-12-thread-3,1e933920f8fe9ab5,05add30f14562a58] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:12:28.594 INFO 21062 --- [ForkJoinPool-11-worker-7,1e933920f8fe9ab5,c0d9b60b2837f38f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:12:28.594 INFO 21062 --- [ForkJoinPool-11-worker-7,1e933920f8fe9ab5,c0d9b60b2837f38f] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:12:28.594 INFO 21062 --- [ForkJoinPool-11-worker-7,1e933920f8fe9ab5,c0d9b60b2837f38f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:12:28.594 INFO 21062 --- [pool-12-thread-4,1e933920f8fe9ab5,f6094d69856c0269] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:12:28.594 INFO 21062 --- [pool-12-thread-1,1e933920f8fe9ab5,7b8704129f95497e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:12:28.594 INFO 21062 --- [pool-12-thread-1,1e933920f8fe9ab5,7b8704129f95497e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:12:28.594 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:12:28.593957+08:00, lastProcessTime=2024-11-04T14:12:18.590140+08:00 2024-11-04 14:12:28.622 INFO 21062 --- [ocp-schedules-27,991bb12bdb4b6378,eff1db80503f8168] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:12:28.623 INFO 21062 --- [ocp-schedules-27,991bb12bdb4b6378,eff1db80503f8168] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:12:28.647 INFO 21062 --- [ocp-async-8,f1d35f79f4e404c6,401a460ba219f960] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:12:28.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:12:28.593957+08:00 2024-11-04 14:12:28.797 INFO 21062 --- [ocp-schedules-6,dc9afbe353757d53,35d3cca54a08ad06] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 14:12:28.798 INFO 21062 --- [ocp-schedules-6,dc9afbe353757d53,35d3cca54a08ad06] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 14:12:28.798 INFO 21062 --- [ocp-schedules-6,dc9afbe353757d53,35d3cca54a08ad06] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=396 2024-11-04 14:12:28.801 INFO 21062 --- [ocp-schedules-11,4bbcbe4a39992ec4,1632cabe99fbcfcb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:12:28.819 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:12:28.823 INFO 21062 --- [ocp-schedules-6,dc9afbe353757d53,35d3cca54a08ad06] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=396 2024-11-04 14:12:28.837 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:12:28.839 WARN 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:12:28.840 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:12:28.841 WARN 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:12:28.849 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:12:28.850 WARN 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:12:28.851 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:12:28.852 WARN 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:12:28.858 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:12:28.859 WARN 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:12:28.860 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:12:28.861 WARN 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:12:28.866 INFO 21062 --- [ocp-schedules-7,916bd1c4f1753ed1,ecc0e973ace6adbd] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:12:29.017 INFO 21062 --- [ocp-schedules-21,ae46462fdf64919d,25fcf457d4dd7b87] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 14:12:29.021 INFO 21062 --- [ocp-schedules-21,ae46462fdf64919d,25fcf457d4dd7b87] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:29.043 INFO 21062 --- [ocp-schedules-3,c4db40e1b3ae284e,3a90d03bfe84b5f1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:12:29.043 INFO 21062 --- [ocp-schedules-3,c4db40e1b3ae284e,3a90d03bfe84b5f1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:12:29.060 ERROR 21062 --- [ocp-schedules-21,ae46462fdf64919d,25fcf457d4dd7b87] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:12:29.060 INFO 21062 --- [ocp-schedules-21,ae46462fdf64919d,25fcf457d4dd7b87] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 14:12:29.103 INFO 21062 --- [ocp-schedules-29,3d76cb74c0815ab6,188faf86e567ab08] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:12:29.117 WARN 21062 --- [ocp-async-2,2e99210bb86275f7,e19f9cbdeb71d71e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:29.136 INFO 21062 --- [ocp-schedules-10,d6e8fefe023433dd,03f82017b37c54a5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:12:29.140 INFO 21062 --- [ocp-schedules-10,d6e8fefe023433dd,03f82017b37c54a5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:12:29.140 INFO 21062 --- [ocp-schedules-10,d6e8fefe023433dd,03f82017b37c54a5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:12:29.154 INFO 21062 --- [ocp-async-1,ecc9c06819ddc93a,aeebb3ac3ba3befa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:29.442 INFO 21062 --- [ocp-schedules-30,b9438183ceefd337,523115228df14f8e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:12:29.442 INFO 21062 --- [ocp-schedules-30,b9438183ceefd337,523115228df14f8e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:12:29.442 INFO 21062 --- [ocp-schedules-30,b9438183ceefd337,523115228df14f8e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:12:29.442 INFO 21062 --- [ocp-schedules-30,b9438183ceefd337,523115228df14f8e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:12:29.443 INFO 21062 --- [ocp-schedules-30,b9438183ceefd337,523115228df14f8e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:12:29.443 INFO 21062 --- [ocp-schedules-30,b9438183ceefd337,523115228df14f8e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:12:29.632 INFO 21062 --- [ocp-schedules-4,e8742cb080acb333,f0278fc9faeabeae] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:12:29.632 INFO 21062 --- [ocp-schedules-4,e8742cb080acb333,f0278fc9faeabeae] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:12:29.662 INFO 21062 --- [pool-39-thread-1,e8742cb080acb333,c44107268c6a488c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:12:29.689 WARN 21062 --- [ocp-schedules-4,e8742cb080acb333,f0278fc9faeabeae] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:12:29.690 INFO 21062 --- [ocp-schedules-4,e8742cb080acb333,f0278fc9faeabeae] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:12:29.985 INFO 21062 --- [prometheus-minute-schedule-1,69fcd488b4eb1079,8ab9eae683004530] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:12:30.283 INFO 21062 --- [ocp-schedules-14,111f269ac2bc43c8,77d2b258e6780100] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:12:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:12:30.781 INFO 21062 --- [ocp-schedules-27,c5e557ede759bfba,a9062027a067e18f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:12:31.118 WARN 21062 --- [ocp-async-9,41cd8cd59e46183c,c4b0c19773f75527] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:31.155 INFO 21062 --- [ocp-async-10,1a4231056c7b54ea,cb653bc0781ed2fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:31.218 INFO 21062 --- [ocp-schedules-9,f442ecf415f59028,7e395fcc9fb31769] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:12:31.224 INFO 21062 --- [ocp-schedules-9,f442ecf415f59028,7e395fcc9fb31769] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:12:31.224 INFO 21062 --- [ocp-schedules-9,f442ecf415f59028,7e395fcc9fb31769] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:12:31.249 INFO 21062 --- [ocp-schedules-8,7084b19133115c77,e787263f745e8a97] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:12:31.249 INFO 21062 --- [ocp-schedules-8,7084b19133115c77,e787263f745e8a97] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:12:31.277 INFO 21062 --- [ocp-schedules-8,7084b19133115c77,e787263f745e8a97] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:12:31.277 INFO 21062 --- [ocp-schedules-8,7084b19133115c77,e787263f745e8a97] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:12:31.277 INFO 21062 --- [ocp-schedules-6,dab789f7342e5f11,fc42395165f040b6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:12:31.453 INFO 21062 --- [ocp-schedules-14,8b888a547564fe52,fea1aa208e498790] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:12:31.453+08:00, clusters=[1] 2024-11-04 14:12:31.455 INFO 21062 --- [ocp-schedules-14,8b888a547564fe52,fea1aa208e498790] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:12:31.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:31.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:31.573 INFO 21062 --- [ocp-schedules-23,33d43178931fa35b,082a7a1011ab8f57] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:12:31.576 INFO 21062 --- [ocp-schedules-23,33d43178931fa35b,082a7a1011ab8f57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:31.602 INFO 21062 --- [ocp-schedules-26,930cedad688d8742,fbeaf862a1fa5d3b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:12:31.602+08:00, clusters=[1] 2024-11-04 14:12:31.605 INFO 21062 --- [ocp-schedules-26,930cedad688d8742,fbeaf862a1fa5d3b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:12:31.637 ERROR 21062 --- [ocp-schedules-23,33d43178931fa35b,082a7a1011ab8f57] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:12:31.637 INFO 21062 --- [ocp-schedules-23,33d43178931fa35b,082a7a1011ab8f57] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:12:32.398 INFO 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:12:32.398 INFO 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:12:32.398 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:12:32.398 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:12:32.406 INFO 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:12:32.408 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:12:32.408 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:12:32.408 WARN 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:12:32.409 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:12:32.409 INFO 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:12:32.409 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:12:32.409 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:12:32.410 WARN 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:12:32.410 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:12:32.411 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:12:32.411 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:12:32.417 INFO 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:12:32.417 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:12:32.417 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:12:32.439 ERROR 21062 --- [ocp-schedules-7,447b4dbb250bfc76,ac2e1fd57492c9c8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:12:32.983 INFO 21062 --- [prometheus-minute-schedule-1,ac04da7df667e6f5,c385598e6e9153b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:33.118 WARN 21062 --- [ocp-async-4,1077112b498f04be,a3185d39d17de148] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:33.155 INFO 21062 --- [ocp-async-6,465a6cc7baa069d1,e4707a4bc24c4616] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:34.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:34.541 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:35.118 WARN 21062 --- [ocp-async-5,dd5a00fa287e94bc,7701865d3f254319] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:35.155 INFO 21062 --- [ocp-async-3,f7d4198af694081f,321ff3b6c05c5808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:35.229 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:12:35.229 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:12:35.237 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:12:35.237 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:12:35.237 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:12:35.237 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:12:35.242 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:12:35.242 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:12:35.242 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:12:35.243 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:12:35.280 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:12:35.288 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:12:35.288 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:12:35.288 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:12:35.288 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:12:35.288 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:12:35.296 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:12:35.369 INFO 21062 --- [ocp-schedules-12,0e045d5c8e7e3073,68a2c4ca77d3fc6e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:12:35.369 INFO 21062 --- [ocp-schedules-12,8d042238656f5ea0,a372f77ba17d95c9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:12:35.373 INFO 21062 --- [ocp-schedules-12,8d042238656f5ea0,a372f77ba17d95c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:35.401 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:12:35.401 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:12:35.827 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:12:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b2b200713ba07766,2e95bde5f69b3855] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:36.429 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.186997163S 2024-11-04 14:12:36.430 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:12:36.536 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:12:36.536 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:12:36.564 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:12:36.627 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:12:36.627 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:12:36.627 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:12:36.627 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:12:36.627 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:12:36.628 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:12:36.640 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:12:36.815 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] o.s.boot.SpringApplication : Started application in 1.578 seconds (JVM running for 6170.494) 2024-11-04 14:12:36.815 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.578755S 2024-11-04 14:12:37.118 WARN 21062 --- [ocp-async-1,cdce4cf9f2570e74,d87851a555a52c33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:37.155 INFO 21062 --- [ocp-async-8,2658ece0b7c72eaa,07598da5bd3a5539] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:37.196 INFO 21062 --- [ocp-schedules-3,bbef0a8e1c77611f,a8de2cf1c403fc74] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:12:37.541 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:37.542 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:37.744 INFO 21062 --- [ocp-schedules-4,9d358799766fd938,b671b39a1912b1db] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:12:37.749 INFO 21062 --- [ocp-schedules-4,9d358799766fd938,b671b39a1912b1db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:37.830 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:12:37.831 ERROR 21062 --- [ocp-schedules-4,9d358799766fd938,b671b39a1912b1db] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:12:37.831 INFO 21062 --- [ocp-schedules-4,9d358799766fd938,b671b39a1912b1db] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:12:38.505 INFO 21062 --- [ocp-schedules-25,bb4b878865cd3ad9,8f61cda37c09239f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:12:38.507 INFO 21062 --- [ocp-schedules-19,ac0aaa170d28143e,c8b214db04c96f05] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:12:38.509 INFO 21062 --- [ocp-schedules-20,7649e5ded144a69d,986b8be7f7269700] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:12:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:12:38.589728+08:00, lastProcessTime=2024-11-04T14:12:28.593957+08:00 2024-11-04 14:12:38.647 INFO 21062 --- [ocp-async-3,a30f180da2d119aa,735b18b98da9336f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:12:38.744 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:12:38.589728+08:00 2024-11-04 14:12:38.966 INFO 21062 --- [ocp-schedules-4,ab86c944a9abf9c3,b95968f66906e192] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:12:38.969 INFO 21062 --- [ocp-schedules-4,ab86c944a9abf9c3,b95968f66906e192] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:38.970 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:12:38.983 INFO 21062 --- [prometheus-minute-schedule-1,333c032067db4bec,8990ca0a4511c4e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:39.118 WARN 21062 --- [ocp-async-7,9ac479d35e778e9b,e00fa1750d9d91c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:39.155 INFO 21062 --- [ocp-async-5,ab792ab22ebdbfb8,9eed639e6b698a3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:12:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:12:40.286 INFO 21062 --- [ocp-schedules-7,026739ccc272817b,8e69366eab13dc01] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:12:40.542 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:40.543 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:40.822 INFO 21062 --- [ocp-schedules-16,f60f8f07337408e5,adb446c1d8eb9c36] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:12:41.119 WARN 21062 --- [ocp-async-2,edbbaed054827bb9,067d44a4ea95a8a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:41.156 INFO 21062 --- [ocp-async-1,339571c7b5dba6e1,9e246b13040cb9f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:12:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:12:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:12:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:12:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:12:41.858 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:12:41.858 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-11-04 14:12:41.983 INFO 21062 --- [prometheus-minute-schedule-1,3840e775e7335541,95005699dd8fb7c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:43.119 WARN 21062 --- [ocp-async-9,eab1662b9356b420,aef8878092f9e297] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:43.156 INFO 21062 --- [ocp-async-10,1c79462c9f3725af,cce6107345966054] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:43.543 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:43.544 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:44.983 INFO 21062 --- [prometheus-minute-schedule-1,cfbee12b3d672792,e856c573481606f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:45.119 WARN 21062 --- [ocp-async-10,49b7e5a5ce2222af,e9a1562e2f1c7feb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:45.156 INFO 21062 --- [ocp-async-6,d284d7269f119b05,cb96896f00797cb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:46.544 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:46.545 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:47.119 WARN 21062 --- [ocp-async-6,f77d3ec3b162b466,b73318fa04a3bfd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:47.156 INFO 21062 --- [ocp-async-3,bffd076001866b8a,e235d2a0435ebb0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b45b4ae39c5fd7ee,bde9f77219509199] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:12:48.590789+08:00, lastProcessTime=2024-11-04T14:12:38.589728+08:00 2024-11-04 14:12:48.647 INFO 21062 --- [ocp-async-7,29d507baffc33ea4,815bd87020667d2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:12:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:12:48.590789+08:00 2024-11-04 14:12:48.970 INFO 21062 --- [ocp-schedules-4,ab86c944a9abf9c3,b95968f66906e192] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:12:49.119 WARN 21062 --- [ocp-async-1,c4f7a4291dc77a9a,b1b4f75865d81de6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:49.156 INFO 21062 --- [ocp-async-10,e3368d87d110be98,86c40e20477ad827] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:49.545 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:49.546 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:50.289 INFO 21062 --- [ocp-schedules-22,a5faa9547225a986,52cb8328c0cd996b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:12:50.848 INFO 21062 --- [ocp-schedules-16,69f3809211c1852c,dcf02a8fffd6f0ad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:12:50.983 INFO 21062 --- [prometheus-minute-schedule-1,f746cd1b32fc83d9,26574b7946f177c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:51.119 WARN 21062 --- [ocp-async-10,3fcbc73560aa89dc,5ad9ff854dcbe8c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:51.156 INFO 21062 --- [ocp-async-6,e7e0de39ef146253,e73d809f2901afa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:51.227 INFO 21062 --- [ocp-schedules-8,96d85b30949e4adf,c11ab525d48c272e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:12:51.232 INFO 21062 --- [ocp-schedules-8,96d85b30949e4adf,c11ab525d48c272e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:12:51.232 INFO 21062 --- [ocp-schedules-8,96d85b30949e4adf,c11ab525d48c272e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:12:52.546 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:52.547 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:53.120 WARN 21062 --- [ocp-async-6,6961fcbbf699a3d1,d4ac8ee42d92ace6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:53.157 INFO 21062 --- [ocp-async-3,46f647ef1990fb47,2c7c9d7c1195a4df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:53.983 INFO 21062 --- [prometheus-minute-schedule-1,0bb6fdbcd92f3295,19014bb671d46d75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:55.120 WARN 21062 --- [ocp-async-3,7bef1e6872d9c047,ca9457d328eeda2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:55.157 INFO 21062 --- [ocp-async-8,d8002d95b8060dd0,d4e58a33abe37fb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:55.548 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:55.548 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:56.983 INFO 21062 --- [prometheus-minute-schedule-1,4e476f7b63a1663a,d655d1704eae94ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:12:57.120 WARN 21062 --- [ocp-async-8,3d041acbd9ae458f,e71491b495274b66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:57.157 INFO 21062 --- [ocp-async-7,a300c719dad614b5,ba6a911ed3f1bcf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:58.500 INFO 21062 --- [ocp-schedules-4,4c660e352d8bcd4c,c92e8f8b69c895f9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:12:58.500 INFO 21062 --- [ocp-schedules-9,053ec3be3f7395b7,b297e9fe607c1884] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:12:58.502 INFO 21062 --- [ocp-schedules-21,c4c751b5a7f52ff6,a3f06d67c98585a0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:12:58.504 INFO 21062 --- [ocp-schedules-9,053ec3be3f7395b7,b297e9fe607c1884] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:58.504 INFO 21062 --- [ocp-schedules-4,4c660e352d8bcd4c,c92e8f8b69c895f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:12:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:12:58.505 INFO 21062 --- [ocp-schedules-23,89e442726b155737,4376202bb3a884e0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:12:58.509 INFO 21062 --- [ocp-schedules-19,a8526696a60be7d4,63fe22fa3378aa62] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:12:58.509 INFO 21062 --- [ocp-schedules-19,a8526696a60be7d4,63fe22fa3378aa62] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:12:58.510 INFO 21062 --- [ocp-schedules-11,14da93c2cedc6845,7f09fef789b7c0d9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:12:58.510 INFO 21062 --- [ocp-schedules-11,14da93c2cedc6845,7f09fef789b7c0d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:12:58.510 INFO 21062 --- [ocp-schedules-15,00ac93428e5f69b6,088ce4a62c3383b0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:12:58.510 INFO 21062 --- [ocp-schedules-15,00ac93428e5f69b6,088ce4a62c3383b0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:12:58.548 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:12:58.549 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:12:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:12:58.589517+08:00, lastProcessTime=2024-11-04T14:12:48.590789+08:00 2024-11-04 14:12:58.647 INFO 21062 --- [ocp-async-9,687f9d5ec42ea5de,a673ead3b90d450a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:12:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:12:58.589517+08:00 2024-11-04 14:12:59.106 INFO 21062 --- [ocp-schedules-6,5d8261ca8e5c07c1,7dd41778b08ce9e2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:12:59.120 WARN 21062 --- [ocp-async-6,8fb04b4b2078c460,adeb18213ecf853d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:12:59.157 INFO 21062 --- [ocp-async-3,6ac514560b907fef,182963c69ce88d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:12:59.983 INFO 21062 --- [prometheus-minute-schedule-1,6843194f5e50f891,186cf661db414c5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700780, interval=60 2024-11-04 14:13:00.291 INFO 21062 --- [ocp-schedules-18,b118ca42448e659d,2256e5b44374d938] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:13:00.886 INFO 21062 --- [ocp-schedules-7,b8d6916de8bd55cc,67508c506cd197c3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:13:01.120 WARN 21062 --- [ocp-async-3,103ee963b93ccdad,bd5484f069ab6955] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:01.157 INFO 21062 --- [ocp-async-8,cdee81a37bbbbd1d,666515281df7ff48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:01.549 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:01.550 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ec0d2a3d2ed91c00,df38434e104542fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:03.121 WARN 21062 --- [ocp-async-8,a6fd482f7281d9ae,3fd041c50e80fba9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:03.158 INFO 21062 --- [ocp-async-7,e872b85509821d2e,3db0e728c3ab61f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:04.550 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:04.551 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:05.121 WARN 21062 --- [ocp-async-7,16f76d07b06592b3,ecfe56994733355f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:05.158 INFO 21062 --- [ocp-async-5,25be8b2c34042fe0,331dcafd4d1ec083] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:05.402 INFO 21062 --- [ocp-schedules-12,8d042238656f5ea0,a372f77ba17d95c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:13:05.402 INFO 21062 --- [ocp-schedules-12,07ef2488923a4615,304f61813661d8a6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:13:05.407 INFO 21062 --- [ocp-schedules-12,07ef2488923a4615,304f61813661d8a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:05.437 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:13:05.437 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:13:05.983 INFO 21062 --- [prometheus-minute-schedule-1,72ecc2fbd9617852,d708ac34919656e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:07.121 WARN 21062 --- [ocp-async-5,8f0b8bec260adfd0,3abd19882e532c91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:07.158 INFO 21062 --- [ocp-async-1,aee25db3d9ca2d78,5e4874c1ecd65a97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:07.552 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:07.552 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:08.504 INFO 21062 --- [ocp-schedules-9,053ec3be3f7395b7,b297e9fe607c1884] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:13:08.505 INFO 21062 --- [ocp-schedules-4,4c660e352d8bcd4c,c92e8f8b69c895f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:13:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:13:08.590960+08:00, lastProcessTime=2024-11-04T14:12:58.589517+08:00 2024-11-04 14:13:08.647 INFO 21062 --- [ocp-async-6,bb534fc47b4a835e,25a06143ee31bb89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:13:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:13:08.590960+08:00 2024-11-04 14:13:08.983 INFO 21062 --- [prometheus-minute-schedule-1,fa07a6c47c09a6f8,98611b58cdfa74e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:09.121 WARN 21062 --- [ocp-async-8,d2594a16032f5607,409b23a99cd6d9ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:09.158 INFO 21062 --- [ocp-async-4,fd84fdd2096ebfb6,2edf0928abc3de6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:13:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:13:10.294 INFO 21062 --- [ocp-schedules-10,922fbb79d357354b,91d79636dc8f6f48] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:13:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:13:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:13:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:13:10.320 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:13:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:13:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:13:10.553 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:10.554 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:10.911 INFO 21062 --- [ocp-schedules-17,97c0bd693f7fad3e,7429b5b0ef7ffc83] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:13:11.122 WARN 21062 --- [ocp-async-4,183682f20ea39e18,513f90ba85d96bb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:11.158 INFO 21062 --- [ocp-async-2,731a38f948394650,3be58cacf69b8c23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:11.235 INFO 21062 --- [ocp-schedules-25,ce1e1116121a13f8,a520e702aad89270] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:13:11.241 INFO 21062 --- [ocp-schedules-25,ce1e1116121a13f8,a520e702aad89270] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:13:11.241 INFO 21062 --- [ocp-schedules-25,ce1e1116121a13f8,a520e702aad89270] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=48, affectRows=48 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=24, affectRows=24 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=50, affectRows=50 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=28, affectRows=28 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=17, affectRows=17 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=1, affectRows=1 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=1, validCount=1 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=23, validCount=23 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=17, validCount=17 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=28, validCount=28 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=48, validCount=48 2024-11-04 14:13:11.512 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=50, validCount=50 2024-11-04 14:13:11.514 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=11, affectRows=11 2024-11-04 14:13:11.514 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=11, validCount=11 2024-11-04 14:13:11.515 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=3, affectRows=3 2024-11-04 14:13:11.515 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=26, affectRows=26 2024-11-04 14:13:11.515 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=3, validCount=3 2024-11-04 14:13:11.515 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=26, validCount=26 2024-11-04 14:13:11.515 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=61, affectRows=61 2024-11-04 14:13:11.516 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=61, validCount=61 2024-11-04 14:13:11.516 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=58, affectRows=58 2024-11-04 14:13:11.516 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700720, nodeCount=69, affectRows=69 2024-11-04 14:13:11.516 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=58, validCount=58 2024-11-04 14:13:11.516 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700720, nodeCount=67, validCount=67 2024-11-04 14:13:11.983 INFO 21062 --- [prometheus-minute-schedule-1,495997107b1b4d54,cafca3fffcd4accb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:13.122 WARN 21062 --- [ocp-async-2,1c52ce2543d32db7,2f7a4f5f70dbb437] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:13.158 INFO 21062 --- [ocp-async-9,464aa82e4fd0f48c,be4cccfd72ff43f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:13.554 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:13.555 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:13.984 INFO 21062 --- [prometheus-cache-monitor1,e8e3479a98739c60,a17162b7d8b0d8be] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1576,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:13:13.985 INFO 21062 --- [prometheus-cache-monitor1,e8e3479a98739c60,a17162b7d8b0d8be] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:13:14.210 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:07:14.193, end=2024-11-04T14:12:14.208} 2024-11-04 14:13:14.210 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:07:14.192, end=2024-11-04T14:12:14.208} 2024-11-04 14:13:14.220 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:13:14.220 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:13:14.220 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:13:14.220 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:13:14.958 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:09:14.936, end=2024-11-04T14:11:14.955} 2024-11-04 14:13:14.962 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:13:14.962 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:13:14.983 INFO 21062 --- [prometheus-minute-schedule-1,71ae05bfc4eb1fa3,14b6f244e3a964be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:15.122 WARN 21062 --- [ocp-async-9,addc0bfd75b39915,00e4b15cbdb3fee9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:15.159 INFO 21062 --- [ocp-async-10,29df82fe4d258813,158543ff4b373c07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:16.555 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:16.557 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:17.122 WARN 21062 --- [ocp-async-7,3ea9ca6904c1bc5e,387f3168443b37b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:17.159 INFO 21062 --- [ocp-async-6,307341055874f883,86f2f62791650aec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:17.983 INFO 21062 --- [prometheus-minute-schedule-1,c606aa723bbf0b81,72120b86b340ef8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:13:18.590698+08:00, lastProcessTime=2024-11-04T14:13:08.590960+08:00 2024-11-04 14:13:18.648 INFO 21062 --- [ocp-async-8,b78484a95022d23a,ec6644315b6efec7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:13:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:13:18.590698+08:00 2024-11-04 14:13:19.122 WARN 21062 --- [ocp-async-2,dfe1527b3db14665,df0e9992e719e61b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:19.159 INFO 21062 --- [ocp-async-1,0122499cf559746d,11711a758b00e985] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:19.557 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:19.558 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:20.297 INFO 21062 --- [ocp-schedules-16,dba840321c2b65e1,a711270dbbd001af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:13:20.952 INFO 21062 --- [ocp-schedules-6,a34da89e4e777c0b,49bce00ade113c73] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:13:20.983 INFO 21062 --- [prometheus-minute-schedule-1,29d9de7e9102ffd7,c5478f263ecebfc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:21.123 WARN 21062 --- [ocp-async-9,32b3d1546ad6061a,6434b51530c61e41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:21.159 INFO 21062 --- [ocp-async-10,f4db94a109424c18,968b00543ec1ef19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:22.558 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:22.559 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:23.123 WARN 21062 --- [ocp-async-7,555a8e10cb008511,a481385ad500bf13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:23.160 INFO 21062 --- [ocp-async-6,3e032de1b10892bd,9cf2d7e3175ba4c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:23.983 INFO 21062 --- [prometheus-minute-schedule-1,92a1d9bc1b99f8fe,a553fdaf08079d0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:25.123 WARN 21062 --- [ocp-async-5,29eda8dbf2963611,b91329de64fa48f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:25.160 INFO 21062 --- [ocp-async-3,3a0f71bd6103f9f0,01558343c88a25e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:25.559 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:25.559 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:26.983 INFO 21062 --- [prometheus-minute-schedule-1,81d4c1d35d0efc75,c3568b8bbf54f7e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:26.984 INFO 21062 --- [prometheus-hour-schedule-1,eb1ce94a5e6add18,f1f5938dc2c513b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:13:27.123 WARN 21062 --- [ocp-async-1,b15eec5f5ff6ce82,f52f9d2cc427cee4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:27.160 INFO 21062 --- [ocp-async-8,c5bd2afd7f1b821e,f3985cba4f5dd7b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:28.498 INFO 21062 --- [ocp-schedules-4,f263ae9bef66b874,f03d1f3c6901e842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:13:28.499 INFO 21062 --- [ocp-schedules-28,7e3b2aa47868c4c8,fb8b884f4f3a979c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:13:28.500 INFO 21062 --- [ocp-schedules-21,08f17cbb24ed4913,27641558429a4d5a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:13:28.500 INFO 21062 --- [ocp-schedules-9,44b3480a388092e6,8790f6444fa1e1c1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:13:28.502 INFO 21062 --- [ocp-schedules-4,f263ae9bef66b874,f03d1f3c6901e842] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:13:28.502 INFO 21062 --- [ocp-schedules-23,cb81473805dfaac8,b2d29905395456ff] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:13:28.504 INFO 21062 --- [ocp-schedules-21,08f17cbb24ed4913,27641558429a4d5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:28.504 INFO 21062 --- [ocp-schedules-9,44b3480a388092e6,8790f6444fa1e1c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:28.504 INFO 21062 --- [ocp-schedules-30,0978572aa9e4e32e,3cf0bfa6095d3855] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:13:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:13:28.505 INFO 21062 --- [ocp-schedules-28,7e3b2aa47868c4c8,fb8b884f4f3a979c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:13:28.510 INFO 21062 --- [ocp-schedules-18,800d7d63d62044a0,6fb7810c75ef5381] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:13:28.510 INFO 21062 --- [ocp-schedules-18,800d7d63d62044a0,6fb7810c75ef5381] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:13:28.510 INFO 21062 --- [ocp-schedules-8,e6395354dd4c5669,1f7875a5a4c9822b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:13:28.510 INFO 21062 --- [ocp-schedules-20,4f7d9ca0db030781,60f7d824402f5b4e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:13:28.510 INFO 21062 --- [ocp-schedules-8,e6395354dd4c5669,1f7875a5a4c9822b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:13:28.510 INFO 21062 --- [ocp-schedules-20,4f7d9ca0db030781,60f7d824402f5b4e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:13:28.559 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:28.560 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:28.574 INFO 21062 --- [ocp-schedules-1,06fb2f87ac0871e2,91cfb50bc8b8df46] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 14:13:28.574 INFO 21062 --- [ocp-schedules-1,06fb2f87ac0871e2,91cfb50bc8b8df46] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 14:13:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:13:28.590352+08:00, lastProcessTime=2024-11-04T14:13:18.590698+08:00 2024-11-04 14:13:28.648 INFO 21062 --- [ocp-async-1,d9156b69a7fed4b5,e10733a1cd95e769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:13:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:13:28.590352+08:00 2024-11-04 14:13:28.804 INFO 21062 --- [ocp-schedules-10,2d14192c453a7a23,d98f4ee25f3823c2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:13:29.048 INFO 21062 --- [ocp-schedules-11,fd6b2b42c3e55b92,ba18e4da98afdd33] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:13:29.048 INFO 21062 --- [ocp-schedules-11,fd6b2b42c3e55b92,ba18e4da98afdd33] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:13:29.108 INFO 21062 --- [ocp-schedules-18,17a46de168d87f72,81b26c9288b5d523] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:13:29.123 WARN 21062 --- [ocp-async-5,d79265028994d47e,8e4198577b4d9ced] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:29.141 INFO 21062 --- [ocp-schedules-22,c27197b89231ef6a,771743c3ffc0ba98] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:13:29.146 INFO 21062 --- [ocp-schedules-22,c27197b89231ef6a,771743c3ffc0ba98] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:13:29.146 INFO 21062 --- [ocp-schedules-22,c27197b89231ef6a,771743c3ffc0ba98] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:13:29.160 INFO 21062 --- [ocp-async-2,ae0e92f4b0f3d32a,9e6098284d921b0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:29.271 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:13:29.271 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:13:29.280 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:13:29.280 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:13:29.280 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:13:29.291 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:13:29.291 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:13:29.291 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:13:29.295 INFO 21062 --- [ocp-schedules-6,e2b7e1dbea576a1f,26ebd637e6775e75] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:13:29.444 INFO 21062 --- [ocp-schedules-14,7440847b7b4d270a,6bae09cf14ad86f8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:13:29.444 INFO 21062 --- [ocp-schedules-14,7440847b7b4d270a,6bae09cf14ad86f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:13:29.451 INFO 21062 --- [ocp-schedules-14,7440847b7b4d270a,6bae09cf14ad86f8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:13:29.451 INFO 21062 --- [ocp-schedules-14,7440847b7b4d270a,6bae09cf14ad86f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:13:29.455 INFO 21062 --- [ocp-schedules-14,7440847b7b4d270a,6bae09cf14ad86f8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:13:29.455 INFO 21062 --- [ocp-schedules-14,7440847b7b4d270a,6bae09cf14ad86f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:13:29.983 INFO 21062 --- [prometheus-minute-schedule-1,0580c05fae91becd,4cd8c3d21e91a439] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:13:30.300 INFO 21062 --- [ocp-schedules-17,cb7e4ec98274266e,7efb990f349f0d5e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:13:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:13:30.978 INFO 21062 --- [ocp-schedules-16,e02eb04a2afcc95c,c07142f337930a48] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:13:31.123 WARN 21062 --- [ocp-async-3,e8b29a07632b98eb,9084c9f9a6d0b257] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:31.160 INFO 21062 --- [ocp-async-9,d1b53baf0860f187,3c1ddbf4569626e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:31.244 INFO 21062 --- [ocp-schedules-25,0b1ff419e335036c,4608c991aa96d75a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:13:31.250 INFO 21062 --- [ocp-schedules-25,0b1ff419e335036c,4608c991aa96d75a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:13:31.250 INFO 21062 --- [ocp-schedules-25,0b1ff419e335036c,4608c991aa96d75a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:13:31.277 INFO 21062 --- [ocp-schedules-29,278363ec95bea496,cacd5154eec1247a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:13:31.278 INFO 21062 --- [ocp-schedules-29,278363ec95bea496,cacd5154eec1247a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:13:31.304 INFO 21062 --- [ocp-schedules-29,278363ec95bea496,cacd5154eec1247a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:13:31.304 INFO 21062 --- [ocp-schedules-29,278363ec95bea496,cacd5154eec1247a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:13:31.306 INFO 21062 --- [ocp-schedules-5,97f4c5c15997df75,f07b1ec44a22b576] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:13:31.481 INFO 21062 --- [ocp-schedules-18,e2e1a46ffe978827,bd2dc291f333ea25] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:13:31.481+08:00, clusters=[1] 2024-11-04 14:13:31.482 INFO 21062 --- [ocp-schedules-18,e2e1a46ffe978827,bd2dc291f333ea25] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:13:31.560 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:31.561 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:31.631 INFO 21062 --- [ocp-schedules-13,374722fe3345fd08,99f84c86f10abce0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:13:31.631+08:00, clusters=[1] 2024-11-04 14:13:31.634 INFO 21062 --- [ocp-schedules-13,374722fe3345fd08,99f84c86f10abce0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:13:32.449 INFO 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:13:32.449 INFO 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:13:32.449 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:13:32.449 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:13:32.459 INFO 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:13:32.459 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:13:32.460 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:13:32.461 WARN 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:13:32.461 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:13:32.461 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:13:32.461 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:13:32.461 INFO 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:13:32.461 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:13:32.463 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:13:32.463 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:13:32.463 WARN 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:13:32.478 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:13:32.479 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:13:32.479 INFO 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:13:32.490 ERROR 21062 --- [ocp-schedules-16,9d89bdd7fddee378,e6e0176d0da8a1c1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:13:32.983 INFO 21062 --- [prometheus-minute-schedule-1,4c7689dfc100ef98,aa39f1db81739b76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:33.123 WARN 21062 --- [ocp-async-8,2cd4f50595508441,92dc29cbb27aa79e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:33.161 INFO 21062 --- [ocp-async-7,7060b45cba8a545a,03b403f03c79be47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:34.561 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:34.562 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:35.124 WARN 21062 --- [ocp-async-4,234cdeb60ee48147,23e441621e560c45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:35.161 INFO 21062 --- [ocp-async-6,dbab10b8a9df6cdf,72b65ec24d58a991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:35.437 INFO 21062 --- [ocp-schedules-12,07ef2488923a4615,304f61813661d8a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:13:35.437 INFO 21062 --- [ocp-schedules-12,427a9e34396c9984,f8d4b0cfc4582e6c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:13:35.441 INFO 21062 --- [ocp-schedules-12,427a9e34396c9984,f8d4b0cfc4582e6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:35.470 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:13:35.470 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:13:35.983 INFO 21062 --- [prometheus-minute-schedule-1,df0697789e3d4f11,026a839bd7859a6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:37.124 WARN 21062 --- [ocp-async-6,788d9f1dd29b3e41,666706f4071242cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:37.161 INFO 21062 --- [ocp-async-1,8a8b6018a8e77c4b,e7045f9c9b81ce42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:37.562 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:37.562 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:37.832 INFO 21062 --- [ocp-schedules-5,a939888a2b12b28f,27bb3bd75cd3fdd0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:13:37.836 INFO 21062 --- [ocp-schedules-5,a939888a2b12b28f,27bb3bd75cd3fdd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:37.920 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:13:37.922 ERROR 21062 --- [ocp-schedules-5,a939888a2b12b28f,27bb3bd75cd3fdd0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:13:37.923 INFO 21062 --- [ocp-schedules-5,a939888a2b12b28f,27bb3bd75cd3fdd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:13:38.504 INFO 21062 --- [ocp-schedules-9,44b3480a388092e6,8790f6444fa1e1c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:13:38.505 INFO 21062 --- [ocp-schedules-21,08f17cbb24ed4913,27641558429a4d5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:13:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:13:38.590357+08:00, lastProcessTime=2024-11-04T14:13:28.590352+08:00 2024-11-04 14:13:38.648 INFO 21062 --- [ocp-async-6,ff256fcd7ddfdbda,3ac6c07b034e8973] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:13:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:13:38.590357+08:00 2024-11-04 14:13:38.983 INFO 21062 --- [prometheus-minute-schedule-1,36710362cd7d44ba,4a0ae01c41d7edc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:39.124 WARN 21062 --- [ocp-async-8,4b8151c42051a7e2,e13927f45599ce5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:39.161 INFO 21062 --- [ocp-async-4,e53b0dee44380cf3,4fb75859659623d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:13:40.004 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:13:40.303 INFO 21062 --- [ocp-schedules-5,990c3cd73d98d674,f341307d5e570339] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:13:40.563 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:40.563 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:41.021 INFO 21062 --- [ocp-schedules-22,b3390c46698f2583,621d2a52b7a12005] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:13:41.124 WARN 21062 --- [ocp-async-4,427ac8f5109094f9,39f04bcabff6ab76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:41.161 INFO 21062 --- [ocp-async-2,eed57c6afd24bee0,9be4b7fa1907c520] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2ceee6a54c11ce4d,f9293ed212c9f59f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:43.124 WARN 21062 --- [ocp-async-2,a9948021784f20f8,d837e15010618a81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:43.161 INFO 21062 --- [ocp-async-9,843bd1832616153f,bb69b80cbdbb0551] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:43.563 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:43.564 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:44.983 INFO 21062 --- [prometheus-minute-schedule-1,bf310b0f5715948b,57d81423b8a81109] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:45.125 WARN 21062 --- [ocp-async-9,b85b280fe566ff7d,7d13563cb372c61a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:45.162 INFO 21062 --- [ocp-async-10,0234443e0458d1c1,b1f1a6284a778bd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:46.564 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:46.565 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:47.125 WARN 21062 --- [ocp-async-10,40ab132ab5952d6d,e0366408c8e47ccb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:47.162 INFO 21062 --- [ocp-async-6,75e70ca60d6b12c2,b2708167567b0857] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:47.983 INFO 21062 --- [prometheus-minute-schedule-1,19712f5b7fef22dd,bb6868124d25e8fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:13:48.591807+08:00, lastProcessTime=2024-11-04T14:13:38.590357+08:00 2024-11-04 14:13:48.648 INFO 21062 --- [ocp-async-7,b87b38e3b2cd1d00,77810ef8e471259d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:13:48.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:13:48.591807+08:00 2024-11-04 14:13:48.970 INFO 21062 --- [ocp-schedules-11,b2337975103fcdf4,bd8420b99de64ca4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:13:48.973 INFO 21062 --- [ocp-schedules-11,b2337975103fcdf4,bd8420b99de64ca4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:48.974 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:13:49.125 WARN 21062 --- [ocp-async-2,1a7e97b49670ae64,03360a7450539889] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:49.162 INFO 21062 --- [ocp-async-9,07df75c2c6065cc9,e04d0df89aba3943] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:49.565 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:49.566 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:50.307 INFO 21062 --- [ocp-schedules-27,3576a55dea74133d,3b1cf2157935219c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:13:50.983 INFO 21062 --- [prometheus-minute-schedule-1,bdd541b9284323dc,61b31b511f3edb6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:51.047 INFO 21062 --- [ocp-schedules-22,7820c7810a95cb4b,4d95a9fd5bfb0ed4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:13:51.125 WARN 21062 --- [ocp-async-9,29971cbcb91286d3,c32576ba960b1472] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:51.162 INFO 21062 --- [ocp-async-10,c4428bece45b6cb0,2b4213943622281b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:51.252 INFO 21062 --- [ocp-schedules-29,3287a3c459cc0292,b306038fbfd68325] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:13:51.258 INFO 21062 --- [ocp-schedules-29,3287a3c459cc0292,b306038fbfd68325] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:13:51.258 INFO 21062 --- [ocp-schedules-29,3287a3c459cc0292,b306038fbfd68325] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:13:52.566 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:52.567 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:53.126 WARN 21062 --- [ocp-async-10,7a8f3d605e02a7fb,d7b454a5d4ee4aa0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:53.162 INFO 21062 --- [ocp-async-6,e29d2ddcf2b895f2,d9a3bbb63c8bdfdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1b0446f37a566ec7,7e029055ca9da2f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:54.352 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f99de703b8dc45fc,77c073f020051bc4] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.130.47.17, traceId f99de703b8dc45fc] 2024-11-04 14:13:54.361 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f99de703b8dc45fc,77c073f020051bc4] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.130.47.17, traceId=f99de703b8dc45fc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2024-11-04 14:13:54.525 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f99de703b8dc45fc,77c073f020051bc4] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 13, current state: FAILED, new state: READY 2024-11-04 14:13:54.525 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f99de703b8dc45fc,77c073f020051bc4] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 42, current state: FAILED, new state: READY 2024-11-04 14:13:54.548 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f99de703b8dc45fc,77c073f020051bc4] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.130.47.17, serverIp=null, startTime=2024-11-04T14:13:54.363+08:00, endTime=2024-11-04T14:13:54.548+08:00, role=null, aliUid=null) 2024-11-04 14:13:54.561 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f99de703b8dc45fc,77c073f020051bc4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.130.47.17, traceId=f99de703b8dc45fc, duration=216 ms] 2024-11-04 14:13:54.971 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,c873b0020037ea02,93ed6e5b21310ead] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c873b0020037ea02, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:13:54.998 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,c873b0020037ea02,93ed6e5b21310ead] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c873b0020037ea02, duration=35 ms] 2024-11-04 14:13:55.126 WARN 21062 --- [ocp-async-6,cb92c5b8c9e8f093,5dadcb9c462c77bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:55.163 INFO 21062 --- [ocp-async-1,05972ed7cbbbe287,366a73e6b78c18ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:55.218 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,13302de07d5c492a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: RUNNING 2024-11-04 14:13:55.225 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,13302de07d5c492a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: RUNNING 2024-11-04 14:13:55.238 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=13, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:13:55.211+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=13}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:13:55.241 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:13:55.243 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:13:55.244 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=42, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:13:55.222+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=42}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:13:55.248 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:13:55.250 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:13:55.255 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:13:55.257 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:13:55.270 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:55.275 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:55.276 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.281 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.322 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:55.327 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:55.327 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.332 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.374 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:13:55.379 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:13:55.404 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.407 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.407 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.410 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.447 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:55.452 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.454 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:55.461 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.499 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:13:55.502 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:13:55.524 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.527 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.535 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.538 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.567 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:55.568 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:55.571 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:55.577 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.582 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:55.588 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.623 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:13:55.634 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:13:55.648 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.652 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.655 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:13:55.668 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.671 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.673 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:13:55.695 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:55.701 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.710 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:55.716 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.747 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 14:13:55.762 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 14:13:55.794 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.798 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.800 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:13:55.802 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:13:55.805 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:13:55.835 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:55.841 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.853 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.856 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.858 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:13:55.860 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:13:55.863 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:13:55.883 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 14:13:55.894 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:55.900 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.927 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.930 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.933 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:13:55.936 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:13:55.938 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:13:55.941 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:13:55.942 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 14:13:55.943 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:13:55.950 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:13:55.954 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:13:55.958 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:13:55.962 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:13:55.965 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:13:55.983 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:55.988 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.989 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.994 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:55.996 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:13:55.999 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:13:56.001 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:13:56.003 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:13:56.008 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:13:56.012 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:13:56.016 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:13:56.020 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:13:56.023 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:13:56.025 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:13:56.035 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:56.040 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.045 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:56.050 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.083 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:13:56.094 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:56.099 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.106 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.109 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.146 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:13:56.151 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:56.156 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.170 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ccf8e82ad306af37,9b1a66fb1d1a1a8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=ccf8e82ad306af37, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:13:56.170 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.173 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.175 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ccf8e82ad306af37,9b1a66fb1d1a1a8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=ccf8e82ad306af37, duration=13 ms] 2024-11-04 14:13:56.203 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:13:56.218 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:56.223 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.227 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.230 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.252 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:13:56.266 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:13:56.275 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:56.281 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.290 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.292 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.310 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:13:56.327 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12 2024-11-04 14:13:56.338 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:56.344 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.349 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,a968e17698671e72,2dd8f228eca38eb9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.12, traceId=a968e17698671e72, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:13:56.353 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,a968e17698671e72,2dd8f228eca38eb9] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:13:56.353 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,a968e17698671e72,2dd8f228eca38eb9] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:13:56.353 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,a968e17698671e72,2dd8f228eca38eb9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.12, traceId=a968e17698671e72, duration=6 ms] 2024-11-04 14:13:56.390 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11 2024-11-04 14:13:56.412 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,117c1058510146d2,37df20d716f7b232] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.11, traceId=117c1058510146d2, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:13:56.416 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,117c1058510146d2,37df20d716f7b232] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:13:56.416 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,117c1058510146d2,37df20d716f7b232] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:13:56.417 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,117c1058510146d2,37df20d716f7b232] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.11, traceId=117c1058510146d2, duration=7 ms] 2024-11-04 14:13:56.435 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.438 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.483 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:56.489 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.508 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.511 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.535 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12 2024-11-04 14:13:56.554 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:56.559 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.602 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11 2024-11-04 14:13:56.683 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.686 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.689 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:13:56.731 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:56.737 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.752 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.755 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.758 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:13:56.783 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/e72f17db51f249ff on 10.46.128.12 2024-11-04 14:13:56.789 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/e72f17db51f249ff on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e72f17db51f249ff, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.792 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/e72f17db51f249ff on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e72f17db51f249ff, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.794 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:56.800 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.835 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:56.840 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.842 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/29c88390ba194be6 on 10.46.128.11 2024-11-04 14:13:56.849 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/29c88390ba194be6 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/29c88390ba194be6, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.852 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/29c88390ba194be6 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/29c88390ba194be6, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.887 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/e72f17db51f249ff/cpio && chmod +x /tmp/e72f17db51f249ff/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e72f17db51f249ff/cpio on 10.46.128.12 2024-11-04 14:13:56.894 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:56.894 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/e72f17db51f249ff/cpio && chmod +x /tmp/e72f17db51f249ff/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e72f17db51f249ff/cpio on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e72f17db51f249ff/cpio && chmod +x /tmp/e72f17db51f249ff/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e72f17db51f249ff/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.899 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.899 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/e72f17db51f249ff/cpio && chmod +x /tmp/e72f17db51f249ff/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e72f17db51f249ff/cpio on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e72f17db51f249ff/cpio && chmod +x /tmp/e72f17db51f249ff/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e72f17db51f249ff/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.939 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:56.944 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.946 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/29c88390ba194be6/cpio && chmod +x /tmp/29c88390ba194be6/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/29c88390ba194be6/cpio on 10.46.128.11 2024-11-04 14:13:56.955 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/29c88390ba194be6/cpio && chmod +x /tmp/29c88390ba194be6/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/29c88390ba194be6/cpio on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/29c88390ba194be6/cpio && chmod +x /tmp/29c88390ba194be6/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/29c88390ba194be6/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.958 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/29c88390ba194be6/cpio && chmod +x /tmp/29c88390ba194be6/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/29c88390ba194be6/cpio on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/29c88390ba194be6/cpio && chmod +x /tmp/29c88390ba194be6/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/29c88390ba194be6/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.983 INFO 21062 --- [prometheus-minute-schedule-1,3f1e8ffbc2eef2ed,6833f01b09fb6e0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:13:56.987 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/e72f17db51f249ff on 10.46.128.12 2024-11-04 14:13:56.993 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/e72f17db51f249ff on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e72f17db51f249ff, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:56.996 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/e72f17db51f249ff on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e72f17db51f249ff, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.002 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:57.007 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.039 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:57.044 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.054 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/29c88390ba194be6 on 10.46.128.11 2024-11-04 14:13:57.060 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/29c88390ba194be6 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/29c88390ba194be6, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.062 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/29c88390ba194be6 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/29c88390ba194be6, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.091 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/e72f17db51f249ff/install_package.sh && chmod +x /tmp/e72f17db51f249ff/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e72f17db51f249ff/install_package.sh on 10.46.128.12 2024-11-04 14:13:57.098 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/e72f17db51f249ff/install_package.sh && chmod +x /tmp/e72f17db51f249ff/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e72f17db51f249ff/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e72f17db51f249ff/install_package.sh && chmod +x /tmp/e72f17db51f249ff/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e72f17db51f249ff/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.101 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/e72f17db51f249ff/install_package.sh && chmod +x /tmp/e72f17db51f249ff/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e72f17db51f249ff/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e72f17db51f249ff/install_package.sh && chmod +x /tmp/e72f17db51f249ff/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e72f17db51f249ff/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.106 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:57.112 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.126 WARN 21062 --- [ocp-async-1,5921d500faffc191,03afe6eefdaf51cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:57.139 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:13:57.144 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.154 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/29c88390ba194be6/install_package.sh && chmod +x /tmp/29c88390ba194be6/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/29c88390ba194be6/install_package.sh on 10.46.128.11 2024-11-04 14:13:57.163 INFO 21062 --- [ocp-async-7,4443b100fb3a67d3,cccf1e4d25b34539] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:57.161 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/29c88390ba194be6/install_package.sh && chmod +x /tmp/29c88390ba194be6/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/29c88390ba194be6/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/29c88390ba194be6/install_package.sh && chmod +x /tmp/29c88390ba194be6/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/29c88390ba194be6/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.164 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/29c88390ba194be6/install_package.sh && chmod +x /tmp/29c88390ba194be6/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/29c88390ba194be6/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/29c88390ba194be6/install_package.sh && chmod +x /tmp/29c88390ba194be6/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/29c88390ba194be6/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.191 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/e72f17db51f249ff/install_package.sh on 10.46.128.12 2024-11-04 14:13:57.202 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:13:57.207 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:13:57.254 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/29c88390ba194be6/install_package.sh on 10.46.128.11 2024-11-04 14:13:57.262 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,ae00002f44fb80f8,7e2e012804544e14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=ae00002f44fb80f8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:13:57.288 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,ae00002f44fb80f8,7e2e012804544e14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=ae00002f44fb80f8, duration=33 ms] 2024-11-04 14:13:58.090 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,1c78ee0a8038a51d,e273d9c5401b248a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=1c78ee0a8038a51d, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:13:58.095 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,1c78ee0a8038a51d,e273d9c5401b248a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=1c78ee0a8038a51d, duration=14 ms] 2024-11-04 14:13:58.500 INFO 21062 --- [ocp-schedules-28,2f631084359948fb,cad480ffd947ac75] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:13:58.500 INFO 21062 --- [ocp-schedules-6,d229b72ab9d41687,cbaf12390e483d12] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:13:58.502 INFO 21062 --- [ocp-schedules-29,db9530642232e388,6a6b2fc4d77b9984] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:13:58.504 INFO 21062 --- [ocp-schedules-28,2f631084359948fb,cad480ffd947ac75] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:58.504 INFO 21062 --- [ocp-schedules-6,d229b72ab9d41687,cbaf12390e483d12] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:13:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:13:58.505 INFO 21062 --- [ocp-schedules-4,91458436fed1c321,d91f0711529b8387] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:13:58.511 INFO 21062 --- [ocp-schedules-18,a0b376740b28e78c,0d35e4e7357f125c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:13:58.511 INFO 21062 --- [ocp-schedules-10,1476b285997fa20a,3da29ece6107bc3f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:13:58.511 INFO 21062 --- [ocp-schedules-10,1476b285997fa20a,3da29ece6107bc3f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:13:58.511 INFO 21062 --- [ocp-schedules-18,a0b376740b28e78c,0d35e4e7357f125c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:13:58.511 INFO 21062 --- [ocp-schedules-17,5efb1776c10013d2,3d705c0f1943fbe2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:13:58.511 INFO 21062 --- [ocp-schedules-17,5efb1776c10013d2,3d705c0f1943fbe2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:13:58.568 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:13:58.569 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:13:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:13:58.590748+08:00, lastProcessTime=2024-11-04T14:13:48.591807+08:00 2024-11-04 14:13:58.648 INFO 21062 --- [ocp-async-2,90384654402895c2,71c0c061a70b80a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:13:58.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:13:58.590748+08:00 2024-11-04 14:13:58.975 INFO 21062 --- [ocp-schedules-11,b2337975103fcdf4,bd8420b99de64ca4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:13:59.111 INFO 21062 --- [ocp-schedules-24,0900c4dc683725cb,e25758a2dfde135e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:13:59.113 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,e9e3230caf44615a,50288fbb464afde4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e9e3230caf44615a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:13:59.126 WARN 21062 --- [ocp-async-10,2b38864ebe186be0,4ef0d4b3f5dc065b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:13:59.140 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,e9e3230caf44615a,50288fbb464afde4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e9e3230caf44615a, duration=35 ms] 2024-11-04 14:13:59.163 INFO 21062 --- [ocp-async-5,e8b6dba9f0d27cc8,1dced068be3cc23d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:13:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7725e6a0a23ca22b,529bb04ac503b081] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700840, interval=60 2024-11-04 14:14:00.310 INFO 21062 --- [ocp-schedules-30,d3d7a4559b67537a,826e7622304c8cde] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:14:00.738 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9c848aca0cf3a7fa,b5a11bae22957e22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9c848aca0cf3a7fa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:14:00.763 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9c848aca0cf3a7fa,b5a11bae22957e22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9c848aca0cf3a7fa, duration=35 ms] 2024-11-04 14:14:01.091 INFO 21062 --- [ocp-schedules-17,4f4b385c6ec81b64,a117d2067afa7c81] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:14:01.126 WARN 21062 --- [ocp-async-5,2318f229adef5d24,17e9ca337de2240d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:01.163 INFO 21062 --- [ocp-async-2,2f428d1add6b8a11,0d437618196bc085] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:01.569 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:01.570 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:02.369 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d80574e837b58baa,99ce788a3924acf2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=d80574e837b58baa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:14:02.395 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d80574e837b58baa,99ce788a3924acf2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=d80574e837b58baa, duration=34 ms] 2024-11-04 14:14:02.983 INFO 21062 --- [prometheus-minute-schedule-1,56e7f2dd55c2b4c2,0bdc1a50b02ed1d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:03.126 WARN 21062 --- [ocp-async-2,91ed8ae5dedc3602,b1db059922346c86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:03.163 INFO 21062 --- [ocp-async-8,36e2555c288fbde0,8f036a29bfedfecd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:03.973 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ab301addba75d713,a70a5e47a151f08e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=ab301addba75d713, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:14:03.999 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ab301addba75d713,a70a5e47a151f08e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=ab301addba75d713, duration=33 ms] 2024-11-04 14:14:04.570 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:04.571 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:05.028 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/e72f17db51f249ff/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/e72f17db51f249ff/install_package.sh, out=, err=sudo: alien: command not found , extOut=null, exitStatus=1) 2024-11-04 14:14:05.031 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/e72f17db51f249ff/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/e72f17db51f249ff/install_package.sh, out=, err=sudo: alien: command not found , extOut=null, exitStatus=1) 2024-11-04 14:14:05.034 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/e72f17db51f249ff/install_package.sh, return code:1, output:, err:sudo: alien: command not found 2024-11-04 14:14:05.036 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/e72f17db51f249ff/install_package.sh, exit code: 1, cause:null 2024-11-04 14:14:05.040 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/e72f17db51f249ff/install_package.sh, exit code: 1 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/e72f17db51f249ff/install_package.sh, exit code: 1 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:14:05.051 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,4cb8109c5f4cebca] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: FAILED 2024-11-04 14:14:05.127 WARN 21062 --- [ocp-async-8,5fa4dc6eb4e1099e,6e827f789d2028cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:05.163 INFO 21062 --- [ocp-async-4,9d90f0237bc8c41c,4ebb495d830115d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:05.232 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/29c88390ba194be6/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/29c88390ba194be6/install_package.sh, out=, err=sudo: alien: command not found , extOut=null, exitStatus=1) 2024-11-04 14:14:05.235 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/29c88390ba194be6/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/29c88390ba194be6/install_package.sh, out=, err=sudo: alien: command not found , extOut=null, exitStatus=1) 2024-11-04 14:14:05.238 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/29c88390ba194be6/install_package.sh, return code:1, output:, err:sudo: alien: command not found 2024-11-04 14:14:05.240 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/29c88390ba194be6/install_package.sh, exit code: 1, cause:null 2024-11-04 14:14:05.242 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/29c88390ba194be6/install_package.sh, exit code: 1 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/29c88390ba194be6/install_package.sh, exit code: 1 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:14:05.252 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,8a37d87e8b9db03f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: FAILED 2024-11-04 14:14:05.433 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=2024-11-04T14:11:33.591+08:00, subtasks=[SubtaskInstanceEntity{id=68, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=33, startTime=2024-11-04T14:11:16.789+08:00, endTime=2024-11-04T14:11:16.854+08:00}, SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busin 2024-11-04 14:14:05.443 WARN 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:14:05.443 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 14:14:05.444 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:2, status:FAILED 2024-11-04 14:14:05.471 INFO 21062 --- [ocp-schedules-12,427a9e34396c9984,f8d4b0cfc4582e6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:14:05.471 INFO 21062 --- [ocp-schedules-12,cb289977817262d1,2d4db605b0032349] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:14:05.474 INFO 21062 --- [ocp-schedules-12,cb289977817262d1,2d4db605b0032349] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:05.498 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,3c82a27e0adcf038,3698b107dd1324b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=3c82a27e0adcf038, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:14:05.502 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:14:05.502 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:14:05.523 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,3c82a27e0adcf038,3698b107dd1324b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=3c82a27e0adcf038, duration=30 ms] 2024-11-04 14:14:05.983 INFO 21062 --- [prometheus-minute-schedule-1,b20533eb2f08d872,1225c67ae25fd6c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:06.028 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,7ca9767829ee8c0c,bc19da039692ced2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=7ca9767829ee8c0c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:14:06.032 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,7ca9767829ee8c0c,bc19da039692ced2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=7ca9767829ee8c0c, duration=11 ms] 2024-11-04 14:14:07.127 WARN 21062 --- [ocp-async-4,5364d97299e364ad,70be6a2d3b9e42e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:07.163 INFO 21062 --- [ocp-async-3,11ef7f808cd2e0d4,32c1c846637d4eeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:07.571 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:07.572 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:08.505 INFO 21062 --- [ocp-schedules-6,d229b72ab9d41687,cbaf12390e483d12] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:14:08.505 INFO 21062 --- [ocp-schedules-28,2f631084359948fb,cad480ffd947ac75] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:14:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:14:08.590550+08:00, lastProcessTime=2024-11-04T14:13:58.590748+08:00 2024-11-04 14:14:08.648 INFO 21062 --- [ocp-async-10,fa78b2a563e92ac2,2ea6e513fb9fd192] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:14:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:14:08.590550+08:00 2024-11-04 14:14:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b14c9582d65fc351,05d59479bffe79b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:09.127 WARN 21062 --- [ocp-async-2,ea26f3f00a6d986b,423903829b7bbddc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:09.164 INFO 21062 --- [ocp-async-7,acdabb75ed711380,d2469e1a6f3986d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:10.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:14:10.004 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:14:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:14:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:14:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:14:10.312 INFO 21062 --- [ocp-schedules-24,c8c252befdef715c,c00eeca40be052f5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:14:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:14:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:14:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:14:10.572 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:10.573 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:11.117 INFO 21062 --- [ocp-schedules-7,18e0adb1019e760c,850f8bb143753f7c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:14:11.127 WARN 21062 --- [ocp-async-8,87144f54f9f42ff4,3183b69678324f26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:11.164 INFO 21062 --- [ocp-async-6,3ab8e0e76856fc98,5d61ba7e03137311] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:11.262 INFO 21062 --- [ocp-schedules-13,51d03f3d6c58b21f,630304ece062fa5e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:14:11.267 INFO 21062 --- [ocp-schedules-13,51d03f3d6c58b21f,630304ece062fa5e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:14:11.267 INFO 21062 --- [ocp-schedules-13,51d03f3d6c58b21f,630304ece062fa5e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:14:11.508 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=5, affectRows=5 2024-11-04 14:14:11.508 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=10, affectRows=10 2024-11-04 14:14:11.508 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=4, validCount=4 2024-11-04 14:14:11.508 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=10, validCount=10 2024-11-04 14:14:11.509 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=17, affectRows=17 2024-11-04 14:14:11.509 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=17, validCount=17 2024-11-04 14:14:11.509 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=18, affectRows=18 2024-11-04 14:14:11.509 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=18, validCount=18 2024-11-04 14:14:11.509 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=53, affectRows=53 2024-11-04 14:14:11.510 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=53, validCount=53 2024-11-04 14:14:11.510 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=48, affectRows=48 2024-11-04 14:14:11.510 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=73, affectRows=73 2024-11-04 14:14:11.510 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=72, affectRows=72 2024-11-04 14:14:11.510 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=48, validCount=48 2024-11-04 14:14:11.510 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=72, validCount=72 2024-11-04 14:14:11.511 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=72, validCount=72 2024-11-04 14:14:11.511 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=48, affectRows=48 2024-11-04 14:14:11.511 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700780, nodeCount=52, affectRows=52 2024-11-04 14:14:11.511 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=48, validCount=48 2024-11-04 14:14:11.511 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700780, nodeCount=51, validCount=51 2024-11-04 14:14:11.983 INFO 21062 --- [prometheus-minute-schedule-1,663cba704fd530ef,d930073bb1e323b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:13.127 WARN 21062 --- [ocp-async-4,840dd2d4345316e3,a79830b397e3caef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:13.164 INFO 21062 --- [ocp-async-3,578a10fbe96172c7,5701a7ebc5ea2e9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:13.573 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:13.574 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:13.984 INFO 21062 --- [prometheus-cache-monitor1,69d8643e299daec8,b0942a7467cdbd4a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1580,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:14:13.984 INFO 21062 --- [prometheus-cache-monitor1,69d8643e299daec8,b0942a7467cdbd4a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:14:14.983 INFO 21062 --- [prometheus-minute-schedule-1,b9e939ad90010d2c,8d5baab96ed09149] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:15.127 WARN 21062 --- [ocp-async-1,69efa78776b30016,d09f2ecec6505214] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:15.164 INFO 21062 --- [ocp-async-9,a6abf510eace67bb,1ff5463e5d213f9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:16.574 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:16.575 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:17.128 WARN 21062 --- [ocp-async-7,a77a26f0d8ea2309,b532348a9bdb98a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:17.164 INFO 21062 --- [ocp-async-10,ce389b1eae63140d,abdf20e30eac2ca7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:17.983 INFO 21062 --- [prometheus-minute-schedule-1,2c8ee037d9c45106,d383c2b7e549fbd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:18.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:14:18.592244+08:00, lastProcessTime=2024-11-04T14:14:08.590550+08:00 2024-11-04 14:14:18.649 INFO 21062 --- [ocp-async-2,c1ae6d7f0de8a91b,e1afa45850b1999a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:14:18.804 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:14:18.592244+08:00 2024-11-04 14:14:19.128 WARN 21062 --- [ocp-async-4,1dcb1859295bb7cd,e640cefa49ef8345] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:19.165 INFO 21062 --- [ocp-async-3,cee50a6fd13fef8a,5344216c17046220] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:19.575 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:19.577 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:20.315 INFO 21062 --- [ocp-schedules-7,d60a64e385186b81,cd1f555074daa5ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:14:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4093157152cec833,b5422a4bb70698dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:21.128 WARN 21062 --- [ocp-async-1,1138a360b942066e,c319a97161d97712] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:21.159 INFO 21062 --- [ocp-schedules-23,a740af98c8c35a28,bd0c7b5e9bd7f625] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:14:21.165 INFO 21062 --- [ocp-async-9,94a530273ad7bc33,91e7d9c7d3a3a30b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:22.577 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:22.578 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:23.128 WARN 21062 --- [ocp-async-7,5f24f942b686a586,0a257aeae716daf8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:23.165 INFO 21062 --- [ocp-async-10,13c6307fb370228b,840029151d73053b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c108565dae88b62e,41ad940f38f5c536] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:25.128 WARN 21062 --- [ocp-async-6,53b2db2d8aff3cff,325652d7a20f8999] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:25.165 INFO 21062 --- [ocp-async-5,dddfdf9fa9a68adb,8ab0776e93a410d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:25.578 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:25.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:26.983 INFO 21062 --- [prometheus-minute-schedule-1,4673ba75c36af873,6d0a1b6134344692] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:26.984 INFO 21062 --- [prometheus-hour-schedule-1,47d98b7847fe47df,da279bd48a3f95d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:14:27.128 WARN 21062 --- [ocp-async-3,217139cb6b691735,432b1b1a193b1e66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:27.165 INFO 21062 --- [ocp-async-2,db3781ec42582337,54d3e4aac5a46e18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:28.498 INFO 21062 --- [ocp-schedules-9,671bc9731c1e207a,fb0145fb1bdcd4b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:14:28.499 INFO 21062 --- [ocp-schedules-23,822b470346f39455,ce3dbf15b0642db9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:14:28.500 INFO 21062 --- [ocp-schedules-1,c00b0f4025eddb7e,d64b2201b4f19339] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:14:28.500 INFO 21062 --- [ocp-schedules-22,321ef33244ec674f,24e85c2293625492] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:14:28.502 INFO 21062 --- [ocp-schedules-9,671bc9731c1e207a,fb0145fb1bdcd4b4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:14:28.502 INFO 21062 --- [ocp-schedules-8,9d3694acfaf96bc4,35ab0563bbf5360a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:14:28.503 INFO 21062 --- [ocp-schedules-1,c00b0f4025eddb7e,d64b2201b4f19339] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:28.503 INFO 21062 --- [ocp-schedules-22,321ef33244ec674f,24e85c2293625492] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:14:28.504 INFO 21062 --- [ocp-schedules-23,822b470346f39455,ce3dbf15b0642db9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:14:28.504 INFO 21062 --- [ocp-schedules-14,9f008fe696f9d983,f6db3dbfd3be937e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:14:28.508 INFO 21062 --- [ocp-schedules-27,8fa33d444295eabf,d919c66167d83737] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:14:28.508 INFO 21062 --- [ocp-schedules-27,8fa33d444295eabf,d919c66167d83737] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:14:28.509 INFO 21062 --- [ocp-schedules-11,efa592e75eea4ec5,212e9981f465a3f9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:14:28.509 INFO 21062 --- [ocp-schedules-11,efa592e75eea4ec5,212e9981f465a3f9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:14:28.509 INFO 21062 --- [ocp-schedules-4,f3f1f2757fd870a7,8e4763fad6bdb978] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:14:28.509 INFO 21062 --- [ocp-schedules-4,f3f1f2757fd870a7,8e4763fad6bdb978] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:14:28.574 INFO 21062 --- [ocp-schedules-7,bc2c513ef6df0626,ec3d4933eb8f959c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1580,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:14:28.574 INFO 21062 --- [ocp-schedules-7,bc2c513ef6df0626,ec3d4933eb8f959c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:14:28.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:28.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:28.583 INFO 21062 --- [ocp-schedules-21,3a1638921bc5805e,dc7cefa15726937c] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:14:28.583+08:00, tenantSize=0 2024-11-04 14:14:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:14:28.588884+08:00, lastProcessTime=2024-11-04T14:14:18.592244+08:00 2024-11-04 14:14:28.649 INFO 21062 --- [ocp-async-3,b04046a7ed58e058,48f2dd841596aad5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:14:28.655 INFO 21062 --- [ocp-schedules-25,ec15178aea379470,71fb6ce75413d2ea] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 14:14:28.655 INFO 21062 --- [ocp-schedules-25,ec15178aea379470,71fb6ce75413d2ea] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 14:14:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:14:28.588884+08:00 2024-11-04 14:14:28.770 INFO 21062 --- [ocp-schedules-18,d98ff3dc76cbbba0,af0d071114d20afb] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:14:28.770 INFO 21062 --- [ocp-schedules-18,d98ff3dc76cbbba0,af0d071114d20afb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:14:28.806 INFO 21062 --- [ocp-schedules-23,738a57f7432ce8ec,e315f2b57de760d8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:14:29.053 INFO 21062 --- [ocp-schedules-30,d762941478e94c30,3167f72ddbbbba6d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:14:29.053 INFO 21062 --- [ocp-schedules-30,d762941478e94c30,3167f72ddbbbba6d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:14:29.113 INFO 21062 --- [ocp-schedules-9,c6542b6bc277008f,414407c8ae932a0b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:14:29.129 WARN 21062 --- [ocp-async-6,1469748b5f31ac22,86aba2346d5e7e5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:29.147 INFO 21062 --- [ocp-schedules-7,bee4d33d0c259a48,3db6fc7df5836fc0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:14:29.151 INFO 21062 --- [ocp-schedules-7,bee4d33d0c259a48,3db6fc7df5836fc0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:14:29.152 INFO 21062 --- [ocp-schedules-7,bee4d33d0c259a48,3db6fc7df5836fc0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:14:29.165 INFO 21062 --- [ocp-async-4,66a54edcebfd82a9,077355b8067b4d79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:29.460 INFO 21062 --- [ocp-schedules-28,260e34b0009ad367,aa45ffc687edf2d9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:14:29.460 INFO 21062 --- [ocp-schedules-28,260e34b0009ad367,aa45ffc687edf2d9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:14:29.461 INFO 21062 --- [ocp-schedules-28,260e34b0009ad367,aa45ffc687edf2d9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:14:29.461 INFO 21062 --- [ocp-schedules-28,260e34b0009ad367,aa45ffc687edf2d9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:14:29.462 INFO 21062 --- [ocp-schedules-28,260e34b0009ad367,aa45ffc687edf2d9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:14:29.462 INFO 21062 --- [ocp-schedules-28,260e34b0009ad367,aa45ffc687edf2d9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:14:29.983 INFO 21062 --- [prometheus-minute-schedule-1,db79a1b596105e06,7f2ff3fb201a96e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:14:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:14:30.318 INFO 21062 --- [ocp-schedules-20,2943c02fd0eb604e,e938949c5b4b5463] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:14:31.129 WARN 21062 --- [ocp-async-5,78c31ce9d6fe69db,631257e68498d57c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:31.166 INFO 21062 --- [ocp-async-1,cc36e2ffe2a797b4,ff8739afb89d1529] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:31.185 INFO 21062 --- [ocp-schedules-10,743eed75c93c4f8b,d3b042127b846637] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:14:31.269 INFO 21062 --- [ocp-schedules-29,5fdf1c2bc6acfeab,a537c9a593a8c241] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:14:31.275 INFO 21062 --- [ocp-schedules-29,5fdf1c2bc6acfeab,a537c9a593a8c241] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:14:31.275 INFO 21062 --- [ocp-schedules-29,5fdf1c2bc6acfeab,a537c9a593a8c241] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:14:31.305 INFO 21062 --- [ocp-schedules-23,da170ccb1e29dbb1,a9d5b779f3ff4110] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:14:31.305 INFO 21062 --- [ocp-schedules-23,da170ccb1e29dbb1,a9d5b779f3ff4110] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:14:31.331 INFO 21062 --- [ocp-schedules-23,da170ccb1e29dbb1,a9d5b779f3ff4110] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:14:31.331 INFO 21062 --- [ocp-schedules-23,da170ccb1e29dbb1,a9d5b779f3ff4110] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:14:31.332 INFO 21062 --- [ocp-schedules-6,7c36660b68281002,7b0646110e587afa] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:14:31.509 INFO 21062 --- [ocp-schedules-30,79db0eb115cbf5e4,0aa11312129b8dae] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:14:31.509+08:00, clusters=[1] 2024-11-04 14:14:31.513 INFO 21062 --- [ocp-schedules-30,79db0eb115cbf5e4,0aa11312129b8dae] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:14:31.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:31.580 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:31.638 INFO 21062 --- [ocp-schedules-15,2ce3f9eaed3ff0e6,b508a22fe59d60d8] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:14:31.643 INFO 21062 --- [ocp-schedules-15,2ce3f9eaed3ff0e6,b508a22fe59d60d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:31.661 INFO 21062 --- [ocp-schedules-24,cdbe3b48ea667b3f,cefc30815e048f38] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:14:31.661+08:00, clusters=[1] 2024-11-04 14:14:31.667 INFO 21062 --- [ocp-schedules-24,cdbe3b48ea667b3f,cefc30815e048f38] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:14:31.699 ERROR 21062 --- [ocp-schedules-15,2ce3f9eaed3ff0e6,b508a22fe59d60d8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:14:31.699 INFO 21062 --- [ocp-schedules-15,2ce3f9eaed3ff0e6,b508a22fe59d60d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:14:32.499 INFO 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:14:32.500 INFO 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:14:32.500 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:14:32.500 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:14:32.508 INFO 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:14:32.509 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:14:32.509 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:14:32.510 WARN 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:14:32.510 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:14:32.510 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:14:32.511 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:14:32.511 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:14:32.511 INFO 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:14:32.512 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:14:32.512 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:14:32.512 WARN 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:14:32.520 INFO 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:14:32.520 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:14:32.520 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:14:32.532 ERROR 21062 --- [ocp-schedules-27,c8c2119d25df4ca6,d4bed549ffb9b6e3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:14:32.983 INFO 21062 --- [prometheus-minute-schedule-1,0f8ac87d52141ae7,dc5ba2bee5f03dfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:33.129 WARN 21062 --- [ocp-async-2,4514acbedd0f0fc8,6239e01896033a26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:33.166 INFO 21062 --- [ocp-async-7,8cc815876d33ccc1,dc3a2498b2db797f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:34.580 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:34.583 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:35.129 WARN 21062 --- [ocp-async-8,5eaab6c1e7717b23,9204a25523443257] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:35.166 INFO 21062 --- [ocp-async-10,02e7ae9160381f30,bc7cf92a5c15687c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:35.502 INFO 21062 --- [ocp-schedules-12,cb289977817262d1,2d4db605b0032349] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:14:35.503 INFO 21062 --- [ocp-schedules-12,c18eabf2b918c78d,b43fe9df52459917] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:14:35.506 INFO 21062 --- [ocp-schedules-12,c18eabf2b918c78d,b43fe9df52459917] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:35.536 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:14:35.536 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:14:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b12e3c276a91b0a3,11d22b220535a789] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:37.129 WARN 21062 --- [ocp-async-4,8a0a1ffef58b8321,715075d00b4d35c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:37.166 INFO 21062 --- [ocp-async-3,c8f852a5e10e9000,1d442f310b8b41fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:37.583 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:37.584 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:37.923 INFO 21062 --- [ocp-schedules-6,86c216a1af3fb266,305ee4d43bafa751] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:14:37.927 INFO 21062 --- [ocp-schedules-6,86c216a1af3fb266,305ee4d43bafa751] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:38.013 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:14:38.015 ERROR 21062 --- [ocp-schedules-6,86c216a1af3fb266,305ee4d43bafa751] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:14:38.015 INFO 21062 --- [ocp-schedules-6,86c216a1af3fb266,305ee4d43bafa751] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:14:38.504 INFO 21062 --- [ocp-schedules-22,321ef33244ec674f,24e85c2293625492] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:14:38.504 INFO 21062 --- [ocp-schedules-1,c00b0f4025eddb7e,d64b2201b4f19339] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:14:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:14:38.589737+08:00, lastProcessTime=2024-11-04T14:14:28.588884+08:00 2024-11-04 14:14:38.649 INFO 21062 --- [ocp-async-10,e5ce0ff5c8487d6f,699d6e6b2f775a17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:14:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:14:38.589737+08:00 2024-11-04 14:14:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e53dbed815d3a007,983d4746dc7d6c5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:39.130 WARN 21062 --- [ocp-async-2,431f0634cf3b4578,275bb1754ad126b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:39.167 INFO 21062 --- [ocp-async-8,ef7a540b46220b90,808661a161cc457b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:14:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:14:40.321 INFO 21062 --- [ocp-schedules-19,76e77d50991c0ae3,764fb2ab3ef41e50] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:14:40.584 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:40.585 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:41.130 WARN 21062 --- [ocp-async-8,5c342b0824730f39,21cfcaa1bef2541f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:41.167 INFO 21062 --- [ocp-async-4,b847a2d4a0c1b6d2,b2e962d8a86c606e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:41.224 INFO 21062 --- [ocp-schedules-14,04cc01534cbb7908,798f0a7f211c2701] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:14:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:14:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:14:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:14:41.983 INFO 21062 --- [prometheus-minute-schedule-1,ac386f19d2e788b2,9357056a31665d69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:42.110 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:14:42.110 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:14:42.112 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.112 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:14:42.113 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.114 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.114 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.115 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:14:42.130 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.131 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.131 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.132 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:14:42.145 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.145 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.146 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.146 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:14:42.161 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.161 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.162 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.162 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:14:42.175 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.175 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.176 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.177 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:14:42.183 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.184 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.184 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.184 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:14:42.189 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.189 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.190 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.190 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:14:42.194 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.194 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.195 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.195 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:14:42.201 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.202 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.203 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.203 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:14:42.209 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.209 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.210 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.210 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:14:42.216 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.217 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.217 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.217 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:14:42.226 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.227 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.228 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.228 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:14:42.243 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.244 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.244 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.245 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:14:42.251 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.251 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.252 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.252 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:14:42.258 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.258 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.259 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.259 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:14:42.266 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.267 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.267 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.268 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:14:42.273 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.273 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.274 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.274 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:14:42.278 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.279 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.279 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.279 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:14:42.284 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.284 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.284 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.285 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:14:42.289 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.290 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.290 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.290 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:14:42.294 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.294 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.295 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.295 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:14:42.300 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.300 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.301 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.301 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:14:42.306 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.306 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.307 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.307 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:14:42.312 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.312 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.313 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.313 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:14:42.317 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.318 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.319 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.319 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:14:42.323 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.323 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.323 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.324 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:14:42.328 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.328 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.329 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.329 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:14:42.334 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.334 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.335 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.335 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:14:42.339 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.340 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.340 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.341 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:14:42.345 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.345 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.346 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.346 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:14:42.350 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.351 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.351 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.351 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:14:42.356 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.356 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.357 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.357 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:14:42.362 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.362 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.363 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.363 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:14:42.368 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.368 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.369 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.369 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:14:42.377 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.377 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.378 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.378 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:14:42.384 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.384 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.385 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.385 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:14:42.391 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.391 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.392 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.392 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:14:42.397 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.397 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.401 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.401 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:14:42.405 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.405 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.406 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.406 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:14:42.409 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.410 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.410 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.411 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:14:42.414 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.414 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.415 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.415 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:14:42.419 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.420 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.420 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.421 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:14:42.430 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.430 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.431 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.431 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:14:42.452 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.452 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.453 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.453 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:14:42.460 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.460 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.461 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.461 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:14:42.466 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.467 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.467 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.468 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:14:42.474 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.474 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.475 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.475 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:14:42.480 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.480 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.481 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.481 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:14:42.485 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.485 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.486 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.486 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:14:42.490 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:14:42.490 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:14:42.491 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.492 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:14:42.493 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.493 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.493 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.494 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:14:42.499 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.499 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:14:42.500 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:14:42.500 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:14:42.504 INFO 21062 --- [ocp-schedules-18,a82674e7765195f1,12b3ddf7c1e80e6f] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:14:43.130 WARN 21062 --- [ocp-async-4,f8b5141650dc6e52,8fabb7d5999e7c48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:43.167 INFO 21062 --- [ocp-async-1,b53919a88386477e,594a58e3427f6c69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:43.585 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:43.586 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f4125a37784bf240,3a11f1b5186184df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:45.130 WARN 21062 --- [ocp-async-1,83f235233cb97dc8,83940f7fbd635ad3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:45.167 INFO 21062 --- [ocp-async-7,38ce70b5d08ab33c,3fcb1076d0957098] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:46.586 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:46.587 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:47.130 WARN 21062 --- [ocp-async-7,0c6d375680e0fa9d,b4ddb8f61f7cf4e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:47.167 INFO 21062 --- [ocp-async-6,7bb60ce70fdd4e49,9326e7c3c259618f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:47.983 INFO 21062 --- [prometheus-minute-schedule-1,092f034c7950296d,08029dc59a2b814a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:14:48.590442+08:00, lastProcessTime=2024-11-04T14:14:38.589737+08:00 2024-11-04 14:14:48.649 INFO 21062 --- [ocp-async-9,54c06b74f4f043d9,967e7fe0bc877d6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:14:48.784 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:14:48.590442+08:00 2024-11-04 14:14:49.130 WARN 21062 --- [ocp-async-4,3ce30c816b5983ac,2a7dfe6f2bd1b68f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:49.167 INFO 21062 --- [ocp-async-1,1ffc1291ffa3b88c,5b93d104c2ac6d3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:49.587 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:49.588 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:50.325 INFO 21062 --- [ocp-schedules-27,4b3a3b422568c385,9fafedfba203f96d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:14:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4a0c96800ac65ea7,433353b737a35987] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:51.131 WARN 21062 --- [ocp-async-1,5d5f39cf8bfc2439,348f5d6e92ab43c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:51.167 INFO 21062 --- [ocp-async-7,67f5c188384e2cc7,53c2837a01b728b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:51.250 INFO 21062 --- [ocp-schedules-16,baddac9465b28946,b496fa90121ed3f7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:14:51.277 INFO 21062 --- [ocp-schedules-13,6247028c44605b98,16302f84e27b9a28] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:14:51.283 INFO 21062 --- [ocp-schedules-13,6247028c44605b98,16302f84e27b9a28] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:14:51.283 INFO 21062 --- [ocp-schedules-13,6247028c44605b98,16302f84e27b9a28] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:14:52.588 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:52.589 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:53.131 WARN 21062 --- [ocp-async-7,844b551c27fcd66e,151c70d06a424c29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:53.168 INFO 21062 --- [ocp-async-6,c84db5511e3ba8ec,0b98d815c76c9bda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:53.983 INFO 21062 --- [prometheus-minute-schedule-1,7130e5b93b354aa5,69bce3714e066a76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:55.131 WARN 21062 --- [ocp-async-6,78597c7bcbbd4710,291cbe1b70766b95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:55.168 INFO 21062 --- [ocp-async-3,cc72a8f5c939b6e3,8908c6b8d0889a61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:55.590 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:55.590 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:56.983 INFO 21062 --- [prometheus-minute-schedule-1,51b00edc0d9a0c08,88939155a8fa6c3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:14:57.131 WARN 21062 --- [ocp-async-3,6ae9c0074d4ceea8,aa02f6754e04cb2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:57.168 INFO 21062 --- [ocp-async-9,40cb03ee800b4f6a,5d2e2f3c9b9e12c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:58.500 INFO 21062 --- [ocp-schedules-6,ef8d14fe3361c942,de0e321bf6f6e609] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:14:58.500 INFO 21062 --- [ocp-schedules-27,945a6d687bf28251,f644ad0f2ccc9a0a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:14:58.503 INFO 21062 --- [ocp-schedules-24,f145bd916425fb4b,546ab54363bd5773] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:14:58.503 INFO 21062 --- [ocp-schedules-6,ef8d14fe3361c942,de0e321bf6f6e609] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:58.504 INFO 21062 --- [ocp-schedules-27,945a6d687bf28251,f644ad0f2ccc9a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:14:58.504 INFO 21062 --- [ocp-schedules-17,50df3098df9919a1,e83a49ac779d378d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:14:58.510 INFO 21062 --- [ocp-schedules-28,7d076ff2a07b96b0,c14c22cf5be328be] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:14:58.510 INFO 21062 --- [ocp-schedules-20,c11c63d50cbca0c2,57c6307fa695ba9a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:14:58.510 INFO 21062 --- [ocp-schedules-28,7d076ff2a07b96b0,c14c22cf5be328be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:14:58.510 INFO 21062 --- [ocp-schedules-20,c11c63d50cbca0c2,57c6307fa695ba9a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:14:58.511 INFO 21062 --- [ocp-schedules-18,3249545fb6a32ab7,2862af846630672b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:14:58.511 INFO 21062 --- [ocp-schedules-18,3249545fb6a32ab7,2862af846630672b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:14:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:14:58.589297+08:00, lastProcessTime=2024-11-04T14:14:48.590442+08:00 2024-11-04 14:14:58.590 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:14:58.591 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:14:58.649 INFO 21062 --- [ocp-async-4,433e40e3929e5423,16f675520b810597] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:14:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:14:58.589297+08:00 2024-11-04 14:14:58.975 INFO 21062 --- [ocp-schedules-26,04ce286ad3bd52f3,60f51080e7882b4e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:14:58.978 INFO 21062 --- [ocp-schedules-26,04ce286ad3bd52f3,60f51080e7882b4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:14:58.979 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:14:59.116 INFO 21062 --- [ocp-schedules-16,c7bd0e71e0ba4d71,4767f4dfe5be5301] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:14:59.131 WARN 21062 --- [ocp-async-7,d19644cb4422dfa6,a06c51b58c07f3ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:14:59.168 INFO 21062 --- [ocp-async-6,7bd8afc6dfe07408,9b285743c7d023e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:14:59.983 INFO 21062 --- [prometheus-minute-schedule-1,5a129b0677b421bb,10bdf0f7fe01a603] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700900, interval=60 2024-11-04 14:15:00.328 INFO 21062 --- [ocp-schedules-4,e143aaaac8a50eb6,21e26a2532ecd79a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:15:00.783 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,dece870af840c15e] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:10:01.0, nextRun=2024-11-04 14:15:00.0, currentDate=Mon Nov 04 14:15:00 CST 2024 2024-11-04 14:15:00.783 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,dece870af840c15e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:10:01.0, nextRunTime=2024-11-04 14:15:00.0} 2024-11-04 14:15:00.810 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,dece870af840c15e] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:15:00.815 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,dece870af840c15e] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:15:00.815 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,dece870af840c15e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:15:01.132 WARN 21062 --- [ocp-async-6,bcd0f076a441bc80,c58cb7fc7d2b9d3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:01.168 INFO 21062 --- [ocp-async-5,524d44c5b44452e0,c2e58a089b26d176] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:01.293 INFO 21062 --- [ocp-schedules-3,574a2b493f2945a5,39b9df22dd266143] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:15:01.591 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:01.592 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4ed53da23a53b5cd,d891307aeb53d6d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:03.132 WARN 21062 --- [ocp-async-5,425a97eb97a69079,f928fccedb1fc1f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:03.169 INFO 21062 --- [ocp-async-2,7361621325944843,da85ddd18d52bc73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:04.592 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:04.594 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:05.132 WARN 21062 --- [ocp-async-2,4471d83649f3d603,b93168e608eeaf09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:05.169 INFO 21062 --- [ocp-async-8,68a2e91e913f24e1,dcc2365c82fc4c70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:05.536 INFO 21062 --- [ocp-schedules-12,c18eabf2b918c78d,b43fe9df52459917] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:15:05.537 INFO 21062 --- [ocp-schedules-12,9bfa8adbb327d0ed,02f696423e130a55] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:15:05.540 INFO 21062 --- [ocp-schedules-12,9bfa8adbb327d0ed,02f696423e130a55] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:15:05.571 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:15:05.571 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:15:05.983 INFO 21062 --- [prometheus-minute-schedule-1,87cd2a2946fab54e,561508ab50ba10f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:07.132 WARN 21062 --- [ocp-async-8,ae63afff8523c18d,72505e6db6de62b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:07.169 INFO 21062 --- [ocp-async-4,72869347f7afb667,b53aaec2581a7298] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:07.594 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:07.595 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:08.504 INFO 21062 --- [ocp-schedules-27,945a6d687bf28251,f644ad0f2ccc9a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:15:08.505 INFO 21062 --- [ocp-schedules-6,ef8d14fe3361c942,de0e321bf6f6e609] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:15:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:15:08.589650+08:00, lastProcessTime=2024-11-04T14:14:58.589297+08:00 2024-11-04 14:15:08.649 INFO 21062 --- [ocp-async-7,7d1302db3eed0baa,e9913a2719892d07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:15:08.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:15:08.589650+08:00 2024-11-04 14:15:08.979 INFO 21062 --- [ocp-schedules-26,04ce286ad3bd52f3,60f51080e7882b4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:15:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d9af12691a5d3f02,3590ce8a084cef8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:09.132 WARN 21062 --- [ocp-async-5,31ff52d931fb2544,38d4f112379e8f77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:09.169 INFO 21062 --- [ocp-async-9,78f08d4a37762816,316d4ec3c1a68255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:10.002 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:15:10.004 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:15:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:15:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:15:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:15:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:15:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:15:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:15:10.332 INFO 21062 --- [ocp-schedules-19,1ad4f147c61b0b4c,037e30cd6d401703] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:15:10.595 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:10.596 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:11.132 WARN 21062 --- [ocp-async-2,f6c05bb213b58605,4cf2685001339f22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:11.169 INFO 21062 --- [ocp-async-10,e433b87b3d5470c5,2fb179355dd8189e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:11.285 INFO 21062 --- [ocp-schedules-22,07b04f09db19dbe1,9ce669ab149f4e26] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:15:11.291 INFO 21062 --- [ocp-schedules-22,07b04f09db19dbe1,9ce669ab149f4e26] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:15:11.291 INFO 21062 --- [ocp-schedules-22,07b04f09db19dbe1,9ce669ab149f4e26] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:15:11.320 INFO 21062 --- [ocp-schedules-28,d09cf65a72e88d0b,60f0e067a5dc38e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:15:11.513 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700840, nodeCount=61, affectRows=61 2024-11-04 14:15:11.514 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700840, nodeCount=71, affectRows=71 2024-11-04 14:15:11.514 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700840, nodeCount=61, affectRows=61 2024-11-04 14:15:11.514 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700840, nodeCount=61, validCount=61 2024-11-04 14:15:11.514 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700840, nodeCount=71, validCount=71 2024-11-04 14:15:11.514 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700840, nodeCount=60, validCount=60 2024-11-04 14:15:11.514 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700840, nodeCount=101, affectRows=101 2024-11-04 14:15:11.515 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700840, nodeCount=99, validCount=99 2024-11-04 14:15:11.516 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700840, nodeCount=102, affectRows=102 2024-11-04 14:15:11.517 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700840, nodeCount=102, validCount=102 2024-11-04 14:15:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b6e0445e565a9e0d,0d8c20eadc415a0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:13.133 WARN 21062 --- [ocp-async-8,fc5aba973bc13e07,ce11afdd9d682cd9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:13.170 INFO 21062 --- [ocp-async-4,4a4d885fc9d24d55,d71df055e2cf46cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:13.596 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:13.597 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:13.984 INFO 21062 --- [prometheus-cache-monitor1,25f3f2d6b3347eb8,c4d7fccd9714b801] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:15:13.984 INFO 21062 --- [prometheus-cache-monitor1,25f3f2d6b3347eb8,c4d7fccd9714b801] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:15:14.974 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:11:14.955, end=2024-11-04T14:13:14.971} 2024-11-04 14:15:14.978 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:15:14.978 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:15:14.983 INFO 21062 --- [prometheus-minute-schedule-1,bb0e1ee1311f2c44,7e6408753d311ec0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:15.133 WARN 21062 --- [ocp-async-3,abd808ef347370b3,b7ceffaa73632713] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:15.170 INFO 21062 --- [ocp-async-1,7539d74d65f47399,96ae7895d52177f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:16.597 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:16.597 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:17.133 WARN 21062 --- [ocp-async-9,f9d1c51798498027,62c2400c3828ad84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:17.170 INFO 21062 --- [ocp-async-7,183d471422b48f0e,3fa6c14a5b53678c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8231cb27f994a680,e6abbe77f8986274] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:15:18.589848+08:00, lastProcessTime=2024-11-04T14:15:08.589650+08:00 2024-11-04 14:15:18.650 INFO 21062 --- [ocp-async-5,1bfbd2b2e5d0a585,04bde9e574aad4b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:15:18.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:15:18.589848+08:00 2024-11-04 14:15:19.133 WARN 21062 --- [ocp-async-8,9cf8c14c7dbd5611,523289a82714545e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:19.170 INFO 21062 --- [ocp-async-4,156b8b0651df5377,bbb757167ff0249d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:19.597 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:19.598 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:20.335 INFO 21062 --- [ocp-schedules-29,3ade0b64312c06b2,8f2278e1d621618a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:15:20.983 INFO 21062 --- [prometheus-minute-schedule-1,69dea703ed4204b6,5da50a78d66d8c1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:21.133 WARN 21062 --- [ocp-async-3,f605e8a484881ab0,6c017f15eb0078ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:21.170 INFO 21062 --- [ocp-async-1,f965503b8f3bfa07,643de82d437d1695] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:21.365 INFO 21062 --- [ocp-schedules-17,bc1a9d47de1daea3,e2ee447101a15d4c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:15:22.598 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:22.599 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:23.134 WARN 21062 --- [ocp-async-9,60e9210107f19184,435bb0b54b955833] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:23.170 INFO 21062 --- [ocp-async-7,a0c0e9661fcdf836,35c7a3afae19699c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:23.983 INFO 21062 --- [prometheus-minute-schedule-1,3b29fba204bc2054,045ba7b18dd6246e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:25.134 WARN 21062 --- [ocp-async-10,3efe0176d6c877f4,f39d817d7eea2334] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:25.171 INFO 21062 --- [ocp-async-6,cef24d923500e37e,42ef0548e234ed02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:25.599 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:25.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9a9929a379ac0a0e,b55e0b52405102ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:26.984 INFO 21062 --- [prometheus-hour-schedule-1,1db14006e630eed9,1f0b584900e393cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:15:27.134 WARN 21062 --- [ocp-async-4,2fd955803496a7ac,d084e0d97de477fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:27.171 INFO 21062 --- [ocp-async-5,70741cefe62c5932,7d4c55d49a3c64f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:28.498 INFO 21062 --- [ocp-schedules-17,4e543e089b29dafe,86e3ee63d5942ca9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:15:28.499 INFO 21062 --- [ocp-schedules-8,ef0b6ac372124f5f,4d47ba5179505694] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:15:28.500 INFO 21062 --- [ocp-schedules-1,5aedd7385be1edbb,92f17bfe966677db] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:15:28.500 INFO 21062 --- [ocp-schedules-22,a09d9d74e63f2037,09a018eb67999c80] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:15:28.502 INFO 21062 --- [ocp-schedules-29,37e590f7259cf152,1c17597ff7c24dec] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:15:28.503 INFO 21062 --- [ocp-schedules-17,4e543e089b29dafe,86e3ee63d5942ca9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:15:28.504 INFO 21062 --- [ocp-schedules-22,a09d9d74e63f2037,09a018eb67999c80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:15:28.504 INFO 21062 --- [ocp-schedules-1,5aedd7385be1edbb,92f17bfe966677db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:15:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:15:28.505 INFO 21062 --- [ocp-schedules-15,a1e226b9dc5af29e,dd66b21187204244] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:15:28.506 INFO 21062 --- [ocp-schedules-8,ef0b6ac372124f5f,4d47ba5179505694] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:15:28.511 INFO 21062 --- [ocp-schedules-3,ed618e86a8ce4eed,5a9a25d17a1f6d90] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:15:28.511 INFO 21062 --- [ocp-schedules-14,ca819487b2723c82,557b86bdf5cb26ae] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:15:28.511 INFO 21062 --- [ocp-schedules-25,b41cd3d656c21f63,32e02f4b280a851f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:15:28.511 INFO 21062 --- [ocp-schedules-3,ed618e86a8ce4eed,5a9a25d17a1f6d90] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:15:28.511 INFO 21062 --- [ocp-schedules-14,ca819487b2723c82,557b86bdf5cb26ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:15:28.511 INFO 21062 --- [ocp-schedules-25,b41cd3d656c21f63,32e02f4b280a851f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:15:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:15:28.590137+08:00, lastProcessTime=2024-11-04T14:15:18.589848+08:00 2024-11-04 14:15:28.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:28.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:28.624 INFO 21062 --- [ocp-schedules-16,2d4b0db7e71667a4,d55fc495fa97be9a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:15:28.625 INFO 21062 --- [ocp-schedules-16,2d4b0db7e71667a4,d55fc495fa97be9a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:15:28.650 INFO 21062 --- [ocp-async-4,0d2902b8f8a57efd,d952f0fae4544a63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:15:28.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:15:28.590137+08:00 2024-11-04 14:15:28.808 INFO 21062 --- [ocp-schedules-20,1786dd6612c527a4,e086671f99788696] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:15:29.057 INFO 21062 --- [ocp-schedules-8,9fb46e646e19a26f,f8c2bdd120d859a4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:15:29.057 INFO 21062 --- [ocp-schedules-8,9fb46e646e19a26f,f8c2bdd120d859a4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:15:29.119 INFO 21062 --- [ocp-schedules-29,391ce299829297d3,40af800facff64fb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:15:29.134 WARN 21062 --- [ocp-async-10,38baefa06cec0296,dcf7bee8f2d079ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:29.152 INFO 21062 --- [ocp-schedules-25,da7c421fbb0626dc,5083fad72ca5a843] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:15:29.157 INFO 21062 --- [ocp-schedules-25,da7c421fbb0626dc,5083fad72ca5a843] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:15:29.157 INFO 21062 --- [ocp-schedules-25,da7c421fbb0626dc,5083fad72ca5a843] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:15:29.171 INFO 21062 --- [ocp-async-8,a9eb741feeadf69a,489414164fd4f579] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:29.463 INFO 21062 --- [ocp-schedules-16,f6bf50acb1a78e99,7fecac2f3876ab96] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:15:29.463 INFO 21062 --- [ocp-schedules-16,f6bf50acb1a78e99,7fecac2f3876ab96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:15:29.464 INFO 21062 --- [ocp-schedules-16,f6bf50acb1a78e99,7fecac2f3876ab96] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:15:29.464 INFO 21062 --- [ocp-schedules-16,f6bf50acb1a78e99,7fecac2f3876ab96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:15:29.465 INFO 21062 --- [ocp-schedules-16,f6bf50acb1a78e99,7fecac2f3876ab96] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:15:29.465 INFO 21062 --- [ocp-schedules-16,f6bf50acb1a78e99,7fecac2f3876ab96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:15:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3d47862b087dd017,37b8c6e8198d00db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:15:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:15:30.338 INFO 21062 --- [ocp-schedules-20,47195accce2b0838,b6d7ad9933a3f846] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:15:31.134 WARN 21062 --- [ocp-async-6,6987c02f4eaa1f94,02831258b5603782] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:31.171 INFO 21062 --- [ocp-async-3,772ea8eb9a22182e,e5d9031841a33075] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:31.293 INFO 21062 --- [ocp-schedules-24,54664c6cb827e273,35d9606b8e302c57] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:15:31.300 INFO 21062 --- [ocp-schedules-24,54664c6cb827e273,35d9606b8e302c57] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:15:31.300 INFO 21062 --- [ocp-schedules-24,54664c6cb827e273,35d9606b8e302c57] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:15:31.331 INFO 21062 --- [ocp-schedules-26,b37a911a7bfc72da,0af2d99ac9bef38c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:15:31.331 INFO 21062 --- [ocp-schedules-26,b37a911a7bfc72da,0af2d99ac9bef38c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:15:31.358 INFO 21062 --- [ocp-schedules-26,b37a911a7bfc72da,0af2d99ac9bef38c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:15:31.358 INFO 21062 --- [ocp-schedules-26,b37a911a7bfc72da,0af2d99ac9bef38c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:15:31.359 INFO 21062 --- [ocp-schedules-29,3950b6d9aaa2d16d,ccb36be041be6f9f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:15:31.398 INFO 21062 --- [ocp-schedules-4,822d0456dc63ef63,225702a23f886814] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:15:31.539 INFO 21062 --- [ocp-schedules-8,125a8fc5c0068bad,b6cb4ece934f78f3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:15:31.539+08:00, clusters=[1] 2024-11-04 14:15:31.541 INFO 21062 --- [ocp-schedules-8,125a8fc5c0068bad,b6cb4ece934f78f3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:15:31.601 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:31.601 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:31.692 INFO 21062 --- [ocp-schedules-17,0f8c052ab786ba66,4952e2bcfd4e94b5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:15:31.692+08:00, clusters=[1] 2024-11-04 14:15:31.696 INFO 21062 --- [ocp-schedules-17,0f8c052ab786ba66,4952e2bcfd4e94b5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:15:32.541 INFO 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:15:32.541 INFO 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:15:32.541 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:15:32.541 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:15:32.551 INFO 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:15:32.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:15:32.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:15:32.553 WARN 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:15:32.554 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:15:32.554 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:15:32.554 INFO 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:15:32.554 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:15:32.554 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:15:32.555 WARN 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:15:32.556 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:15:32.556 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:15:32.563 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:15:32.563 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:15:32.563 INFO 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:15:32.577 ERROR 21062 --- [ocp-schedules-11,a14f27accb83bc19,dd266e103d579779] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:15:32.983 INFO 21062 --- [prometheus-minute-schedule-1,bcf3bdb7ffb128f4,526cb63af64fc549] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:33.134 WARN 21062 --- [ocp-async-5,5879686f1c729eb4,7fc2373225527864] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:33.171 INFO 21062 --- [ocp-async-9,781b145372436e79,4b2da8a7d115778a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:34.602 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:34.602 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:35.135 WARN 21062 --- [ocp-async-2,98bd4fe0f1b33f2b,6597fe947a9f3073] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:35.171 INFO 21062 --- [ocp-async-7,cadafbecbf94b1a8,8d0703abca3e54ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:35.572 INFO 21062 --- [ocp-schedules-12,9bfa8adbb327d0ed,02f696423e130a55] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:15:35.572 INFO 21062 --- [ocp-schedules-12,9ba347eaefbf1765,49a3bdda99e3960e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:15:35.576 INFO 21062 --- [ocp-schedules-12,9ba347eaefbf1765,49a3bdda99e3960e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:15:35.606 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:15:35.606 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:15:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c48feab9857c8e9e,159ce0b078e889fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:37.135 WARN 21062 --- [ocp-async-8,9adacf3e0b2c3cb3,b1ff1ac9d7229e65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:37.172 INFO 21062 --- [ocp-async-4,b9bcd9101dddd223,8bc7866146f7b964] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:37.197 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:15:37.197 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:15:37.205 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:15:37.205 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:15:37.205 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:15:37.205 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:15:37.210 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:15:37.210 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:15:37.210 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:15:37.210 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:15:37.246 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:15:37.254 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:15:37.254 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:15:37.254 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:15:37.254 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:15:37.254 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:15:37.262 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:15:37.602 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:37.603 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:37.825 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:15:38.015 INFO 21062 --- [ocp-schedules-15,8b56a6f71f4b710c,0ee70317ef70828e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:15:38.020 INFO 21062 --- [ocp-schedules-15,8b56a6f71f4b710c,0ee70317ef70828e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:15:38.104 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:15:38.111 ERROR 21062 --- [ocp-schedules-15,8b56a6f71f4b710c,0ee70317ef70828e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:15:38.112 INFO 21062 --- [ocp-schedules-15,8b56a6f71f4b710c,0ee70317ef70828e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:15:38.387 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.176926618S 2024-11-04 14:15:38.387 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:15:38.483 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:15:38.484 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:15:38.504 INFO 21062 --- [ocp-schedules-22,a09d9d74e63f2037,09a018eb67999c80] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:15:38.505 INFO 21062 --- [ocp-schedules-1,5aedd7385be1edbb,92f17bfe966677db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:15:38.512 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:15:38.571 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:15:38.571 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:15:38.571 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:15:38.571 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:15:38.571 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:15:38.572 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:15:38.582 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:15:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:15:38.589693+08:00, lastProcessTime=2024-11-04T14:15:28.590137+08:00 2024-11-04 14:15:38.650 INFO 21062 --- [ocp-async-7,e9130861c2dab433,0c8cd4b1f1bec312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:15:38.735 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] o.s.boot.SpringApplication : Started application in 1.53 seconds (JVM running for 6352.414) 2024-11-04 14:15:38.735 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.530983287S 2024-11-04 14:15:38.752 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:15:38.589693+08:00 2024-11-04 14:15:38.983 INFO 21062 --- [prometheus-minute-schedule-1,818ade82e2d233e3,29cbc616359d27c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:39.099 INFO 21062 --- [ocp-schedules-13,312f87780ce4064e,237b3ae048c77597] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:15:39.135 WARN 21062 --- [ocp-async-5,60628797f89a8e92,8fb7ab73edd86e91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:39.172 INFO 21062 --- [ocp-async-9,0d9a1ca90dafa5f5,9b150108ebab2083] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:15:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:15:40.341 INFO 21062 --- [ocp-schedules-18,90a7dbf28803df78,736c5b0af99eff35] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:15:40.604 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:40.604 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:41.135 WARN 21062 --- [ocp-async-9,a383f277220b5435,1c50ce617946a8d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:41.172 INFO 21062 --- [ocp-async-8,e382c1bb8a39d03e,79c33fb91e94dec2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:41.442 INFO 21062 --- [ocp-schedules-4,23977975472c840d,2c10f3e657457868] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:15:41.983 INFO 21062 --- [prometheus-minute-schedule-1,1d39d148911a642f,04b9814770aef69b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:43.135 WARN 21062 --- [ocp-async-8,18ee314e9a46c511,eb7bf030a20a0946] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:43.172 INFO 21062 --- [ocp-async-3,ab21f01a0055547b,b1ee99433ace5859] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:43.604 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:43.605 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:44.983 INFO 21062 --- [prometheus-minute-schedule-1,729835fece9e7d65,f3217574abf0970a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:44.988 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9cbdd94f9773e09a,9c79f5c82df74857] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.130.47.17, traceId 9cbdd94f9773e09a] 2024-11-04 14:15:44.993 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9cbdd94f9773e09a,9c79f5c82df74857] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.130.47.17, traceId=9cbdd94f9773e09a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2024-11-04 14:15:45.136 WARN 21062 --- [ocp-async-3,df7a199ce3cd5e73,b39ec368a56b57c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:45.158 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9cbdd94f9773e09a,9c79f5c82df74857] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 13, current state: FAILED, new state: READY 2024-11-04 14:15:45.159 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9cbdd94f9773e09a,9c79f5c82df74857] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 42, current state: FAILED, new state: READY 2024-11-04 14:15:45.172 INFO 21062 --- [ocp-async-2,37a92e3e85eb850c,de3ef70c93ef7840] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:45.179 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9cbdd94f9773e09a,9c79f5c82df74857] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.130.47.17, serverIp=null, startTime=2024-11-04T14:15:44.998+08:00, endTime=2024-11-04T14:15:45.179+08:00, role=null, aliUid=null) 2024-11-04 14:15:45.191 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9cbdd94f9773e09a,9c79f5c82df74857] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.130.47.17, traceId=9cbdd94f9773e09a, duration=210 ms] 2024-11-04 14:15:45.480 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,8255aa79595d4ad0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: RUNNING 2024-11-04 14:15:45.486 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,8255aa79595d4ad0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: RUNNING 2024-11-04 14:15:45.500 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=13, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:15:45.472+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=13}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:15:45.502 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:15:45.502 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=42, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:15:45.484+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=42}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:15:45.506 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:15:45.509 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:15:45.512 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:15:45.514 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:15:45.520 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:15:45.533 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:45.537 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:45.539 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.542 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.582 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:45.583 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:45.587 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.589 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.598 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,03b2488babf85907,ec903ae2e043be76] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=03b2488babf85907, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:45.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,03b2488babf85907,ec903ae2e043be76] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=03b2488babf85907, duration=30 ms] 2024-11-04 14:15:45.630 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:15:45.632 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:15:45.659 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.662 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.664 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.667 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.707 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:45.710 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:45.712 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.715 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.755 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:15:45.758 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:15:45.781 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.783 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.792 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.795 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.827 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:45.832 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.834 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:45.840 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.875 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:15:45.882 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:15:45.900 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.903 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.905 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:15:45.914 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.917 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.919 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:15:45.943 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:45.948 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.958 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:45.963 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:45.991 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 14:15:46.010 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 14:15:46.035 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.038 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.041 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:15:46.044 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:15:46.046 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:15:46.055 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.058 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.060 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:15:46.062 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:15:46.065 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:15:46.083 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:46.088 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.102 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:46.107 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.131 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 14:15:46.154 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 14:15:46.175 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.178 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.181 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:15:46.183 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:15:46.185 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:15:46.188 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:15:46.190 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:15:46.196 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:15:46.199 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.199 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:15:46.201 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.206 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:15:46.208 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:15:46.210 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:15:46.212 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:15:46.205 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:15:46.215 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:15:46.218 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:15:46.221 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:15:46.222 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:15:46.225 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:15:46.230 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:15:46.233 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:15:46.236 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:15:46.243 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:46.249 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.253 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:46.258 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.295 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:46.300 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.302 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:46.308 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.343 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:15:46.354 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:15:46.367 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.370 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.380 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.383 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.411 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:46.417 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.422 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:46.427 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.459 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:15:46.474 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:15:46.482 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.485 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.498 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.503 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.507 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:15:46.522 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:15:46.527 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:46.532 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.546 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:46.551 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.579 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12 2024-11-04 14:15:46.598 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11 2024-11-04 14:15:46.601 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,444a43cf823e46e2,ccd4862290ef87fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.12, traceId=444a43cf823e46e2, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:15:46.605 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,444a43cf823e46e2,ccd4862290ef87fe] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:15:46.605 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,444a43cf823e46e2,ccd4862290ef87fe] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:15:46.605 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:46.605 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,444a43cf823e46e2,ccd4862290ef87fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.12, traceId=444a43cf823e46e2, duration=7 ms] 2024-11-04 14:15:46.606 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:46.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0e799b856d4660c4,84e764855dd92160] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.11, traceId=0e799b856d4660c4, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:15:46.624 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0e799b856d4660c4,84e764855dd92160] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:15:46.624 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0e799b856d4660c4,84e764855dd92160] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:15:46.624 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,0e799b856d4660c4,84e764855dd92160] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.11, traceId=0e799b856d4660c4, duration=7 ms] 2024-11-04 14:15:46.705 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.709 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.720 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.723 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.751 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:46.757 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.766 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:46.771 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.803 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12 2024-11-04 14:15:46.814 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11 2024-11-04 14:15:46.864 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,202d13669aa967fa,803f91319948831b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=202d13669aa967fa, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:15:46.873 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,202d13669aa967fa,803f91319948831b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=202d13669aa967fa, duration=16 ms] 2024-11-04 14:15:46.949 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.953 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.955 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:15:46.964 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.967 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:46.969 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:15:46.991 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:46.997 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.010 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:47.015 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.043 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/4cdf314703f641d7 on 10.46.128.12 2024-11-04 14:15:47.049 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/4cdf314703f641d7 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/4cdf314703f641d7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.054 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/4cdf314703f641d7 on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/4cdf314703f641d7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.062 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d0883d062ac24505 on 10.46.128.11 2024-11-04 14:15:47.068 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d0883d062ac24505 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d0883d062ac24505, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.070 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d0883d062ac24505 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d0883d062ac24505, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.091 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:47.096 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.110 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:47.115 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.136 WARN 21062 --- [ocp-async-2,c556b2ef64dd136d,40e2eae3bd050e8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:47.139 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/4cdf314703f641d7/cpio && chmod +x /tmp/4cdf314703f641d7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/4cdf314703f641d7/cpio on 10.46.128.12 2024-11-04 14:15:47.146 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/4cdf314703f641d7/cpio && chmod +x /tmp/4cdf314703f641d7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/4cdf314703f641d7/cpio on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/4cdf314703f641d7/cpio && chmod +x /tmp/4cdf314703f641d7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/4cdf314703f641d7/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.149 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/4cdf314703f641d7/cpio && chmod +x /tmp/4cdf314703f641d7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/4cdf314703f641d7/cpio on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/4cdf314703f641d7/cpio && chmod +x /tmp/4cdf314703f641d7/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/4cdf314703f641d7/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.162 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d0883d062ac24505/cpio && chmod +x /tmp/d0883d062ac24505/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d0883d062ac24505/cpio on 10.46.128.11 2024-11-04 14:15:47.169 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d0883d062ac24505/cpio && chmod +x /tmp/d0883d062ac24505/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d0883d062ac24505/cpio on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d0883d062ac24505/cpio && chmod +x /tmp/d0883d062ac24505/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d0883d062ac24505/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.172 INFO 21062 --- [ocp-async-10,abf360fcd17730fc,d8f45de0f8436806] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:47.172 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d0883d062ac24505/cpio && chmod +x /tmp/d0883d062ac24505/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d0883d062ac24505/cpio on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d0883d062ac24505/cpio && chmod +x /tmp/d0883d062ac24505/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d0883d062ac24505/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.191 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:47.196 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.210 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:47.215 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.243 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/4cdf314703f641d7 on 10.46.128.12 2024-11-04 14:15:47.249 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/4cdf314703f641d7 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/4cdf314703f641d7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.252 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/4cdf314703f641d7 on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/4cdf314703f641d7, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.262 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d0883d062ac24505 on 10.46.128.11 2024-11-04 14:15:47.269 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d0883d062ac24505 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d0883d062ac24505, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.271 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d0883d062ac24505 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d0883d062ac24505, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.295 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:47.301 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.310 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:47.315 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.347 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/4cdf314703f641d7/install_package.sh && chmod +x /tmp/4cdf314703f641d7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/4cdf314703f641d7/install_package.sh on 10.46.128.12 2024-11-04 14:15:47.354 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/4cdf314703f641d7/install_package.sh && chmod +x /tmp/4cdf314703f641d7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/4cdf314703f641d7/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/4cdf314703f641d7/install_package.sh && chmod +x /tmp/4cdf314703f641d7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/4cdf314703f641d7/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.356 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/4cdf314703f641d7/install_package.sh && chmod +x /tmp/4cdf314703f641d7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/4cdf314703f641d7/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/4cdf314703f641d7/install_package.sh && chmod +x /tmp/4cdf314703f641d7/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/4cdf314703f641d7/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.362 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d0883d062ac24505/install_package.sh && chmod +x /tmp/d0883d062ac24505/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d0883d062ac24505/install_package.sh on 10.46.128.11 2024-11-04 14:15:47.369 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d0883d062ac24505/install_package.sh && chmod +x /tmp/d0883d062ac24505/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d0883d062ac24505/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d0883d062ac24505/install_package.sh && chmod +x /tmp/d0883d062ac24505/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d0883d062ac24505/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.372 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d0883d062ac24505/install_package.sh && chmod +x /tmp/d0883d062ac24505/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d0883d062ac24505/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d0883d062ac24505/install_package.sh && chmod +x /tmp/d0883d062ac24505/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d0883d062ac24505/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.399 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:15:47.404 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.414 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:15:47.419 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:15:47.451 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/4cdf314703f641d7/install_package.sh on 10.46.128.12 2024-11-04 14:15:47.462 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/d0883d062ac24505/install_package.sh on 10.46.128.11 2024-11-04 14:15:47.757 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,c2bc3afeded6b4e4,b706deb6cb98e535] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c2bc3afeded6b4e4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:47.784 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,c2bc3afeded6b4e4,b706deb6cb98e535] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c2bc3afeded6b4e4, duration=35 ms] 2024-11-04 14:15:47.983 INFO 21062 --- [prometheus-minute-schedule-1,09b442641a033492,ffacf4842b4e7ecb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:48.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:15:48.589563+08:00, lastProcessTime=2024-11-04T14:15:38.589693+08:00 2024-11-04 14:15:48.650 INFO 21062 --- [ocp-async-5,4aa574352e643360,4b262a0b1a7fe3de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:15:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:15:48.589563+08:00 2024-11-04 14:15:49.136 WARN 21062 --- [ocp-async-8,acfd0af813dfd5d0,62fc7beedb6a5d28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:49.173 INFO 21062 --- [ocp-async-3,c984c1f7e62fed90,1f92c48cba7adc77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:49.604 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,67aef3135b1f003e,5a70c287ecaf99b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=67aef3135b1f003e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:49.606 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:49.607 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:49.630 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,67aef3135b1f003e,5a70c287ecaf99b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=67aef3135b1f003e, duration=36 ms] 2024-11-04 14:15:50.343 INFO 21062 --- [ocp-schedules-7,fa7da33b21f6dde6,74e4b3f5b9737322] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:15:50.983 INFO 21062 --- [prometheus-minute-schedule-1,709e921a33882c6b,9facd25b271c2119] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:51.121 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4a2f3677e2482916,58e841373299e759] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=4a2f3677e2482916, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:51.136 WARN 21062 --- [ocp-async-3,aa1e1b2b340fb3a8,291fc9906d3efcd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:51.149 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4a2f3677e2482916,58e841373299e759] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=4a2f3677e2482916, duration=36 ms] 2024-11-04 14:15:51.173 INFO 21062 --- [ocp-async-2,77330aba9551f32c,03062078abfa23c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:51.303 INFO 21062 --- [ocp-schedules-25,90a6421300754e78,df56397a49e284e2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:15:51.308 INFO 21062 --- [ocp-schedules-25,90a6421300754e78,df56397a49e284e2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:15:51.308 INFO 21062 --- [ocp-schedules-25,90a6421300754e78,df56397a49e284e2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:15:51.468 INFO 21062 --- [ocp-schedules-4,f25fa3531d84e19a,238290c6b486ead8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:15:52.607 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:52.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:52.638 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,91278c95177fd56f,80f4a7c3d77f2263] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=91278c95177fd56f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:52.664 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,91278c95177fd56f,80f4a7c3d77f2263] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=91278c95177fd56f, duration=33 ms] 2024-11-04 14:15:53.136 WARN 21062 --- [ocp-async-2,66629ec861e32abf,5ca85202925a95bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:53.173 INFO 21062 --- [ocp-async-10,3a33584c5054e375,0f824241e9163aca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:53.983 INFO 21062 --- [prometheus-minute-schedule-1,0ac380542fbc3c62,3f4b1ac4b0eacebe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:54.223 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,d39fa415958fbc5c,3e0d9bad44be9b27] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=d39fa415958fbc5c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:54.251 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,d39fa415958fbc5c,3e0d9bad44be9b27] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=d39fa415958fbc5c, duration=36 ms] 2024-11-04 14:15:55.137 WARN 21062 --- [ocp-async-10,f760c6510a26d133,4bbc5fc4a4a602ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:55.173 INFO 21062 --- [ocp-async-6,49d237c202cf6d08,1e8004a9db42d494] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:55.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:55.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:55.780 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,35e5018a1c793f57,0861d1578fb7d688] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=35e5018a1c793f57, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:55.806 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,35e5018a1c793f57,0861d1578fb7d688] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=35e5018a1c793f57, duration=33 ms] 2024-11-04 14:15:56.983 INFO 21062 --- [prometheus-minute-schedule-1,0e2af1a360c9ceab,8399760ea9abc36f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:15:57.137 WARN 21062 --- [ocp-async-6,087b6b407fe49c19,ef43d2f1a5661301] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:57.173 INFO 21062 --- [ocp-async-5,d4c38cc7934d6dcc,7c9441b087da4bed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:57.319 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,33b398aa3f78e5c6,bb5904d66fcf4ed7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=33b398aa3f78e5c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:57.344 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,33b398aa3f78e5c6,bb5904d66fcf4ed7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=33b398aa3f78e5c6, duration=31 ms] 2024-11-04 14:15:58.500 INFO 21062 --- [ocp-schedules-30,569a3c12247af71e,ded0763aa65a090c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:15:58.500 INFO 21062 --- [ocp-schedules-28,bab61441369ef4aa,dc80d2374253fceb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:15:58.502 INFO 21062 --- [ocp-schedules-25,b6f722ec89696cc2,6fd9794f563fab44] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:15:58.503 INFO 21062 --- [ocp-schedules-30,569a3c12247af71e,ded0763aa65a090c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:15:58.504 INFO 21062 --- [ocp-schedules-28,bab61441369ef4aa,dc80d2374253fceb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:15:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:15:58.504 INFO 21062 --- [ocp-schedules-19,4257e64d18dc2dbe,33b23403fdb15ba7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:15:58.511 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,582367de492e4593,a925a02e126c1afc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=582367de492e4593, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:15:58.513 INFO 21062 --- [ocp-schedules-16,b74241caaf34f127,6d0c285d7c4564be] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:15:58.513 INFO 21062 --- [ocp-schedules-16,b74241caaf34f127,6d0c285d7c4564be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:15:58.513 INFO 21062 --- [ocp-schedules-7,0786176970f10bcb,bdbe039ef5b5f0d6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:15:58.513 INFO 21062 --- [ocp-schedules-7,0786176970f10bcb,bdbe039ef5b5f0d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:15:58.514 INFO 21062 --- [ocp-schedules-17,dc983c212914f69d,622bdbd5bf315e11] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:15:58.514 INFO 21062 --- [ocp-schedules-17,dc983c212914f69d,622bdbd5bf315e11] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:15:58.522 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,582367de492e4593,a925a02e126c1afc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=582367de492e4593, duration=17 ms] 2024-11-04 14:15:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:15:58.591731+08:00, lastProcessTime=2024-11-04T14:15:48.589563+08:00 2024-11-04 14:15:58.609 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:15:58.609 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:15:58.651 INFO 21062 --- [ocp-async-8,698e67b7e08ebd24,5468e8ffb47ed1a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:15:58.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:15:58.591731+08:00 2024-11-04 14:15:59.104 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,699b5bedc19a8488,dff040e9386986f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=699b5bedc19a8488, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:15:59.122 INFO 21062 --- [ocp-schedules-22,d5b46bc83246f0e0,1b8c85f08ea4d2ea] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:15:59.129 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,699b5bedc19a8488,dff040e9386986f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=699b5bedc19a8488, duration=33 ms] 2024-11-04 14:15:59.137 WARN 21062 --- [ocp-async-2,280f0677b6306bc3,511b0130d1dca841] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:15:59.173 INFO 21062 --- [ocp-async-10,55f12e40f6768be2,f4116b3198329eba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:15:59.983 INFO 21062 --- [prometheus-minute-schedule-1,60f968f95f1e43ab,8ccda9d4d8903ddf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730700960, interval=60 2024-11-04 14:16:00.346 INFO 21062 --- [ocp-schedules-27,c7c7dd963be93b9e,ed0b0dcdc2938b47] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:16:00.691 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,30eb216838f131ff,f4a9007179de420a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=30eb216838f131ff, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:00.718 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,30eb216838f131ff,f4a9007179de420a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=30eb216838f131ff, duration=35 ms] 2024-11-04 14:16:01.137 WARN 21062 --- [ocp-async-10,95cabf16707eac8b,983ba4b893de50b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:01.174 INFO 21062 --- [ocp-async-4,7f72873f8e765e1f,01d4501623cda7fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:01.510 INFO 21062 --- [ocp-schedules-3,889bb7a619fedfe5,d3724ece1970a0a0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:16:01.609 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:01.610 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:02.174 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5707dd66def7d65a,06a287b92f4ca2d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=5707dd66def7d65a, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:16:02.180 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5707dd66def7d65a,06a287b92f4ca2d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=5707dd66def7d65a, duration=15 ms] 2024-11-04 14:16:02.372 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,a34221012b8e09fc,0865b6a3044fe074] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=a34221012b8e09fc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:02.399 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,a34221012b8e09fc,0865b6a3044fe074] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=a34221012b8e09fc, duration=33 ms] 2024-11-04 14:16:02.983 INFO 21062 --- [prometheus-minute-schedule-1,6b07a96665f84a04,21772750d490f235] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:03.137 WARN 21062 --- [ocp-async-4,99858d393a90dd18,d6db705bdbb28528] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:03.174 INFO 21062 --- [ocp-async-1,c74ceb543341bc6b,0391622a12a0f6eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:04.041 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,adc7e67f0ea6b77e,d860c2d192b48b98] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=adc7e67f0ea6b77e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:04.069 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,adc7e67f0ea6b77e,d860c2d192b48b98] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=adc7e67f0ea6b77e, duration=34 ms] 2024-11-04 14:16:04.610 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:04.611 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:05.137 WARN 21062 --- [ocp-async-1,a835c5439e5a6eb7,afbfdd76ce2e660c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:05.174 INFO 21062 --- [ocp-async-9,b3c880d32da78565,bb61aa80a4b6545b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:05.590 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,261a3cf2ac4ae568,a97531980af224c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=261a3cf2ac4ae568, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:05.606 INFO 21062 --- [ocp-schedules-12,9ba347eaefbf1765,49a3bdda99e3960e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:16:05.606 INFO 21062 --- [ocp-schedules-12,ab5f6fd8b6cb1cb4,dfa7e0e33360f20f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:16:05.610 INFO 21062 --- [ocp-schedules-12,ab5f6fd8b6cb1cb4,dfa7e0e33360f20f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:05.617 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,261a3cf2ac4ae568,a97531980af224c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=261a3cf2ac4ae568, duration=34 ms] 2024-11-04 14:16:05.638 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:16:05.638 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:16:05.983 INFO 21062 --- [prometheus-minute-schedule-1,b6b27b7e2ec5efd9,2ca2ab9093248f9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:07.128 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,2fb13617ee266a5d,d61b364b79a6c08e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=2fb13617ee266a5d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:07.138 WARN 21062 --- [ocp-async-9,a2e6ab6dd18f2a37,4353e7b0aa1fb16a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:07.154 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,2fb13617ee266a5d,d61b364b79a6c08e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=2fb13617ee266a5d, duration=33 ms] 2024-11-04 14:16:07.174 INFO 21062 --- [ocp-async-8,17822da521a6552c,e44f60e87b26c3af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:07.611 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:07.612 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:08.504 INFO 21062 --- [ocp-schedules-28,bab61441369ef4aa,dc80d2374253fceb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:16:08.505 INFO 21062 --- [ocp-schedules-30,569a3c12247af71e,ded0763aa65a090c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:16:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:16:08.591139+08:00, lastProcessTime=2024-11-04T14:15:58.591731+08:00 2024-11-04 14:16:08.651 INFO 21062 --- [ocp-async-2,9d138cab24717734,f4fc57eca3df732b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:16:08.666 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e9e8c90f7ab87f96,00a9f303a9b28e15] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e9e8c90f7ab87f96, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:08.692 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,e9e8c90f7ab87f96,00a9f303a9b28e15] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e9e8c90f7ab87f96, duration=32 ms] 2024-11-04 14:16:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:16:08.591139+08:00 2024-11-04 14:16:08.980 INFO 21062 --- [ocp-schedules-14,41ccd75b547e72b4,e4a5f7b20383bca2] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:16:08.983 INFO 21062 --- [prometheus-minute-schedule-1,142fefdd39f8812d,cf10f92011f79d76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:08.983 INFO 21062 --- [ocp-schedules-14,41ccd75b547e72b4,e4a5f7b20383bca2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:08.984 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:16:09.138 WARN 21062 --- [ocp-async-4,a795d47688139b39,4a21f832a6d69887] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:09.174 INFO 21062 --- [ocp-async-5,d2bfc7db6948b6f7,c16ede0804c5424e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:10.002 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:16:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:16:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:16:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:16:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:16:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:16:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:16:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:16:10.328 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,a5fa90e5d2c6f62e,d2f4031d63587f02] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=a5fa90e5d2c6f62e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:10.349 INFO 21062 --- [ocp-schedules-22,2e2e05f8d422952b,2e75b3626fec4486] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:16:10.355 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,a5fa90e5d2c6f62e,d2f4031d63587f02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=a5fa90e5d2c6f62e, duration=34 ms] 2024-11-04 14:16:10.612 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:10.613 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:11.138 WARN 21062 --- [ocp-async-1,62a37136f5398dfe,7f0390bb9b57bab2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:11.174 INFO 21062 --- [ocp-async-7,ceece1bef85fc18e,917f9ed229d6a637] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:11.311 INFO 21062 --- [ocp-schedules-20,0191bd98bb1f0581,3882f94aec1b68dc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:16:11.316 INFO 21062 --- [ocp-schedules-20,0191bd98bb1f0581,3882f94aec1b68dc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:16:11.316 INFO 21062 --- [ocp-schedules-20,0191bd98bb1f0581,3882f94aec1b68dc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:16:11.517 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700900, nodeCount=45, affectRows=45 2024-11-04 14:16:11.517 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700900, nodeCount=44, validCount=44 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700900, nodeCount=64, affectRows=64 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700900, nodeCount=71, affectRows=71 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700900, nodeCount=84, affectRows=84 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700900, nodeCount=80, affectRows=80 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700900, nodeCount=52, affectRows=52 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700900, nodeCount=64, validCount=64 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700900, nodeCount=69, validCount=69 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700900, nodeCount=80, validCount=80 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700900, nodeCount=52, validCount=52 2024-11-04 14:16:11.518 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700900, nodeCount=84, validCount=84 2024-11-04 14:16:11.537 INFO 21062 --- [ocp-schedules-30,f1cc328a5db8b9df,9a8d251d66a24c5d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:16:11.983 INFO 21062 --- [prometheus-minute-schedule-1,5ba4024e4f6286eb,396b40406a60a7b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:12.011 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,eaa5a2ddb5595afd,2bf2e97579daea27] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=eaa5a2ddb5595afd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:12.038 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,eaa5a2ddb5595afd,2bf2e97579daea27] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=eaa5a2ddb5595afd, duration=35 ms] 2024-11-04 14:16:13.138 WARN 21062 --- [ocp-async-9,5f2fc116ae584b3f,25728c9a2c3b70dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:13.175 INFO 21062 --- [ocp-async-8,776d4cdfa4c30b6e,06d212594fd530cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:13.581 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ceb89a2f0d11fb32,cb0eb4bbe9549196] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=ceb89a2f0d11fb32, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:13.608 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ceb89a2f0d11fb32,cb0eb4bbe9549196] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=ceb89a2f0d11fb32, duration=37 ms] 2024-11-04 14:16:13.613 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:13.614 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:13.985 INFO 21062 --- [prometheus-cache-monitor1,f1a15102290c8fa7,252fe09c4f24dd66] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:16:13.985 INFO 21062 --- [prometheus-cache-monitor1,f1a15102290c8fa7,252fe09c4f24dd66] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:16:14.983 INFO 21062 --- [prometheus-minute-schedule-1,27393c9bd613d091,6565520374a58b3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:15.138 WARN 21062 --- [ocp-async-6,36aa355d4e7a3dcb,675c396f7dc819bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:15.157 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,6297d477f96c9440,257bb9b1d5e818c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=6297d477f96c9440, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:15.175 INFO 21062 --- [ocp-async-3,eee7f070c9d9e946,6039bf4a35d8e991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:15.183 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,6297d477f96c9440,257bb9b1d5e818c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=6297d477f96c9440, duration=34 ms] 2024-11-04 14:16:16.614 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:16.615 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:16.766 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,1addfe75e82edf15,f0864c796ca40d7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=1addfe75e82edf15, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:16.792 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,1addfe75e82edf15,f0864c796ca40d7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=1addfe75e82edf15, duration=32 ms] 2024-11-04 14:16:17.139 WARN 21062 --- [ocp-async-5,90f8dd9c0a5070a2,a6f7a9adeaa62cf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:17.175 INFO 21062 --- [ocp-async-2,7e7378fc848fbd6f,d17a55944a7458cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d2709f35944e3afa,4f8dc6e0f1c11ca4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:18.304 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,01d17b9878fb8c40,f3043149e6ef3e0b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=01d17b9878fb8c40, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:18.332 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,01d17b9878fb8c40,f3043149e6ef3e0b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=01d17b9878fb8c40, duration=36 ms] 2024-11-04 14:16:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:16:18.590502+08:00, lastProcessTime=2024-11-04T14:16:08.591139+08:00 2024-11-04 14:16:18.651 INFO 21062 --- [ocp-async-4,33400b07825436d4,a3665fc8ca38e00e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:16:18.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:16:18.590502+08:00 2024-11-04 14:16:18.984 INFO 21062 --- [ocp-schedules-14,41ccd75b547e72b4,e4a5f7b20383bca2] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:16:19.139 WARN 21062 --- [ocp-async-9,dcc27cc40594d725,0c7c55acf860ee9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:19.175 INFO 21062 --- [ocp-async-8,570b6203ad4a6428,75cdfc9c2ce736bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:19.615 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:19.616 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:19.831 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d90b1fa7ff798042,ccdbf60e7b547045] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=d90b1fa7ff798042, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:19.858 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d90b1fa7ff798042,ccdbf60e7b547045] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=d90b1fa7ff798042, duration=34 ms] 2024-11-04 14:16:20.352 INFO 21062 --- [ocp-schedules-5,4a99f394fd4314ef,cbcfc821ea36adad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:16:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f4102cfb80f24166,1d3562b800b9222c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:21.139 WARN 21062 --- [ocp-async-6,cc5e423019602034,617f9c9a63b52070] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:21.175 INFO 21062 --- [ocp-async-3,4911ab76b2f1f09f,f1ed5bd7e1358409] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:21.372 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,c1212eff4b9fc01f,d578e2b1fb892ab0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c1212eff4b9fc01f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:21.399 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,c1212eff4b9fc01f,d578e2b1fb892ab0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c1212eff4b9fc01f, duration=34 ms] 2024-11-04 14:16:21.580 INFO 21062 --- [ocp-schedules-19,79c0e387b73b8f6b,9579a248ef608d26] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:16:22.616 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:22.616 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:22.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,bb7fc36a90ef5bbb,1cd8a00ac12ddcdc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=bb7fc36a90ef5bbb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:22.944 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,bb7fc36a90ef5bbb,1cd8a00ac12ddcdc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=bb7fc36a90ef5bbb, duration=35 ms] 2024-11-04 14:16:23.139 WARN 21062 --- [ocp-async-5,ba700cebe8228eef,0cc0340265387f03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:23.175 INFO 21062 --- [ocp-async-2,ce2614b89f4e8770,03104f5776b820db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6b7e509b02671190,0339e6a03b4b7b22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:24.464 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,6108112bfbfde4d5,10c69d67b09d81e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=6108112bfbfde4d5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:24.491 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,6108112bfbfde4d5,10c69d67b09d81e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=6108112bfbfde4d5, duration=37 ms] 2024-11-04 14:16:25.139 WARN 21062 --- [ocp-async-2,1d981c419c196708,b5369160c70f87bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:25.176 INFO 21062 --- [ocp-async-10,ff070003f370a1c6,910c25d3d614d3c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:25.617 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:25.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:26.073 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,c27b8a1b09212081,409fec7289d45440] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c27b8a1b09212081, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:26.101 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,c27b8a1b09212081,409fec7289d45440] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c27b8a1b09212081, duration=38 ms] 2024-11-04 14:16:26.983 INFO 21062 --- [prometheus-minute-schedule-1,45f6e6de3207ca46,24db901f1c94ff2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5f6a54db770081df,3e7a846842396f51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:16:27.139 WARN 21062 --- [ocp-async-10,c2e57f72d3ee71ba,4aa5c8a862df58c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:27.176 INFO 21062 --- [ocp-async-4,7fff43b466ca7b68,62ab76433b391c90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:27.614 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,41e0fd2a76daae72,9cc6d5555d42bd3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=41e0fd2a76daae72, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:27.641 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,41e0fd2a76daae72,9cc6d5555d42bd3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=41e0fd2a76daae72, duration=34 ms] 2024-11-04 14:16:28.253 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/4cdf314703f641d7/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/4cdf314703f641d7/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:16:28.257 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/4cdf314703f641d7/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/4cdf314703f641d7/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:16:28.260 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/4cdf314703f641d7/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:16:28.263 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/4cdf314703f641d7/install_package.sh, exit code: 2, cause:null 2024-11-04 14:16:28.266 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/4cdf314703f641d7/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/4cdf314703f641d7/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:16:28.277 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,b66ae92d79a37928] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: FAILED 2024-11-04 14:16:28.498 INFO 21062 --- [ocp-schedules-5,0b82e3523e193fe6,2d6df7b96488d478] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:16:28.499 INFO 21062 --- [ocp-schedules-8,cf8a9482a48fa9e7,9d1904110d19f1ef] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:16:28.500 INFO 21062 --- [ocp-schedules-19,d4bd0e2b0d66deea,a397a259a8664c5f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:16:28.500 INFO 21062 --- [ocp-schedules-9,c0e71b5c396ed09d,640d20876ce06d41] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:16:28.502 INFO 21062 --- [ocp-schedules-22,4094a36167326260,4ad7185ac3d33ad7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:16:28.503 INFO 21062 --- [ocp-schedules-5,0b82e3523e193fe6,2d6df7b96488d478] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:16:28.504 INFO 21062 --- [ocp-schedules-19,d4bd0e2b0d66deea,a397a259a8664c5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:28.504 INFO 21062 --- [ocp-schedules-9,c0e71b5c396ed09d,640d20876ce06d41] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:16:28.505 INFO 21062 --- [ocp-schedules-26,3c6faacc3b5c8c7d,68dfba7b1a5983f1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:16:28.506 INFO 21062 --- [ocp-schedules-8,cf8a9482a48fa9e7,9d1904110d19f1ef] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:16:28.510 INFO 21062 --- [ocp-schedules-16,5c6958b53546db94,3397aab267257f0e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:16:28.510 INFO 21062 --- [ocp-schedules-16,5c6958b53546db94,3397aab267257f0e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:16:28.510 INFO 21062 --- [ocp-schedules-24,e0090dec19849a45,5f4b99bfd6bfbb13] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:16:28.510 INFO 21062 --- [ocp-schedules-24,e0090dec19849a45,5f4b99bfd6bfbb13] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:16:28.511 INFO 21062 --- [ocp-schedules-18,1288037e1a332581,b30e30bcdb96dd2c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:16:28.511 INFO 21062 --- [ocp-schedules-18,1288037e1a332581,b30e30bcdb96dd2c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:16:28.574 INFO 21062 --- [ocp-schedules-6,ed5243a3efadecc6,9eff34d09ae64738] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:16:28.574 INFO 21062 --- [ocp-schedules-6,ed5243a3efadecc6,9eff34d09ae64738] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:16:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:16:28.590907+08:00, lastProcessTime=2024-11-04T14:16:18.590502+08:00 2024-11-04 14:16:28.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:28.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:28.651 INFO 21062 --- [ocp-async-6,ec0e5ee321f7433c,430174801f6737fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:16:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:16:28.590907+08:00 2024-11-04 14:16:28.811 INFO 21062 --- [ocp-schedules-13,7303fc6c73278105,a19533ba7784a62d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:16:29.062 INFO 21062 --- [ocp-schedules-8,6b72a3a3fa8d4939,970ca5871e99db2d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:16:29.062 INFO 21062 --- [ocp-schedules-8,6b72a3a3fa8d4939,970ca5871e99db2d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:16:29.126 INFO 21062 --- [ocp-schedules-26,00c13827ce468371,bbd475512cd2ac96] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:16:29.140 WARN 21062 --- [ocp-async-1,70074d0f6c69fd78,91541ce9f11c16b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:29.157 INFO 21062 --- [ocp-schedules-5,616a5720e268d99e,d71c6b252dec6d8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:16:29.163 INFO 21062 --- [ocp-schedules-5,616a5720e268d99e,d71c6b252dec6d8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:16:29.163 INFO 21062 --- [ocp-schedules-5,616a5720e268d99e,d71c6b252dec6d8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:16:29.165 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,edc058456f3d290b,a80f156ca2802f14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=edc058456f3d290b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:29.176 INFO 21062 --- [ocp-async-9,3480368b51907fe3,65d839e0186074d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:29.191 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,edc058456f3d290b,a80f156ca2802f14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=edc058456f3d290b, duration=33 ms] 2024-11-04 14:16:29.295 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:16:29.295 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:16:29.303 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:16:29.303 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:16:29.303 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:16:29.314 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:16:29.314 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:16:29.314 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:16:29.317 INFO 21062 --- [ocp-schedules-6,5970b869f1a733d7,d582b924837b262c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:16:29.468 INFO 21062 --- [ocp-schedules-17,b886fa72d4d75c3f,db7dacbde768a369] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:16:29.468 INFO 21062 --- [ocp-schedules-17,b886fa72d4d75c3f,db7dacbde768a369] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:16:29.477 INFO 21062 --- [ocp-schedules-17,b886fa72d4d75c3f,db7dacbde768a369] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:16:29.477 INFO 21062 --- [ocp-schedules-17,b886fa72d4d75c3f,db7dacbde768a369] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:16:29.481 INFO 21062 --- [ocp-schedules-17,b886fa72d4d75c3f,db7dacbde768a369] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:16:29.481 INFO 21062 --- [ocp-schedules-17,b886fa72d4d75c3f,db7dacbde768a369] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:16:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6ebce41c98e5590f,ec280328cab8085a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:16:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:16:30.356 INFO 21062 --- [ocp-schedules-8,8048304f8eae177e,50a4ee61d0db50f5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:16:30.879 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,5d5dded49369e338,22dc432ad471a705] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=5d5dded49369e338, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:30.906 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,5d5dded49369e338,22dc432ad471a705] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=5d5dded49369e338, duration=36 ms] 2024-11-04 14:16:31.140 WARN 21062 --- [ocp-async-9,4f7fcaadb5ea5894,3435f90de43e015a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:31.176 INFO 21062 --- [ocp-async-8,6e8759fe82ce6107,f926056b67c69bd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:31.318 INFO 21062 --- [ocp-schedules-13,c9d98c3d047ef725,06abe652c4f7bd65] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:16:31.323 INFO 21062 --- [ocp-schedules-13,c9d98c3d047ef725,06abe652c4f7bd65] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:16:31.323 INFO 21062 --- [ocp-schedules-13,c9d98c3d047ef725,06abe652c4f7bd65] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:16:31.359 INFO 21062 --- [ocp-schedules-3,3cb0c26a867e4f13,0b3c65aa33f1f254] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:16:31.359 INFO 21062 --- [ocp-schedules-3,3cb0c26a867e4f13,0b3c65aa33f1f254] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:16:31.385 INFO 21062 --- [ocp-schedules-3,3cb0c26a867e4f13,0b3c65aa33f1f254] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:16:31.385 INFO 21062 --- [ocp-schedules-3,3cb0c26a867e4f13,0b3c65aa33f1f254] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:16:31.387 INFO 21062 --- [ocp-schedules-26,f96bb918cf9fc410,40c953776a03eb1f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:16:31.438 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/d0883d062ac24505/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/d0883d062ac24505/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:16:31.442 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/d0883d062ac24505/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/d0883d062ac24505/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:16:31.445 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/d0883d062ac24505/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:16:31.448 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/d0883d062ac24505/install_package.sh, exit code: 2, cause:null 2024-11-04 14:16:31.451 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/d0883d062ac24505/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/d0883d062ac24505/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:16:31.462 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,e9fe2df8666ffc97] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: FAILED 2024-11-04 14:16:31.568 INFO 21062 --- [ocp-schedules-24,d6991edf9083fd52,f8a5c40103715ccb] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:16:31.568+08:00, clusters=[1] 2024-11-04 14:16:31.570 INFO 21062 --- [ocp-schedules-24,d6991edf9083fd52,f8a5c40103715ccb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:16:31.605 INFO 21062 --- [ocp-schedules-8,da223c0615c3bf52,13f846544024bed3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:16:31.619 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:31.619 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:31.699 INFO 21062 --- [ocp-schedules-28,365147daea51b5bf,309dc31b75887307] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:16:31.704 INFO 21062 --- [ocp-schedules-28,365147daea51b5bf,309dc31b75887307] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:31.722 INFO 21062 --- [ocp-schedules-4,280c963d92ef55b6,fb03963f395c5ec2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:16:31.722+08:00, clusters=[1] 2024-11-04 14:16:31.725 INFO 21062 --- [ocp-schedules-4,280c963d92ef55b6,fb03963f395c5ec2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:16:31.760 ERROR 21062 --- [ocp-schedules-28,365147daea51b5bf,309dc31b75887307] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:16:31.760 INFO 21062 --- [ocp-schedules-28,365147daea51b5bf,309dc31b75887307] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:16:32.459 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=2024-11-04T14:14:05.434+08:00, subtasks=[SubtaskInstanceEntity{id=68, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=33, startTime=2024-11-04T14:11:16.789+08:00, endTime=2024-11-04T14:11:16.854+08:00}, SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busin 2024-11-04 14:16:32.470 WARN 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:16:32.471 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 14:16:32.472 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:2, status:FAILED 2024-11-04 14:16:32.582 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,e3c70392995e416c,d973ffb46b608a78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e3c70392995e416c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:16:32.587 INFO 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:16:32.587 INFO 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:16:32.587 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:16:32.587 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:16:32.596 INFO 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:16:32.596 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:16:32.596 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:16:32.598 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:16:32.598 WARN 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:16:32.598 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:16:32.599 INFO 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:16:32.599 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:16:32.599 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:16:32.600 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:16:32.600 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:16:32.600 WARN 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:16:32.607 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:16:32.607 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:16:32.607 INFO 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:16:32.608 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,e3c70392995e416c,d973ffb46b608a78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e3c70392995e416c, duration=35 ms] 2024-11-04 14:16:32.618 ERROR 21062 --- [ocp-schedules-20,889f6a54de7055ae,e2266ef9d3ee4864] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:16:32.983 INFO 21062 --- [prometheus-minute-schedule-1,313b739977ef800b,330a2556b1da7b32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:33.140 WARN 21062 --- [ocp-async-8,cc54b681770902a1,ad8354f24872feae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:33.176 INFO 21062 --- [ocp-async-5,f8bd79b8e0bf70d6,7d0ea200e49a2d37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:33.198 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b3117ccabac077bb,d2afda09dd5e97e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=b3117ccabac077bb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:16:33.204 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b3117ccabac077bb,d2afda09dd5e97e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=b3117ccabac077bb, duration=13 ms] 2024-11-04 14:16:34.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:34.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:35.140 WARN 21062 --- [ocp-async-5,0253fa31ee02be7b,a0b744f8d5808cab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:35.176 INFO 21062 --- [ocp-async-7,c228f2fdc9963776,030d1cfbe1e1322f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:35.639 INFO 21062 --- [ocp-schedules-12,ab5f6fd8b6cb1cb4,dfa7e0e33360f20f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:16:35.639 INFO 21062 --- [ocp-schedules-12,7ad97f0bbacc120e,3608e1724454adfd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:16:35.643 INFO 21062 --- [ocp-schedules-12,7ad97f0bbacc120e,3608e1724454adfd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:35.670 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:16:35.670 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:16:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e5c8b5e8e3bfbdd2,200c8d7a53ef52f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:37.140 WARN 21062 --- [ocp-async-7,dad50eb834032e25,0ec85ea3c73aab44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:37.176 INFO 21062 --- [ocp-async-6,127098bdcd22a761,d327ff0da8708c7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:37.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:37.621 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:38.112 INFO 21062 --- [ocp-schedules-21,99e723c124988f39,91d3c5e7e9589fb3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:16:38.116 INFO 21062 --- [ocp-schedules-21,99e723c124988f39,91d3c5e7e9589fb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:38.201 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:16:38.207 ERROR 21062 --- [ocp-schedules-21,99e723c124988f39,91d3c5e7e9589fb3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:16:38.207 INFO 21062 --- [ocp-schedules-21,99e723c124988f39,91d3c5e7e9589fb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:16:38.505 INFO 21062 --- [ocp-schedules-9,c0e71b5c396ed09d,640d20876ce06d41] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:16:38.505 INFO 21062 --- [ocp-schedules-19,d4bd0e2b0d66deea,a397a259a8664c5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:16:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:16:38.590224+08:00, lastProcessTime=2024-11-04T14:16:28.590907+08:00 2024-11-04 14:16:38.651 INFO 21062 --- [ocp-async-2,884059cba3e7e4ac,a081294186b27f7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:16:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:16:38.590224+08:00 2024-11-04 14:16:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ba79c88b34750018,cc1820159bba9c9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:39.141 WARN 21062 --- [ocp-async-4,2d400ddef0d82790,fbc4a26b9e86d7d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:39.177 INFO 21062 --- [ocp-async-1,79a2732ad2b7d40f,6788303a8339225c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:40.002 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:16:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:16:40.359 INFO 21062 --- [ocp-schedules-22,e9cee1d69b00c37d,3286c7a239db9678] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:16:40.621 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:40.622 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:41.141 WARN 21062 --- [ocp-async-1,a3bff80765c4d28c,f18bb8fbed785a9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:41.177 INFO 21062 --- [ocp-async-9,c1ad25664fa275c9,95564f4d407488b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:41.644 INFO 21062 --- [ocp-schedules-28,26d03e29c818dd67,166589ea3f2d98a4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:16:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:16:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:16:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:16:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:16:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:16:41.852 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:16:41.852 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:16:41.983 INFO 21062 --- [prometheus-minute-schedule-1,7c9ba55c9937be93,2bb6beeff9152ef9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:43.141 WARN 21062 --- [ocp-async-9,b946aea49f0d4a05,5ac0dbab4d694b82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:43.177 INFO 21062 --- [ocp-async-8,a31d8689ba4248b9,2142883ad88b49c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:43.622 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:43.623 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:44.983 INFO 21062 --- [prometheus-minute-schedule-1,1ed95770558e373a,599babd08ebd0231] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:45.141 WARN 21062 --- [ocp-async-8,e02ba76e5eae9503,ac085da9d5526991] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:45.177 INFO 21062 --- [ocp-async-5,16b3812904d43c95,554d29c3444d75cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:46.623 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:46.624 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:47.141 WARN 21062 --- [ocp-async-5,8bf34e97a3104a78,aaac5b0e84357bac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:47.177 INFO 21062 --- [ocp-async-2,4e16ecc9b52eddbf,2ad81af86fd0550a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:47.983 INFO 21062 --- [prometheus-minute-schedule-1,a0c6f0339af993ce,d0a0be02bfde2b92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:16:48.591542+08:00, lastProcessTime=2024-11-04T14:16:38.590224+08:00 2024-11-04 14:16:48.651 INFO 21062 --- [ocp-async-4,783b3a171930b6d4,4873bb8cb71b1cbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:16:48.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:16:48.591542+08:00 2024-11-04 14:16:49.142 WARN 21062 --- [ocp-async-9,131b47e59c05f528,fa63f18c2bfbb372] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:49.178 INFO 21062 --- [ocp-async-6,d38f058839a09826,3b8747329c3c8d63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:49.624 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:49.625 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:50.363 INFO 21062 --- [ocp-schedules-26,3fa72b70340a51c7,3ed3e4d45434745e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:16:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d7d2b5794e788de7,d10fbd04d4e00571] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:51.142 WARN 21062 --- [ocp-async-6,78aebda20bf6beab,729a5412166e467e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:51.178 INFO 21062 --- [ocp-async-3,98a2921c486e7540,c3c952f0ba5a98df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:51.326 INFO 21062 --- [ocp-schedules-25,beab6dfc65369987,4be980ab6b70c815] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:16:51.332 INFO 21062 --- [ocp-schedules-25,beab6dfc65369987,4be980ab6b70c815] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:16:51.332 INFO 21062 --- [ocp-schedules-25,beab6dfc65369987,4be980ab6b70c815] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:16:51.668 INFO 21062 --- [ocp-schedules-22,83faa9c1b647b100,77cd4d67d2babf64] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:16:52.625 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:52.626 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:53.142 WARN 21062 --- [ocp-async-3,4da1fed1343f008c,d1e3aa276d6106ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:53.178 INFO 21062 --- [ocp-async-7,f5efb26564935bae,536d39868dee95cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:53.983 INFO 21062 --- [prometheus-minute-schedule-1,91bbd69fcea5a053,0103bcf5ec599a23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:55.142 WARN 21062 --- [ocp-async-7,1f5a2e4e9e56676b,0c7e202d46b3bd8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:55.178 INFO 21062 --- [ocp-async-10,3b793c56db5132f6,e32e8e3be23ca369] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:55.626 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:55.627 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d14cce8d1c2ac890,81865866afc69405] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:16:57.142 WARN 21062 --- [ocp-async-8,87c857203989ff2d,6692300e321e4827] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:57.178 INFO 21062 --- [ocp-async-4,3a86303512a56f52,83ddc17628ec36d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:58.500 INFO 21062 --- [ocp-schedules-19,275304b08c9e4a90,81b723ed8d49f4db] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:16:58.500 INFO 21062 --- [ocp-schedules-2,59ed9ad01da9ed87,9d924f9e13d47e23] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:16:58.502 INFO 21062 --- [ocp-schedules-17,188f1af5e96e4a4c,b7288f3223a3e8fc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:16:58.504 INFO 21062 --- [ocp-schedules-19,275304b08c9e4a90,81b723ed8d49f4db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:58.504 INFO 21062 --- [ocp-schedules-2,59ed9ad01da9ed87,9d924f9e13d47e23] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:16:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:16:58.505 INFO 21062 --- [ocp-schedules-4,219cc97adbbf9bb1,3e907ccb1f3d6863] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:16:58.512 INFO 21062 --- [ocp-schedules-20,b2d478e6907cab20,b3f3b9ab22c7cad9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:16:58.512 INFO 21062 --- [ocp-schedules-20,b2d478e6907cab20,b3f3b9ab22c7cad9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:16:58.513 INFO 21062 --- [ocp-schedules-3,b63e3b3bf2f01cd0,613a759c25df6623] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:16:58.513 INFO 21062 --- [ocp-schedules-3,b63e3b3bf2f01cd0,613a759c25df6623] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:16:58.513 INFO 21062 --- [ocp-schedules-15,d80ae8ec25478f80,49fc6b62ac27a0b8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:16:58.513 INFO 21062 --- [ocp-schedules-15,d80ae8ec25478f80,49fc6b62ac27a0b8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:16:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:16:58.590134+08:00, lastProcessTime=2024-11-04T14:16:48.591542+08:00 2024-11-04 14:16:58.627 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:16:58.628 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:16:58.652 INFO 21062 --- [ocp-async-9,5f4418b6ab689095,0336ed427dd2d84a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:16:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:16:58.590134+08:00 2024-11-04 14:16:59.129 INFO 21062 --- [ocp-schedules-21,b9543eced2bd5c3a,b4a624b8c3aab01c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:16:59.143 WARN 21062 --- [ocp-async-3,b358f8d3b0dd2ec5,e77915bd91b99c1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:16:59.179 INFO 21062 --- [ocp-async-2,e6061ea32ff3d934,8bdc2658dbe64935] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:16:59.983 INFO 21062 --- [prometheus-minute-schedule-1,4e872c882e240d8c,da4352b339434bb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701020, interval=60 2024-11-04 14:17:00.366 INFO 21062 --- [ocp-schedules-21,9df18a5331df9a12,a184ca14558ad525] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:17:01.143 WARN 21062 --- [ocp-async-7,c73e5029fcf9b817,950811d9c2443e3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:01.179 INFO 21062 --- [ocp-async-10,63b63baa7326d678,220a57363be0c3b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:01.628 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:01.629 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:01.711 INFO 21062 --- [ocp-schedules-20,38d33f77ca7d2e8b,df367ca44f286179] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:17:02.983 INFO 21062 --- [prometheus-minute-schedule-1,164ba08e060a58b7,6bb7b84476a8805e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:03.143 WARN 21062 --- [ocp-async-8,3960ce8b71b188ae,845c33d34356b3d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:03.179 INFO 21062 --- [ocp-async-4,93d638704a5effa6,53961e68e8cdb0ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:04.629 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:04.630 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:05.143 WARN 21062 --- [ocp-async-5,1f325cd66230abfc,d405e83643395e9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:05.179 INFO 21062 --- [ocp-async-1,a5f3a8d46f3774a5,8793d58467401900] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:05.671 INFO 21062 --- [ocp-schedules-12,7ad97f0bbacc120e,3608e1724454adfd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:17:05.671 INFO 21062 --- [ocp-schedules-12,64036d3defbac0c0,84427f09d7b7e323] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:17:05.676 INFO 21062 --- [ocp-schedules-12,64036d3defbac0c0,84427f09d7b7e323] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:05.706 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:17:05.706 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:17:05.983 INFO 21062 --- [prometheus-minute-schedule-1,90d00e969946b828,5603b39ec1abb2af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:07.143 WARN 21062 --- [ocp-async-2,1aad839b3c3b178f,b542d74b5929166b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:07.179 INFO 21062 --- [ocp-async-9,1ab3b99a83536c68,87edc14cf2b3cf39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:07.630 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:07.630 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:08.505 INFO 21062 --- [ocp-schedules-2,59ed9ad01da9ed87,9d924f9e13d47e23] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:17:08.505 INFO 21062 --- [ocp-schedules-19,275304b08c9e4a90,81b723ed8d49f4db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:17:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:17:08.590963+08:00, lastProcessTime=2024-11-04T14:16:58.590134+08:00 2024-11-04 14:17:08.652 INFO 21062 --- [ocp-async-3,fe8402bad622c554,e63aaf9c792acaba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:17:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:17:08.590963+08:00 2024-11-04 14:17:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a5e04eba26bb3c4d,0cdf0fa5522f343f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:09.143 WARN 21062 --- [ocp-async-8,8ed9ecc05cccfe8b,4992b75da9e82630] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:09.179 INFO 21062 --- [ocp-async-4,06314eb877735b23,8020295e6d6c8186] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:17:10.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:17:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:17:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:17:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:17:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:17:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:17:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:17:10.369 INFO 21062 --- [ocp-schedules-27,34b575dd14a05e65,f8b0f4f845060746] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:17:10.630 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:10.631 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:11.144 WARN 21062 --- [ocp-async-5,8fd56874d2ad6ce6,f5ca8395ec8c293c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:11.180 INFO 21062 --- [ocp-async-1,df8f73f050a76a55,f558cb7a4b90834f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:11.333 INFO 21062 --- [ocp-schedules-11,393814a368548bf9,e35d921a1cfee9da] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:17:11.338 INFO 21062 --- [ocp-schedules-11,393814a368548bf9,e35d921a1cfee9da] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:17:11.338 INFO 21062 --- [ocp-schedules-11,393814a368548bf9,e35d921a1cfee9da] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:17:11.520 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=12, affectRows=12 2024-11-04 14:17:11.520 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=12, validCount=12 2024-11-04 14:17:11.521 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=33, affectRows=33 2024-11-04 14:17:11.521 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=49, affectRows=49 2024-11-04 14:17:11.521 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=33, validCount=33 2024-11-04 14:17:11.521 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=49, validCount=49 2024-11-04 14:17:11.522 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=24, affectRows=24 2024-11-04 14:17:11.522 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=24, validCount=24 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=54, affectRows=54 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=76, affectRows=76 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=62, affectRows=62 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730700960, nodeCount=86, affectRows=86 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=54, validCount=54 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=62, validCount=62 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=75, validCount=75 2024-11-04 14:17:11.523 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730700960, nodeCount=84, validCount=84 2024-11-04 14:17:11.736 INFO 21062 --- [ocp-schedules-30,5f402a79e0a36520,feb660d7f3016b0c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:17:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a5dd64945a42c407,9e43cf3c870c6510] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:13.144 WARN 21062 --- [ocp-async-2,64b54046518bb189,33dc02f247666ab8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:13.180 INFO 21062 --- [ocp-async-9,e40985f1890a5611,82607807b6b318cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:13.631 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:13.632 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:13.984 INFO 21062 --- [prometheus-cache-monitor1,820d08e607734055,e4517f54bdfba683] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:17:13.984 INFO 21062 --- [prometheus-cache-monitor1,820d08e607734055,e4517f54bdfba683] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:17:14.983 INFO 21062 --- [prometheus-minute-schedule-1,37d6aedb7a4fc2c5,d8a3896a96f772ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:14.990 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:13:14.971, end=2024-11-04T14:15:14.987} 2024-11-04 14:17:14.999 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:17:14.999 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:17:15.144 WARN 21062 --- [ocp-async-10,bd4a332a11048854,16a38aec48727e19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:15.180 INFO 21062 --- [ocp-async-6,4b21e5fe4eda0330,b70a6fb16de2120c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:16.632 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:16.633 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:17.144 WARN 21062 --- [ocp-async-4,a7acf589da18c02c,237b8c4cdcc49160] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:17.180 INFO 21062 --- [ocp-async-3,ae51ce13a5bd11b0,c4cb27860f444767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:17.983 INFO 21062 --- [prometheus-minute-schedule-1,66526968bd30fa89,229aa9ba93a2a9bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:17:18.591186+08:00, lastProcessTime=2024-11-04T14:17:08.590963+08:00 2024-11-04 14:17:18.652 INFO 21062 --- [ocp-async-8,d530ce370a37d020,7cc39df74f48f427] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:17:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:17:18.591186+08:00 2024-11-04 14:17:18.985 INFO 21062 --- [ocp-schedules-16,c9ed717ed0cdf3ef,f6b8d026bec3b220] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:17:18.990 INFO 21062 --- [ocp-schedules-16,c9ed717ed0cdf3ef,f6b8d026bec3b220] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:18.991 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:17:19.144 WARN 21062 --- [ocp-async-2,e36c5eec4ab9108e,89ce052bb8ff0f27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:19.180 INFO 21062 --- [ocp-async-9,33c3eb870eabdac0,41625373c38792ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:19.633 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:19.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:20.372 INFO 21062 --- [ocp-schedules-18,bb6b58be30666143,2e7563d6d37ff71e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:17:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5d29a31dbd909ae1,236abc716c29e1dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:21.145 WARN 21062 --- [ocp-async-10,1df3d670cbac1346,c5108e5c6944c31a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:21.181 INFO 21062 --- [ocp-async-6,1f446f219229acff,4cbca3b293f576cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:21.780 INFO 21062 --- [ocp-schedules-22,ac77925fade877ad,c1dd42b740405cb9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:17:22.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:22.635 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:23.145 WARN 21062 --- [ocp-async-4,a13d1fbe2aa007f5,b1bf27916f379b45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:23.181 INFO 21062 --- [ocp-async-3,f4def940fef3c982,72f6eaa3ddf30e31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:23.983 INFO 21062 --- [prometheus-minute-schedule-1,21f91356cd1c5186,82bd1a067f825d43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:25.145 WARN 21062 --- [ocp-async-1,6131e1b4c9a9bbd6,113f5eb738a27bea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:25.181 INFO 21062 --- [ocp-async-7,db8b1c698fb1ab98,be3c9b7f2b73f03e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:25.635 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:25.635 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:26.983 INFO 21062 --- [prometheus-minute-schedule-1,4ed217b6a20608cf,d8a5bc6dcee1ce41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a0e12da3e4894358,e373218d498eefd9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:17:27.145 WARN 21062 --- [ocp-async-7,02ddc9ea7aa061c4,a7b4c9f49ab79fb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:27.181 INFO 21062 --- [ocp-async-8,6a0182330cb6e413,da91afae5593dead] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:28.498 INFO 21062 --- [ocp-schedules-19,ed08bd83bc622004,e119884df9ed8992] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:17:28.500 INFO 21062 --- [ocp-schedules-8,89951b6f39fa85ea,98eceb04deb79c5b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:17:28.500 INFO 21062 --- [ocp-schedules-14,e7e8b281a83a309e,775f8b8457598570] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:17:28.500 INFO 21062 --- [ocp-schedules-22,175744dca43ba6d2,16f23d76230fd9a8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:17:28.502 INFO 21062 --- [ocp-schedules-24,7ea23f0c83d92d4f,3f306516e67e741d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:17:28.503 INFO 21062 --- [ocp-schedules-19,ed08bd83bc622004,e119884df9ed8992] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:17:28.503 INFO 21062 --- [ocp-schedules-8,89951b6f39fa85ea,98eceb04deb79c5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:17:28.504 INFO 21062 --- [ocp-schedules-22,175744dca43ba6d2,16f23d76230fd9a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:28.505 INFO 21062 --- [ocp-schedules-2,be77c4681864404d,44f549045b4fcac2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:17:28.505 INFO 21062 --- [ocp-schedules-21,6029e8385864c7db,8e77dc7f97d258d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:17:28.506 INFO 21062 --- [ocp-schedules-14,e7e8b281a83a309e,775f8b8457598570] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:17:28.508 INFO 21062 --- [ocp-schedules-21,6029e8385864c7db,8e77dc7f97d258d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:28.512 INFO 21062 --- [ocp-schedules-30,c8c36a4bec689b22,1bba51b03c9a3b18] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:17:28.512 INFO 21062 --- [ocp-schedules-6,b3eca9255e52f2c8,6d8f4bfbe2a4c533] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:17:28.512 INFO 21062 --- [ocp-schedules-6,b3eca9255e52f2c8,6d8f4bfbe2a4c533] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:17:28.512 INFO 21062 --- [ocp-schedules-30,c8c36a4bec689b22,1bba51b03c9a3b18] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:17:28.512 INFO 21062 --- [ocp-schedules-11,2177225e62c6f51f,5cfcf1c1073cf77c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:17:28.512 INFO 21062 --- [ocp-schedules-11,2177225e62c6f51f,5cfcf1c1073cf77c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:17:28.536 INFO 21062 --- [ocp-schedules-20,edc622fbd9783715,21dd82ac6a263d69] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:17:28.536 INFO 21062 --- [ocp-schedules-20,edc622fbd9783715,21dd82ac6a263d69] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:17:28.605 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:17:28.605080+08:00, lastProcessTime=2024-11-04T14:17:18.591186+08:00 2024-11-04 14:17:28.621 INFO 21062 --- [pool-12-thread-5,94ffa7438591f5a2,fde18fc1db80fc3c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:17:28.621 INFO 21062 --- [pool-12-thread-8,94ffa7438591f5a2,6b6562b6096826d3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:17:28.621 INFO 21062 --- [pool-12-thread-5,94ffa7438591f5a2,fde18fc1db80fc3c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:17:28.622 INFO 21062 --- [pool-12-thread-6,94ffa7438591f5a2,891a2f472dc1c292] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:17:28.622 INFO 21062 --- [pool-12-thread-7,94ffa7438591f5a2,dd52749cb4faf94b] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:17:28.623 INFO 21062 --- [pool-12-thread-5,94ffa7438591f5a2,fde18fc1db80fc3c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:17:28.625 INFO 21062 --- [pool-12-thread-5,94ffa7438591f5a2,fde18fc1db80fc3c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:17:28.627 INFO 21062 --- [ForkJoinPool-11-worker-0,94ffa7438591f5a2,3df58f21ab839451] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:17:28.627 INFO 21062 --- [pool-12-thread-6,94ffa7438591f5a2,891a2f472dc1c292] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:17:28.628 INFO 21062 --- [ForkJoinPool-11-worker-9,94ffa7438591f5a2,9528dec1f173ad95] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:17:28.628 INFO 21062 --- [ForkJoinPool-11-worker-9,94ffa7438591f5a2,9528dec1f173ad95] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:17:28.628 INFO 21062 --- [ForkJoinPool-11-worker-9,94ffa7438591f5a2,9528dec1f173ad95] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:17:28.628 INFO 21062 --- [pool-12-thread-8,94ffa7438591f5a2,6b6562b6096826d3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:17:28.630 INFO 21062 --- [pool-12-thread-7,94ffa7438591f5a2,dd52749cb4faf94b] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:17:28.630 INFO 21062 --- [pool-12-thread-7,94ffa7438591f5a2,dd52749cb4faf94b] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:17:28.636 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:28.636 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:28.652 INFO 21062 --- [ocp-async-10,cddec3b0e569d554,cf382fe282643ec4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:17:28.780 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:17:28.605080+08:00 2024-11-04 14:17:28.814 INFO 21062 --- [ocp-schedules-24,0ec94bb4a89eefc2,89f83c2a45fd64d3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:17:28.867 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:17:28.883 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:17:28.885 WARN 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:17:28.886 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:17:28.887 WARN 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:17:28.893 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:17:28.894 WARN 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:17:28.896 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:17:28.897 WARN 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:17:28.903 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:17:28.904 WARN 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:17:28.905 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:17:28.906 WARN 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:17:28.912 INFO 21062 --- [ocp-schedules-6,d89d82526c0af5dd,e2fec1a811ec9664] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:17:28.991 INFO 21062 --- [ocp-schedules-16,c9ed717ed0cdf3ef,f6b8d026bec3b220] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:17:29.068 INFO 21062 --- [ocp-schedules-20,efb23df2cb9d87e8,3b77a5345547e937] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:17:29.068 INFO 21062 --- [ocp-schedules-20,efb23df2cb9d87e8,3b77a5345547e937] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:17:29.132 INFO 21062 --- [ocp-schedules-11,25228d919ad734bb,f7913d47b1cefbbf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:17:29.145 WARN 21062 --- [ocp-async-5,6b223c0ff0ffc37b,202cf3c41c33310a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:29.163 INFO 21062 --- [ocp-schedules-26,141e5842c12333b7,81f368e85dfd8854] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:17:29.169 INFO 21062 --- [ocp-schedules-26,141e5842c12333b7,81f368e85dfd8854] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:17:29.169 INFO 21062 --- [ocp-schedules-26,141e5842c12333b7,81f368e85dfd8854] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:17:29.181 INFO 21062 --- [ocp-async-2,b6aab23a87035fcc,bfd4ee852587ae8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:29.486 INFO 21062 --- [ocp-schedules-17,8e99b86ae50a2467,5624606ae631fc3a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:17:29.486 INFO 21062 --- [ocp-schedules-17,8e99b86ae50a2467,5624606ae631fc3a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:17:29.487 INFO 21062 --- [ocp-schedules-17,8e99b86ae50a2467,5624606ae631fc3a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:17:29.487 INFO 21062 --- [ocp-schedules-17,8e99b86ae50a2467,5624606ae631fc3a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:17:29.488 INFO 21062 --- [ocp-schedules-17,8e99b86ae50a2467,5624606ae631fc3a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:17:29.488 INFO 21062 --- [ocp-schedules-17,8e99b86ae50a2467,5624606ae631fc3a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:17:29.690 INFO 21062 --- [ocp-schedules-3,1015768b4a625723,acad95a71995daed] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:17:29.690 INFO 21062 --- [ocp-schedules-3,1015768b4a625723,acad95a71995daed] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:17:29.719 INFO 21062 --- [pool-40-thread-1,1015768b4a625723,8bd9e6c022f7963a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:17:29.746 WARN 21062 --- [ocp-schedules-3,1015768b4a625723,acad95a71995daed] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:17:29.746 INFO 21062 --- [ocp-schedules-3,1015768b4a625723,acad95a71995daed] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:17:29.983 INFO 21062 --- [prometheus-minute-schedule-1,63a5c71e2e598cb3,bfb06361511f85b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:17:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:17:30.374 INFO 21062 --- [ocp-schedules-13,b70a9031b4893680,914dec19e265e517] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:17:31.145 WARN 21062 --- [ocp-async-2,7a355decf886dc34,e87206d7ac25f2d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:31.181 INFO 21062 --- [ocp-async-9,ccf257c882e0b442,21481fe9bf46262d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:31.342 INFO 21062 --- [ocp-schedules-18,a12cde4adde3c010,e31e438b4b28ab31] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:17:31.348 INFO 21062 --- [ocp-schedules-18,a12cde4adde3c010,e31e438b4b28ab31] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:17:31.348 INFO 21062 --- [ocp-schedules-18,a12cde4adde3c010,e31e438b4b28ab31] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:17:31.386 INFO 21062 --- [ocp-schedules-16,878b3dc53ab67be5,2490cb05df807a00] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:17:31.386 INFO 21062 --- [ocp-schedules-16,878b3dc53ab67be5,2490cb05df807a00] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:17:31.413 INFO 21062 --- [ocp-schedules-16,878b3dc53ab67be5,2490cb05df807a00] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:17:31.413 INFO 21062 --- [ocp-schedules-16,878b3dc53ab67be5,2490cb05df807a00] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:17:31.414 INFO 21062 --- [ocp-schedules-6,98a3e5c94a398d29,dce38910fa176a42] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:17:31.595 INFO 21062 --- [ocp-schedules-13,e6b4b94233bc5ed1,3f25d8ada1645a8a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:17:31.595+08:00, clusters=[1] 2024-11-04 14:17:31.597 INFO 21062 --- [ocp-schedules-13,e6b4b94233bc5ed1,3f25d8ada1645a8a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:17:31.636 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:31.637 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:31.750 INFO 21062 --- [ocp-schedules-29,44a7865d67190b26,e5dbd5e3d345900a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:17:31.750+08:00, clusters=[1] 2024-11-04 14:17:31.753 INFO 21062 --- [ocp-schedules-29,44a7865d67190b26,e5dbd5e3d345900a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:17:31.807 INFO 21062 --- [ocp-schedules-17,10e7475e9819fe52,5b7c956ab78079d9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:17:32.628 INFO 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:17:32.628 INFO 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:17:32.628 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:17:32.628 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:17:32.637 INFO 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:17:32.637 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:17:32.638 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:17:32.638 WARN 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:17:32.639 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:17:32.639 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:17:32.639 INFO 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:17:32.639 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:17:32.640 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:17:32.640 WARN 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:17:32.641 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:17:32.641 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:17:32.647 INFO 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:17:32.648 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:17:32.648 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:17:32.658 ERROR 21062 --- [ocp-schedules-3,3c8f4445e5f4a1d0,79f2bd905b496e4e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:17:32.983 INFO 21062 --- [prometheus-minute-schedule-1,39c1649b95a609fe,09d4c19b416c9f3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:33.145 WARN 21062 --- [ocp-async-9,7b6500840cfe1b7e,2612a9cfef8701d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:33.181 INFO 21062 --- [ocp-async-4,273a6266a172ccfb,9d9168bc778f632e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:34.637 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:34.638 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:35.146 WARN 21062 --- [ocp-async-4,fa63407508eadfac,bd3d386d6909eb8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:35.182 INFO 21062 --- [ocp-async-1,04f78ff92b7eb0a8,8b12e369d7350915] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:35.707 INFO 21062 --- [ocp-schedules-12,64036d3defbac0c0,84427f09d7b7e323] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:17:35.707 INFO 21062 --- [ocp-schedules-12,beab15a3678b9adf,d55fa4ba9162c2f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:17:35.711 INFO 21062 --- [ocp-schedules-12,beab15a3678b9adf,d55fa4ba9162c2f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:35.741 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:17:35.741 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:17:35.983 INFO 21062 --- [prometheus-minute-schedule-1,f18553fc65915f3d,a2fcab5bf3b51e29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:37.146 WARN 21062 --- [ocp-async-1,c9dc1348b472d33f,c42c216c7b250d99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:37.182 INFO 21062 --- [ocp-async-10,c5715b59ce3a3b9e,450873892cac329f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:37.638 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:37.639 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:38.208 INFO 21062 --- [ocp-schedules-18,e76a9048cca1d8e9,ff886f543582e4dd] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:17:38.211 INFO 21062 --- [ocp-schedules-18,e76a9048cca1d8e9,ff886f543582e4dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:38.297 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:17:38.299 ERROR 21062 --- [ocp-schedules-18,e76a9048cca1d8e9,ff886f543582e4dd] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:17:38.299 INFO 21062 --- [ocp-schedules-18,e76a9048cca1d8e9,ff886f543582e4dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:17:38.504 INFO 21062 --- [ocp-schedules-8,89951b6f39fa85ea,98eceb04deb79c5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:17:38.505 INFO 21062 --- [ocp-schedules-22,175744dca43ba6d2,16f23d76230fd9a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:17:38.509 INFO 21062 --- [ocp-schedules-21,6029e8385864c7db,8e77dc7f97d258d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:17:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:17:38.590565+08:00, lastProcessTime=2024-11-04T14:17:28.605080+08:00 2024-11-04 14:17:38.652 INFO 21062 --- [ocp-async-3,ef73b8bc42006610,a91bc9935c071884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:17:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:17:38.590565+08:00 2024-11-04 14:17:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a73586d309b57c3d,3ae9f0d8a9139ddb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:39.146 WARN 21062 --- [ocp-async-8,fc645f84750cb674,6552f49af1b0ae0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:39.182 INFO 21062 --- [ocp-async-5,6f1d02555c3c471b,c61a838eb7262923] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:17:40.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:17:40.377 INFO 21062 --- [ocp-schedules-27,a72524594f4a3033,34f9e77022d9df42] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:17:40.639 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:40.640 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:41.146 WARN 21062 --- [ocp-async-5,486cb72bda822631,4f48664c073e435b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:41.182 INFO 21062 --- [ocp-async-2,b5432ecc42b0631e,d73e09562b2d2c71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:41.848 INFO 21062 --- [ocp-schedules-22,69793f273ce2f69b,1bd3b1d046bd7c4d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:17:41.983 INFO 21062 --- [prometheus-minute-schedule-1,836673a6cda5bd00,613245136cf84e1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:43.147 WARN 21062 --- [ocp-async-2,68707aa938c866f4,13c10b5efa01f8fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:43.182 INFO 21062 --- [ocp-async-9,07aedf0a1948c825,f3ed0001a982360f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:43.640 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:43.641 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:44.983 INFO 21062 --- [prometheus-minute-schedule-1,4ad329527e5e0e6a,ce5c635360d2439d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:45.147 WARN 21062 --- [ocp-async-9,ba5da9962ca3b732,e92868d40c248f06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:45.182 INFO 21062 --- [ocp-async-6,567c8b5a73f49444,00426352f8f02cc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:46.641 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:46.642 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:47.147 WARN 21062 --- [ocp-async-6,d15ecca66a607b1f,72acea76222c0f88] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:47.182 INFO 21062 --- [ocp-async-3,9e505f742689b996,d03e982eabfaa969] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:47.983 INFO 21062 --- [prometheus-minute-schedule-1,95378725c46688ad,64fada3bda77a1ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:17:48.591893+08:00, lastProcessTime=2024-11-04T14:17:38.590565+08:00 2024-11-04 14:17:48.653 INFO 21062 --- [ocp-async-8,71b9164deadd9519,16d92582e0161d61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:17:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:17:48.591893+08:00 2024-11-04 14:17:49.147 WARN 21062 --- [ocp-async-2,63dbc3a1d3becb22,14c02d5f694b8931] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:49.183 INFO 21062 --- [ocp-async-10,0c2f465279d21757,1602611e396a70b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:49.642 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:49.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:50.380 INFO 21062 --- [ocp-schedules-6,4db4b878d68372ce,5383b7dd25dabdd6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:17:50.983 INFO 21062 --- [prometheus-minute-schedule-1,9e48dc75d6af0fdb,73dbcb8e1d03a3ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:51.147 WARN 21062 --- [ocp-async-10,699de76803e51568,9bd60ec57a4e4fba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:51.183 INFO 21062 --- [ocp-async-4,22551f5745bf8436,c62a4a55395d4402] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:51.351 INFO 21062 --- [ocp-schedules-20,b14ce0e0d1feee27,1179bd3ca3ef6b71] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:17:51.357 INFO 21062 --- [ocp-schedules-20,b14ce0e0d1feee27,1179bd3ca3ef6b71] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:17:51.357 INFO 21062 --- [ocp-schedules-20,b14ce0e0d1feee27,1179bd3ca3ef6b71] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:17:51.875 INFO 21062 --- [ocp-schedules-10,ed9fb8124d98ef8c,14115f9b317dd4c7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:17:52.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:52.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:53.147 WARN 21062 --- [ocp-async-6,01a0df49b20b02aa,3d4cf8553824c190] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:53.183 INFO 21062 --- [ocp-async-1,a219133506f8f8ff,156e4c8a39f5bafa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:53.983 INFO 21062 --- [prometheus-minute-schedule-1,fd3f963e1432bbc7,9f4ece3c6fa90006] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:55.148 WARN 21062 --- [ocp-async-3,c96230caea02d88d,331133dde02393d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:55.183 INFO 21062 --- [ocp-async-7,59503a115c114341,0e1ed326ac20c055] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:55.644 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:55.644 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d7aca67cbc4a5b8c,60e50f84847ef7fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:17:57.148 WARN 21062 --- [ocp-async-9,47b27140b9dfbbe8,a510464b206ba3c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:57.184 INFO 21062 --- [ocp-async-8,2d43d42d0462cec6,b38986757b981a6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:58.500 INFO 21062 --- [ocp-schedules-24,89be13e28c02ae57,81d571d3a9be6f70] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:17:58.500 INFO 21062 --- [ocp-schedules-27,f34d780530ee602e,f4d2c2d8ed25d657] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:17:58.502 INFO 21062 --- [ocp-schedules-18,42634e7036bb220e,49f381825703d210] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:17:58.503 INFO 21062 --- [ocp-schedules-24,89be13e28c02ae57,81d571d3a9be6f70] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:58.504 INFO 21062 --- [ocp-schedules-27,f34d780530ee602e,f4d2c2d8ed25d657] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:17:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:17:58.504 INFO 21062 --- [ocp-schedules-23,f82dc54feff79d3f,90cba42802c4d0cc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:17:58.511 INFO 21062 --- [ocp-schedules-4,1932f0d2068e6b1b,faf4b489dee70e7d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:17:58.511 INFO 21062 --- [ocp-schedules-4,1932f0d2068e6b1b,faf4b489dee70e7d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:17:58.511 INFO 21062 --- [ocp-schedules-22,6eef45c76d610909,8af4c8f49a7f0bad] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:17:58.511 INFO 21062 --- [ocp-schedules-15,bed0b6660474c9ac,16ad586ce031df14] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:17:58.511 INFO 21062 --- [ocp-schedules-22,6eef45c76d610909,8af4c8f49a7f0bad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:17:58.511 INFO 21062 --- [ocp-schedules-15,bed0b6660474c9ac,16ad586ce031df14] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:17:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:17:58.590690+08:00, lastProcessTime=2024-11-04T14:17:48.591893+08:00 2024-11-04 14:17:58.644 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:17:58.645 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:17:58.653 INFO 21062 --- [ocp-async-2,4498546793da3b8f,494812d799e31e7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:17:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:17:58.590690+08:00 2024-11-04 14:17:59.134 INFO 21062 --- [ocp-schedules-26,0daa3be5583770bb,12162d15d99cca42] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:17:59.148 WARN 21062 --- [ocp-async-6,a58f6311f1c31c34,2382baa04da35ecb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:17:59.184 INFO 21062 --- [ocp-async-1,2d872c8e97599070,3e666699e5e6f93a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:17:59.983 INFO 21062 --- [prometheus-minute-schedule-1,919f48e46ab1d72a,00e447dd00524ccd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701080, interval=60 2024-11-04 14:18:00.383 INFO 21062 --- [ocp-schedules-21,00b3ca5552cebbc8,d2337badd0697f69] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:18:01.148 WARN 21062 --- [ocp-async-3,cbe28c6da0c29c77,1aa2b6ff3dd39eb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:01.184 INFO 21062 --- [ocp-async-7,b2de7cc2bffe7291,d51dd080870ff4aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:01.645 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:01.646 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:01.917 INFO 21062 --- [ocp-schedules-30,8f602c29e857d2b0,ea68a8842c15c806] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:18:02.983 INFO 21062 --- [prometheus-minute-schedule-1,68798c829b345c2a,fe283cd3a3ad857d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:03.148 WARN 21062 --- [ocp-async-9,4ada046f11938f94,172a5d69789f5480] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:03.184 INFO 21062 --- [ocp-async-8,cc35cdc96a879b60,8e0f65661a1ff6bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:04.646 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:04.647 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:05.148 WARN 21062 --- [ocp-async-4,edd51d0b69d98eba,15f7a756618a643a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:05.184 INFO 21062 --- [ocp-async-5,f80a8df938ca72f7,b0da86ced07ff429] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:05.741 INFO 21062 --- [ocp-schedules-12,beab15a3678b9adf,d55fa4ba9162c2f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:18:05.742 INFO 21062 --- [ocp-schedules-12,0b16f2c2f2642bda,5eee117674688f3b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:18:05.746 INFO 21062 --- [ocp-schedules-12,0b16f2c2f2642bda,5eee117674688f3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:05.776 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:18:05.776 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:18:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d8ea4059db39c2a8,a1032dba9879eabd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:07.149 WARN 21062 --- [ocp-async-1,f2613605e05ec25c,4686d0cf8d26a9a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:07.184 INFO 21062 --- [ocp-async-2,0f1b6d81bb62328f,d9130f4a6115ff8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:07.647 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:07.648 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:08.504 INFO 21062 --- [ocp-schedules-27,f34d780530ee602e,f4d2c2d8ed25d657] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:18:08.504 INFO 21062 --- [ocp-schedules-24,89be13e28c02ae57,81d571d3a9be6f70] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:18:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:18:08.590454+08:00, lastProcessTime=2024-11-04T14:17:58.590690+08:00 2024-11-04 14:18:08.653 INFO 21062 --- [ocp-async-6,a13bbcf5f15c9da6,ed1b945362c1b68e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:18:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:18:08.590454+08:00 2024-11-04 14:18:08.983 INFO 21062 --- [prometheus-minute-schedule-1,0dc7ed8ae9e0c18f,077591fc9eb95dbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:09.149 WARN 21062 --- [ocp-async-9,f0b762c56d8df9eb,66d540f36d1fd270] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:09.184 INFO 21062 --- [ocp-async-8,69d6ebd4ec98ca50,ce4555ab3031c5a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:18:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:18:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:18:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:18:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:18:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:18:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:18:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:18:10.388 INFO 21062 --- [ocp-schedules-19,89e8d09afcd734fb,f6974176e19a419e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:18:10.648 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:10.649 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:11.149 WARN 21062 --- [ocp-async-4,cd32891f65f7046c,96be8ca6cd492d72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:11.185 INFO 21062 --- [ocp-async-5,05cab9a46eb5c29f,dadeef6afeacc41f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:11.360 INFO 21062 --- [ocp-schedules-6,a4378305303f4505,bd384cd92e6a4c02] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:18:11.366 INFO 21062 --- [ocp-schedules-6,a4378305303f4505,bd384cd92e6a4c02] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:18:11.366 INFO 21062 --- [ocp-schedules-6,a4378305303f4505,bd384cd92e6a4c02] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:18:11.526 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701020, nodeCount=54, affectRows=54 2024-11-04 14:18:11.526 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701020, nodeCount=52, affectRows=52 2024-11-04 14:18:11.527 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701020, nodeCount=52, validCount=52 2024-11-04 14:18:11.527 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701020, nodeCount=54, validCount=54 2024-11-04 14:18:11.527 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701020, nodeCount=79, affectRows=79 2024-11-04 14:18:11.527 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701020, nodeCount=109, affectRows=109 2024-11-04 14:18:11.527 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701020, nodeCount=102, affectRows=102 2024-11-04 14:18:11.527 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701020, nodeCount=79, validCount=79 2024-11-04 14:18:11.528 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701020, nodeCount=107, validCount=107 2024-11-04 14:18:11.528 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701020, nodeCount=101, validCount=101 2024-11-04 14:18:11.941 INFO 21062 --- [ocp-schedules-5,e3989dd542d64bca,e9ae52f27437f323] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:18:11.983 INFO 21062 --- [prometheus-minute-schedule-1,2afe522bdc922026,153c6bc3bdda475e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:13.149 WARN 21062 --- [ocp-async-1,69e0228ad539b68c,aea8e88600a9d9ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:13.185 INFO 21062 --- [ocp-async-2,13eeb380552a7b6a,9b38d6722785ef89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:13.649 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:13.650 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:13.984 INFO 21062 --- [prometheus-cache-monitor1,8bcbd76d3c4fffd2,b14948c5d9eed092] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:18:13.984 INFO 21062 --- [prometheus-cache-monitor1,8bcbd76d3c4fffd2,b14948c5d9eed092] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:18:14.231 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:12:14.208, end=2024-11-04T14:17:14.228} 2024-11-04 14:18:14.231 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:12:14.208, end=2024-11-04T14:17:14.229} 2024-11-04 14:18:14.235 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:18:14.235 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:18:14.236 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:18:14.236 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:18:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5c5c9e7178dd9dd4,395e6eed68eaba90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:15.149 WARN 21062 --- [ocp-async-7,59e029b6ff615247,948e90513ae1b76f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:15.185 INFO 21062 --- [ocp-async-10,37875ebf5d06c06b,fbe2ab2e71588408] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:16.650 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:16.651 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:17.149 WARN 21062 --- [ocp-async-8,556bf7d5b8c617e4,fd03247095549f84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:17.185 INFO 21062 --- [ocp-async-6,f46e9cacfbddd138,d87cc6d546320d68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4c8868e8e9161698,9d3882a867faa798] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:18:18.590784+08:00, lastProcessTime=2024-11-04T14:18:08.590454+08:00 2024-11-04 14:18:18.653 INFO 21062 --- [ocp-async-9,ae88c3737bf73c09,2781d4a9d691a637] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:18:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:18:18.590784+08:00 2024-11-04 14:18:19.149 WARN 21062 --- [ocp-async-1,65f481430544f760,3495a585788d2409] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:19.185 INFO 21062 --- [ocp-async-2,94f8a96f27a7c098,162aebdedae99343] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:19.651 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:19.652 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:20.391 INFO 21062 --- [ocp-schedules-6,a3d91ecfb95cf246,3f510e2da8d90fc4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:18:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ce7e853ad5dfa5d4,f054cc96f7cd796b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:21.150 WARN 21062 --- [ocp-async-7,f76982c8fd7a6420,288e2e5945d510ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:21.185 INFO 21062 --- [ocp-async-10,169e64619b22976c,a5bdaf2afe86df68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:21.983 INFO 21062 --- [ocp-schedules-18,4e087cf446ec6db0,fc89fe4ef280c261] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:18:22.652 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:22.652 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:23.150 WARN 21062 --- [ocp-async-8,0d4585d3deebaef0,86c8a210afc34846] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:23.186 INFO 21062 --- [ocp-async-6,9f90713b08acc35c,fe5ba035761f3550] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:23.983 INFO 21062 --- [prometheus-minute-schedule-1,16f6c8eef5ea84f3,2e0ec5110151644b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:25.150 WARN 21062 --- [ocp-async-6,2f9e7bf49656fc73,2ef794046f189234] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:25.186 INFO 21062 --- [ocp-async-3,c031b4d0e591c50e,a45bf759336b7baf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:25.653 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:25.653 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:26.983 INFO 21062 --- [prometheus-minute-schedule-1,6b038417d2b50707,6c670e3f5e24578e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4587a1f0bc1ff6b0,eaa4f71f43793456] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:18:27.150 WARN 21062 --- [ocp-async-3,2b8777784ef222cd,8b81825a57e6c257] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:27.186 INFO 21062 --- [ocp-async-9,7b9af3fc37d75d22,f2ae86289be80c8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:28.498 INFO 21062 --- [ocp-schedules-11,00db1b71abe03305,97be097a324c48c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:18:28.499 INFO 21062 --- [ocp-schedules-5,7f242d827020afeb,4e1f14a55ff43b09] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:18:28.500 INFO 21062 --- [ocp-schedules-18,1229d2a39001c772,b45cd4fa515de16f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:18:28.500 INFO 21062 --- [ocp-schedules-4,be6840ed0e3c6148,88cf44a890ecd57c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:18:28.502 INFO 21062 --- [ocp-schedules-11,00db1b71abe03305,97be097a324c48c5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:18:28.502 INFO 21062 --- [ocp-schedules-30,9b6384c5ff7dfcac,e88ff036ba005f56] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:18:28.503 INFO 21062 --- [ocp-schedules-18,1229d2a39001c772,b45cd4fa515de16f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:28.504 INFO 21062 --- [ocp-schedules-4,be6840ed0e3c6148,88cf44a890ecd57c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:18:28.504 INFO 21062 --- [ocp-schedules-24,4a42dc782c933983,c9416fd296ffd897] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:18:28.504 INFO 21062 --- [ocp-schedules-5,7f242d827020afeb,4e1f14a55ff43b09] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:18:28.509 INFO 21062 --- [ocp-schedules-26,c0a2063f17e735a1,005aa3da34eb49f2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:18:28.509 INFO 21062 --- [ocp-schedules-26,c0a2063f17e735a1,005aa3da34eb49f2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:18:28.510 INFO 21062 --- [ocp-schedules-16,44ddb5facecb2d88,44f71d331fa3b4ae] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:18:28.510 INFO 21062 --- [ocp-schedules-16,44ddb5facecb2d88,44f71d331fa3b4ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:18:28.510 INFO 21062 --- [ocp-schedules-19,d1e4b3b1b3e477fb,5a0526f49161e7ac] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:18:28.510 INFO 21062 --- [ocp-schedules-19,d1e4b3b1b3e477fb,5a0526f49161e7ac] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:18:28.575 INFO 21062 --- [ocp-schedules-22,5b6a970d021401e3,eb8e3c3039539f20] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:18:28.575 INFO 21062 --- [ocp-schedules-22,5b6a970d021401e3,eb8e3c3039539f20] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:18:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:18:28.590350+08:00, lastProcessTime=2024-11-04T14:18:18.590784+08:00 2024-11-04 14:18:28.626 INFO 21062 --- [ocp-schedules-7,9bccb3b1faf8f686,6232c92b0baa8070] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:18:28.627 INFO 21062 --- [ocp-schedules-7,9bccb3b1faf8f686,6232c92b0baa8070] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:18:28.653 INFO 21062 --- [ocp-async-3,5e7ba33e60049e15,13696c30425ce93a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:18:28.654 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:28.654 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:18:28.590350+08:00 2024-11-04 14:18:28.782 INFO 21062 --- [ocp-schedules-23,885bcc301b9f0c10,14dd542211d37a22] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:18:28.783 INFO 21062 --- [ocp-schedules-23,885bcc301b9f0c10,14dd542211d37a22] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:18:28.816 INFO 21062 --- [ocp-schedules-10,8537ee391c71b365,1c4a445ec609456a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:18:28.991 INFO 21062 --- [ocp-schedules-24,1b29a924bf0f90bb,736faeed8a41cda3] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:18:28.995 INFO 21062 --- [ocp-schedules-24,1b29a924bf0f90bb,736faeed8a41cda3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:28.995 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:18:29.072 INFO 21062 --- [ocp-schedules-16,7aa992c011641615,04ffaa6b66a0bff7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:18:29.072 INFO 21062 --- [ocp-schedules-16,7aa992c011641615,04ffaa6b66a0bff7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:18:29.137 INFO 21062 --- [ocp-schedules-19,264d5404ef3ad98d,c6c0321fa2819f68] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:18:29.150 WARN 21062 --- [ocp-async-4,19e723704583ee3f,12fb99d0995995d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:29.169 INFO 21062 --- [ocp-schedules-25,2b981c8035a6f988,bff27e154af288cc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:18:29.174 INFO 21062 --- [ocp-schedules-25,2b981c8035a6f988,bff27e154af288cc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:18:29.174 INFO 21062 --- [ocp-schedules-25,2b981c8035a6f988,bff27e154af288cc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:18:29.186 INFO 21062 --- [ocp-async-1,87acd8ccf699c83a,d9bd497324e67834] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:29.489 INFO 21062 --- [ocp-schedules-1,7e39d73611259b09,33e80d1329186ce7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:18:29.489 INFO 21062 --- [ocp-schedules-1,7e39d73611259b09,33e80d1329186ce7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:18:29.491 INFO 21062 --- [ocp-schedules-1,7e39d73611259b09,33e80d1329186ce7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:18:29.491 INFO 21062 --- [ocp-schedules-1,7e39d73611259b09,33e80d1329186ce7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:18:29.491 INFO 21062 --- [ocp-schedules-1,7e39d73611259b09,33e80d1329186ce7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:18:29.491 INFO 21062 --- [ocp-schedules-1,7e39d73611259b09,33e80d1329186ce7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:18:29.983 INFO 21062 --- [prometheus-minute-schedule-1,95fa9d2ddd5a7bef,88b2dfdc24e303a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:18:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:18:30.394 INFO 21062 --- [ocp-schedules-15,8d5ed927725711e8,f72d2ef1dcd648b4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:18:31.151 WARN 21062 --- [ocp-async-1,014d1749f7fa61fe,c92bb425d214ac89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:31.186 INFO 21062 --- [ocp-async-2,169092d90fc14ec7,10ded2735f9f7b2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:31.368 INFO 21062 --- [ocp-schedules-5,f7a7780cec71f7cf,fbccfde2197b6093] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:18:31.374 INFO 21062 --- [ocp-schedules-5,f7a7780cec71f7cf,fbccfde2197b6093] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:18:31.374 INFO 21062 --- [ocp-schedules-5,f7a7780cec71f7cf,fbccfde2197b6093] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:18:31.413 INFO 21062 --- [ocp-schedules-16,0badc3e06ca3a5d1,1005aed48da85cee] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:18:31.413 INFO 21062 --- [ocp-schedules-16,0badc3e06ca3a5d1,1005aed48da85cee] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:18:31.441 INFO 21062 --- [ocp-schedules-16,0badc3e06ca3a5d1,1005aed48da85cee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:18:31.441 INFO 21062 --- [ocp-schedules-16,0badc3e06ca3a5d1,1005aed48da85cee] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:18:31.441 INFO 21062 --- [ocp-schedules-23,c783159e28122206,73866c9e676312a8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:18:31.623 INFO 21062 --- [ocp-schedules-22,b3d9805839be6d5c,87690876e104ee43] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:18:31.623+08:00, clusters=[1] 2024-11-04 14:18:31.625 INFO 21062 --- [ocp-schedules-22,b3d9805839be6d5c,87690876e104ee43] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:18:31.654 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:31.655 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:31.760 INFO 21062 --- [ocp-schedules-21,398db8af8c381f2e,2dd3f4a94b7942fc] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:18:31.764 INFO 21062 --- [ocp-schedules-21,398db8af8c381f2e,2dd3f4a94b7942fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:31.781 INFO 21062 --- [ocp-schedules-9,201cbcbb96affaba,4838382449bd41fe] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:18:31.781+08:00, clusters=[1] 2024-11-04 14:18:31.785 INFO 21062 --- [ocp-schedules-9,201cbcbb96affaba,4838382449bd41fe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:18:31.823 ERROR 21062 --- [ocp-schedules-21,398db8af8c381f2e,2dd3f4a94b7942fc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:18:31.823 INFO 21062 --- [ocp-schedules-21,398db8af8c381f2e,2dd3f4a94b7942fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:18:32.008 INFO 21062 --- [ocp-schedules-6,fc36846ad2c57204,f83f65862f6dc9aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:18:32.668 INFO 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:18:32.669 INFO 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:18:32.669 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:18:32.669 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:18:32.679 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:18:32.679 INFO 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:18:32.679 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:18:32.681 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:18:32.681 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:18:32.681 WARN 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:18:32.682 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:18:32.682 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:18:32.682 INFO 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:18:32.683 WARN 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:18:32.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:18:32.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:18:32.694 INFO 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:18:32.694 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:18:32.694 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:18:32.705 ERROR 21062 --- [ocp-schedules-19,c9933da625d68cef,615ccddb4cc7c19d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:18:32.983 INFO 21062 --- [prometheus-minute-schedule-1,9233b1699af79717,b53346546e6a0ee1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:33.151 WARN 21062 --- [ocp-async-2,5ecaaee5dc1fca05,e4ffb60869b69eca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:33.187 INFO 21062 --- [ocp-async-8,9d07d0b890fe6f82,9054c103da1ccce2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:34.656 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:34.656 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:35.151 WARN 21062 --- [ocp-async-8,46f71b92c66da508,3827c4d59b9a5123] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:35.187 INFO 21062 --- [ocp-async-6,c02366719877fa36,85d8a7816d730732] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:35.776 INFO 21062 --- [ocp-schedules-12,0b16f2c2f2642bda,5eee117674688f3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:18:35.777 INFO 21062 --- [ocp-schedules-12,1be7d552124d9890,c46dcd36e06a9d59] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:18:35.781 INFO 21062 --- [ocp-schedules-12,1be7d552124d9890,c46dcd36e06a9d59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:35.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:18:35.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:18:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4c40301858b2347f,793135e33d5deb41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:37.151 WARN 21062 --- [ocp-async-6,218121e1557051af,f9c0227f08630127] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:37.187 INFO 21062 --- [ocp-async-3,b72ba26b38c76f60,472ae77ee84c9435] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:37.656 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:37.657 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:38.299 INFO 21062 --- [ocp-schedules-3,498085dcfbc333ba,c09bc911a565a086] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:18:38.303 INFO 21062 --- [ocp-schedules-3,498085dcfbc333ba,c09bc911a565a086] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:38.389 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:18:38.394 ERROR 21062 --- [ocp-schedules-3,498085dcfbc333ba,c09bc911a565a086] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:18:38.394 INFO 21062 --- [ocp-schedules-3,498085dcfbc333ba,c09bc911a565a086] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:18:38.504 INFO 21062 --- [ocp-schedules-18,1229d2a39001c772,b45cd4fa515de16f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:18:38.504 INFO 21062 --- [ocp-schedules-4,be6840ed0e3c6148,88cf44a890ecd57c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:18:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:18:38.590580+08:00, lastProcessTime=2024-11-04T14:18:28.590350+08:00 2024-11-04 14:18:38.653 INFO 21062 --- [ocp-async-5,8d707219bdffa8ee,32974c3c48f8c9e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:18:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:18:38.590580+08:00 2024-11-04 14:18:38.983 INFO 21062 --- [prometheus-minute-schedule-1,2cc14b14a606ee0c,29772bfa733581f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:38.996 INFO 21062 --- [ocp-schedules-24,1b29a924bf0f90bb,736faeed8a41cda3] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:18:39.099 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:18:39.099 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:18:39.108 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:18:39.108 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:18:39.108 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:18:39.108 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:18:39.114 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:18:39.114 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:18:39.114 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:18:39.114 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:18:39.151 WARN 21062 --- [ocp-async-9,c9620cc2a6c33a4b,49fed344e3751ca8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:39.153 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:18:39.162 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:18:39.162 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:18:39.162 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:18:39.162 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:18:39.162 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:18:39.207 INFO 21062 --- [ocp-async-8,bda6fdbebf1866b1,5862e4616d648ef3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:39.211 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:18:39.726 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:18:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:18:40.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:18:40.276 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.162657519S 2024-11-04 14:18:40.277 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:18:40.371 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:18:40.372 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:18:40.398 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:18:40.398 INFO 21062 --- [ocp-schedules-11,da8c27d6014a079b,7f1497f2f9a1dc61] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:18:40.458 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:18:40.458 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:18:40.458 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:18:40.458 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:18:40.458 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:18:40.459 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:18:40.469 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:18:40.638 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] o.s.boot.SpringApplication : Started application in 1.529 seconds (JVM running for 6534.317) 2024-11-04 14:18:40.638 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.530076978S 2024-11-04 14:18:40.658 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:40.658 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:41.095 INFO 21062 --- [ocp-schedules-29,773f57fc932627d7,8033573ac0ba9471] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:18:41.152 WARN 21062 --- [ocp-async-8,f8bcb197193e7ac9,051530ae72db35b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:41.207 INFO 21062 --- [ocp-async-6,04b8a7c6c6a0163a,a37a72c8eacb9f2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:18:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:18:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:18:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:18:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:18:41.856 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:18:41.856 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 14:18:41.983 INFO 21062 --- [prometheus-minute-schedule-1,048a0b65d472c781,3a750e229aa8089e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:42.051 INFO 21062 --- [ocp-schedules-23,887ab2be7f580122,f67f3d79602e25bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:18:43.152 WARN 21062 --- [ocp-async-6,e5f22653ddd4aec0,06bda1d199fc4ff5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:43.207 INFO 21062 --- [ocp-async-3,d7f5b7ea42d62dca,ec1b3e7669040108] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:43.659 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:43.659 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:44.983 INFO 21062 --- [prometheus-minute-schedule-1,660a5fa23ffe5bc5,8fe4ed4a3300d95a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:45.152 WARN 21062 --- [ocp-async-3,1107e93ea82e8ea5,06d48395425cda0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:45.207 INFO 21062 --- [ocp-async-10,18a8851aea1d7cbf,7359cbe77fec9991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:46.659 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:46.661 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:47.152 WARN 21062 --- [ocp-async-10,6442f77e4a8ab8bf,1049142c8255867b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:47.208 INFO 21062 --- [ocp-async-5,f4012ee9196a81c5,f48aee6c82fd3d44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:47.983 INFO 21062 --- [prometheus-minute-schedule-1,c1f50e09c2b2a2c7,1c5a53e034148358] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:18:48.590421+08:00, lastProcessTime=2024-11-04T14:18:38.590580+08:00 2024-11-04 14:18:48.653 INFO 21062 --- [ocp-async-9,bed33bff3bb1e746,2dc1a6ec2398bfb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:18:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:18:48.590421+08:00 2024-11-04 14:18:49.152 WARN 21062 --- [ocp-async-6,1fee1edeb2e68bc9,82227bf671202e5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:49.208 INFO 21062 --- [ocp-async-2,74ad41fff0891c04,0a60cf018f0ddec6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:49.661 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:49.662 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:50.401 INFO 21062 --- [ocp-schedules-3,7ef3814a92799ace,8a7a3dc9265b1e06] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:18:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d83cf69462dc6167,a4a199ad10531c8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:51.152 WARN 21062 --- [ocp-async-3,6af7d59fd2c5b42b,2cb2cc65d24c7372] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:51.208 INFO 21062 --- [ocp-async-4,8845f8c3d3974168,02d3f3b9282a9d29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:51.376 INFO 21062 --- [ocp-schedules-28,6a6422ead8be2c33,b8845d538a2c9539] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:18:51.383 INFO 21062 --- [ocp-schedules-28,6a6422ead8be2c33,b8845d538a2c9539] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:18:51.384 INFO 21062 --- [ocp-schedules-28,6a6422ead8be2c33,b8845d538a2c9539] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:18:52.080 INFO 21062 --- [ocp-schedules-10,fcf62b87d2fff37f,05020da183bf9be9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:18:52.662 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:52.663 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:53.152 WARN 21062 --- [ocp-async-10,4585f5343ae8b43d,b96e55e31fabe0ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:53.208 INFO 21062 --- [ocp-async-5,683757fcfd4bd5f7,78c500a6568bdd00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a43fa0edcdbf7e11,3a07dcc8c2ce51db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:55.153 WARN 21062 --- [ocp-async-1,a0c5eeceb6505e78,c61bbbf4620cd68a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:55.208 INFO 21062 --- [ocp-async-7,11777421deeb04b3,a2b5cf27b54536c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:55.663 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:55.664 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:56.983 INFO 21062 --- [prometheus-minute-schedule-1,efd52765f6840528,0c20da19b5aeefce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:18:57.153 WARN 21062 --- [ocp-async-2,49582b19d8caf7c9,23128dcd9ce65ed9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:57.209 INFO 21062 --- [ocp-async-9,36da7b1d4b6fb33a,8fefbb55d8c84769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:58.500 INFO 21062 --- [ocp-schedules-10,506bfa9af95575ac,28835dfbc7bf7d10] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:18:58.500 INFO 21062 --- [ocp-schedules-1,c7178544a674fa86,222e9f984a594b63] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:18:58.502 INFO 21062 --- [ocp-schedules-11,da9ac619d537d643,9a7055dd5bceb1b5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:18:58.503 INFO 21062 --- [ocp-schedules-10,506bfa9af95575ac,28835dfbc7bf7d10] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:58.503 INFO 21062 --- [ocp-schedules-1,c7178544a674fa86,222e9f984a594b63] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:18:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:18:58.505 INFO 21062 --- [ocp-schedules-5,ae5f77eb98759f43,3a809c7f49f50ff2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:18:58.509 INFO 21062 --- [ocp-schedules-22,c969da2565aaea24,bd0a95e3c9da03b9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:18:58.509 INFO 21062 --- [ocp-schedules-22,c969da2565aaea24,bd0a95e3c9da03b9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:18:58.509 INFO 21062 --- [ocp-schedules-14,ace74aa1f8d814bc,e65ae018e33cf08b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:18:58.509 INFO 21062 --- [ocp-schedules-18,429ea69e98662a58,1d0b7b2e7d5e6b74] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:18:58.509 INFO 21062 --- [ocp-schedules-14,ace74aa1f8d814bc,e65ae018e33cf08b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:18:58.510 INFO 21062 --- [ocp-schedules-18,429ea69e98662a58,1d0b7b2e7d5e6b74] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:18:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:18:58.589730+08:00, lastProcessTime=2024-11-04T14:18:48.590421+08:00 2024-11-04 14:18:58.654 INFO 21062 --- [ocp-async-6,b7ac544e329b81c8,486acad66efb485c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:18:58.664 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:18:58.665 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:18:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:18:58.589730+08:00 2024-11-04 14:18:59.141 INFO 21062 --- [ocp-schedules-24,e3298fadc2df87a8,cb06d18775102280] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:18:59.153 WARN 21062 --- [ocp-async-10,a1a2ab3a53be1954,05aed6c5e6079063] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:18:59.209 INFO 21062 --- [ocp-async-5,ae0769485d4a6008,6c686a3d6c950767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:18:59.983 INFO 21062 --- [prometheus-minute-schedule-1,5fe9425af9edc0fe,c8d5aa4686d1346d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701140, interval=60 2024-11-04 14:19:00.404 INFO 21062 --- [ocp-schedules-24,1eeaf20b01fdced8,17eb402ff85554db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:19:01.153 WARN 21062 --- [ocp-async-1,ec81c8d72eab5afd,b6cb8d097055465d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:01.209 INFO 21062 --- [ocp-async-7,91b726de2edb12bf,87c9fe53d4c07813] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:01.665 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:01.665 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:02.106 INFO 21062 --- [ocp-schedules-9,cc84432782b941aa,a72cbfc166a8b9fc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:19:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c051eaada878a98f,0f6ded81d65b622d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:03.153 WARN 21062 --- [ocp-async-2,3225519da07e7eea,b75f6370cb4bcb3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:03.209 INFO 21062 --- [ocp-async-9,dd72b4109a45ecd2,cb35af12e607c46c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:04.666 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:04.667 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:05.153 WARN 21062 --- [ocp-async-4,d3e3e3756b4ef0b8,1e7b9ada6a290595] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:05.209 INFO 21062 --- [ocp-async-8,250cc26eef554aad,1a3a4b7f4c6387a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:05.811 INFO 21062 --- [ocp-schedules-12,1be7d552124d9890,c46dcd36e06a9d59] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:19:05.812 INFO 21062 --- [ocp-schedules-12,b8283286cfbb190f,d950e8972b12a798] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:19:05.815 INFO 21062 --- [ocp-schedules-12,b8283286cfbb190f,d950e8972b12a798] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:05.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:19:05.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:19:05.983 INFO 21062 --- [prometheus-minute-schedule-1,6af5c52a49ee91fc,17a330553b7ce4c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:07.154 WARN 21062 --- [ocp-async-8,00f3944e4b06ce0a,08bfe848e2756410] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:07.209 INFO 21062 --- [ocp-async-6,4ffb237f950b5c99,eb026169781aacef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:07.667 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:07.668 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:08.504 INFO 21062 --- [ocp-schedules-1,c7178544a674fa86,222e9f984a594b63] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:19:08.504 INFO 21062 --- [ocp-schedules-10,506bfa9af95575ac,28835dfbc7bf7d10] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:19:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:19:08.589943+08:00, lastProcessTime=2024-11-04T14:18:58.589730+08:00 2024-11-04 14:19:08.656 INFO 21062 --- [ocp-async-4,8794108c9344d2cb,d0a4f6d8e5d6b53a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:19:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:19:08.589943+08:00 2024-11-04 14:19:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d4abb71f1a93f242,56239bed78087e3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:09.154 WARN 21062 --- [ocp-async-7,0bfad6222fc871d3,dee64cf84ed00656] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:09.209 INFO 21062 --- [ocp-async-9,72b5771295d001ec,7e5b7ee37ec592a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:19:10.004 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:19:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:19:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:19:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:19:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:19:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:19:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:19:10.407 INFO 21062 --- [ocp-schedules-18,38cf1f54c41282e2,c7b3f92a64dcfc1e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:19:10.668 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:10.669 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:11.154 WARN 21062 --- [ocp-async-9,2ba54537d64b11f4,56511441fd171fd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:11.210 INFO 21062 --- [ocp-async-8,7904919e2a864cee,47a95db63c7920b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:11.386 INFO 21062 --- [ocp-schedules-7,61e3de883573a035,1a67f6602b38a631] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:19:11.392 INFO 21062 --- [ocp-schedules-7,61e3de883573a035,1a67f6602b38a631] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:19:11.392 INFO 21062 --- [ocp-schedules-7,61e3de883573a035,1a67f6602b38a631] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:19:11.532 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=1, affectRows=1 2024-11-04 14:19:11.532 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=0, validCount=0 2024-11-04 14:19:11.533 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=10, affectRows=10 2024-11-04 14:19:11.533 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=8, affectRows=8 2024-11-04 14:19:11.533 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=8, validCount=8 2024-11-04 14:19:11.533 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=10, validCount=10 2024-11-04 14:19:11.533 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=31, affectRows=31 2024-11-04 14:19:11.533 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=48, affectRows=48 2024-11-04 14:19:11.534 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=64, affectRows=64 2024-11-04 14:19:11.534 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=31, validCount=31 2024-11-04 14:19:11.534 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=48, validCount=48 2024-11-04 14:19:11.534 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=64, validCount=64 2024-11-04 14:19:11.534 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=13, affectRows=13 2024-11-04 14:19:11.534 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=13, validCount=13 2024-11-04 14:19:11.535 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=43, affectRows=43 2024-11-04 14:19:11.535 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=25, affectRows=25 2024-11-04 14:19:11.535 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=43, validCount=43 2024-11-04 14:19:11.535 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=25, validCount=25 2024-11-04 14:19:11.537 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701080, nodeCount=153, affectRows=153 2024-11-04 14:19:11.537 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701080, nodeCount=151, validCount=151 2024-11-04 14:19:11.983 INFO 21062 --- [prometheus-minute-schedule-1,3c1f40687f894ae8,04b872f7d707f2bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:12.132 INFO 21062 --- [ocp-schedules-6,7a85859f18b3e1a5,7e1a2a6788de34d3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:19:13.154 WARN 21062 --- [ocp-async-8,a38da672d6f1354d,b9c76484994b0fc3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:13.210 INFO 21062 --- [ocp-async-6,e595e8941f9e3247,0cd5898958c14ac9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:13.669 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:13.670 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:13.984 INFO 21062 --- [prometheus-cache-monitor1,149345c4b94fddc1,ee8df9cfc83be87a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:19:13.984 INFO 21062 --- [prometheus-cache-monitor1,149345c4b94fddc1,ee8df9cfc83be87a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:19:14.983 INFO 21062 --- [prometheus-minute-schedule-1,94515e7f800a4855,ae8ea7f230c11ace] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:15.010 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:15:14.987, end=2024-11-04T14:17:15.007} 2024-11-04 14:19:15.021 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:19:15.021 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:19:15.154 WARN 21062 --- [ocp-async-6,e0cb4de3513f08f8,acc8da12f91dd651] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:15.210 INFO 21062 --- [ocp-async-2,51cc30dd2f1726a2,ba7598365e4b4872] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:16.670 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:16.671 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:17.155 WARN 21062 --- [ocp-async-2,b214b48750226cea,2f8245d99dd9ff6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:17.210 INFO 21062 --- [ocp-async-4,680d31c30b363928,6f6101cd638c836e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:17.983 INFO 21062 --- [prometheus-minute-schedule-1,375534c94f41b367,36c61e4c2dda25d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:19:18.590790+08:00, lastProcessTime=2024-11-04T14:19:08.589943+08:00 2024-11-04 14:19:18.654 INFO 21062 --- [ocp-async-7,f4af2f18ce746363,d3627256fe6bbfd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:19:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:19:18.590790+08:00 2024-11-04 14:19:19.155 WARN 21062 --- [ocp-async-8,e7e2690edf44822e,e5a0b6cb7530299d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:19.210 INFO 21062 --- [ocp-async-6,824c271e676de9dc,b1d8ddeca3b69e3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:19.671 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:19.672 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:20.410 INFO 21062 --- [ocp-schedules-22,4c6c62aeb85658d1,b73fa703aeec201c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:19:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ea679c52ad0c9763,35ae50bfec0ec65a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:21.155 WARN 21062 --- [ocp-async-6,c3dadd5879defd30,f7379f3c26d189ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:21.210 INFO 21062 --- [ocp-async-2,fb941aff56998cde,900b3229db34940a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:22.173 INFO 21062 --- [ocp-schedules-30,04996c04e83766d6,a5e4a18b9d18db50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:19:22.672 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:22.672 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:23.155 WARN 21062 --- [ocp-async-2,01cba522c53f5193,7fbd9cf4189d7eff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:23.211 INFO 21062 --- [ocp-async-4,f3d50bfd28b9fb34,386964d9551173cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:23.983 INFO 21062 --- [prometheus-minute-schedule-1,7d0f4fcba140e5b3,aad770cfef6fd2e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:25.155 WARN 21062 --- [ocp-async-4,ba0d485ec96b4829,25a481607d4824ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:25.211 INFO 21062 --- [ocp-async-5,caa6c2fde8bf6250,92c86881963a7d72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:25.673 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:25.673 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:26.983 INFO 21062 --- [prometheus-minute-schedule-1,707871a4b81493b9,13936e4a74fadd66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:26.984 INFO 21062 --- [prometheus-hour-schedule-1,3e170c404188aeeb,6e98426f0b040003] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:19:27.156 WARN 21062 --- [ocp-async-5,10e97cc40273a3c5,0359b691e1235670] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:27.211 INFO 21062 --- [ocp-async-3,e9f31d9894533a37,1ffb26984c1cceb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:28.498 INFO 21062 --- [ocp-schedules-17,e11dfa17fc2111b4,a31069d242ce6e45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:19:28.499 INFO 21062 --- [ocp-schedules-28,e18e547c8fe65631,6d89bcf8a47bba87] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:19:28.500 INFO 21062 --- [ocp-schedules-7,7379650c57c1c487,a3ff5dc5d023fd36] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:19:28.500 INFO 21062 --- [ocp-schedules-10,944e4e02af1fded9,e5e68ad91028f413] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:19:28.502 INFO 21062 --- [ocp-schedules-17,e11dfa17fc2111b4,a31069d242ce6e45] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:19:28.502 INFO 21062 --- [ocp-schedules-21,4b005f831c6efe91,1560e2505662afad] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:19:28.503 INFO 21062 --- [ocp-schedules-7,7379650c57c1c487,a3ff5dc5d023fd36] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:19:28.504 INFO 21062 --- [ocp-schedules-10,944e4e02af1fded9,e5e68ad91028f413] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:28.504 INFO 21062 --- [ocp-schedules-13,c3c4aec13fbf9884,1bf96f4b0f7f8ec8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:19:28.505 INFO 21062 --- [ocp-schedules-28,e18e547c8fe65631,6d89bcf8a47bba87] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:19:28.509 INFO 21062 --- [ocp-schedules-30,dc92028f22c1200d,007fae547d91119d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:19:28.509 INFO 21062 --- [ocp-schedules-30,dc92028f22c1200d,007fae547d91119d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:19:28.509 INFO 21062 --- [ocp-schedules-6,daca7c7173e83109,dc7396dda1d83dfa] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:19:28.509 INFO 21062 --- [ocp-schedules-6,daca7c7173e83109,dc7396dda1d83dfa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:19:28.510 INFO 21062 --- [ocp-schedules-27,9c0947645a712814,8a3d4d54a0adf6e2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:19:28.510 INFO 21062 --- [ocp-schedules-27,9c0947645a712814,8a3d4d54a0adf6e2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:19:28.586 INFO 21062 --- [ocp-schedules-9,6a7c5678b5cd1fba,bebc9f319c46ab3a] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:19:28.586+08:00, tenantSize=0 2024-11-04 14:19:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:19:28.589955+08:00, lastProcessTime=2024-11-04T14:19:18.590790+08:00 2024-11-04 14:19:28.654 INFO 21062 --- [ocp-async-5,5361a03bfee452db,33ae06953a446f35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:19:28.673 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:28.674 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:19:28.589955+08:00 2024-11-04 14:19:28.819 INFO 21062 --- [ocp-schedules-4,37e05c7508aaa9be,197847ed1a1f8b99] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:19:29.076 INFO 21062 --- [ocp-schedules-3,862784bce92a9083,441de7ba8c376c9f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:19:29.076 INFO 21062 --- [ocp-schedules-3,862784bce92a9083,441de7ba8c376c9f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:19:29.144 INFO 21062 --- [ocp-schedules-30,4749e1be1b09af4b,7a1161e76f6a64a1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:19:29.156 WARN 21062 --- [ocp-async-10,13c23fc3c8ef177d,96fd3a82a69cfc64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:29.174 INFO 21062 --- [ocp-schedules-17,f045c069a25c9183,6951b00891a69901] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:19:29.179 INFO 21062 --- [ocp-schedules-17,f045c069a25c9183,6951b00891a69901] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:19:29.179 INFO 21062 --- [ocp-schedules-17,f045c069a25c9183,6951b00891a69901] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:19:29.211 INFO 21062 --- [ocp-async-1,536b3bcc764cbf7b,100f4325b6de2941] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:29.318 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:19:29.318 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:19:29.326 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:19:29.326 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:19:29.326 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:19:29.336 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:19:29.336 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:19:29.336 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:19:29.339 INFO 21062 --- [ocp-schedules-26,9ecc9b02c30c3360,10569c733330b09f] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:19:29.492 INFO 21062 --- [ocp-schedules-29,533bcf2dd1b7cec8,f7d724e6cc9c6d31] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:19:29.492 INFO 21062 --- [ocp-schedules-29,533bcf2dd1b7cec8,f7d724e6cc9c6d31] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:19:29.500 INFO 21062 --- [ocp-schedules-29,533bcf2dd1b7cec8,f7d724e6cc9c6d31] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:19:29.500 INFO 21062 --- [ocp-schedules-29,533bcf2dd1b7cec8,f7d724e6cc9c6d31] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:19:29.506 INFO 21062 --- [ocp-schedules-29,533bcf2dd1b7cec8,f7d724e6cc9c6d31] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:19:29.506 INFO 21062 --- [ocp-schedules-29,533bcf2dd1b7cec8,f7d724e6cc9c6d31] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:19:29.983 INFO 21062 --- [prometheus-minute-schedule-1,cbceda23b62ccc4b,1ef3602bfc604ceb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:19:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:19:30.414 INFO 21062 --- [ocp-schedules-1,2a0106915224d331,8b0bc0906deb6f20] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:19:31.156 WARN 21062 --- [ocp-async-1,c835ce35baaeeafd,537d5d6d6c236636] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:31.212 INFO 21062 --- [ocp-async-7,2fcca2413718ac4b,cff0912d14cab16a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:31.395 INFO 21062 --- [ocp-schedules-6,704db4584b61bb97,c634430051bc14e1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:19:31.400 INFO 21062 --- [ocp-schedules-6,704db4584b61bb97,c634430051bc14e1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:19:31.400 INFO 21062 --- [ocp-schedules-6,704db4584b61bb97,c634430051bc14e1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:19:31.441 INFO 21062 --- [ocp-schedules-23,ad6fe47454deed14,8017392a059f4797] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:19:31.441 INFO 21062 --- [ocp-schedules-23,ad6fe47454deed14,8017392a059f4797] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:19:31.469 INFO 21062 --- [ocp-schedules-23,ad6fe47454deed14,8017392a059f4797] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:19:31.469 INFO 21062 --- [ocp-schedules-23,ad6fe47454deed14,8017392a059f4797] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:19:31.472 INFO 21062 --- [ocp-schedules-2,9f970c7fa72515f0,9127f4d93b36864a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:19:31.653 INFO 21062 --- [ocp-schedules-24,f66f2cfa782537bd,6ab07afbc559fdb6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:19:31.653+08:00, clusters=[1] 2024-11-04 14:19:31.656 INFO 21062 --- [ocp-schedules-24,f66f2cfa782537bd,6ab07afbc559fdb6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:19:31.674 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:31.675 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:31.813 INFO 21062 --- [ocp-schedules-17,d8b12bfc4a019885,9dfbf37f2d3ab1ca] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:19:31.813+08:00, clusters=[1] 2024-11-04 14:19:31.816 INFO 21062 --- [ocp-schedules-17,d8b12bfc4a019885,9dfbf37f2d3ab1ca] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:19:32.197 INFO 21062 --- [ocp-schedules-4,89a60cd524b52601,dfb253203f5068d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:19:32.715 INFO 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:19:32.715 INFO 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:19:32.715 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:19:32.715 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:19:32.724 INFO 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:19:32.725 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:19:32.725 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:19:32.727 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:19:32.727 WARN 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:19:32.727 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:19:32.727 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:19:32.727 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:19:32.727 INFO 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:19:32.728 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:19:32.728 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:19:32.728 WARN 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:19:32.736 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:19:32.736 INFO 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:19:32.736 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:19:32.746 ERROR 21062 --- [ocp-schedules-30,bb030ff2c3198795,d64263ec3124c00f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:19:32.983 INFO 21062 --- [prometheus-minute-schedule-1,4a80ef651d1b7bc4,17187026eb0fa0ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:33.156 WARN 21062 --- [ocp-async-7,ecb6b4b91b03efe0,d11a70b38ce3dd88] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:33.212 INFO 21062 --- [ocp-async-9,32d4d89527eedb7f,7b8185e1248b6c85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:34.675 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:34.677 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:35.156 WARN 21062 --- [ocp-async-9,26adbf7d44b8d109,704f8b29463cad25] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:35.212 INFO 21062 --- [ocp-async-4,65c7c7f567d93e23,297cc14bc7c310a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:35.846 INFO 21062 --- [ocp-schedules-12,b8283286cfbb190f,d950e8972b12a798] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:19:35.846 INFO 21062 --- [ocp-schedules-12,15af9f1c719ef100,3aad1d79e853f5bd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:19:35.851 INFO 21062 --- [ocp-schedules-12,15af9f1c719ef100,3aad1d79e853f5bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:35.882 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:19:35.882 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:19:35.983 INFO 21062 --- [prometheus-minute-schedule-1,bd773380d425d536,c7b68d259d99d082] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:37.156 WARN 21062 --- [ocp-async-4,95b68f78007482f4,bb7e69b45ac4623f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:37.212 INFO 21062 --- [ocp-async-6,d68687d3c4adfb2b,610c7f028dcc2def] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:37.677 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:37.678 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:38.394 INFO 21062 --- [ocp-schedules-3,cea60ab9c8eefde2,b95c7dff70b4f8c3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:19:38.399 INFO 21062 --- [ocp-schedules-3,cea60ab9c8eefde2,b95c7dff70b4f8c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:38.484 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:19:38.488 ERROR 21062 --- [ocp-schedules-3,cea60ab9c8eefde2,b95c7dff70b4f8c3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:19:38.488 INFO 21062 --- [ocp-schedules-3,cea60ab9c8eefde2,b95c7dff70b4f8c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:19:38.504 INFO 21062 --- [ocp-schedules-7,7379650c57c1c487,a3ff5dc5d023fd36] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:19:38.505 INFO 21062 --- [ocp-schedules-10,944e4e02af1fded9,e5e68ad91028f413] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:19:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:19:38.589371+08:00, lastProcessTime=2024-11-04T14:19:28.589955+08:00 2024-11-04 14:19:38.654 INFO 21062 --- [ocp-async-8,d5daa336c06e2184,6cf09b629fc77b3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:19:38.751 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:19:38.589371+08:00 2024-11-04 14:19:38.983 INFO 21062 --- [prometheus-minute-schedule-1,4ef77aa8e5f4d2c7,282570a01025b0c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:38.996 INFO 21062 --- [ocp-schedules-19,e862e40ffeec0e7e,45fc56b7420f5065] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:19:38.999 INFO 21062 --- [ocp-schedules-19,e862e40ffeec0e7e,45fc56b7420f5065] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:39.000 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:19:39.157 WARN 21062 --- [ocp-async-3,c1c3f4fd8a83a837,a971e287683bfd22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:39.212 INFO 21062 --- [ocp-async-10,bd5ac7ca2bcfcdf0,a76281cbec7f4769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:19:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:19:40.416 INFO 21062 --- [ocp-schedules-18,b0e345de64dc5b3f,3d255e8230375f1d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:19:40.678 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:40.679 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:41.157 WARN 21062 --- [ocp-async-10,656e1172af5c03a5,7f764db70f4d8841] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:41.212 INFO 21062 --- [ocp-async-4,8b06cbedc1c2d761,b681900f415a8ac1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:41.983 INFO 21062 --- [prometheus-minute-schedule-1,4908ff28fd972c72,18ab87b6bb66404f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:42.236 INFO 21062 --- [ocp-schedules-9,78e1172a66bc6c79,7367198182b337bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:19:43.157 WARN 21062 --- [ocp-async-4,c5c5124a1c3628bc,9004a0dde0df9256] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:43.213 INFO 21062 --- [ocp-async-6,31c4f8c2644e5a03,f888f8a7aa906181] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:43.679 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:43.680 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3c53d60514246e9f,a42614bc8a30f01e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:45.157 WARN 21062 --- [ocp-async-6,e091e86e1555a550,1d09afb23e9581c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:45.213 INFO 21062 --- [ocp-async-2,37e4bba5a14ed03d,933aa2842365f041] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:46.680 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:46.681 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:47.157 WARN 21062 --- [ocp-async-2,c97a12f7e5cc7063,5ab6dbb1f11d065a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:47.213 INFO 21062 --- [ocp-async-8,d2cfbb78366154dc,a0019d41978cdab6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:47.983 INFO 21062 --- [prometheus-minute-schedule-1,04ca68b0ae3c6846,5367ed6c98639bb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:19:48.592995+08:00, lastProcessTime=2024-11-04T14:19:38.589371+08:00 2024-11-04 14:19:48.655 INFO 21062 --- [ocp-async-3,d16818eecb1774fc,a80449c6523f3bd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:19:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:19:48.592995+08:00 2024-11-04 14:19:49.000 INFO 21062 --- [ocp-schedules-19,e862e40ffeec0e7e,45fc56b7420f5065] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:19:49.158 WARN 21062 --- [ocp-async-4,4abf92678586ff1a,79b92770d3d8b76e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:49.213 INFO 21062 --- [ocp-async-7,2548322883769376,88b8204facac80d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:49.681 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:49.682 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:50.419 INFO 21062 --- [ocp-schedules-22,0757c25f2256e008,a4219d89cceaefa2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:19:50.983 INFO 21062 --- [prometheus-minute-schedule-1,fc896c032fd0812f,703188976e72ab0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:51.158 WARN 21062 --- [ocp-async-6,fc73b867a0d75bbb,b4b3dc1d13922f36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:51.213 INFO 21062 --- [ocp-async-9,e99435b2cc37c4d6,5fcaf78eab593ddc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:51.403 INFO 21062 --- [ocp-schedules-11,cc4eec5bb21a4258,f58e9c161f6ea665] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:19:51.409 INFO 21062 --- [ocp-schedules-11,cc4eec5bb21a4258,f58e9c161f6ea665] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:19:51.409 INFO 21062 --- [ocp-schedules-11,cc4eec5bb21a4258,f58e9c161f6ea665] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:19:52.264 INFO 21062 --- [ocp-schedules-29,c259ef334e35e027,5a8f6f6ddf45e712] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:19:52.682 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:52.682 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:53.158 WARN 21062 --- [ocp-async-2,5c2d3ef67a56d128,b370e3a0bafb94e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:53.213 INFO 21062 --- [ocp-async-1,5a2f1816b5d26f0f,3e6390f3d4a89574] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4536d7eb0b4a4190,7680758ac67df342] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:55.158 WARN 21062 --- [ocp-async-8,17fac93a7d27f393,fc9a4c44c68ce22a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:55.214 INFO 21062 --- [ocp-async-5,827fff53636e4eb9,51bdd678fe6cb884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:55.683 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:55.683 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fb7c5d80aa8c5360,0fb890a450351900] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:19:57.158 WARN 21062 --- [ocp-async-7,2aa17b5f3b3e9070,b04c2fdb5b1d9143] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:57.214 INFO 21062 --- [ocp-async-3,3becbdb9a3a2fa3e,b2f11d8474a1bf34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:58.499 INFO 21062 --- [ocp-schedules-2,1b321b74c1c8888b,de610db342e3d061] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:19:58.500 INFO 21062 --- [ocp-schedules-9,1ec7e247a24f615b,366c6fcd8266b83b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:19:58.502 INFO 21062 --- [ocp-schedules-24,52748f4cf81a25dc,f57ce07d9ba02816] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:19:58.504 INFO 21062 --- [ocp-schedules-9,1ec7e247a24f615b,366c6fcd8266b83b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:58.505 INFO 21062 --- [ocp-schedules-2,1b321b74c1c8888b,de610db342e3d061] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:19:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:19:58.507 INFO 21062 --- [ocp-schedules-11,68686b2aff0dca5c,ae52ad6e183ce42d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:19:58.509 INFO 21062 --- [ocp-schedules-5,2c21530247a9aad9,7896df23bac4aa3d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:19:58.510 INFO 21062 --- [ocp-schedules-8,2522b0d211c70c42,d419f9409ff0dcd1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:19:58.510 INFO 21062 --- [ocp-schedules-5,2c21530247a9aad9,7896df23bac4aa3d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:19:58.510 INFO 21062 --- [ocp-schedules-8,2522b0d211c70c42,d419f9409ff0dcd1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:19:58.511 INFO 21062 --- [ocp-schedules-28,0de96d196e939d96,6ad0a66c56e8f978] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:19:58.511 INFO 21062 --- [ocp-schedules-28,0de96d196e939d96,6ad0a66c56e8f978] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:19:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:19:58.589297+08:00, lastProcessTime=2024-11-04T14:19:48.592995+08:00 2024-11-04 14:19:58.655 INFO 21062 --- [ocp-async-4,cac7356eac5012a3,5ee5529a13df4860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:19:58.683 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:19:58.684 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:19:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:19:58.589297+08:00 2024-11-04 14:19:59.146 INFO 21062 --- [ocp-schedules-1,96fd96eec01e9ce7,88450b333e89a890] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:19:59.159 WARN 21062 --- [ocp-async-2,4ace7ec27391b8ac,782bb444b48976c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:19:59.214 INFO 21062 --- [ocp-async-1,9c8e6aec0c814350,26ec85dd9596300c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:19:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d1e46034f0522514,27efec56350673ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701200, interval=60 2024-11-04 14:20:00.423 INFO 21062 --- [ocp-schedules-20,2608356acd813f7b,85ef273edf9e241f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:20:00.599 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,38e86f87b31fb905] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:15:01.0, nextRun=2024-11-04 14:20:00.0, currentDate=Mon Nov 04 14:20:00 CST 2024 2024-11-04 14:20:00.599 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,38e86f87b31fb905] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:15:01.0, nextRunTime=2024-11-04 14:20:00.0} 2024-11-04 14:20:00.628 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,38e86f87b31fb905] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:20:00.631 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,38e86f87b31fb905] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:20:00.631 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,38e86f87b31fb905] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:20:01.159 WARN 21062 --- [ocp-async-8,f5ae03559a70a90e,8b3020a0542c8c4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:01.214 INFO 21062 --- [ocp-async-5,cfc23e889f342f61,57ed8edfb4bd78cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:01.684 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:01.685 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:02.311 INFO 21062 --- [ocp-schedules-15,45ecc435385ccce6,96f4ad96c4880d2a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:20:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a950f1d0935cc775,0f56de0748ab11b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:03.159 WARN 21062 --- [ocp-async-7,3291715b18ea65bf,fd47d8eb628df4c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:03.215 INFO 21062 --- [ocp-async-3,b4a9ead065a986d2,f702ded1c4664a17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:04.685 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:04.686 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:04.704 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,effcc6ca67a295b3] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:10:04.0, nextRun=2024-11-04 14:20:04.0, currentDate=Mon Nov 04 14:20:04 CST 2024 2024-11-04 14:20:04.704 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,effcc6ca67a295b3] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 14:10:04.0, nextRunTime=2024-11-04 14:20:04.0} 2024-11-04 14:20:04.709 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,effcc6ca67a295b3] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 14:20:04.709 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,effcc6ca67a295b3] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T14:20:04.709+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 14:20:05.117 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,b07ddc94401ccb86] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 80, operation:EXECUTE, state: RUNNING 2024-11-04 14:20:05.136 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=80, context=Context{parallelIdx=-1, stringMap={task_instance_id=16, task_operation=execute, latest_execution_start_time=2024-11-04T14:20:05.110+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=80}, listMap={}}, executor=10.46.128.24 2024-11-04 14:20:05.140 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:20:05.140 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:20:05.142 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.142 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:20:05.144 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.144 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.144 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.145 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:20:05.159 WARN 21062 --- [ocp-async-9,5df94968d38c3aa1,972efd31f1a6823b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:05.160 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.160 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.161 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.162 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:20:05.175 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.175 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.176 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.176 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:20:05.190 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.191 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.191 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.192 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:20:05.204 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.205 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.205 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.206 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:20:05.211 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.212 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.212 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.213 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:20:05.215 INFO 21062 --- [ocp-async-10,53251eca0b39efac,0e8602b8595bce44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:05.217 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.218 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.218 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.219 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:20:05.223 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.224 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.225 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.225 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:20:05.231 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.232 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.232 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.233 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:20:05.238 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.239 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.239 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.240 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:20:05.245 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.246 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.246 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.247 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:20:05.255 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.256 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.256 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.257 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:20:05.269 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.269 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.270 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.271 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:20:05.275 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.276 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.276 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.277 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:20:05.282 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.283 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.283 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.284 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:20:05.289 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.290 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.290 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.291 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:20:05.295 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.295 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.296 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.297 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:20:05.300 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.301 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.301 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.302 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:20:05.307 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.307 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.307 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.308 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:20:05.312 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.313 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.313 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.314 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:20:05.317 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.318 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.318 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.319 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:20:05.324 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.324 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.325 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.325 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:20:05.329 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.330 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.330 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.331 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:20:05.335 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.335 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.336 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.336 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:20:05.341 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.342 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.342 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.343 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:20:05.347 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.347 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.348 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.348 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:20:05.352 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.353 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.353 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.354 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:20:05.358 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.359 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.359 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.360 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:20:05.364 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.364 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.365 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.365 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:20:05.370 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.370 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.371 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.372 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:20:05.376 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.377 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.377 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.378 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:20:05.382 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.382 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.383 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.383 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:20:05.388 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.388 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.389 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.389 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:20:05.393 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.394 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.394 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.395 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:20:05.402 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.402 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.403 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.404 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:20:05.410 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.410 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.411 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.412 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:20:05.417 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.417 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.418 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.418 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:20:05.423 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.424 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.425 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.425 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:20:05.429 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.429 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.429 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.430 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:20:05.434 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.434 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.435 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.435 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:20:05.438 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.439 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.439 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.440 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:20:05.444 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.444 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.445 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.445 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:20:05.454 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.454 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.455 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.455 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:20:05.475 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.476 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.476 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.477 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:20:05.483 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.483 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.484 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.484 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:20:05.490 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.490 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.490 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.491 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:20:05.497 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.497 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.498 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.498 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:20:05.502 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.502 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.503 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.503 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:20:05.507 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.508 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.508 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.509 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:20:05.513 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:20:05.513 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:20:05.514 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.514 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:20:05.516 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.516 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.516 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.517 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:20:05.522 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.522 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:20:05.523 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:20:05.523 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:20:05.527 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:20:05.536 INFO 21062 --- [subtask-executor13,dd193f8f67a5a423,80b8598faf86a5a9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 80, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:20:05.563 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:16, status:SUCCESS 2024-11-04 14:20:05.564 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:16, status:SUCCESS 2024-11-04 14:20:05.882 INFO 21062 --- [ocp-schedules-12,15af9f1c719ef100,3aad1d79e853f5bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:20:05.882 INFO 21062 --- [ocp-schedules-12,773078d74014e466,1c3b1f949162bf04] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:20:05.886 INFO 21062 --- [ocp-schedules-12,773078d74014e466,1c3b1f949162bf04] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:05.915 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:20:05.915 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:20:05.983 INFO 21062 --- [prometheus-minute-schedule-1,146d7961b87aaf6d,76b0d621ca540f2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:07.159 WARN 21062 --- [ocp-async-10,acafd496f342727d,cdfe7a9bf54b39a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:07.215 INFO 21062 --- [ocp-async-4,abfca35590ea26db,2ddbef582e7dcbad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:07.686 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:07.687 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:08.505 INFO 21062 --- [ocp-schedules-9,1ec7e247a24f615b,366c6fcd8266b83b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:20:08.506 INFO 21062 --- [ocp-schedules-2,1b321b74c1c8888b,de610db342e3d061] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:20:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:20:08.590575+08:00, lastProcessTime=2024-11-04T14:19:58.589297+08:00 2024-11-04 14:20:08.655 INFO 21062 --- [ocp-async-9,97654848ceb4a100,13e0f70fd754bc67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:20:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:20:08.590575+08:00 2024-11-04 14:20:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a42c99848ab8123b,dde89d883e1254f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:09.159 WARN 21062 --- [ocp-async-5,e6227425702de3c5,0d911f3febd1616f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:09.215 INFO 21062 --- [ocp-async-3,9bf52440c67f6f73,6a71e8ff4c054c06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:10.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:20:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:20:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:20:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:20:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:20:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:20:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:20:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:20:10.426 INFO 21062 --- [ocp-schedules-6,b7d7e969247b9bc2,5e98e563ae55cd9a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:20:10.687 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:10.688 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:11.160 WARN 21062 --- [ocp-async-3,7f4fec773239caf9,d79bee5f8c34732b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:11.215 INFO 21062 --- [ocp-async-10,ce86e59cee1560e1,8c12f553b9cf9fe6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:11.411 INFO 21062 --- [ocp-schedules-21,edfd22147ae8e796,cfe0c5098d025022] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:20:11.417 INFO 21062 --- [ocp-schedules-21,edfd22147ae8e796,cfe0c5098d025022] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:20:11.417 INFO 21062 --- [ocp-schedules-21,edfd22147ae8e796,cfe0c5098d025022] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=2, affectRows=2 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=2, validCount=2 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=8, affectRows=8 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=8, validCount=8 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=26, affectRows=26 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=26, validCount=26 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=6, affectRows=6 2024-11-04 14:20:11.536 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=6, validCount=6 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=23, affectRows=23 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=23, validCount=23 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=36, affectRows=36 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=33, affectRows=33 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=55, affectRows=55 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=33, validCount=33 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=36, validCount=36 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=54, validCount=54 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=76, affectRows=76 2024-11-04 14:20:11.537 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=79, affectRows=79 2024-11-04 14:20:11.538 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=79, validCount=79 2024-11-04 14:20:11.538 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=75, validCount=75 2024-11-04 14:20:11.538 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701140, nodeCount=52, affectRows=52 2024-11-04 14:20:11.538 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701140, nodeCount=51, validCount=51 2024-11-04 14:20:11.983 INFO 21062 --- [prometheus-minute-schedule-1,f9142ed1686966e3,2b216e02fdfbf0ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:12.336 INFO 21062 --- [ocp-schedules-17,e774e7065ba66219,678eabbb27e01f83] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:20:13.160 WARN 21062 --- [ocp-async-10,5231c4d112e7b1dc,063dc1b78c57ebe2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:13.215 INFO 21062 --- [ocp-async-4,b00ead7fcc214716,6f01ca65dfbab5bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:13.688 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:13.689 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:13.984 INFO 21062 --- [prometheus-cache-monitor1,a58ea9d1dff5ea1d,a1258ae9758b4f55] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:20:13.984 INFO 21062 --- [prometheus-cache-monitor1,a58ea9d1dff5ea1d,a1258ae9758b4f55] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:20:14.983 INFO 21062 --- [prometheus-minute-schedule-1,7f3752f59a5de12e,ac0588852e2d62b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:15.160 WARN 21062 --- [ocp-async-4,fd358eec062442f8,68ba5364932df35d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:15.216 INFO 21062 --- [ocp-async-7,9748e36ee00d7eca,4f52146832b8c61d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:16.689 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:16.690 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:17.160 WARN 21062 --- [ocp-async-7,079341f0b1cef8ce,58aecb8f35b99664] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:17.216 INFO 21062 --- [ocp-async-2,10e775f00f7916bd,623b6fe754be750b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:17.983 INFO 21062 --- [prometheus-minute-schedule-1,35ab5263b6950ab3,81a263354d976374] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:20:18.591118+08:00, lastProcessTime=2024-11-04T14:20:08.590575+08:00 2024-11-04 14:20:18.655 INFO 21062 --- [ocp-async-5,8449eac2aa45f468,70f522d27f3b68fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:20:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:20:18.591118+08:00 2024-11-04 14:20:19.160 WARN 21062 --- [ocp-async-10,4f519b329636d203,df7efcdc94edf1cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:19.216 INFO 21062 --- [ocp-async-4,dc7a04d815ee61a0,9259fb45d9c073fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:19.690 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:19.691 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:20.430 INFO 21062 --- [ocp-schedules-2,95583226ca78eb26,3908be527fe0b167] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:20:20.983 INFO 21062 --- [prometheus-minute-schedule-1,381cdb60dca186ba,8c8803e5a943768e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:21.161 WARN 21062 --- [ocp-async-4,199c9fcf3c2e2f76,3ef8645e65ef66f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:21.216 INFO 21062 --- [ocp-async-7,0669309ff013df20,7abc07f870f0202e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:22.378 INFO 21062 --- [ocp-schedules-22,3ef9e342cd06c639,0ffe7f53cf4cc6c6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:20:22.691 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:22.692 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:23.161 WARN 21062 --- [ocp-async-7,f89f9914962eeff2,e2f27fbc05277a6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:23.216 INFO 21062 --- [ocp-async-2,784b9182a188f707,86b8a25ee64eb3e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c02f646c30b74c12,e10bddf1eb19ee6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:25.161 WARN 21062 --- [ocp-async-2,96f706c6253ec6d8,03a912c31b703142] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:25.216 INFO 21062 --- [ocp-async-8,2680f86e9b844051,b71ff5c32964f5fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:25.692 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:25.693 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:26.983 INFO 21062 --- [prometheus-minute-schedule-1,992f95c07003e6e5,96845743ecb06427] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:26.984 INFO 21062 --- [prometheus-hour-schedule-1,3234a89ff5984cbe,0e8f0643eeded092] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:20:27.161 WARN 21062 --- [ocp-async-8,933eb768f33638c8,6c88d1175d28a0ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:27.216 INFO 21062 --- [ocp-async-6,94647b85c0a0a174,446ea9604aa37dba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:28.498 INFO 21062 --- [ocp-schedules-26,2876ae67eecd6af3,5bd2116a34cebfd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:20:28.500 INFO 21062 --- [ocp-schedules-24,23a1f8b666fff87a,dc76973e3fdabc34] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:20:28.500 INFO 21062 --- [ocp-schedules-3,e58aecd7b095ffef,a1fda1233acd6a22] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:20:28.500 INFO 21062 --- [ocp-schedules-18,02d3d213dafd4c24,88df264fe1cb9288] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:20:28.502 INFO 21062 --- [ocp-schedules-11,e693e571c856490f,260451c8db5bab60] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:20:28.503 INFO 21062 --- [ocp-schedules-26,2876ae67eecd6af3,5bd2116a34cebfd8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:20:28.505 INFO 21062 --- [ocp-schedules-18,02d3d213dafd4c24,88df264fe1cb9288] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:28.505 INFO 21062 --- [ocp-schedules-3,e58aecd7b095ffef,a1fda1233acd6a22] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:28.506 INFO 21062 --- [ocp-schedules-16,0df2700f1e4b99df,abb47b59cb750222] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:20:28.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:20:28.506 INFO 21062 --- [ocp-schedules-24,23a1f8b666fff87a,dc76973e3fdabc34] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:20:28.512 INFO 21062 --- [ocp-schedules-29,129bfe0a433b01b9,96090dc17343a035] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:20:28.512 INFO 21062 --- [ocp-schedules-29,129bfe0a433b01b9,96090dc17343a035] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:20:28.512 INFO 21062 --- [ocp-schedules-7,0ec87252ee36a4c9,4bf4e6c28b79807e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:20:28.512 INFO 21062 --- [ocp-schedules-7,0ec87252ee36a4c9,4bf4e6c28b79807e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:20:28.513 INFO 21062 --- [ocp-schedules-4,fe8d73c2006cf141,af92a38063d64bcc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:20:28.513 INFO 21062 --- [ocp-schedules-4,fe8d73c2006cf141,af92a38063d64bcc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:20:28.575 INFO 21062 --- [ocp-schedules-2,b66597f4227a4e1e,0cb1774cb9768807] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:20:28.575 INFO 21062 --- [ocp-schedules-2,b66597f4227a4e1e,0cb1774cb9768807] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:20:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:20:28.589280+08:00, lastProcessTime=2024-11-04T14:20:18.591118+08:00 2024-11-04 14:20:28.655 INFO 21062 --- [ocp-async-8,a215a8eb7a80c688,6e33ba91221f45f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:20:28.693 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:28.694 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:20:28.589280+08:00 2024-11-04 14:20:28.822 INFO 21062 --- [ocp-schedules-20,ea15911c98b8ce50,17623d4d87d46897] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:20:29.081 INFO 21062 --- [ocp-schedules-8,6156dd9dce447b80,cf23f5bf3204c950] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:20:29.081 INFO 21062 --- [ocp-schedules-8,6156dd9dce447b80,cf23f5bf3204c950] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:20:29.149 INFO 21062 --- [ocp-schedules-16,c87b018b1a0d8c5e,c1c1b766c2d261c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:20:29.161 WARN 21062 --- [ocp-async-9,9e1c6b79f3d065cb,218c90b0a443b601] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:29.179 INFO 21062 --- [ocp-schedules-7,0643048ec894110a,d72980b8a82c7bcb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:20:29.185 INFO 21062 --- [ocp-schedules-7,0643048ec894110a,d72980b8a82c7bcb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:20:29.185 INFO 21062 --- [ocp-schedules-7,0643048ec894110a,d72980b8a82c7bcb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:20:29.217 INFO 21062 --- [ocp-async-1,0d65a584b70759fe,8b68898944e61598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:29.510 INFO 21062 --- [ocp-schedules-10,71672472b1eb931c,e8bfece0e87ffbaa] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:20:29.510 INFO 21062 --- [ocp-schedules-10,71672472b1eb931c,e8bfece0e87ffbaa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:20:29.512 INFO 21062 --- [ocp-schedules-10,71672472b1eb931c,e8bfece0e87ffbaa] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:20:29.512 INFO 21062 --- [ocp-schedules-10,71672472b1eb931c,e8bfece0e87ffbaa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:20:29.512 INFO 21062 --- [ocp-schedules-10,71672472b1eb931c,e8bfece0e87ffbaa] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:20:29.512 INFO 21062 --- [ocp-schedules-10,71672472b1eb931c,e8bfece0e87ffbaa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:20:29.983 INFO 21062 --- [prometheus-minute-schedule-1,80b49b26eb5a7841,5d2a5f70dce65f23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:20:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:20:30.435 INFO 21062 --- [ocp-schedules-4,914af369a0c39f76,ca71898af3ed7d32] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:20:31.161 WARN 21062 --- [ocp-async-1,f220a17e3ecdf17d,2f724429a5fc2e78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:31.217 INFO 21062 --- [ocp-async-5,7dcac6c1d48925cb,623bc23139a4018e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:31.420 INFO 21062 --- [ocp-schedules-11,90144f60789a87b5,b2d57ac89adfcc0b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:20:31.425 INFO 21062 --- [ocp-schedules-11,90144f60789a87b5,b2d57ac89adfcc0b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:20:31.425 INFO 21062 --- [ocp-schedules-11,90144f60789a87b5,b2d57ac89adfcc0b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:20:31.470 INFO 21062 --- [ocp-schedules-1,63ed81382dadce5f,61b20ed33cf2a02b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:20:31.470 INFO 21062 --- [ocp-schedules-1,63ed81382dadce5f,61b20ed33cf2a02b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:20:31.497 INFO 21062 --- [ocp-schedules-1,63ed81382dadce5f,61b20ed33cf2a02b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:20:31.497 INFO 21062 --- [ocp-schedules-1,63ed81382dadce5f,61b20ed33cf2a02b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:20:31.498 INFO 21062 --- [ocp-schedules-27,7bfa5fe6808e402a,8603d577a9f63d7d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:20:31.683 INFO 21062 --- [ocp-schedules-17,a7fd6f4f2950dc55,f27b5a944495e378] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:20:31.683+08:00, clusters=[1] 2024-11-04 14:20:31.684 INFO 21062 --- [ocp-schedules-17,a7fd6f4f2950dc55,f27b5a944495e378] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:20:31.694 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:31.695 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:31.823 INFO 21062 --- [ocp-schedules-24,59034e85b32affc0,6e47bf94ae0b2e03] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:20:31.827 INFO 21062 --- [ocp-schedules-24,59034e85b32affc0,6e47bf94ae0b2e03] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:31.841 INFO 21062 --- [ocp-schedules-4,cad97f4843f099c7,711745e4dbe5076f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:20:31.841+08:00, clusters=[1] 2024-11-04 14:20:31.844 INFO 21062 --- [ocp-schedules-4,cad97f4843f099c7,711745e4dbe5076f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:20:31.886 ERROR 21062 --- [ocp-schedules-24,59034e85b32affc0,6e47bf94ae0b2e03] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:20:31.886 INFO 21062 --- [ocp-schedules-24,59034e85b32affc0,6e47bf94ae0b2e03] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:20:32.403 INFO 21062 --- [ocp-schedules-15,f573d133ba6c3e3b,3fc91211428c08ef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:20:32.755 INFO 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:20:32.755 INFO 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:20:32.756 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:20:32.756 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:20:32.766 INFO 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:20:32.766 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:20:32.766 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:20:32.767 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:20:32.767 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:20:32.767 WARN 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:20:32.768 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:20:32.768 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:20:32.768 INFO 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:20:32.769 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:20:32.769 WARN 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:20:32.769 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:20:32.777 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:20:32.777 INFO 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:20:32.777 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:20:32.789 ERROR 21062 --- [ocp-schedules-5,ab7daa273de0f9fb,f0b1de15165ae833] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:20:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3c99a28126360230,ec7e53b903d56f39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:33.162 WARN 21062 --- [ocp-async-5,d0f1eb9e5ff68bbf,c62974e12d9f76a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:33.217 INFO 21062 --- [ocp-async-3,b9502132651149f7,f927ca756cf6e4a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:34.695 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:34.696 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:35.162 WARN 21062 --- [ocp-async-3,da68c0dcc24a4707,16cbaca622891730] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:35.217 INFO 21062 --- [ocp-async-2,a6f77319f7182be2,c2555819ef2fb4ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:35.916 INFO 21062 --- [ocp-schedules-12,773078d74014e466,1c3b1f949162bf04] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:20:35.916 INFO 21062 --- [ocp-schedules-12,82da8a6247526a52,9e9ae7009b7e3319] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:20:35.920 INFO 21062 --- [ocp-schedules-12,82da8a6247526a52,9e9ae7009b7e3319] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:35.950 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:20:35.950 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:20:35.983 INFO 21062 --- [prometheus-minute-schedule-1,2883069af811ca88,86b15892aaf87aa7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:37.162 WARN 21062 --- [ocp-async-2,bd216e4662eae896,17990049a1aeac31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:37.217 INFO 21062 --- [ocp-async-4,54a784fe5b5cf8d4,dbf4774c0f4d8110] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:37.696 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:37.697 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:38.488 INFO 21062 --- [ocp-schedules-22,741e1d19533e5054,2df26263a89848d7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:20:38.493 INFO 21062 --- [ocp-schedules-22,741e1d19533e5054,2df26263a89848d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:38.506 INFO 21062 --- [ocp-schedules-18,02d3d213dafd4c24,88df264fe1cb9288] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:20:38.506 INFO 21062 --- [ocp-schedules-3,e58aecd7b095ffef,a1fda1233acd6a22] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:20:38.582 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:20:38.584 ERROR 21062 --- [ocp-schedules-22,741e1d19533e5054,2df26263a89848d7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:20:38.584 INFO 21062 --- [ocp-schedules-22,741e1d19533e5054,2df26263a89848d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:20:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:20:38.589952+08:00, lastProcessTime=2024-11-04T14:20:28.589280+08:00 2024-11-04 14:20:38.656 INFO 21062 --- [ocp-async-10,76146bdfbe9f3c96,55b8b47cb14527b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:20:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:20:38.589952+08:00 2024-11-04 14:20:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a12fb604d755c9cc,5bc42358596e4dbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:39.162 WARN 21062 --- [ocp-async-6,1fa98bf5138a998e,72066d2a61252720] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:39.217 INFO 21062 --- [ocp-async-9,92dd6bd506ba320e,ac704094e171d03a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:20:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:20:40.439 INFO 21062 --- [ocp-schedules-11,89d224c4dcead654,b25b7a47f0fd23d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:20:40.697 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:40.698 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:41.162 WARN 21062 --- [ocp-async-9,ff02e2fad995a11b,13f061d40b28a41b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:41.218 INFO 21062 --- [ocp-async-2,1c421482807db0a6,412599ca52d207a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:20:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:20:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=30ms, period=120000ms 2024-11-04 14:20:41.983 INFO 21062 --- [prometheus-minute-schedule-1,41ab3f690b0dcabe,437561cb28a2cd87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:42.445 INFO 21062 --- [ocp-schedules-25,6aae320d5db394e5,643d5f9dabc4561c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:20:43.162 WARN 21062 --- [ocp-async-2,b53014747b97d85e,3ad0e9e47cce0f0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:43.218 INFO 21062 --- [ocp-async-4,d6019540e5a58fe9,71de517c33c4a67c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:43.698 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:43.699 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:44.983 INFO 21062 --- [prometheus-minute-schedule-1,04ed1ad7f10cb967,5862c9b2ca31fef5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:45.163 WARN 21062 --- [ocp-async-4,5db2aed226a11393,fed03501619280b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:45.218 INFO 21062 --- [ocp-async-7,dad76873aaa064b8,bcf55b961dfc1e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:46.699 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:46.700 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:47.163 WARN 21062 --- [ocp-async-7,5098521d0ee1109b,b1ea050baa837877] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:47.218 INFO 21062 --- [ocp-async-10,ab94de24cb833d59,9203bff094443f4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b7146575a9455b2c,2b80ae795fbab695] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:20:48.593284+08:00, lastProcessTime=2024-11-04T14:20:38.589952+08:00 2024-11-04 14:20:48.656 INFO 21062 --- [ocp-async-6,05ef740659acf821,29aa3a77a0de299a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:20:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:20:48.593284+08:00 2024-11-04 14:20:49.000 INFO 21062 --- [ocp-schedules-28,40d743a98f71bf21,bf5fcb096c3d3886] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:20:49.004 INFO 21062 --- [ocp-schedules-28,40d743a98f71bf21,bf5fcb096c3d3886] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:49.005 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:20:49.163 WARN 21062 --- [ocp-async-2,4369c29b2861f1ea,eb02a9e4d95a5742] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:49.218 INFO 21062 --- [ocp-async-5,1cdb417b5fbef862,fecf2b79baa5ee2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:49.700 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:49.701 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:50.442 INFO 21062 --- [ocp-schedules-10,74a5a54635d0140c,9bec09b80a5a7c46] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:20:50.983 INFO 21062 --- [prometheus-minute-schedule-1,2e27b3c8eeb4118e,53a6767c06b12213] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:51.163 WARN 21062 --- [ocp-async-4,98be7b35d6573958,9a815fc0ececb144] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:51.219 INFO 21062 --- [ocp-async-3,8cb9fb2818b3dd89,a7ba8abc647ea39b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:51.428 INFO 21062 --- [ocp-schedules-19,5a03c85de13aa826,2aa018dc3ac91da4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:20:51.434 INFO 21062 --- [ocp-schedules-19,5a03c85de13aa826,2aa018dc3ac91da4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:20:51.434 INFO 21062 --- [ocp-schedules-19,5a03c85de13aa826,2aa018dc3ac91da4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:20:52.473 INFO 21062 --- [ocp-schedules-2,47692b9c83a04991,7b6b052f90dc0d22] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:20:52.701 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:52.702 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:53.163 WARN 21062 --- [ocp-async-7,454a3eb9f12b05d5,a32b4fac1d179583] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:53.219 INFO 21062 --- [ocp-async-10,4f0d3e8cb11477e2,a86e3f8ce6197039] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:53.983 INFO 21062 --- [prometheus-minute-schedule-1,61fceff7b0a430a0,3500c1deff53b2ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:55.164 WARN 21062 --- [ocp-async-1,ace55683118f7fb3,c0e5d11b41141993] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:55.219 INFO 21062 --- [ocp-async-8,cf32212679d5c84d,2de997cda2700f44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:55.702 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:55.703 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:56.983 INFO 21062 --- [prometheus-minute-schedule-1,1e8a9196ad913052,7efd70569ba8de83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:20:57.164 WARN 21062 --- [ocp-async-5,d5dd255f58273e8c,a9ca8ec1eceb3bcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:57.219 INFO 21062 --- [ocp-async-6,2e57c7945152117d,754071ee5360ff09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:58.500 INFO 21062 --- [ocp-schedules-2,6ba94eaffbe1278a,e10ff8c6b9b55c11] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:20:58.500 INFO 21062 --- [ocp-schedules-30,1272431fc5be4769,7d9e56032b1fa413] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:20:58.502 INFO 21062 --- [ocp-schedules-10,84b14abd4cbad0b6,f5a94aa93d42ac48] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:20:58.504 INFO 21062 --- [ocp-schedules-2,6ba94eaffbe1278a,e10ff8c6b9b55c11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:58.504 INFO 21062 --- [ocp-schedules-30,1272431fc5be4769,7d9e56032b1fa413] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:20:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:20:58.505 INFO 21062 --- [ocp-schedules-11,c7321f1391c00990,025d5b1e94d11490] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:20:58.510 INFO 21062 --- [ocp-schedules-14,dfa5da97c8232dbf,1e849ffb4fda9c29] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:20:58.510 INFO 21062 --- [ocp-schedules-14,dfa5da97c8232dbf,1e849ffb4fda9c29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:20:58.510 INFO 21062 --- [ocp-schedules-6,fe2419abfe295d6b,83f1876cf7944c87] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:20:58.511 INFO 21062 --- [ocp-schedules-6,fe2419abfe295d6b,83f1876cf7944c87] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:20:58.511 INFO 21062 --- [ocp-schedules-26,b80f4b427ab14277,a53510b52d18887a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:20:58.511 INFO 21062 --- [ocp-schedules-26,b80f4b427ab14277,a53510b52d18887a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:20:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:20:58.590549+08:00, lastProcessTime=2024-11-04T14:20:48.593284+08:00 2024-11-04 14:20:58.656 INFO 21062 --- [ocp-async-2,7299912011bbd6b6,901ebe4007a81e23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:20:58.703 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:20:58.704 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:20:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:20:58.590549+08:00 2024-11-04 14:20:59.009 INFO 21062 --- [ocp-schedules-28,40d743a98f71bf21,bf5fcb096c3d3886] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:20:59.152 INFO 21062 --- [ocp-schedules-17,5cfa6dd7be4f40c4,fa515fc56e5362ec] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:20:59.164 WARN 21062 --- [ocp-async-7,f8961cb7bd831e0c,66a7452f3f1c1904] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:20:59.219 INFO 21062 --- [ocp-async-10,832e0c93b3496f95,5015af4ed32bfb40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:20:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f8ca78b955e112e5,93dff891fe6bdd91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701260, interval=60 2024-11-04 14:21:00.445 INFO 21062 --- [ocp-schedules-16,06696d2361d13509,d868be1519136c86] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:21:01.164 WARN 21062 --- [ocp-async-1,2ebd86cb4cebe9d0,7cc00b32b52f5332] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:01.219 INFO 21062 --- [ocp-async-8,d2f774a5cd362d3c,b6878efa68474cb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:01.704 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:01.704 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:02.512 INFO 21062 --- [ocp-schedules-1,0f5e5ee3484fcba7,d259fd7b8631e3d3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:21:02.983 INFO 21062 --- [prometheus-minute-schedule-1,9198405dd8e95f50,4155632eb82cec7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:03.164 WARN 21062 --- [ocp-async-5,1071c85a7c2906f5,722c79dfa9a32b4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:03.220 INFO 21062 --- [ocp-async-6,28612f3b4b214225,2717ce48bc264357] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:04.704 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:04.705 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:05.165 WARN 21062 --- [ocp-async-3,27ca29342447c708,d55752bbec31c575] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:05.220 INFO 21062 --- [ocp-async-9,02715f9a6cacd3ee,62f9ff7a9ebf9d33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:05.951 INFO 21062 --- [ocp-schedules-12,82da8a6247526a52,9e9ae7009b7e3319] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:21:05.951 INFO 21062 --- [ocp-schedules-12,2ac8d3ce25d62349,65a113a5afab00a9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:21:05.955 INFO 21062 --- [ocp-schedules-12,2ac8d3ce25d62349,65a113a5afab00a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e81a4453df49b04d,f02b2ad522e6c79c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:05.987 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:21:05.987 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:21:07.165 WARN 21062 --- [ocp-async-10,42aa2d17b0abfb09,12af0881ce050e13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:07.220 INFO 21062 --- [ocp-async-2,8477ccd3b2bd4337,2923136670751de4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:07.705 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:07.706 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:08.505 INFO 21062 --- [ocp-schedules-2,6ba94eaffbe1278a,e10ff8c6b9b55c11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:21:08.505 INFO 21062 --- [ocp-schedules-30,1272431fc5be4769,7d9e56032b1fa413] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:21:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:21:08.591606+08:00, lastProcessTime=2024-11-04T14:20:58.590549+08:00 2024-11-04 14:21:08.656 INFO 21062 --- [ocp-async-3,4669da09c3f1f5a2,ae6dc30c027fe764] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:21:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:21:08.591606+08:00 2024-11-04 14:21:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f911b32062f9f1d9,501972ecb4fcd797] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:09.165 WARN 21062 --- [ocp-async-8,95aee77f5ca25737,d5ba31bb5b7a0760] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:09.220 INFO 21062 --- [ocp-async-6,9d8b90835e591e2f,cedcad110e91fa78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:21:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:21:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:21:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:21:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:21:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:21:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:21:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:21:10.448 INFO 21062 --- [ocp-schedules-5,12f27bd13d4b5f74,f920c3cb2405c598] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:21:10.706 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:10.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:11.165 WARN 21062 --- [ocp-async-6,3a091f2c4fe5e2bd,52b68a1beeb5122f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:11.221 INFO 21062 --- [ocp-async-9,87bde00c144f14ce,3fbfd6ab0fbd090b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:11.438 INFO 21062 --- [ocp-schedules-28,cfd976cf1942e3e3,8ce75f1b4bef0724] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:21:11.444 INFO 21062 --- [ocp-schedules-28,cfd976cf1942e3e3,8ce75f1b4bef0724] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:21:11.444 INFO 21062 --- [ocp-schedules-28,cfd976cf1942e3e3,8ce75f1b4bef0724] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:21:11.547 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=10, affectRows=10 2024-11-04 14:21:11.547 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=11, affectRows=11 2024-11-04 14:21:11.547 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=10, validCount=10 2024-11-04 14:21:11.547 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=11, validCount=11 2024-11-04 14:21:11.547 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=5, affectRows=5 2024-11-04 14:21:11.548 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=5, validCount=5 2024-11-04 14:21:11.548 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=36, affectRows=36 2024-11-04 14:21:11.548 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=15, affectRows=15 2024-11-04 14:21:11.548 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=2, affectRows=2 2024-11-04 14:21:11.548 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=2, validCount=2 2024-11-04 14:21:11.548 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=15, validCount=15 2024-11-04 14:21:11.548 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=44, affectRows=44 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=36, validCount=36 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=44, validCount=44 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=43, affectRows=43 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=19, affectRows=19 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=19, validCount=19 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=46, affectRows=46 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=43, validCount=43 2024-11-04 14:21:11.549 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=46, validCount=46 2024-11-04 14:21:11.550 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=31, affectRows=31 2024-11-04 14:21:11.550 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=27, affectRows=27 2024-11-04 14:21:11.550 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=31, validCount=31 2024-11-04 14:21:11.551 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=26, validCount=26 2024-11-04 14:21:11.551 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=56, affectRows=56 2024-11-04 14:21:11.551 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701200, nodeCount=51, affectRows=51 2024-11-04 14:21:11.551 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=49, validCount=49 2024-11-04 14:21:11.551 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701200, nodeCount=56, validCount=56 2024-11-04 14:21:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9aa8f2fb37027c67,ef31de4a78103cb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:12.539 INFO 21062 --- [ocp-schedules-26,319909b7f58038e7,15d6105c081d51a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:21:13.165 WARN 21062 --- [ocp-async-9,a6c856824f328193,b1f217bb702fa539] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:13.221 INFO 21062 --- [ocp-async-2,8ad51035c6640ddf,742563d404588e07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:13.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:13.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:13.984 INFO 21062 --- [prometheus-cache-monitor1,27ce4f3eb2253e59,e8706109860dfa39] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:21:13.984 INFO 21062 --- [prometheus-cache-monitor1,27ce4f3eb2253e59,e8706109860dfa39] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:21:14.983 INFO 21062 --- [prometheus-minute-schedule-1,147ba580622af7dd,b5e1efffb2895179] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:15.031 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:17:15.007, end=2024-11-04T14:19:15.028} 2024-11-04 14:21:15.035 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:21:15.035 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:21:15.165 WARN 21062 --- [ocp-async-2,5f83d50c65bc6101,c42fd2cff4f65bd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:15.221 INFO 21062 --- [ocp-async-4,f35024dcb98890f3,0525dd734ec7c1e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:16.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:16.708 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:17.166 WARN 21062 --- [ocp-async-4,5f4cdaba455e2c7f,e8c58807140a54f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:17.221 INFO 21062 --- [ocp-async-7,15ecb43a6a6f2afd,d17ab759ef42aa77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:17.983 INFO 21062 --- [prometheus-minute-schedule-1,b448ae511610881e,51888253fb73094f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:21:18.590845+08:00, lastProcessTime=2024-11-04T14:21:08.591606+08:00 2024-11-04 14:21:18.656 INFO 21062 --- [ocp-async-8,a9cef72d1363c202,a8a374b71d82fc41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:21:18.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:21:18.590845+08:00 2024-11-04 14:21:19.166 WARN 21062 --- [ocp-async-9,aa3bfc12630a5744,08a34294e221bbcc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:19.221 INFO 21062 --- [ocp-async-2,abc47e5ba3f7b8a6,ede6760ccf64147a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:19.708 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:19.709 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:20.451 INFO 21062 --- [ocp-schedules-30,0c2e55f9c89df7e3,a0bf25c20af8e3b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:21:20.983 INFO 21062 --- [prometheus-minute-schedule-1,70d6a9eb1b175ff0,8b722ff139322fb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:21.166 WARN 21062 --- [ocp-async-2,93d0b11961a2c819,27ac272f6aa6e562] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:21.222 INFO 21062 --- [ocp-async-4,89c8a3f2f271df3d,132c8cd43e7d5e76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:22.622 INFO 21062 --- [ocp-schedules-15,38026dfe040fd5b7,00fa27d37a99a349] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:21:22.709 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:22.710 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:23.166 WARN 21062 --- [ocp-async-4,13818c817b9a08a5,9404ffdd812a93de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:23.222 INFO 21062 --- [ocp-async-7,a9f08e58211484c5,af6657c39e4fe0c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:23.983 INFO 21062 --- [prometheus-minute-schedule-1,26d94e2d66f0410c,f68f9263d30d5c38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:25.166 WARN 21062 --- [ocp-async-7,56c5835ab596f496,626f96984f6fb081] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:25.222 INFO 21062 --- [ocp-async-10,2acca552633e0fd7,41038436e587666a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:25.710 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:25.711 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:26.983 INFO 21062 --- [prometheus-minute-schedule-1,775d7bcf36c4545c,16d2816e8e1dbe20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:26.984 INFO 21062 --- [prometheus-hour-schedule-1,af58c5dea5c594d1,7c8c420c365b9055] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:21:27.166 WARN 21062 --- [ocp-async-10,f90a0f239f043a3f,93336eb33af5501e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:27.222 INFO 21062 --- [ocp-async-8,52d4384f0670b0a6,e45155af623f04ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:28.498 INFO 21062 --- [ocp-schedules-22,20670a7dbf5b0425,fb43f9e4cee19656] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:21:28.499 INFO 21062 --- [ocp-schedules-16,8c155c30c569672e,b3c7d23c60364b70] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:21:28.500 INFO 21062 --- [ocp-schedules-26,1459490bbcd94197,9149a1910a8a06f6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:21:28.500 INFO 21062 --- [ocp-schedules-19,375f3bb447281871,c1038715ba095ae8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:21:28.502 INFO 21062 --- [ocp-schedules-27,394da12e7ef50342,65cff2aa5f3986d2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:21:28.503 INFO 21062 --- [ocp-schedules-22,20670a7dbf5b0425,fb43f9e4cee19656] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:21:28.503 INFO 21062 --- [ocp-schedules-26,1459490bbcd94197,9149a1910a8a06f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:21:28.504 INFO 21062 --- [ocp-schedules-19,375f3bb447281871,c1038715ba095ae8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:28.505 INFO 21062 --- [ocp-schedules-9,435b63a92f9c0b6f,03009565b1617841] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:21:28.505 INFO 21062 --- [ocp-schedules-16,8c155c30c569672e,b3c7d23c60364b70] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:21:28.511 INFO 21062 --- [ocp-schedules-20,6ae44130513c2aa5,0af1c7e8849653f5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:21:28.511 INFO 21062 --- [ocp-schedules-20,6ae44130513c2aa5,0af1c7e8849653f5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:21:28.511 INFO 21062 --- [ocp-schedules-15,b49977fbd4ae00d8,dcc05f6079f04be3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:21:28.511 INFO 21062 --- [ocp-schedules-15,b49977fbd4ae00d8,dcc05f6079f04be3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:21:28.512 INFO 21062 --- [ocp-schedules-28,43fd07a059cec59e,7d020b46b8d63683] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:21:28.512 INFO 21062 --- [ocp-schedules-28,43fd07a059cec59e,7d020b46b8d63683] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:21:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:21:28.589512+08:00, lastProcessTime=2024-11-04T14:21:18.590845+08:00 2024-11-04 14:21:28.628 INFO 21062 --- [ocp-schedules-1,576f182e2f31c0fb,4bf24946fa5fd935] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:21:28.629 INFO 21062 --- [ocp-schedules-1,576f182e2f31c0fb,4bf24946fa5fd935] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:21:28.656 INFO 21062 --- [ocp-async-10,874bbd477fcd13d5,050e36583bfec411] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:21:28.711 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:28.712 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:28.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:21:28.589512+08:00 2024-11-04 14:21:28.824 INFO 21062 --- [ocp-schedules-30,1b0157b3b357db19,954a8eae92b32e6e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:21:29.085 INFO 21062 --- [ocp-schedules-3,88259dd560b57cb3,db8134872111bcd5] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:21:29.085 INFO 21062 --- [ocp-schedules-3,88259dd560b57cb3,db8134872111bcd5] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:21:29.154 INFO 21062 --- [ocp-schedules-20,c64cad212cd409bf,e91b44ca011aa381] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:21:29.167 WARN 21062 --- [ocp-async-7,70d327986803f85f,bb92efc22f89e1a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:29.185 INFO 21062 --- [ocp-schedules-22,f87b9810e4b04a14,96a1f12279dc0e4b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:21:29.190 INFO 21062 --- [ocp-schedules-22,f87b9810e4b04a14,96a1f12279dc0e4b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:21:29.190 INFO 21062 --- [ocp-schedules-22,f87b9810e4b04a14,96a1f12279dc0e4b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:21:29.222 INFO 21062 --- [ocp-async-9,309633c325fecd22,35ee9854d40af236] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:29.513 INFO 21062 --- [ocp-schedules-1,2911fdc9022f9dfa,2b6480ed75921579] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:21:29.513 INFO 21062 --- [ocp-schedules-1,2911fdc9022f9dfa,2b6480ed75921579] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:21:29.516 INFO 21062 --- [ocp-schedules-1,2911fdc9022f9dfa,2b6480ed75921579] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:21:29.517 INFO 21062 --- [ocp-schedules-1,2911fdc9022f9dfa,2b6480ed75921579] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:21:29.517 INFO 21062 --- [ocp-schedules-1,2911fdc9022f9dfa,2b6480ed75921579] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:21:29.517 INFO 21062 --- [ocp-schedules-1,2911fdc9022f9dfa,2b6480ed75921579] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:21:29.983 INFO 21062 --- [prometheus-minute-schedule-1,9ebe3657447e3ae1,fa684b7876a75877] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:21:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:21:30.455 INFO 21062 --- [ocp-schedules-28,60f1dbfdf507fe83,1d678f6d07d5a766] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:21:31.167 WARN 21062 --- [ocp-async-9,96e16875cc69db73,1b93351da2492ace] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:31.222 INFO 21062 --- [ocp-async-2,64ef4cb8c75f1424,78c5a2d19b6cc57c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:31.447 INFO 21062 --- [ocp-schedules-7,f02185d6deeb3cb1,71c1232603affa48] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:21:31.453 INFO 21062 --- [ocp-schedules-7,f02185d6deeb3cb1,71c1232603affa48] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:21:31.453 INFO 21062 --- [ocp-schedules-7,f02185d6deeb3cb1,71c1232603affa48] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:21:31.497 INFO 21062 --- [ocp-schedules-20,25038383a6b7a0ad,055b6792294babd3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:21:31.497 INFO 21062 --- [ocp-schedules-20,25038383a6b7a0ad,055b6792294babd3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:21:31.524 INFO 21062 --- [ocp-schedules-10,81964e70eab981b9,01e315abb0941608] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:21:31.524 INFO 21062 --- [ocp-schedules-20,25038383a6b7a0ad,055b6792294babd3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:21:31.524 INFO 21062 --- [ocp-schedules-20,25038383a6b7a0ad,055b6792294babd3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:21:31.711 INFO 21062 --- [ocp-schedules-9,fa272ab54ee5541c,cafb3d097c90d5c6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:21:31.711+08:00, clusters=[1] 2024-11-04 14:21:31.712 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:31.712 INFO 21062 --- [ocp-schedules-9,fa272ab54ee5541c,cafb3d097c90d5c6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:21:31.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:31.870 INFO 21062 --- [ocp-schedules-21,04089a9118ad27d7,484afd1c7190acc8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:21:31.870+08:00, clusters=[1] 2024-11-04 14:21:31.873 INFO 21062 --- [ocp-schedules-21,04089a9118ad27d7,484afd1c7190acc8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:21:32.649 INFO 21062 --- [ocp-schedules-16,974d13501e7318be,5e236da2ffa3cd1a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:21:32.798 INFO 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:21:32.798 INFO 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:21:32.798 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:21:32.798 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:21:32.808 INFO 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:21:32.808 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:21:32.809 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:21:32.810 WARN 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:21:32.810 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:21:32.810 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:21:32.810 INFO 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:21:32.811 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:21:32.811 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:21:32.812 WARN 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:21:32.812 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:21:32.812 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:21:32.820 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:21:32.820 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:21:32.820 INFO 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:21:32.831 ERROR 21062 --- [ocp-schedules-15,bfe6378453599ffa,088ea31debe41374] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:21:32.983 INFO 21062 --- [prometheus-minute-schedule-1,69d618d225468fc4,7a48972dda6ea200] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:33.167 WARN 21062 --- [ocp-async-2,d7c8250dc034df67,14fd32bdaa036b64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:33.223 INFO 21062 --- [ocp-async-4,e1e7cc1d0dc16664,8c3b59dd5b01aa3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:34.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:34.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:35.167 WARN 21062 --- [ocp-async-6,15909970a58f87bb,49ffc59fb6cf5ee1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:35.223 INFO 21062 --- [ocp-async-3,a7373edc0b00109a,d79487fdc741586a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:35.983 INFO 21062 --- [prometheus-minute-schedule-1,61bcffca3275e712,00d51d7f6cc9d7ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:35.987 INFO 21062 --- [ocp-schedules-12,2ac8d3ce25d62349,65a113a5afab00a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:21:35.988 INFO 21062 --- [ocp-schedules-12,e981c6d0238eb0fb,9c281a351586e23d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:21:35.992 INFO 21062 --- [ocp-schedules-12,e981c6d0238eb0fb,9c281a351586e23d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:36.022 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:21:36.022 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:21:37.167 WARN 21062 --- [ocp-async-1,2e4e816af8930d70,60e18adb45d6e658] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:37.223 INFO 21062 --- [ocp-async-10,0642c8ad3be20ad2,7f5798b85b4f7239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:37.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:37.715 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:38.504 INFO 21062 --- [ocp-schedules-26,1459490bbcd94197,9149a1910a8a06f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:21:38.505 INFO 21062 --- [ocp-schedules-19,375f3bb447281871,c1038715ba095ae8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:21:38.584 INFO 21062 --- [ocp-schedules-17,f6880641734c5c74,223fe81e99531be9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:21:38.587 INFO 21062 --- [ocp-schedules-17,f6880641734c5c74,223fe81e99531be9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:21:38.590498+08:00, lastProcessTime=2024-11-04T14:21:28.589512+08:00 2024-11-04 14:21:38.657 INFO 21062 --- [ocp-async-3,bdbde596735ffec3,58e06db32629dbe4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:21:38.672 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:21:38.675 ERROR 21062 --- [ocp-schedules-17,f6880641734c5c74,223fe81e99531be9] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:21:38.675 INFO 21062 --- [ocp-schedules-17,f6880641734c5c74,223fe81e99531be9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:21:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:21:38.590498+08:00 2024-11-04 14:21:38.983 INFO 21062 --- [prometheus-minute-schedule-1,8156e0811c656feb,411bd18beeec8f15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:39.168 WARN 21062 --- [ocp-async-8,73cf9138dab4ea5b,035b51659d48f139] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:39.223 INFO 21062 --- [ocp-async-4,eca170bec309f7a5,21551b28a42b235f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:21:40.004 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:21:40.458 INFO 21062 --- [ocp-schedules-1,438d07b349fdbf37,b1df3a76e460d79d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:21:40.715 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:40.715 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:41.095 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:21:41.095 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:21:41.103 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:21:41.103 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:21:41.103 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:21:41.103 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:21:41.109 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:21:41.109 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:21:41.109 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:21:41.109 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:21:41.142 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:21:41.151 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:21:41.151 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:21:41.151 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:21:41.151 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:21:41.151 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:21:41.159 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:21:41.168 WARN 21062 --- [ocp-async-7,444c5429b8abbfbe,a467cbb2d231d6cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:41.223 INFO 21062 --- [ocp-async-1,8339a095ad4a3d9d,9b30015701396364] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:41.692 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:21:41.983 INFO 21062 --- [prometheus-minute-schedule-1,5c93291c578e6428,a9f93bea4e1d88ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:42.257 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.148057716S 2024-11-04 14:21:42.257 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:21:42.362 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:21:42.362 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:21:42.389 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:21:42.490 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:21:42.490 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:21:42.490 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:21:42.490 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:21:42.490 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:21:42.490 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:21:42.502 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:21:42.677 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] o.s.boot.SpringApplication : Started application in 1.573 seconds (JVM running for 6716.356) 2024-11-04 14:21:42.677 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.573735446S 2024-11-04 14:21:42.692 INFO 21062 --- [ocp-schedules-19,722f6019dbed804b,58931462fa889467] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:21:43.086 INFO 21062 --- [ocp-schedules-27,697f13eaff77fd65,f10c1c426d09b5f5] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:21:43.168 WARN 21062 --- [ocp-async-9,81620a5e82592608,687c0b7e50ca98b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:43.224 INFO 21062 --- [ocp-async-5,f6857e216abc4404,b5a860642e48a4e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:43.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:43.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e10a32d184eb164e,a75395aa1de6228d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:45.168 WARN 21062 --- [ocp-async-2,3f992f9bc49b6485,4f21a428b3acaa78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:45.224 INFO 21062 --- [ocp-async-6,b97aaa2d6978b6da,f4f8bc94a7dda90a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:46.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:46.717 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:47.168 WARN 21062 --- [ocp-async-4,3b017389ad75a047,ae2b853832f0b15c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:47.224 INFO 21062 --- [ocp-async-3,143d6b6134f33644,b900198b09286410] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:47.983 INFO 21062 --- [prometheus-minute-schedule-1,75be6117540ce4d0,6edec4ec810b128c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:21:48.591107+08:00, lastProcessTime=2024-11-04T14:21:38.590498+08:00 2024-11-04 14:21:48.657 INFO 21062 --- [ocp-async-2,a398a2c58d8c5acb,a3a9f8ddad478388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:21:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:21:48.591107+08:00 2024-11-04 14:21:49.168 WARN 21062 --- [ocp-async-9,5b69631237b99268,d1e3d26ef12be9eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:49.224 INFO 21062 --- [ocp-async-5,8798323e16a8bb46,99ac021d3d4245d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:49.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:49.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:50.460 INFO 21062 --- [ocp-schedules-7,b3f7d0a31935b940,e0c76d1333fae404] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:21:50.983 INFO 21062 --- [prometheus-minute-schedule-1,feca2177440f9e0e,cfcd0844ba84dc64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:51.169 WARN 21062 --- [ocp-async-5,cd8ae0eb5f99983c,2344616ccb0754bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:51.224 INFO 21062 --- [ocp-async-6,0da26625e7578410,2f1d27ad9dd40da0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:51.455 INFO 21062 --- [ocp-schedules-25,ec408b537e912771,4af4047271f15548] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:21:51.461 INFO 21062 --- [ocp-schedules-25,ec408b537e912771,4af4047271f15548] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:21:51.461 INFO 21062 --- [ocp-schedules-25,ec408b537e912771,4af4047271f15548] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:21:52.718 INFO 21062 --- [ocp-schedules-16,b7ac8eafb674a765,339db48766469839] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:21:52.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:52.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:53.169 WARN 21062 --- [ocp-async-6,1bda21c274020d5a,5321972226af047a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:53.225 INFO 21062 --- [ocp-async-3,3d89fc3ece325382,c9951bc5bc41b75e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:53.983 INFO 21062 --- [prometheus-minute-schedule-1,cbe4c6bd56d8fb8f,b7a5350f64630705] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:55.169 WARN 21062 --- [ocp-async-3,79b48b7fa27fec6f,3e09b1e63e62bb11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:55.225 INFO 21062 --- [ocp-async-10,35b2c1cd571c1842,ceb89aa566a2704b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:55.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:55.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:56.983 INFO 21062 --- [prometheus-minute-schedule-1,2a97ecadeb44d1f6,f8b300f6269eb24b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:21:57.169 WARN 21062 --- [ocp-async-10,e96aa5150f6d6995,68f4c8261ac507e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:57.225 INFO 21062 --- [ocp-async-2,d6d193a3b08a3ee9,1c810d4dd280f263] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:58.500 INFO 21062 --- [ocp-schedules-26,451979e1e47633ba,6956fc5232b0a2da] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:21:58.500 INFO 21062 --- [ocp-schedules-7,f324034dafffe1e7,70cd4495ccfdcb48] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:21:58.502 INFO 21062 --- [ocp-schedules-21,07ec263a80ae6d43,e1fc707cf3c5d145] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:21:58.503 INFO 21062 --- [ocp-schedules-26,451979e1e47633ba,6956fc5232b0a2da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:21:58.505 INFO 21062 --- [ocp-schedules-13,bc3743b25c8bb2de,06e757e343f14183] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:21:58.505 INFO 21062 --- [ocp-schedules-7,f324034dafffe1e7,70cd4495ccfdcb48] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:58.511 INFO 21062 --- [ocp-schedules-1,4dae056af64af55c,28f895d41f5ffcc2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:21:58.511 INFO 21062 --- [ocp-schedules-18,778031d34bb182e8,ee3a3f8010dce03d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:21:58.511 INFO 21062 --- [ocp-schedules-1,4dae056af64af55c,28f895d41f5ffcc2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:21:58.511 INFO 21062 --- [ocp-schedules-18,778031d34bb182e8,ee3a3f8010dce03d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:21:58.512 INFO 21062 --- [ocp-schedules-20,5f849fa32ba6f906,9047344e787555b4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:21:58.512 INFO 21062 --- [ocp-schedules-20,5f849fa32ba6f906,9047344e787555b4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:21:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:21:58.589736+08:00, lastProcessTime=2024-11-04T14:21:48.591107+08:00 2024-11-04 14:21:58.657 INFO 21062 --- [ocp-async-1,3b976e7ea747cd76,381744812a6a4442] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:21:58.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:21:58.721 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:21:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:21:58.589736+08:00 2024-11-04 14:21:59.009 INFO 21062 --- [ocp-schedules-16,b46e37b041a06f27,94cf803ec09ffed3] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:21:59.013 INFO 21062 --- [ocp-schedules-16,b46e37b041a06f27,94cf803ec09ffed3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:21:59.016 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:21:59.157 INFO 21062 --- [ocp-schedules-14,1faf3f995f292333,efa1a9a2f69a6a8c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:21:59.169 WARN 21062 --- [ocp-async-6,614646d7675edfab,0f66986ff46aa186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:21:59.225 INFO 21062 --- [ocp-async-3,9cf1925485ac8bb6,2517abceb4d53897] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:21:59.983 INFO 21062 --- [prometheus-minute-schedule-1,9160c76e5a724468,71fe7ea784ca7286] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701320, interval=60 2024-11-04 14:22:00.464 INFO 21062 --- [ocp-schedules-30,39f95dd5d9971b93,09155d5ead921720] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:22:01.169 WARN 21062 --- [ocp-async-3,d1a3f2ccc72f46e8,7565273a44b921e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:01.225 INFO 21062 --- [ocp-async-10,8020eade97f49bc9,19ba4dd829c6fb8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:01.721 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:01.721 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:02.762 INFO 21062 --- [ocp-schedules-9,c641fc60d86f8f6d,f21c80d07f7b3dbe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:22:02.983 INFO 21062 --- [prometheus-minute-schedule-1,9f3a665dbfcc3812,51b28dcbfb80c008] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:03.169 WARN 21062 --- [ocp-async-10,08b5989e72be1d2c,2806faa4d6fbba82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:03.226 INFO 21062 --- [ocp-async-2,9442f0dafcbb57e1,4626c3b95abc0743] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:04.722 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:04.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:05.170 WARN 21062 --- [ocp-async-2,d4f020148e68828d,1c667a68c6572973] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:05.226 INFO 21062 --- [ocp-async-4,9f43605b1c32e2b4,70449dec58371b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ec6b6726b17269a1,a906f415c794fb6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:06.023 INFO 21062 --- [ocp-schedules-12,e981c6d0238eb0fb,9c281a351586e23d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:22:06.023 INFO 21062 --- [ocp-schedules-12,516e96b54fb6cd02,0cc83c56479dbc74] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:22:06.028 INFO 21062 --- [ocp-schedules-12,516e96b54fb6cd02,0cc83c56479dbc74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:06.057 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:22:06.057 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:22:07.170 WARN 21062 --- [ocp-async-4,0ddaa59edd609c78,f2c03a3f8e21ce66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:07.226 INFO 21062 --- [ocp-async-9,6d3e547d1b9b92be,9c33621da0b6fd3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:07.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:07.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:08.504 INFO 21062 --- [ocp-schedules-26,451979e1e47633ba,6956fc5232b0a2da] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:22:08.505 INFO 21062 --- [ocp-schedules-7,f324034dafffe1e7,70cd4495ccfdcb48] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:22:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:22:08.590133+08:00, lastProcessTime=2024-11-04T14:21:58.589736+08:00 2024-11-04 14:22:08.657 INFO 21062 --- [ocp-async-6,ecc5b59844d6ec36,c2801b750445db9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:22:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:22:08.590133+08:00 2024-11-04 14:22:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d420f503d4d5e013,676f35d3e2365827] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:09.016 INFO 21062 --- [ocp-schedules-16,b46e37b041a06f27,94cf803ec09ffed3] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:22:09.170 WARN 21062 --- [ocp-async-10,b0da5270ccd42952,5dbe84b58d7064e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:09.226 INFO 21062 --- [ocp-async-2,7c327c49f57b0e62,c2bcb8b42f95b97a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:22:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:22:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:22:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:22:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:22:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:22:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:22:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:22:10.468 INFO 21062 --- [ocp-schedules-23,72303d7b4d4e1796,e82807fb7c247211] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:22:10.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:10.724 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:11.170 WARN 21062 --- [ocp-async-2,ef29ca4966d7aaf7,75178ae45f478c3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:11.226 INFO 21062 --- [ocp-async-4,c0a35a4bfb39ebe0,1ad5d8830f3af723] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:11.464 INFO 21062 --- [ocp-schedules-29,6512171cb7a6b796,f69ccd66983db103] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:22:11.471 INFO 21062 --- [ocp-schedules-29,6512171cb7a6b796,f69ccd66983db103] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:22:11.471 INFO 21062 --- [ocp-schedules-29,6512171cb7a6b796,f69ccd66983db103] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:22:11.542 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701260, nodeCount=86, affectRows=86 2024-11-04 14:22:11.542 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701260, nodeCount=121, affectRows=121 2024-11-04 14:22:11.542 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701260, nodeCount=92, affectRows=92 2024-11-04 14:22:11.543 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701260, nodeCount=86, validCount=86 2024-11-04 14:22:11.543 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701260, nodeCount=91, validCount=91 2024-11-04 14:22:11.543 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701260, nodeCount=97, affectRows=97 2024-11-04 14:22:11.543 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701260, nodeCount=119, validCount=119 2024-11-04 14:22:11.543 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701260, nodeCount=97, validCount=97 2024-11-04 14:22:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c7dfbe5bd8423f71,de23817854414b74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:12.786 INFO 21062 --- [ocp-schedules-9,c3d059eaaaf4361e,82f19746400cc589] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:22:13.170 WARN 21062 --- [ocp-async-4,cbdd88d5762dbd56,dde4d2fb7b98cc81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:13.226 INFO 21062 --- [ocp-async-9,bd0896a8e452bbfd,2612ec379154c800] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:13.724 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:13.725 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:13.984 INFO 21062 --- [prometheus-cache-monitor1,da21aa31b9253c3f,02460ece3a746d8d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:22:13.985 INFO 21062 --- [prometheus-cache-monitor1,da21aa31b9253c3f,02460ece3a746d8d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:22:14.983 INFO 21062 --- [prometheus-minute-schedule-1,33f16e16e759cfa8,bb83c9ecc3d6a973] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:15.170 WARN 21062 --- [ocp-async-9,c26f32735db8bd7d,e6aff21dff9aa95c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:15.227 INFO 21062 --- [ocp-async-8,5c1983bc9997c9bf,b7f5f0448807b9b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:16.725 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:16.726 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:17.170 WARN 21062 --- [ocp-async-8,05fc558bbd339302,1d080ae60e553a02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:17.227 INFO 21062 --- [ocp-async-7,77640e6072bebee4,9c4eea8ee5e0b163] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d9f08c0842202e9e,bd8d4b036523bb25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:22:18.590320+08:00, lastProcessTime=2024-11-04T14:22:08.590133+08:00 2024-11-04 14:22:18.657 INFO 21062 --- [ocp-async-10,e4d6ccc333444cbd,785004d82e0a9015] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:22:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:22:18.590320+08:00 2024-11-04 14:22:19.171 WARN 21062 --- [ocp-async-4,d043654c794313d8,070612b0531a2ddd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:19.227 INFO 21062 --- [ocp-async-9,5e7177722a956b77,b461d477f675201d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:19.726 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:19.727 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:20.471 INFO 21062 --- [ocp-schedules-7,f8264aeb3895d83f,7a08639ae7fd82eb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:22:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9aa58480ab533238,17d7fbdbc85fd730] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:21.171 WARN 21062 --- [ocp-async-9,d898263401f839e9,2ffa7bedff75a7aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:21.227 INFO 21062 --- [ocp-async-8,f927a4e1211dc2a2,60411ae70ecd713e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:22.727 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:22.728 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:22.827 INFO 21062 --- [ocp-schedules-10,8e0365e9350bac31,c317e25d07b2cf40] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:22:23.171 WARN 21062 --- [ocp-async-8,d08f4712af0bd25b,427803c02b177372] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:23.227 INFO 21062 --- [ocp-async-7,64c28a616aa62aee,12d85c38de60ed77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:23.983 INFO 21062 --- [prometheus-minute-schedule-1,2e9ac5448ebe9792,ed7722a4a133ce62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:25.171 WARN 21062 --- [ocp-async-7,19a67bbeeed3df53,b32bfc77a71987ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:25.227 INFO 21062 --- [ocp-async-1,82dbbc2b6efea2f4,1549d562ccc01305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:25.728 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:25.728 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:26.983 INFO 21062 --- [prometheus-minute-schedule-1,25b0138d2c1cfb72,f94e99f9aaef0299] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:26.984 INFO 21062 --- [prometheus-hour-schedule-1,641f91cd9058217c,3aa74ee8ce790c42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:22:27.171 WARN 21062 --- [ocp-async-1,46362ed8698530f9,79a24e417b95de8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:27.228 INFO 21062 --- [ocp-async-10,1ee0cfc401322437,beebc3ede58e72ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:28.498 INFO 21062 --- [ocp-schedules-23,92244dfe6f71519e,68e74d429d31dbe4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:22:28.499 INFO 21062 --- [ocp-schedules-21,f0e0094afad8dc6d,8f632941cb7598f1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:22:28.500 INFO 21062 --- [ocp-schedules-9,9c8b1ad08a12c39e,49920a0443ade973] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:22:28.500 INFO 21062 --- [ocp-schedules-29,e7fd36c4553d2104,89dcbe45053f10e6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:22:28.502 INFO 21062 --- [ocp-schedules-19,402e5b11e2776629,2cb783c024224a55] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:22:28.503 INFO 21062 --- [ocp-schedules-23,92244dfe6f71519e,68e74d429d31dbe4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:22:28.504 INFO 21062 --- [ocp-schedules-9,9c8b1ad08a12c39e,49920a0443ade973] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:28.504 INFO 21062 --- [ocp-schedules-5,2efc5102d954accb,e2ba3810519f8454] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:22:28.504 INFO 21062 --- [ocp-schedules-13,c7c105f8a4ddde49,e2ec00936776177d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:22:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:22:28.504 INFO 21062 --- [ocp-schedules-21,f0e0094afad8dc6d,8f632941cb7598f1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:22:28.505 INFO 21062 --- [ocp-schedules-29,e7fd36c4553d2104,89dcbe45053f10e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:28.507 INFO 21062 --- [ocp-schedules-13,c7c105f8a4ddde49,e2ec00936776177d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:28.510 INFO 21062 --- [ocp-schedules-22,16b09a1e4bd2dae8,a13410b439ae60a3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:22:28.510 INFO 21062 --- [ocp-schedules-22,16b09a1e4bd2dae8,a13410b439ae60a3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:22:28.511 INFO 21062 --- [ocp-schedules-26,5cf1855a7b631588,25f1ae9015ea36f1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:22:28.511 INFO 21062 --- [ocp-schedules-26,5cf1855a7b631588,25f1ae9015ea36f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:22:28.511 INFO 21062 --- [ocp-schedules-20,7a997e143d6a1261,df9d3907c30392ec] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:22:28.511 INFO 21062 --- [ocp-schedules-20,7a997e143d6a1261,df9d3907c30392ec] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:22:28.536 INFO 21062 --- [ocp-schedules-27,47b7d28d54c844cf,75ea40899d459d67] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:22:28.536 INFO 21062 --- [ocp-schedules-27,47b7d28d54c844cf,75ea40899d459d67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:22:28.576 INFO 21062 --- [ocp-schedules-7,850aed48806f5ebb,57a105ec6d8b9b63] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:22:28.576 INFO 21062 --- [ocp-schedules-7,850aed48806f5ebb,57a105ec6d8b9b63] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:22:28.577 INFO 21062 --- [pool-12-thread-10,5bfd4d81c231ef4e,09ebfaade1cde09a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:22:28.577 INFO 21062 --- [pool-12-thread-11,5bfd4d81c231ef4e,ba10085c9304d334] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:22:28.577 INFO 21062 --- [pool-12-thread-12,5bfd4d81c231ef4e,9258f2028953c558] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:22:28.578 INFO 21062 --- [pool-12-thread-9,5bfd4d81c231ef4e,87cdc2a97716a62c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:22:28.583 INFO 21062 --- [ForkJoinPool-11-worker-2,5bfd4d81c231ef4e,a388649ce3f6029e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:22:28.583 INFO 21062 --- [ForkJoinPool-11-worker-2,5bfd4d81c231ef4e,a388649ce3f6029e] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:22:28.583 INFO 21062 --- [ForkJoinPool-11-worker-2,5bfd4d81c231ef4e,a388649ce3f6029e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:22:28.583 INFO 21062 --- [pool-12-thread-11,5bfd4d81c231ef4e,ba10085c9304d334] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:22:28.585 INFO 21062 --- [pool-12-thread-9,5bfd4d81c231ef4e,87cdc2a97716a62c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:22:28.585 INFO 21062 --- [pool-12-thread-9,5bfd4d81c231ef4e,87cdc2a97716a62c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:22:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:22:28.589350+08:00, lastProcessTime=2024-11-04T14:22:18.590320+08:00 2024-11-04 14:22:28.604 INFO 21062 --- [pool-12-thread-12,5bfd4d81c231ef4e,9258f2028953c558] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:22:28.605 INFO 21062 --- [pool-12-thread-12,5bfd4d81c231ef4e,9258f2028953c558] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:22:28.607 INFO 21062 --- [pool-12-thread-12,5bfd4d81c231ef4e,9258f2028953c558] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:22:28.609 INFO 21062 --- [ForkJoinPool-11-worker-2,5bfd4d81c231ef4e,760c6d78855aa065] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:22:28.609 INFO 21062 --- [pool-12-thread-10,5bfd4d81c231ef4e,09ebfaade1cde09a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:22:28.657 INFO 21062 --- [ocp-async-1,29477017d5f1696e,a45889b7cfda0b0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:22:28.729 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:28.729 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:28.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:22:28.589350+08:00 2024-11-04 14:22:28.794 INFO 21062 --- [ocp-schedules-5,ff33ca8216a66f74,99bd8a3ab468b88c] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:22:28.794 INFO 21062 --- [ocp-schedules-5,ff33ca8216a66f74,99bd8a3ab468b88c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:22:28.827 INFO 21062 --- [ocp-schedules-22,d9d274c5bcdd5c5b,83fe8ca34273d569] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:22:28.828 INFO 21062 --- [ocp-schedules-19,d56956376bda6c32,2277b43b7629a112] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 14:22:28.828 INFO 21062 --- [ocp-schedules-19,d56956376bda6c32,2277b43b7629a112] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 14:22:28.828 INFO 21062 --- [ocp-schedules-19,d56956376bda6c32,2277b43b7629a112] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=396 2024-11-04 14:22:28.850 INFO 21062 --- [ocp-schedules-19,d56956376bda6c32,2277b43b7629a112] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=396 2024-11-04 14:22:28.912 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:22:28.930 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:22:28.932 WARN 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:22:28.932 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:22:28.933 WARN 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:22:28.940 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:22:28.941 WARN 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:22:28.942 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:22:28.943 WARN 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:22:28.949 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:22:28.950 WARN 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:22:28.951 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:22:28.952 WARN 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:22:28.958 INFO 21062 --- [ocp-schedules-20,81b6b27c9330468c,8ddd6ccd3a0f8333] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:22:29.060 INFO 21062 --- [ocp-schedules-8,d7fe5802e07db5fd,62de4e0679b28d3f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 14:22:29.064 INFO 21062 --- [ocp-schedules-8,d7fe5802e07db5fd,62de4e0679b28d3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:29.089 INFO 21062 --- [ocp-schedules-28,427b290e85dc1f7a,ca6a7bdd84b1dc9b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:22:29.089 INFO 21062 --- [ocp-schedules-28,427b290e85dc1f7a,ca6a7bdd84b1dc9b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:22:29.100 ERROR 21062 --- [ocp-schedules-8,d7fe5802e07db5fd,62de4e0679b28d3f] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:22:29.100 INFO 21062 --- [ocp-schedules-8,d7fe5802e07db5fd,62de4e0679b28d3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 14:22:29.159 INFO 21062 --- [ocp-schedules-7,e264cc5930f2a72d,60e512a3c184d549] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:22:29.172 WARN 21062 --- [ocp-async-7,cfc7f328e69dce17,613dd71f8410ff63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:29.191 INFO 21062 --- [ocp-schedules-4,6abbf136e6554162,d0bee8ee3565b0a6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:22:29.196 INFO 21062 --- [ocp-schedules-4,6abbf136e6554162,d0bee8ee3565b0a6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:22:29.197 INFO 21062 --- [ocp-schedules-4,6abbf136e6554162,d0bee8ee3565b0a6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:22:29.228 INFO 21062 --- [ocp-async-4,f27928f09f957c90,1663e217774a4025] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:29.340 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:22:29.340 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:22:29.347 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:22:29.347 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:22:29.347 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:22:29.357 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:22:29.357 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:22:29.357 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:22:29.360 INFO 21062 --- [ocp-schedules-3,4a9af9d85b7087db,e635867e2c650a6f] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:22:29.520 INFO 21062 --- [ocp-schedules-17,7cd74687c66beacf,b80e1c4e4afefb7d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:22:29.520 INFO 21062 --- [ocp-schedules-17,7cd74687c66beacf,b80e1c4e4afefb7d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:22:29.525 INFO 21062 --- [ocp-schedules-17,7cd74687c66beacf,b80e1c4e4afefb7d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:22:29.525 INFO 21062 --- [ocp-schedules-17,7cd74687c66beacf,b80e1c4e4afefb7d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:22:29.529 INFO 21062 --- [ocp-schedules-17,7cd74687c66beacf,b80e1c4e4afefb7d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:22:29.529 INFO 21062 --- [ocp-schedules-17,7cd74687c66beacf,b80e1c4e4afefb7d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:22:29.746 INFO 21062 --- [ocp-schedules-10,0088980bbf8d289d,ee450d3d18596bb0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:22:29.747 INFO 21062 --- [ocp-schedules-10,0088980bbf8d289d,ee450d3d18596bb0] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:22:29.776 INFO 21062 --- [pool-41-thread-1,0088980bbf8d289d,9290cfe9ff5a5be7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:22:29.803 WARN 21062 --- [ocp-schedules-10,0088980bbf8d289d,ee450d3d18596bb0] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:22:29.803 INFO 21062 --- [ocp-schedules-10,0088980bbf8d289d,ee450d3d18596bb0] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:22:29.983 INFO 21062 --- [prometheus-minute-schedule-1,667f41b1fbab10fd,2d28e22a123bb0bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:22:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:22:30.476 INFO 21062 --- [ocp-schedules-26,fb331c3d1203c011,6117adf8712effd3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:22:31.172 WARN 21062 --- [ocp-async-4,a872056be77993bb,de1954f4f7c31111] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:31.228 INFO 21062 --- [ocp-async-9,125fd3dfa151195f,9a00ba333dd6956c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:31.473 INFO 21062 --- [ocp-schedules-30,5173edc8dae1803e,3611d0d364ffc876] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:22:31.478 INFO 21062 --- [ocp-schedules-30,5173edc8dae1803e,3611d0d364ffc876] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:22:31.478 INFO 21062 --- [ocp-schedules-30,5173edc8dae1803e,3611d0d364ffc876] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:22:31.524 INFO 21062 --- [ocp-schedules-23,41cc14f96d917830,da975f7b135bdf48] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:22:31.524 INFO 21062 --- [ocp-schedules-23,41cc14f96d917830,da975f7b135bdf48] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:22:31.549 INFO 21062 --- [ocp-schedules-23,41cc14f96d917830,da975f7b135bdf48] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:22:31.549 INFO 21062 --- [ocp-schedules-23,41cc14f96d917830,da975f7b135bdf48] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:22:31.551 INFO 21062 --- [ocp-schedules-27,efdf62438576731c,898e1e5e96b4dd4b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:22:31.729 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:31.730 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:31.740 INFO 21062 --- [ocp-schedules-1,3f973a486e62a357,b6a4c3d4458b6d64] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:22:31.740+08:00, clusters=[1] 2024-11-04 14:22:31.742 INFO 21062 --- [ocp-schedules-1,3f973a486e62a357,b6a4c3d4458b6d64] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:22:31.886 INFO 21062 --- [ocp-schedules-10,4782d74443cf9107,3d4c66052f59043a] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:22:31.889 INFO 21062 --- [ocp-schedules-10,4782d74443cf9107,3d4c66052f59043a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:31.899 INFO 21062 --- [ocp-schedules-22,72acdc65af24176e,10e6687c402bca5f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:22:31.899+08:00, clusters=[1] 2024-11-04 14:22:31.903 INFO 21062 --- [ocp-schedules-22,72acdc65af24176e,10e6687c402bca5f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:22:31.944 ERROR 21062 --- [ocp-schedules-10,4782d74443cf9107,3d4c66052f59043a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:22:31.944 INFO 21062 --- [ocp-schedules-10,4782d74443cf9107,3d4c66052f59043a] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:22:32.840 INFO 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:22:32.841 INFO 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:22:32.841 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:22:32.841 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:22:32.850 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:22:32.851 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:22:32.851 INFO 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:22:32.852 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:22:32.852 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:22:32.852 WARN 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:22:32.853 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:22:32.853 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:22:32.853 INFO 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:22:32.854 INFO 21062 --- [ocp-schedules-6,15beb4eea4f7f210,f5dc351364ef6e1c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:22:32.854 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:22:32.854 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:22:32.855 WARN 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:22:32.861 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:22:32.861 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:22:32.862 INFO 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:22:32.881 ERROR 21062 --- [ocp-schedules-2,1c692bbce625e47d,fa9b12ebda4c1711] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:22:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f13d8f7d8502452d,0653b3133e792ae0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:33.172 WARN 21062 --- [ocp-async-9,1b06f7227724a75c,19c58541699a81fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:33.228 INFO 21062 --- [ocp-async-8,fa4bce53ba3824a3,cce4bdbae3a8e156] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:34.730 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:34.731 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:35.172 WARN 21062 --- [ocp-async-2,34f596e755ab0ad7,ae30c354280e5513] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:35.228 INFO 21062 --- [ocp-async-6,448ff0bd3c8ff7b7,22bdc4fce25357f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:35.983 INFO 21062 --- [prometheus-minute-schedule-1,6c48ea9befc4dd4e,934ba512e6d2e7c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:36.058 INFO 21062 --- [ocp-schedules-12,516e96b54fb6cd02,0cc83c56479dbc74] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:22:36.058 INFO 21062 --- [ocp-schedules-12,590ce76696ff04ce,b7ded97dea8be98a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:22:36.062 INFO 21062 --- [ocp-schedules-12,590ce76696ff04ce,b7ded97dea8be98a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:36.093 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:22:36.093 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:22:37.172 WARN 21062 --- [ocp-async-3,d1d31603c67bff56,11546b271be8a6f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:37.229 INFO 21062 --- [ocp-async-1,3d2c17bd274f4845,08767e43b0497421] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:37.731 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:37.732 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:38.505 INFO 21062 --- [ocp-schedules-9,9c8b1ad08a12c39e,49920a0443ade973] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:22:38.506 INFO 21062 --- [ocp-schedules-29,e7fd36c4553d2104,89dcbe45053f10e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:22:38.508 INFO 21062 --- [ocp-schedules-13,c7c105f8a4ddde49,e2ec00936776177d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:22:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:22:38.591402+08:00, lastProcessTime=2024-11-04T14:22:28.589350+08:00 2024-11-04 14:22:38.657 INFO 21062 --- [ocp-async-6,424e33e04d240194,ada331df6b7dbc1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:22:38.675 INFO 21062 --- [ocp-schedules-18,f5463618dbdcc9e4,f9b9b7bfaa3629aa] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:22:38.678 INFO 21062 --- [ocp-schedules-18,f5463618dbdcc9e4,f9b9b7bfaa3629aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:22:38.591402+08:00 2024-11-04 14:22:38.764 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:22:38.766 ERROR 21062 --- [ocp-schedules-18,f5463618dbdcc9e4,f9b9b7bfaa3629aa] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:22:38.766 INFO 21062 --- [ocp-schedules-18,f5463618dbdcc9e4,f9b9b7bfaa3629aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:22:38.983 INFO 21062 --- [prometheus-minute-schedule-1,dc239d3d5872a2bb,a6f96e72c768dcd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:39.173 WARN 21062 --- [ocp-async-10,8697b13fa4db0adf,ce7753937ae2285b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:39.229 INFO 21062 --- [ocp-async-2,7042a5c62f7594a7,7b34879e4b2f1ada] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:22:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:22:40.479 INFO 21062 --- [ocp-schedules-29,642df9413a90ae27,3ddd87fae408b799] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:22:40.732 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:40.733 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:41.173 WARN 21062 --- [ocp-async-7,36f42c30b1cc8505,0427a1a2ed4c7089] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:41.229 INFO 21062 --- [ocp-async-3,0a652f3d2050e722,03d9439315963724] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:22:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:22:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:22:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:22:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:22:41.830 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:22:41.830 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-11-04 14:22:41.983 INFO 21062 --- [prometheus-minute-schedule-1,ec7d8502a1283b8b,d9051c9ea6c2b499] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:42.939 INFO 21062 --- [ocp-schedules-7,97b76c3c45991a2d,260a8f6b85da7f2c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:22:43.173 WARN 21062 --- [ocp-async-4,2c19f9190037ceaa,59927136cd06b61a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:43.229 INFO 21062 --- [ocp-async-5,33db4a53d72c67e9,2bac26aedea003bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:43.733 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:43.733 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3d1424a5212e47ea,3e2fb4792644b69c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:45.173 WARN 21062 --- [ocp-async-9,6317e68da53906bb,8adc0073a5ece9b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:45.229 INFO 21062 --- [ocp-async-8,9cd96a907c51ecbf,9e8c2af3ad7f73ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:46.734 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:46.734 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:47.173 WARN 21062 --- [ocp-async-2,1e5e9c69080beea1,e10111d407ba606a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:47.229 INFO 21062 --- [ocp-async-6,05ec3bef180cc857,22c1a8c3e52252e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:47.983 INFO 21062 --- [prometheus-minute-schedule-1,d1c6d004c247ea8c,88f6360bf95b5c15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:22:48.591738+08:00, lastProcessTime=2024-11-04T14:22:38.591402+08:00 2024-11-04 14:22:48.658 INFO 21062 --- [ocp-async-9,883a90049b75b919,4a46f22b200f0a55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:22:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:22:48.591738+08:00 2024-11-04 14:22:49.173 WARN 21062 --- [ocp-async-4,f96aff84c6263980,654f60362927d2cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:49.230 INFO 21062 --- [ocp-async-5,49299051af97207e,cc62bb0347edf5d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:49.735 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:49.735 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:50.482 INFO 21062 --- [ocp-schedules-11,4b63a2ca671185cc,4fecff7aef72c415] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:22:50.983 INFO 21062 --- [prometheus-minute-schedule-1,58bcd310209aa0cc,10a32786641fca34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:51.174 WARN 21062 --- [ocp-async-10,35f1b37f3ef7f82b,175daa15adf40bd1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:51.230 INFO 21062 --- [ocp-async-8,53c548a4d0a399c1,09f767f1e988ebb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:51.481 INFO 21062 --- [ocp-schedules-26,f4adb40e838f00e7,ec334475ff26a386] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:22:51.488 INFO 21062 --- [ocp-schedules-26,f4adb40e838f00e7,ec334475ff26a386] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:22:51.488 INFO 21062 --- [ocp-schedules-26,f4adb40e838f00e7,ec334475ff26a386] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:22:52.735 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:52.736 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:52.963 INFO 21062 --- [ocp-schedules-13,3b6f5ac529451b9b,e4bebb8571e803cb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:22:53.174 WARN 21062 --- [ocp-async-8,26699759623e5401,a2271cbda3d26d4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:53.230 INFO 21062 --- [ocp-async-6,2c1917207c183eb4,f6e95a82d59fceb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6b1677f9ccb8e4f1,26a26fa4f91a16d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:55.174 WARN 21062 --- [ocp-async-6,bd9ed93b43f6dc5a,a5b9ef7db70e2182] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:55.230 INFO 21062 --- [ocp-async-1,f552095846856ecc,c5ed649dc669d8d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:55.736 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:55.737 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:56.983 INFO 21062 --- [prometheus-minute-schedule-1,639eeb098ce65a30,dbd60326b9bd3724] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:22:57.174 WARN 21062 --- [ocp-async-1,82420b5b54c95cf9,97ea40947ca0fd10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:57.230 INFO 21062 --- [ocp-async-9,3abe454b54aae7a1,5a22900932258c17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:58.500 INFO 21062 --- [ocp-schedules-20,17e84797eebe029b,b9a9b8f00109d119] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:22:58.500 INFO 21062 --- [ocp-schedules-27,0b44dc864fcc91b4,f769d3b76e2823f6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:22:58.502 INFO 21062 --- [ocp-schedules-11,e6d71591408e0586,b6591f424dcedd99] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:22:58.503 INFO 21062 --- [ocp-schedules-20,17e84797eebe029b,b9a9b8f00109d119] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:58.503 INFO 21062 --- [ocp-schedules-27,0b44dc864fcc91b4,f769d3b76e2823f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:22:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:22:58.505 INFO 21062 --- [ocp-schedules-2,640ae576c848d1b5,bdd2575ec54b5f62] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:22:58.511 INFO 21062 --- [ocp-schedules-13,b8c4f69e3c6bc17b,41702c60abcea1da] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:22:58.511 INFO 21062 --- [ocp-schedules-24,e225841e7339e3e5,d757b5653e87bd63] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:22:58.511 INFO 21062 --- [ocp-schedules-13,b8c4f69e3c6bc17b,41702c60abcea1da] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:22:58.511 INFO 21062 --- [ocp-schedules-24,e225841e7339e3e5,d757b5653e87bd63] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:22:58.511 INFO 21062 --- [ocp-schedules-28,cc3464f1299d95e0,84d93c5411ed2662] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:22:58.511 INFO 21062 --- [ocp-schedules-28,cc3464f1299d95e0,84d93c5411ed2662] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:22:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:22:58.589685+08:00, lastProcessTime=2024-11-04T14:22:48.591738+08:00 2024-11-04 14:22:58.658 INFO 21062 --- [ocp-async-4,f68313afe3aaa373,4a45398f43bf594b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:22:58.737 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:22:58.738 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:22:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:22:58.589685+08:00 2024-11-04 14:22:59.162 INFO 21062 --- [ocp-schedules-14,db88064a350bd9c0,44842f8a15c57997] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:22:59.174 WARN 21062 --- [ocp-async-8,cb4890cf3867453d,3260692e6bf83bd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:22:59.230 INFO 21062 --- [ocp-async-6,0d4281f60493056c,8fb6ed179af7b7af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:22:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e6fad155d24b3208,d2eee3ca331f7dbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701380, interval=60 2024-11-04 14:23:00.484 INFO 21062 --- [ocp-schedules-14,bfd53f6761c157d7,55517ccda8782beb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:23:01.174 WARN 21062 --- [ocp-async-6,515934507ceaa014,de6aeb4a17d3cce2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:01.230 INFO 21062 --- [ocp-async-1,4d5e286038ddab85,32656473dcd7d9aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:01.738 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:01.739 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5bd81ab261c693be,eb5942fa6d6a9eba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:03.003 INFO 21062 --- [ocp-schedules-17,b06eb6413d420a04,d40564c4c213a6b4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:23:03.175 WARN 21062 --- [ocp-async-1,c43044f649526158,07d725346d561c30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:03.231 INFO 21062 --- [ocp-async-9,047c2527903cb24e,7cd57274a312d326] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:04.739 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:04.740 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:05.175 WARN 21062 --- [ocp-async-9,3528fbd43acaaf01,0f0bb2ff1c99a5ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:05.231 INFO 21062 --- [ocp-async-7,9cc26f8645f521d4,2c356cecc6bf3731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:05.983 INFO 21062 --- [prometheus-minute-schedule-1,22d55d2754b5cd70,d61e5c528bc412b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:06.093 INFO 21062 --- [ocp-schedules-12,590ce76696ff04ce,b7ded97dea8be98a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:23:06.094 INFO 21062 --- [ocp-schedules-12,b0ad352c6da5ba31,7cd32334dc255bcb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:23:06.097 INFO 21062 --- [ocp-schedules-12,b0ad352c6da5ba31,7cd32334dc255bcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:06.126 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:23:06.126 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:23:07.175 WARN 21062 --- [ocp-async-7,8be09f0b83b83f8a,ff4f0e2b1b05f510] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:07.231 INFO 21062 --- [ocp-async-3,8a935fc0862e49d9,d1f08a256612df89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:07.740 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:07.741 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:08.504 INFO 21062 --- [ocp-schedules-27,0b44dc864fcc91b4,f769d3b76e2823f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:23:08.505 INFO 21062 --- [ocp-schedules-20,17e84797eebe029b,b9a9b8f00109d119] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:23:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:23:08.590353+08:00, lastProcessTime=2024-11-04T14:22:58.589685+08:00 2024-11-04 14:23:08.658 INFO 21062 --- [ocp-async-8,4e516650c26f22a8,798d71261265ff79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:23:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:23:08.590353+08:00 2024-11-04 14:23:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d4b4e63652b0952f,b73bf7119d7b81b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:09.017 INFO 21062 --- [ocp-schedules-2,5292a7712a2e9145,4e3b07914a9fc34a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:23:09.020 INFO 21062 --- [ocp-schedules-2,5292a7712a2e9145,4e3b07914a9fc34a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:09.021 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:23:09.175 WARN 21062 --- [ocp-async-1,b3d3f40e51de020d,ad7eee815c714d6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:09.231 INFO 21062 --- [ocp-async-9,88e3feb57ad6540e,b8acafc02b7b6880] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:23:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:23:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:23:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:23:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:23:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:23:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:23:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:23:10.488 INFO 21062 --- [ocp-schedules-14,4c0a3782a28068ae,3af817a5ecf84e84] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:23:10.741 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:10.741 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:11.175 WARN 21062 --- [ocp-async-9,3b9b0c039b596672,60d4170121b287dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:11.231 INFO 21062 --- [ocp-async-7,2d5baf833c83fd94,b4ad0f38d6f8341e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:11.490 INFO 21062 --- [ocp-schedules-19,e5aed43cfcf5087a,426602d6473a9f6b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:23:11.497 INFO 21062 --- [ocp-schedules-19,e5aed43cfcf5087a,426602d6473a9f6b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:23:11.497 INFO 21062 --- [ocp-schedules-19,e5aed43cfcf5087a,426602d6473a9f6b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:23:11.552 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=4, affectRows=4 2024-11-04 14:23:11.552 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=3, validCount=3 2024-11-04 14:23:11.552 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=18, affectRows=18 2024-11-04 14:23:11.552 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=18, validCount=18 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=40, affectRows=40 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=23, affectRows=23 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=23, validCount=23 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=40, validCount=40 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=41, affectRows=41 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=46, affectRows=46 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=74, affectRows=74 2024-11-04 14:23:11.553 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=41, validCount=41 2024-11-04 14:23:11.554 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=46, validCount=46 2024-11-04 14:23:11.554 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=53, affectRows=53 2024-11-04 14:23:11.554 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701320, nodeCount=97, affectRows=97 2024-11-04 14:23:11.554 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=74, validCount=74 2024-11-04 14:23:11.554 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=52, validCount=52 2024-11-04 14:23:11.554 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701320, nodeCount=96, validCount=96 2024-11-04 14:23:11.983 INFO 21062 --- [prometheus-minute-schedule-1,5e8c5be713e56304,c5ad5adb8798079c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:13.029 INFO 21062 --- [ocp-schedules-7,382d3bb17d9da6d8,5d7a64e4522c1d44] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:23:13.175 WARN 21062 --- [ocp-async-7,9cbef166d621ded4,23a6d26ab3eee1f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:13.232 INFO 21062 --- [ocp-async-3,b6d7709e8b906c47,4ba64ad7c6dd312e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:13.741 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:13.742 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:13.984 INFO 21062 --- [prometheus-cache-monitor1,7eb7db76b7098fa6,5131cc3718c96828] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:23:13.984 INFO 21062 --- [prometheus-cache-monitor1,7eb7db76b7098fa6,5131cc3718c96828] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:23:14.247 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:17:14.228, end=2024-11-04T14:22:14.244} 2024-11-04 14:23:14.247 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:17:14.229, end=2024-11-04T14:22:14.244} 2024-11-04 14:23:14.257 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:23:14.257 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:23:14.257 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:23:14.257 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:23:14.983 INFO 21062 --- [prometheus-minute-schedule-1,713639c31a161d00,fb9d768fee02aed7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:15.045 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:19:15.028, end=2024-11-04T14:21:15.043} 2024-11-04 14:23:15.050 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:23:15.050 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:23:15.176 WARN 21062 --- [ocp-async-3,98aa85efc004beff,d9880e0a26af309d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:15.232 INFO 21062 --- [ocp-async-5,e891c481eebd2b27,df602656cd74c001] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:16.742 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:16.743 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:17.176 WARN 21062 --- [ocp-async-5,30b98c5a6c715db6,b9613e2044dcdbbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:17.232 INFO 21062 --- [ocp-async-2,d13005e1cf7db4d4,b21d8b8646b11c59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:17.983 INFO 21062 --- [prometheus-minute-schedule-1,42692e747b63c5f5,6a79ef83b71c24b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:23:18.590572+08:00, lastProcessTime=2024-11-04T14:23:08.590353+08:00 2024-11-04 14:23:18.658 INFO 21062 --- [ocp-async-1,a67d223cb130a92d,ea0f9b2a8e99d5fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:23:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:23:18.590572+08:00 2024-11-04 14:23:19.021 INFO 21062 --- [ocp-schedules-2,5292a7712a2e9145,4e3b07914a9fc34a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:23:19.176 WARN 21062 --- [ocp-async-7,221e1a223a048434,97fc873f8f6fcb21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:19.232 INFO 21062 --- [ocp-async-3,ce9fc791248bb02b,4a98d52eeb41dd09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:19.743 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:19.744 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:20.490 INFO 21062 --- [ocp-schedules-8,d72b8ba12f401c4a,8cb04ef23e0045a2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:23:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e334082c6085ed0b,91b3e99024a82163] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:21.176 WARN 21062 --- [ocp-async-3,a3908893ed0883fe,7fca92d84e23b267] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:21.232 INFO 21062 --- [ocp-async-5,c9c6f83ff5346ba1,5f01a83625c15685] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:22.744 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:22.745 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:23.070 INFO 21062 --- [ocp-schedules-6,7a2ba2e0f8920a48,4aff3c9613f284b6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:23:23.176 WARN 21062 --- [ocp-async-5,6e4ab2807e6fceb9,ab28d7ac02a24b31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:23.233 INFO 21062 --- [ocp-async-2,31c7acd661d1aa91,6a66f9ee9313127b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:23.983 INFO 21062 --- [prometheus-minute-schedule-1,482ac6d674c9d310,bfa280cecbda1462] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:25.176 WARN 21062 --- [ocp-async-2,8b99d1fecd13c20a,878e28dcebd73360] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:25.233 INFO 21062 --- [ocp-async-4,959bf36807603909,5e6129f2b7d4339a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:25.745 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:25.746 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2a6577f3ed452cbf,0b72d698819a8d05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b96c7af9e9d19a15,3462617adb04025c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:23:27.177 WARN 21062 --- [ocp-async-4,423bad34dd56f8a1,20b7290df87676f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:27.233 INFO 21062 --- [ocp-async-1,8ccfbb63ac0aa548,230ae20e90289490] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:28.498 INFO 21062 --- [ocp-schedules-27,5882065076defa2b,2b1586029ad822e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:23:28.499 INFO 21062 --- [ocp-schedules-13,3bcb2936d27d3d6e,98ddbe975c96cb86] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:23:28.500 INFO 21062 --- [ocp-schedules-22,6fe49b852678144c,2ac2fdc991af65b3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:23:28.500 INFO 21062 --- [ocp-schedules-15,6d73989edab3741b,b9295a4e67423047] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:23:28.502 INFO 21062 --- [ocp-schedules-27,5882065076defa2b,2b1586029ad822e3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:23:28.503 INFO 21062 --- [ocp-schedules-30,93f5b82146b5d03a,f868cc0fb87d5324] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:23:28.504 INFO 21062 --- [ocp-schedules-22,6fe49b852678144c,2ac2fdc991af65b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:28.504 INFO 21062 --- [ocp-schedules-15,6d73989edab3741b,b9295a4e67423047] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:23:28.505 INFO 21062 --- [ocp-schedules-23,78a3cbc270324c86,8575620602458c83] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:23:28.507 INFO 21062 --- [ocp-schedules-13,3bcb2936d27d3d6e,98ddbe975c96cb86] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:23:28.510 INFO 21062 --- [ocp-schedules-28,34514e79f964df76,f820b5a04aa73cb8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:23:28.510 INFO 21062 --- [ocp-schedules-18,a1948c3c92134b51,7570c865eb9b3847] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:23:28.510 INFO 21062 --- [ocp-schedules-28,34514e79f964df76,f820b5a04aa73cb8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:23:28.510 INFO 21062 --- [ocp-schedules-18,a1948c3c92134b51,7570c865eb9b3847] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:23:28.512 INFO 21062 --- [ocp-schedules-24,088e1e72b41094ed,f140284e47297836] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:23:28.512 INFO 21062 --- [ocp-schedules-24,088e1e72b41094ed,f140284e47297836] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:23:28.578 INFO 21062 --- [ocp-schedules-14,2f114188dcb367d9,a2df71b01604f091] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 14:23:28.578 INFO 21062 --- [ocp-schedules-14,2f114188dcb367d9,a2df71b01604f091] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 14:23:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:23:28.590067+08:00, lastProcessTime=2024-11-04T14:23:18.590572+08:00 2024-11-04 14:23:28.658 INFO 21062 --- [ocp-async-4,d66fb2af9bb528ba,452725f43845af7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:23:28.746 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:28.747 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:23:28.590067+08:00 2024-11-04 14:23:28.829 INFO 21062 --- [ocp-schedules-11,51a1b8b7779058cc,bcdd2ed84295c7b6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:23:29.093 INFO 21062 --- [ocp-schedules-28,165a39f8940a7f17,98459712cbf13911] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:23:29.093 INFO 21062 --- [ocp-schedules-28,165a39f8940a7f17,98459712cbf13911] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:23:29.165 INFO 21062 --- [ocp-schedules-24,3cee3a3d1a8baa1d,17548970fc2c0eb0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:23:29.177 WARN 21062 --- [ocp-async-2,cb9c9cb592304ab2,aae293eed99e310c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:29.197 INFO 21062 --- [ocp-schedules-4,7a6a5e26ab5870e7,5304fbb4219edf31] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:23:29.202 INFO 21062 --- [ocp-schedules-4,7a6a5e26ab5870e7,5304fbb4219edf31] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:23:29.202 INFO 21062 --- [ocp-schedules-4,7a6a5e26ab5870e7,5304fbb4219edf31] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:23:29.233 INFO 21062 --- [ocp-async-7,8517ee6fdbfde203,318773cff1cf4094] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:29.530 INFO 21062 --- [ocp-schedules-29,793c00e8d24c5111,b83523053b5a13da] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:23:29.530 INFO 21062 --- [ocp-schedules-29,793c00e8d24c5111,b83523053b5a13da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:23:29.531 INFO 21062 --- [ocp-schedules-29,793c00e8d24c5111,b83523053b5a13da] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:23:29.531 INFO 21062 --- [ocp-schedules-29,793c00e8d24c5111,b83523053b5a13da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:23:29.532 INFO 21062 --- [ocp-schedules-29,793c00e8d24c5111,b83523053b5a13da] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:23:29.532 INFO 21062 --- [ocp-schedules-29,793c00e8d24c5111,b83523053b5a13da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:23:29.983 INFO 21062 --- [prometheus-minute-schedule-1,f2df9b889b7487b9,18a2391972ca5920] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:23:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:23:30.493 INFO 21062 --- [ocp-schedules-4,25f143f18de6dd71,b977fea262712f0a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:23:31.177 WARN 21062 --- [ocp-async-7,b309993a5787ceca,632fdc891cbe2bf5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:31.233 INFO 21062 --- [ocp-async-3,4d515802bcc39999,2f12ddaa8d752d1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:31.499 INFO 21062 --- [ocp-schedules-21,6cadbf0099fabfb3,2d7f534b14be99f1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:23:31.505 INFO 21062 --- [ocp-schedules-21,6cadbf0099fabfb3,2d7f534b14be99f1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:23:31.505 INFO 21062 --- [ocp-schedules-21,6cadbf0099fabfb3,2d7f534b14be99f1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:23:31.550 INFO 21062 --- [ocp-schedules-19,347020f0b550e62e,5b9fc24268167f7b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:23:31.550 INFO 21062 --- [ocp-schedules-19,347020f0b550e62e,5b9fc24268167f7b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:23:31.576 INFO 21062 --- [ocp-schedules-19,347020f0b550e62e,5b9fc24268167f7b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:23:31.576 INFO 21062 --- [ocp-schedules-19,347020f0b550e62e,5b9fc24268167f7b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:23:31.578 INFO 21062 --- [ocp-schedules-25,44cac3badd94e0fc,7ac864b0f9f57457] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:23:31.747 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:31.748 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:31.768 INFO 21062 --- [ocp-schedules-27,d76f7c61d83d1d88,33a9658dd7aee8d1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:23:31.768+08:00, clusters=[1] 2024-11-04 14:23:31.769 INFO 21062 --- [ocp-schedules-27,d76f7c61d83d1d88,33a9658dd7aee8d1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:23:31.929 INFO 21062 --- [ocp-schedules-17,bed12135c92d5790,9f32b3f7070c612c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:23:31.929+08:00, clusters=[1] 2024-11-04 14:23:31.932 INFO 21062 --- [ocp-schedules-17,bed12135c92d5790,9f32b3f7070c612c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:23:32.891 INFO 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:23:32.891 INFO 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:23:32.891 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:23:32.891 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:23:32.902 INFO 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:23:32.902 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:23:32.902 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:23:32.904 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:23:32.904 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:23:32.904 WARN 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:23:32.904 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:23:32.904 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:23:32.905 INFO 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:23:32.906 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:23:32.906 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:23:32.906 WARN 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:23:32.913 INFO 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:23:32.914 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:23:32.914 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:23:32.926 ERROR 21062 --- [ocp-schedules-11,fe4cfc56032a2971,70444adb6e40f9a8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:23:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b57e58624a499e9b,43b9eef25edd05a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:33.096 INFO 21062 --- [ocp-schedules-1,4ad5782920cab33b,70481b518fc6f5f7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:23:33.177 WARN 21062 --- [ocp-async-3,8e073e120342f4fb,1ec45d63f37841b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:33.233 INFO 21062 --- [ocp-async-5,6b21f99752e1c244,6d2f8c637c6702b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:34.748 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:34.749 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:35.177 WARN 21062 --- [ocp-async-9,45e1f9abf526ea79,de1a1d498101a3dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:35.234 INFO 21062 --- [ocp-async-8,4e2dba11a6d56a58,f9dc9ca415184b1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c713d097dc8bb6c5,688a78480cb01f00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:36.126 INFO 21062 --- [ocp-schedules-12,b0ad352c6da5ba31,7cd32334dc255bcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:23:36.126 INFO 21062 --- [ocp-schedules-12,fce7735581f5164c,4b9f94d7431d0767] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:23:36.130 INFO 21062 --- [ocp-schedules-12,fce7735581f5164c,4b9f94d7431d0767] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:36.160 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:23:36.160 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:23:37.178 WARN 21062 --- [ocp-async-6,b232daadd9ff70d6,934a390a3b4722ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:37.234 INFO 21062 --- [ocp-async-4,5120865e39786edd,59e458a57e97a17c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:37.749 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:37.750 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:38.505 INFO 21062 --- [ocp-schedules-15,6d73989edab3741b,b9295a4e67423047] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:23:38.505 INFO 21062 --- [ocp-schedules-22,6fe49b852678144c,2ac2fdc991af65b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:23:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:23:38.590795+08:00, lastProcessTime=2024-11-04T14:23:28.590067+08:00 2024-11-04 14:23:38.658 INFO 21062 --- [ocp-async-8,cbd5c07ca9eacd62,69211796a2cddec0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:23:38.767 INFO 21062 --- [ocp-schedules-25,1661418995e64402,096cf6dd06b9e377] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:23:38.771 INFO 21062 --- [ocp-schedules-25,1661418995e64402,096cf6dd06b9e377] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:38.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:23:38.590795+08:00 2024-11-04 14:23:38.863 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:23:38.865 ERROR 21062 --- [ocp-schedules-25,1661418995e64402,096cf6dd06b9e377] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:23:38.865 INFO 21062 --- [ocp-schedules-25,1661418995e64402,096cf6dd06b9e377] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:23:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a6fdc55bd44e190f,2c1b5d8b45bb2d7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:39.178 WARN 21062 --- [ocp-async-1,ab5e6975f56b6559,d83d0ef56e92ddd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:39.234 INFO 21062 --- [ocp-async-9,db93cfb4bc0499bb,8ffdd127405a9717] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:23:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:23:40.497 INFO 21062 --- [ocp-schedules-18,f44bbf6f704d49a5,e6b4ba1921ded799] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:23:40.750 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:40.751 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:41.178 WARN 21062 --- [ocp-async-2,5f556da0ca4e76b2,241c265f2fbd1065] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:41.234 INFO 21062 --- [ocp-async-6,efde18c8a614e3cf,56a864fbf38cf05e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:41.983 INFO 21062 --- [prometheus-minute-schedule-1,8feff667846dfb4f,e36ab7dee1c5d01c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:43.137 INFO 21062 --- [ocp-schedules-9,3b1e93d358c6b56f,97c8d10055bcfad3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:23:43.178 WARN 21062 --- [ocp-async-7,31476122a3399e10,05a48a6fdf314f08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:43.234 INFO 21062 --- [ocp-async-10,5a94cc79ec6312a7,5c578d24e934a62f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:43.751 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:43.752 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:44.018 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fc045f4950a4a1e,da6faac4cfda3fa7] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.130.47.17, traceId 8fc045f4950a4a1e] 2024-11-04 14:23:44.025 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fc045f4950a4a1e,da6faac4cfda3fa7] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.130.47.17, traceId=8fc045f4950a4a1e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2024-11-04 14:23:44.195 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fc045f4950a4a1e,da6faac4cfda3fa7] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 42, current state: FAILED, new state: READY 2024-11-04 14:23:44.195 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fc045f4950a4a1e,da6faac4cfda3fa7] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 13, current state: FAILED, new state: READY 2024-11-04 14:23:44.221 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fc045f4950a4a1e,da6faac4cfda3fa7] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.130.47.17, serverIp=null, startTime=2024-11-04T14:23:44.028+08:00, endTime=2024-11-04T14:23:44.220+08:00, role=null, aliUid=null) 2024-11-04 14:23:44.235 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8fc045f4950a4a1e,da6faac4cfda3fa7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.130.47.17, traceId=8fc045f4950a4a1e, duration=226 ms] 2024-11-04 14:23:44.614 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,8a84de5d74b1ad1f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: RUNNING 2024-11-04 14:23:44.621 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,8a84de5d74b1ad1f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: RUNNING 2024-11-04 14:23:44.635 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=13, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:23:44.607+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=13}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:23:44.637 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:23:44.638 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=42, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:23:44.618+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=42}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:23:44.647 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,2d2fd0b687793eaa,1882fa2eebd98d12] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=2d2fd0b687793eaa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:44.642 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:23:44.645 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:23:44.647 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:23:44.650 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:23:44.655 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:23:44.668 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:44.674 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,2d2fd0b687793eaa,1882fa2eebd98d12] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=2d2fd0b687793eaa, duration=33 ms] 2024-11-04 14:23:44.672 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:44.674 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.678 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.722 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:44.723 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:44.728 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.730 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.771 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:23:44.774 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:23:44.796 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.799 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.808 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.811 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.843 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:44.848 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.854 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:44.859 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.891 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:23:44.906 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:23:44.915 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.918 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.939 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.942 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.963 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:44.968 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a6c56f558f160f14,530b14a7741b846b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:44.986 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:44.991 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.011 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:23:45.034 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:23:45.035 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.039 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.042 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:23:45.067 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.070 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.073 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:23:45.079 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:45.084 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.114 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:45.119 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.131 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 14:23:45.162 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 14:23:45.178 WARN 21062 --- [ocp-async-3,4b8ba1d0838289d5,eb3f8533714ab2fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:45.177 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.179 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.182 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:23:45.184 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:23:45.187 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:23:45.208 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.211 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.213 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:23:45.215 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:23:45.218 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:23:45.219 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:45.226 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.235 INFO 21062 --- [ocp-async-5,86d0c17b67ad00ab,78cf5f31dae96418] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:45.254 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:45.259 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.271 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 14:23:45.306 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 14:23:45.316 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.319 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.321 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:23:45.324 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:23:45.326 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:23:45.329 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:23:45.332 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:23:45.337 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:23:45.341 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:23:45.344 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:23:45.347 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:23:45.350 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:23:45.352 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.355 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.357 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:23:45.360 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:23:45.362 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:23:45.364 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:23:45.367 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:23:45.369 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:45.371 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:23:45.374 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.375 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:23:45.380 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:23:45.383 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:23:45.386 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:23:45.403 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:45.409 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.415 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:45.421 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.450 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:45.455 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.467 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:23:45.490 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.493 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.502 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:23:45.526 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.529 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.535 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:45.540 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.570 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:45.575 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.587 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:23:45.610 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.612 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.618 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:23:45.633 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:23:45.642 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.644 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.651 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:45.656 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.660 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:23:45.686 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:45.691 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.703 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12 2024-11-04 14:23:45.727 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,f1af6651c92dff51,9dbc9ea6ae9ba45a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.12, traceId=f1af6651c92dff51, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:23:45.733 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,f1af6651c92dff51,9dbc9ea6ae9ba45a] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:23:45.733 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,f1af6651c92dff51,9dbc9ea6ae9ba45a] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:23:45.734 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,f1af6651c92dff51,9dbc9ea6ae9ba45a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.12, traceId=f1af6651c92dff51, duration=12 ms] 2024-11-04 14:23:45.738 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11 2024-11-04 14:23:45.760 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,180969d336b46917,f380832b0fb15228] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.11, traceId=180969d336b46917, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:23:45.763 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,180969d336b46917,f380832b0fb15228] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:23:45.763 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,180969d336b46917,f380832b0fb15228] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:23:45.764 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,180969d336b46917,f380832b0fb15228] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.11, traceId=180969d336b46917, duration=6 ms] 2024-11-04 14:23:45.893 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.896 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.903 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:45.907 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.909 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.910 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.956 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,9d6869256946c568,70c34088f1906a1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=9d6869256946c568, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:23:45.954 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:45.955 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12 2024-11-04 14:23:45.960 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:45.965 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,9d6869256946c568,70c34088f1906a1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=9d6869256946c568, duration=16 ms] 2024-11-04 14:23:46.006 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11 2024-11-04 14:23:46.105 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.109 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.111 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:23:46.147 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:46.153 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.154 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.159 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.164 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:23:46.195 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/779f7704ceb74e8e on 10.46.128.12 2024-11-04 14:23:46.198 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:46.201 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/779f7704ceb74e8e on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/779f7704ceb74e8e, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.204 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.204 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/779f7704ceb74e8e on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/779f7704ceb74e8e, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.243 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:46.246 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d23577ca06c94022 on 10.46.128.11 2024-11-04 14:23:46.248 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.252 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d23577ca06c94022 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d23577ca06c94022, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.254 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d23577ca06c94022 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d23577ca06c94022, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.295 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/779f7704ceb74e8e/cpio && chmod +x /tmp/779f7704ceb74e8e/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/779f7704ceb74e8e/cpio on 10.46.128.12 2024-11-04 14:23:46.298 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:46.302 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/779f7704ceb74e8e/cpio && chmod +x /tmp/779f7704ceb74e8e/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/779f7704ceb74e8e/cpio on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/779f7704ceb74e8e/cpio && chmod +x /tmp/779f7704ceb74e8e/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/779f7704ceb74e8e/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.303 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.304 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/779f7704ceb74e8e/cpio && chmod +x /tmp/779f7704ceb74e8e/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/779f7704ceb74e8e/cpio on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/779f7704ceb74e8e/cpio && chmod +x /tmp/779f7704ceb74e8e/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/779f7704ceb74e8e/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.347 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:46.350 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d23577ca06c94022/cpio && chmod +x /tmp/d23577ca06c94022/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d23577ca06c94022/cpio on 10.46.128.11 2024-11-04 14:23:46.352 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.357 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d23577ca06c94022/cpio && chmod +x /tmp/d23577ca06c94022/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d23577ca06c94022/cpio on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d23577ca06c94022/cpio && chmod +x /tmp/d23577ca06c94022/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d23577ca06c94022/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.360 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d23577ca06c94022/cpio && chmod +x /tmp/d23577ca06c94022/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d23577ca06c94022/cpio on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d23577ca06c94022/cpio && chmod +x /tmp/d23577ca06c94022/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d23577ca06c94022/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.399 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/779f7704ceb74e8e on 10.46.128.12 2024-11-04 14:23:46.402 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:46.405 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/779f7704ceb74e8e on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/779f7704ceb74e8e, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.407 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.408 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/779f7704ceb74e8e on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/779f7704ceb74e8e, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.451 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:46.454 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d23577ca06c94022 on 10.46.128.11 2024-11-04 14:23:46.457 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.461 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d23577ca06c94022 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d23577ca06c94022, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.463 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d23577ca06c94022 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d23577ca06c94022, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.503 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/779f7704ceb74e8e/install_package.sh && chmod +x /tmp/779f7704ceb74e8e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/779f7704ceb74e8e/install_package.sh on 10.46.128.12 2024-11-04 14:23:46.506 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:46.510 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/779f7704ceb74e8e/install_package.sh && chmod +x /tmp/779f7704ceb74e8e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/779f7704ceb74e8e/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/779f7704ceb74e8e/install_package.sh && chmod +x /tmp/779f7704ceb74e8e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/779f7704ceb74e8e/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.511 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.513 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/779f7704ceb74e8e/install_package.sh && chmod +x /tmp/779f7704ceb74e8e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/779f7704ceb74e8e/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/779f7704ceb74e8e/install_package.sh && chmod +x /tmp/779f7704ceb74e8e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/779f7704ceb74e8e/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.555 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:23:46.558 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d23577ca06c94022/install_package.sh && chmod +x /tmp/d23577ca06c94022/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d23577ca06c94022/install_package.sh on 10.46.128.11 2024-11-04 14:23:46.560 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.566 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d23577ca06c94022/install_package.sh && chmod +x /tmp/d23577ca06c94022/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d23577ca06c94022/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d23577ca06c94022/install_package.sh && chmod +x /tmp/d23577ca06c94022/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d23577ca06c94022/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.569 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d23577ca06c94022/install_package.sh && chmod +x /tmp/d23577ca06c94022/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d23577ca06c94022/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d23577ca06c94022/install_package.sh && chmod +x /tmp/d23577ca06c94022/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d23577ca06c94022/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.603 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/779f7704ceb74e8e/install_package.sh on 10.46.128.12 2024-11-04 14:23:46.610 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:23:46.615 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:23:46.658 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/d23577ca06c94022/install_package.sh on 10.46.128.11 2024-11-04 14:23:46.752 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:46.753 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:46.898 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,f85c53653deb5b30,7da5f7ff8eb70494] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=f85c53653deb5b30, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:46.925 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,f85c53653deb5b30,7da5f7ff8eb70494] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=f85c53653deb5b30, duration=34 ms] 2024-11-04 14:23:47.178 WARN 21062 --- [ocp-async-5,d4473bf2930c2b68,98d8b2b67e33c5fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:47.235 INFO 21062 --- [ocp-async-8,305957fc7fa8d8ae,199aaeec53b89015] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:47.983 INFO 21062 --- [prometheus-minute-schedule-1,077a294bdff714fe,fa8c2db512543413] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:23:48.592316+08:00, lastProcessTime=2024-11-04T14:23:38.590795+08:00 2024-11-04 14:23:48.659 INFO 21062 --- [ocp-async-3,d25dee79f58c6162,702f9f4b0ca2bcf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:23:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:23:48.592316+08:00 2024-11-04 14:23:49.058 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,abbdd545913c2305,5e180d52f8e21dff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=abbdd545913c2305, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:49.083 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,abbdd545913c2305,5e180d52f8e21dff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=abbdd545913c2305, duration=32 ms] 2024-11-04 14:23:49.179 WARN 21062 --- [ocp-async-6,2cd8c51b39df8363,f341661477ef6eb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:49.235 INFO 21062 --- [ocp-async-10,8f07018ed77212c1,54512de68df06ab2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:49.753 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:49.754 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:50.499 INFO 21062 --- [ocp-schedules-11,a9475e5007fb3437,2a43f8811d9495b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:23:50.617 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4399e40ced09d2f2,e235e7f37a2b24df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=4399e40ced09d2f2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:50.644 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4399e40ced09d2f2,e235e7f37a2b24df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=4399e40ced09d2f2, duration=34 ms] 2024-11-04 14:23:50.983 INFO 21062 --- [prometheus-minute-schedule-1,9128f09cb92340e4,ae2585c6badd35f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:51.179 WARN 21062 --- [ocp-async-10,acc735099a19d6cf,e3c094263269b555] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:51.235 INFO 21062 --- [ocp-async-5,3e7d997261b7905d,9b43aeb9f025757e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:51.509 INFO 21062 --- [ocp-schedules-15,7e274ab0355f6ede,eb8ca99e56549b6a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:23:51.515 INFO 21062 --- [ocp-schedules-15,7e274ab0355f6ede,eb8ca99e56549b6a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:23:51.515 INFO 21062 --- [ocp-schedules-15,7e274ab0355f6ede,eb8ca99e56549b6a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:23:52.179 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f0c772936bc25410,81aa12a9627b1f0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=f0c772936bc25410, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:52.207 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f0c772936bc25410,81aa12a9627b1f0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=f0c772936bc25410, duration=35 ms] 2024-11-04 14:23:52.754 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:52.755 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:53.165 INFO 21062 --- [ocp-schedules-6,6e0bda9ca6e93f55,5b16009f86631c4d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:23:53.179 WARN 21062 --- [ocp-async-5,25a29b5f933202cd,88ffd7d3f900b789] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:53.235 INFO 21062 --- [ocp-async-8,50c92c2dfaabedf4,1aed58605d01873c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:53.721 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,2ca8ef49793df22e,0f75565aace52f4d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=2ca8ef49793df22e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:53.748 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,2ca8ef49793df22e,0f75565aace52f4d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=2ca8ef49793df22e, duration=36 ms] 2024-11-04 14:23:53.983 INFO 21062 --- [prometheus-minute-schedule-1,eacea5f7bd60b8b4,e5980e5869466dad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:55.179 WARN 21062 --- [ocp-async-8,3696eafb5ec5b627,f14269dba35c8a1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:55.235 INFO 21062 --- [ocp-async-4,dc4388a99648e3f7,9dca23d443d4f6f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:55.341 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,730812f2ed5376ad,7d6b53e9ca65b6eb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=730812f2ed5376ad, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:55.369 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,730812f2ed5376ad,7d6b53e9ca65b6eb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=730812f2ed5376ad, duration=35 ms] 2024-11-04 14:23:55.755 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:55.756 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:56.938 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,06cef78b683759a5,e3460534bc5d8f3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=06cef78b683759a5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:56.964 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,06cef78b683759a5,e3460534bc5d8f3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=06cef78b683759a5, duration=33 ms] 2024-11-04 14:23:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c42aec5434956f1e,217e6f978f64980b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:57.179 WARN 21062 --- [ocp-async-4,e144d40a2d37c878,059cc13521f3123f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:57.236 INFO 21062 --- [ocp-async-1,9279e6b12615c922,07b97d14d77fbf98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:58.458 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,26b1ac87ed00b2d5,67e503039e935183] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=26b1ac87ed00b2d5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:58.484 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,26b1ac87ed00b2d5,67e503039e935183] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=26b1ac87ed00b2d5, duration=33 ms] 2024-11-04 14:23:58.499 INFO 21062 --- [ocp-schedules-17,831b1a03880620a0,e4652488e8f8ebb5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:23:58.500 INFO 21062 --- [ocp-schedules-11,61a1bd64df8914ae,a660f442dbdafa0d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:23:58.502 INFO 21062 --- [ocp-schedules-1,4858c9df2ffa4930,e26933cc135dae3a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:23:58.503 INFO 21062 --- [ocp-schedules-17,831b1a03880620a0,e4652488e8f8ebb5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:23:58.504 INFO 21062 --- [ocp-schedules-11,61a1bd64df8914ae,a660f442dbdafa0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:23:58.505 INFO 21062 --- [ocp-schedules-5,af937b1865b2d03b,c13bbdb4998d7824] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:23:58.510 INFO 21062 --- [ocp-schedules-22,c88fbfa6ce743001,5a9a0cccd0059917] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:23:58.510 INFO 21062 --- [ocp-schedules-22,c88fbfa6ce743001,5a9a0cccd0059917] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:23:58.510 INFO 21062 --- [ocp-schedules-9,2be041c434472c51,ed28b1430c74aa49] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:23:58.510 INFO 21062 --- [ocp-schedules-9,2be041c434472c51,ed28b1430c74aa49] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:23:58.511 INFO 21062 --- [ocp-schedules-3,7cac8deddaae391c,ac39066c765a78a0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:23:58.511 INFO 21062 --- [ocp-schedules-3,7cac8deddaae391c,ac39066c765a78a0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:23:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:23:58.590953+08:00, lastProcessTime=2024-11-04T14:23:48.592316+08:00 2024-11-04 14:23:58.659 INFO 21062 --- [ocp-async-6,b359bec6b24d11aa,ff7999def44191f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:23:58.756 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:23:58.757 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:23:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:23:58.590953+08:00 2024-11-04 14:23:59.168 INFO 21062 --- [ocp-schedules-29,7855ec4d73a75995,89e633504d0efc3d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:23:59.179 WARN 21062 --- [ocp-async-5,6bdf7e767a372759,a7541e9e7862f15c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:23:59.236 INFO 21062 --- [ocp-async-8,09ca7746272cc7a6,4a46abed7e3e6fbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:23:59.957 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,69b07816c882ae29,8a95c8a2f90bc2ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=69b07816c882ae29, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:23:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7139f96802cf1c20,7e015a1f4788bbd1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701440, interval=60 2024-11-04 14:23:59.984 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,69b07816c882ae29,8a95c8a2f90bc2ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=69b07816c882ae29, duration=34 ms] 2024-11-04 14:24:00.503 INFO 21062 --- [ocp-schedules-29,b850a75a28e6b2ab,207484d86677c44c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:24:01.074 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,93e1951de9fa0441,6c4188b474a54862] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=93e1951de9fa0441, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:24:01.083 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,93e1951de9fa0441,6c4188b474a54862] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=93e1951de9fa0441, duration=17 ms] 2024-11-04 14:24:01.180 WARN 21062 --- [ocp-async-8,8deac0ed77d5e3d2,643ffd3160c110fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:01.236 INFO 21062 --- [ocp-async-4,322de4e1cf2f22fe,e57b3085c3ee8541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:01.757 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:01.758 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:01.794 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6b0660ccbc2b582a,c530d0c781bd04bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=6b0660ccbc2b582a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:01.819 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6b0660ccbc2b582a,c530d0c781bd04bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=6b0660ccbc2b582a, duration=34 ms] 2024-11-04 14:24:02.441 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ed34d6447207f4f7,43009798726c7daa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=ed34d6447207f4f7, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:24:02.449 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ed34d6447207f4f7,43009798726c7daa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=ed34d6447207f4f7, duration=16 ms] 2024-11-04 14:24:02.983 INFO 21062 --- [prometheus-minute-schedule-1,64dbc3e7bbe57c04,cd0fd25adb88aac6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:03.180 WARN 21062 --- [ocp-async-4,8f9401989a0c46a4,9bee2516ecf21c63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:03.214 INFO 21062 --- [ocp-schedules-9,4f28e67f2128a127,c1f9d1982889f15d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:24:03.236 INFO 21062 --- [ocp-async-1,b3c7f12901267722,bdc96d80ea700e34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:04.115 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6b9b4fe2fe0690b8,604bf89bb5ed2f19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=6b9b4fe2fe0690b8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:24:04.122 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6b9b4fe2fe0690b8,604bf89bb5ed2f19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=6b9b4fe2fe0690b8, duration=16 ms] 2024-11-04 14:24:04.333 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,787ae0f73cad5cbf,8e003eed341b70c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=787ae0f73cad5cbf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:04.360 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,787ae0f73cad5cbf,8e003eed341b70c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=787ae0f73cad5cbf, duration=33 ms] 2024-11-04 14:24:04.758 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:04.759 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:05.124 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e8cb69a6cf8d873a,639e48de6ddfcfe1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=e8cb69a6cf8d873a, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:24:05.132 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,e8cb69a6cf8d873a,639e48de6ddfcfe1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=e8cb69a6cf8d873a, duration=15 ms] 2024-11-04 14:24:05.180 WARN 21062 --- [ocp-async-1,4fc4dedb157d0404,327b2f0c2b3610da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:05.236 INFO 21062 --- [ocp-async-2,1b600400b04d26a8,e96bec793214308e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f4098de082e322bf,b460b1b7624495db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:06.161 INFO 21062 --- [ocp-schedules-12,fce7735581f5164c,4b9f94d7431d0767] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:24:06.161 INFO 21062 --- [ocp-schedules-12,7a1075f08e1726c9,6b870d4a75ba3fe3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:24:06.164 INFO 21062 --- [ocp-schedules-12,7a1075f08e1726c9,6b870d4a75ba3fe3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:06.194 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:24:06.194 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:24:06.442 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,01c7533c6bdf7537,789682c3555260e7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=01c7533c6bdf7537, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:24:06.450 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,01c7533c6bdf7537,789682c3555260e7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=01c7533c6bdf7537, duration=15 ms] 2024-11-04 14:24:07.180 WARN 21062 --- [ocp-async-2,0bb8782d2b8f4c66,b7d7065c05234ed6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:07.185 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,819b0efb32fec9c2,6ac1f4a88fa18938] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=819b0efb32fec9c2, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:24:07.193 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,819b0efb32fec9c2,6ac1f4a88fa18938] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=819b0efb32fec9c2, duration=16 ms] 2024-11-04 14:24:07.236 INFO 21062 --- [ocp-async-7,2f939fa7eb583640,c26e9e15a92204a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:07.390 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,c27ea5a2263a4f93,f91acc03a5af53a7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c27ea5a2263a4f93, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:07.417 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,c27ea5a2263a4f93,f91acc03a5af53a7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c27ea5a2263a4f93, duration=33 ms] 2024-11-04 14:24:07.759 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:07.760 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:08.504 INFO 21062 --- [ocp-schedules-17,831b1a03880620a0,e4652488e8f8ebb5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:24:08.505 INFO 21062 --- [ocp-schedules-11,61a1bd64df8914ae,a660f442dbdafa0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:24:08.548 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ecf72dc8578f1e07,b994c62563e20cc3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=ecf72dc8578f1e07, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:24:08.556 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ecf72dc8578f1e07,b994c62563e20cc3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=ecf72dc8578f1e07, duration=17 ms] 2024-11-04 14:24:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:24:08.590134+08:00, lastProcessTime=2024-11-04T14:23:58.590953+08:00 2024-11-04 14:24:08.659 INFO 21062 --- [ocp-async-5,051d209428faffb5,acc86845b9a37310] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:24:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:24:08.590134+08:00 2024-11-04 14:24:08.983 INFO 21062 --- [prometheus-minute-schedule-1,5678e5d4a37b8d2c,448d91012f099073] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:09.180 WARN 21062 --- [ocp-async-4,f35cbe8ac01f6ccc,d3ac5f753139f338] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:09.237 INFO 21062 --- [ocp-async-1,1ab843b985905984,896af8acfecc3555] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:09.601 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,9db49f5c2c94545c,145298a6fc3e8e7e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=9db49f5c2c94545c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:24:09.610 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,9db49f5c2c94545c,145298a6fc3e8e7e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=9db49f5c2c94545c, duration=16 ms] 2024-11-04 14:24:09.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,01eb11972bfd3f7b,ef17ecfdb54cc9b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=01eb11972bfd3f7b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:09.837 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,01eb11972bfd3f7b,ef17ecfdb54cc9b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=01eb11972bfd3f7b, duration=33 ms] 2024-11-04 14:24:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:24:10.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:24:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:24:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:24:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:24:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:24:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:24:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:24:10.506 INFO 21062 --- [ocp-schedules-18,4fbacf51355f7f0a,af818c0eca8d1e5d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:24:10.760 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:10.761 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:11.180 WARN 21062 --- [ocp-async-1,dca9e2bc3295a25c,5327df732ceac63b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:11.237 INFO 21062 --- [ocp-async-2,91daa9e77316a192,1b7ab3d753c95e87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:11.459 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,3dde760a391725dc,a9773fa921e33d8d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=3dde760a391725dc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:11.485 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,3dde760a391725dc,a9773fa921e33d8d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=3dde760a391725dc, duration=34 ms] 2024-11-04 14:24:11.518 INFO 21062 --- [ocp-schedules-26,2b97585a9bf14fee,590fc85b323fb42c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:24:11.524 INFO 21062 --- [ocp-schedules-26,2b97585a9bf14fee,590fc85b323fb42c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:24:11.524 INFO 21062 --- [ocp-schedules-26,2b97585a9bf14fee,590fc85b323fb42c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:24:11.565 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701380, nodeCount=396, affectRows=396 2024-11-04 14:24:11.567 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701380, nodeCount=393, validCount=393 2024-11-04 14:24:11.983 INFO 21062 --- [prometheus-minute-schedule-1,125f8a0849be8bf7,46eb12970c5b39f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:12.996 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,87fd498886f2379a,f4624cf1d8737ac6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=87fd498886f2379a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:13.023 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,87fd498886f2379a,f4624cf1d8737ac6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=87fd498886f2379a, duration=34 ms] 2024-11-04 14:24:13.181 WARN 21062 --- [ocp-async-2,ef7e816a29c41bae,0fa7c5a791d25aa6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:13.237 INFO 21062 --- [ocp-async-6,9c630f124d96bf10,22fd8d543cc083ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:13.239 INFO 21062 --- [ocp-schedules-18,14d8be01b9a7220d,35d8ed7ced7929ed] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:24:13.761 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:13.762 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:13.984 INFO 21062 --- [prometheus-cache-monitor1,b6463f13f1bceef6,3b80821dc2eb42c9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:24:13.984 INFO 21062 --- [prometheus-cache-monitor1,b6463f13f1bceef6,3b80821dc2eb42c9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:24:14.542 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9f0d7fcf2e34ef2a,e57f21f532b79904] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9f0d7fcf2e34ef2a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:14.571 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9f0d7fcf2e34ef2a,e57f21f532b79904] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9f0d7fcf2e34ef2a, duration=39 ms] 2024-11-04 14:24:14.983 INFO 21062 --- [prometheus-minute-schedule-1,e81e0466a4d24282,6c027c9513c86708] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:15.181 WARN 21062 --- [ocp-async-6,038d30131b829625,753692618e4a28f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:15.237 INFO 21062 --- [ocp-async-10,9fc77ef2bd15cad1,f189f232347d47c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:16.147 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,21ff26351e5b1423,ca267f3f1ec191c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=21ff26351e5b1423, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:16.175 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,21ff26351e5b1423,ca267f3f1ec191c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=21ff26351e5b1423, duration=34 ms] 2024-11-04 14:24:16.762 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:16.763 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:17.181 WARN 21062 --- [ocp-async-10,693cec53dfb59eb5,31a96c56970fb0d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:17.237 INFO 21062 --- [ocp-async-5,ca2cf7f9b5bfcc98,e637a5883fe09874] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:17.668 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,92b6cc6feb1d4a59,1211a4a1a45ec6de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=92b6cc6feb1d4a59, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:17.693 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,92b6cc6feb1d4a59,1211a4a1a45ec6de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=92b6cc6feb1d4a59, duration=32 ms] 2024-11-04 14:24:17.983 INFO 21062 --- [prometheus-minute-schedule-1,6e3762cad2a515ad,b7ef5238ade7bb08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:24:18.591149+08:00, lastProcessTime=2024-11-04T14:24:08.590134+08:00 2024-11-04 14:24:18.659 INFO 21062 --- [ocp-async-4,c55a967c01533a56,a377bcd54d0eef4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:24:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:24:18.591149+08:00 2024-11-04 14:24:19.022 INFO 21062 --- [ocp-schedules-1,38a95ee73d3b42d0,66b53bac80996605] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:24:19.026 INFO 21062 --- [ocp-schedules-1,38a95ee73d3b42d0,66b53bac80996605] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:19.026 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:24:19.181 WARN 21062 --- [ocp-async-2,ed1f74a54815e2f7,c923c6938ff6fba3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:19.193 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,c431cba70bf1afc5,1597f3787e497fb4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c431cba70bf1afc5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:19.225 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,c431cba70bf1afc5,1597f3787e497fb4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c431cba70bf1afc5, duration=41 ms] 2024-11-04 14:24:19.237 INFO 21062 --- [ocp-async-7,45236c1d6a738550,f71d0c9301a68f6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:19.763 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:19.764 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:20.509 INFO 21062 --- [ocp-schedules-21,0bbf14f953ba6cdd,00ff8661f1eb7ca7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:24:20.755 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,00dc6ee815f831b7,15a327bdd54c0a6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=00dc6ee815f831b7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:20.781 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,00dc6ee815f831b7,15a327bdd54c0a6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=00dc6ee815f831b7, duration=33 ms] 2024-11-04 14:24:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9a10f48eac40c7d9,c5b1fab023fc8619] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:21.181 WARN 21062 --- [ocp-async-6,79109aba56867e83,7eed37c31dca6b46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:21.238 INFO 21062 --- [ocp-async-3,74b92234072a47a6,2007d6d608cb98d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:22.318 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,fc8c513c81319ff8,0f24ec998ae0004a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=fc8c513c81319ff8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:22.345 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,fc8c513c81319ff8,0f24ec998ae0004a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=fc8c513c81319ff8, duration=35 ms] 2024-11-04 14:24:22.764 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:22.765 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:23.181 WARN 21062 --- [ocp-async-10,91d5334dded3e99e,a1fccd549afecff3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:23.238 INFO 21062 --- [ocp-async-9,5e39d849c52c931c,d48e90487a6182ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:23.282 INFO 21062 --- [ocp-schedules-17,ec53aaf6a72b3c66,d6f729e0bd9deedc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:24:23.832 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,05f2de5d0e51b43e,a5b5cfa2a9f4b07c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=05f2de5d0e51b43e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:23.859 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,05f2de5d0e51b43e,a5b5cfa2a9f4b07c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=05f2de5d0e51b43e, duration=33 ms] 2024-11-04 14:24:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c1730b178a32ea6d,2ee88943fd81d1ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:25.182 WARN 21062 --- [ocp-async-5,8bd09590b50cf63f,12add79208913683] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:25.238 INFO 21062 --- [ocp-async-8,cd926b0ead72aeed,843d3d908b0ddd42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:25.416 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,fd08899d9714de1b,e1c9095f8c069ec6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=fd08899d9714de1b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:25.443 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,fd08899d9714de1b,e1c9095f8c069ec6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=fd08899d9714de1b, duration=34 ms] 2024-11-04 14:24:25.765 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:25.766 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:25.957 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/779f7704ceb74e8e/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/779f7704ceb74e8e/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:24:25.961 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/779f7704ceb74e8e/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/779f7704ceb74e8e/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:24:25.964 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/779f7704ceb74e8e/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:24:25.967 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/779f7704ceb74e8e/install_package.sh, exit code: 2, cause:null 2024-11-04 14:24:25.971 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/779f7704ceb74e8e/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/779f7704ceb74e8e/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:24:25.982 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,ab154986cf6da77d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: FAILED 2024-11-04 14:24:26.527 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/d23577ca06c94022/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/d23577ca06c94022/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:24:26.531 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/d23577ca06c94022/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/d23577ca06c94022/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:24:26.534 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/d23577ca06c94022/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:24:26.537 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/d23577ca06c94022/install_package.sh, exit code: 2, cause:null 2024-11-04 14:24:26.539 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/d23577ca06c94022/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/d23577ca06c94022/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:24:26.550 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,a5a8d0807b6f2241] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: FAILED 2024-11-04 14:24:26.983 INFO 21062 --- [prometheus-minute-schedule-1,86c7968ec2d9e30e,dd2092c30ac1271b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f6c8ba0aee6f0fa6,217c6eea4097cb7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:24:27.065 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,91bee65f2544498b,08a686a2e2fa4654] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=91bee65f2544498b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:27.093 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,91bee65f2544498b,08a686a2e2fa4654] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=91bee65f2544498b, duration=35 ms] 2024-11-04 14:24:27.182 WARN 21062 --- [ocp-async-7,155a8365ec69dede,9e0c3aa6b3994383] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:27.238 INFO 21062 --- [ocp-async-4,9c07444981286735,f005a686c2697738] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:27.333 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=2024-11-04T14:16:32.461+08:00, subtasks=[SubtaskInstanceEntity{id=68, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=33, startTime=2024-11-04T14:11:16.789+08:00, endTime=2024-11-04T14:11:16.854+08:00}, SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busin 2024-11-04 14:24:27.343 WARN 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:24:27.343 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 14:24:27.344 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:2, status:FAILED 2024-11-04 14:24:27.597 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,7094f8e5a243e296,a30b42ac3b0eae75] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=7094f8e5a243e296, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:24:27.604 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,7094f8e5a243e296,a30b42ac3b0eae75] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=7094f8e5a243e296, duration=14 ms] 2024-11-04 14:24:28.498 INFO 21062 --- [ocp-schedules-15,599bda479594e2d1,f4d295b9eb4024a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:24:28.499 INFO 21062 --- [ocp-schedules-14,94b4e3fde0f84d2c,3be55b1a5b48b1de] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:24:28.500 INFO 21062 --- [ocp-schedules-24,a514f4e6d453825a,dcba542e700374ef] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:24:28.500 INFO 21062 --- [ocp-schedules-26,6e286de60a517de8,14f5c66ae296f518] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:24:28.502 INFO 21062 --- [ocp-schedules-23,e9046a0480430feb,7f440af744bbc80c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:24:28.503 INFO 21062 --- [ocp-schedules-15,599bda479594e2d1,f4d295b9eb4024a5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:24:28.504 INFO 21062 --- [ocp-schedules-26,6e286de60a517de8,14f5c66ae296f518] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:28.504 INFO 21062 --- [ocp-schedules-24,a514f4e6d453825a,dcba542e700374ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:24:28.505 INFO 21062 --- [ocp-schedules-14,94b4e3fde0f84d2c,3be55b1a5b48b1de] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:24:28.505 INFO 21062 --- [ocp-schedules-16,4af5a4407fa5ada0,8ee5321ed11331dc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:24:28.510 INFO 21062 --- [ocp-schedules-6,b36e9d401cabdfc8,cf14c8db1e0a67c8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:24:28.510 INFO 21062 --- [ocp-schedules-27,04eb713c39ad3409,cb487d93dc8db02e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:24:28.510 INFO 21062 --- [ocp-schedules-6,b36e9d401cabdfc8,cf14c8db1e0a67c8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:24:28.510 INFO 21062 --- [ocp-schedules-27,04eb713c39ad3409,cb487d93dc8db02e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:24:28.510 INFO 21062 --- [ocp-schedules-11,d89ee90269065cc0,7b8cac0fd106cd2f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:24:28.510 INFO 21062 --- [ocp-schedules-11,d89ee90269065cc0,7b8cac0fd106cd2f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:24:28.576 INFO 21062 --- [ocp-schedules-20,ece8a9214a9dce7d,6dd913b9c1670f39] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:24:28.576 INFO 21062 --- [ocp-schedules-20,ece8a9214a9dce7d,6dd913b9c1670f39] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:24:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:24:28.588841+08:00, lastProcessTime=2024-11-04T14:24:18.591149+08:00 2024-11-04 14:24:28.589 INFO 21062 --- [ocp-schedules-22,1fe69f21f807c5ef,20a76078467f80f5] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:24:28.589+08:00, tenantSize=0 2024-11-04 14:24:28.610 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,bd20d8a9ac318a00,adfeaedaa03a804c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=bd20d8a9ac318a00, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:24:28.629 INFO 21062 --- [ocp-schedules-5,0c454b9258bf332a,f6399e94e05334b6] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:24:28.630 INFO 21062 --- [ocp-schedules-5,0c454b9258bf332a,f6399e94e05334b6] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:24:28.637 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,bd20d8a9ac318a00,adfeaedaa03a804c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=bd20d8a9ac318a00, duration=33 ms] 2024-11-04 14:24:28.659 INFO 21062 --- [ocp-async-8,378ff00bf155084f,c6b0b51e9dcd4ab5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:24:28.665 INFO 21062 --- [ocp-schedules-17,617bf0b8529331b2,1694f48de7a2faab] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 14:24:28.665 INFO 21062 --- [ocp-schedules-17,617bf0b8529331b2,1694f48de7a2faab] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 14:24:28.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:24:28.588841+08:00 2024-11-04 14:24:28.766 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:28.767 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:28.831 INFO 21062 --- [ocp-schedules-14,9479edadc3081e10,40c5ef70b355b4c8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:24:29.031 INFO 21062 --- [ocp-schedules-1,38a95ee73d3b42d0,66b53bac80996605] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:24:29.098 INFO 21062 --- [ocp-schedules-27,8aedd29afbac5507,547c8c5b940c60c1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:24:29.098 INFO 21062 --- [ocp-schedules-27,8aedd29afbac5507,547c8c5b940c60c1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:24:29.170 INFO 21062 --- [ocp-schedules-4,c7f6466aba7f07a4,418bfca3c2d0d6f1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:24:29.182 WARN 21062 --- [ocp-async-5,f36241c2b50bdb31,95c1bda904ada8b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:29.202 INFO 21062 --- [ocp-schedules-25,b11f6d58e4fcdcf2,690ba3bbcf75286f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:24:29.208 INFO 21062 --- [ocp-schedules-25,b11f6d58e4fcdcf2,690ba3bbcf75286f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:24:29.208 INFO 21062 --- [ocp-schedules-25,b11f6d58e4fcdcf2,690ba3bbcf75286f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:24:29.238 INFO 21062 --- [ocp-async-2,7ec937259da3349a,461a4b8fcf546d13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:29.534 INFO 21062 --- [ocp-schedules-21,990bf5b53900dac9,954cd8a564010cb1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:24:29.534 INFO 21062 --- [ocp-schedules-21,990bf5b53900dac9,954cd8a564010cb1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:24:29.535 INFO 21062 --- [ocp-schedules-21,990bf5b53900dac9,954cd8a564010cb1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:24:29.535 INFO 21062 --- [ocp-schedules-21,990bf5b53900dac9,954cd8a564010cb1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:24:29.536 INFO 21062 --- [ocp-schedules-21,990bf5b53900dac9,954cd8a564010cb1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:24:29.536 INFO 21062 --- [ocp-schedules-21,990bf5b53900dac9,954cd8a564010cb1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:24:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3699f5725cf6ad09,cabe7bed846c6de8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:24:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:24:30.512 INFO 21062 --- [ocp-schedules-2,d901813af571345d,8667cbc9c4557c33] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:24:31.182 WARN 21062 --- [ocp-async-7,ad1119eca516a4bb,9d0568237f62ba7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:31.239 INFO 21062 --- [ocp-async-6,4fbabdbe5fcf19d8,dbe7ccbdbd953473] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:31.527 INFO 21062 --- [ocp-schedules-4,c80fdb6c303ca512,9625fc44d122261f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:24:31.533 INFO 21062 --- [ocp-schedules-4,c80fdb6c303ca512,9625fc44d122261f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:24:31.533 INFO 21062 --- [ocp-schedules-4,c80fdb6c303ca512,9625fc44d122261f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:24:31.577 INFO 21062 --- [ocp-schedules-11,c2c29727a7f963d5,50074bfb5aa6a2f9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:24:31.577 INFO 21062 --- [ocp-schedules-11,c2c29727a7f963d5,50074bfb5aa6a2f9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:24:31.604 INFO 21062 --- [ocp-schedules-11,c2c29727a7f963d5,50074bfb5aa6a2f9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:24:31.604 INFO 21062 --- [ocp-schedules-11,c2c29727a7f963d5,50074bfb5aa6a2f9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:24:31.604 INFO 21062 --- [ocp-schedules-14,88ea47d82192bc43,6266a6f9a69446eb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:24:31.767 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:31.768 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:31.794 INFO 21062 --- [ocp-schedules-27,815f69bc0d6557f3,b1157aebc7e7e962] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:24:31.794+08:00, clusters=[1] 2024-11-04 14:24:31.795 INFO 21062 --- [ocp-schedules-27,815f69bc0d6557f3,b1157aebc7e7e962] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:24:31.945 INFO 21062 --- [ocp-schedules-5,ff9423718b3fb695,2f4845263d748063] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:24:31.948 INFO 21062 --- [ocp-schedules-5,ff9423718b3fb695,2f4845263d748063] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:31.959 INFO 21062 --- [ocp-schedules-8,ca1708d5170b27d1,2624e809afda94fc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:24:31.959+08:00, clusters=[1] 2024-11-04 14:24:31.963 INFO 21062 --- [ocp-schedules-8,ca1708d5170b27d1,2624e809afda94fc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:24:32.007 ERROR 21062 --- [ocp-schedules-5,ff9423718b3fb695,2f4845263d748063] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:24:32.008 INFO 21062 --- [ocp-schedules-5,ff9423718b3fb695,2f4845263d748063] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:24:32.935 INFO 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:24:32.935 INFO 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:24:32.935 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:24:32.935 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:24:32.943 INFO 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:24:32.944 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:24:32.944 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:24:32.945 WARN 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:24:32.945 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:24:32.945 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:24:32.946 INFO 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:24:32.946 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:24:32.946 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:24:32.947 WARN 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:24:32.947 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:24:32.947 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:24:32.954 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:24:32.954 INFO 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:24:32.954 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:24:32.969 ERROR 21062 --- [ocp-schedules-14,0b631e8dc4204bf7,038dbcb723ae9612] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:24:32.983 INFO 21062 --- [prometheus-minute-schedule-1,83e95151febcc75d,331635eda0488b59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:33.182 WARN 21062 --- [ocp-async-4,654094a5860191a6,282ecf472efee8c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:33.239 INFO 21062 --- [ocp-async-10,58f7145b6dc07e2d,c69fd4b90c54398c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:33.309 INFO 21062 --- [ocp-schedules-19,137941a8f94ebbae,5f17cde10b2b4ef4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:24:34.768 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:34.769 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:35.182 WARN 21062 --- [ocp-async-1,877ea50ab76ce140,43649659c6859f20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:35.239 INFO 21062 --- [ocp-async-9,3831f83904a7cf51,ab64221d52306123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:35.983 INFO 21062 --- [prometheus-minute-schedule-1,76a089a33b2877f2,8cd1f9deba27435c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:36.195 INFO 21062 --- [ocp-schedules-12,7a1075f08e1726c9,6b870d4a75ba3fe3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:24:36.195 INFO 21062 --- [ocp-schedules-12,32a001080abd0676,0efe415669aee586] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:24:36.198 INFO 21062 --- [ocp-schedules-12,32a001080abd0676,0efe415669aee586] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:36.228 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:24:36.228 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:24:37.183 WARN 21062 --- [ocp-async-2,5580d9bd99d17a86,641aa5ad3a02ad6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:37.239 INFO 21062 --- [ocp-async-8,973e370ee6da4bca,48b106347f6687a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:37.769 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:37.771 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:38.504 INFO 21062 --- [ocp-schedules-26,6e286de60a517de8,14f5c66ae296f518] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:24:38.505 INFO 21062 --- [ocp-schedules-24,a514f4e6d453825a,dcba542e700374ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:24:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:24:38.590787+08:00, lastProcessTime=2024-11-04T14:24:28.588841+08:00 2024-11-04 14:24:38.660 INFO 21062 --- [ocp-async-5,6577c8599ac07266,32380f12a9a57aab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:24:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:24:38.590787+08:00 2024-11-04 14:24:38.866 INFO 21062 --- [ocp-schedules-11,6595fa7da092ec43,9ddc68242bebc5db] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:24:38.869 INFO 21062 --- [ocp-schedules-11,6595fa7da092ec43,9ddc68242bebc5db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:38.954 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:24:38.956 ERROR 21062 --- [ocp-schedules-11,6595fa7da092ec43,9ddc68242bebc5db] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:24:38.956 INFO 21062 --- [ocp-schedules-11,6595fa7da092ec43,9ddc68242bebc5db] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:24:38.983 INFO 21062 --- [prometheus-minute-schedule-1,861f13981d7ca0df,9cfd96a7ba694b5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:39.183 WARN 21062 --- [ocp-async-4,f2c34958dfe22de3,6994c29f1637c240] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:39.239 INFO 21062 --- [ocp-async-1,1018fe560cebcb85,b3df29ff279bd437] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:24:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:24:40.515 INFO 21062 --- [ocp-schedules-3,bb0f9f08b69f0893,362e220b3d665b5e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:24:40.771 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:40.771 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:41.183 WARN 21062 --- [ocp-async-1,791666382b026831,969d97d5921a1eb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:41.240 INFO 21062 --- [ocp-async-2,0c430a94b7d3c321,71c29904df4d4a3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:24:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:24:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:24:41.983 INFO 21062 --- [prometheus-minute-schedule-1,dfdbb16e8c791154,28f4d0669a5866cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:42.511 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:24:42.512 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:24:42.513 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.514 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:24:42.515 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.516 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.516 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.517 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:24:42.535 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.536 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.536 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.537 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:24:42.550 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.551 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.551 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.552 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:24:42.566 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.566 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.567 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.567 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:24:42.580 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.581 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.581 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.582 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:24:42.587 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.588 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.588 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.588 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:24:42.593 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.593 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.593 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.594 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:24:42.598 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.598 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.599 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.599 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:24:42.605 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.605 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.606 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.606 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:24:42.611 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.612 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.612 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.613 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:24:42.618 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.619 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.619 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.620 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:24:42.627 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.628 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.628 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.628 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:24:42.642 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.642 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.643 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.643 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:24:42.648 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.648 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.649 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.649 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:24:42.655 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.655 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.656 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.656 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:24:42.662 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.662 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.663 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.663 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:24:42.667 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.668 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.668 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.669 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:24:42.673 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.673 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.673 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.674 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:24:42.678 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.678 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.679 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.679 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:24:42.684 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.684 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.685 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.685 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:24:42.689 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.689 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.690 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.690 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:24:42.694 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.695 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.695 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.696 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:24:42.699 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.700 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.701 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.701 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:24:42.705 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.706 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.706 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.707 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:24:42.711 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.712 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.712 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.713 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:24:42.717 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.717 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.718 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.718 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:24:42.722 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.723 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.723 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.724 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:24:42.728 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.728 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.729 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.729 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:24:42.734 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.734 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.735 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.735 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:24:42.739 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.740 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.740 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.740 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:24:42.745 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.745 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.746 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.746 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:24:42.750 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.751 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.751 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.752 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:24:42.756 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.757 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.757 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.758 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:24:42.762 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.762 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.763 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.763 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:24:42.770 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.770 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.771 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.771 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:24:42.776 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.777 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.777 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.777 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:24:42.783 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.783 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.784 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.784 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:24:42.789 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.790 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.790 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.790 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:24:42.794 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.795 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.795 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.795 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:24:42.799 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.799 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.800 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.800 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:24:42.804 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.804 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.805 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.805 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:24:42.809 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.810 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.810 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.811 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:24:42.820 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.821 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.821 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.822 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:24:42.841 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.841 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.842 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.842 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:24:42.848 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.848 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.849 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.849 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:24:42.855 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.856 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.856 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.857 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:24:42.862 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.862 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.863 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.863 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:24:42.867 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.867 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.868 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.868 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:24:42.872 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.872 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.873 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.873 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:24:42.877 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:24:42.877 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:24:42.878 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.879 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:24:42.880 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.880 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.881 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.881 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:24:42.887 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.888 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:24:42.888 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:24:42.889 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:24:42.896 INFO 21062 --- [ocp-schedules-26,f190f47e6786adb0,deaaa85704ff647e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:24:43.086 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:24:43.086 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:24:43.093 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:24:43.093 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:24:43.093 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:24:43.093 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:24:43.100 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:24:43.100 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:24:43.100 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:24:43.100 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:24:43.135 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:24:43.143 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:24:43.143 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:24:43.144 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:24:43.144 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:24:43.144 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:24:43.151 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:24:43.183 WARN 21062 --- [ocp-async-2,635f64cdcf5d08f9,1e056477726c78c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:43.240 INFO 21062 --- [ocp-async-6,11fd80551908ce91,23faec2fa1d8f263] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:43.350 INFO 21062 --- [ocp-schedules-29,da29a7a3f996173f,c7e861e7f3361345] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:24:43.701 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:24:43.772 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:43.772 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:44.263 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.163664646S 2024-11-04 14:24:44.264 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:24:44.361 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:24:44.362 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:24:44.389 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:24:44.450 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:24:44.450 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:24:44.450 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:24:44.450 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:24:44.450 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:24:44.451 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:24:44.462 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:24:44.668 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] o.s.boot.SpringApplication : Started application in 1.575 seconds (JVM running for 6898.348) 2024-11-04 14:24:44.668 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.575794649S 2024-11-04 14:24:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e0c0dcc25f94b509,78a1a0d7eb789b0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:45.112 INFO 21062 --- [ocp-schedules-20,beec8c4ef5c0de7e,4496f05e2ab13156] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:24:45.183 WARN 21062 --- [ocp-async-6,b94422e6987cbaf4,21924839a9fdb8a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:45.240 INFO 21062 --- [ocp-async-10,eeae56767d8a4a9f,8ea5dfa98c4fbfab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:46.773 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:46.774 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:47.184 WARN 21062 --- [ocp-async-10,0c06c8928d116c66,6f2d5c45e33b6d6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:47.240 INFO 21062 --- [ocp-async-9,23bbd71e075d0a93,49962a4dfbed16cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:47.983 INFO 21062 --- [prometheus-minute-schedule-1,7def75f6102bda60,0560d06d4abead70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:24:48.590997+08:00, lastProcessTime=2024-11-04T14:24:38.590787+08:00 2024-11-04 14:24:48.660 INFO 21062 --- [ocp-async-3,13ec706161b85e3b,d1baba6b6595e761] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:24:48.744 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:24:48.590997+08:00 2024-11-04 14:24:49.184 WARN 21062 --- [ocp-async-2,9ae7205ebae753a1,e9aaadc1d350aaa7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:49.240 INFO 21062 --- [ocp-async-6,c244407464ea9276,f7f0a42194af1c4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:49.774 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:49.775 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:50.518 INFO 21062 --- [ocp-schedules-2,bafb127868f86e99,e3e65a6b313464ff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:24:50.983 INFO 21062 --- [prometheus-minute-schedule-1,981438de23c7f6aa,c1cbddb982f192bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:51.185 WARN 21062 --- [ocp-async-6,1fc91394b4fe1ce5,5e137255fa092abe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:51.241 INFO 21062 --- [ocp-async-10,d0d964494b702683,2af98468be8fb3b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:51.535 INFO 21062 --- [ocp-schedules-8,ab8a0a464e335145,c4e687a58644223c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:24:51.542 INFO 21062 --- [ocp-schedules-8,ab8a0a464e335145,c4e687a58644223c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:24:51.542 INFO 21062 --- [ocp-schedules-8,ab8a0a464e335145,c4e687a58644223c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:24:52.775 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:52.777 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:53.186 WARN 21062 --- [ocp-async-10,972abdc097e2f406,722de3b0a0e5e1e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:53.241 INFO 21062 --- [ocp-async-9,d5aeec26a666e8da,589e77feda736aca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:53.378 INFO 21062 --- [ocp-schedules-22,e357aa9a5152d043,27a2fb771ba7bc4d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:24:53.983 INFO 21062 --- [prometheus-minute-schedule-1,774ae6d816484d84,f08f7591d9cc925a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:55.186 WARN 21062 --- [ocp-async-9,3395a6c459b3f038,5603de804f30804d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:55.241 INFO 21062 --- [ocp-async-7,b07fb0a2b738dcdb,792bbaf12de91dd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:55.777 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:55.778 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f9600051fec4fa98,bf2d32a957f3ed66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:24:57.186 WARN 21062 --- [ocp-async-7,8fdb6afcb7a69cdf,a45c498bac2bca3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:57.241 INFO 21062 --- [ocp-async-4,fdfecaf7f9facf63,7b9cca949e3ae015] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:58.500 INFO 21062 --- [ocp-schedules-29,2738e74d19b80b81,4890ea93319ca8d1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:24:58.500 INFO 21062 --- [ocp-schedules-10,3763af58e59800a4,1590eb21d492d1f8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:24:58.503 INFO 21062 --- [ocp-schedules-21,383876a80419c9d3,e0aad5b05455179f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:24:58.503 INFO 21062 --- [ocp-schedules-10,3763af58e59800a4,1590eb21d492d1f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:58.503 INFO 21062 --- [ocp-schedules-29,2738e74d19b80b81,4890ea93319ca8d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:24:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:24:58.505 INFO 21062 --- [ocp-schedules-14,a4c01023c0a6a46f,8c44812aa35da378] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:24:58.511 INFO 21062 --- [ocp-schedules-26,cca943df26558d78,0506db8a38176864] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:24:58.511 INFO 21062 --- [ocp-schedules-26,cca943df26558d78,0506db8a38176864] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:24:58.511 INFO 21062 --- [ocp-schedules-25,86ee3daa5c9774cc,d873b2a5587df6c5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:24:58.512 INFO 21062 --- [ocp-schedules-25,86ee3daa5c9774cc,d873b2a5587df6c5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:24:58.512 INFO 21062 --- [ocp-schedules-3,20baacc0e4fecc7c,3629dc511ea912a8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:24:58.512 INFO 21062 --- [ocp-schedules-3,20baacc0e4fecc7c,3629dc511ea912a8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:24:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:24:58.590402+08:00, lastProcessTime=2024-11-04T14:24:48.590997+08:00 2024-11-04 14:24:58.660 INFO 21062 --- [ocp-async-2,36dead9aa5f51ca9,a7d9596d39f97ef4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:24:58.743 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:24:58.590402+08:00 2024-11-04 14:24:58.778 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:24:58.778 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:24:59.174 INFO 21062 --- [ocp-schedules-27,6f9afafd98d17e87,eeb1f259595c087e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:24:59.186 WARN 21062 --- [ocp-async-10,d60d2678d23b42b6,8581ecbb10f57e9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:24:59.241 INFO 21062 --- [ocp-async-9,4260c82f4ae58f2f,a4a346f92c857831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:24:59.983 INFO 21062 --- [prometheus-minute-schedule-1,8b0b1c75711025aa,b6bc208637827919] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701500, interval=60 2024-11-04 14:25:00.468 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,f9a5c3bcda832a4f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:20:01.0, nextRun=2024-11-04 14:25:00.0, currentDate=Mon Nov 04 14:25:00 CST 2024 2024-11-04 14:25:00.469 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,f9a5c3bcda832a4f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:20:01.0, nextRunTime=2024-11-04 14:25:00.0} 2024-11-04 14:25:00.496 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,f9a5c3bcda832a4f] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:25:00.521 INFO 21062 --- [ocp-schedules-23,f4bbcfa7e7bce73e,c504ae028c444baf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:25:00.524 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,f9a5c3bcda832a4f] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:25:00.524 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,f9a5c3bcda832a4f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:25:01.186 WARN 21062 --- [ocp-async-9,5b5f59f4e65ddcd2,44633490f02ce637] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:01.241 INFO 21062 --- [ocp-async-7,d25233bd46470e3b,1beac608bfb7ae73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:01.779 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:01.779 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:02.983 INFO 21062 --- [prometheus-minute-schedule-1,977e9e2acee072b9,eac68b1ef78e0943] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:03.186 WARN 21062 --- [ocp-async-7,f1e93bd640d56fd8,8178457c16fdb0b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:03.241 INFO 21062 --- [ocp-async-4,8b529dbed2b0b91f,c81a21574a8b80be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:03.406 INFO 21062 --- [ocp-schedules-4,5ef6950b4abca710,7b4b3e1118f61eec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:25:04.780 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:04.780 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:05.187 WARN 21062 --- [ocp-async-4,c738c3dd9cef42eb,1e5a08b201c7fc88] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:05.242 INFO 21062 --- [ocp-async-5,69c95068cad97ecf,e3de989f41541a70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:05.983 INFO 21062 --- [prometheus-minute-schedule-1,94d172aca1500430,b3fce6d711328f09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:06.229 INFO 21062 --- [ocp-schedules-12,32a001080abd0676,0efe415669aee586] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:25:06.229 INFO 21062 --- [ocp-schedules-12,3aa448a581f56cf8,27a7be45d2a2d9cf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:25:06.233 INFO 21062 --- [ocp-schedules-12,3aa448a581f56cf8,27a7be45d2a2d9cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:06.261 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:25:06.261 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:25:07.187 WARN 21062 --- [ocp-async-5,3f905c63554b68d9,89aaa2f83a0c8357] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:07.242 INFO 21062 --- [ocp-async-8,a6890dbd92580502,6074d7427b06851f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:07.781 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:07.781 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:08.504 INFO 21062 --- [ocp-schedules-10,3763af58e59800a4,1590eb21d492d1f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:25:08.504 INFO 21062 --- [ocp-schedules-29,2738e74d19b80b81,4890ea93319ca8d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:25:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:25:08.589669+08:00, lastProcessTime=2024-11-04T14:24:58.590402+08:00 2024-11-04 14:25:08.660 INFO 21062 --- [ocp-async-10,4d06348e36b01360,17162bdce5641a24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:25:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:25:08.589669+08:00 2024-11-04 14:25:08.983 INFO 21062 --- [prometheus-minute-schedule-1,6ba0c13ea114b07d,f8dc3baf5032f760] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:09.187 WARN 21062 --- [ocp-async-7,6a92b90f9dfb901c,4769cb09f5baa43a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:09.242 INFO 21062 --- [ocp-async-4,708d1687139b945b,9695c94729bd7a09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:25:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:25:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:25:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:25:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:25:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:25:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:25:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:25:10.524 INFO 21062 --- [ocp-schedules-15,6b6d3d438734a203,c236570426ee1862] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:25:10.781 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:10.782 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:11.187 WARN 21062 --- [ocp-async-4,6c299f4aefb64678,d831d3363986f4ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:11.242 INFO 21062 --- [ocp-async-5,cb1c309a5893a454,f8956a5c50fcd543] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:11.545 INFO 21062 --- [ocp-schedules-5,10ff01dec3cf0aff,23cdbb56120ff0fe] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:25:11.550 INFO 21062 --- [ocp-schedules-5,10ff01dec3cf0aff,23cdbb56120ff0fe] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:25:11.550 INFO 21062 --- [ocp-schedules-5,10ff01dec3cf0aff,23cdbb56120ff0fe] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:25:11.571 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=6, affectRows=6 2024-11-04 14:25:11.571 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=6, validCount=6 2024-11-04 14:25:11.571 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=1, affectRows=1 2024-11-04 14:25:11.572 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=0, validCount=0 2024-11-04 14:25:11.572 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=2, affectRows=2 2024-11-04 14:25:11.572 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=2, validCount=2 2024-11-04 14:25:11.573 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=4, affectRows=4 2024-11-04 14:25:11.573 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=4, validCount=4 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=10, affectRows=10 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=17, affectRows=17 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=10, validCount=10 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=26, affectRows=26 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=31, affectRows=31 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=23, affectRows=23 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=26, affectRows=26 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=19, affectRows=19 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=9, affectRows=9 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=8, affectRows=8 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=17, validCount=17 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=9, validCount=9 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=40, affectRows=40 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=8, validCount=8 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=19, validCount=19 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=26, validCount=26 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=34, affectRows=34 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=23, validCount=23 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=26, validCount=26 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=50, affectRows=50 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=31, validCount=31 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=40, validCount=40 2024-11-04 14:25:11.574 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=34, validCount=34 2024-11-04 14:25:11.575 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=49, validCount=49 2024-11-04 14:25:11.575 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=39, affectRows=39 2024-11-04 14:25:11.576 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=38, validCount=38 2024-11-04 14:25:11.576 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701440, nodeCount=51, affectRows=51 2024-11-04 14:25:11.577 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701440, nodeCount=51, validCount=51 2024-11-04 14:25:11.983 INFO 21062 --- [prometheus-minute-schedule-1,5f57dac4fb301087,4496d9fd9a068063] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:13.187 WARN 21062 --- [ocp-async-5,a8ef7088e789e97a,18d4cf45b9b19f1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:13.242 INFO 21062 --- [ocp-async-2,e5e92f2a974c5cbd,de5b7bb837c47dd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:13.431 INFO 21062 --- [ocp-schedules-1,d60e726f276caca0,d554e710f7169b43] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:25:13.782 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:13.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:13.984 INFO 21062 --- [prometheus-cache-monitor1,040cf28e9dacbd6f,375676310e786728] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:25:13.984 INFO 21062 --- [prometheus-cache-monitor1,040cf28e9dacbd6f,375676310e786728] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:25:14.983 INFO 21062 --- [prometheus-minute-schedule-1,59f3d4c5929382c4,7e1c7fcf40d0f650] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:15.062 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:21:15.043, end=2024-11-04T14:23:15.059} 2024-11-04 14:25:15.071 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:25:15.071 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:25:15.187 WARN 21062 --- [ocp-async-2,b467ddb75b57109a,064f3088c8372592] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:15.242 INFO 21062 --- [ocp-async-6,0faac9ca9e07dbdc,45bdedc41bccd13d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:16.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:16.784 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:17.188 WARN 21062 --- [ocp-async-6,f3f7231ecaac34b7,1ddced158f11fdbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:17.243 INFO 21062 --- [ocp-async-10,a834e440dfe46302,3c6f6f3cadcbf556] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:17.983 INFO 21062 --- [prometheus-minute-schedule-1,57b795c3e3b3fbdc,38ddc4185be22d0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:25:18.591409+08:00, lastProcessTime=2024-11-04T14:25:08.589669+08:00 2024-11-04 14:25:18.660 INFO 21062 --- [ocp-async-7,7b68e505c9006a6a,5db2cb97e3134325] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:25:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:25:18.591409+08:00 2024-11-04 14:25:19.188 WARN 21062 --- [ocp-async-5,228602a817724bdd,6b2cdb6a352451ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:19.243 INFO 21062 --- [ocp-async-8,e588388149b82c54,e7b0fd097f944b49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:19.785 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:19.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:20.528 INFO 21062 --- [ocp-schedules-20,c85ea72168830765,ea980ba568933274] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:25:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f55f221e41ecca3a,ac1bb28504e39211] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:21.188 WARN 21062 --- [ocp-async-8,06a2431bdf8e6499,7b36c1a0c125548e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:21.243 INFO 21062 --- [ocp-async-3,310e1423bd81a2b2,0efc8e3f3aa4f9bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:22.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:22.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:23.188 WARN 21062 --- [ocp-async-6,cec25e4c62aa6d26,e7e411dadf090759] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:23.243 INFO 21062 --- [ocp-async-1,72dbaf5ebbd34592,56a1640d306969bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:23.475 INFO 21062 --- [ocp-schedules-14,ea40dfb5dfe60f3a,8ced62fcc973a04a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:25:23.983 INFO 21062 --- [prometheus-minute-schedule-1,3b7900532b56f941,61d38e8a91227ed1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:25.188 WARN 21062 --- [ocp-async-10,15b2e496f8953cbd,a9689ddb8a73f912] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:25.243 INFO 21062 --- [ocp-async-9,9af09bf76fd1e008,71fe9fbaffe163e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:25.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:25.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3b65a33df4955715,b2ffe9179b850aa3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:26.984 INFO 21062 --- [prometheus-hour-schedule-1,7c5c0617c4fdb2b6,8fd7bebdf83ca645] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:25:27.189 WARN 21062 --- [ocp-async-2,00f5227050f0b01a,cc2dc631cc396a03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:27.243 INFO 21062 --- [ocp-async-7,9044051559c52a43,1d46bd4eadfb3c96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:28.498 INFO 21062 --- [ocp-schedules-26,8b2cb83234c5fa36,4c43d1ec80465a1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:25:28.500 INFO 21062 --- [ocp-schedules-28,82ce43fe81e2389e,632d336a04176bf8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:25:28.500 INFO 21062 --- [ocp-schedules-13,b3af986aa25c5b48,8e991e2bab4da292] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:25:28.500 INFO 21062 --- [ocp-schedules-8,c2737d4a1021f86a,8e0facc86627657c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:25:28.502 INFO 21062 --- [ocp-schedules-10,16f10ef643cf335f,c93b23291b130e35] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:25:28.503 INFO 21062 --- [ocp-schedules-26,8b2cb83234c5fa36,4c43d1ec80465a1e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:25:28.504 INFO 21062 --- [ocp-schedules-28,82ce43fe81e2389e,632d336a04176bf8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:28.504 INFO 21062 --- [ocp-schedules-8,c2737d4a1021f86a,8e0facc86627657c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:25:28.506 INFO 21062 --- [ocp-schedules-13,b3af986aa25c5b48,8e991e2bab4da292] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:25:28.506 INFO 21062 --- [ocp-schedules-15,c7e1295881d06940,63d1e347816906f3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:25:28.511 INFO 21062 --- [ocp-schedules-18,29d8595365c92e8c,9e92b6771cabef8d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:25:28.511 INFO 21062 --- [ocp-schedules-14,5fa0cb9b3a446ae5,f42c3715f40ba63c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:25:28.511 INFO 21062 --- [ocp-schedules-30,609aad80508ab074,39f6c840eba1061d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:25:28.511 INFO 21062 --- [ocp-schedules-30,609aad80508ab074,39f6c840eba1061d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:25:28.511 INFO 21062 --- [ocp-schedules-18,29d8595365c92e8c,9e92b6771cabef8d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:25:28.511 INFO 21062 --- [ocp-schedules-14,5fa0cb9b3a446ae5,f42c3715f40ba63c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:25:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:25:28.590095+08:00, lastProcessTime=2024-11-04T14:25:18.591409+08:00 2024-11-04 14:25:28.660 INFO 21062 --- [ocp-async-9,f0746592adf00884,a19fb692e0fedca1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:25:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:25:28.590095+08:00 2024-11-04 14:25:28.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:28.790 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:28.834 INFO 21062 --- [ocp-schedules-22,51d91e63c04506fb,26bb740f51103e25] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:25:29.031 INFO 21062 --- [ocp-schedules-1,b72b4f04c6c51434,5186b1f222f68f5e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:25:29.035 INFO 21062 --- [ocp-schedules-1,b72b4f04c6c51434,5186b1f222f68f5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:29.035 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:25:29.102 INFO 21062 --- [ocp-schedules-13,2fb6f8c92def1f57,a57491c36ea3999f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:25:29.102 INFO 21062 --- [ocp-schedules-13,2fb6f8c92def1f57,a57491c36ea3999f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:25:29.177 INFO 21062 --- [ocp-schedules-15,57378af25821c468,9be80b07933d77db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:25:29.189 WARN 21062 --- [ocp-async-10,6b7df38906718e9b,a0d6368cbe7a123a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:29.208 INFO 21062 --- [ocp-schedules-14,1725d2037f3db25c,91bec52d98aa4ad9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:25:29.214 INFO 21062 --- [ocp-schedules-14,1725d2037f3db25c,91bec52d98aa4ad9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:25:29.214 INFO 21062 --- [ocp-schedules-14,1725d2037f3db25c,91bec52d98aa4ad9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:25:29.244 INFO 21062 --- [ocp-async-5,f0860c27f23f089d,fa36843e74d3dca9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:29.361 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:25:29.361 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:25:29.369 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:25:29.369 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:25:29.369 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:25:29.380 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:25:29.380 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:25:29.380 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:25:29.384 INFO 21062 --- [ocp-schedules-19,78862bb1b84c7740,6650708008693542] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:25:29.537 INFO 21062 --- [ocp-schedules-2,3b788300967f8b97,420fa3b6a12c01b2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:25:29.537 INFO 21062 --- [ocp-schedules-2,3b788300967f8b97,420fa3b6a12c01b2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:25:29.544 INFO 21062 --- [ocp-schedules-2,3b788300967f8b97,420fa3b6a12c01b2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:25:29.544 INFO 21062 --- [ocp-schedules-2,3b788300967f8b97,420fa3b6a12c01b2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:25:29.548 INFO 21062 --- [ocp-schedules-2,3b788300967f8b97,420fa3b6a12c01b2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:25:29.548 INFO 21062 --- [ocp-schedules-2,3b788300967f8b97,420fa3b6a12c01b2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:25:29.983 INFO 21062 --- [prometheus-minute-schedule-1,cf067672d7c3912c,725fa73ff4701f65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:25:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:25:30.532 INFO 21062 --- [ocp-schedules-17,ddd97d38ab8f4626,0fa74caf48667d63] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:25:31.189 WARN 21062 --- [ocp-async-2,2883789513ce7c6f,707a8ae0207035c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:31.244 INFO 21062 --- [ocp-async-8,1932ee73c14b1e6c,f9785507a8c855ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:31.554 INFO 21062 --- [ocp-schedules-24,75078cb63e843007,6558a754a37226ae] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:25:31.559 INFO 21062 --- [ocp-schedules-24,75078cb63e843007,6558a754a37226ae] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:25:31.559 INFO 21062 --- [ocp-schedules-24,75078cb63e843007,6558a754a37226ae] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:25:31.604 INFO 21062 --- [ocp-schedules-3,853c0d9f833fd019,a722b6b235053a21] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:25:31.604 INFO 21062 --- [ocp-schedules-3,853c0d9f833fd019,a722b6b235053a21] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:25:31.633 INFO 21062 --- [ocp-schedules-3,853c0d9f833fd019,a722b6b235053a21] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:25:31.633 INFO 21062 --- [ocp-schedules-3,853c0d9f833fd019,a722b6b235053a21] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:25:31.635 INFO 21062 --- [ocp-schedules-4,2f45bc498ea17cc1,bb334812f5913be5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:25:31.790 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:31.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:31.822 INFO 21062 --- [ocp-schedules-21,74c2ca9537ea8551,92f2bef3be413fa5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:25:31.822+08:00, clusters=[1] 2024-11-04 14:25:31.824 INFO 21062 --- [ocp-schedules-21,74c2ca9537ea8551,92f2bef3be413fa5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:25:31.988 INFO 21062 --- [ocp-schedules-27,d72901216ed86bbf,7527e42135a06930] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:25:31.988+08:00, clusters=[1] 2024-11-04 14:25:31.992 INFO 21062 --- [ocp-schedules-27,d72901216ed86bbf,7527e42135a06930] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:25:32.979 INFO 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:25:32.980 INFO 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:25:32.980 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:25:32.980 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:25:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3b204d560a23ac5a,ae506b2b995f9e3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:32.990 INFO 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:25:32.990 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:25:32.991 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:25:32.992 WARN 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:25:32.992 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:25:32.992 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:25:32.992 INFO 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:25:32.992 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:25:32.992 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:25:32.993 WARN 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:25:32.993 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:25:32.993 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:25:33.001 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:25:33.001 INFO 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:25:33.001 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:25:33.024 ERROR 21062 --- [ocp-schedules-6,7f00c6827f051bd6,78dfd8d5776348d1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:25:33.189 WARN 21062 --- [ocp-async-7,5ee5272487e2c24e,4ba1487e071b130d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:33.244 INFO 21062 --- [ocp-async-6,b81024109be40c2d,adeecba745e795c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:33.501 INFO 21062 --- [ocp-schedules-2,cde5e4339bddc80b,610ea3100e0e09e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:25:34.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:34.794 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:35.189 WARN 21062 --- [ocp-async-4,5cf47a68162f8bdb,ef61d4dc58bab911] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:35.244 INFO 21062 --- [ocp-async-1,bde25074f00accbe,9459d0d121327ad7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:35.983 INFO 21062 --- [prometheus-minute-schedule-1,701a878a41b169a9,3af165f438d0273e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:36.262 INFO 21062 --- [ocp-schedules-12,3aa448a581f56cf8,27a7be45d2a2d9cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:25:36.262 INFO 21062 --- [ocp-schedules-12,0e5aa3bb9cf8b1b6,d24c73c46b0f51b6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:25:36.268 INFO 21062 --- [ocp-schedules-12,0e5aa3bb9cf8b1b6,d24c73c46b0f51b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:36.297 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:25:36.297 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:25:37.190 WARN 21062 --- [ocp-async-5,d9e252ded82404d2,bf16e18210e5f816] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:37.245 INFO 21062 --- [ocp-async-9,b80e5e7076e35d58,b4f38d774d2e3ed5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:37.794 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:37.795 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:38.505 INFO 21062 --- [ocp-schedules-8,c2737d4a1021f86a,8e0facc86627657c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:25:38.505 INFO 21062 --- [ocp-schedules-28,82ce43fe81e2389e,632d336a04176bf8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:25:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:25:38.589729+08:00, lastProcessTime=2024-11-04T14:25:28.590095+08:00 2024-11-04 14:25:38.661 INFO 21062 --- [ocp-async-10,6bec87a3dcf9b912,dc687de437228863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:25:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:25:38.589729+08:00 2024-11-04 14:25:38.957 INFO 21062 --- [ocp-schedules-4,90a30b0bd4732814,82e915bf09cec557] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:25:38.960 INFO 21062 --- [ocp-schedules-4,90a30b0bd4732814,82e915bf09cec557] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a15019b95cf13b77,328905c83d1372c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:39.036 INFO 21062 --- [ocp-schedules-1,b72b4f04c6c51434,5186b1f222f68f5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:25:39.046 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:25:39.050 ERROR 21062 --- [ocp-schedules-4,90a30b0bd4732814,82e915bf09cec557] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:25:39.050 INFO 21062 --- [ocp-schedules-4,90a30b0bd4732814,82e915bf09cec557] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:25:39.190 WARN 21062 --- [ocp-async-3,6f02d99e30c107dc,b113887159c1fa57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:39.245 INFO 21062 --- [ocp-async-4,1bd05ba2a321e964,275b0faaaed43587] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:40.004 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:25:40.004 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:25:40.535 INFO 21062 --- [ocp-schedules-1,091256138db72207,aec08d021aefb10d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:25:40.795 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:40.797 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:41.190 WARN 21062 --- [ocp-async-4,e47f3a64555a57f4,252b623e2cbd5014] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:41.245 INFO 21062 --- [ocp-async-5,d5b5c346f9c4dbf7,41c80e9fd6fab605] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:41.983 INFO 21062 --- [prometheus-minute-schedule-1,7adf1a6ca4cf7247,dea288a369073726] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:43.190 WARN 21062 --- [ocp-async-5,e11b8c2b9669f449,90e3191d4ae39225] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:43.245 INFO 21062 --- [ocp-async-8,2266a2ea6ea8495b,0dc76c314fb23a65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:43.544 INFO 21062 --- [ocp-schedules-5,7259531d0d4fa154,e1aeb134d669547c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:25:43.797 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:43.797 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:44.983 INFO 21062 --- [prometheus-minute-schedule-1,43ca32b0d55fc9a5,726e8ee3a88ae459] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:45.190 WARN 21062 --- [ocp-async-8,f500457248523d1d,7b8e4ef7c6b1e7c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:45.245 INFO 21062 --- [ocp-async-6,ad83995041f60139,e738bd96a6599dde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:46.798 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:46.799 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:47.190 WARN 21062 --- [ocp-async-6,383b1bf13e89c2f8,e29b5096a9075167] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:47.246 INFO 21062 --- [ocp-async-1,825c83ca2fecc8f8,271ea173f60a36ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:47.983 INFO 21062 --- [prometheus-minute-schedule-1,a16c594dfdd0fd8f,693f0dadc7c6f720] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:25:48.591144+08:00, lastProcessTime=2024-11-04T14:25:38.589729+08:00 2024-11-04 14:25:48.661 INFO 21062 --- [ocp-async-7,a3530e3eaef22028,8d1b5b0b78b8d68b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:25:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:25:48.591144+08:00 2024-11-04 14:25:49.190 WARN 21062 --- [ocp-async-5,2e0af94cba0ec096,8bbd186c220c11e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:49.246 INFO 21062 --- [ocp-async-8,60241b3edf7cc3e1,e7aecd2228eab667] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:49.799 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:49.801 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:50.538 INFO 21062 --- [ocp-schedules-25,15178add98926476,bf55d8bf1f85de3b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:25:50.983 INFO 21062 --- [prometheus-minute-schedule-1,e4f865ac4d8d22b6,aa393fccb5248440] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:51.191 WARN 21062 --- [ocp-async-8,ee627088c48c3d98,a37c019a1a4c3683] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:51.246 INFO 21062 --- [ocp-async-6,f40d652f14169923,360ceebf67f8a701] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:51.561 INFO 21062 --- [ocp-schedules-23,74c03d0997ee3b90,3bde0b27e5abf3fb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:25:51.566 INFO 21062 --- [ocp-schedules-23,74c03d0997ee3b90,3bde0b27e5abf3fb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:25:51.566 INFO 21062 --- [ocp-schedules-23,74c03d0997ee3b90,3bde0b27e5abf3fb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:25:52.801 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:52.802 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:53.191 WARN 21062 --- [ocp-async-6,aaf71f7240352830,d35dbe52923a7642] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:53.246 INFO 21062 --- [ocp-async-1,ea23ac47f94d3505,c3452e55106ce441] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:53.571 INFO 21062 --- [ocp-schedules-2,522dc9c903a79d5c,d5bf1fc34bc6b75b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:25:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c489b1bf098e80c7,6096b719a3764ca1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:55.191 WARN 21062 --- [ocp-async-1,ac52f271e8dffed5,ccbdcd7bfcbe15f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:55.246 INFO 21062 --- [ocp-async-9,94c01ab3d1d1aec4,20f3a22badf52ae1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:55.803 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:55.804 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:56.983 INFO 21062 --- [prometheus-minute-schedule-1,85f718161caacf9c,4718ed4545cfc368] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:25:57.191 WARN 21062 --- [ocp-async-9,e66b0d6a8f6dfbc8,71b4779b26dc63e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:57.246 INFO 21062 --- [ocp-async-3,01eaa1852199745c,4fbc0a5af296a3dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:58.499 INFO 21062 --- [ocp-schedules-2,d334b2ca59ccfbd3,9b88eec8d6e39298] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:25:58.500 INFO 21062 --- [ocp-schedules-1,3db6a81388e8dd49,9715cea18caf4182] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:25:58.503 INFO 21062 --- [ocp-schedules-17,8235d136d5f966de,96d300d1f9b17411] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:25:58.503 INFO 21062 --- [ocp-schedules-2,d334b2ca59ccfbd3,9b88eec8d6e39298] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:25:58.505 INFO 21062 --- [ocp-schedules-1,3db6a81388e8dd49,9715cea18caf4182] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:25:58.506 INFO 21062 --- [ocp-schedules-24,d167d5b6fb8f2a6f,73278a9195d7420a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:25:58.511 INFO 21062 --- [ocp-schedules-23,a5b816e58dd2fd81,fe20e23026ae7022] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:25:58.511 INFO 21062 --- [ocp-schedules-16,48d5353f575ce67a,1ebadae055dc6001] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:25:58.511 INFO 21062 --- [ocp-schedules-16,48d5353f575ce67a,1ebadae055dc6001] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:25:58.511 INFO 21062 --- [ocp-schedules-23,a5b816e58dd2fd81,fe20e23026ae7022] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:25:58.512 INFO 21062 --- [ocp-schedules-13,b9297f91ed1e4045,e548cc4e02896225] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:25:58.512 INFO 21062 --- [ocp-schedules-13,b9297f91ed1e4045,e548cc4e02896225] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:25:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:25:58.589950+08:00, lastProcessTime=2024-11-04T14:25:48.591144+08:00 2024-11-04 14:25:58.661 INFO 21062 --- [ocp-async-5,91eb7cf817884f4b,83e10b6fb532e4ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:25:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:25:58.589950+08:00 2024-11-04 14:25:58.804 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:25:58.805 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:25:59.180 INFO 21062 --- [ocp-schedules-6,2833ddaa5b2cb90e,71dfd906b5d56e45] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:25:59.191 WARN 21062 --- [ocp-async-6,b2db765ed3f5e95b,7401ebbef282345c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:25:59.247 INFO 21062 --- [ocp-async-1,1049738beaacf441,b34607e034215000] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:25:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1d5bb3a2cb4bff3f,870105e3271162ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701560, interval=60 2024-11-04 14:26:00.544 INFO 21062 --- [ocp-schedules-15,fdaa6fc34e626737,fd4071c34132e608] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:26:01.191 WARN 21062 --- [ocp-async-1,4a72382ed09e7476,537a34160d9d999c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:01.247 INFO 21062 --- [ocp-async-9,a26d0663e4604eb8,90db88d352eca836] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:01.805 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:01.806 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d56e245bd7c2b0e5,4f3a6889400a7d5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:03.192 WARN 21062 --- [ocp-async-9,fad408c5f7548ef5,48b97f3f968d4b5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:03.247 INFO 21062 --- [ocp-async-3,4708519240170ee8,2e1330686c207dce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:03.614 INFO 21062 --- [ocp-schedules-4,c34441d1d79bf7f8,5a24ac737b3fad7f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:26:04.807 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:04.808 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:05.192 WARN 21062 --- [ocp-async-3,2da2973311b90dca,405546d150f69fd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:05.247 INFO 21062 --- [ocp-async-10,5542dde425be7560,3031fc2fda7cbf45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ee618faec8787782,65c74d76c58f6686] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:06.297 INFO 21062 --- [ocp-schedules-12,0e5aa3bb9cf8b1b6,d24c73c46b0f51b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:26:06.297 INFO 21062 --- [ocp-schedules-12,4ad72c3237442321,c486447136dbf0d5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:26:06.302 INFO 21062 --- [ocp-schedules-12,4ad72c3237442321,c486447136dbf0d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:06.332 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:26:06.332 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:26:07.192 WARN 21062 --- [ocp-async-10,83dd8ed4aacce395,baa8b21a9ddc6014] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:07.247 INFO 21062 --- [ocp-async-2,2b8d763b41d1524e,dc557238b27087de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:07.808 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:07.809 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:08.505 INFO 21062 --- [ocp-schedules-2,d334b2ca59ccfbd3,9b88eec8d6e39298] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:26:08.505 INFO 21062 --- [ocp-schedules-1,3db6a81388e8dd49,9715cea18caf4182] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:26:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:26:08.589701+08:00, lastProcessTime=2024-11-04T14:25:58.589950+08:00 2024-11-04 14:26:08.661 INFO 21062 --- [ocp-async-6,24a762c2b9066a71,5d6f5361c80bfd52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:26:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:26:08.589701+08:00 2024-11-04 14:26:08.983 INFO 21062 --- [prometheus-minute-schedule-1,9f08eb0f46369844,0fbe7c803cf07a5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:09.192 WARN 21062 --- [ocp-async-9,c85be35f9b6d095d,a155b5590b6643a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:09.247 INFO 21062 --- [ocp-async-7,0147c905a827be37,c4e6b8c802ac142a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:26:10.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:26:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:26:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:26:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:26:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:26:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:26:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:26:10.548 INFO 21062 --- [ocp-schedules-6,0e9d2197baee57a9,bc828cd6ce3c7a22] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:26:10.809 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:10.810 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:11.192 WARN 21062 --- [ocp-async-7,7bea10b9aa04a50c,8f4e716fd31ecccf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:11.248 INFO 21062 --- [ocp-async-4,45dd9e07bd45f9a1,937cbb8e740c7c95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:11.570 INFO 21062 --- [ocp-schedules-29,b627e593222dfd13,38464e4464292308] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=7, affectRows=7 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=22, affectRows=22 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=7, validCount=7 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=18, affectRows=18 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=22, validCount=22 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=18, validCount=18 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=32, affectRows=32 2024-11-04 14:26:11.571 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=38, affectRows=38 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=32, validCount=32 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=38, validCount=38 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=50, affectRows=50 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=39, affectRows=39 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=53, affectRows=53 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=38, validCount=38 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=50, validCount=50 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=53, validCount=53 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=75, affectRows=75 2024-11-04 14:26:11.572 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701500, nodeCount=62, affectRows=62 2024-11-04 14:26:11.573 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=74, validCount=74 2024-11-04 14:26:11.573 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701500, nodeCount=61, validCount=61 2024-11-04 14:26:11.574 INFO 21062 --- [ocp-schedules-29,b627e593222dfd13,38464e4464292308] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:26:11.574 INFO 21062 --- [ocp-schedules-29,b627e593222dfd13,38464e4464292308] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:26:11.983 INFO 21062 --- [prometheus-minute-schedule-1,cc16a3e03dfba49c,a37f07e30f01e6e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:13.192 WARN 21062 --- [ocp-async-4,018224695a7441e7,5c764dc4497d38e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:13.248 INFO 21062 --- [ocp-async-5,de39bbab9449db96,2a3c3bb0eec8f7a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:13.639 INFO 21062 --- [ocp-schedules-16,712371e22eaf5787,ada5c859fd06ae16] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:26:13.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:13.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:13.984 INFO 21062 --- [prometheus-cache-monitor1,f8757dc1a78f7411,dda8e5cb7db414e7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:26:13.984 INFO 21062 --- [prometheus-cache-monitor1,f8757dc1a78f7411,dda8e5cb7db414e7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:26:14.983 INFO 21062 --- [prometheus-minute-schedule-1,4cc60aaa991929d3,0d701105e7ff3873] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:15.193 WARN 21062 --- [ocp-async-5,cf55c08f89dbfd5e,d4df1243337f6b5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:15.248 INFO 21062 --- [ocp-async-8,50f7d49db9e284ca,80f34637db3b22ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:16.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:16.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:17.193 WARN 21062 --- [ocp-async-8,facdfe946623b3bc,f4fda373cad6ba0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:17.248 INFO 21062 --- [ocp-async-6,7e79358b8efca34c,91766b39d76c049b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:17.983 INFO 21062 --- [prometheus-minute-schedule-1,304149db4358f4c4,a1fe45bec9ab97c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:26:18.590537+08:00, lastProcessTime=2024-11-04T14:26:08.589701+08:00 2024-11-04 14:26:18.662 INFO 21062 --- [ocp-async-5,6118bd9a3c84bd64,53ac0f4f132ca7db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:26:18.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:26:18.590537+08:00 2024-11-04 14:26:19.193 WARN 21062 --- [ocp-async-4,9a0962c36ddab9da,8fab99a9a14ab31b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:19.248 INFO 21062 --- [ocp-async-2,0e194e5122f08095,c2ad5ea3440e24ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:19.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:19.814 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:20.551 INFO 21062 --- [ocp-schedules-5,32300a4c367066d4,562453dfdc74003f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:26:20.983 INFO 21062 --- [prometheus-minute-schedule-1,c4dc224bc7fc13ba,31a9ffe30f55e115] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:21.193 WARN 21062 --- [ocp-async-9,fa49c6e81671a40d,5b4a09586c70a67a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:21.249 INFO 21062 --- [ocp-async-3,1fac70a4e06a111b,2b5f717dcbd2f628] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:22.814 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:22.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:23.193 WARN 21062 --- [ocp-async-8,747d88356ef98b8f,b02be1bd5247397b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:23.249 INFO 21062 --- [ocp-async-6,9d29d62c8821de04,456aff54d4038d9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:23.681 INFO 21062 --- [ocp-schedules-6,fe5fae522f4432ae,b4802eb7ae83475d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:26:23.983 INFO 21062 --- [prometheus-minute-schedule-1,2bb805aa2a47cc2c,073e7317e22aaccc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:25.193 WARN 21062 --- [ocp-async-10,e220478bb0759616,52e18fb30071b021] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:25.249 INFO 21062 --- [ocp-async-1,d7456f8b755f20b4,fcc660e833468bda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:25.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:25.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f950d93fc2f9933f,681f8359d7f2700c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ec3f1313220f3657,2fabc7e9bf2388de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:26:27.194 WARN 21062 --- [ocp-async-2,895c9e04a5deceb3,6e67571721c47034] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:27.249 INFO 21062 --- [ocp-async-5,69185fece6b2173b,cfd3ff859413b95e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:28.498 INFO 21062 --- [ocp-schedules-19,0fbb9104d0782d6e,1541573b7cbd8a3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:26:28.499 INFO 21062 --- [ocp-schedules-25,96799557835932eb,3317be6e009ece8d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:26:28.499 INFO 21062 --- [ocp-schedules-10,45f61dc99216a912,e5e948a13cfa5eed] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:26:28.500 INFO 21062 --- [ocp-schedules-27,fbc1b6c9e4bd3cf4,d716fb3cba8b7f21] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:26:28.502 INFO 21062 --- [ocp-schedules-18,8898b9e515acfd7c,297f923e3e2481e2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:26:28.503 INFO 21062 --- [ocp-schedules-19,0fbb9104d0782d6e,1541573b7cbd8a3d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:26:28.503 INFO 21062 --- [ocp-schedules-10,45f61dc99216a912,e5e948a13cfa5eed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:28.503 INFO 21062 --- [ocp-schedules-27,fbc1b6c9e4bd3cf4,d716fb3cba8b7f21] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:26:28.504 INFO 21062 --- [ocp-schedules-17,8290e9ba05574a57,c8767b16d4b9431e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:26:28.506 INFO 21062 --- [ocp-schedules-25,96799557835932eb,3317be6e009ece8d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:26:28.511 INFO 21062 --- [ocp-schedules-5,e2569fa1b4ff61ad,abd520180ad135cd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:26:28.511 INFO 21062 --- [ocp-schedules-5,e2569fa1b4ff61ad,abd520180ad135cd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:26:28.511 INFO 21062 --- [ocp-schedules-9,405309928c374320,a0d15a4554ac3f4f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:26:28.511 INFO 21062 --- [ocp-schedules-9,405309928c374320,a0d15a4554ac3f4f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:26:28.512 INFO 21062 --- [ocp-schedules-6,01e4f410a6b6fdce,a1a9fe28a2a35d5c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:26:28.512 INFO 21062 --- [ocp-schedules-6,01e4f410a6b6fdce,a1a9fe28a2a35d5c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:26:28.577 INFO 21062 --- [ocp-schedules-20,8623d6f7f11647de,aef2ef71e0cf6aeb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:26:28.577 INFO 21062 --- [ocp-schedules-20,8623d6f7f11647de,aef2ef71e0cf6aeb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:26:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:26:28.590784+08:00, lastProcessTime=2024-11-04T14:26:18.590537+08:00 2024-11-04 14:26:28.662 INFO 21062 --- [ocp-async-1,3ad2b5bb49188f14,696e21404955487e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:26:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:26:28.590784+08:00 2024-11-04 14:26:28.806 INFO 21062 --- [ocp-schedules-8,60d2379ea2ea5d70,d1b8141e91d0bf7a] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:26:28.806 INFO 21062 --- [ocp-schedules-8,60d2379ea2ea5d70,d1b8141e91d0bf7a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:26:28.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:28.819 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:28.836 INFO 21062 --- [ocp-schedules-29,0b3a4bee2f6b0a9f,3503f04bddea99f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:26:29.107 INFO 21062 --- [ocp-schedules-25,eca29acde44fdad8,9c36f361bae0980e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:26:29.107 INFO 21062 --- [ocp-schedules-25,eca29acde44fdad8,9c36f361bae0980e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:26:29.183 INFO 21062 --- [ocp-schedules-18,e79ad1e861bc4fcf,faa15a209352afef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:26:29.194 WARN 21062 --- [ocp-async-10,3912ff2d816ea889,bb7d73f44cf2e946] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:29.214 INFO 21062 --- [ocp-schedules-2,bd74867f99356092,419eb4714b214e18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:26:29.219 INFO 21062 --- [ocp-schedules-2,bd74867f99356092,419eb4714b214e18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:26:29.219 INFO 21062 --- [ocp-schedules-2,bd74867f99356092,419eb4714b214e18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:26:29.250 INFO 21062 --- [ocp-async-4,c23fd4a32b788895,d40f5d2be7de21ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:29.552 INFO 21062 --- [ocp-schedules-14,739134bd523d852d,65e43be7313ff0b7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:26:29.553 INFO 21062 --- [ocp-schedules-14,739134bd523d852d,65e43be7313ff0b7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:26:29.554 INFO 21062 --- [ocp-schedules-14,739134bd523d852d,65e43be7313ff0b7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:26:29.554 INFO 21062 --- [ocp-schedules-14,739134bd523d852d,65e43be7313ff0b7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:26:29.554 INFO 21062 --- [ocp-schedules-14,739134bd523d852d,65e43be7313ff0b7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:26:29.554 INFO 21062 --- [ocp-schedules-14,739134bd523d852d,65e43be7313ff0b7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:26:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3fd7f2bfdfe097a1,012c0f10dc463bc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:26:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:26:30.554 INFO 21062 --- [ocp-schedules-20,f052ac0474dfec87,2cad7ef289ff3727] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:26:31.194 WARN 21062 --- [ocp-async-1,73f0d417d22c7218,046ce9d37bf524b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:31.250 INFO 21062 --- [ocp-async-9,ee860a5cf1b8ab09,2f4e2cfc2df33594] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:31.578 INFO 21062 --- [ocp-schedules-17,0cce3d1101d7dea8,b1f4264e5432bf98] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:26:31.584 INFO 21062 --- [ocp-schedules-17,0cce3d1101d7dea8,b1f4264e5432bf98] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:26:31.584 INFO 21062 --- [ocp-schedules-17,0cce3d1101d7dea8,b1f4264e5432bf98] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:26:31.634 INFO 21062 --- [ocp-schedules-5,993527080bb9a4eb,75509c4db7bcc8f4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:26:31.634 INFO 21062 --- [ocp-schedules-5,993527080bb9a4eb,75509c4db7bcc8f4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:26:31.662 INFO 21062 --- [ocp-schedules-5,993527080bb9a4eb,75509c4db7bcc8f4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:26:31.662 INFO 21062 --- [ocp-schedules-5,993527080bb9a4eb,75509c4db7bcc8f4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:26:31.665 INFO 21062 --- [ocp-schedules-11,3873917acfd133c6,00c4be2c1912ec94] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:26:31.819 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:31.820 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:31.849 INFO 21062 --- [ocp-schedules-16,1ed6b0473f9594e7,2adabb729113e4d7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:26:31.849+08:00, clusters=[1] 2024-11-04 14:26:31.851 INFO 21062 --- [ocp-schedules-16,1ed6b0473f9594e7,2adabb729113e4d7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:26:32.008 INFO 21062 --- [ocp-schedules-30,c016d035d3cea0c1,1fec29fd50f17896] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:26:32.011 INFO 21062 --- [ocp-schedules-30,c016d035d3cea0c1,1fec29fd50f17896] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:32.018 INFO 21062 --- [ocp-schedules-8,dc3e2dc270d99806,404671c42c523cdc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:26:32.018+08:00, clusters=[1] 2024-11-04 14:26:32.022 INFO 21062 --- [ocp-schedules-8,dc3e2dc270d99806,404671c42c523cdc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:26:32.069 ERROR 21062 --- [ocp-schedules-30,c016d035d3cea0c1,1fec29fd50f17896] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:26:32.069 INFO 21062 --- [ocp-schedules-30,c016d035d3cea0c1,1fec29fd50f17896] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:26:32.983 INFO 21062 --- [prometheus-minute-schedule-1,9f3ffbe1d84948c7,78b62fca579f63a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:33.033 INFO 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:26:33.033 INFO 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:26:33.033 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:26:33.033 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:26:33.043 INFO 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:26:33.044 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:26:33.044 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:26:33.046 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:26:33.046 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:26:33.046 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:26:33.046 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:26:33.047 WARN 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:26:33.048 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:26:33.048 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:26:33.048 INFO 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:26:33.049 WARN 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:26:33.056 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:26:33.058 INFO 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:26:33.058 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:26:33.070 ERROR 21062 --- [ocp-schedules-19,bc6811ade6ba3dc2,bbc5cb86bd4c19e7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:26:33.194 WARN 21062 --- [ocp-async-5,e80db7346611a507,16cc60ec8c2cf61c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:33.250 INFO 21062 --- [ocp-async-8,0b801f078b936b94,2c21091e7f15d5ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:33.706 INFO 21062 --- [ocp-schedules-21,4caa30a75b26821b,5125020b00fa5ab8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:26:34.821 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:34.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:35.194 WARN 21062 --- [ocp-async-8,e22c84eda1fc102c,af9c053db3dbfdeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:35.250 INFO 21062 --- [ocp-async-6,5c2d62f013b77750,81c6e25f9a14fbf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1abd8bf7433b7890,a8df53056efe1511] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:36.333 INFO 21062 --- [ocp-schedules-12,4ad72c3237442321,c486447136dbf0d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:26:36.333 INFO 21062 --- [ocp-schedules-12,e41efc17c3f16e22,9b16f7cb867afcb2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:26:36.337 INFO 21062 --- [ocp-schedules-12,e41efc17c3f16e22,9b16f7cb867afcb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:36.365 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:26:36.365 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:26:37.195 WARN 21062 --- [ocp-async-6,6a26b239e722929d,79a009639af6b856] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:37.250 INFO 21062 --- [ocp-async-2,ba18514a0369552f,880bba07a74a489c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:37.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:37.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:38.504 INFO 21062 --- [ocp-schedules-27,fbc1b6c9e4bd3cf4,d716fb3cba8b7f21] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:26:38.504 INFO 21062 --- [ocp-schedules-10,45f61dc99216a912,e5e948a13cfa5eed] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:26:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:26:38.590568+08:00, lastProcessTime=2024-11-04T14:26:28.590784+08:00 2024-11-04 14:26:38.662 INFO 21062 --- [ocp-async-10,44cca6844ee3bd16,e4f9890e671db1d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:26:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:26:38.590568+08:00 2024-11-04 14:26:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e3f8a4662202dc7a,312a083824a1565f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:39.036 INFO 21062 --- [ocp-schedules-22,055ba31e1ec033b4,b040821fad4faff1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:26:39.040 INFO 21062 --- [ocp-schedules-22,055ba31e1ec033b4,b040821fad4faff1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:39.041 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:26:39.050 INFO 21062 --- [ocp-schedules-23,374f48553ca7562f,a6304422a773fd69] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:26:39.054 INFO 21062 --- [ocp-schedules-23,374f48553ca7562f,a6304422a773fd69] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:39.137 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:26:39.139 ERROR 21062 --- [ocp-schedules-23,374f48553ca7562f,a6304422a773fd69] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:26:39.139 INFO 21062 --- [ocp-schedules-23,374f48553ca7562f,a6304422a773fd69] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:26:39.195 WARN 21062 --- [ocp-async-5,bb06508a5ca1de69,986d873a83083543] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:39.250 INFO 21062 --- [ocp-async-7,97b6d3591ee15459,5a42541d78ef2743] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:26:40.005 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:26:40.557 INFO 21062 --- [ocp-schedules-17,6094ff8f4c0a0207,95e77dc70f52d8e9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:26:40.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:40.824 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:41.195 WARN 21062 --- [ocp-async-7,82cb67175ce1dded,11fbe0d7f2a4bd98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:41.251 INFO 21062 --- [ocp-async-4,f97cce8d4e2081a0,91ef72db40f584a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:26:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:26:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:26:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:26:41.802 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:26:41.834 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:26:41.834 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=32ms, period=120000ms 2024-11-04 14:26:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2097e1826f9d42c3,50e70f783f861aac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:43.195 WARN 21062 --- [ocp-async-4,64395e30b3a99771,3fde9014905958be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:43.251 INFO 21062 --- [ocp-async-9,8b47bfd3ffa78680,8cda36ae6dfc2570] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:43.746 INFO 21062 --- [ocp-schedules-21,0767dd85c82737cf,f3c0c09d9d1f5338] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:26:43.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:43.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:44.983 INFO 21062 --- [prometheus-minute-schedule-1,cd5f12dc36235dcc,bf89f0334603406b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:45.195 WARN 21062 --- [ocp-async-9,38341a56fdf025c8,616c2c394fca1ff0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:45.251 INFO 21062 --- [ocp-async-3,c0a883c9f729c670,9500811882e106e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:46.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:46.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:47.196 WARN 21062 --- [ocp-async-3,4ff35018e6b1419a,577d1fb2d9141d0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:47.251 INFO 21062 --- [ocp-async-6,cdcf212cc8cbb1e8,b36dd13329c59407] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:47.983 INFO 21062 --- [prometheus-minute-schedule-1,ed50294555d93d8d,ee7310ac7161e2c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:26:48.590790+08:00, lastProcessTime=2024-11-04T14:26:38.590568+08:00 2024-11-04 14:26:48.662 INFO 21062 --- [ocp-async-5,2ecc68aae7059764,aa7f45372a6e5343] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:26:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:26:48.590790+08:00 2024-11-04 14:26:49.041 INFO 21062 --- [ocp-schedules-22,055ba31e1ec033b4,b040821fad4faff1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:26:49.196 WARN 21062 --- [ocp-async-4,2abab53a3057b6a2,6ad2c751af0d845d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:49.251 INFO 21062 --- [ocp-async-9,80980380f6320ddb,82673206230abaf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:49.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:49.829 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:50.560 INFO 21062 --- [ocp-schedules-8,540e13cb4871a289,e79b07ccac451fb4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:26:50.983 INFO 21062 --- [prometheus-minute-schedule-1,bde274bef9183ce6,ad973f4ebc3e5a5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:51.196 WARN 21062 --- [ocp-async-9,d50f1062473de1da,abe00b132c76dbcd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:51.252 INFO 21062 --- [ocp-async-3,b9c61909104cf09f,17dec69c7951cd2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:51.587 INFO 21062 --- [ocp-schedules-26,b4d7d9a54c3f67da,5112f2710134b16e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:26:51.593 INFO 21062 --- [ocp-schedules-26,b4d7d9a54c3f67da,5112f2710134b16e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:26:51.593 INFO 21062 --- [ocp-schedules-26,b4d7d9a54c3f67da,5112f2710134b16e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:26:52.829 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:52.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:53.196 WARN 21062 --- [ocp-async-3,34ea7d3ecb075eba,238b1eb9c6478186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:53.252 INFO 21062 --- [ocp-async-6,6fba187132698bfe,47748abc39994617] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:53.771 INFO 21062 --- [ocp-schedules-11,0097b7ee22b19867,3d4f7b1f9b3d2808] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:26:53.983 INFO 21062 --- [prometheus-minute-schedule-1,15508d605f935116,c304159cf9edfff9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:55.196 WARN 21062 --- [ocp-async-6,79df84506cb1a58b,0c009b4c80b5a0fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:55.252 INFO 21062 --- [ocp-async-2,a715691c2532eb93,580bc82b224912d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:55.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:55.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:56.983 INFO 21062 --- [prometheus-minute-schedule-1,229473881b7ad90d,8491684e94c78743] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:26:57.197 WARN 21062 --- [ocp-async-2,41b201c283e41429,d703016e5da0845b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:57.252 INFO 21062 --- [ocp-async-8,5d0f306124ea56db,86c7413e4f711ddc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:58.500 INFO 21062 --- [ocp-schedules-9,40822cc168ee652c,842225ad1a13b09c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:26:58.500 INFO 21062 --- [ocp-schedules-21,265f921c639a1dbf,62be3008efb90dbd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:26:58.502 INFO 21062 --- [ocp-schedules-2,146ebe4a0b9d5b80,29b1180d677a0d15] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:26:58.504 INFO 21062 --- [ocp-schedules-9,40822cc168ee652c,842225ad1a13b09c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:58.504 INFO 21062 --- [ocp-schedules-21,265f921c639a1dbf,62be3008efb90dbd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:26:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:26:58.505 INFO 21062 --- [ocp-schedules-4,761b0f703374e40c,fb521b04c62fa0ea] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:26:58.512 INFO 21062 --- [ocp-schedules-18,a1a8ecf06ffcb8dc,26f993a7a05e83b5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:26:58.512 INFO 21062 --- [ocp-schedules-18,a1a8ecf06ffcb8dc,26f993a7a05e83b5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:26:58.513 INFO 21062 --- [ocp-schedules-29,ab980c95517c1369,8179425b6d091c3e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:26:58.513 INFO 21062 --- [ocp-schedules-29,ab980c95517c1369,8179425b6d091c3e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:26:58.513 INFO 21062 --- [ocp-schedules-17,747bfee4012209e2,f53ccd2cbab4c155] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:26:58.513 INFO 21062 --- [ocp-schedules-17,747bfee4012209e2,f53ccd2cbab4c155] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:26:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:26:58.590351+08:00, lastProcessTime=2024-11-04T14:26:48.590790+08:00 2024-11-04 14:26:58.662 INFO 21062 --- [ocp-async-4,6647550580b547a1,47391c3c3f722e33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:26:58.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:26:58.590351+08:00 2024-11-04 14:26:58.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:26:58.833 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:26:59.185 INFO 21062 --- [ocp-schedules-10,42fd346c10ac8225,bbc0e295fac626e4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:26:59.197 WARN 21062 --- [ocp-async-3,dbfa7d0e2da49376,90118671ef8de497] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:26:59.252 INFO 21062 --- [ocp-async-6,4cfed5237acc3056,ab8ba8fef1ec0c7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:26:59.983 INFO 21062 --- [prometheus-minute-schedule-1,be05d1d1df8177fc,b562956bad6ec065] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701620, interval=60 2024-11-04 14:27:00.563 INFO 21062 --- [ocp-schedules-5,21181a7c4915fc21,1e83792f4282cca5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:27:01.197 WARN 21062 --- [ocp-async-6,577a7100b9106afa,0b4b54f0a2486889] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:01.252 INFO 21062 --- [ocp-async-2,abc3cf3dc1e99ade,515c18966f5c89af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:01.833 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:01.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ef47c067ea0eae7d,e3258ffa173950a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:03.197 WARN 21062 --- [ocp-async-2,a5bcb1543b6b974c,ea261d68353cb829] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:03.252 INFO 21062 --- [ocp-async-8,a0e4af58bebf2a56,269bfd10e0189950] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:03.850 INFO 21062 --- [ocp-schedules-25,52344fba5482c522,39b07a8249552a70] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:27:04.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:04.836 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:05.197 WARN 21062 --- [ocp-async-8,31080fbcea19e56f,0501a31553d89024] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:05.253 INFO 21062 --- [ocp-async-10,bd75245327278482,338b7c7969f622c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2c7dc1f71f5bbc6c,b4743e909a2f04b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:06.365 INFO 21062 --- [ocp-schedules-12,e41efc17c3f16e22,9b16f7cb867afcb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:27:06.365 INFO 21062 --- [ocp-schedules-12,7ce24b65636357bf,a2a1ffbf460f2996] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:27:06.369 INFO 21062 --- [ocp-schedules-12,7ce24b65636357bf,a2a1ffbf460f2996] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:06.400 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:27:06.400 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:27:07.197 WARN 21062 --- [ocp-async-10,1b6c1659ba55a90b,13ed5763b4758afd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:07.253 INFO 21062 --- [ocp-async-1,d53d624b21547df8,71691225c47e4192] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:07.836 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:07.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:08.505 INFO 21062 --- [ocp-schedules-9,40822cc168ee652c,842225ad1a13b09c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:27:08.505 INFO 21062 --- [ocp-schedules-21,265f921c639a1dbf,62be3008efb90dbd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:27:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:27:08.590328+08:00, lastProcessTime=2024-11-04T14:26:58.590351+08:00 2024-11-04 14:27:08.662 INFO 21062 --- [ocp-async-3,5004cef4962880f4,54eb62caee91af9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:27:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:27:08.590328+08:00 2024-11-04 14:27:08.983 INFO 21062 --- [prometheus-minute-schedule-1,538bc9ed0fb91dcb,081f45079fb66238] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:09.197 WARN 21062 --- [ocp-async-2,57aaf5e6fe589653,3bb42b367390873c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:09.253 INFO 21062 --- [ocp-async-5,19d71469b761908d,11d553e866b2138d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:27:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:27:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:27:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:27:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:27:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:27:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:27:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:27:10.566 INFO 21062 --- [ocp-schedules-30,757e2132072f046a,de1b572f120d3edd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:27:10.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:10.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:11.198 WARN 21062 --- [ocp-async-5,183e0b525995e8da,93f570fcda44a765] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:11.253 INFO 21062 --- [ocp-async-7,efe347e727b0a34b,9ed8b61010c51fec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:11.573 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=2, affectRows=2 2024-11-04 14:27:11.573 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=2, validCount=2 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=10, affectRows=10 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=19, affectRows=19 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=12, affectRows=12 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=10, validCount=10 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=28, affectRows=28 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=12, validCount=12 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=19, validCount=19 2024-11-04 14:27:11.574 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=28, validCount=28 2024-11-04 14:27:11.575 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=42, affectRows=42 2024-11-04 14:27:11.575 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=61, affectRows=61 2024-11-04 14:27:11.575 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=42, validCount=42 2024-11-04 14:27:11.575 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=50, affectRows=50 2024-11-04 14:27:11.575 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=61, validCount=61 2024-11-04 14:27:11.575 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=50, validCount=50 2024-11-04 14:27:11.576 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=84, affectRows=84 2024-11-04 14:27:11.576 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701560, nodeCount=88, affectRows=88 2024-11-04 14:27:11.576 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=82, validCount=82 2024-11-04 14:27:11.577 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701560, nodeCount=87, validCount=87 2024-11-04 14:27:11.595 INFO 21062 --- [ocp-schedules-20,c49b103623caa768,11fe56ee56ae0600] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:27:11.601 INFO 21062 --- [ocp-schedules-20,c49b103623caa768,11fe56ee56ae0600] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:27:11.601 INFO 21062 --- [ocp-schedules-20,c49b103623caa768,11fe56ee56ae0600] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:27:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6069a36b4029d29b,98ca4d1633494cf9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:13.198 WARN 21062 --- [ocp-async-7,3e55d224ef66ca25,011f55bb810d23a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:13.253 INFO 21062 --- [ocp-async-4,d3529c7fe2e46cd7,d703e8918db4f4db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:13.839 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:13.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:13.875 INFO 21062 --- [ocp-schedules-23,01a0ec6e69a6942b,977528b9fdf9ea07] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:27:13.984 INFO 21062 --- [prometheus-cache-monitor1,cfd509cfb9a8f0c9,f9e50b9ad0304982] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:27:13.984 INFO 21062 --- [prometheus-cache-monitor1,cfd509cfb9a8f0c9,f9e50b9ad0304982] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:27:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5ac7d84465f71b45,4ec1259e3f2df6bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:15.082 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:23:15.059, end=2024-11-04T14:25:15.079} 2024-11-04 14:27:15.086 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:27:15.086 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:27:15.198 WARN 21062 --- [ocp-async-4,5f0e752d65712f0e,f2c890bc2770adf7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:15.254 INFO 21062 --- [ocp-async-9,e6a801f7b60b2ed8,ecfaec974e7cf855] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:16.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:16.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:17.198 WARN 21062 --- [ocp-async-8,bb5711557fd89623,7298a94232228105] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:17.254 INFO 21062 --- [ocp-async-3,9c2839a598fe19d3,12db87da935c2e4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:17.983 INFO 21062 --- [prometheus-minute-schedule-1,71bc87ee3a13b690,a0b50e2175442a67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:27:18.590590+08:00, lastProcessTime=2024-11-04T14:27:08.590328+08:00 2024-11-04 14:27:18.663 INFO 21062 --- [ocp-async-4,9274cf300817a3a0,de5e279da04c0f0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:27:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:27:18.590590+08:00 2024-11-04 14:27:19.198 WARN 21062 --- [ocp-async-7,63a5ec26233cb73d,74a98112fb096e71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:19.254 INFO 21062 --- [ocp-async-1,ec87e679ca5a1fd5,d69e10781cd25b91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:19.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:19.843 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:20.569 INFO 21062 --- [ocp-schedules-20,40ee0191d4d9934f,f8e5fd1f1ca20d0b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:27:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ea82ce6af8059a3d,ac0b7817d89631af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:21.198 WARN 21062 --- [ocp-async-2,d0233d02f60796bf,fdc9d2ac623d2721] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:21.254 INFO 21062 --- [ocp-async-9,cee99ceedeaa6e51,2958764f147d52dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:22.843 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:22.844 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:23.199 WARN 21062 --- [ocp-async-8,5c43de4e0573b643,1e8dac36753b6941] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:23.254 INFO 21062 --- [ocp-async-3,17bb0e9d759976d6,b32514c90d305fd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:23.917 INFO 21062 --- [ocp-schedules-25,e49c24332d045b88,f4c95d0696073814] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:27:23.983 INFO 21062 --- [prometheus-minute-schedule-1,3e7eaf8494de7331,6328bdf366866c20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:25.199 WARN 21062 --- [ocp-async-10,77c5f6bfc25490e6,21c07721675b826b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:25.254 INFO 21062 --- [ocp-async-6,45a518845e487183,367c46b797f3c427] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:25.844 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:25.846 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:26.983 INFO 21062 --- [prometheus-minute-schedule-1,dee0164dcd72c30b,a83a80ca05a61e70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2d8309b9dd989d8e,984ad6e1dd87932a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:27:27.199 WARN 21062 --- [ocp-async-1,dc3f09f557fdcc91,df0b741a31113c81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:27.255 INFO 21062 --- [ocp-async-4,f6009e9488c6bcfc,759179f56151dad1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:28.498 INFO 21062 --- [ocp-schedules-3,14b2a67874d817b5,76e5c0b63b082130] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:27:28.499 INFO 21062 --- [ocp-schedules-30,ed54fd08186e59d0,582323a163683028] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:27:28.500 INFO 21062 --- [ocp-schedules-11,98e8d397fcd7ff60,e589225f3d519a79] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:27:28.500 INFO 21062 --- [ocp-schedules-22,afd18629f05a0d1c,4ac598e49c517d09] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:27:28.503 INFO 21062 --- [ocp-schedules-2,d4290d7afdd91dce,346005a0863a3592] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:27:28.503 INFO 21062 --- [ocp-schedules-3,14b2a67874d817b5,76e5c0b63b082130] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:27:28.504 INFO 21062 --- [ocp-schedules-11,98e8d397fcd7ff60,e589225f3d519a79] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:28.504 INFO 21062 --- [ocp-schedules-22,afd18629f05a0d1c,4ac598e49c517d09] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:28.504 INFO 21062 --- [ocp-schedules-1,f9a3e3da2cda763f,6710103a19ce5314] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:27:28.504 INFO 21062 --- [ocp-schedules-30,ed54fd08186e59d0,582323a163683028] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:27:28.504 INFO 21062 --- [ocp-schedules-14,21c007890157081a,b3bc539e479a8a5a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:27:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:27:28.507 INFO 21062 --- [ocp-schedules-14,21c007890157081a,b3bc539e479a8a5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:28.511 INFO 21062 --- [ocp-schedules-17,1d7935ff352d22ac,2edd991ad1d36f06] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:27:28.511 INFO 21062 --- [ocp-schedules-8,e10faef9f1a33cde,0fb4f7a5d8c8131a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:27:28.511 INFO 21062 --- [ocp-schedules-17,1d7935ff352d22ac,2edd991ad1d36f06] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:27:28.511 INFO 21062 --- [ocp-schedules-8,e10faef9f1a33cde,0fb4f7a5d8c8131a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:27:28.511 INFO 21062 --- [ocp-schedules-21,af831ddd488fb752,aeef6c35e287a0f2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:27:28.511 INFO 21062 --- [ocp-schedules-21,af831ddd488fb752,aeef6c35e287a0f2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:27:28.537 INFO 21062 --- [ocp-schedules-26,7e74445497230153,d0928d3ec9ae9550] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:27:28.537 INFO 21062 --- [ocp-schedules-26,7e74445497230153,d0928d3ec9ae9550] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:27:28.581 INFO 21062 --- [pool-12-thread-15,66ad5415e8d5a8fd,c2432e7a6f0ad838] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:27:28.581 INFO 21062 --- [pool-12-thread-14,66ad5415e8d5a8fd,6ca05ec6955ad0a4] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:27:28.581 INFO 21062 --- [pool-12-thread-13,66ad5415e8d5a8fd,cb32ba2a3b8dbdc1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:27:28.581 INFO 21062 --- [pool-12-thread-13,66ad5415e8d5a8fd,cb32ba2a3b8dbdc1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:27:28.581 INFO 21062 --- [pool-12-thread-16,66ad5415e8d5a8fd,306b78590c168080] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:27:28.585 INFO 21062 --- [pool-12-thread-13,66ad5415e8d5a8fd,cb32ba2a3b8dbdc1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:27:28.586 INFO 21062 --- [pool-12-thread-13,66ad5415e8d5a8fd,cb32ba2a3b8dbdc1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:27:28.586 INFO 21062 --- [ForkJoinPool-11-worker-11,66ad5415e8d5a8fd,b2305bff447d5191] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:27:28.586 INFO 21062 --- [ForkJoinPool-11-worker-11,66ad5415e8d5a8fd,b2305bff447d5191] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:27:28.586 INFO 21062 --- [ForkJoinPool-11-worker-4,66ad5415e8d5a8fd,740f2223e21a2a2a] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:27:28.586 INFO 21062 --- [ForkJoinPool-11-worker-11,66ad5415e8d5a8fd,b2305bff447d5191] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:27:28.587 INFO 21062 --- [pool-12-thread-15,66ad5415e8d5a8fd,c2432e7a6f0ad838] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:27:28.587 INFO 21062 --- [pool-12-thread-14,66ad5415e8d5a8fd,6ca05ec6955ad0a4] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:27:28.589 INFO 21062 --- [pool-12-thread-16,66ad5415e8d5a8fd,306b78590c168080] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:27:28.589 INFO 21062 --- [pool-12-thread-16,66ad5415e8d5a8fd,306b78590c168080] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:27:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:27:28.589913+08:00, lastProcessTime=2024-11-04T14:27:18.590590+08:00 2024-11-04 14:27:28.631 INFO 21062 --- [ocp-schedules-25,5c8d10b472f5c3b3,fcbcda55ff7d5e16] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:27:28.632 INFO 21062 --- [ocp-schedules-25,5c8d10b472f5c3b3,fcbcda55ff7d5e16] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:27:28.663 INFO 21062 --- [ocp-async-6,c23b1640afbfcae7,ff924314e48d057b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:27:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:27:28.589913+08:00 2024-11-04 14:27:28.839 INFO 21062 --- [ocp-schedules-30,dd9703ad54302c08,0fb7520a1d9fa974] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:27:28.846 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:28.847 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:28.958 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:27:28.976 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:27:28.977 WARN 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:27:28.978 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:27:28.979 WARN 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:27:28.986 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:27:28.987 WARN 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:27:28.988 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:27:28.989 WARN 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:27:28.995 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:27:28.996 WARN 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:27:28.996 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:27:28.997 WARN 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:27:29.003 INFO 21062 --- [ocp-schedules-2,a7303ec3ec98e5fb,f8f504d9b63fb6c3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:27:29.114 INFO 21062 --- [ocp-schedules-3,46ead4c517a3a35c,72886dadfad3ed98] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:27:29.114 INFO 21062 --- [ocp-schedules-3,46ead4c517a3a35c,72886dadfad3ed98] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:27:29.188 INFO 21062 --- [ocp-schedules-26,e177a130b99269de,252fc2844e205b83] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:27:29.199 WARN 21062 --- [ocp-async-10,b7031d4ac6b3df57,637d6a4e637a8f02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:29.220 INFO 21062 --- [ocp-schedules-7,27b537e396a05d86,5bb48c3a190cc300] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:27:29.224 INFO 21062 --- [ocp-schedules-7,27b537e396a05d86,5bb48c3a190cc300] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:27:29.224 INFO 21062 --- [ocp-schedules-7,27b537e396a05d86,5bb48c3a190cc300] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:27:29.255 INFO 21062 --- [ocp-async-7,4342027d480627e2,4f7a5f6c4cb98ddc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:29.555 INFO 21062 --- [ocp-schedules-18,fd01495ee9b43b11,8d3ecd9fdbe94a63] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:27:29.555 INFO 21062 --- [ocp-schedules-18,fd01495ee9b43b11,8d3ecd9fdbe94a63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:27:29.557 INFO 21062 --- [ocp-schedules-18,fd01495ee9b43b11,8d3ecd9fdbe94a63] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:27:29.557 INFO 21062 --- [ocp-schedules-18,fd01495ee9b43b11,8d3ecd9fdbe94a63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:27:29.557 INFO 21062 --- [ocp-schedules-18,fd01495ee9b43b11,8d3ecd9fdbe94a63] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:27:29.557 INFO 21062 --- [ocp-schedules-18,fd01495ee9b43b11,8d3ecd9fdbe94a63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:27:29.803 INFO 21062 --- [ocp-schedules-29,cc428aedcea29c15,f199fcadd609e3a0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:27:29.803 INFO 21062 --- [ocp-schedules-29,cc428aedcea29c15,f199fcadd609e3a0] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:27:29.830 INFO 21062 --- [pool-42-thread-1,cc428aedcea29c15,a67a6d986f24992d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:27:29.858 WARN 21062 --- [ocp-schedules-29,cc428aedcea29c15,f199fcadd609e3a0] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:27:29.858 INFO 21062 --- [ocp-schedules-29,cc428aedcea29c15,f199fcadd609e3a0] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:27:29.983 INFO 21062 --- [prometheus-minute-schedule-1,273ffd830d0ca8c6,5ab8802e0a9d1b83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:27:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:27:30.572 INFO 21062 --- [ocp-schedules-27,8ea342557834b4c1,df51b8e62840a059] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:27:31.199 WARN 21062 --- [ocp-async-6,cab62380a22ae1b4,493e01e7ed88b349] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:31.255 INFO 21062 --- [ocp-async-2,2e5055dcbeb9e3fd,5ca293d1709289d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:31.604 INFO 21062 --- [ocp-schedules-26,ce06032dcd373b04,15cdcb2ee60e207f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:27:31.609 INFO 21062 --- [ocp-schedules-26,ce06032dcd373b04,15cdcb2ee60e207f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:27:31.609 INFO 21062 --- [ocp-schedules-26,ce06032dcd373b04,15cdcb2ee60e207f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:27:31.662 INFO 21062 --- [ocp-schedules-20,f268162c0e471bed,fe07d90d95c1c7a3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:27:31.662 INFO 21062 --- [ocp-schedules-20,f268162c0e471bed,fe07d90d95c1c7a3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:27:31.689 INFO 21062 --- [ocp-schedules-20,f268162c0e471bed,fe07d90d95c1c7a3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:27:31.689 INFO 21062 --- [ocp-schedules-20,f268162c0e471bed,fe07d90d95c1c7a3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:27:31.689 INFO 21062 --- [ocp-schedules-7,640983c144bc8333,c6ed634d50225734] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:27:31.847 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:31.848 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:31.877 INFO 21062 --- [ocp-schedules-5,4a1875fa60e037b9,105fda926fa157ec] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:27:31.877+08:00, clusters=[1] 2024-11-04 14:27:31.878 INFO 21062 --- [ocp-schedules-5,4a1875fa60e037b9,105fda926fa157ec] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:27:32.047 INFO 21062 --- [ocp-schedules-24,3be52611ef6422f8,29744f3fb6731b77] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:27:32.047+08:00, clusters=[1] 2024-11-04 14:27:32.050 INFO 21062 --- [ocp-schedules-24,3be52611ef6422f8,29744f3fb6731b77] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:27:32.983 INFO 21062 --- [prometheus-minute-schedule-1,18427b7603c36fd8,c7f75e28b57044e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:33.080 INFO 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:27:33.080 INFO 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:27:33.080 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:27:33.080 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:27:33.089 INFO 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:27:33.090 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:27:33.090 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:27:33.091 WARN 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:27:33.091 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:27:33.092 INFO 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:27:33.092 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:27:33.092 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:27:33.092 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:27:33.093 WARN 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:27:33.093 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:27:33.093 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:27:33.099 INFO 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:27:33.099 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:27:33.099 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:27:33.111 ERROR 21062 --- [ocp-schedules-2,09e63f7409f2d2cd,e3f1769963d12382] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:27:33.200 WARN 21062 --- [ocp-async-4,6f0aec0e0177fd8f,0488c6b3a6e68738] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:33.255 INFO 21062 --- [ocp-async-8,811ddddc6064870d,524264565052a550] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:33.943 INFO 21062 --- [ocp-schedules-19,ae21bf53d8d9cfe9,5ab5f1593ac5d67a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:27:34.848 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:34.849 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:35.200 WARN 21062 --- [ocp-async-8,3ba776470673aba4,9f719dc88be1038f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:35.255 INFO 21062 --- [ocp-async-3,eba86b69dffea270,4cdd47dfc0d78f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1104a7baf0b78b3a,49df5171dad46ba3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:36.400 INFO 21062 --- [ocp-schedules-12,7ce24b65636357bf,a2a1ffbf460f2996] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:27:36.400 INFO 21062 --- [ocp-schedules-12,84aef6d082fcc052,02665a921dac05b4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:27:36.404 INFO 21062 --- [ocp-schedules-12,84aef6d082fcc052,02665a921dac05b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:36.434 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:27:36.434 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:27:37.200 WARN 21062 --- [ocp-async-3,e020c090cc1371e7,c5977c8593464059] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:37.255 INFO 21062 --- [ocp-async-1,8ed4a214546115bd,99f830f162a7c64f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:37.849 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:37.850 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:38.505 INFO 21062 --- [ocp-schedules-22,afd18629f05a0d1c,4ac598e49c517d09] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:27:38.505 INFO 21062 --- [ocp-schedules-11,98e8d397fcd7ff60,e589225f3d519a79] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:27:38.508 INFO 21062 --- [ocp-schedules-14,21c007890157081a,b3bc539e479a8a5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:27:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:27:38.590120+08:00, lastProcessTime=2024-11-04T14:27:28.589913+08:00 2024-11-04 14:27:38.663 INFO 21062 --- [ocp-async-10,17bc8c67e52defc0,d6fbd451de4e3516] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:27:38.807 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:27:38.590120+08:00 2024-11-04 14:27:38.983 INFO 21062 --- [prometheus-minute-schedule-1,02ed46320bbc62bf,af779ebc5308ab4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:39.139 INFO 21062 --- [ocp-schedules-13,437cd808e347a0d7,5cc89d4f59bea6a0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:27:39.143 INFO 21062 --- [ocp-schedules-13,437cd808e347a0d7,5cc89d4f59bea6a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:39.200 WARN 21062 --- [ocp-async-4,5984bb89e926e949,b962acc53dc38a5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:39.229 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:27:39.232 ERROR 21062 --- [ocp-schedules-13,437cd808e347a0d7,5cc89d4f59bea6a0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:27:39.232 INFO 21062 --- [ocp-schedules-13,437cd808e347a0d7,5cc89d4f59bea6a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:27:39.256 INFO 21062 --- [ocp-async-5,e338bffe8e335a23,eb8daad676d1ed87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:40.002 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:27:40.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:27:40.575 INFO 21062 --- [ocp-schedules-17,a3218c304ebdbb11,4505b32fec1f3f86] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:27:40.851 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:40.852 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:41.200 WARN 21062 --- [ocp-async-5,7b901c926bd15277,a360478720f59ff7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:41.256 INFO 21062 --- [ocp-async-6,ebc2092a428312db,4abe9fd80b19ba6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2f027fbeb7fcd979,7e71f54be29d905c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:43.200 WARN 21062 --- [ocp-async-6,b3b6d08711d24935,bc2c6007b33c47b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:43.256 INFO 21062 --- [ocp-async-2,40d7881cf4d151da,2563c4f53f12c7e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:43.852 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:43.854 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:43.986 INFO 21062 --- [ocp-schedules-16,b146125d7773ea60,1085820715901774] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:27:44.983 INFO 21062 --- [prometheus-minute-schedule-1,c90346e805b6f30d,1d6932f88442eb55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:45.112 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:27:45.112 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:27:45.119 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:27:45.119 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:27:45.119 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:27:45.120 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:27:45.127 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:27:45.127 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:27:45.127 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:27:45.127 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:27:45.162 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:27:45.170 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:27:45.170 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:27:45.171 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:27:45.171 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:27:45.171 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:27:45.181 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:27:45.201 WARN 21062 --- [ocp-async-2,e9b27a47ff28f0b6,a20bda26a5054bf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:45.256 INFO 21062 --- [ocp-async-9,661443ad8149f116,1bfced732f55f463] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:45.707 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:27:46.327 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.199934154S 2024-11-04 14:27:46.327 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:27:46.425 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:27:46.425 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:27:46.452 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:27:46.510 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:27:46.510 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:27:46.510 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:27:46.510 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:27:46.510 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:27:46.510 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:27:46.523 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:27:46.701 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] o.s.boot.SpringApplication : Started application in 1.581 seconds (JVM running for 7080.38) 2024-11-04 14:27:46.701 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.581657714S 2024-11-04 14:27:46.854 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:46.855 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:47.106 INFO 21062 --- [ocp-schedules-21,5a3e754cd6b26ef8,f4838367a2cb3841] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:27:47.201 WARN 21062 --- [ocp-async-9,e6276a4a4d44f8bb,bd6d2c1ab9b5f94d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:47.256 INFO 21062 --- [ocp-async-3,67125332eeb9b6d0,ff42475d8df6a033] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:47.983 INFO 21062 --- [prometheus-minute-schedule-1,a84dd13a4ea0ea45,43552997f66311aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:27:48.590878+08:00, lastProcessTime=2024-11-04T14:27:38.590120+08:00 2024-11-04 14:27:48.663 INFO 21062 --- [ocp-async-4,789bc02ac34b6bb7,6e341351f3efb662] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:27:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:27:48.590878+08:00 2024-11-04 14:27:49.041 INFO 21062 --- [ocp-schedules-9,b42aeb56a90511d0,8b08468d1d1e914b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:27:49.045 INFO 21062 --- [ocp-schedules-9,b42aeb56a90511d0,8b08468d1d1e914b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:49.046 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:27:49.201 WARN 21062 --- [ocp-async-6,91d77381b1fb6dec,919151893ecf8ef3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:49.256 INFO 21062 --- [ocp-async-2,290592257934cb16,7306c754d848d9af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:49.855 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:49.857 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:50.578 INFO 21062 --- [ocp-schedules-13,8ce6308b5219fc92,5a892f026a4472ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:27:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8cd123b74657f569,c1cd2f819988d9b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:51.201 WARN 21062 --- [ocp-async-2,ba9a0eac3ae3740a,fada7283578b2812] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:51.256 INFO 21062 --- [ocp-async-9,9be8d26679f28ddc,45770305635d9f2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:51.612 INFO 21062 --- [ocp-schedules-11,66bb8f604d7c6b75,c3fa171288b7ad4d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:27:51.617 INFO 21062 --- [ocp-schedules-11,66bb8f604d7c6b75,c3fa171288b7ad4d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:27:51.617 INFO 21062 --- [ocp-schedules-11,66bb8f604d7c6b75,c3fa171288b7ad4d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:27:52.857 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:52.858 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:53.201 WARN 21062 --- [ocp-async-9,fa3e5a4351f6123f,b889af657e141ef4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:53.257 INFO 21062 --- [ocp-async-3,b784434947e5b8d7,a208ec1ee814f929] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:53.983 INFO 21062 --- [prometheus-minute-schedule-1,b996351ce37c388d,8525251bea062ea3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:54.021 INFO 21062 --- [ocp-schedules-27,b448b13d733de559,785023c1e84ecb4a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:27:55.201 WARN 21062 --- [ocp-async-3,a0adeaff3d317616,233be466ad530005] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:55.257 INFO 21062 --- [ocp-async-1,5e2cdad5b54682a9,13473698196d9679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:55.858 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:55.859 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:56.983 INFO 21062 --- [prometheus-minute-schedule-1,4f51a5d892ef4565,a3a4810384db43d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:27:57.202 WARN 21062 --- [ocp-async-1,ba6684b77f6b277e,c6d3730d47d852f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:57.257 INFO 21062 --- [ocp-async-8,65f255e3147d1b5a,e0b565d5f2f842af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:58.500 INFO 21062 --- [ocp-schedules-2,12d365784fdcca65,caf190c7758f910c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:27:58.500 INFO 21062 --- [ocp-schedules-17,e7586e79582d84b9,f22d74023b7a15db] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:27:58.502 INFO 21062 --- [ocp-schedules-29,2449247ad71ce142,c5e7a895c539f921] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:27:58.504 INFO 21062 --- [ocp-schedules-2,12d365784fdcca65,caf190c7758f910c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:58.504 INFO 21062 --- [ocp-schedules-17,e7586e79582d84b9,f22d74023b7a15db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:27:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:27:58.505 INFO 21062 --- [ocp-schedules-28,84ccb5edbbce9acf,c0c6494df2c220f5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:27:58.510 INFO 21062 --- [ocp-schedules-10,cac33413be4929ee,7c04e16825eff5ad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:27:58.510 INFO 21062 --- [ocp-schedules-22,1ef3d7d473c728b6,a8d0419b08c60631] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:27:58.510 INFO 21062 --- [ocp-schedules-10,cac33413be4929ee,7c04e16825eff5ad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:27:58.510 INFO 21062 --- [ocp-schedules-22,1ef3d7d473c728b6,a8d0419b08c60631] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:27:58.510 INFO 21062 --- [ocp-schedules-23,b35658528f24d130,718d4c9892ed3ab7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:27:58.510 INFO 21062 --- [ocp-schedules-23,b35658528f24d130,718d4c9892ed3ab7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:27:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:27:58.590312+08:00, lastProcessTime=2024-11-04T14:27:48.590878+08:00 2024-11-04 14:27:58.663 INFO 21062 --- [ocp-async-6,e34f46d5d2055a84,4d9c697e5cb988a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:27:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:27:58.590312+08:00 2024-11-04 14:27:58.859 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:27:58.860 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:27:59.046 INFO 21062 --- [ocp-schedules-9,b42aeb56a90511d0,8b08468d1d1e914b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:27:59.192 INFO 21062 --- [ocp-schedules-16,7a61d31fa05106b2,22489c1b6a47c3da] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:27:59.202 WARN 21062 --- [ocp-async-9,6f1445519dc46dd3,19ea1c5df91e36ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:27:59.257 INFO 21062 --- [ocp-async-3,abe5ab3b397335fa,7f1c29a4c298d5dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:27:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2247b551b1fe1675,24c90e313691c880] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701680, interval=60 2024-11-04 14:28:00.580 INFO 21062 --- [ocp-schedules-18,85f82327e47a9d3c,39ad5aa7ef727be8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:28:01.202 WARN 21062 --- [ocp-async-3,7a4c30a6d1b9ee2e,6b054ebf77ef77be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:01.257 INFO 21062 --- [ocp-async-1,d4ce05246ebe63ca,7071d516c88d1a52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:01.860 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:01.862 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:02.983 INFO 21062 --- [prometheus-minute-schedule-1,e247d2c5c7d751c1,a3b996b9b3c1b7c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:03.202 WARN 21062 --- [ocp-async-1,346a102b7c9f859e,25e6c20544411649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:03.257 INFO 21062 --- [ocp-async-8,1cc2f0d817ba53e8,17e162bc34acfde6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:04.062 INFO 21062 --- [ocp-schedules-15,debe9c05eff5519a,a5e235caf6fc4eec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:28:04.862 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:04.863 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:05.202 WARN 21062 --- [ocp-async-8,a59a421fd5576865,9edd479de3d95200] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:05.258 INFO 21062 --- [ocp-async-5,e3657a5fd2550a19,feca86e6f81a9894] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:05.983 INFO 21062 --- [prometheus-minute-schedule-1,9d2c24b502a965cb,23e68fdc3bfdbfb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:06.434 INFO 21062 --- [ocp-schedules-12,84aef6d082fcc052,02665a921dac05b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:28:06.435 INFO 21062 --- [ocp-schedules-12,9c76a62092478aa0,83b5a8d384024285] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:28:06.439 INFO 21062 --- [ocp-schedules-12,9c76a62092478aa0,83b5a8d384024285] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:06.469 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:28:06.469 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:28:07.202 WARN 21062 --- [ocp-async-5,24aa9edda88e46ac,fe2a500b5038a3b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:07.258 INFO 21062 --- [ocp-async-6,8b2e349dd0ace13a,8ce4ecc454963439] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:07.864 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:07.865 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:08.505 INFO 21062 --- [ocp-schedules-17,e7586e79582d84b9,f22d74023b7a15db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:28:08.505 INFO 21062 --- [ocp-schedules-2,12d365784fdcca65,caf190c7758f910c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:28:08.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:28:08.592860+08:00, lastProcessTime=2024-11-04T14:27:58.590312+08:00 2024-11-04 14:28:08.664 INFO 21062 --- [ocp-async-8,369afb346a8d94e5,0a68046d181b37ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:28:08.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:28:08.592860+08:00 2024-11-04 14:28:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ba3a0c341efb2c33,a202aba715770a7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:09.203 WARN 21062 --- [ocp-async-1,9048941bbb994c40,c05967f6b86f99dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:09.258 INFO 21062 --- [ocp-async-4,4197cbc77fac35a5,bad60079439c54b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:28:10.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:28:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:28:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:28:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:28:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:28:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:28:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:28:10.583 INFO 21062 --- [ocp-schedules-19,ad25f0220110cec7,7e929d3457f9b034] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:28:10.865 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:10.866 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:11.203 WARN 21062 --- [ocp-async-4,eca8e45c5fb392f4,51972e1d955eeef1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:11.258 INFO 21062 --- [ocp-async-10,6d29bad02cf2db0c,4e1a63728cdf61a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=22, affectRows=22 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=15, affectRows=15 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=5, affectRows=5 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=9, affectRows=9 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=5, validCount=5 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=17, affectRows=17 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=9, validCount=9 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=15, validCount=15 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=27, affectRows=27 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=17, validCount=17 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=22, validCount=22 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=21, affectRows=21 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=32, affectRows=32 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=21, validCount=21 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=27, validCount=27 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=31, validCount=31 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=47, affectRows=47 2024-11-04 14:28:11.578 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=73, affectRows=73 2024-11-04 14:28:11.579 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=47, validCount=47 2024-11-04 14:28:11.579 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=71, validCount=71 2024-11-04 14:28:11.579 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=59, affectRows=59 2024-11-04 14:28:11.579 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=59, validCount=59 2024-11-04 14:28:11.580 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701620, nodeCount=69, affectRows=69 2024-11-04 14:28:11.580 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701620, nodeCount=69, validCount=69 2024-11-04 14:28:11.619 INFO 21062 --- [ocp-schedules-28,e1ce863181beb1d8,8c82cfee6665c880] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:28:11.624 INFO 21062 --- [ocp-schedules-28,e1ce863181beb1d8,8c82cfee6665c880] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:28:11.624 INFO 21062 --- [ocp-schedules-28,e1ce863181beb1d8,8c82cfee6665c880] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:28:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9357c30792627c00,fd1de049db1217da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:13.203 WARN 21062 --- [ocp-async-10,513c3d5d6cf02d26,daafbbd58623ba9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:13.258 INFO 21062 --- [ocp-async-7,8db5a4c736a12a45,54fe22207c734163] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:13.866 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:13.868 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:13.984 INFO 21062 --- [prometheus-cache-monitor1,5148edb6cf2f7477,30f6b8e300447821] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:28:13.984 INFO 21062 --- [prometheus-cache-monitor1,5148edb6cf2f7477,30f6b8e300447821] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:28:14.087 INFO 21062 --- [ocp-schedules-25,71703cc9c759ef3a,f89ce18f1fab1669] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:28:14.270 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:22:14.244, end=2024-11-04T14:27:14.267} 2024-11-04 14:28:14.270 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:22:14.244, end=2024-11-04T14:27:14.267} 2024-11-04 14:28:14.280 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:28:14.280 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:28:14.280 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:28:14.280 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:28:14.983 INFO 21062 --- [prometheus-minute-schedule-1,daf22fa5e1329dff,aff819745673d96d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:15.203 WARN 21062 --- [ocp-async-6,b7265b526dd365cb,9a851d6ec56902e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:15.258 INFO 21062 --- [ocp-async-2,81ff8b1ab683f615,836beb293d1c492f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:16.868 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:16.870 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:17.203 WARN 21062 --- [ocp-async-9,162c16c6e3993b3e,fc3b30edccb83234] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:17.259 INFO 21062 --- [ocp-async-8,8117cab933cd10da,36f2425656f25832] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:17.983 INFO 21062 --- [prometheus-minute-schedule-1,3b67693e6b1283e1,4036d96b6daa6774] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:28:18.590350+08:00, lastProcessTime=2024-11-04T14:28:08.592860+08:00 2024-11-04 14:28:18.664 INFO 21062 --- [ocp-async-6,dc8de98493e2e9b2,d2c262123c6d8d20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:28:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:28:18.590350+08:00 2024-11-04 14:28:19.203 WARN 21062 --- [ocp-async-10,8c04d81e07d4e1a6,c25d8e9ed7ab1361] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:19.259 INFO 21062 --- [ocp-async-7,a031f7af22ade57f,37212b2222f01334] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:19.870 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:19.871 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:20.585 INFO 21062 --- [ocp-schedules-10,8944e41255dd977b,619c8f5da207abb3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:28:20.983 INFO 21062 --- [prometheus-minute-schedule-1,53ed4f862af8b19d,505f1b63372a3462] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:21.204 WARN 21062 --- [ocp-async-6,658b4fd4ef28503a,6ffeeeaf307a584c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:21.259 INFO 21062 --- [ocp-async-2,23cbc3897e74f9a7,6518ae670a41b1ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:22.871 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:22.872 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:23.204 WARN 21062 --- [ocp-async-9,0f26243b48e48dc9,f1154c70ef2dffec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:23.259 INFO 21062 --- [ocp-async-8,c416aacfdce3aed5,f97995fe9bac9293] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:23.983 INFO 21062 --- [prometheus-minute-schedule-1,2b9e1dbab52eb74b,cb92decaf089cca5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:24.160 INFO 21062 --- [ocp-schedules-13,83fc56e4e680b10a,ce0381172d12e16c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:28:25.204 WARN 21062 --- [ocp-async-5,9fffd6e600d6bab9,ebb0973c87b7f1ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:25.259 INFO 21062 --- [ocp-async-3,7e3de7c47bae1499,522f756936ed8c21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:25.873 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:25.874 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ba33e34472b69aff,f00cb439232817fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2bbd0d34df986c74,f2dc48941ecb0161] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:28:27.204 WARN 21062 --- [ocp-async-7,150f4f1a9d01bb22,117810cc90c9b0e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:27.259 INFO 21062 --- [ocp-async-1,206ab8c34cd26f05,d4433e67153d2ab6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:28.498 INFO 21062 --- [ocp-schedules-19,ec2b6ae5f6d31688,622c846691178350] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:28:28.499 INFO 21062 --- [ocp-schedules-5,e85296f49d08a260,9fa26f780ed38597] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:28:28.500 INFO 21062 --- [ocp-schedules-20,4294a368a81dc3de,eeb44092bba93a23] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:28:28.500 INFO 21062 --- [ocp-schedules-25,d03d70baef2af644,4932663637fa7e1a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:28:28.502 INFO 21062 --- [ocp-schedules-24,138c6ea509d443ee,7a584a814bf3f151] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:28:28.503 INFO 21062 --- [ocp-schedules-20,4294a368a81dc3de,eeb44092bba93a23] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:28.504 INFO 21062 --- [ocp-schedules-19,ec2b6ae5f6d31688,622c846691178350] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:28:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:28:28.504 INFO 21062 --- [ocp-schedules-25,d03d70baef2af644,4932663637fa7e1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:28.504 INFO 21062 --- [ocp-schedules-4,b761d79dea20e95c,5c4bbe2678cbfc80] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:28:28.506 INFO 21062 --- [ocp-schedules-5,e85296f49d08a260,9fa26f780ed38597] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:28:28.510 INFO 21062 --- [ocp-schedules-29,9130aca531a95db6,7c4d67db0a46211f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:28:28.511 INFO 21062 --- [ocp-schedules-27,bdd9edfb204cedb8,c7ca31e765d2332c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:28:28.511 INFO 21062 --- [ocp-schedules-30,7a07d3d0e1046d7b,f26d0d5c27fc2c41] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:28:28.511 INFO 21062 --- [ocp-schedules-29,9130aca531a95db6,7c4d67db0a46211f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:28:28.511 INFO 21062 --- [ocp-schedules-30,7a07d3d0e1046d7b,f26d0d5c27fc2c41] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:28:28.511 INFO 21062 --- [ocp-schedules-27,bdd9edfb204cedb8,c7ca31e765d2332c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:28:28.577 INFO 21062 --- [ocp-schedules-6,cfd2bcf529043cc3,ba0be776cf9e9ccd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:28:28.577 INFO 21062 --- [ocp-schedules-6,cfd2bcf529043cc3,ba0be776cf9e9ccd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:28:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:28:28.590765+08:00, lastProcessTime=2024-11-04T14:28:18.590350+08:00 2024-11-04 14:28:28.664 INFO 21062 --- [ocp-async-3,b60ed0eee258f1f7,8e0c271ac3eb9c84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:28:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:28:28.590765+08:00 2024-11-04 14:28:28.842 INFO 21062 --- [ocp-schedules-28,89ae9e05dbc2f00f,20fc3ea45bafcdfa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:28:28.874 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:28.875 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:29.118 INFO 21062 --- [ocp-schedules-4,da4ea00bdd73ac4f,2c0375822eb8fe74] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:28:29.118 INFO 21062 --- [ocp-schedules-4,da4ea00bdd73ac4f,2c0375822eb8fe74] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:28:29.194 INFO 21062 --- [ocp-schedules-23,8bae20f8ab8e466c,fb1650207cff0235] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:28:29.204 WARN 21062 --- [ocp-async-5,fb10083c7175dd2e,4ef13db6514933aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:29.224 INFO 21062 --- [ocp-schedules-6,0c33ba789bd1e6e3,7dda4ad89fd3d8cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:28:29.229 INFO 21062 --- [ocp-schedules-6,0c33ba789bd1e6e3,7dda4ad89fd3d8cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:28:29.229 INFO 21062 --- [ocp-schedules-6,0c33ba789bd1e6e3,7dda4ad89fd3d8cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:28:29.260 INFO 21062 --- [ocp-async-10,de4a5ba1e297667a,1a38f5f367de4d34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:29.384 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:28:29.384 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:28:29.393 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:28:29.393 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:28:29.393 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:28:29.405 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:28:29.405 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:28:29.405 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:28:29.409 INFO 21062 --- [ocp-schedules-8,b1e2357b2ea79a93,11cb45d66598e237] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:28:29.560 INFO 21062 --- [ocp-schedules-11,b5c3a8a660534901,daec5a6e46db1e21] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:28:29.560 INFO 21062 --- [ocp-schedules-11,b5c3a8a660534901,daec5a6e46db1e21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:28:29.566 INFO 21062 --- [ocp-schedules-11,b5c3a8a660534901,daec5a6e46db1e21] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:28:29.566 INFO 21062 --- [ocp-schedules-11,b5c3a8a660534901,daec5a6e46db1e21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:28:29.571 INFO 21062 --- [ocp-schedules-11,b5c3a8a660534901,daec5a6e46db1e21] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:28:29.571 INFO 21062 --- [ocp-schedules-11,b5c3a8a660534901,daec5a6e46db1e21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:28:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7bc7044d36fe9893,b47ea10d7ecede68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:28:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:28:30.590 INFO 21062 --- [ocp-schedules-9,532327e985903545,f95987604659dadc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:28:31.204 WARN 21062 --- [ocp-async-10,65bc98c9b729b41e,44f416d559edbd26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:31.260 INFO 21062 --- [ocp-async-6,88b473f90fc1737a,8100028e6e8699aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:31.626 INFO 21062 --- [ocp-schedules-19,40b528db601ebd47,5074cf468276935e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:28:31.632 INFO 21062 --- [ocp-schedules-19,40b528db601ebd47,5074cf468276935e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:28:31.632 INFO 21062 --- [ocp-schedules-19,40b528db601ebd47,5074cf468276935e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:28:31.689 INFO 21062 --- [ocp-schedules-1,54a69ff3b7a94466,443bf9ea8b0cff27] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:28:31.689 INFO 21062 --- [ocp-schedules-1,54a69ff3b7a94466,443bf9ea8b0cff27] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:28:31.716 INFO 21062 --- [ocp-schedules-1,54a69ff3b7a94466,443bf9ea8b0cff27] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:28:31.716 INFO 21062 --- [ocp-schedules-1,54a69ff3b7a94466,443bf9ea8b0cff27] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:28:31.718 INFO 21062 --- [ocp-schedules-14,7197018eca2f5f3a,e76eb34e209ca0e9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:28:31.875 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:31.876 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:31.903 INFO 21062 --- [ocp-schedules-18,7ed498f8c093463b,c43a064d533ad043] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:28:31.903+08:00, clusters=[1] 2024-11-04 14:28:31.905 INFO 21062 --- [ocp-schedules-18,7ed498f8c093463b,c43a064d533ad043] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:28:32.069 INFO 21062 --- [ocp-schedules-17,f39367dc7df9d75e,6c71ebc0b20c9c05] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:28:32.073 INFO 21062 --- [ocp-schedules-17,f39367dc7df9d75e,6c71ebc0b20c9c05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:32.077 INFO 21062 --- [ocp-schedules-10,bbefa9a257611325,e57ee9d118a6e1ab] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:28:32.077+08:00, clusters=[1] 2024-11-04 14:28:32.080 INFO 21062 --- [ocp-schedules-10,bbefa9a257611325,e57ee9d118a6e1ab] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:28:32.137 ERROR 21062 --- [ocp-schedules-17,f39367dc7df9d75e,6c71ebc0b20c9c05] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:28:32.137 INFO 21062 --- [ocp-schedules-17,f39367dc7df9d75e,6c71ebc0b20c9c05] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:28:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d7c1f89feb8cc729,948735c1af020c1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:33.121 INFO 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:28:33.122 INFO 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:28:33.122 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:28:33.122 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:28:33.132 INFO 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:28:33.132 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:28:33.132 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:28:33.133 WARN 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:28:33.134 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:28:33.134 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:28:33.134 INFO 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:28:33.134 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:28:33.134 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:28:33.135 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:28:33.135 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:28:33.135 WARN 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:28:33.142 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:28:33.143 INFO 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:28:33.143 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:28:33.154 ERROR 21062 --- [ocp-schedules-18,0110e3152507c1fe,7f52eea369abc839] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:28:33.205 WARN 21062 --- [ocp-async-6,bb8058313ffeafe4,1b3e10c3dc3b51f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:33.260 INFO 21062 --- [ocp-async-9,4d840179c08f82cc,111741a429a327ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:34.185 INFO 21062 --- [ocp-schedules-11,080a712368c20d86,35745d21983d18f9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:28:34.876 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:34.878 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:35.205 WARN 21062 --- [ocp-async-9,a27d9ef4c4cf27d7,d7575774065d1376] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:35.260 INFO 21062 --- [ocp-async-4,ac072d3f9d2881d4,05b2f015501477d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:35.983 INFO 21062 --- [prometheus-minute-schedule-1,50683cdd69435e78,6dded93e49a2a25a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:36.469 INFO 21062 --- [ocp-schedules-12,9c76a62092478aa0,83b5a8d384024285] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:28:36.470 INFO 21062 --- [ocp-schedules-12,f7d0f0b6cedaa680,81e28a2ad950c8bd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:28:36.473 INFO 21062 --- [ocp-schedules-12,f7d0f0b6cedaa680,81e28a2ad950c8bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:36.502 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:28:36.502 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:28:37.205 WARN 21062 --- [ocp-async-4,338b2678fb1aa3ac,fe06ffb19c019018] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:37.260 INFO 21062 --- [ocp-async-7,e8b3635e60897bcf,915a539b5f4429df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:37.878 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:37.879 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:38.504 INFO 21062 --- [ocp-schedules-20,4294a368a81dc3de,eeb44092bba93a23] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:28:38.504 INFO 21062 --- [ocp-schedules-25,d03d70baef2af644,4932663637fa7e1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:28:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:28:38.589659+08:00, lastProcessTime=2024-11-04T14:28:28.590765+08:00 2024-11-04 14:28:38.664 INFO 21062 --- [ocp-async-8,72c0023a51dc45c7,232c12a41c257a06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:28:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:28:38.589659+08:00 2024-11-04 14:28:38.983 INFO 21062 --- [prometheus-minute-schedule-1,89d170cde3d9e4f4,d22e8eb49ef58fb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:39.205 WARN 21062 --- [ocp-async-1,22139039b6dd3ebe,58543c16df0a5a53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:39.232 INFO 21062 --- [ocp-schedules-4,d1e422f8f86cafe4,312a7834d6a815af] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:28:39.236 INFO 21062 --- [ocp-schedules-4,d1e422f8f86cafe4,312a7834d6a815af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:39.260 INFO 21062 --- [ocp-async-5,af96299cc1c63804,983a245edd2fa779] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:39.317 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:28:39.319 ERROR 21062 --- [ocp-schedules-4,d1e422f8f86cafe4,312a7834d6a815af] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:28:39.319 INFO 21062 --- [ocp-schedules-4,d1e422f8f86cafe4,312a7834d6a815af] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:28:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:28:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:28:40.594 INFO 21062 --- [ocp-schedules-25,d5663a9e7e19830b,f43745c512ae22dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:28:40.879 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:40.880 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:41.206 WARN 21062 --- [ocp-async-5,5bd6d068f39e08ad,ecfbf253d3d513b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:41.261 INFO 21062 --- [ocp-async-10,f641f636abe5ed24,fdd49781b6866291] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:28:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:28:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:28:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:28:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:28:41.854 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:28:41.854 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:28:41.983 INFO 21062 --- [prometheus-minute-schedule-1,9559f0fb092be693,7ef892a6b78737e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:43.206 WARN 21062 --- [ocp-async-10,8a5d62c01d72d4b8,b66e124e87ed8dcf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:43.261 INFO 21062 --- [ocp-async-6,511137a41f7dbe68,b54d2dd7b096af7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:43.880 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:43.881 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:44.224 INFO 21062 --- [ocp-schedules-2,3792bca90b0a2917,03a0f5ecae674afe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:28:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9b3473bb3c1c9bc0,6d999ad4dfc7845b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:45.206 WARN 21062 --- [ocp-async-6,2520e8a195c286f0,e0528709e2a77bba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:45.261 INFO 21062 --- [ocp-async-9,9c04b026b1452a14,fbf7ae7c08d2d5d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:46.881 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:46.883 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:47.206 WARN 21062 --- [ocp-async-9,809e8da7487f35ff,e960a6675259e575] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:47.261 INFO 21062 --- [ocp-async-4,b7e6228d2bf0d354,29e5252b7a2ada79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:47.983 INFO 21062 --- [prometheus-minute-schedule-1,036d9c8aa1cadecd,7832b67a252539a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:28:48.591387+08:00, lastProcessTime=2024-11-04T14:28:38.589659+08:00 2024-11-04 14:28:48.664 INFO 21062 --- [ocp-async-1,dde5a1c67d3059b1,68cfa97a435fc7c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:28:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:28:48.591387+08:00 2024-11-04 14:28:49.206 WARN 21062 --- [ocp-async-10,3ae31f7f2b080a43,ee8cbbfc54150b58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:49.261 INFO 21062 --- [ocp-async-6,c1dc54d1f38f3f70,986571cc38661801] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:49.883 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:49.884 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:50.597 INFO 21062 --- [ocp-schedules-28,c04cd24f2affd2d0,eb04fa75f2f1f378] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:28:50.983 INFO 21062 --- [prometheus-minute-schedule-1,96a22d9d541856a8,1557cb298e7b8142] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:51.207 WARN 21062 --- [ocp-async-6,edc9d1b27adb1b1f,f24255ef65eba891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:51.262 INFO 21062 --- [ocp-async-9,088f00e0165da1e1,2ff35274a6a41e27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:51.635 INFO 21062 --- [ocp-schedules-5,f50078aee37716d2,4d666063d217b099] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:28:51.641 INFO 21062 --- [ocp-schedules-5,f50078aee37716d2,4d666063d217b099] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:28:51.641 INFO 21062 --- [ocp-schedules-5,f50078aee37716d2,4d666063d217b099] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:28:52.884 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:52.885 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:53.207 WARN 21062 --- [ocp-async-9,448533ff484d4cc7,e7274ff33ba6229b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:53.262 INFO 21062 --- [ocp-async-4,c1e9b938e40ab4da,1b93ea35388a3bee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6065014d535ab9af,4d46b5eadd89551b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:54.249 INFO 21062 --- [ocp-schedules-1,85860922100d8c42,e7f16a9f0cfc3024] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:28:55.207 WARN 21062 --- [ocp-async-4,f394eb1bde604143,e5c96cc45ed0aeda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:55.262 INFO 21062 --- [ocp-async-7,8b038cc41826fee3,5581d279775b595e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:55.886 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:55.887 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fa333c1042414551,a6da29c55ab03bb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:28:57.207 WARN 21062 --- [ocp-async-7,4c2f9be7e1d494f7,c30dc273ca3fb86e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:57.262 INFO 21062 --- [ocp-async-2,6f0024c32578816b,9b473693054023b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:58.500 INFO 21062 --- [ocp-schedules-8,b821a3426af76a9a,3cd1f7d99df5c700] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:28:58.500 INFO 21062 --- [ocp-schedules-10,b755db931c1a5b54,2897655aa52a20d4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:28:58.502 INFO 21062 --- [ocp-schedules-4,9380b7cd55c70624,24c50076f4c9ba2e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:28:58.504 INFO 21062 --- [ocp-schedules-8,b821a3426af76a9a,3cd1f7d99df5c700] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:58.504 INFO 21062 --- [ocp-schedules-10,b755db931c1a5b54,2897655aa52a20d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:28:58.504 INFO 21062 --- [ocp-schedules-19,c7b14a53010c6305,dfb03767d3ef3dde] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:28:58.511 INFO 21062 --- [ocp-schedules-11,0beb840649655ac3,2db487475eb0f504] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:28:58.511 INFO 21062 --- [ocp-schedules-18,33f9eeb851606e0c,eafa6cd82451ef06] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:28:58.511 INFO 21062 --- [ocp-schedules-11,0beb840649655ac3,2db487475eb0f504] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:28:58.511 INFO 21062 --- [ocp-schedules-18,33f9eeb851606e0c,eafa6cd82451ef06] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:28:58.512 INFO 21062 --- [ocp-schedules-3,786ba4a14482d0da,440b7ef6c48650e8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:28:58.512 INFO 21062 --- [ocp-schedules-3,786ba4a14482d0da,440b7ef6c48650e8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:28:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:28:58.589295+08:00, lastProcessTime=2024-11-04T14:28:48.591387+08:00 2024-11-04 14:28:58.664 INFO 21062 --- [ocp-async-10,b9afb2f665fd8cd5,46cbda7960e58566] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:28:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:28:58.589295+08:00 2024-11-04 14:28:58.887 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:28:58.888 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:28:59.046 INFO 21062 --- [ocp-schedules-27,22ef5fd595ae4cee,edaeaf2470b41655] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:28:59.050 INFO 21062 --- [ocp-schedules-27,22ef5fd595ae4cee,edaeaf2470b41655] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:28:59.050 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:28:59.197 INFO 21062 --- [ocp-schedules-7,1d8babab520e4464,179376d0490e04c2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:28:59.207 WARN 21062 --- [ocp-async-9,55da5be455ea9747,264a87190309343d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:28:59.262 INFO 21062 --- [ocp-async-4,daa02d0bec517c5b,80790b9582767b79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:28:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ab77fff9c95a8d16,b4b4fa6c0b5d99b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701740, interval=60 2024-11-04 14:29:00.600 INFO 21062 --- [ocp-schedules-20,9f2d77d0ea2bad74,8d4653c33dfb1848] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:29:01.207 WARN 21062 --- [ocp-async-4,79d3d6f8422a5a2d,c33a48f3e0561374] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:01.263 INFO 21062 --- [ocp-async-7,3bfaf644733e6bf3,3cc23c7fdb38202d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:01.888 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:01.890 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:02.983 INFO 21062 --- [prometheus-minute-schedule-1,6a151e8828662795,c055d1bccc3b05e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:03.208 WARN 21062 --- [ocp-async-7,707eb5ddbd812569,6cbc41d80e48a573] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:03.263 INFO 21062 --- [ocp-async-1,d33cab18f6482437,943628b65ee48296] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:04.292 INFO 21062 --- [ocp-schedules-22,b81e9ec671ff1419,e440d86e3918531d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:29:04.890 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:04.891 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:05.208 WARN 21062 --- [ocp-async-1,aad75755348e86cf,8028aee2709c0d82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:05.263 INFO 21062 --- [ocp-async-5,2ba1c9a1b6af91a9,9e5fb01139cfa4e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:05.983 INFO 21062 --- [prometheus-minute-schedule-1,51378bf76f7f2227,52eea5d1c844c98e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:06.503 INFO 21062 --- [ocp-schedules-12,f7d0f0b6cedaa680,81e28a2ad950c8bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:29:06.503 INFO 21062 --- [ocp-schedules-12,4ec1187cdd249735,1a33547271c66501] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:29:06.508 INFO 21062 --- [ocp-schedules-12,4ec1187cdd249735,1a33547271c66501] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:29:06.537 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:29:06.537 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:29:07.208 WARN 21062 --- [ocp-async-5,fbf9d9b1c8a65da0,47dda5b3ca45d775] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:07.263 INFO 21062 --- [ocp-async-10,01e0af57bf51637f,8c7f63c7be0e6f9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:07.891 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:07.892 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:08.505 INFO 21062 --- [ocp-schedules-10,b755db931c1a5b54,2897655aa52a20d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:29:08.505 INFO 21062 --- [ocp-schedules-8,b821a3426af76a9a,3cd1f7d99df5c700] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:29:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:29:08.590134+08:00, lastProcessTime=2024-11-04T14:28:58.589295+08:00 2024-11-04 14:29:08.665 INFO 21062 --- [ocp-async-1,f975c8861cfe1e10,2cc0e3a4ef27addb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:29:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:29:08.590134+08:00 2024-11-04 14:29:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a8fcbdd2ef38e76b,281604bdf3813f3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:09.051 INFO 21062 --- [ocp-schedules-27,22ef5fd595ae4cee,edaeaf2470b41655] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:29:09.208 WARN 21062 --- [ocp-async-7,9f74e8b4b755ade6,c0a0b62b481d61f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:09.263 INFO 21062 --- [ocp-async-2,d6791e9ba359e930,15a28c4cc3203044] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:29:10.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:29:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:29:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:29:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:29:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:29:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:29:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:29:10.604 INFO 21062 --- [ocp-schedules-26,c3f782a4f5bbce61,9c5a969e1ecac351] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:29:10.892 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:10.893 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:11.208 WARN 21062 --- [ocp-async-2,1cbfe328bf66cada,ad96e537f044ff8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:11.263 INFO 21062 --- [ocp-async-8,5e7e788091f878f4,85ee4dc743b1262b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:11.583 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701680, nodeCount=57, affectRows=57 2024-11-04 14:29:11.583 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701680, nodeCount=80, affectRows=80 2024-11-04 14:29:11.583 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701680, nodeCount=57, validCount=57 2024-11-04 14:29:11.584 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701680, nodeCount=53, affectRows=53 2024-11-04 14:29:11.584 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701680, nodeCount=103, affectRows=103 2024-11-04 14:29:11.584 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701680, nodeCount=80, validCount=80 2024-11-04 14:29:11.584 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701680, nodeCount=103, affectRows=103 2024-11-04 14:29:11.584 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701680, nodeCount=53, validCount=53 2024-11-04 14:29:11.584 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701680, nodeCount=101, validCount=101 2024-11-04 14:29:11.584 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701680, nodeCount=102, validCount=102 2024-11-04 14:29:11.643 INFO 21062 --- [ocp-schedules-11,0ac8b48f6a4379f1,f78f547ef6793a8b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:29:11.648 INFO 21062 --- [ocp-schedules-11,0ac8b48f6a4379f1,f78f547ef6793a8b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:29:11.648 INFO 21062 --- [ocp-schedules-11,0ac8b48f6a4379f1,f78f547ef6793a8b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:29:11.983 INFO 21062 --- [prometheus-minute-schedule-1,3bb2926fa5ebfd07,71451cd580979ce9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:13.208 WARN 21062 --- [ocp-async-5,2763dc4675842999,eb9dcc50fee5662d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:13.264 INFO 21062 --- [ocp-async-3,23ff15c189ede590,3b37a5c1270a7831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:13.893 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:13.895 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:13.984 INFO 21062 --- [prometheus-cache-monitor1,58530d6245aaff84,234709add205d98a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:29:13.984 INFO 21062 --- [prometheus-cache-monitor1,58530d6245aaff84,234709add205d98a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:29:14.319 INFO 21062 --- [ocp-schedules-20,ad2f9ea3342ed7e8,763c98d2712edd94] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:29:14.983 INFO 21062 --- [prometheus-minute-schedule-1,4f84e72e4951609e,898a83e32cafc0cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:15.097 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:25:15.079, end=2024-11-04T14:27:15.094} 2024-11-04 14:29:15.101 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:29:15.101 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:29:15.208 WARN 21062 --- [ocp-async-10,958b4f470dde7137,1ea29b11a3f512e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:15.264 INFO 21062 --- [ocp-async-6,e6355649981c3d28,a400d06b8d28b770] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:16.895 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:16.896 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:17.209 WARN 21062 --- [ocp-async-9,83e1c8eec7bb8a40,ec28b8d96f318821] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:17.264 INFO 21062 --- [ocp-async-1,c04ceb9a504bc2e1,6a56a3dcde9bc6ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:17.983 INFO 21062 --- [prometheus-minute-schedule-1,acf708119b0f9823,c8180a4adbe616f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:18.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:29:18.592024+08:00, lastProcessTime=2024-11-04T14:29:08.590134+08:00 2024-11-04 14:29:18.665 INFO 21062 --- [ocp-async-10,f7161d3282c16f66,136b572ad8eb7457] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:29:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:29:18.592024+08:00 2024-11-04 14:29:19.209 WARN 21062 --- [ocp-async-5,9431adaa3324b733,2b16c90f857fe3ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:19.264 INFO 21062 --- [ocp-async-3,12c1b35951226236,ffa181e142203a07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:19.896 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:19.897 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:20.606 INFO 21062 --- [ocp-schedules-3,1f79a595664ee1f9,fad32917d1ffcaf9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:29:20.983 INFO 21062 --- [prometheus-minute-schedule-1,1547c9e9b9dc0310,ee96ed5c960a62cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:21.209 WARN 21062 --- [ocp-async-10,5baf20c6f52ed4e7,02029ac38b98fe71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:21.264 INFO 21062 --- [ocp-async-6,7d28382fbaaf9b14,4705bbeab6a9edfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:22.897 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:22.898 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:23.209 WARN 21062 --- [ocp-async-9,086614480151bf8c,06a14fac48128572] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:23.265 INFO 21062 --- [ocp-async-1,dbe47efc2391bdcf,d17dec8c60e1e0f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ed7bf96ae0cfcc66,6785078a2b3a8b4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:24.364 INFO 21062 --- [ocp-schedules-10,7dc80c8838788efb,c9f8130af474162a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:29:25.209 WARN 21062 --- [ocp-async-8,27c532e6d1a11621,377692dde61a6c73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:25.265 INFO 21062 --- [ocp-async-4,6dd2168f06f1bf27,6acffa16a1561513] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:25.898 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:25.899 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ce2bc3a1fe9a049f,2e6d44d0bf6334cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f69c3271f3bd35f5,58ccfa9f872bb5a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:29:27.209 WARN 21062 --- [ocp-async-3,138ffa75606aa299,7e865036922c2e0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:27.265 INFO 21062 --- [ocp-async-7,0645c046e211ecee,3d2cd35b741b31f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:28.498 INFO 21062 --- [ocp-schedules-9,0c755525287cca97,06e4b715e7518acd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:29:28.500 INFO 21062 --- [ocp-schedules-23,fa0ba9c613ed6308,1f7e339381dcf1f7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:29:28.500 INFO 21062 --- [ocp-schedules-22,88fe8aafd12da1d4,449e94adae3fa1cf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:29:28.500 INFO 21062 --- [ocp-schedules-17,23594d500e0ffa0e,41a521e0f7a3e761] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:29:28.503 INFO 21062 --- [ocp-schedules-30,1f12cf32e9c93064,94b95dbdfb7b2848] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:29:28.503 INFO 21062 --- [ocp-schedules-9,0c755525287cca97,06e4b715e7518acd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:29:28.504 INFO 21062 --- [ocp-schedules-23,fa0ba9c613ed6308,1f7e339381dcf1f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:29:28.504 INFO 21062 --- [ocp-schedules-17,23594d500e0ffa0e,41a521e0f7a3e761] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:29:28.505 INFO 21062 --- [ocp-schedules-15,6f0970cfe50e437c,85a44c4cffd3072d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:29:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:29:28.506 INFO 21062 --- [ocp-schedules-22,88fe8aafd12da1d4,449e94adae3fa1cf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:29:28.511 INFO 21062 --- [ocp-schedules-20,f60b21e1a49bf846,b1916b6719d2b450] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:29:28.511 INFO 21062 --- [ocp-schedules-20,f60b21e1a49bf846,b1916b6719d2b450] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:29:28.511 INFO 21062 --- [ocp-schedules-6,722cf1d15c716bb7,5e98c18a3dfa4823] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:29:28.511 INFO 21062 --- [ocp-schedules-6,722cf1d15c716bb7,5e98c18a3dfa4823] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:29:28.512 INFO 21062 --- [ocp-schedules-10,c12b5b5a3dd8af65,653588170dfbf1c7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:29:28.512 INFO 21062 --- [ocp-schedules-10,c12b5b5a3dd8af65,653588170dfbf1c7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:29:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:29:28.590565+08:00, lastProcessTime=2024-11-04T14:29:18.592024+08:00 2024-11-04 14:29:28.592 INFO 21062 --- [ocp-schedules-3,ff52b8cea4191280,4dacbd904c8708d3] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:29:28.592+08:00, tenantSize=0 2024-11-04 14:29:28.665 INFO 21062 --- [ocp-async-4,ae3c2f9c9898c5dc,20e45e45b029542d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:29:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:29:28.590565+08:00 2024-11-04 14:29:28.844 INFO 21062 --- [ocp-schedules-11,91b7797b7dad5303,bcd9be566fe039f8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:29:28.899 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:28.900 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:29.124 INFO 21062 --- [ocp-schedules-1,9895b8cd5e8fac99,6ab54c28f04aa474] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:29:29.124 INFO 21062 --- [ocp-schedules-1,9895b8cd5e8fac99,6ab54c28f04aa474] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:29:29.200 INFO 21062 --- [ocp-schedules-6,b4611ac23c42b7a5,a0e23989979266f4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:29:29.210 WARN 21062 --- [ocp-async-2,5ae53c1bf0876281,6286586afefcf678] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:29.230 INFO 21062 --- [ocp-schedules-8,3cad6f2ee0c2ac07,b89de0e5092e7c82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:29:29.235 INFO 21062 --- [ocp-schedules-8,3cad6f2ee0c2ac07,b89de0e5092e7c82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:29:29.235 INFO 21062 --- [ocp-schedules-8,3cad6f2ee0c2ac07,b89de0e5092e7c82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:29:29.265 INFO 21062 --- [ocp-async-5,df758d3c65092cda,919f8e992c5d064c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:29.576 INFO 21062 --- [ocp-schedules-29,777b0d420b7f9f63,29b90dfd985fcd1e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:29:29.576 INFO 21062 --- [ocp-schedules-29,777b0d420b7f9f63,29b90dfd985fcd1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:29:29.577 INFO 21062 --- [ocp-schedules-29,777b0d420b7f9f63,29b90dfd985fcd1e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:29:29.577 INFO 21062 --- [ocp-schedules-29,777b0d420b7f9f63,29b90dfd985fcd1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:29:29.578 INFO 21062 --- [ocp-schedules-29,777b0d420b7f9f63,29b90dfd985fcd1e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:29:29.578 INFO 21062 --- [ocp-schedules-29,777b0d420b7f9f63,29b90dfd985fcd1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:29:29.983 INFO 21062 --- [prometheus-minute-schedule-1,bdeb57e093846e14,4db846e77589e13e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:29:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:29:30.609 INFO 21062 --- [ocp-schedules-8,445995f5f3a10754,9209aca304404888] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:29:31.210 WARN 21062 --- [ocp-async-5,80ac413723adef92,f3241dfeb096572f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:31.265 INFO 21062 --- [ocp-async-10,7e7c292c2e7b8e4e,1a4f498ea4a69f70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:31.650 INFO 21062 --- [ocp-schedules-20,3a431ac5060dd782,c281e4d5c921b0ab] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:29:31.657 INFO 21062 --- [ocp-schedules-20,3a431ac5060dd782,c281e4d5c921b0ab] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:29:31.657 INFO 21062 --- [ocp-schedules-20,3a431ac5060dd782,c281e4d5c921b0ab] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:29:31.717 INFO 21062 --- [ocp-schedules-22,c2c20d71e681ce0f,3d03e7bc41e5043a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:29:31.717 INFO 21062 --- [ocp-schedules-22,c2c20d71e681ce0f,3d03e7bc41e5043a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:29:31.744 INFO 21062 --- [ocp-schedules-30,d634642d1f5dce4f,22c0b837919ecba6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:29:31.745 INFO 21062 --- [ocp-schedules-22,c2c20d71e681ce0f,3d03e7bc41e5043a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:29:31.745 INFO 21062 --- [ocp-schedules-22,c2c20d71e681ce0f,3d03e7bc41e5043a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:29:31.900 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:31.901 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:31.931 INFO 21062 --- [ocp-schedules-8,418aee4c173133f4,0fa8041bcc928cb2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:29:31.931+08:00, clusters=[1] 2024-11-04 14:29:31.933 INFO 21062 --- [ocp-schedules-8,418aee4c173133f4,0fa8041bcc928cb2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:29:32.106 INFO 21062 --- [ocp-schedules-19,6784131e8321254a,171984e4ae579665] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:29:32.106+08:00, clusters=[1] 2024-11-04 14:29:32.110 INFO 21062 --- [ocp-schedules-19,6784131e8321254a,171984e4ae579665] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:29:32.983 INFO 21062 --- [prometheus-minute-schedule-1,adbbded403c4c9a8,95227c4e478e8af5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:33.164 INFO 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:29:33.165 INFO 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:29:33.165 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:29:33.165 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:29:33.175 INFO 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:29:33.176 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:29:33.176 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:29:33.177 WARN 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:29:33.177 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:29:33.177 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:29:33.178 INFO 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:29:33.178 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:29:33.178 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:29:33.179 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:29:33.179 WARN 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:29:33.179 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:29:33.187 INFO 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:29:33.187 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:29:33.187 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:29:33.201 ERROR 21062 --- [ocp-schedules-5,7ef492e5ef53c655,71e556f1e56fafc8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:29:33.210 WARN 21062 --- [ocp-async-10,2dc9659b3792b993,840824e6b2acbf5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:33.266 INFO 21062 --- [ocp-async-9,95fe23564b421ffa,4023d9b64692ffb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:34.395 INFO 21062 --- [ocp-schedules-16,72ffb57ff36f5f7b,54a0a70e33550f10] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:29:34.901 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:34.901 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:35.210 WARN 21062 --- [ocp-async-9,98b0bbd383f902ce,229a0f1704a1194e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:35.266 INFO 21062 --- [ocp-async-8,fdacd6b544a61b0e,2edc3d92f99dc7b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4cdf899a4a6a8b62,d77ceb5fbfbb4a29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:36.537 INFO 21062 --- [ocp-schedules-12,4ec1187cdd249735,1a33547271c66501] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:29:36.537 INFO 21062 --- [ocp-schedules-12,c9c611b9443bb307,2421807e23aeed17] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:29:36.541 INFO 21062 --- [ocp-schedules-12,c9c611b9443bb307,2421807e23aeed17] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:29:36.569 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:29:36.569 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:29:37.210 WARN 21062 --- [ocp-async-8,099f5ee0168dad00,9c9b9d9264f36b10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:37.266 INFO 21062 --- [ocp-async-3,e853f724c29d5006,b4ba0e59aae8d047] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:37.902 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:37.902 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:38.505 INFO 21062 --- [ocp-schedules-17,23594d500e0ffa0e,41a521e0f7a3e761] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:29:38.505 INFO 21062 --- [ocp-schedules-23,fa0ba9c613ed6308,1f7e339381dcf1f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:29:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:29:38.589517+08:00, lastProcessTime=2024-11-04T14:29:28.590565+08:00 2024-11-04 14:29:38.665 INFO 21062 --- [ocp-async-1,8d22c47e5e34898f,9f35a1f04dad68ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:29:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:29:38.589517+08:00 2024-11-04 14:29:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e8802a93e15faf05,ae0a422d83785161] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:39.211 WARN 21062 --- [ocp-async-6,1d9b86b1be119d47,b6684b65574823c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:39.266 INFO 21062 --- [ocp-async-9,ee8598c14d3aeb7d,a06e304bedef98b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:39.319 INFO 21062 --- [ocp-schedules-23,7b70826d74084c15,eac603af90126e0a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:29:39.323 INFO 21062 --- [ocp-schedules-23,7b70826d74084c15,eac603af90126e0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:29:39.407 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:29:39.409 ERROR 21062 --- [ocp-schedules-23,7b70826d74084c15,eac603af90126e0a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:29:39.409 INFO 21062 --- [ocp-schedules-23,7b70826d74084c15,eac603af90126e0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:29:40.002 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:29:40.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:29:40.612 INFO 21062 --- [ocp-schedules-8,4aee4c36d5e48a52,036420782194ffaf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:29:40.903 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:40.903 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:41.211 WARN 21062 --- [ocp-async-9,1a8c1dbd6b7d119f,6f8702dcabb6a48d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:41.266 INFO 21062 --- [ocp-async-8,55be790d52d995ad,9209d25f2f8afbc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:41.983 INFO 21062 --- [prometheus-minute-schedule-1,12988ec7e442f877,072527aaba355470] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:43.211 WARN 21062 --- [ocp-async-8,82275dc6a808d5ba,ed04e7b7e2c5359d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:43.266 INFO 21062 --- [ocp-async-3,4b9c96c41c4aea2b,ddcbfefd94e66e9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:43.904 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:43.904 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:44.479 INFO 21062 --- [ocp-schedules-1,03aa8560696ecd5e,33f0de405bb29052] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:29:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6ecfdcb1f9169ab4,d900d806b90b6c30] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:45.211 WARN 21062 --- [ocp-async-3,eca9deacf0b3dd98,762caa3535866e62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:45.267 INFO 21062 --- [ocp-async-7,a6e15ce737eac2b2,ed68065180395bf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:46.905 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:46.905 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:47.211 WARN 21062 --- [ocp-async-2,6c66f2e7275de028,b64af7b02d94e1f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:47.267 INFO 21062 --- [ocp-async-1,24c736ec824804d3,2edc2412e3ee231a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:47.983 INFO 21062 --- [prometheus-minute-schedule-1,1b37a18d728e8bf1,46076b73594c3dfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:29:48.590853+08:00, lastProcessTime=2024-11-04T14:29:38.589517+08:00 2024-11-04 14:29:48.665 INFO 21062 --- [ocp-async-3,88e0679c47bf8576,0236e8a83f24c7ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:29:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:29:48.590853+08:00 2024-11-04 14:29:49.211 WARN 21062 --- [ocp-async-8,d438c9d631c1565b,010dc4ac0005a915] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:49.267 INFO 21062 --- [ocp-async-10,cedc183eb1da3ef1,ab084d089999074f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:49.905 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:49.906 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:50.615 INFO 21062 --- [ocp-schedules-16,d2fd2476d8c71aaa,54f9cab5d1d06ee1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:29:50.983 INFO 21062 --- [prometheus-minute-schedule-1,e70d33a02ba22a53,3b4bd33304583f38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:51.212 WARN 21062 --- [ocp-async-6,1d1d9fc9e492ae71,bd099baea10e0d22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:51.267 INFO 21062 --- [ocp-async-7,770264b49b8b943f,9f3e01d39e723c47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:51.659 INFO 21062 --- [ocp-schedules-14,88a5162e5ef975fd,90c4f63b2338109b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:29:51.664 INFO 21062 --- [ocp-schedules-14,88a5162e5ef975fd,90c4f63b2338109b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:29:51.664 INFO 21062 --- [ocp-schedules-14,88a5162e5ef975fd,90c4f63b2338109b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:29:52.906 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:52.907 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:53.212 WARN 21062 --- [ocp-async-2,b057eb2b624a8656,a5132d9aef03096a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:53.267 INFO 21062 --- [ocp-async-1,bbe6df9f239e9c0b,a42b32ef3badebca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c478980c2cdee976,1d255d802baa4588] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:54.518 INFO 21062 --- [ocp-schedules-21,cbff2f5b3265acb9,cc337febc7928046] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:29:55.212 WARN 21062 --- [ocp-async-5,96b05528596be9e8,3ea5ead88b965968] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:55.267 INFO 21062 --- [ocp-async-4,126c0e6f684d063f,22f14c22e810cfac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:55.907 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:55.908 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7eb54751dae0683d,4974958325c340e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:29:57.212 WARN 21062 --- [ocp-async-10,5289f43b46dd40c0,662d456273767da5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:57.267 INFO 21062 --- [ocp-async-3,9027c4a5fe6af0c7,a64814115c41b6e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:58.499 INFO 21062 --- [ocp-schedules-29,0f885b712b16c781,45762633182bdc56] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:29:58.500 INFO 21062 --- [ocp-schedules-21,65a0b041c7be7478,56052a01b848c62c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:29:58.502 INFO 21062 --- [ocp-schedules-16,a702751f66d8ab6e,1b3b8dcd5e86d622] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:29:58.504 INFO 21062 --- [ocp-schedules-29,0f885b712b16c781,45762633182bdc56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:29:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:29:58.505 INFO 21062 --- [ocp-schedules-21,65a0b041c7be7478,56052a01b848c62c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:29:58.506 INFO 21062 --- [ocp-schedules-5,b2e790a5798e1916,e6549bec5c35c4a3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:29:58.511 INFO 21062 --- [ocp-schedules-20,b486a9834dc80825,2a70bf528c81e12d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:29:58.512 INFO 21062 --- [ocp-schedules-20,b486a9834dc80825,2a70bf528c81e12d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:29:58.512 INFO 21062 --- [ocp-schedules-2,1220c38378ef075b,2a6c6bb1cf040312] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:29:58.512 INFO 21062 --- [ocp-schedules-2,1220c38378ef075b,2a6c6bb1cf040312] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:29:58.512 INFO 21062 --- [ocp-schedules-18,e4465a61e25bf45a,c7501b747c84b18f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:29:58.512 INFO 21062 --- [ocp-schedules-18,e4465a61e25bf45a,c7501b747c84b18f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:29:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:29:58.589949+08:00, lastProcessTime=2024-11-04T14:29:48.590853+08:00 2024-11-04 14:29:58.666 INFO 21062 --- [ocp-async-5,d97bc37181f78526,5400aa64b14568cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:29:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:29:58.589949+08:00 2024-11-04 14:29:58.908 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:29:58.909 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:29:59.202 INFO 21062 --- [ocp-schedules-6,9b2ff983aa2c648e,5531b25cf8b8891e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:29:59.212 WARN 21062 --- [ocp-async-2,e375276edc8f07d2,c40eedbc91b64054] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:29:59.268 INFO 21062 --- [ocp-async-1,a32c6a549e3aec41,d65594af74aea575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:29:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d8683bb81ee9b571,b2324e4ec654f3a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701800, interval=60 2024-11-04 14:30:00.389 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2e1f4f1a6c3323af] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:25:00.0, nextRun=2024-11-04 14:30:00.0, currentDate=Mon Nov 04 14:30:00 CST 2024 2024-11-04 14:30:00.389 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2e1f4f1a6c3323af] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:25:00.0, nextRunTime=2024-11-04 14:30:00.0} 2024-11-04 14:30:00.416 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2e1f4f1a6c3323af] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:30:00.420 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2e1f4f1a6c3323af] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:30:00.420 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2e1f4f1a6c3323af] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:30:00.619 INFO 21062 --- [ocp-schedules-10,0639499d6570ab9a,3b72f4ba28461983] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:30:01.212 WARN 21062 --- [ocp-async-5,1d60af0f192d5063,e9e38269385c6845] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:01.268 INFO 21062 --- [ocp-async-4,802b0de6b220447e,72ff5135f64624b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:01.909 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:01.910 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ccad89980c339c0a,99fd805747f0105b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:03.213 WARN 21062 --- [ocp-async-10,1e0670ad924dc0ca,457248a9e9b5f91e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:03.268 INFO 21062 --- [ocp-async-3,026bf8dcb0cef0ca,58248ed0b5e28212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:04.490 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2c79c50ef7f0d17a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:20:05.0, nextRun=2024-11-04 14:30:04.0, currentDate=Mon Nov 04 14:30:04 CST 2024 2024-11-04 14:30:04.490 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2c79c50ef7f0d17a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 14:20:05.0, nextRunTime=2024-11-04 14:30:04.0} 2024-11-04 14:30:04.496 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2c79c50ef7f0d17a] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 14:30:04.496 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,2c79c50ef7f0d17a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T14:30:04.496+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 14:30:04.563 INFO 21062 --- [ocp-schedules-22,da24ffc407976f20,3e5c65a34702400f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:30:04.583 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,4ec00d876bb6abdd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 81, operation:EXECUTE, state: RUNNING 2024-11-04 14:30:04.610 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=81, context=Context{parallelIdx=-1, stringMap={task_instance_id=17, task_operation=execute, latest_execution_start_time=2024-11-04T14:30:04.563+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=81}, listMap={}}, executor=10.46.128.24 2024-11-04 14:30:04.615 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:30:04.616 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:30:04.618 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.618 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:30:04.619 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.620 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.620 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.621 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:30:04.637 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.638 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.638 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.639 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:30:04.652 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.652 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.653 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.653 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:30:04.668 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.668 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.669 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.669 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:30:04.682 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.683 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.683 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.684 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:30:04.689 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.690 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.690 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.690 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:30:04.695 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.695 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.695 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.696 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:30:04.700 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.700 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.701 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.701 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:30:04.707 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.708 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.708 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.709 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:30:04.714 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.714 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.714 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.715 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:30:04.721 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.721 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.722 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.722 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:30:04.730 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.731 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.731 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.732 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:30:04.745 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.745 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.745 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.746 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:30:04.751 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.751 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.752 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.752 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:30:04.758 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.759 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.759 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.760 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:30:04.766 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.766 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.767 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.767 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:30:04.772 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.772 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.773 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.773 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:30:04.777 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.778 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.778 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.779 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:30:04.783 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.784 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.784 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.785 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:30:04.790 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.790 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.791 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.791 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:30:04.795 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.796 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.796 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.797 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:30:04.801 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.802 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.802 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.803 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:30:04.807 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.807 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.808 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.808 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:30:04.813 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.813 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.814 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.814 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:30:04.819 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.819 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.820 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.820 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:30:04.825 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.825 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.826 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.826 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:30:04.831 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.831 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.832 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.832 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:30:04.836 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.837 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.837 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.838 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:30:04.842 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.843 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.843 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.843 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:30:04.847 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.848 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.848 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.849 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:30:04.853 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.853 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.854 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.854 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:30:04.858 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.859 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.859 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.860 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:30:04.864 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.864 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.865 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.865 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:30:04.869 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.870 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.870 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.871 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:30:04.878 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.879 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.879 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.880 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:30:04.886 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.886 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.887 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.887 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:30:04.893 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.893 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.894 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.894 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:30:04.900 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.900 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.901 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.901 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:30:04.905 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.905 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.906 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.906 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:30:04.910 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.910 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.910 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:04.911 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.911 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:04.911 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:30:04.915 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.916 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.917 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.917 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:30:04.921 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.922 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.923 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.924 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:30:04.934 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.934 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.935 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.935 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:30:04.956 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.957 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.957 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.958 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:30:04.964 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.964 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.965 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.965 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:30:04.971 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.971 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.972 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.972 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:30:04.978 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.978 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.979 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.979 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:30:04.983 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.984 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.984 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.985 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:30:04.988 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.988 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.989 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.989 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:30:04.993 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:30:04.993 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:30:04.995 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.995 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:30:04.997 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.997 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:04.997 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:04.998 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:30:05.003 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:05.003 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:30:05.004 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:30:05.004 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:30:05.008 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:30:05.018 INFO 21062 --- [subtask-executor14,dd193f8f67a5a423,4238a12ee89dd6db] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 81, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:30:05.213 WARN 21062 --- [ocp-async-3,f4e729255e60e707,191992d55a3942d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:05.268 INFO 21062 --- [ocp-async-9,7669688679545568,76a70300270633ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:05.400 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:17, status:SUCCESS 2024-11-04 14:30:05.401 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:17, status:SUCCESS 2024-11-04 14:30:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ac33206b5a88836f,613712d499c5fd18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:06.570 INFO 21062 --- [ocp-schedules-12,c9c611b9443bb307,2421807e23aeed17] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:30:06.570 INFO 21062 --- [ocp-schedules-12,8f4d1bde8d96d658,987b4eae9e3baacc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:30:06.573 INFO 21062 --- [ocp-schedules-12,8f4d1bde8d96d658,987b4eae9e3baacc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:06.602 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:30:06.602 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:30:07.213 WARN 21062 --- [ocp-async-9,c8c8b9d07a474943,fde9eff8ed1581ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:07.268 INFO 21062 --- [ocp-async-8,f2df415d21a33781,c7ab961c3e55322c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:07.911 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:07.912 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:08.505 INFO 21062 --- [ocp-schedules-29,0f885b712b16c781,45762633182bdc56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:30:08.506 INFO 21062 --- [ocp-schedules-21,65a0b041c7be7478,56052a01b848c62c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:30:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:30:08.590567+08:00, lastProcessTime=2024-11-04T14:29:58.589949+08:00 2024-11-04 14:30:08.666 INFO 21062 --- [ocp-async-7,df53db499cb6f33e,a4bececd108a20f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:30:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:30:08.590567+08:00 2024-11-04 14:30:08.983 INFO 21062 --- [prometheus-minute-schedule-1,49aebe90f57a9bc4,92bd439839b7d0e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:09.051 INFO 21062 --- [ocp-schedules-23,0afdcf19515740f0,f5f06b3032935eb6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:30:09.055 INFO 21062 --- [ocp-schedules-23,0afdcf19515740f0,f5f06b3032935eb6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:09.056 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:30:09.213 WARN 21062 --- [ocp-async-4,e7cad5a37f458a36,fcf437e88be984c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:09.268 INFO 21062 --- [ocp-async-3,d5b38c4befcf5f50,86b951919058be27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:30:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:30:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:30:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:30:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:30:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:30:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:30:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:30:10.622 INFO 21062 --- [ocp-schedules-9,f076c9d36ebf2e12,c1670f7cce72d45b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:30:10.912 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:10.913 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:11.213 WARN 21062 --- [ocp-async-3,19fd95a25e763959,4ed6e07faf704813] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:11.269 INFO 21062 --- [ocp-async-9,29159df216dbc023,f8db0f5d80f54ac2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=4, affectRows=4 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=4, validCount=4 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=11, affectRows=11 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=18, affectRows=18 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=11, validCount=11 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=18, validCount=18 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=3, affectRows=3 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=14, affectRows=14 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=3, validCount=3 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=25, affectRows=25 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=14, validCount=14 2024-11-04 14:30:11.587 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=21, affectRows=21 2024-11-04 14:30:11.588 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=25, validCount=25 2024-11-04 14:30:11.588 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=21, validCount=21 2024-11-04 14:30:11.588 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=34, affectRows=34 2024-11-04 14:30:11.588 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=34, validCount=34 2024-11-04 14:30:11.588 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=39, affectRows=39 2024-11-04 14:30:11.588 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=49, affectRows=49 2024-11-04 14:30:11.588 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=47, affectRows=47 2024-11-04 14:30:11.589 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=38, validCount=38 2024-11-04 14:30:11.589 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=49, validCount=49 2024-11-04 14:30:11.589 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=47, validCount=47 2024-11-04 14:30:11.589 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=53, affectRows=53 2024-11-04 14:30:11.589 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=52, validCount=52 2024-11-04 14:30:11.590 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701740, nodeCount=78, affectRows=78 2024-11-04 14:30:11.590 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701740, nodeCount=77, validCount=77 2024-11-04 14:30:11.666 INFO 21062 --- [ocp-schedules-26,de0ca81d49ab98a1,7a69273fd8244922] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:30:11.671 INFO 21062 --- [ocp-schedules-26,de0ca81d49ab98a1,7a69273fd8244922] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:30:11.671 INFO 21062 --- [ocp-schedules-26,de0ca81d49ab98a1,7a69273fd8244922] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:30:11.983 INFO 21062 --- [prometheus-minute-schedule-1,fcc9be093adc8216,0913cf3a1f43fc4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:13.213 WARN 21062 --- [ocp-async-9,0930c94f118740b5,050a0b76da092fa4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:13.269 INFO 21062 --- [ocp-async-8,d5f3ef2c1003276d,e4f93ddbfaca34fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:13.913 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:13.914 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:13.985 INFO 21062 --- [prometheus-cache-monitor1,979461d9c3a5e5cd,f9643ba7782a8f4a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:30:13.985 INFO 21062 --- [prometheus-cache-monitor1,979461d9c3a5e5cd,f9643ba7782a8f4a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:30:14.600 INFO 21062 --- [ocp-schedules-8,a29f5c763d6e53bf,5e91fcf5422a1f1f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:30:14.983 INFO 21062 --- [prometheus-minute-schedule-1,16707c85c043f40d,f320ef00a8f833f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:15.214 WARN 21062 --- [ocp-async-8,7b7c9dc8d3fbfb38,dd5c7e41fd8349e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:15.269 INFO 21062 --- [ocp-async-10,91278dd1191cc73e,7011d4c81cc27209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:16.914 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:16.914 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:17.214 WARN 21062 --- [ocp-async-10,beb35c129187212e,7a82ffd39e55b766] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:17.269 INFO 21062 --- [ocp-async-7,e0c0ff54f32a0538,31e91e38d1c1a8a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4754ca6bb353ba70,fb64b5ef832f97ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:30:18.589296+08:00, lastProcessTime=2024-11-04T14:30:08.590567+08:00 2024-11-04 14:30:18.666 INFO 21062 --- [ocp-async-4,fbf9039be1dca09c,e82816e5c997929e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:30:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:30:18.589296+08:00 2024-11-04 14:30:19.056 INFO 21062 --- [ocp-schedules-23,0afdcf19515740f0,f5f06b3032935eb6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:30:19.214 WARN 21062 --- [ocp-async-9,3acb7d6f526a13e2,22591dd44418aad4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:19.269 INFO 21062 --- [ocp-async-8,1dcf7fb3f25d6b90,b166d4df2e120665] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:19.915 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:19.915 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:20.624 INFO 21062 --- [ocp-schedules-27,6aa9f3a04f023147,8967f148743db76e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:30:20.983 INFO 21062 --- [prometheus-minute-schedule-1,86a14c1f17ed8fc4,2a6318180554c1f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:21.214 WARN 21062 --- [ocp-async-8,efe8bbfdaebd53d7,2b6ddea38f0d84dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:21.270 INFO 21062 --- [ocp-async-10,167d3b49a1ee0576,2f107aba9a667081] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:22.915 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:22.916 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:23.214 WARN 21062 --- [ocp-async-10,74f72811b4e5407d,e060c9f40053fbf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:23.270 INFO 21062 --- [ocp-async-7,c637b1f43bdada26,0408136778c72567] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4d0069bd96cc46cd,a9634fc25ea73fcf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:24.644 INFO 21062 --- [ocp-schedules-17,39116645037632f3,f332d7d77f088614] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:30:25.215 WARN 21062 --- [ocp-async-7,ce7bac8db06bda6a,55dd4824de47ea0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:25.270 INFO 21062 --- [ocp-async-5,984afd3a6e68b19d,381ee5b4011a2272] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:25.916 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:25.917 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f47d5b13091c8164,d5a0850da84794d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9e44443466c01095,ce5bf898dc67ef2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:30:27.215 WARN 21062 --- [ocp-async-5,b90eb0cde2b52d61,1baf69ada0ab0bf0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:27.270 INFO 21062 --- [ocp-async-6,3ac7c0c2d8cefd72,f9524659df2a95f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:28.498 INFO 21062 --- [ocp-schedules-28,5f06dac5f6132539,0cd7baac889654d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:30:28.500 INFO 21062 --- [ocp-schedules-22,d2d205cf967ab92d,eddd868142aa0c52] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:30:28.500 INFO 21062 --- [ocp-schedules-8,1eb0d96352aaf75c,fa4b9921dd2cbc06] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:30:28.500 INFO 21062 --- [ocp-schedules-21,9e84c1e533c1fb7e,bc9c2e1e34b4e3fe] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:30:28.502 INFO 21062 --- [ocp-schedules-25,e6eb85e695036600,8d850cf1fee7d130] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:30:28.503 INFO 21062 --- [ocp-schedules-28,5f06dac5f6132539,0cd7baac889654d2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:30:28.504 INFO 21062 --- [ocp-schedules-8,1eb0d96352aaf75c,fa4b9921dd2cbc06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:28.504 INFO 21062 --- [ocp-schedules-21,9e84c1e533c1fb7e,bc9c2e1e34b4e3fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:28.504 INFO 21062 --- [ocp-schedules-6,dce76c361c3e1eee,2e178b1a166ebb42] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:30:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:30:28.506 INFO 21062 --- [ocp-schedules-22,d2d205cf967ab92d,eddd868142aa0c52] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:30:28.509 INFO 21062 --- [ocp-schedules-9,12e0dc6351b97335,78eea9fda43a70cd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:30:28.509 INFO 21062 --- [ocp-schedules-9,12e0dc6351b97335,78eea9fda43a70cd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:30:28.510 INFO 21062 --- [ocp-schedules-10,7ada7bc801348277,a6754bcf6f9ec7b8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:30:28.510 INFO 21062 --- [ocp-schedules-10,7ada7bc801348277,a6754bcf6f9ec7b8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:30:28.510 INFO 21062 --- [ocp-schedules-19,66b6a39f93453494,81c479aebfb913ae] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:30:28.511 INFO 21062 --- [ocp-schedules-19,66b6a39f93453494,81c479aebfb913ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:30:28.577 INFO 21062 --- [ocp-schedules-14,f5cccb4df322667b,6124942b124ca58a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:30:28.577 INFO 21062 --- [ocp-schedules-14,f5cccb4df322667b,6124942b124ca58a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:30:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:30:28.590992+08:00, lastProcessTime=2024-11-04T14:30:18.589296+08:00 2024-11-04 14:30:28.633 INFO 21062 --- [ocp-schedules-15,fb6d0734dd8d6e4f,f5b6a6d38fffb894] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:30:28.634 INFO 21062 --- [ocp-schedules-15,fb6d0734dd8d6e4f,f5b6a6d38fffb894] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:30:28.666 INFO 21062 --- [ocp-async-5,aca840bb01506063,1d918e82a8c79977] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:30:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:30:28.590992+08:00 2024-11-04 14:30:28.818 INFO 21062 --- [ocp-schedules-13,f1aee36b659dce8a,b70a02733826a0c7] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:30:28.818 INFO 21062 --- [ocp-schedules-13,f1aee36b659dce8a,b70a02733826a0c7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:30:28.846 INFO 21062 --- [ocp-schedules-27,d32c1202f3072534,bc5951f31909ed75] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:30:28.917 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:28.918 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:29.128 INFO 21062 --- [ocp-schedules-22,88ee65cab3a0d70b,94efd72d1a59ac67] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:30:29.128 INFO 21062 --- [ocp-schedules-22,88ee65cab3a0d70b,94efd72d1a59ac67] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:30:29.205 INFO 21062 --- [ocp-schedules-19,77a83bfb3ee2b37c,633b3ae9361cb80f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:30:29.215 WARN 21062 --- [ocp-async-7,04d1c2db7792927e,0f938cc345340ac0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:29.237 INFO 21062 --- [ocp-schedules-14,0ffe954077242e49,919d29318dbd0ca0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:30:29.242 INFO 21062 --- [ocp-schedules-14,0ffe954077242e49,919d29318dbd0ca0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:30:29.242 INFO 21062 --- [ocp-schedules-14,0ffe954077242e49,919d29318dbd0ca0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:30:29.270 INFO 21062 --- [ocp-async-1,06e5beabf7407ca2,ddddddd1bac4371f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:29.579 INFO 21062 --- [ocp-schedules-1,66f81e823bdd6c64,4a37bcadc4212d29] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:30:29.579 INFO 21062 --- [ocp-schedules-1,66f81e823bdd6c64,4a37bcadc4212d29] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:30:29.580 INFO 21062 --- [ocp-schedules-1,66f81e823bdd6c64,4a37bcadc4212d29] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:30:29.580 INFO 21062 --- [ocp-schedules-1,66f81e823bdd6c64,4a37bcadc4212d29] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:30:29.580 INFO 21062 --- [ocp-schedules-1,66f81e823bdd6c64,4a37bcadc4212d29] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:30:29.580 INFO 21062 --- [ocp-schedules-1,66f81e823bdd6c64,4a37bcadc4212d29] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:30:29.983 INFO 21062 --- [prometheus-minute-schedule-1,c64d54c419338699,c3b2b9330fc48e5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:30:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:30:30.626 INFO 21062 --- [ocp-schedules-16,a3c93c880143889c,c65b7b905094b884] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:30:31.215 WARN 21062 --- [ocp-async-1,b0d6a373ecbd00aa,132cebb3d8f354c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:31.270 INFO 21062 --- [ocp-async-4,1abeb5dfd35657c2,9daf4918c8e8d955] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:31.674 INFO 21062 --- [ocp-schedules-28,01d924399400566d,4106d33bca31e0c0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:30:31.680 INFO 21062 --- [ocp-schedules-28,01d924399400566d,4106d33bca31e0c0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:30:31.680 INFO 21062 --- [ocp-schedules-28,01d924399400566d,4106d33bca31e0c0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:30:31.746 INFO 21062 --- [ocp-schedules-27,a2fc4e55639ea4b2,a067c125d91705f1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:30:31.746 INFO 21062 --- [ocp-schedules-27,a2fc4e55639ea4b2,a067c125d91705f1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:30:31.771 INFO 21062 --- [ocp-schedules-27,a2fc4e55639ea4b2,a067c125d91705f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:30:31.771 INFO 21062 --- [ocp-schedules-27,a2fc4e55639ea4b2,a067c125d91705f1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:30:31.771 INFO 21062 --- [ocp-schedules-3,9efe80dc2374ef76,727f3c3f1c62993a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:30:31.918 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:31.918 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:31.958 INFO 21062 --- [ocp-schedules-26,6c0f659cf4edc5eb,cad8cef7b0ac1f25] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:30:31.958+08:00, clusters=[1] 2024-11-04 14:30:31.959 INFO 21062 --- [ocp-schedules-26,6c0f659cf4edc5eb,cad8cef7b0ac1f25] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:30:32.137 INFO 21062 --- [ocp-schedules-15,cc853da9ac2a5337,235e1fdcbd1a6482] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:30:32.137 INFO 21062 --- [ocp-schedules-17,2e02cf7e92e90f10,48b7c69bf9b6a233] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:30:32.137+08:00, clusters=[1] 2024-11-04 14:30:32.141 INFO 21062 --- [ocp-schedules-17,2e02cf7e92e90f10,48b7c69bf9b6a233] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:30:32.141 INFO 21062 --- [ocp-schedules-15,cc853da9ac2a5337,235e1fdcbd1a6482] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:32.199 ERROR 21062 --- [ocp-schedules-15,cc853da9ac2a5337,235e1fdcbd1a6482] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:30:32.199 INFO 21062 --- [ocp-schedules-15,cc853da9ac2a5337,235e1fdcbd1a6482] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:30:32.983 INFO 21062 --- [prometheus-minute-schedule-1,100477c2f1ddfd87,f82a84b8b3c77129] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:33.212 INFO 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:30:33.212 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:30:33.212 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:30:33.212 INFO 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:30:33.215 WARN 21062 --- [ocp-async-4,07722c4e69c9512b,b3bc3c7681687004] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:33.222 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:30:33.222 INFO 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:30:33.223 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:30:33.224 WARN 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:30:33.224 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:30:33.224 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:30:33.225 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:30:33.225 INFO 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:30:33.225 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:30:33.226 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:30:33.226 WARN 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:30:33.226 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:30:33.234 INFO 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:30:33.234 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:30:33.235 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:30:33.259 ERROR 21062 --- [ocp-schedules-3,999212f907714050,7aa57ee6fcb02e48] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:30:33.271 INFO 21062 --- [ocp-async-3,4c381eb56f8db5b3,2121408e5481653b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:34.668 INFO 21062 --- [ocp-schedules-26,c0d92768d40f0ada,11388861705bcac1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:30:34.918 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:34.919 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:35.215 WARN 21062 --- [ocp-async-3,612637575c3ab6d0,c1a402043cdb68d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:35.271 INFO 21062 --- [ocp-async-9,13808bceaefb73bd,d13becd541d18420] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:35.983 INFO 21062 --- [prometheus-minute-schedule-1,066f8deaae599981,fafb622f02a99bf6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:36.602 INFO 21062 --- [ocp-schedules-12,8f4d1bde8d96d658,987b4eae9e3baacc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:30:36.603 INFO 21062 --- [ocp-schedules-12,224626b2e780ac1c,f3f92c7b73b7a5c7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:30:36.607 INFO 21062 --- [ocp-schedules-12,224626b2e780ac1c,f3f92c7b73b7a5c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:36.637 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:30:36.637 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:30:37.216 WARN 21062 --- [ocp-async-9,d3ce5c841eb70260,0679a1dbeb33a509] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:37.271 INFO 21062 --- [ocp-async-8,e5aefed750e6d171,1480afee64f5a490] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:37.919 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:37.920 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:38.505 INFO 21062 --- [ocp-schedules-21,9e84c1e533c1fb7e,bc9c2e1e34b4e3fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:30:38.505 INFO 21062 --- [ocp-schedules-8,1eb0d96352aaf75c,fa4b9921dd2cbc06] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:30:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:30:38.590768+08:00, lastProcessTime=2024-11-04T14:30:28.590992+08:00 2024-11-04 14:30:38.666 INFO 21062 --- [ocp-async-7,f3d2bcf3cb60a329,f8a08d8d1e5a5f9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:30:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:30:38.590768+08:00 2024-11-04 14:30:38.983 INFO 21062 --- [prometheus-minute-schedule-1,1eb206aba317b977,82875c57df4a6fcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:39.216 WARN 21062 --- [ocp-async-6,2004f0e2b120d674,2c588866102f8b47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:39.271 INFO 21062 --- [ocp-async-3,3d6846026785d363,13afb59b72ebb638] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:39.409 INFO 21062 --- [ocp-schedules-5,1293e3a47451fa3d,cd976b9cb7926710] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:30:39.413 INFO 21062 --- [ocp-schedules-5,1293e3a47451fa3d,cd976b9cb7926710] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:39.500 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:30:39.503 ERROR 21062 --- [ocp-schedules-5,1293e3a47451fa3d,cd976b9cb7926710] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:30:39.503 INFO 21062 --- [ocp-schedules-5,1293e3a47451fa3d,cd976b9cb7926710] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:30:40.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:30:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:30:40.630 INFO 21062 --- [ocp-schedules-7,85bea4b5d2fcc946,ee17cbaff6917768] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:30:40.920 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:40.921 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:41.216 WARN 21062 --- [ocp-async-3,d9af2d9a9cbbf2a1,c0a9e57d1e898009] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:41.272 INFO 21062 --- [ocp-async-9,d1303e3cc9de2d93,c7ba11bf0ecf6464] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:30:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:30:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 14:30:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a35e64292f84d9c3,caedbea4975056dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:43.216 WARN 21062 --- [ocp-async-9,c500fed861cbf040,360353107cd0a071] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:43.272 INFO 21062 --- [ocp-async-8,fbad9007567a4d01,fb644e5562d28064] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:43.921 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:43.922 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:44.710 INFO 21062 --- [ocp-schedules-10,3de4368e9e2a00b0,7c30de77b5ca916a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:30:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a794298371a1f7ba,926e21645684c0d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:45.216 WARN 21062 --- [ocp-async-8,0a6fbb7ec647eb3c,5f1d53bf5bbf78f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:45.272 INFO 21062 --- [ocp-async-10,99352d6e02d89076,795bfc283664ca1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:46.922 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:46.922 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:47.107 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:30:47.107 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:30:47.114 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:30:47.114 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:30:47.114 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:30:47.114 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:30:47.120 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:30:47.120 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:30:47.120 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:30:47.120 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:30:47.156 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:30:47.164 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:30:47.164 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:30:47.164 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:30:47.164 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:30:47.164 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:30:47.172 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:30:47.233 WARN 21062 --- [ocp-async-10,02f7201c49870ee7,2100b421559a33e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:47.272 INFO 21062 --- [ocp-async-7,c058cd2742037428,619f4374d6e3cad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:47.727 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:30:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b85f44c3eed14f74,a4d9eac911f0fe11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:48.290 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.169368101S 2024-11-04 14:30:48.290 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:30:48.395 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:30:48.395 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:30:48.424 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:30:48.490 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:30:48.490 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:30:48.490 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:30:48.490 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:30:48.490 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:30:48.490 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:30:48.501 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:30:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:30:48.590581+08:00, lastProcessTime=2024-11-04T14:30:38.590768+08:00 2024-11-04 14:30:48.667 INFO 21062 --- [ocp-async-8,4a9679d15d0b1fbd,3060d358b75af7d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:30:48.673 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] o.s.boot.SpringApplication : Started application in 1.559 seconds (JVM running for 7262.352) 2024-11-04 14:30:48.673 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.559584153S 2024-11-04 14:30:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:30:48.590581+08:00 2024-11-04 14:30:49.132 INFO 21062 --- [ocp-schedules-6,19c746b34a3b5826,60bfe06aa478a3e3] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:30:49.234 WARN 21062 --- [ocp-async-5,14ebf44c6c9d1c3b,5b91dec68618253a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:49.272 INFO 21062 --- [ocp-async-4,51f591540769a5ed,6200fb3e495fc0cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:49.923 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:49.924 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:50.633 INFO 21062 --- [ocp-schedules-30,420671f8a27c24ed,8a86c888d492d05f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:30:50.983 INFO 21062 --- [prometheus-minute-schedule-1,c10214b8e5ee895a,08d55fcd32fd17a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:51.234 WARN 21062 --- [ocp-async-6,f99361014d808930,0d8db0aebf4b2f24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:51.272 INFO 21062 --- [ocp-async-2,fcb72c75cf9c0ee3,0766ebfe61a82356] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:51.683 INFO 21062 --- [ocp-schedules-16,81203966404fe916,49dc344158b7ade2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:30:51.690 INFO 21062 --- [ocp-schedules-16,81203966404fe916,49dc344158b7ade2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:30:51.690 INFO 21062 --- [ocp-schedules-16,81203966404fe916,49dc344158b7ade2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:30:52.924 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:52.925 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:53.240 WARN 21062 --- [ocp-async-3,c43d956d1fb61883,c7385ef222429b95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:53.272 INFO 21062 --- [ocp-async-7,7754e6635170804f,469fea4435071e66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ca38ca8af8e0cffd,efeafcfae4bc5c13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:54.738 INFO 21062 --- [ocp-schedules-3,6e68b03c3aa914df,c3167d60921c8a99] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:30:55.240 WARN 21062 --- [ocp-async-1,6f4240d8ad2e748c,11125d4b026d3170] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:55.273 INFO 21062 --- [ocp-async-9,00b07a6830a6a86d,5ced00c19be79a66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:55.925 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:55.926 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f95657f70d64469a,6e1da28aab3662da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:30:57.240 WARN 21062 --- [ocp-async-6,b151b45eb8617a1e,974fbed32f771336] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:57.273 INFO 21062 --- [ocp-async-8,6718982de6730f8c,260681f7bc517cbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:58.500 INFO 21062 --- [ocp-schedules-22,041a306a9eedb7c2,11eef651fa6544d2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:30:58.500 INFO 21062 --- [ocp-schedules-1,beae9eb777f6b7a8,2ddc98e7ff20b539] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:30:58.502 INFO 21062 --- [ocp-schedules-10,75f8c955438ad9bc,04abd8d15c76d285] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:30:58.504 INFO 21062 --- [ocp-schedules-22,041a306a9eedb7c2,11eef651fa6544d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:58.504 INFO 21062 --- [ocp-schedules-1,beae9eb777f6b7a8,2ddc98e7ff20b539] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:30:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:30:58.505 INFO 21062 --- [ocp-schedules-13,5c634000e0096043,79589806c58dab98] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:30:58.511 INFO 21062 --- [ocp-schedules-30,3648cb96bfa09725,cea759a415269ee6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:30:58.511 INFO 21062 --- [ocp-schedules-23,cb9990d196046fa4,0f7a27cc91e76c9d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:30:58.511 INFO 21062 --- [ocp-schedules-30,3648cb96bfa09725,cea759a415269ee6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:30:58.511 INFO 21062 --- [ocp-schedules-23,cb9990d196046fa4,0f7a27cc91e76c9d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:30:58.511 INFO 21062 --- [ocp-schedules-9,28537fe0972d0812,17bcae614d05ebd9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:30:58.511 INFO 21062 --- [ocp-schedules-9,28537fe0972d0812,17bcae614d05ebd9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:30:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:30:58.590799+08:00, lastProcessTime=2024-11-04T14:30:48.590581+08:00 2024-11-04 14:30:58.667 INFO 21062 --- [ocp-async-1,c01711d007f155a9,04efc4024754949b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:30:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:30:58.590799+08:00 2024-11-04 14:30:58.926 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:30:58.927 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:30:59.208 INFO 21062 --- [ocp-schedules-21,cd769e804b5d8906,d06f84d0b09b4bfb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:30:59.241 WARN 21062 --- [ocp-async-10,4cb3a1f1552d4bf1,1ef6cc39b7828f69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:30:59.273 INFO 21062 --- [ocp-async-7,2626ce7d0f7e9a8f,a77bf014d5fffeb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:30:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e8b82a759cf5c9d0,2c8cde087848838b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701860, interval=60 2024-11-04 14:31:00.636 INFO 21062 --- [ocp-schedules-19,44d1fc2eddef7143,90f32059672542e3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:31:01.241 WARN 21062 --- [ocp-async-5,724340a5001fbab5,b2267e0c325061cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:01.273 INFO 21062 --- [ocp-async-9,1d31e8f85c98fb16,38477174ca4038a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:01.927 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:01.928 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1620dec15e3b2828,c00c2410afaaa1f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:03.241 WARN 21062 --- [ocp-async-6,5a9024b165952c94,8e3eece0f9fa5fd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:03.273 INFO 21062 --- [ocp-async-8,da5932355bd4301b,c469169ab9ac8bc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:04.764 INFO 21062 --- [ocp-schedules-24,ded87940364a3aa4,79edff896ff5db2e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:31:04.928 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:04.928 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:05.241 WARN 21062 --- [ocp-async-2,74681610f56bef4e,fbd7cfd6e35d15ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:05.273 INFO 21062 --- [ocp-async-3,de53959ad937f65d,236be916f8117d36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e0566c6cb964fd4e,79ea7fa45d4dfcbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:06.638 INFO 21062 --- [ocp-schedules-12,224626b2e780ac1c,f3f92c7b73b7a5c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:31:06.638 INFO 21062 --- [ocp-schedules-12,713fe3e7aeb1b72d,52135ac1ca449cef] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:31:06.641 INFO 21062 --- [ocp-schedules-12,713fe3e7aeb1b72d,52135ac1ca449cef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:06.672 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:31:06.672 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:31:07.241 WARN 21062 --- [ocp-async-5,6a2516f74cd1aa1b,b96d8395a4b7ee90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:07.274 INFO 21062 --- [ocp-async-1,acb841372c59c838,bf72f0f4e15af1e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:07.929 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:07.929 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:08.505 INFO 21062 --- [ocp-schedules-1,beae9eb777f6b7a8,2ddc98e7ff20b539] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:31:08.505 INFO 21062 --- [ocp-schedules-22,041a306a9eedb7c2,11eef651fa6544d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:31:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:31:08.589728+08:00, lastProcessTime=2024-11-04T14:30:58.590799+08:00 2024-11-04 14:31:08.667 INFO 21062 --- [ocp-async-2,29a425f665d586b6,75f7052a657fc68e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:31:08.741 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:31:08.589728+08:00 2024-11-04 14:31:08.983 INFO 21062 --- [prometheus-minute-schedule-1,595d495648a02847,8a081bf6b9788403] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:09.241 WARN 21062 --- [ocp-async-4,7cb10d6b9efa8c03,4aefaa35cfd872c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:09.274 INFO 21062 --- [ocp-async-8,3015456d8a289871,1c079fb56483b16b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:31:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:31:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:31:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:31:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:31:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:31:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:31:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:31:10.639 INFO 21062 --- [ocp-schedules-2,2401a877120e8f4e,9992e59f5c8cd6ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:31:10.930 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:10.930 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:11.241 WARN 21062 --- [ocp-async-2,f3030cc0c0c1c4dd,cc4fd5668dce0613] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:11.274 INFO 21062 --- [ocp-async-3,87ff3c54cc2e0ebe,98f7b9fc2a8a1a48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:11.589 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701800, nodeCount=43, affectRows=43 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701800, nodeCount=43, validCount=43 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701800, nodeCount=52, affectRows=52 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701800, nodeCount=43, affectRows=43 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701800, nodeCount=55, affectRows=55 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701800, nodeCount=54, affectRows=54 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701800, nodeCount=43, validCount=43 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701800, nodeCount=55, validCount=55 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701800, nodeCount=51, validCount=51 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701800, nodeCount=54, validCount=54 2024-11-04 14:31:11.590 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701800, nodeCount=69, affectRows=69 2024-11-04 14:31:11.591 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701800, nodeCount=69, validCount=69 2024-11-04 14:31:11.591 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701800, nodeCount=80, affectRows=80 2024-11-04 14:31:11.591 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701800, nodeCount=78, validCount=78 2024-11-04 14:31:11.692 INFO 21062 --- [ocp-schedules-10,fd8b92f64aa497af,45c71ab7a3f64955] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:31:11.697 INFO 21062 --- [ocp-schedules-10,fd8b92f64aa497af,45c71ab7a3f64955] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:31:11.697 INFO 21062 --- [ocp-schedules-10,fd8b92f64aa497af,45c71ab7a3f64955] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:31:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0faf0d6099ca83b2,7fb0f602c42b1fc2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:13.242 WARN 21062 --- [ocp-async-7,5bb321cca51cebde,20b24dc0460655c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:13.274 INFO 21062 --- [ocp-async-1,8e52b465fcc6e703,328a9eff15eec2fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:13.931 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:13.931 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:13.984 INFO 21062 --- [prometheus-cache-monitor1,044f946ba457759c,39d09a267f67e6e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:31:13.984 INFO 21062 --- [prometheus-cache-monitor1,044f946ba457759c,39d09a267f67e6e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:31:14.788 INFO 21062 --- [ocp-schedules-7,c3a1c465631beaad,5bb12813686a45ad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:31:14.983 INFO 21062 --- [prometheus-minute-schedule-1,98086b0397d6a09f,0435e8532235ea91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:15.112 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:27:15.094, end=2024-11-04T14:29:15.109} 2024-11-04 14:31:15.121 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:31:15.121 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:31:15.242 WARN 21062 --- [ocp-async-4,f81cba1b7be8f71a,24d4084e68439baa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:15.274 INFO 21062 --- [ocp-async-6,f56e1a862688032f,5bd632116616cd72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:16.931 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:16.932 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:17.242 WARN 21062 --- [ocp-async-2,4cc303d505794afd,3fcfb2c7e4bfec07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:17.274 INFO 21062 --- [ocp-async-10,ec9ff026fbc91c25,4a3b695c40a7a2d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:17.983 INFO 21062 --- [prometheus-minute-schedule-1,46451fb399ff9063,eda1b014ed2d4057] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:31:18.590548+08:00, lastProcessTime=2024-11-04T14:31:08.589728+08:00 2024-11-04 14:31:18.667 INFO 21062 --- [ocp-async-4,f2b703b6e6da7a85,04dfbf56c5bafcf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:31:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:31:18.590548+08:00 2024-11-04 14:31:19.056 INFO 21062 --- [ocp-schedules-2,7af8805d5883d5be,624374c78aa0c9ff] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:31:19.060 INFO 21062 --- [ocp-schedules-2,7af8805d5883d5be,624374c78aa0c9ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:19.060 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:31:19.242 WARN 21062 --- [ocp-async-5,9fb353b100685756,c25e94120e18b45d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:19.275 INFO 21062 --- [ocp-async-1,99362da7f83cd77b,293031183372722b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:19.932 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:19.933 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:20.642 INFO 21062 --- [ocp-schedules-13,0b28a698946d4a21,b398c5b20839b977] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:31:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ceeb8b6544add25c,410101f354c876b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:21.242 WARN 21062 --- [ocp-async-9,04b154fcc4e1981b,f58bb4e81dcf0b88] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:21.275 INFO 21062 --- [ocp-async-6,801bc784ae35c301,b5d2a28f054cb541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:22.933 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:22.935 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:23.242 WARN 21062 --- [ocp-async-2,544b0061b0113d04,949c74b78058f9e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:23.275 INFO 21062 --- [ocp-async-10,e147b9b7e87de347,ca97b94e673c4e27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b0267f0bcaba68ee,7221024ca0f1297e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:24.829 INFO 21062 --- [ocp-schedules-27,6c0599a27f5a1321,143d4f7de72a3888] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:31:25.243 WARN 21062 --- [ocp-async-3,7c4f811af80718f9,45a2c923ed8efc1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:25.275 INFO 21062 --- [ocp-async-7,8cceb92923100891,9225c2a274972624] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:25.935 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:25.936 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:26.983 INFO 21062 --- [prometheus-minute-schedule-1,8adddfe2ea21a556,b39d89d45a128177] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:26.984 INFO 21062 --- [prometheus-hour-schedule-1,235f37e88623b1fe,0636b12868b148ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:31:27.243 WARN 21062 --- [ocp-async-7,6bb08e17630f9169,860b48472788921a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:27.275 INFO 21062 --- [ocp-async-4,d855c3dc254bdfc2,06c8d18f3bc4a0b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:28.498 INFO 21062 --- [ocp-schedules-13,308b3c04af412425,4c765a5e494e46b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:31:28.500 INFO 21062 --- [ocp-schedules-23,756371e72a14f794,f76639b7342c89ab] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:31:28.500 INFO 21062 --- [ocp-schedules-11,a0a9f9be394b536e,a9f9958ecff3b67a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:31:28.501 INFO 21062 --- [ocp-schedules-22,6c37773694718a81,fe7529d2bdb8df83] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:31:28.502 INFO 21062 --- [ocp-schedules-7,46247d8bed819118,6dfa8fde638143c8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:31:28.503 INFO 21062 --- [ocp-schedules-13,308b3c04af412425,4c765a5e494e46b2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:31:28.504 INFO 21062 --- [ocp-schedules-23,756371e72a14f794,f76639b7342c89ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:28.504 INFO 21062 --- [ocp-schedules-11,a0a9f9be394b536e,a9f9958ecff3b67a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:31:28.504 INFO 21062 --- [ocp-schedules-14,317f7ad9dd6510e1,47588d4ae2774f66] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:31:28.507 INFO 21062 --- [ocp-schedules-22,6c37773694718a81,fe7529d2bdb8df83] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:31:28.510 INFO 21062 --- [ocp-schedules-20,1c6e779c3b6e0f04,c00eb2b7c1a856ad] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:31:28.510 INFO 21062 --- [ocp-schedules-20,1c6e779c3b6e0f04,c00eb2b7c1a856ad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:31:28.511 INFO 21062 --- [ocp-schedules-26,a22f5a8b53904a67,c967f06066b104b4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:31:28.511 INFO 21062 --- [ocp-schedules-26,a22f5a8b53904a67,c967f06066b104b4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:31:28.511 INFO 21062 --- [ocp-schedules-17,191620960a60f349,ab781be9aa758886] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:31:28.511 INFO 21062 --- [ocp-schedules-17,191620960a60f349,ab781be9aa758886] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:31:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:31:28.589951+08:00, lastProcessTime=2024-11-04T14:31:18.590548+08:00 2024-11-04 14:31:28.667 INFO 21062 --- [ocp-async-7,83281e3ac4aec759,ee1d36c758ee2c18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:31:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:31:28.589951+08:00 2024-11-04 14:31:28.849 INFO 21062 --- [ocp-schedules-21,30996eda8b9db9ab,afb2c9c2cb7a1f70] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:31:28.936 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:28.937 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:29.061 INFO 21062 --- [ocp-schedules-2,7af8805d5883d5be,624374c78aa0c9ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:31:29.133 INFO 21062 --- [ocp-schedules-14,5c82599a48dc9758,9e0ac5ba9a8deeaa] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:31:29.133 INFO 21062 --- [ocp-schedules-14,5c82599a48dc9758,9e0ac5ba9a8deeaa] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:31:29.210 INFO 21062 --- [ocp-schedules-26,19e7b0dbf628903e,c7ad199702dd3222] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:31:29.242 INFO 21062 --- [ocp-schedules-17,271f0c51792121a8,79e5cbaee3cabc89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:31:29.243 WARN 21062 --- [ocp-async-6,5847590d845ab86e,c6d00e034619c72d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:29.248 INFO 21062 --- [ocp-schedules-17,271f0c51792121a8,79e5cbaee3cabc89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:31:29.248 INFO 21062 --- [ocp-schedules-17,271f0c51792121a8,79e5cbaee3cabc89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:31:29.275 INFO 21062 --- [ocp-async-10,7c7d7e29b9abe18c,45835820ad64bc53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:29.409 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:31:29.409 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:31:29.418 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:31:29.418 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:31:29.418 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:31:29.429 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:31:29.429 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:31:29.429 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:31:29.433 INFO 21062 --- [ocp-schedules-9,047ab0b14db7e5b7,6267a494beb76f5c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:31:29.581 INFO 21062 --- [ocp-schedules-1,777cb5d586e2848e,ed41b339e84e43dc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:31:29.581 INFO 21062 --- [ocp-schedules-1,777cb5d586e2848e,ed41b339e84e43dc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:31:29.588 INFO 21062 --- [ocp-schedules-1,777cb5d586e2848e,ed41b339e84e43dc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:31:29.588 INFO 21062 --- [ocp-schedules-1,777cb5d586e2848e,ed41b339e84e43dc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:31:29.594 INFO 21062 --- [ocp-schedules-1,777cb5d586e2848e,ed41b339e84e43dc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:31:29.594 INFO 21062 --- [ocp-schedules-1,777cb5d586e2848e,ed41b339e84e43dc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:31:29.983 INFO 21062 --- [prometheus-minute-schedule-1,5cb5f52c47902b52,6650daddbd4007d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:31:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:31:30.645 INFO 21062 --- [ocp-schedules-17,4d976a233c29baec,4b955d6b2897cb83] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:31:31.243 WARN 21062 --- [ocp-async-7,1fbd29b1a8de06df,d4deb52319e44fda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:31.276 INFO 21062 --- [ocp-async-9,d8f998d032795fd2,1e61a44e5b46e87c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:31.699 INFO 21062 --- [ocp-schedules-10,e224ceddcaae82b3,1ab5179618dbb2ce] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:31:31.705 INFO 21062 --- [ocp-schedules-10,e224ceddcaae82b3,1ab5179618dbb2ce] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:31:31.705 INFO 21062 --- [ocp-schedules-10,e224ceddcaae82b3,1ab5179618dbb2ce] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:31:31.771 INFO 21062 --- [ocp-schedules-3,2ed336ec2a7fd125,41d1846c1c8e7599] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:31:31.771 INFO 21062 --- [ocp-schedules-3,2ed336ec2a7fd125,41d1846c1c8e7599] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:31:31.800 INFO 21062 --- [ocp-schedules-20,ce4cd309e7b7ce6e,026f838ecf9acd8c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:31:31.801 INFO 21062 --- [ocp-schedules-3,2ed336ec2a7fd125,41d1846c1c8e7599] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:31:31.801 INFO 21062 --- [ocp-schedules-3,2ed336ec2a7fd125,41d1846c1c8e7599] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:31:31.937 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:31.938 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:31.985 INFO 21062 --- [ocp-schedules-28,4316d1014dfcf9eb,4689116a6b2164e8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:31:31.985+08:00, clusters=[1] 2024-11-04 14:31:31.987 INFO 21062 --- [ocp-schedules-28,4316d1014dfcf9eb,4689116a6b2164e8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:31:32.169 INFO 21062 --- [ocp-schedules-19,4096b9563a101fd4,3b5dea8bfb00c730] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:31:32.169+08:00, clusters=[1] 2024-11-04 14:31:32.173 INFO 21062 --- [ocp-schedules-19,4096b9563a101fd4,3b5dea8bfb00c730] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:31:32.983 INFO 21062 --- [prometheus-minute-schedule-1,16bfc404e980dd2d,91535e12a96be703] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:33.243 WARN 21062 --- [ocp-async-4,1f4d4eef4d83b344,babfec3a0eb6eeeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:33.270 INFO 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:31:33.271 INFO 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:31:33.271 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:31:33.271 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:31:33.276 INFO 21062 --- [ocp-async-8,c56907684af1277c,4c8c0387b1d92677] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:33.281 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:31:33.281 INFO 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:31:33.281 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:31:33.283 WARN 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:31:33.283 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:31:33.283 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:31:33.284 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:31:33.284 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:31:33.284 INFO 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:31:33.285 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:31:33.285 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:31:33.285 WARN 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:31:33.292 INFO 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:31:33.292 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:31:33.293 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:31:33.308 ERROR 21062 --- [ocp-schedules-30,3d363cf73ed6ea54,9a7ed30b0069a4fd] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:31:34.854 INFO 21062 --- [ocp-schedules-30,980b57c2365507b1,5415e47f31b88cf3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:31:34.938 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:34.939 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:35.243 WARN 21062 --- [ocp-async-6,5e0edeaa47923de1,9b1c2571a50a8b6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:35.276 INFO 21062 --- [ocp-async-5,e7b6c4c7d72b065c,aad3f7156c0a21e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0ee1d8f8b3f85d2a,80170bb131969111] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:36.672 INFO 21062 --- [ocp-schedules-12,713fe3e7aeb1b72d,52135ac1ca449cef] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:31:36.673 INFO 21062 --- [ocp-schedules-12,ce4f010ee03b4b54,3dfb10d84debd7a0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:31:36.677 INFO 21062 --- [ocp-schedules-12,ce4f010ee03b4b54,3dfb10d84debd7a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:36.707 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:31:36.707 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:31:37.244 WARN 21062 --- [ocp-async-7,31d11fa32501440e,f34f8f00f72135e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:37.276 INFO 21062 --- [ocp-async-1,36acc681418ec265,e645b54fa6755f63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:37.939 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:37.940 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:38.505 INFO 21062 --- [ocp-schedules-11,a0a9f9be394b536e,a9f9958ecff3b67a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:31:38.505 INFO 21062 --- [ocp-schedules-23,756371e72a14f794,f76639b7342c89ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:31:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:31:38.590692+08:00, lastProcessTime=2024-11-04T14:31:28.589951+08:00 2024-11-04 14:31:38.667 INFO 21062 --- [ocp-async-8,5b6c212e906288aa,f091959fd5e678d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:31:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:31:38.590692+08:00 2024-11-04 14:31:38.983 INFO 21062 --- [prometheus-minute-schedule-1,59d3a6b58f8bc6e4,f420e1e672c842e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:39.244 WARN 21062 --- [ocp-async-9,83106365563bb75d,982575624519bf87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:39.276 INFO 21062 --- [ocp-async-3,a2d91f8835763990,fa1c58cb6d5c4322] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:39.503 INFO 21062 --- [ocp-schedules-3,a6e460ce29fb543d,ca0d2e5ffce43266] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:31:39.506 INFO 21062 --- [ocp-schedules-3,a6e460ce29fb543d,ca0d2e5ffce43266] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:39.590 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:31:39.592 ERROR 21062 --- [ocp-schedules-3,a6e460ce29fb543d,ca0d2e5ffce43266] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:31:39.592 INFO 21062 --- [ocp-schedules-3,a6e460ce29fb543d,ca0d2e5ffce43266] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:31:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:31:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:31:40.649 INFO 21062 --- [ocp-schedules-20,c6b960be0d712c82,59f1f1a421749ac7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:31:40.940 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:40.940 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:41.244 WARN 21062 --- [ocp-async-8,9c884a59ef925a36,c5630ebb427b24ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:41.277 INFO 21062 --- [ocp-async-7,aee71df949aca696,3310c577662316c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:41.983 INFO 21062 --- [prometheus-minute-schedule-1,389c9825b75f7fb0,15efe63864c2268d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:43.244 WARN 21062 --- [ocp-async-10,a1db0b396b4b25eb,011b5f48b9ebfba3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:43.277 INFO 21062 --- [ocp-async-1,7af02be49ff7d38c,03f83823982837f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:43.941 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:43.942 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:44.893 INFO 21062 --- [ocp-schedules-24,3daf3aa1a434ec5b,a0342710d2b98577] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:31:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e1f69069f4441b14,1c2ebe8f6191245f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:45.244 WARN 21062 --- [ocp-async-9,ff5f1cc63c71c0ba,9f62ace426821499] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:45.277 INFO 21062 --- [ocp-async-2,d62b13f38f5e06ec,2ac90cad3c1fda30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:46.942 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:46.943 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:47.244 WARN 21062 --- [ocp-async-8,7ea88211416ef356,58677b0e1d74e266] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:47.277 INFO 21062 --- [ocp-async-3,ce69655bd2ef0259,3574535dfc74a699] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8813a133cbe68ddc,80446eddf681991d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:31:48.591813+08:00, lastProcessTime=2024-11-04T14:31:38.590692+08:00 2024-11-04 14:31:48.668 INFO 21062 --- [ocp-async-9,c5bf0110018d1905,9a32ac4a69f188cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:31:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:31:48.591813+08:00 2024-11-04 14:31:49.245 WARN 21062 --- [ocp-async-7,8a78eda30c0748a5,fb20ec47e580f45d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:49.277 INFO 21062 --- [ocp-async-4,d4284c4d375207e2,17d9aa33dfe5e403] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:49.943 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:49.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:50.652 INFO 21062 --- [ocp-schedules-4,b432a8aef38f13e5,8771de55fbe5d477] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:31:50.983 INFO 21062 --- [prometheus-minute-schedule-1,e3d7a33bbaf396b1,7c4535e2118fb752] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:51.245 WARN 21062 --- [ocp-async-1,8a7af2a9d60e0b9d,9ee578a95369fd7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:51.277 INFO 21062 --- [ocp-async-6,9e43ffb9fa696e78,173175ea4f1a84a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:51.707 INFO 21062 --- [ocp-schedules-17,b5589b12db5d814d,34cf7b3ccdb822cf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:31:51.713 INFO 21062 --- [ocp-schedules-17,b5589b12db5d814d,34cf7b3ccdb822cf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:31:51.713 INFO 21062 --- [ocp-schedules-17,b5589b12db5d814d,34cf7b3ccdb822cf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:31:52.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:52.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:53.245 WARN 21062 --- [ocp-async-2,04efbcfb110ffe56,f9234dd15b18920b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:53.277 INFO 21062 --- [ocp-async-3,07341d032f1057f5,243f7e14718e854c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1355b4c8a3745888,93062a155e114a08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:54.919 INFO 21062 --- [ocp-schedules-23,2381d0cd8a6d9215,62c1a8293e58b966] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:31:55.245 WARN 21062 --- [ocp-async-7,fcc3dfdecd7496e2,a67d5971df274135] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:55.278 INFO 21062 --- [ocp-async-10,82422ef553bb722f,4eb963fb4e9e3b60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:55.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:55.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e7c9146b8de3dcd5,ef56efe8895d5722] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:31:57.245 WARN 21062 --- [ocp-async-1,251701041f87203f,d80b278610588cc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:57.278 INFO 21062 --- [ocp-async-9,02699d8e2b72a68e,d04d698ddaca348a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:58.500 INFO 21062 --- [ocp-schedules-30,903714c915d17dbe,89e56a8a643fa46d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:31:58.500 INFO 21062 --- [ocp-schedules-11,3ea6f0472274f103,a8e2a773af164509] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:31:58.502 INFO 21062 --- [ocp-schedules-1,7f32da8ffdd8c1b9,ec2ffb3db29ba2e3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:31:58.504 INFO 21062 --- [ocp-schedules-30,903714c915d17dbe,89e56a8a643fa46d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:58.504 INFO 21062 --- [ocp-schedules-11,3ea6f0472274f103,a8e2a773af164509] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:31:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:31:58.505 INFO 21062 --- [ocp-schedules-25,10b33bd7f7c9d230,1925e93e7773923c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:31:58.510 INFO 21062 --- [ocp-schedules-27,aef38bf59e0a5c56,352c2b69313010b9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:31:58.510 INFO 21062 --- [ocp-schedules-27,aef38bf59e0a5c56,352c2b69313010b9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:31:58.511 INFO 21062 --- [ocp-schedules-21,c1a8de38c36ece25,f31a2733e4f8f497] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:31:58.511 INFO 21062 --- [ocp-schedules-21,c1a8de38c36ece25,f31a2733e4f8f497] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:31:58.512 INFO 21062 --- [ocp-schedules-23,4c6cec04ffb503f5,96fcaff31ae795f6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:31:58.512 INFO 21062 --- [ocp-schedules-23,4c6cec04ffb503f5,96fcaff31ae795f6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:31:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:31:58.589078+08:00, lastProcessTime=2024-11-04T14:31:48.591813+08:00 2024-11-04 14:31:58.668 INFO 21062 --- [ocp-async-7,c2ad648797abd8eb,c27f381b35da363f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:31:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:31:58.589078+08:00 2024-11-04 14:31:58.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:31:58.946 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:31:59.213 INFO 21062 --- [ocp-schedules-17,b5b3b9b92e252807,d7bcd498b6d09729] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:31:59.245 WARN 21062 --- [ocp-async-8,d0f592c5692edde8,3aa117981ec3bb4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:31:59.278 INFO 21062 --- [ocp-async-3,cf120910704dbca3,8f6bbe1e7ff3e0d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:31:59.983 INFO 21062 --- [prometheus-minute-schedule-1,35f0e1151971acfa,c2290f40cf851a37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701920, interval=60 2024-11-04 14:32:00.655 INFO 21062 --- [ocp-schedules-14,8e4b97c186fe76c0,b93251ecdb43ec58] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:32:01.246 WARN 21062 --- [ocp-async-7,813841f76d65ed74,670924d0075a07a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:01.278 INFO 21062 --- [ocp-async-10,2706e927639a0225,0879723b55cfc4ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:01.946 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:01.947 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:02.983 INFO 21062 --- [prometheus-minute-schedule-1,55632c01c0166d24,05314f0eacae53b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:03.246 WARN 21062 --- [ocp-async-1,610e3832f2c72527,9b600760fba3bccd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:03.278 INFO 21062 --- [ocp-async-9,a80eead54acb85ba,b9a3f10436912a31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:04.947 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:04.948 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:04.961 INFO 21062 --- [ocp-schedules-25,5d0e7f023134452f,fa440ff6c7b2a7f5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:32:05.246 WARN 21062 --- [ocp-async-8,d5949ca830fddcb8,7331f60a1d941d20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:05.279 INFO 21062 --- [ocp-async-2,7c91b4e302ac3f36,7afb656f73bfafc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8c939cd1bf21eeb4,5cc3a700e4dc3247] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:06.708 INFO 21062 --- [ocp-schedules-12,ce4f010ee03b4b54,3dfb10d84debd7a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:32:06.708 INFO 21062 --- [ocp-schedules-12,cb55a8422420ff99,3eeb53ecfe225105] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:32:06.712 INFO 21062 --- [ocp-schedules-12,cb55a8422420ff99,3eeb53ecfe225105] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:06.742 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:32:06.742 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:32:07.246 WARN 21062 --- [ocp-async-7,375dd726a1dde327,e1112e2919a42202] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:07.279 INFO 21062 --- [ocp-async-5,7bf2d373e2ac039c,aa0ad0f5bc756255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:07.948 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:07.949 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:08.505 INFO 21062 --- [ocp-schedules-11,3ea6f0472274f103,a8e2a773af164509] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:32:08.505 INFO 21062 --- [ocp-schedules-30,903714c915d17dbe,89e56a8a643fa46d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:32:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:32:08.589081+08:00, lastProcessTime=2024-11-04T14:31:58.589078+08:00 2024-11-04 14:32:08.668 INFO 21062 --- [ocp-async-8,fd2244ab810af36b,ccd3d3a56e811190] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:32:08.802 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:32:08.589081+08:00 2024-11-04 14:32:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f06aaba9ae5847bb,f138eea7116d622c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:09.247 WARN 21062 --- [ocp-async-1,71bfbd9f1dd9d351,2e9acd72969a9567] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:09.279 INFO 21062 --- [ocp-async-9,9ebb99cfe778c68a,76bb20fbfd03de35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:32:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:32:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:32:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:32:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:32:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:32:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:32:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:32:10.658 INFO 21062 --- [ocp-schedules-26,84b2c95892d6b772,f62ca02e83ea3413] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:32:10.949 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:10.950 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:11.247 WARN 21062 --- [ocp-async-8,c18953dfd3d5a3f4,df70d2a3c30889a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:11.279 INFO 21062 --- [ocp-async-2,77e9018c1eca1330,90e18e619eb2ea51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:11.595 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=1, affectRows=1 2024-11-04 14:32:11.595 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=1, validCount=1 2024-11-04 14:32:11.596 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=5, affectRows=5 2024-11-04 14:32:11.596 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=5, validCount=5 2024-11-04 14:32:11.596 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=19, affectRows=19 2024-11-04 14:32:11.596 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=23, affectRows=23 2024-11-04 14:32:11.596 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=17, affectRows=17 2024-11-04 14:32:11.596 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=5, affectRows=5 2024-11-04 14:32:11.596 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=5, validCount=5 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=19, validCount=19 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=23, validCount=23 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=17, validCount=17 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=25, affectRows=25 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=33, affectRows=33 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=25, validCount=25 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=62, affectRows=62 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=33, validCount=33 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=61, validCount=61 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=57, affectRows=57 2024-11-04 14:32:11.597 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=51, affectRows=51 2024-11-04 14:32:11.598 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=51, affectRows=51 2024-11-04 14:32:11.598 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=56, validCount=56 2024-11-04 14:32:11.598 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701860, nodeCount=47, affectRows=47 2024-11-04 14:32:11.598 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=50, validCount=50 2024-11-04 14:32:11.598 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=51, validCount=51 2024-11-04 14:32:11.598 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701860, nodeCount=47, validCount=47 2024-11-04 14:32:11.716 INFO 21062 --- [ocp-schedules-19,c5eb10cbcf931b7c,3e5bbbe80b875839] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:32:11.723 INFO 21062 --- [ocp-schedules-19,c5eb10cbcf931b7c,3e5bbbe80b875839] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:32:11.723 INFO 21062 --- [ocp-schedules-19,c5eb10cbcf931b7c,3e5bbbe80b875839] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:32:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8a4ac5eca199c95e,9c0f22b87711b39b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:13.247 WARN 21062 --- [ocp-async-7,506d7e69868f9fbc,64220b59e6b0ed90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:13.279 INFO 21062 --- [ocp-async-5,0a2ddde1af35b70e,4325cc66871bab90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:13.950 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:13.951 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:13.984 INFO 21062 --- [prometheus-cache-monitor1,eab267fa5c0ec8c3,7f716b92cf029280] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:32:13.984 INFO 21062 --- [prometheus-cache-monitor1,eab267fa5c0ec8c3,7f716b92cf029280] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:32:14.983 INFO 21062 --- [prometheus-minute-schedule-1,37ff6c4134ca887d,5329156a29da1741] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:15.000 INFO 21062 --- [ocp-schedules-24,1501bb65b96a8c15,613e16513ac0bb74] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:32:15.247 WARN 21062 --- [ocp-async-5,2518bcb3602a66e7,c71dfa1a81ef92a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:15.279 INFO 21062 --- [ocp-async-10,394038340d73e557,c3cdc53d2dd3c148] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:16.951 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:16.952 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:17.247 WARN 21062 --- [ocp-async-8,766a7641b0c47818,b3a2b50391a4d482] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:17.280 INFO 21062 --- [ocp-async-6,47a64971fac002e1,590e711e9065a6be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4a4eda0f2202acc1,013b2deff1a53bce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:32:18.589728+08:00, lastProcessTime=2024-11-04T14:32:08.589081+08:00 2024-11-04 14:32:18.668 INFO 21062 --- [ocp-async-5,7f32edc06e0effb8,36d9c831d9eefb8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:32:18.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:32:18.589728+08:00 2024-11-04 14:32:19.247 WARN 21062 --- [ocp-async-7,5254b29b33be8433,c7d837bcf8ecc75b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:19.280 INFO 21062 --- [ocp-async-1,d0a873585220104f,abb04ef6e38a7982] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:19.952 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:19.953 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:20.661 INFO 21062 --- [ocp-schedules-27,b84f88cde73273b3,dd85d89f5d391406] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:32:20.983 INFO 21062 --- [prometheus-minute-schedule-1,93ba9e4c214c8a8e,d65483539901a218] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:21.248 WARN 21062 --- [ocp-async-1,42b9352c7d7de371,b49894129d4c0267] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:21.280 INFO 21062 --- [ocp-async-8,2eb3f75adae7edcd,31b398f7c595605e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:22.953 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:22.954 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:23.248 WARN 21062 --- [ocp-async-10,971d05fcc837de86,cdff2dbfb08f70a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:23.280 INFO 21062 --- [ocp-async-6,f0cb763aa0d82870,480260c32c780f1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:23.983 INFO 21062 --- [prometheus-minute-schedule-1,35c9e9e2a0a6d309,5d422c52fd3335f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:25.042 INFO 21062 --- [ocp-schedules-30,1cf9a2ae0c142d58,cf3e7567f132cede] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:32:25.248 WARN 21062 --- [ocp-async-6,e28751d5a4886710,355e80d67ab0bc5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:25.280 INFO 21062 --- [ocp-async-3,5b16712d67e6f9fe,2974a6e9c2208b59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:25.954 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:25.955 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:26.983 INFO 21062 --- [prometheus-minute-schedule-1,45a113cc8d5792ce,be8484e3c2e81708] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b168d41c3a096aaf,1de7e6211d42a215] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:32:27.248 WARN 21062 --- [ocp-async-3,71b46539fc274b6f,5debe8674d143aab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:27.280 INFO 21062 --- [ocp-async-5,8eeac833b67a440d,c8a20e5db9b3b475] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:28.498 INFO 21062 --- [ocp-schedules-19,894615780b434cec,344764abe4016260] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:32:28.500 INFO 21062 --- [ocp-schedules-27,b43722dfb02d3177,d25f29dfaa3dbd59] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:32:28.500 INFO 21062 --- [ocp-schedules-4,53adc0964e2252e7,cc8ffb9447251cef] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:32:28.500 INFO 21062 --- [ocp-schedules-8,2b97c44a2d7949c3,65a158797a2fd968] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:32:28.502 INFO 21062 --- [ocp-schedules-18,0ff91eec35f8d450,c084543e765879c2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:32:28.503 INFO 21062 --- [ocp-schedules-19,894615780b434cec,344764abe4016260] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:32:28.503 INFO 21062 --- [ocp-schedules-27,b43722dfb02d3177,d25f29dfaa3dbd59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:28.503 INFO 21062 --- [ocp-schedules-8,2b97c44a2d7949c3,65a158797a2fd968] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:32:28.504 INFO 21062 --- [ocp-schedules-5,7195d714b5bc5d2e,3b6569db96d5d8c1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:32:28.505 INFO 21062 --- [ocp-schedules-25,c6e5e6dbfb6265b4,635c29a1e371069d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:32:28.506 INFO 21062 --- [ocp-schedules-4,53adc0964e2252e7,cc8ffb9447251cef] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:32:28.508 INFO 21062 --- [ocp-schedules-25,c6e5e6dbfb6265b4,635c29a1e371069d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:28.510 INFO 21062 --- [ocp-schedules-22,9beddfc5147cb009,3acc59f3e2cf8d7f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:32:28.510 INFO 21062 --- [ocp-schedules-22,9beddfc5147cb009,3acc59f3e2cf8d7f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:32:28.511 INFO 21062 --- [ocp-schedules-1,c03136fb44abdf16,656f7ea697af43cc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:32:28.511 INFO 21062 --- [ocp-schedules-1,c03136fb44abdf16,656f7ea697af43cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:32:28.511 INFO 21062 --- [ocp-schedules-3,e4eef19905835200,929b967916fd3326] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:32:28.511 INFO 21062 --- [ocp-schedules-3,e4eef19905835200,929b967916fd3326] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:32:28.537 INFO 21062 --- [ocp-schedules-11,b4bf74b83a9f5aa7,43d85a5ba027aa99] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:32:28.537 INFO 21062 --- [ocp-schedules-11,b4bf74b83a9f5aa7,43d85a5ba027aa99] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:32:28.578 INFO 21062 --- [ocp-schedules-21,acb078224a548b3e,a3171b30f9f12809] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:32:28.578 INFO 21062 --- [ocp-schedules-21,acb078224a548b3e,a3171b30f9f12809] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:32:28.580 INFO 21062 --- [pool-12-thread-4,e04d87b512ebb469,c1323a0f2f48c7b7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:32:28.580 INFO 21062 --- [pool-12-thread-3,e04d87b512ebb469,f3153b9964c609b3] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:32:28.581 INFO 21062 --- [pool-12-thread-2,e04d87b512ebb469,0586bcb3aa1dbab9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:32:28.581 INFO 21062 --- [pool-12-thread-1,e04d87b512ebb469,918bf53238cccee0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:32:28.586 INFO 21062 --- [ForkJoinPool-11-worker-13,e04d87b512ebb469,05d3e43a9064b34f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:32:28.586 INFO 21062 --- [ForkJoinPool-11-worker-13,e04d87b512ebb469,05d3e43a9064b34f] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:32:28.586 INFO 21062 --- [ForkJoinPool-11-worker-13,e04d87b512ebb469,05d3e43a9064b34f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:32:28.586 INFO 21062 --- [pool-12-thread-4,e04d87b512ebb469,c1323a0f2f48c7b7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:32:28.586 INFO 21062 --- [pool-12-thread-3,e04d87b512ebb469,f3153b9964c609b3] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:32:28.586 INFO 21062 --- [pool-12-thread-3,e04d87b512ebb469,f3153b9964c609b3] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:32:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:32:28.589403+08:00, lastProcessTime=2024-11-04T14:32:18.589728+08:00 2024-11-04 14:32:28.609 INFO 21062 --- [pool-12-thread-2,e04d87b512ebb469,0586bcb3aa1dbab9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:32:28.609 INFO 21062 --- [pool-12-thread-2,e04d87b512ebb469,0586bcb3aa1dbab9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:32:28.610 INFO 21062 --- [pool-12-thread-2,e04d87b512ebb469,0586bcb3aa1dbab9] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:32:28.613 INFO 21062 --- [ForkJoinPool-11-worker-13,e04d87b512ebb469,3e8fb05fc2a5657c] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:32:28.613 INFO 21062 --- [pool-12-thread-1,e04d87b512ebb469,918bf53238cccee0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:32:28.668 INFO 21062 --- [ocp-async-2,2e10b9e3b0b4cbaa,191fce50690c85cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:32:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:32:28.589403+08:00 2024-11-04 14:32:28.853 INFO 21062 --- [ocp-schedules-4,4fed65c50512ea89,1f5ec8589d9f856b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:32:28.856 INFO 21062 --- [ocp-schedules-5,b377997e904d4422,1e47682cfd45c5f7] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 14:32:28.856 INFO 21062 --- [ocp-schedules-5,b377997e904d4422,1e47682cfd45c5f7] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 14:32:28.856 INFO 21062 --- [ocp-schedules-5,b377997e904d4422,1e47682cfd45c5f7] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=396 2024-11-04 14:32:28.874 INFO 21062 --- [ocp-schedules-5,b377997e904d4422,1e47682cfd45c5f7] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=396 2024-11-04 14:32:28.955 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:28.956 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:29.003 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:32:29.020 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:32:29.022 WARN 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:32:29.023 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:32:29.024 WARN 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:32:29.030 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:32:29.031 WARN 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:32:29.032 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:32:29.033 WARN 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:32:29.038 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:32:29.040 WARN 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:32:29.040 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:32:29.041 WARN 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:32:29.047 INFO 21062 --- [ocp-schedules-19,fb3ec2a57805947f,083acb9db4538747] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:32:29.061 INFO 21062 --- [ocp-schedules-11,ce25523989adf754,f4948e480a503a18] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:32:29.064 INFO 21062 --- [ocp-schedules-11,ce25523989adf754,f4948e480a503a18] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:29.065 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:32:29.101 INFO 21062 --- [ocp-schedules-17,13ebd85859d68266,0783a8eebc17c730] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 14:32:29.104 INFO 21062 --- [ocp-schedules-17,13ebd85859d68266,0783a8eebc17c730] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:29.139 INFO 21062 --- [ocp-schedules-24,4cfd474b43189888,d08bd6aeee8d7c86] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:32:29.139 INFO 21062 --- [ocp-schedules-24,4cfd474b43189888,d08bd6aeee8d7c86] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:32:29.139 ERROR 21062 --- [ocp-schedules-17,13ebd85859d68266,0783a8eebc17c730] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:32:29.139 INFO 21062 --- [ocp-schedules-17,13ebd85859d68266,0783a8eebc17c730] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 14:32:29.215 INFO 21062 --- [ocp-schedules-9,7570c1b06b863414,c8cd75aeff7aa4d5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:32:29.248 WARN 21062 --- [ocp-async-6,5b48345c50c95fd6,f0390b23b05b8cf6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:29.249 INFO 21062 --- [ocp-schedules-2,15588ab50f29aba5,8dd8e4ece2c0ae71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:32:29.254 INFO 21062 --- [ocp-schedules-2,15588ab50f29aba5,8dd8e4ece2c0ae71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:32:29.254 INFO 21062 --- [ocp-schedules-2,15588ab50f29aba5,8dd8e4ece2c0ae71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:32:29.281 INFO 21062 --- [ocp-async-7,cbce7df2c2e50402,78d2d65acb299445] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:29.600 INFO 21062 --- [ocp-schedules-16,83cc0558e3a16575,c84e055451cd05b8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:32:29.600 INFO 21062 --- [ocp-schedules-16,83cc0558e3a16575,c84e055451cd05b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:32:29.601 INFO 21062 --- [ocp-schedules-16,83cc0558e3a16575,c84e055451cd05b8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:32:29.601 INFO 21062 --- [ocp-schedules-16,83cc0558e3a16575,c84e055451cd05b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:32:29.602 INFO 21062 --- [ocp-schedules-16,83cc0558e3a16575,c84e055451cd05b8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:32:29.602 INFO 21062 --- [ocp-schedules-16,83cc0558e3a16575,c84e055451cd05b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:32:29.858 INFO 21062 --- [ocp-schedules-4,425df51aef8c9b1b,2686bf856e5dd12a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:32:29.858 INFO 21062 --- [ocp-schedules-4,425df51aef8c9b1b,2686bf856e5dd12a] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:32:29.886 INFO 21062 --- [pool-43-thread-1,425df51aef8c9b1b,b6e3c736002d22ec] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:32:29.916 WARN 21062 --- [ocp-schedules-4,425df51aef8c9b1b,2686bf856e5dd12a] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:32:29.916 INFO 21062 --- [ocp-schedules-4,425df51aef8c9b1b,2686bf856e5dd12a] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:32:29.983 INFO 21062 --- [prometheus-minute-schedule-1,fbaa1bff90af9ba0,db6a07c91ecbdddd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:32:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:32:30.664 INFO 21062 --- [ocp-schedules-3,13b88dbd68a65513,2fd1c25e974c62f3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:32:31.249 WARN 21062 --- [ocp-async-7,d4161dbd64905aeb,7c443bc235649d87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:31.281 INFO 21062 --- [ocp-async-1,f9c47384ed82fd56,b017f5d528bc87ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:31.725 INFO 21062 --- [ocp-schedules-24,12615afe65fcfa94,f5168afa767ea559] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:32:31.730 INFO 21062 --- [ocp-schedules-24,12615afe65fcfa94,f5168afa767ea559] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:32:31.730 INFO 21062 --- [ocp-schedules-24,12615afe65fcfa94,f5168afa767ea559] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:32:31.801 INFO 21062 --- [ocp-schedules-3,b3f040ac1b8e761c,c08e3f833306827f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:32:31.801 INFO 21062 --- [ocp-schedules-3,b3f040ac1b8e761c,c08e3f833306827f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:32:31.828 INFO 21062 --- [ocp-schedules-3,b3f040ac1b8e761c,c08e3f833306827f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:32:31.828 INFO 21062 --- [ocp-schedules-3,b3f040ac1b8e761c,c08e3f833306827f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:32:31.828 INFO 21062 --- [ocp-schedules-7,3f55fdacf5c35975,9465ab071b50aba7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:32:31.956 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:31.957 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:32.013 INFO 21062 --- [ocp-schedules-15,f19638516df29e4a,f6c2b2e927b5e473] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:32:32.013+08:00, clusters=[1] 2024-11-04 14:32:32.016 INFO 21062 --- [ocp-schedules-15,f19638516df29e4a,f6c2b2e927b5e473] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:32:32.199 INFO 21062 --- [ocp-schedules-30,f1228c6d517b4d9a,73da29a3260be3b5] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:32:32.200 INFO 21062 --- [ocp-schedules-29,e23afe70266c2dd6,98925966d5919fd0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:32:32.200+08:00, clusters=[1] 2024-11-04 14:32:32.203 INFO 21062 --- [ocp-schedules-30,f1228c6d517b4d9a,73da29a3260be3b5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:32.203 INFO 21062 --- [ocp-schedules-29,e23afe70266c2dd6,98925966d5919fd0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:32:32.257 ERROR 21062 --- [ocp-schedules-30,f1228c6d517b4d9a,73da29a3260be3b5] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:32:32.257 INFO 21062 --- [ocp-schedules-30,f1228c6d517b4d9a,73da29a3260be3b5] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:32:32.983 INFO 21062 --- [prometheus-minute-schedule-1,44b2ad9569b6f0d3,5d051e7212fddaf2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:33.249 WARN 21062 --- [ocp-async-5,3560d40a778c1fb3,6e765375ceeab494] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:33.281 INFO 21062 --- [ocp-async-10,0fc5e45c0513e676,df41b6b608fb9d73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:33.317 INFO 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:32:33.317 INFO 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:32:33.317 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:32:33.317 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:32:33.327 INFO 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:32:33.327 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:32:33.327 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:32:33.328 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:32:33.329 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:32:33.329 WARN 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:32:33.329 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:32:33.329 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:32:33.329 INFO 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:32:33.330 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:32:33.330 WARN 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:32:33.330 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:32:33.337 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:32:33.337 INFO 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:32:33.337 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:32:33.349 ERROR 21062 --- [ocp-schedules-16,9442ee0c1067ce5a,00a04b59e8317ad9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:32:34.957 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:34.958 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:35.075 INFO 21062 --- [ocp-schedules-21,5610eecb21fcc247,e98525bfb3e42ec3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:32:35.249 WARN 21062 --- [ocp-async-6,b7153f8a049acdf5,f28894c7d8898922] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:35.281 INFO 21062 --- [ocp-async-9,f4dda30cd0f6a85f,ab4deb02cdb023bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0f267806720abd3a,e6050e5456d61e67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:36.743 INFO 21062 --- [ocp-schedules-12,cb55a8422420ff99,3eeb53ecfe225105] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:32:36.743 INFO 21062 --- [ocp-schedules-12,ff56503e85a6129c,81a5ff49eb63b564] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:32:36.748 INFO 21062 --- [ocp-schedules-12,ff56503e85a6129c,81a5ff49eb63b564] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:36.776 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:32:36.776 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:32:37.249 WARN 21062 --- [ocp-async-7,015383ad95e55a85,fb162c00ebced1ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:37.281 INFO 21062 --- [ocp-async-3,8b563e7afcafa1cb,147152fdf6714372] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:37.958 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:37.959 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:38.504 INFO 21062 --- [ocp-schedules-8,2b97c44a2d7949c3,65a158797a2fd968] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:32:38.504 INFO 21062 --- [ocp-schedules-27,b43722dfb02d3177,d25f29dfaa3dbd59] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:32:38.508 INFO 21062 --- [ocp-schedules-25,c6e5e6dbfb6265b4,635c29a1e371069d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:32:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:32:38.590569+08:00, lastProcessTime=2024-11-04T14:32:28.589403+08:00 2024-11-04 14:32:38.669 INFO 21062 --- [ocp-async-8,54c025f45b13e9c9,dd115bc8fca9c0b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:32:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:32:38.590569+08:00 2024-11-04 14:32:38.983 INFO 21062 --- [prometheus-minute-schedule-1,531e4236c75bf40b,7f6ed8551c04fbb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:39.065 INFO 21062 --- [ocp-schedules-11,ce25523989adf754,f4948e480a503a18] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:32:39.249 WARN 21062 --- [ocp-async-10,8eee094a1766626b,0ac7c04a43dca900] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:39.281 INFO 21062 --- [ocp-async-6,f7fef943b0d64339,91dfaffc05873a75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:39.592 INFO 21062 --- [ocp-schedules-8,5f29f96e0ea219be,872f9a3ed3f03ea8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:32:39.596 INFO 21062 --- [ocp-schedules-8,5f29f96e0ea219be,872f9a3ed3f03ea8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:39.684 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:32:39.687 ERROR 21062 --- [ocp-schedules-8,5f29f96e0ea219be,872f9a3ed3f03ea8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:32:39.687 INFO 21062 --- [ocp-schedules-8,5f29f96e0ea219be,872f9a3ed3f03ea8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:32:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:32:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:32:40.668 INFO 21062 --- [ocp-schedules-19,2aadcf3608004a29,c6f62dea7a81cae9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:32:40.959 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:40.960 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:41.249 WARN 21062 --- [ocp-async-9,94b00fe51f0f0836,b1b3b731450bc89b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:41.281 INFO 21062 --- [ocp-async-7,3a42e0d50d1f565b,249767c4dfc43f19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:32:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:32:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:32:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:32:41.802 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:32:41.831 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:32:41.831 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-11-04 14:32:41.983 INFO 21062 --- [prometheus-minute-schedule-1,85d717c7b9d46d5d,1c49a14fca3fb792] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:43.250 WARN 21062 --- [ocp-async-3,889a1c83ca7ff17b,0e07b01a2493927b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:43.282 INFO 21062 --- [ocp-async-4,bbb608c21a597002,fd703b1d8ed017f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:43.960 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:43.961 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3aa357bb5211ae6f,eff9e3bb3b4d1034] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:45.124 INFO 21062 --- [ocp-schedules-27,fc793cdb0ef98e8c,6487d61ac183dec8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:32:45.250 WARN 21062 --- [ocp-async-5,f3700f5ccaa021d1,58c779aacc88abe9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:45.282 INFO 21062 --- [ocp-async-1,eb252511c0bc3e6f,366e1a53710fa7bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:46.961 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:46.962 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:47.250 WARN 21062 --- [ocp-async-6,159322cbadb7b0ad,50c5d24259915fa8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:47.282 INFO 21062 --- [ocp-async-8,d2969711634038d3,6f89434e09f0b040] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:47.983 INFO 21062 --- [prometheus-minute-schedule-1,1c598b824eea41e4,122e67694e1ab353] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:32:48.590571+08:00, lastProcessTime=2024-11-04T14:32:38.590569+08:00 2024-11-04 14:32:48.669 INFO 21062 --- [ocp-async-5,6844e06e8a340f97,3665d6ed7c0feb30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:32:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:32:48.590571+08:00 2024-11-04 14:32:49.250 WARN 21062 --- [ocp-async-3,3aaa38de381c54a3,adebf3c70d8795ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:49.282 INFO 21062 --- [ocp-async-4,c93bbde6dc92fcc8,ea4b5851a79fed81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:49.962 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:49.964 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:50.670 INFO 21062 --- [ocp-schedules-17,73bab7bc058827e2,182c4966951e7155] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:32:50.983 INFO 21062 --- [prometheus-minute-schedule-1,74d529ad1a325453,777c289cda0e3e1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:51.250 WARN 21062 --- [ocp-async-10,c6a9134ecbbac030,144a1c59f2de24bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:51.283 INFO 21062 --- [ocp-async-1,81bf21086c851144,285de4cf879a514d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:51.733 INFO 21062 --- [ocp-schedules-11,3a125494f1e24ede,60bab4da8b034a7d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:32:51.738 INFO 21062 --- [ocp-schedules-11,3a125494f1e24ede,60bab4da8b034a7d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:32:51.738 INFO 21062 --- [ocp-schedules-11,3a125494f1e24ede,60bab4da8b034a7d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:32:52.964 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:52.965 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:53.251 WARN 21062 --- [ocp-async-9,8d6d144742d085f9,bb4337bc6c6c5dd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:53.283 INFO 21062 --- [ocp-async-8,55beaac4fb9613b9,90d9f2cb955b8b88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ed7528e799c19421,7277b3415c54ff67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:55.161 INFO 21062 --- [ocp-schedules-19,8f141d9fa1ca9d20,1b7a74aeab88d4b6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:32:55.251 WARN 21062 --- [ocp-async-3,b2b2006a5c35570d,4ea4148e8ee34e97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:55.283 INFO 21062 --- [ocp-async-2,dce653a66816b9da,b68ba272b0ed7d2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:55.965 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:55.966 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7e5a6205ffd13f2b,7f529e49e2182af0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:32:57.251 WARN 21062 --- [ocp-async-10,fc2d88de78131c65,9f0e4ff2b20e8c46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:57.283 INFO 21062 --- [ocp-async-5,c30757a1e3e120bb,c827baf25d859964] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:58.500 INFO 21062 --- [ocp-schedules-26,4ad5a5d362924ad3,ad5f0b1f432fa8e9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:32:58.500 INFO 21062 --- [ocp-schedules-20,ea50030c5d323591,d25373f44816d018] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:32:58.502 INFO 21062 --- [ocp-schedules-21,276eba752b489304,a29cc1e3fbabfed4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:32:58.504 INFO 21062 --- [ocp-schedules-26,4ad5a5d362924ad3,ad5f0b1f432fa8e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:58.504 INFO 21062 --- [ocp-schedules-20,ea50030c5d323591,d25373f44816d018] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:32:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:32:58.505 INFO 21062 --- [ocp-schedules-24,2464277e3e8a8d95,56d0d0b02bfaf252] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:32:58.512 INFO 21062 --- [ocp-schedules-13,2c07f89ec52f7bdd,d1d14e441323419b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:32:58.512 INFO 21062 --- [ocp-schedules-13,2c07f89ec52f7bdd,d1d14e441323419b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:32:58.512 INFO 21062 --- [ocp-schedules-27,2a95061ec10e4cc9,78a907089604af4e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:32:58.512 INFO 21062 --- [ocp-schedules-2,21122094b1026743,5bc1e9f8af2a9fc2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:32:58.513 INFO 21062 --- [ocp-schedules-27,2a95061ec10e4cc9,78a907089604af4e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:32:58.513 INFO 21062 --- [ocp-schedules-2,21122094b1026743,5bc1e9f8af2a9fc2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:32:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:32:58.589875+08:00, lastProcessTime=2024-11-04T14:32:48.590571+08:00 2024-11-04 14:32:58.669 INFO 21062 --- [ocp-async-3,e994448cb5fca29d,e122bdec8e0a4458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:32:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:32:58.589875+08:00 2024-11-04 14:32:58.966 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:32:58.967 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:32:59.218 INFO 21062 --- [ocp-schedules-3,f0cc234447b09e50,6c85e2f4a147e2cd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:32:59.251 WARN 21062 --- [ocp-async-9,b7fe93450a2bd4dd,74a8bf871d6d48f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:32:59.283 INFO 21062 --- [ocp-async-8,69285de0b98e4399,48b108df00765654] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:32:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1b85218a559e8aaa,e8ef93f4fd258c20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730701980, interval=60 2024-11-04 14:33:00.674 INFO 21062 --- [ocp-schedules-22,9e047b2a67841e0c,2eee7af0074223f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:33:01.251 WARN 21062 --- [ocp-async-3,f899f7d41b6fa5c8,83a64ca16fb6f714] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:01.283 INFO 21062 --- [ocp-async-2,471ea1b0da6293a0,33a8e9c35c663fce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:01.967 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:01.968 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a6bb76cfe7fd18ed,8e0102dd0cb82861] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:03.252 WARN 21062 --- [ocp-async-10,50bf76323cd0ee1f,9faee1996b9b6442] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:03.284 INFO 21062 --- [ocp-async-5,34f7ff6656010cad,018c7331eac99c18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:04.969 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:04.969 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:05.213 INFO 21062 --- [ocp-schedules-5,76df1925881aa42f,13f749e7e1f101f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:33:05.252 WARN 21062 --- [ocp-async-9,c9e8d7b198f4fd02,b75f097f36c1dfb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:05.284 INFO 21062 --- [ocp-async-6,d7df1f3bb2eee73f,2276fd4b2fd33f92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f7bae3f63b2b95da,f39d59de53d0345a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:06.777 INFO 21062 --- [ocp-schedules-12,ff56503e85a6129c,81a5ff49eb63b564] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:33:06.777 INFO 21062 --- [ocp-schedules-12,d66b8c5c718425de,a86600867752ec58] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:33:06.781 INFO 21062 --- [ocp-schedules-12,d66b8c5c718425de,a86600867752ec58] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:06.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:33:06.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:33:07.252 WARN 21062 --- [ocp-async-8,65f99047caad7862,7cc671b94c254fcf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:07.284 INFO 21062 --- [ocp-async-7,bb8bc4491012a971,8e94c64f37597ff0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:07.970 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:07.970 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:08.505 INFO 21062 --- [ocp-schedules-20,ea50030c5d323591,d25373f44816d018] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:33:08.505 INFO 21062 --- [ocp-schedules-26,4ad5a5d362924ad3,ad5f0b1f432fa8e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:33:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:33:08.589952+08:00, lastProcessTime=2024-11-04T14:32:58.589875+08:00 2024-11-04 14:33:08.669 INFO 21062 --- [ocp-async-9,69e53028b38034ff,b2353fa5f581c610] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:33:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:33:08.589952+08:00 2024-11-04 14:33:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b04ee8013f1a35e2,7d6b89baafd0a911] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:09.252 WARN 21062 --- [ocp-async-2,24652e811c08661e,d695b2d26cbea3a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:09.284 INFO 21062 --- [ocp-async-5,95e799824d564195,28a7821c22df4f42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:33:10.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:33:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:33:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:33:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:33:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:33:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:33:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:33:10.676 INFO 21062 --- [ocp-schedules-23,dde62d40544bec04,763457c7b7e33372] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:33:10.971 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:10.971 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:11.252 WARN 21062 --- [ocp-async-9,138c677fc3d79a0a,d6e375f032411594] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:11.284 INFO 21062 --- [ocp-async-8,4162b9d566a97a0d,54040584722a511e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=21, affectRows=21 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=4, affectRows=4 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=4, validCount=4 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=9, affectRows=9 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=28, affectRows=28 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=8, validCount=8 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=21, validCount=21 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=28, validCount=28 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=35, affectRows=35 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=37, affectRows=37 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=38, affectRows=38 2024-11-04 14:33:11.598 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=35, validCount=35 2024-11-04 14:33:11.599 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=37, validCount=37 2024-11-04 14:33:11.599 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=38, validCount=38 2024-11-04 14:33:11.599 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=67, affectRows=67 2024-11-04 14:33:11.599 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=66, validCount=66 2024-11-04 14:33:11.599 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=70, affectRows=70 2024-11-04 14:33:11.599 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701920, nodeCount=87, affectRows=87 2024-11-04 14:33:11.599 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=70, validCount=70 2024-11-04 14:33:11.600 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701920, nodeCount=86, validCount=86 2024-11-04 14:33:11.740 INFO 21062 --- [ocp-schedules-29,904de6bed4e1b91f,4b6ec8501ab33e1e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:33:11.746 INFO 21062 --- [ocp-schedules-29,904de6bed4e1b91f,4b6ec8501ab33e1e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:33:11.746 INFO 21062 --- [ocp-schedules-29,904de6bed4e1b91f,4b6ec8501ab33e1e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:33:11.983 INFO 21062 --- [prometheus-minute-schedule-1,3ffc61d8122d340d,a7b7d773dc560468] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:13.253 WARN 21062 --- [ocp-async-8,2220752e34fde826,e0b0ede4ec1a62bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:13.285 INFO 21062 --- [ocp-async-7,62d94e57f9c4cb12,d1e03e98ac319278] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:13.972 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:13.973 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:13.984 INFO 21062 --- [prometheus-cache-monitor1,a1df4e4df6c9a72d,bf6f279b76924619] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:33:13.984 INFO 21062 --- [prometheus-cache-monitor1,a1df4e4df6c9a72d,bf6f279b76924619] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:33:14.291 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:27:14.267, end=2024-11-04T14:32:14.289} 2024-11-04 14:33:14.292 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:27:14.267, end=2024-11-04T14:32:14.289} 2024-11-04 14:33:14.296 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:33:14.296 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:33:14.296 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:33:14.296 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:33:14.983 INFO 21062 --- [prometheus-minute-schedule-1,adb4fc7f3e2a4173,6983bb38b8d44bbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:15.131 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:29:15.109, end=2024-11-04T14:31:15.129} 2024-11-04 14:33:15.136 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:33:15.136 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:33:15.244 INFO 21062 --- [ocp-schedules-8,d755f46fd89efaf4,013d2536187278bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:33:15.253 WARN 21062 --- [ocp-async-2,4bd897d53696b2d8,e43aba1570b879e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:15.285 INFO 21062 --- [ocp-async-10,e43992a87767eb47,46f6283365531fd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:16.973 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:16.974 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:17.253 WARN 21062 --- [ocp-async-10,1ddd1b7dd7e5e5e9,3d898b7564f94467] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:17.285 INFO 21062 --- [ocp-async-1,7464079469a12a5b,dec6413523e8f024] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:17.983 INFO 21062 --- [prometheus-minute-schedule-1,6487c2cce10d2b16,17b54b4e2b305742] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:33:18.591187+08:00, lastProcessTime=2024-11-04T14:33:08.589952+08:00 2024-11-04 14:33:18.669 INFO 21062 --- [ocp-async-2,2bffa49317a96918,c09216a84565fc63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:33:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:33:18.591187+08:00 2024-11-04 14:33:19.253 WARN 21062 --- [ocp-async-8,ffcdc74fa464a75d,13ad8f4ed6bc6c9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:19.285 INFO 21062 --- [ocp-async-7,f6f4fdf57528ba5c,28e40f1c966b183a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:19.974 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:19.975 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:20.679 INFO 21062 --- [ocp-schedules-29,bc7641df289a1743,1845903fcc1f31b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:33:20.983 INFO 21062 --- [prometheus-minute-schedule-1,d2b91102f262e037,62c9ee94f427fd3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:21.253 WARN 21062 --- [ocp-async-7,c078897d19bb2bb5,ff8cb3e84a985727] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:21.285 INFO 21062 --- [ocp-async-10,55f0f196eade32ec,cf071c4a87f8e2f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:22.975 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:22.976 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:23.254 WARN 21062 --- [ocp-async-10,eaf3055ada75d06d,46b91c123aa5839e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:23.286 INFO 21062 --- [ocp-async-1,8e38174a89ef8d2b,4c148c81dd21b70e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:23.983 INFO 21062 --- [prometheus-minute-schedule-1,56147fc4c11cee3a,2598ada016e61a44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:25.254 WARN 21062 --- [ocp-async-8,7fd692134af90e91,617f727b68773d6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:25.286 INFO 21062 --- [ocp-async-3,8987bb04820b0b50,1a42fcfaeb88a6e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:25.287 INFO 21062 --- [ocp-schedules-9,9d3f05a926bd1a5c,449fc6a6df7648f0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:33:25.976 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:25.977 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:26.983 INFO 21062 --- [prometheus-minute-schedule-1,5cf154d870702aa9,650689adbfb8f9fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:26.984 INFO 21062 --- [prometheus-hour-schedule-1,6bfd9aa4b549a29b,c29357972958b241] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:33:27.254 WARN 21062 --- [ocp-async-3,7300bfe241f3d8e2,59b05fd07000f199] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:27.286 INFO 21062 --- [ocp-async-2,7eba13c042dc417a,32c733265949d372] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:28.498 INFO 21062 --- [ocp-schedules-24,ab7a981cb2b7816d,2967d865b8191397] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:33:28.500 INFO 21062 --- [ocp-schedules-4,8711ddbf12e6e600,137c4e3fd8568a74] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:33:28.500 INFO 21062 --- [ocp-schedules-19,0cabd5c3d9145dcf,7bf9a8c07096ad64] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:33:28.500 INFO 21062 --- [ocp-schedules-15,b167d0c807a52f44,bf5c04830e9152c4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:33:28.502 INFO 21062 --- [ocp-schedules-7,058039d58443b037,d19dc29f2b650d3d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:33:28.503 INFO 21062 --- [ocp-schedules-24,ab7a981cb2b7816d,2967d865b8191397] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:33:28.503 INFO 21062 --- [ocp-schedules-19,0cabd5c3d9145dcf,7bf9a8c07096ad64] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:28.504 INFO 21062 --- [ocp-schedules-15,b167d0c807a52f44,bf5c04830e9152c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:33:28.504 INFO 21062 --- [ocp-schedules-29,e07f144f8f44fb9e,17f341a92392769a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:33:28.505 INFO 21062 --- [ocp-schedules-4,8711ddbf12e6e600,137c4e3fd8568a74] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:33:28.509 INFO 21062 --- [ocp-schedules-16,4459f588a7d28255,1398cb22b6bf57ce] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:33:28.509 INFO 21062 --- [ocp-schedules-16,4459f588a7d28255,1398cb22b6bf57ce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:33:28.509 INFO 21062 --- [ocp-schedules-2,8a7e2539beb7229d,ae86bcb5d8d40127] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:33:28.509 INFO 21062 --- [ocp-schedules-2,8a7e2539beb7229d,ae86bcb5d8d40127] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:33:28.509 INFO 21062 --- [ocp-schedules-25,01a50655bb50c3f1,2f487fdfdda139eb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:33:28.509 INFO 21062 --- [ocp-schedules-25,01a50655bb50c3f1,2f487fdfdda139eb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:33:28.581 INFO 21062 --- [ocp-schedules-20,c3945421b7d11b39,731246db4c6279cc] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 14:33:28.581 INFO 21062 --- [ocp-schedules-20,c3945421b7d11b39,731246db4c6279cc] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 14:33:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:33:28.589078+08:00, lastProcessTime=2024-11-04T14:33:18.591187+08:00 2024-11-04 14:33:28.635 INFO 21062 --- [ocp-schedules-18,9d16617aaaa0adf9,a0f43872befa74f4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:33:28.635 INFO 21062 --- [ocp-schedules-18,9d16617aaaa0adf9,a0f43872befa74f4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:33:28.670 INFO 21062 --- [ocp-async-6,0fda8f08609ed9b0,33bd307c675d0604] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:33:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:33:28.589078+08:00 2024-11-04 14:33:28.855 INFO 21062 --- [ocp-schedules-11,3bf6f790686019e5,2adf0e0544dd0067] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:33:28.977 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:28.977 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:29.143 INFO 21062 --- [ocp-schedules-29,0a2bb885332f5e77,bfe2b005d98b0a0f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:33:29.143 INFO 21062 --- [ocp-schedules-29,0a2bb885332f5e77,bfe2b005d98b0a0f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:33:29.221 INFO 21062 --- [ocp-schedules-25,62dabb93e085a751,7608f55dd566e076] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:33:29.254 INFO 21062 --- [ocp-schedules-24,63f9cac0ebacfb99,bcf85e5575ab50b0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:33:29.254 WARN 21062 --- [ocp-async-8,ff7c978047b093f3,a0db06cad41b8817] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:29.259 INFO 21062 --- [ocp-schedules-24,63f9cac0ebacfb99,bcf85e5575ab50b0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:33:29.259 INFO 21062 --- [ocp-schedules-24,63f9cac0ebacfb99,bcf85e5575ab50b0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:33:29.286 INFO 21062 --- [ocp-async-1,49fef2d11b9f6835,e45c6bf2970489c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:29.603 INFO 21062 --- [ocp-schedules-6,1f59f6be04e4466e,daca52841446bdb0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:33:29.603 INFO 21062 --- [ocp-schedules-6,1f59f6be04e4466e,daca52841446bdb0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:33:29.604 INFO 21062 --- [ocp-schedules-6,1f59f6be04e4466e,daca52841446bdb0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:33:29.604 INFO 21062 --- [ocp-schedules-6,1f59f6be04e4466e,daca52841446bdb0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:33:29.605 INFO 21062 --- [ocp-schedules-6,1f59f6be04e4466e,daca52841446bdb0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:33:29.605 INFO 21062 --- [ocp-schedules-6,1f59f6be04e4466e,daca52841446bdb0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:33:29.983 INFO 21062 --- [prometheus-minute-schedule-1,cd75c92200f0b1ab,5039a05b3b677839] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:33:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:33:30.682 INFO 21062 --- [ocp-schedules-5,74e31a723efab82b,5445a8de804c4ee6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:33:31.254 WARN 21062 --- [ocp-async-1,44c0fd5eb1d71f24,a000b911e7af13b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:31.286 INFO 21062 --- [ocp-async-7,e11b510ddb901168,743348b74690b791] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:31.749 INFO 21062 --- [ocp-schedules-9,48ad99fe2b3219fd,5c166452b0bfbe38] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:33:31.753 INFO 21062 --- [ocp-schedules-9,48ad99fe2b3219fd,5c166452b0bfbe38] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:33:31.753 INFO 21062 --- [ocp-schedules-9,48ad99fe2b3219fd,5c166452b0bfbe38] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:33:31.828 INFO 21062 --- [ocp-schedules-5,45acb81f0f6c9a67,6c4f1f11d51ecdd4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:33:31.828 INFO 21062 --- [ocp-schedules-5,45acb81f0f6c9a67,6c4f1f11d51ecdd4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:33:31.857 INFO 21062 --- [ocp-schedules-5,45acb81f0f6c9a67,6c4f1f11d51ecdd4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:33:31.857 INFO 21062 --- [ocp-schedules-5,45acb81f0f6c9a67,6c4f1f11d51ecdd4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:33:31.858 INFO 21062 --- [ocp-schedules-20,5f3ddc54eb0c5a85,8d35cd038023e6b8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:33:31.978 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:31.978 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:32.041 INFO 21062 --- [ocp-schedules-3,5c5fe72ff9c09761,92f4ef8263101a5a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:33:32.041+08:00, clusters=[1] 2024-11-04 14:33:32.043 INFO 21062 --- [ocp-schedules-3,5c5fe72ff9c09761,92f4ef8263101a5a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:33:32.229 INFO 21062 --- [ocp-schedules-26,e18740022ed40c29,4246ccd84fb7cffe] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:33:32.229+08:00, clusters=[1] 2024-11-04 14:33:32.233 INFO 21062 --- [ocp-schedules-26,e18740022ed40c29,4246ccd84fb7cffe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:33:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a7c5b0900e57a23e,eb740ba6c0830a40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:33.254 WARN 21062 --- [ocp-async-7,12aa054440ad8efd,a5859d9c09420215] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:33.286 INFO 21062 --- [ocp-async-9,8bddfe75d1777797,27d23690bd0cfe1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:33.359 INFO 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:33:33.359 INFO 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:33:33.359 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:33:33.359 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:33:33.369 INFO 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:33:33.369 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:33:33.369 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:33:33.371 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:33:33.371 WARN 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:33:33.371 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:33:33.371 INFO 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:33:33.372 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:33:33.372 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:33:33.373 WARN 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:33:33.373 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:33:33.373 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:33:33.380 INFO 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:33:33.380 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:33:33.380 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:33:33.390 ERROR 21062 --- [ocp-schedules-27,798f65461104f52c,be167370e5b98d8e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:33:34.978 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:34.979 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:35.255 WARN 21062 --- [ocp-async-10,f80c6968424fdf7a,a87252c63e1b5446] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:35.287 INFO 21062 --- [ocp-async-5,d9e837cdb487d44e,dfcb80f3f2c7e146] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:35.318 INFO 21062 --- [ocp-schedules-9,d28f8fbf530fc4b3,d992cd6145d380df] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:33:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e7b60dfdc9a580ba,0d2ee52c350bd700] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:36.811 INFO 21062 --- [ocp-schedules-12,d66b8c5c718425de,a86600867752ec58] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:33:36.811 INFO 21062 --- [ocp-schedules-12,b9264d2ecb5c4e96,ddad8257d4f1929d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:33:36.816 INFO 21062 --- [ocp-schedules-12,b9264d2ecb5c4e96,ddad8257d4f1929d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:36.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:33:36.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:33:37.255 WARN 21062 --- [ocp-async-1,244bb45f0b66985b,904722762ae3e050] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:37.287 INFO 21062 --- [ocp-async-3,eeecc244462631ad,a1b8a8ec964ec777] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:37.979 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:37.980 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:38.504 INFO 21062 --- [ocp-schedules-15,b167d0c807a52f44,bf5c04830e9152c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:33:38.504 INFO 21062 --- [ocp-schedules-19,0cabd5c3d9145dcf,7bf9a8c07096ad64] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:33:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:33:38.589727+08:00, lastProcessTime=2024-11-04T14:33:28.589078+08:00 2024-11-04 14:33:38.670 INFO 21062 --- [ocp-async-8,3a8e8ad3696a4258,89007b7373bafb44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:33:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:33:38.589727+08:00 2024-11-04 14:33:38.983 INFO 21062 --- [prometheus-minute-schedule-1,4c113d34c2073b36,75bd88746097e0d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:39.065 INFO 21062 --- [ocp-schedules-13,6116eaf7fa06ef33,67049552cad52a1f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:33:39.069 INFO 21062 --- [ocp-schedules-13,6116eaf7fa06ef33,67049552cad52a1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:39.069 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:33:39.255 WARN 21062 --- [ocp-async-7,056b1ccb61c55c1c,5026e777f6415e43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:39.287 INFO 21062 --- [ocp-async-10,da75ebc9222345e8,4e36ca3af07bd702] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:39.687 INFO 21062 --- [ocp-schedules-30,b49f1f1181e27fd9,0543c5b9f86a8a15] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:33:39.691 INFO 21062 --- [ocp-schedules-30,b49f1f1181e27fd9,0543c5b9f86a8a15] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:39.776 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:33:39.782 ERROR 21062 --- [ocp-schedules-30,b49f1f1181e27fd9,0543c5b9f86a8a15] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:33:39.782 INFO 21062 --- [ocp-schedules-30,b49f1f1181e27fd9,0543c5b9f86a8a15] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:33:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:33:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:33:40.686 INFO 21062 --- [ocp-schedules-19,bdb410f3a75a4384,551ec4c5bc0a30cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:33:40.980 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:40.981 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:41.255 WARN 21062 --- [ocp-async-8,f7aa0d85c855caaa,69d7d286c3274604] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:41.287 INFO 21062 --- [ocp-async-1,89d9347a32b87bed,91981eb9557bf806] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:41.983 INFO 21062 --- [prometheus-minute-schedule-1,86b1a4729c116ee2,ec7e53242af0d460] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:43.255 WARN 21062 --- [ocp-async-6,0f84981adf9835eb,25f689c97fc4ccac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:43.287 INFO 21062 --- [ocp-async-4,61530b97e26ba0cf,edec043e76b0ede1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:43.981 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:43.982 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:44.983 INFO 21062 --- [prometheus-minute-schedule-1,363235b1be5e2988,fb1bb7e7024d7a5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:45.255 WARN 21062 --- [ocp-async-2,a3847d4c3f66a361,cf0e412ae4899119] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:45.287 INFO 21062 --- [ocp-async-9,c09d45b0edce3846,ed5a1b902f15bf45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:45.359 INFO 21062 --- [ocp-schedules-11,e5973ed84687abc2,66e443c212a1144e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:33:46.982 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:46.983 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:47.256 WARN 21062 --- [ocp-async-10,2c403d354cfd919b,cf4d92009b1c1bf6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:47.288 INFO 21062 --- [ocp-async-8,c8e8e1b289c21948,245157b42a384f7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:47.983 INFO 21062 --- [prometheus-minute-schedule-1,4a426853bff50223,f39f5a47a2d4bd1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:33:48.591203+08:00, lastProcessTime=2024-11-04T14:33:38.589727+08:00 2024-11-04 14:33:48.670 INFO 21062 --- [ocp-async-2,e08099a577e3f2cc,f0c6502fb53598cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:33:48.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:33:48.591203+08:00 2024-11-04 14:33:49.069 INFO 21062 --- [ocp-schedules-13,6116eaf7fa06ef33,67049552cad52a1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:33:49.132 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:33:49.132 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:33:49.140 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:33:49.140 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:33:49.140 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:33:49.140 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:33:49.146 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:33:49.146 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:33:49.146 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:33:49.146 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:33:49.182 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:33:49.190 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:33:49.190 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:33:49.191 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:33:49.191 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:33:49.191 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:33:49.199 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:33:49.256 WARN 21062 --- [ocp-async-6,a92aae79efaf0738,32ff2531db4168a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:49.288 INFO 21062 --- [ocp-async-4,e1dbcf62be810731,1f3454347926ea0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:49.715 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:33:49.983 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:49.984 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:50.292 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.145838977S 2024-11-04 14:33:50.292 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:33:50.392 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:33:50.392 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:33:50.415 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:33:50.513 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:33:50.513 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:33:50.513 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:33:50.513 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:33:50.513 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:33:50.513 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:33:50.524 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:33:50.690 INFO 21062 --- [ocp-schedules-1,e7894e42513629ec,3f7728ce63855361] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:33:50.702 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] o.s.boot.SpringApplication : Started application in 1.562 seconds (JVM running for 7444.381) 2024-11-04 14:33:50.702 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.562697235S 2024-11-04 14:33:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a13128e0faffd4fa,cfe7691751c328d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:51.169 INFO 21062 --- [ocp-schedules-8,ccecf672e0181f7f,2ce1c9eea3b8710d] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:33:51.256 WARN 21062 --- [ocp-async-4,6f02335a144caffc,ffc1b8ddad1d41bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:51.288 INFO 21062 --- [ocp-async-9,597857ab76139753,95872e122871b376] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:51.755 INFO 21062 --- [ocp-schedules-24,30f2033ff296bfea,f6ba028d56128233] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:33:51.761 INFO 21062 --- [ocp-schedules-24,30f2033ff296bfea,f6ba028d56128233] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:33:51.761 INFO 21062 --- [ocp-schedules-24,30f2033ff296bfea,f6ba028d56128233] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:33:52.984 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:52.986 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:53.256 WARN 21062 --- [ocp-async-10,7471cee0d5b5aa74,34fddfe370d0a66d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:53.288 INFO 21062 --- [ocp-async-8,37961af170c09f69,64583b6d9070bcff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:53.983 INFO 21062 --- [prometheus-minute-schedule-1,bd5db762369a4f71,1fc5ab65b06c873e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:55.256 WARN 21062 --- [ocp-async-1,187dce7bcf005d04,2d841ba95434e9bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:55.288 INFO 21062 --- [ocp-async-3,cbcc603a455cecdf,d3c4fac3242be3f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:55.384 INFO 21062 --- [ocp-schedules-29,772cd864ad2ea918,4d2b41358dda8ffe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:33:55.986 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:55.987 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fc7d25554ac09db6,73cfa3e24c156168] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:33:57.256 WARN 21062 --- [ocp-async-4,f353c879be36e1cb,780150a4f346b276] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:57.289 INFO 21062 --- [ocp-async-2,cd2d436484b7a435,2a330139b52d4c0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:58.499 INFO 21062 --- [ocp-schedules-1,a2d9df6ae62d1c77,52ad817694365ee6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:33:58.500 INFO 21062 --- [ocp-schedules-24,29985fabe096719f,b0094cb405ad15c5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:33:58.502 INFO 21062 --- [ocp-schedules-22,f092098761944761,6339e8606f74477f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:33:58.503 INFO 21062 --- [ocp-schedules-1,a2d9df6ae62d1c77,52ad817694365ee6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:33:58.505 INFO 21062 --- [ocp-schedules-24,29985fabe096719f,b0094cb405ad15c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:33:58.505 INFO 21062 --- [ocp-schedules-19,5eadbd8d53d832a6,4b31d42fe8687183] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:33:58.512 INFO 21062 --- [ocp-schedules-23,419b534625e1ab43,0d6f223358688187] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:33:58.512 INFO 21062 --- [ocp-schedules-21,08959ec526e1136d,a47e3f771aa93dfb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:33:58.512 INFO 21062 --- [ocp-schedules-14,aef39765c65f8079,e73046b07b72e9e9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:33:58.512 INFO 21062 --- [ocp-schedules-14,aef39765c65f8079,e73046b07b72e9e9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:33:58.512 INFO 21062 --- [ocp-schedules-21,08959ec526e1136d,a47e3f771aa93dfb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:33:58.512 INFO 21062 --- [ocp-schedules-23,419b534625e1ab43,0d6f223358688187] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:33:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:33:58.591385+08:00, lastProcessTime=2024-11-04T14:33:48.591203+08:00 2024-11-04 14:33:58.670 INFO 21062 --- [ocp-async-1,59ef56914ede7bbe,74aae7371ed93ca8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:33:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:33:58.591385+08:00 2024-11-04 14:33:58.987 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:33:58.988 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:33:59.224 INFO 21062 --- [ocp-schedules-20,943cab75fcc71cc8,7826bee073d74ab4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:33:59.257 WARN 21062 --- [ocp-async-9,f2d6decfd3303022,3ca73c94bf21dc5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:33:59.289 INFO 21062 --- [ocp-async-8,61376ea7f78a444c,f4b0e049b01173fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:33:59.983 INFO 21062 --- [prometheus-minute-schedule-1,19b69b3a9debe5aa,65e2ef889638e78c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702040, interval=60 2024-11-04 14:34:00.692 INFO 21062 --- [ocp-schedules-4,f1c2ffd6db29a9b8,ffbb42a842102a89] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:34:01.257 WARN 21062 --- [ocp-async-1,3e8e56b1a3a54ebf,2462568cf0e58353] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:01.289 INFO 21062 --- [ocp-async-4,ecebe3e1853c0861,b82b31dc5f5fc624] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:01.988 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:01.989 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:02.983 INFO 21062 --- [prometheus-minute-schedule-1,9d3bff2c2af1ddd2,0dc29f18ce0f4bb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:03.257 WARN 21062 --- [ocp-async-3,92b96603e42bc317,105e8be6037598cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:03.289 INFO 21062 --- [ocp-async-2,5378fa130c796cc7,fee6896e9702c505] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:04.990 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:04.991 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:05.257 WARN 21062 --- [ocp-async-9,3458bd5e4a6560be,4425aa36c70d1861] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:05.289 INFO 21062 --- [ocp-async-5,fe84b2a7458a5b90,435cb7a8969e80f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:05.430 INFO 21062 --- [ocp-schedules-3,322129e08c1b90f1,7e73b61d33ea498d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:34:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ca334ce7a033d17f,0cd0976008ed2951] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:06.845 INFO 21062 --- [ocp-schedules-12,b9264d2ecb5c4e96,ddad8257d4f1929d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:34:06.846 INFO 21062 --- [ocp-schedules-12,092e4f7ae90eaa18,365b432fde39ff0b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:34:06.851 INFO 21062 --- [ocp-schedules-12,092e4f7ae90eaa18,365b432fde39ff0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:06.881 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:34:06.881 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:34:07.257 WARN 21062 --- [ocp-async-1,75c88d4ecbe1078e,656f1d52c4e3d94f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:07.289 INFO 21062 --- [ocp-async-6,ff806a4ca6a36fa7,b23d63cf2693ddfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:07.991 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:07.992 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:08.504 INFO 21062 --- [ocp-schedules-1,a2d9df6ae62d1c77,52ad817694365ee6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:34:08.505 INFO 21062 --- [ocp-schedules-24,29985fabe096719f,b0094cb405ad15c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:34:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:34:08.590502+08:00, lastProcessTime=2024-11-04T14:33:58.591385+08:00 2024-11-04 14:34:08.670 INFO 21062 --- [ocp-async-9,3e51c7714888d599,28b41dba52a445f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:34:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:34:08.590502+08:00 2024-11-04 14:34:08.983 INFO 21062 --- [prometheus-minute-schedule-1,cb95eb8c25f75ba8,16e0ad214375df39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:09.257 WARN 21062 --- [ocp-async-3,b88d7cd657c93598,eebbeecbf93c2d23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:09.290 INFO 21062 --- [ocp-async-7,a2224dd598374ed6,2e47689f13887410] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:34:10.005 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:34:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:34:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:34:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:34:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:34:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:34:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:34:10.695 INFO 21062 --- [ocp-schedules-14,382316e16eeccfcb,29d2aec038e96e78] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:34:10.992 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:10.993 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:11.258 WARN 21062 --- [ocp-async-2,2910a76970911cc0,30a6a46dcca10510] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:11.290 INFO 21062 --- [ocp-async-10,3f4c9281fd13174f,6397602d93209b5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:11.603 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701980, nodeCount=49, affectRows=49 2024-11-04 14:34:11.603 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701980, nodeCount=54, affectRows=54 2024-11-04 14:34:11.603 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701980, nodeCount=48, affectRows=48 2024-11-04 14:34:11.603 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701980, nodeCount=48, validCount=48 2024-11-04 14:34:11.603 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701980, nodeCount=67, affectRows=67 2024-11-04 14:34:11.603 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701980, nodeCount=54, validCount=54 2024-11-04 14:34:11.603 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701980, nodeCount=48, validCount=48 2024-11-04 14:34:11.604 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701980, nodeCount=67, validCount=67 2024-11-04 14:34:11.604 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701980, nodeCount=87, affectRows=87 2024-11-04 14:34:11.604 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730701980, nodeCount=91, affectRows=91 2024-11-04 14:34:11.604 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701980, nodeCount=85, validCount=85 2024-11-04 14:34:11.605 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730701980, nodeCount=91, validCount=91 2024-11-04 14:34:11.763 INFO 21062 --- [ocp-schedules-1,5e02c9dd86dfa987,1c7d6122a621aaf8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:34:11.769 INFO 21062 --- [ocp-schedules-1,5e02c9dd86dfa987,1c7d6122a621aaf8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:34:11.769 INFO 21062 --- [ocp-schedules-1,5e02c9dd86dfa987,1c7d6122a621aaf8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:34:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8259e71854e8d914,663c4c0a67f4ca4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:13.258 WARN 21062 --- [ocp-async-5,43103053751ac318,00e6fb0b3baa8bb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:13.290 INFO 21062 --- [ocp-async-8,784bf7d6947af88d,8e505ee760cbf1d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:13.984 INFO 21062 --- [prometheus-cache-monitor1,f98fdeae79eefdfd,2ca865156d6ddb20] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:34:13.984 INFO 21062 --- [prometheus-cache-monitor1,f98fdeae79eefdfd,2ca865156d6ddb20] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:34:13.993 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:13.994 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:14.983 INFO 21062 --- [prometheus-minute-schedule-1,ad1069d2634a5802,b6c98f59059fc41f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:15.258 WARN 21062 --- [ocp-async-3,1eff7ffd937a2097,676fe53ceac1c74e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:15.290 INFO 21062 --- [ocp-async-4,83b10bada0d7b96b,26bb5e28ba502de5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:15.457 INFO 21062 --- [ocp-schedules-8,465c780845a43f20,b67d2281392eaa80] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:34:16.994 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:16.996 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:17.258 WARN 21062 --- [ocp-async-2,95c53d9d7f2e38e8,b8e882e439a16968] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:17.290 INFO 21062 --- [ocp-async-7,0a2d80a57cb15c38,d5f03eebedfbd8fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8da258f530631fa2,ba847d98bef9be90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:34:18.590561+08:00, lastProcessTime=2024-11-04T14:34:08.590502+08:00 2024-11-04 14:34:18.670 INFO 21062 --- [ocp-async-3,fb4ff15b87926828,a097a477465c6fd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:34:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:34:18.590561+08:00 2024-11-04 14:34:19.258 WARN 21062 --- [ocp-async-10,8c146afbd2988eaf,e8b8ef5664a30582] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:19.291 INFO 21062 --- [ocp-async-6,4826cd29d3ac10e8,34f836fc72629d70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:19.996 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:19.998 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:20.698 INFO 21062 --- [ocp-schedules-13,78d0b75b3b4b87ac,e1c7b5388f07e792] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:34:20.983 INFO 21062 --- [prometheus-minute-schedule-1,1e74fa9324cb7fc1,0ec9d808efd1bc45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:21.258 WARN 21062 --- [ocp-async-8,cff545a89d81c8d8,105ca095b4204553] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:21.291 INFO 21062 --- [ocp-async-9,95a2a610c72f8a41,19b5b813c46b14a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:22.998 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:23.000 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:23.259 WARN 21062 --- [ocp-async-2,01dcfb4ee5330dcf,3413e5b0f5aa0463] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:23.291 INFO 21062 --- [ocp-async-7,2f08aa8ad1ea2de8,9bef231030dbe7c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:23.983 INFO 21062 --- [prometheus-minute-schedule-1,fc9deed8c38f4e4e,55f3d09244e6f193] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:25.259 WARN 21062 --- [ocp-async-10,e2626d1a81a1707d,0dbf2b201dc4716f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:25.291 INFO 21062 --- [ocp-async-5,49b6da63c110dee8,dbc887d481d299b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:25.498 INFO 21062 --- [ocp-schedules-14,498cfa6152e336c6,574f88cf87e27239] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:34:26.000 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:26.001 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:26.983 INFO 21062 --- [prometheus-minute-schedule-1,135e3ac7312d6104,b95eff775a306235] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:26.984 INFO 21062 --- [prometheus-hour-schedule-1,45afaafaab5f7e30,84cbeb8f11a65bbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:34:27.259 WARN 21062 --- [ocp-async-8,b715141b6ce345be,76402d2d1e4d84e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:27.291 INFO 21062 --- [ocp-async-3,6a39006d89d9f1b9,857a2eb3cebb5f88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:28.498 INFO 21062 --- [ocp-schedules-11,4d97d502af42b2f3,573e30475f7f8ca3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:34:28.500 INFO 21062 --- [ocp-schedules-21,6cb23c144b02005e,49b566c93d701d26] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:34:28.500 INFO 21062 --- [ocp-schedules-6,f9823e5c6e094999,c9d9db68b976232a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:34:28.500 INFO 21062 --- [ocp-schedules-4,1165ffc0a4ce8236,ae3ec2583f3e3791] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:34:28.502 INFO 21062 --- [ocp-schedules-11,4d97d502af42b2f3,573e30475f7f8ca3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:34:28.502 INFO 21062 --- [ocp-schedules-30,01dbf32014a20ad0,d8d1dd21f53d7dd2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:34:28.503 INFO 21062 --- [ocp-schedules-6,f9823e5c6e094999,c9d9db68b976232a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:34:28.504 INFO 21062 --- [ocp-schedules-4,1165ffc0a4ce8236,ae3ec2583f3e3791] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:28.507 INFO 21062 --- [ocp-schedules-29,2a83aefe925ee1c6,2e4e8f9bdf1882fe] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:34:28.508 INFO 21062 --- [ocp-schedules-21,6cb23c144b02005e,49b566c93d701d26] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:34:28.510 INFO 21062 --- [ocp-schedules-24,e9488f594da30ed9,4b5a31950e55956a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:34:28.510 INFO 21062 --- [ocp-schedules-2,f4dbac2abadaae42,c74e04b62db51afb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:34:28.510 INFO 21062 --- [ocp-schedules-24,e9488f594da30ed9,4b5a31950e55956a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:34:28.510 INFO 21062 --- [ocp-schedules-2,f4dbac2abadaae42,c74e04b62db51afb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:34:28.510 INFO 21062 --- [ocp-schedules-28,fa170b96d4938f9f,dd6b965172570942] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:34:28.510 INFO 21062 --- [ocp-schedules-28,fa170b96d4938f9f,dd6b965172570942] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:34:28.578 INFO 21062 --- [ocp-schedules-27,a87801732f6c32db,96e80d8daa3ad46f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:34:28.578 INFO 21062 --- [ocp-schedules-27,a87801732f6c32db,96e80d8daa3ad46f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:34:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:34:28.589304+08:00, lastProcessTime=2024-11-04T14:34:18.590561+08:00 2024-11-04 14:34:28.595 INFO 21062 --- [ocp-schedules-22,f7440177e2f41aa0,25326dde4314511d] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:34:28.595+08:00, tenantSize=0 2024-11-04 14:34:28.671 INFO 21062 --- [ocp-async-5,2bfedd7e5f6351b0,8358bcc3b12e5739] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:34:28.674 INFO 21062 --- [ocp-schedules-18,10f3c3709415347f,c7a687d76416bd76] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 14:34:28.674 INFO 21062 --- [ocp-schedules-18,10f3c3709415347f,c7a687d76416bd76] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 14:34:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:34:28.589304+08:00 2024-11-04 14:34:28.828 INFO 21062 --- [ocp-schedules-26,3927c3e89cef6a1e,b2b00579ac5a5c20] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:34:28.828 INFO 21062 --- [ocp-schedules-26,3927c3e89cef6a1e,b2b00579ac5a5c20] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:34:28.860 INFO 21062 --- [ocp-schedules-21,f89b8f229e442a87,078120743a9ee30f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:34:29.001 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:29.001 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:29.147 INFO 21062 --- [ocp-schedules-29,5dd77f8cf16895c0,c3b30dc2877fbe54] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:34:29.148 INFO 21062 --- [ocp-schedules-29,5dd77f8cf16895c0,c3b30dc2877fbe54] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:34:29.227 INFO 21062 --- [ocp-schedules-20,73883f5e3cefd238,8f6d65cdc6e60571] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:34:29.259 WARN 21062 --- [ocp-async-10,7cc9051073212e5b,a4ae72cb991546f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:29.259 INFO 21062 --- [ocp-schedules-3,fb11db59eb2da666,0a342ad343bbe12d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:34:29.265 INFO 21062 --- [ocp-schedules-3,fb11db59eb2da666,0a342ad343bbe12d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:34:29.265 INFO 21062 --- [ocp-schedules-3,fb11db59eb2da666,0a342ad343bbe12d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:34:29.291 INFO 21062 --- [ocp-async-1,4b2843d79ec2fdc8,f87f70c2a8d9083d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:29.433 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:34:29.433 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:34:29.442 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:34:29.442 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:34:29.442 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:34:29.453 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:34:29.453 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:34:29.453 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:34:29.456 INFO 21062 --- [ocp-schedules-18,985723c0564d5391,1ad5edb6e1dd61a9] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:34:29.607 INFO 21062 --- [ocp-schedules-17,267ca16331a04c6a,b6294a5ec6d27180] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:34:29.607 INFO 21062 --- [ocp-schedules-17,267ca16331a04c6a,b6294a5ec6d27180] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:34:29.614 INFO 21062 --- [ocp-schedules-17,267ca16331a04c6a,b6294a5ec6d27180] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:34:29.614 INFO 21062 --- [ocp-schedules-17,267ca16331a04c6a,b6294a5ec6d27180] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:34:29.618 INFO 21062 --- [ocp-schedules-17,267ca16331a04c6a,b6294a5ec6d27180] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:34:29.618 INFO 21062 --- [ocp-schedules-17,267ca16331a04c6a,b6294a5ec6d27180] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:34:29.983 INFO 21062 --- [prometheus-minute-schedule-1,d63502b5fedb5572,8c3041aef6871555] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:34:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:34:30.701 INFO 21062 --- [ocp-schedules-11,a21bf1c1e09474bd,888e3a27ead4c068] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:34:31.259 WARN 21062 --- [ocp-async-8,49543c554277fe4f,b3e48bd20c6a5a02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:31.291 INFO 21062 --- [ocp-async-6,e5b9de4bdc6edbdd,e5171f2f55afaa0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:31.771 INFO 21062 --- [ocp-schedules-7,058b2fa2de20f2e8,d96e1e3d97136eac] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:34:31.777 INFO 21062 --- [ocp-schedules-7,058b2fa2de20f2e8,d96e1e3d97136eac] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:34:31.777 INFO 21062 --- [ocp-schedules-7,058b2fa2de20f2e8,d96e1e3d97136eac] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:34:31.858 INFO 21062 --- [ocp-schedules-10,79db7927eececae2,59fa93d258e03060] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:34:31.858 INFO 21062 --- [ocp-schedules-10,79db7927eececae2,59fa93d258e03060] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:34:31.883 INFO 21062 --- [ocp-schedules-10,79db7927eececae2,59fa93d258e03060] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:34:31.883 INFO 21062 --- [ocp-schedules-10,79db7927eececae2,59fa93d258e03060] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:34:31.885 INFO 21062 --- [ocp-schedules-3,02b9337ed65e4e89,6974929f84b32b30] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:34:32.002 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:32.002 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:32.069 INFO 21062 --- [ocp-schedules-18,ff38249f3da83829,eb3ae0f55b6319e1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:34:32.069+08:00, clusters=[1] 2024-11-04 14:34:32.071 INFO 21062 --- [ocp-schedules-18,ff38249f3da83829,eb3ae0f55b6319e1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:34:32.258 INFO 21062 --- [ocp-schedules-5,dfc5df2d9830d09f,2e43c22fbb17ad2f] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:34:32.258 INFO 21062 --- [ocp-schedules-24,fad9c4a4d26306f3,4218c00e20ed8fbc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:34:32.258+08:00, clusters=[1] 2024-11-04 14:34:32.261 INFO 21062 --- [ocp-schedules-5,dfc5df2d9830d09f,2e43c22fbb17ad2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:32.262 INFO 21062 --- [ocp-schedules-24,fad9c4a4d26306f3,4218c00e20ed8fbc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:34:32.318 ERROR 21062 --- [ocp-schedules-5,dfc5df2d9830d09f,2e43c22fbb17ad2f] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:34:32.318 INFO 21062 --- [ocp-schedules-5,dfc5df2d9830d09f,2e43c22fbb17ad2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:34:32.983 INFO 21062 --- [prometheus-minute-schedule-1,00536f208a8e507c,893fa2b8d1b29ed2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:33.259 WARN 21062 --- [ocp-async-9,927593bdadbd76e9,9517efc84cd13909] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:33.292 INFO 21062 --- [ocp-async-2,10f89eda16df52b8,9ee2ff9db0528548] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:33.398 INFO 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:34:33.399 INFO 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:34:33.399 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:34:33.399 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:34:33.408 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:34:33.409 INFO 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:34:33.409 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:34:33.410 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:34:33.410 WARN 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:34:33.411 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:34:33.411 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:34:33.411 INFO 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:34:33.411 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:34:33.412 WARN 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:34:33.412 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:34:33.412 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:34:33.420 INFO 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:34:33.420 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:34:33.420 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:34:33.431 ERROR 21062 --- [ocp-schedules-18,7c1c9bcddc5a37dc,6ad2c81206cd1c43] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:34:35.003 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:35.003 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:35.260 WARN 21062 --- [ocp-async-7,573870c541903eb3,63067c362da301c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:35.292 INFO 21062 --- [ocp-async-4,9b6b60702bc8d228,4e2b0dfe5a158726] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:35.525 INFO 21062 --- [ocp-schedules-3,f385c86bceaa889d,45fa2db7b1d11ed8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:34:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1ec27f87a18ec703,7b4b9af07ff732a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:36.881 INFO 21062 --- [ocp-schedules-12,092e4f7ae90eaa18,365b432fde39ff0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:34:36.881 INFO 21062 --- [ocp-schedules-12,f13d976afc8cc0f9,a32d2b3deacd3533] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:34:36.885 INFO 21062 --- [ocp-schedules-12,f13d976afc8cc0f9,a32d2b3deacd3533] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:36.914 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:34:36.914 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:34:37.260 WARN 21062 --- [ocp-async-1,87e31e3214b835e4,500ee9cdc793d3dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:37.292 INFO 21062 --- [ocp-async-5,099e4b4927d1edd0,163c1fe0b8c45e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:38.003 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:38.004 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:38.504 INFO 21062 --- [ocp-schedules-6,f9823e5c6e094999,c9d9db68b976232a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:34:38.507 INFO 21062 --- [ocp-schedules-4,1165ffc0a4ce8236,ae3ec2583f3e3791] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:34:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:34:38.589907+08:00, lastProcessTime=2024-11-04T14:34:28.589304+08:00 2024-11-04 14:34:38.671 INFO 21062 --- [ocp-async-10,a6248412ddd43aa3,76b73801734e56ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:34:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:34:38.589907+08:00 2024-11-04 14:34:38.983 INFO 21062 --- [prometheus-minute-schedule-1,b56b76938801e641,f58f0cfab3fe0930] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:39.260 WARN 21062 --- [ocp-async-3,c8950e1f624d9c40,20b52017fadf874c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:39.292 INFO 21062 --- [ocp-async-7,af0a132faab6fc67,7beef5a202ec64d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:39.782 INFO 21062 --- [ocp-schedules-8,3c52373e193c5bf4,a52c54696be1550a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:34:39.785 INFO 21062 --- [ocp-schedules-8,3c52373e193c5bf4,a52c54696be1550a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:39.871 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:34:39.874 ERROR 21062 --- [ocp-schedules-8,3c52373e193c5bf4,a52c54696be1550a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:34:39.874 INFO 21062 --- [ocp-schedules-8,3c52373e193c5bf4,a52c54696be1550a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:34:40.002 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:34:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:34:40.704 INFO 21062 --- [ocp-schedules-24,9c70293a1c100bce,560eb299d33de59c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:34:41.004 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:41.005 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:41.260 WARN 21062 --- [ocp-async-4,6123a80d8b7664ba,c4746ded2d286a6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:41.292 INFO 21062 --- [ocp-async-1,49b3fc6995344d85,15118ddd8d3f7906] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:34:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:34:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:34:41.856 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:34:41.856 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:34:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6c0c5380ac39853f,2f91e98148e17beb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:42.904 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:34:42.905 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:34:42.906 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.906 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:34:42.908 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.908 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.910 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.910 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:34:42.925 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.926 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.926 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.927 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:34:42.941 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.942 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.942 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.943 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:34:42.956 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.957 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.958 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.958 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:34:42.972 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.972 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.973 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.973 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:34:42.979 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.979 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.980 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.980 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:34:42.984 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.985 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.986 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.986 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:34:42.991 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.991 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.992 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.992 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:34:42.998 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:42.999 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:42.999 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.000 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:34:43.004 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.005 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.005 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.006 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:34:43.011 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.011 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.012 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.012 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:34:43.021 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.022 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.022 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.023 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:34:43.036 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.036 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.037 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.037 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:34:43.041 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.042 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.042 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.043 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:34:43.049 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.049 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.052 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.053 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:34:43.058 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.059 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.059 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.060 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:34:43.064 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.064 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.065 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.065 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:34:43.069 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.070 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.071 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.071 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:34:43.076 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.076 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.077 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.077 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:34:43.081 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.082 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.083 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.083 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:34:43.087 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.088 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.088 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.088 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:34:43.093 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.093 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.094 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.094 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:34:43.098 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.099 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.099 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.100 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:34:43.104 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.104 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.105 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.105 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:34:43.110 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.111 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.111 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.112 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:34:43.116 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.116 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.117 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.117 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:34:43.121 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.121 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.122 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.122 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:34:43.126 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.127 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.127 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.128 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:34:43.132 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.133 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.133 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.134 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:34:43.138 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.139 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.139 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.139 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:34:43.143 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.144 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.144 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.144 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:34:43.149 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.149 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.149 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.150 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:34:43.154 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.154 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.155 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.155 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:34:43.159 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.159 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.160 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.160 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:34:43.167 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.167 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.168 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.169 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:34:43.175 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.175 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.176 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.176 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:34:43.181 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.182 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.182 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.183 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:34:43.188 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.188 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.189 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.189 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:34:43.192 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.193 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.193 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.194 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:34:43.197 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.198 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.198 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.198 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:34:43.202 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.202 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.203 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.203 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:34:43.206 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.206 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.207 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.207 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:34:43.216 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.216 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.217 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.217 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:34:43.239 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.239 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.240 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.240 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:34:43.246 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.246 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.247 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.247 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:34:43.253 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.253 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.254 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.254 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:34:43.259 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.260 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.260 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.260 WARN 21062 --- [ocp-async-5,7ecb884cce0d763b,d8061a5558f0f55a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:43.260 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:34:43.264 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.265 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.265 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.266 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:34:43.270 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.270 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.270 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.271 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:34:43.275 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:34:43.275 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:34:43.276 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.277 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:34:43.278 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.278 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.279 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.279 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:34:43.284 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.285 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:34:43.285 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:34:43.286 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:34:43.289 INFO 21062 --- [ocp-schedules-24,07ecb674b679d6e5,e4b7f3625319b3d7] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:34:43.293 INFO 21062 --- [ocp-async-6,5de9f2fc7d7554d2,9f36e992ead8c08d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:44.005 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:44.006 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3c1438a74b2df36a,d28387046fe7a5c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:45.260 WARN 21062 --- [ocp-async-9,602048843cbd96ce,6efa34a730b31dd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:45.293 INFO 21062 --- [ocp-async-2,1e7a7fcd3894266b,03baffeaea7ec385] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:45.567 INFO 21062 --- [ocp-schedules-28,f4e254a752652f17,e28898a79b2ee722] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:34:47.006 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:47.007 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:47.261 WARN 21062 --- [ocp-async-7,1ce0e3a5530fc219,29cbe579e16b87c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:47.293 INFO 21062 --- [ocp-async-10,dc914a87b1855612,f916d43d0d8641b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:47.983 INFO 21062 --- [prometheus-minute-schedule-1,45bdfd51665d6c09,8f5fbd549bde4dbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:34:48.590958+08:00, lastProcessTime=2024-11-04T14:34:38.589907+08:00 2024-11-04 14:34:48.671 INFO 21062 --- [ocp-async-3,a4d600e1aca2207e,c9c410e0a9de5583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:34:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:34:48.590958+08:00 2024-11-04 14:34:49.070 INFO 21062 --- [ocp-schedules-26,4b0a804cb9786e66,3e427c1a7e1b98a6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:34:49.073 INFO 21062 --- [ocp-schedules-26,4b0a804cb9786e66,3e427c1a7e1b98a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:49.074 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:34:49.261 WARN 21062 --- [ocp-async-1,42f114b4a9187a0e,9bd59c92ed605ee4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:49.293 INFO 21062 --- [ocp-async-6,caaa20d4b17f1776,1742a51dbccbfbca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:50.007 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:50.008 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:50.707 INFO 21062 --- [ocp-schedules-19,84b4300787bde5fb,0b81cc4bf27f0abb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:34:50.983 INFO 21062 --- [prometheus-minute-schedule-1,75975db75214c215,efe9cb9f27835b3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:51.261 WARN 21062 --- [ocp-async-6,018b319cf803e6dd,148ff0577a798fe7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:51.293 INFO 21062 --- [ocp-async-2,5072449d05ec4328,979cbcde1567cd02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:51.779 INFO 21062 --- [ocp-schedules-8,5694701e76da243a,6d6f6d35adeb8848] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:34:51.785 INFO 21062 --- [ocp-schedules-8,5694701e76da243a,6d6f6d35adeb8848] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:34:51.785 INFO 21062 --- [ocp-schedules-8,5694701e76da243a,6d6f6d35adeb8848] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:34:53.008 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:53.009 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:53.261 WARN 21062 --- [ocp-async-2,08423abe5f23c847,8c1136839d097b1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:53.293 INFO 21062 --- [ocp-async-10,1dd389810dfc774b,508466c2c10a3493] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:53.983 INFO 21062 --- [prometheus-minute-schedule-1,af4904d0ee47ee50,ff885d99bc2c6943] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:55.261 WARN 21062 --- [ocp-async-1,0a1d079375a202d7,307b55124b25347a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:55.293 INFO 21062 --- [ocp-async-5,2a2d584d130dc65f,582ceabe33b2e64d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:55.629 INFO 21062 --- [ocp-schedules-24,80cd870b31dd1df3,a0bc0041faedd46b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:34:56.009 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:56.010 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d2629d0162c34999,a2f8077abf3c4027] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:34:57.261 WARN 21062 --- [ocp-async-6,edd60412f4a86a85,ff733fa226b443ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:57.293 INFO 21062 --- [ocp-async-3,7b15355bae84bd5a,1c458568cba2f2cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:58.500 INFO 21062 --- [ocp-schedules-6,d546c8d092e87e3c,addd31730671e7fa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:34:58.500 INFO 21062 --- [ocp-schedules-3,ce741ce1d0eb1c55,1d315d9d170415db] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:34:58.502 INFO 21062 --- [ocp-schedules-25,da526e556b3b1e98,938ca84948dbf525] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:34:58.504 INFO 21062 --- [ocp-schedules-6,d546c8d092e87e3c,addd31730671e7fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:58.504 INFO 21062 --- [ocp-schedules-3,ce741ce1d0eb1c55,1d315d9d170415db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:34:58.505 INFO 21062 --- [ocp-schedules-5,a0ec0473d6fa8f70,f660593d93d346a7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:34:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:34:58.509 INFO 21062 --- [ocp-schedules-7,37afe2a4b29f5462,7eaf77b596314e20] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:34:58.509 INFO 21062 --- [ocp-schedules-7,37afe2a4b29f5462,7eaf77b596314e20] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:34:58.509 INFO 21062 --- [ocp-schedules-18,0568cf5e55c15647,4c27f5bd49300981] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:34:58.510 INFO 21062 --- [ocp-schedules-18,0568cf5e55c15647,4c27f5bd49300981] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:34:58.510 INFO 21062 --- [ocp-schedules-8,7d38a900bcb3c171,2f4115baac9883c6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:34:58.510 INFO 21062 --- [ocp-schedules-8,7d38a900bcb3c171,2f4115baac9883c6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:34:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:34:58.590535+08:00, lastProcessTime=2024-11-04T14:34:48.590958+08:00 2024-11-04 14:34:58.671 INFO 21062 --- [ocp-async-1,0f72bb2e3291aaab,07d8ef8984a2b12a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:34:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:34:58.590535+08:00 2024-11-04 14:34:59.010 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:34:59.011 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:34:59.074 INFO 21062 --- [ocp-schedules-26,4b0a804cb9786e66,3e427c1a7e1b98a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:34:59.229 INFO 21062 --- [ocp-schedules-23,ab9bd483076d3882,2e802164ff193c8d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:34:59.261 WARN 21062 --- [ocp-async-2,166eefd443cf5f28,b127bea199992a38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:34:59.294 INFO 21062 --- [ocp-async-10,c75cc95d0f9e8b54,d9fc5520c6283732] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:34:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e7a4e24ceeeb2c9c,aabbe9794000a082] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702100, interval=60 2024-11-04 14:35:00.309 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,470cd7e768d9c72c] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:30:00.0, nextRun=2024-11-04 14:35:00.0, currentDate=Mon Nov 04 14:35:00 CST 2024 2024-11-04 14:35:00.309 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,470cd7e768d9c72c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:30:00.0, nextRunTime=2024-11-04 14:35:00.0} 2024-11-04 14:35:00.340 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,470cd7e768d9c72c] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:35:00.345 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,470cd7e768d9c72c] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:35:00.345 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,470cd7e768d9c72c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:35:00.710 INFO 21062 --- [ocp-schedules-25,b8a28fe81efd9ac8,0df75e766e92698c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:35:01.262 WARN 21062 --- [ocp-async-10,41812ed3af50ca92,97542a46f6c36f1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:01.294 INFO 21062 --- [ocp-async-6,2129e7ef1aab70f7,65f1072d3304c36b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:02.011 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:02.012 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ef7a1804ab1008d4,ffaf1dcabe829660] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:03.262 WARN 21062 --- [ocp-async-6,dbf08607a91ad35d,6ffb936df115f865] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:03.294 INFO 21062 --- [ocp-async-3,bc642255e2d78654,04a579078b9a0ad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:05.012 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:05.014 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:05.262 WARN 21062 --- [ocp-async-2,343fbaecd92127b8,5e81b55613ca2b2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:05.294 INFO 21062 --- [ocp-async-7,3632f7cb004bc5fa,f46b0a3ee9e13700] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:05.670 INFO 21062 --- [ocp-schedules-15,3d5d15d698c555b3,44bfc52ebe7b89c4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:35:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2f82834619495165,a5aecf8bab7988df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:06.914 INFO 21062 --- [ocp-schedules-12,f13d976afc8cc0f9,a32d2b3deacd3533] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:35:06.915 INFO 21062 --- [ocp-schedules-12,8cd3364a3c545e8d,a848be842cb44633] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:35:06.918 INFO 21062 --- [ocp-schedules-12,8cd3364a3c545e8d,a848be842cb44633] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:06.948 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:35:06.948 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:35:07.262 WARN 21062 --- [ocp-async-7,3ecba774a413420a,460125c9057be148] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:07.294 INFO 21062 --- [ocp-async-1,37626a4517f3dfe1,2dfebf214235f055] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:08.014 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:08.015 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:08.506 INFO 21062 --- [ocp-schedules-3,ce741ce1d0eb1c55,1d315d9d170415db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:35:08.506 INFO 21062 --- [ocp-schedules-6,d546c8d092e87e3c,addd31730671e7fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:35:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:35:08.590138+08:00, lastProcessTime=2024-11-04T14:34:58.590535+08:00 2024-11-04 14:35:08.671 INFO 21062 --- [ocp-async-2,fa88cd5ca9d5d690,1491a92d2d2e3f5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:35:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:35:08.590138+08:00 2024-11-04 14:35:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a3691493a82caf64,e1b881c05218f51b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:09.262 WARN 21062 --- [ocp-async-6,b6efb45212be3619,ea3d9426d87a6f1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:09.294 INFO 21062 --- [ocp-async-9,ee7f9b63065f845c,b88c1c575453109f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:35:10.004 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:35:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:35:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:35:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:35:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:35:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:35:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:35:10.713 INFO 21062 --- [ocp-schedules-16,ff2d18842b22299c,60886e181958ae13] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:35:11.015 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:11.016 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:11.262 WARN 21062 --- [ocp-async-9,f252babd010abf5f,15b310ea25b7bbdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:11.294 INFO 21062 --- [ocp-async-4,20c48b43ff0280a1,b9c32bfbc2919111] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:11.610 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=4, affectRows=4 2024-11-04 14:35:11.610 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=4, validCount=4 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=3, affectRows=3 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=11, affectRows=11 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=3, validCount=3 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=11, validCount=11 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=34, affectRows=34 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=34, validCount=34 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=31, affectRows=31 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=59, affectRows=59 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=50, affectRows=50 2024-11-04 14:35:11.611 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=75, affectRows=75 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=58, affectRows=58 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=29, affectRows=29 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=31, validCount=31 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=57, validCount=57 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=29, validCount=29 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702040, nodeCount=42, affectRows=42 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=50, validCount=50 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=58, validCount=58 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=74, validCount=74 2024-11-04 14:35:11.612 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702040, nodeCount=42, validCount=42 2024-11-04 14:35:11.788 INFO 21062 --- [ocp-schedules-27,78dda3d99f9685f2,dd1de36d370c1759] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:35:11.793 INFO 21062 --- [ocp-schedules-27,78dda3d99f9685f2,dd1de36d370c1759] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:35:11.793 INFO 21062 --- [ocp-schedules-27,78dda3d99f9685f2,dd1de36d370c1759] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:35:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8958031a14bbd5c5,4b9838679cc5ed4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:13.263 WARN 21062 --- [ocp-async-4,4b9d774df20dbe31,10b830718bec3da0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:13.295 INFO 21062 --- [ocp-async-8,07a5baa2a693d726,1801784b801b2133] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:13.985 INFO 21062 --- [prometheus-cache-monitor1,b6e9e60956d6529a,9f7ad4fb1d5b1b0c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:35:13.985 INFO 21062 --- [prometheus-cache-monitor1,b6e9e60956d6529a,9f7ad4fb1d5b1b0c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:35:14.016 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:14.017 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:14.983 INFO 21062 --- [prometheus-minute-schedule-1,49b06c919bb1f53a,5a0d2c044d910a14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:15.146 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:31:15.129, end=2024-11-04T14:33:15.144} 2024-11-04 14:35:15.155 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:35:15.155 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:35:15.263 WARN 21062 --- [ocp-async-1,bfc498afae8a5bc1,7312d7abda49d501] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:15.295 INFO 21062 --- [ocp-async-5,397c3d610766c7ea,45bc4c9a4eddbbdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:15.694 INFO 21062 --- [ocp-schedules-3,5647e8ffc457785e,7aa9f64a53527d09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:35:17.017 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:17.018 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:17.263 WARN 21062 --- [ocp-async-2,04c4e29c1fe41a49,f6dfd179a7f3aac5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:17.295 INFO 21062 --- [ocp-async-3,ae3a991cf5882001,e3e9fa51f6e91b65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8fc0bdbcbab5b79b,1ab50b0ade8bd231] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:35:18.589948+08:00, lastProcessTime=2024-11-04T14:35:08.590138+08:00 2024-11-04 14:35:18.672 INFO 21062 --- [ocp-async-6,21c094a52bafb509,3739d90a1b56b92f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:35:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:35:18.589948+08:00 2024-11-04 14:35:19.263 WARN 21062 --- [ocp-async-10,3b4df960865f2f1a,5d42b52613942c5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:19.295 INFO 21062 --- [ocp-async-8,e0c93dea710e7164,57a84268d9b02c84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:20.018 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:20.019 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:20.716 INFO 21062 --- [ocp-schedules-30,2163723f37576e92,a059b27cd3b1d044] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:35:20.983 INFO 21062 --- [prometheus-minute-schedule-1,af422c43a7a43491,64cba12fb610af19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:21.263 WARN 21062 --- [ocp-async-1,71fb081fc148a804,37ca4913dfb9d341] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:21.296 INFO 21062 --- [ocp-async-5,815f71ce6367ac39,4ba9e2f5b85d0bf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:23.019 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:23.020 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:23.263 WARN 21062 --- [ocp-async-2,6683ab91bf6be850,56a79ff42ce63cda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:23.296 INFO 21062 --- [ocp-async-3,39ae980d2699b5bb,c7068863051c1b68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:23.983 INFO 21062 --- [prometheus-minute-schedule-1,71776746b2035fd1,e19e60aabc4f39d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:25.264 WARN 21062 --- [ocp-async-10,379f946f7f41a89a,d26ecec0ecec977b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:25.296 INFO 21062 --- [ocp-async-4,88a8e37226e54514,5c1e32eb780fdf7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:25.739 INFO 21062 --- [ocp-schedules-14,011a4ab4eeb2aa56,965015fcbcbd15ad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:35:26.020 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:26.021 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:26.983 INFO 21062 --- [prometheus-minute-schedule-1,dfff12e00b35adb6,af1d6a2288bc8b43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5b80d0af414beda9,ec687b1ff6b2d677] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:35:27.264 WARN 21062 --- [ocp-async-1,170521559e1299fc,0503120264afc9fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:27.296 INFO 21062 --- [ocp-async-6,7f2b1977e18f4094,ee4f78a722314bc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:28.498 INFO 21062 --- [ocp-schedules-20,02b88f37be1258e3,e8c1c0a6cb472f36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:35:28.499 INFO 21062 --- [ocp-schedules-23,dacb468065084408,f545aba5fb046449] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:35:28.499 INFO 21062 --- [ocp-schedules-26,0172d4b4aa756442,d211a418cdd2c6a7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:35:28.500 INFO 21062 --- [ocp-schedules-22,deee2233046c84a3,3c614efa0b2c932a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:35:28.502 INFO 21062 --- [ocp-schedules-13,5de51adede8fd07f,1b1fb66d43359a3a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:35:28.503 INFO 21062 --- [ocp-schedules-20,02b88f37be1258e3,e8c1c0a6cb472f36] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:35:28.503 INFO 21062 --- [ocp-schedules-26,0172d4b4aa756442,d211a418cdd2c6a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:35:28.504 INFO 21062 --- [ocp-schedules-22,deee2233046c84a3,3c614efa0b2c932a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:28.504 INFO 21062 --- [ocp-schedules-6,7a3f0fed61e79a72,408d6249099d788c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:35:28.505 INFO 21062 --- [ocp-schedules-23,dacb468065084408,f545aba5fb046449] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:35:28.510 INFO 21062 --- [ocp-schedules-10,5d540a6e0ed72792,966723eb6d27ff17] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:35:28.510 INFO 21062 --- [ocp-schedules-9,a62785c36d3f02f3,b553f3716db0c30e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:35:28.510 INFO 21062 --- [ocp-schedules-10,5d540a6e0ed72792,966723eb6d27ff17] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:35:28.510 INFO 21062 --- [ocp-schedules-9,a62785c36d3f02f3,b553f3716db0c30e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:35:28.512 INFO 21062 --- [ocp-schedules-4,e7c2ed1a999a80f1,72e3f5b46e8dd959] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:35:28.512 INFO 21062 --- [ocp-schedules-4,e7c2ed1a999a80f1,72e3f5b46e8dd959] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:35:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:35:28.588668+08:00, lastProcessTime=2024-11-04T14:35:18.589948+08:00 2024-11-04 14:35:28.672 INFO 21062 --- [ocp-async-4,4674a9ba42ab5354,838c8e1b321ed256] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:35:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:35:28.588668+08:00 2024-11-04 14:35:28.863 INFO 21062 --- [ocp-schedules-11,cefc844cd2706a9a,d6c27894547e0d86] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:35:29.021 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:29.022 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:29.152 INFO 21062 --- [ocp-schedules-23,8067b6d24c523d68,55b6fa426a08d31a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:35:29.152 INFO 21062 --- [ocp-schedules-23,8067b6d24c523d68,55b6fa426a08d31a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:35:29.231 INFO 21062 --- [ocp-schedules-8,ce2b6d849b927ac1,44f81c16187bbc52] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:35:29.264 WARN 21062 --- [ocp-async-10,fa7acc19d20b7fae,5bcd4b03bb193c00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:29.265 INFO 21062 --- [ocp-schedules-9,e857eb2581152a3a,1fa2a07d392b304a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:35:29.270 INFO 21062 --- [ocp-schedules-9,e857eb2581152a3a,1fa2a07d392b304a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:35:29.270 INFO 21062 --- [ocp-schedules-9,e857eb2581152a3a,1fa2a07d392b304a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:35:29.296 INFO 21062 --- [ocp-async-7,b38540f61a0d0835,6acf73a60828417c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:29.625 INFO 21062 --- [ocp-schedules-16,05a3e0f357987f29,0351b6b21d687e6c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:35:29.625 INFO 21062 --- [ocp-schedules-16,05a3e0f357987f29,0351b6b21d687e6c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:35:29.626 INFO 21062 --- [ocp-schedules-16,05a3e0f357987f29,0351b6b21d687e6c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:35:29.626 INFO 21062 --- [ocp-schedules-16,05a3e0f357987f29,0351b6b21d687e6c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:35:29.627 INFO 21062 --- [ocp-schedules-16,05a3e0f357987f29,0351b6b21d687e6c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:35:29.627 INFO 21062 --- [ocp-schedules-16,05a3e0f357987f29,0351b6b21d687e6c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:35:29.983 INFO 21062 --- [prometheus-minute-schedule-1,71fcd33909b999ff,47a94ac282c8450b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:35:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:35:30.718 INFO 21062 --- [ocp-schedules-11,a2526aa1a996e658,c1a85c03ffa86d62] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:35:31.264 WARN 21062 --- [ocp-async-1,8a5e452a0c060c5a,2ba6841b26e45f79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:31.296 INFO 21062 --- [ocp-async-8,15918f4e28270dda,47f14d8f03ee0c9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:31.796 INFO 21062 --- [ocp-schedules-3,22b4583790d5b334,39537864d1b163a7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:35:31.802 INFO 21062 --- [ocp-schedules-3,22b4583790d5b334,39537864d1b163a7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:35:31.802 INFO 21062 --- [ocp-schedules-3,22b4583790d5b334,39537864d1b163a7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:35:31.884 INFO 21062 --- [ocp-schedules-30,f6d1cdd59c954e94,ff20932dcba2926c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:35:31.884 INFO 21062 --- [ocp-schedules-30,f6d1cdd59c954e94,ff20932dcba2926c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:35:31.911 INFO 21062 --- [ocp-schedules-30,f6d1cdd59c954e94,ff20932dcba2926c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:35:31.911 INFO 21062 --- [ocp-schedules-30,f6d1cdd59c954e94,ff20932dcba2926c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:35:31.912 INFO 21062 --- [ocp-schedules-8,419dfac04f82685e,a731c442181b2751] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:35:32.022 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:32.023 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:32.100 INFO 21062 --- [ocp-schedules-29,c31a49cbcd9b2ef1,a8bcb48c353fa0ac] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:35:32.100+08:00, clusters=[1] 2024-11-04 14:35:32.102 INFO 21062 --- [ocp-schedules-29,c31a49cbcd9b2ef1,a8bcb48c353fa0ac] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:35:32.289 INFO 21062 --- [ocp-schedules-6,17d17d0bbbf58873,433af8ca58718291] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:35:32.289+08:00, clusters=[1] 2024-11-04 14:35:32.292 INFO 21062 --- [ocp-schedules-6,17d17d0bbbf58873,433af8ca58718291] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:35:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d98984e26668a1d8,5021694188fccbc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:33.264 WARN 21062 --- [ocp-async-5,a4139e5f394b68cb,8f56a835219357a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:33.297 INFO 21062 --- [ocp-async-2,f9ca00b590f6516a,4fb4bdecb6efaa1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:33.442 INFO 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:35:33.442 INFO 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:35:33.442 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:35:33.442 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:35:33.454 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:35:33.454 INFO 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:35:33.454 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:35:33.456 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:35:33.456 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:35:33.456 WARN 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:35:33.456 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:35:33.456 INFO 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:35:33.456 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:35:33.458 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:35:33.458 WARN 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:35:33.458 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:35:33.466 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:35:33.467 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:35:33.467 INFO 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:35:33.480 ERROR 21062 --- [ocp-schedules-4,9c482b23dc42c6ac,87bba452f6018b39] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:35:35.023 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:35.024 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:35.265 WARN 21062 --- [ocp-async-3,e5f21b577b5778d3,075ec2f06fb97a9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:35.297 INFO 21062 --- [ocp-async-9,713f29d795be3781,2011f05d7bd5b909] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:35.764 INFO 21062 --- [ocp-schedules-28,73320589ade13f55,05164a2862f67567] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:35:35.983 INFO 21062 --- [prometheus-minute-schedule-1,cb0e813ce370302e,330e75e4c06c88bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:36.948 INFO 21062 --- [ocp-schedules-12,8cd3364a3c545e8d,a848be842cb44633] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:35:36.948 INFO 21062 --- [ocp-schedules-12,6ec7e5bea62db07c,90826b224b63bb51] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:35:36.952 INFO 21062 --- [ocp-schedules-12,6ec7e5bea62db07c,90826b224b63bb51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:36.981 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:35:36.981 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:35:37.265 WARN 21062 --- [ocp-async-1,6b7d9bec6c2e489b,eecb4a235d7e6198] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:37.297 INFO 21062 --- [ocp-async-4,9dec495b46cfd226,7c6ee81ecfbe0889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:38.024 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:38.025 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:38.504 INFO 21062 --- [ocp-schedules-26,0172d4b4aa756442,d211a418cdd2c6a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:35:38.504 INFO 21062 --- [ocp-schedules-22,deee2233046c84a3,3c614efa0b2c932a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:35:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:35:38.589946+08:00, lastProcessTime=2024-11-04T14:35:28.588668+08:00 2024-11-04 14:35:38.672 INFO 21062 --- [ocp-async-10,8792e075f9cfeabd,ac1d1ed6a51fc35a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:35:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:35:38.589946+08:00 2024-11-04 14:35:38.983 INFO 21062 --- [prometheus-minute-schedule-1,7177a41754f4f91c,7335777e2e982fe6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:39.265 WARN 21062 --- [ocp-async-6,77f25562cc1dc93f,7a7ec7c90b0c2ca8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:39.297 INFO 21062 --- [ocp-async-3,4944407c28bac12d,d00564048e45f683] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:39.874 INFO 21062 --- [ocp-schedules-22,ecb34d8fd691a2b6,2072ec60c7e73f2d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:35:39.878 INFO 21062 --- [ocp-schedules-22,ecb34d8fd691a2b6,2072ec60c7e73f2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:39.963 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:35:39.965 ERROR 21062 --- [ocp-schedules-22,ecb34d8fd691a2b6,2072ec60c7e73f2d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:35:39.965 INFO 21062 --- [ocp-schedules-22,ecb34d8fd691a2b6,2072ec60c7e73f2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:35:40.002 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:35:40.002 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:35:40.722 INFO 21062 --- [ocp-schedules-15,cb68191f975c8569,a365aefdba98545b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:35:41.025 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:41.026 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:41.265 WARN 21062 --- [ocp-async-10,c4067da854dfebba,563ab0f02ee80aed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:41.297 INFO 21062 --- [ocp-async-1,9a8566331617f82b,0d329785cc029e9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:41.983 INFO 21062 --- [prometheus-minute-schedule-1,7c1c785a15fb8602,a158308f3b3a7d28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:43.265 WARN 21062 --- [ocp-async-4,9a751e4c66ed8da4,3179fde6ad801ac7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:43.298 INFO 21062 --- [ocp-async-8,5453202a00da454d,40ea461f0f6b2078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:44.026 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:44.027 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e2d99ba9c7794ae7,8ac8b8f67a8eb6ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:45.265 WARN 21062 --- [ocp-async-8,93f916491c817ce0,52676b0143586c48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:45.298 INFO 21062 --- [ocp-async-2,0dc6b5398fc255b5,2ddc0f933464d245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:45.804 INFO 21062 --- [ocp-schedules-6,893b0600c3dc3cde,55eca559f4495dee] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:35:47.027 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:47.028 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:47.265 WARN 21062 --- [ocp-async-2,1c72cda34af961c5,7d349b597e69e404] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:47.298 INFO 21062 --- [ocp-async-9,a83a75d9ba68b2ba,664152e48130212f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:47.983 INFO 21062 --- [prometheus-minute-schedule-1,0fca1de5d776777f,230bf5677097f73e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:35:48.591199+08:00, lastProcessTime=2024-11-04T14:35:38.589946+08:00 2024-11-04 14:35:48.672 INFO 21062 --- [ocp-async-6,17c05035712efe8d,35099204b9332970] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:35:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:35:48.591199+08:00 2024-11-04 14:35:49.266 WARN 21062 --- [ocp-async-1,f0dee5e0e94aac05,b18c588fbf0f25a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:49.298 INFO 21062 --- [ocp-async-8,2961922608bc06b7,3487bd652c8e3079] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:50.028 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:50.029 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:50.725 INFO 21062 --- [ocp-schedules-10,9a23f5168b7630f4,3f08e195042eb826] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:35:50.983 INFO 21062 --- [prometheus-minute-schedule-1,747ab66eb35a5ac1,7d654510906074cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:51.266 WARN 21062 --- [ocp-async-8,4b5ab030dfaabc93,789ab91aaa7bc96b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:51.298 INFO 21062 --- [ocp-async-2,cd078652eb47138d,ecfa3fbeacc0e746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:51.804 INFO 21062 --- [ocp-schedules-30,2a5194cb354825d8,31999b8b5f3a91cb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:35:51.810 INFO 21062 --- [ocp-schedules-30,2a5194cb354825d8,31999b8b5f3a91cb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:35:51.810 INFO 21062 --- [ocp-schedules-30,2a5194cb354825d8,31999b8b5f3a91cb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:35:53.029 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:53.030 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:53.266 WARN 21062 --- [ocp-async-10,f7306afec94ad82a,bea296cf62ed3e19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:53.298 INFO 21062 --- [ocp-async-9,1f70e902211f63b6,a37d0f078ecf1d3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:53.983 INFO 21062 --- [prometheus-minute-schedule-1,be25b5feb37f18f2,493f351dfea796b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:55.266 WARN 21062 --- [ocp-async-1,d267e4d89775bd68,abdbeb702675d48b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:55.298 INFO 21062 --- [ocp-async-7,7099c114f01b63a5,1d4b33029909828e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:55.831 INFO 21062 --- [ocp-schedules-9,dde2f28cbaa92cef,094314a0e916d8a7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:35:56.030 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:56.031 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:56.983 INFO 21062 --- [prometheus-minute-schedule-1,04d94473464db012,043b2ac38f3ea50f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:35:57.266 WARN 21062 --- [ocp-async-6,41ab33b343e2a6dc,958bd595ab5a7495] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:57.299 INFO 21062 --- [ocp-async-5,af4573dc97ca56ee,6f73a1241bd09174] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:58.500 INFO 21062 --- [ocp-schedules-4,77bbdd8ddb6ef353,b459368600bd82d7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:35:58.500 INFO 21062 --- [ocp-schedules-6,95d2723bb9565ac2,708e74661eee5f03] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:35:58.502 INFO 21062 --- [ocp-schedules-26,74b7f6cdac5eb1d5,29ffb4d562cfcfe3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:35:58.504 INFO 21062 --- [ocp-schedules-4,77bbdd8ddb6ef353,b459368600bd82d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:35:58.505 INFO 21062 --- [ocp-schedules-1,a661694e37d1e13e,075145b617bcd6c5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:35:58.505 INFO 21062 --- [ocp-schedules-6,95d2723bb9565ac2,708e74661eee5f03] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:58.510 INFO 21062 --- [ocp-schedules-18,2342854b51cfc658,adbe924bd6265017] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:35:58.510 INFO 21062 --- [ocp-schedules-18,2342854b51cfc658,adbe924bd6265017] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:35:58.510 INFO 21062 --- [ocp-schedules-11,9890de302c260735,1c878b16253ecf3b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:35:58.510 INFO 21062 --- [ocp-schedules-11,9890de302c260735,1c878b16253ecf3b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:35:58.510 INFO 21062 --- [ocp-schedules-9,ded88d8a9d0e260f,b5e115db3660a8bd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:35:58.510 INFO 21062 --- [ocp-schedules-9,ded88d8a9d0e260f,b5e115db3660a8bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:35:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:35:58.590586+08:00, lastProcessTime=2024-11-04T14:35:48.591199+08:00 2024-11-04 14:35:58.672 INFO 21062 --- [ocp-async-1,b23e7e116d5f0d59,804ff5e60d5cfbd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:35:58.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:35:58.590586+08:00 2024-11-04 14:35:59.031 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:35:59.032 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:35:59.075 INFO 21062 --- [ocp-schedules-7,281a1575540a0a74,b90a1da122120192] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:35:59.078 INFO 21062 --- [ocp-schedules-7,281a1575540a0a74,b90a1da122120192] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:35:59.079 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:35:59.235 INFO 21062 --- [ocp-schedules-28,aa6a5057c8d6ca5e,d13623786b5dbdf0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:35:59.267 WARN 21062 --- [ocp-async-2,73f2515533ab73cf,ef4da0f92edd777c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:35:59.299 INFO 21062 --- [ocp-async-1,fdb39a59e3da1fa0,3b7050d1b9dd471d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:35:59.983 INFO 21062 --- [prometheus-minute-schedule-1,0470f5fb05839837,538cd751e8401678] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702160, interval=60 2024-11-04 14:36:00.727 INFO 21062 --- [ocp-schedules-28,198209719ff33384,b0f43ae3e7386f09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:36:01.267 WARN 21062 --- [ocp-async-9,c4d35ac20204bae2,78afecbafddad6c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:01.299 INFO 21062 --- [ocp-async-6,22ebfe6bffd0e04b,4d7dd7530459ca9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:02.032 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:02.033 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f10769c1af74c9ab,ecc9606d2a59ffaa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:03.267 WARN 21062 --- [ocp-async-7,7f9e41a896ce5112,219508b1423e0ca7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:03.299 INFO 21062 --- [ocp-async-5,3aebef6869cbfdb9,6aaaa67aa4f11517] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:05.033 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:05.034 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:05.267 WARN 21062 --- [ocp-async-2,278b817eb412ce7f,b738db18121bc932] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:05.299 INFO 21062 --- [ocp-async-10,8c5b3049c2df5594,eef49647bd8dec87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:05.873 INFO 21062 --- [ocp-schedules-23,3b637e67404bcc8e,7cf0258b8f17d867] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:36:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f280eac4c8add11f,bd49aa2052c42ed1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:06.982 INFO 21062 --- [ocp-schedules-12,6ec7e5bea62db07c,90826b224b63bb51] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:36:06.982 INFO 21062 --- [ocp-schedules-12,367064f5ffcdc78d,21a555c94e59f9f3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:36:06.986 INFO 21062 --- [ocp-schedules-12,367064f5ffcdc78d,21a555c94e59f9f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:07.017 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:36:07.017 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:36:07.267 WARN 21062 --- [ocp-async-9,4ed85df51417535f,e66e906a47704eb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:07.300 INFO 21062 --- [ocp-async-1,aaab06a51459cca0,8af9c693d1994c0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:08.034 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:08.035 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:08.505 INFO 21062 --- [ocp-schedules-4,77bbdd8ddb6ef353,b459368600bd82d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:36:08.506 INFO 21062 --- [ocp-schedules-6,95d2723bb9565ac2,708e74661eee5f03] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:36:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:36:08.591189+08:00, lastProcessTime=2024-11-04T14:35:58.590586+08:00 2024-11-04 14:36:08.673 INFO 21062 --- [ocp-async-2,ec993e1ad807d4fc,a91336930fad4d46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:36:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:36:08.591189+08:00 2024-11-04 14:36:08.983 INFO 21062 --- [prometheus-minute-schedule-1,178b6dcf8c755833,e095df1beaebcde0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:09.079 INFO 21062 --- [ocp-schedules-7,281a1575540a0a74,b90a1da122120192] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:36:09.267 WARN 21062 --- [ocp-async-6,7c71437ed8c477f7,b58c1ce35b3383e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:09.300 INFO 21062 --- [ocp-async-8,c9432c17f7b55c51,976f66a3eca721ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:10.002 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:36:10.002 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:36:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:36:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:36:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:36:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:36:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:36:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:36:10.730 INFO 21062 --- [ocp-schedules-25,767eeb2c9418b10b,425c08b54fb9c036] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:36:11.035 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:11.036 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:11.268 WARN 21062 --- [ocp-async-5,2e536e53daf24198,88109cd93f7191de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:11.300 INFO 21062 --- [ocp-async-3,2449adee61c088bd,0eef35abeda9ef61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:11.610 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=5, affectRows=5 2024-11-04 14:36:11.610 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=5, validCount=5 2024-11-04 14:36:11.610 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=27, affectRows=27 2024-11-04 14:36:11.610 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=15, affectRows=15 2024-11-04 14:36:11.610 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=15, validCount=15 2024-11-04 14:36:11.610 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=27, validCount=27 2024-11-04 14:36:11.611 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=41, affectRows=41 2024-11-04 14:36:11.611 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=42, affectRows=42 2024-11-04 14:36:11.611 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=41, validCount=41 2024-11-04 14:36:11.611 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=42, validCount=42 2024-11-04 14:36:11.611 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=28, affectRows=28 2024-11-04 14:36:11.611 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=28, validCount=28 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=67, affectRows=67 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=69, affectRows=69 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=24, affectRows=24 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702100, nodeCount=78, affectRows=78 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=24, validCount=24 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=66, validCount=66 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=69, validCount=69 2024-11-04 14:36:11.612 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702100, nodeCount=76, validCount=76 2024-11-04 14:36:11.813 INFO 21062 --- [ocp-schedules-27,2a4b96732457daca,1fac3ebec32b3997] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:36:11.820 INFO 21062 --- [ocp-schedules-27,2a4b96732457daca,1fac3ebec32b3997] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:36:11.820 INFO 21062 --- [ocp-schedules-27,2a4b96732457daca,1fac3ebec32b3997] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:36:11.983 INFO 21062 --- [prometheus-minute-schedule-1,d44dcb22d3d023e2,9add36415f2fe30b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:13.268 WARN 21062 --- [ocp-async-10,1d76ab654a6c111a,3a25dc1af300c89b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:13.300 INFO 21062 --- [ocp-async-4,035601fea49b214e,405c6dd0d2064ce5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:13.984 INFO 21062 --- [prometheus-cache-monitor1,29fab1dee2eadac4,a0e163af735fc366] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:36:13.984 INFO 21062 --- [prometheus-cache-monitor1,29fab1dee2eadac4,a0e163af735fc366] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:36:14.036 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:14.037 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5b1e658f499cb92d,445f99abd24dbb21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:15.268 WARN 21062 --- [ocp-async-1,b64d232636c0a845,645b916769f5bc70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:15.300 INFO 21062 --- [ocp-async-7,3909ae6f840cfddd,407cdf530ee5014c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:15.898 INFO 21062 --- [ocp-schedules-28,0819e736b1902e30,6b4a32fac3a3b2de] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:36:17.037 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:17.038 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:17.268 WARN 21062 --- [ocp-async-2,a6de3318b1e9631a,f20fbb4c8e6e832b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:17.301 INFO 21062 --- [ocp-async-8,5e966552af637f06,9b926c27fc7a705e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d1dd9663de5192d7,85e5c21a0a61b5ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:36:18.590464+08:00, lastProcessTime=2024-11-04T14:36:08.591189+08:00 2024-11-04 14:36:18.673 INFO 21062 --- [ocp-async-1,c38793b8a1b56ef0,49f610ffd11bf279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:36:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:36:18.590464+08:00 2024-11-04 14:36:19.268 WARN 21062 --- [ocp-async-10,84edf26d48efec72,b6ffe88828041f41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:19.301 INFO 21062 --- [ocp-async-4,98c1dae9a5cc4e13,491231695b98ce25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:20.038 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:20.039 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:20.734 INFO 21062 --- [ocp-schedules-27,84cb1a9dadbd970d,389dc3a014a0ef9a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:36:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ea5f7a28397989cf,307e3cfeead9299e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:21.268 WARN 21062 --- [ocp-async-1,f3e0ef0de392c8ac,8f6e0c45134d6da7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:21.301 INFO 21062 --- [ocp-async-7,d7f6511c3de4d1c3,f455828c1a212db8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:23.040 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:23.040 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:23.269 WARN 21062 --- [ocp-async-5,0a62da25594e3290,057d48d4354ddd84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:23.301 INFO 21062 --- [ocp-async-8,f9b312ac63359ae8,a982403bf0e3d162] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:23.983 INFO 21062 --- [prometheus-minute-schedule-1,963b76ec634396b3,c390a38b94c36a01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:25.269 WARN 21062 --- [ocp-async-10,23feeb8e8179b4b5,d6dee6123c0ae564] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:25.302 INFO 21062 --- [ocp-async-3,52e3e8cb46b44639,8942d3f2768a2afa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:25.940 INFO 21062 --- [ocp-schedules-10,5c10e3076fc1493a,4b8710f674886880] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:36:26.041 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:26.041 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:26.983 INFO 21062 --- [prometheus-minute-schedule-1,71aa8088ec480b88,e61a9917398c02b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:26.984 INFO 21062 --- [prometheus-hour-schedule-1,57fbb44237d918c0,1868902dc1fc15ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:36:27.269 WARN 21062 --- [ocp-async-4,a7e5992f7c4e8cbb,df1f252d1f151ce8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:27.302 INFO 21062 --- [ocp-async-6,1de0e13ecfe98922,561b207d3f3d35e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:28.498 INFO 21062 --- [ocp-schedules-24,cd3e9bb654164596,c081b1d18665b363] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:36:28.499 INFO 21062 --- [ocp-schedules-1,db2ae0e21c7fcd3f,5971855833a4bc63] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:36:28.500 INFO 21062 --- [ocp-schedules-6,51e002d0ac02f126,09408bb2894db339] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:36:28.500 INFO 21062 --- [ocp-schedules-8,9fba40aa3c79421a,0c279862775ac48b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:36:28.502 INFO 21062 --- [ocp-schedules-2,dec65b9f223263b0,45fd5822baa66193] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:36:28.503 INFO 21062 --- [ocp-schedules-24,cd3e9bb654164596,c081b1d18665b363] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:36:28.504 INFO 21062 --- [ocp-schedules-8,9fba40aa3c79421a,0c279862775ac48b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:28.505 INFO 21062 --- [ocp-schedules-10,52606efb9776e7e6,1039e764bfeb2170] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:36:28.505 INFO 21062 --- [ocp-schedules-6,51e002d0ac02f126,09408bb2894db339] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:36:28.506 INFO 21062 --- [ocp-schedules-1,db2ae0e21c7fcd3f,5971855833a4bc63] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:36:28.510 INFO 21062 --- [ocp-schedules-27,59d80566788d6c7d,82529402e744061d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:36:28.510 INFO 21062 --- [ocp-schedules-15,92373eff4850a892,ebf12c1e56f65fac] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:36:28.510 INFO 21062 --- [ocp-schedules-27,59d80566788d6c7d,82529402e744061d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:36:28.510 INFO 21062 --- [ocp-schedules-15,92373eff4850a892,ebf12c1e56f65fac] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:36:28.511 INFO 21062 --- [ocp-schedules-21,971e85575bce5c82,d7633c9ccca1c346] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:36:28.511 INFO 21062 --- [ocp-schedules-21,971e85575bce5c82,d7633c9ccca1c346] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:36:28.578 INFO 21062 --- [ocp-schedules-25,ab9a536a3035f6ea,f58382355f8b69bd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:36:28.579 INFO 21062 --- [ocp-schedules-25,ab9a536a3035f6ea,f58382355f8b69bd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:36:28.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:36:28.592249+08:00, lastProcessTime=2024-11-04T14:36:18.590464+08:00 2024-11-04 14:36:28.636 INFO 21062 --- [ocp-schedules-18,6d88f2f03e91b1ed,6f9a1d2d3e236be4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:36:28.637 INFO 21062 --- [ocp-schedules-18,6d88f2f03e91b1ed,6f9a1d2d3e236be4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:36:28.673 INFO 21062 --- [ocp-async-3,6cafea220bcf0482,bb1bc98e1ba72844] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:36:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:36:28.592249+08:00 2024-11-04 14:36:28.865 INFO 21062 --- [ocp-schedules-28,221edf60a3a02b4e,6e8300a3d6daef0b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:36:29.042 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:29.042 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:29.157 INFO 21062 --- [ocp-schedules-21,456b317fa321747f,b470dcdb5771a9de] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:36:29.157 INFO 21062 --- [ocp-schedules-21,456b317fa321747f,b470dcdb5771a9de] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:36:29.238 INFO 21062 --- [ocp-schedules-24,115ca8ef7dfab5f1,54c7327dad0538e9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:36:29.269 WARN 21062 --- [ocp-async-2,d9bd77689381ad67,24b8129f81fc7578] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:29.270 INFO 21062 --- [ocp-schedules-11,28a8f2d35c535073,03c4e1b29b3463b9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:36:29.276 INFO 21062 --- [ocp-schedules-11,28a8f2d35c535073,03c4e1b29b3463b9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:36:29.276 INFO 21062 --- [ocp-schedules-11,28a8f2d35c535073,03c4e1b29b3463b9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:36:29.302 INFO 21062 --- [ocp-async-9,fab826d571eeea34,d074738467c98260] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:29.628 INFO 21062 --- [ocp-schedules-19,02f87ec2b6affe5f,ea28fcb72768a4ae] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:36:29.628 INFO 21062 --- [ocp-schedules-19,02f87ec2b6affe5f,ea28fcb72768a4ae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:36:29.629 INFO 21062 --- [ocp-schedules-19,02f87ec2b6affe5f,ea28fcb72768a4ae] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:36:29.629 INFO 21062 --- [ocp-schedules-19,02f87ec2b6affe5f,ea28fcb72768a4ae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:36:29.631 INFO 21062 --- [ocp-schedules-19,02f87ec2b6affe5f,ea28fcb72768a4ae] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:36:29.631 INFO 21062 --- [ocp-schedules-19,02f87ec2b6affe5f,ea28fcb72768a4ae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:36:29.983 INFO 21062 --- [prometheus-minute-schedule-1,adcafaccb9cc65f1,cdb2e577ecaac39c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:36:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:36:30.738 INFO 21062 --- [ocp-schedules-29,189a32c3de28876d,c5feb93b438e268f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:36:31.269 WARN 21062 --- [ocp-async-3,f9c00e449faa5d19,864f3a07756cacd9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:31.302 INFO 21062 --- [ocp-async-4,84b3b60df79119cd,dba5fa9cb149a9f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:31.823 INFO 21062 --- [ocp-schedules-16,25b2ed03c227a7b5,b70209935732bf61] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:36:31.828 INFO 21062 --- [ocp-schedules-16,25b2ed03c227a7b5,b70209935732bf61] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:36:31.828 INFO 21062 --- [ocp-schedules-16,25b2ed03c227a7b5,b70209935732bf61] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:36:31.911 INFO 21062 --- [ocp-schedules-28,5cfab322c4188525,a388eb7fe3254696] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:36:31.912 INFO 21062 --- [ocp-schedules-28,5cfab322c4188525,a388eb7fe3254696] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:36:31.939 INFO 21062 --- [ocp-schedules-28,5cfab322c4188525,a388eb7fe3254696] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:36:31.939 INFO 21062 --- [ocp-schedules-28,5cfab322c4188525,a388eb7fe3254696] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:36:31.941 INFO 21062 --- [ocp-schedules-2,5f7cf13a7314c749,53745bdc6dd1cdfb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:36:32.042 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:32.043 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:32.130 INFO 21062 --- [ocp-schedules-19,cd3b235e85c33b45,d078ec210ff316a4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:36:32.130+08:00, clusters=[1] 2024-11-04 14:36:32.131 INFO 21062 --- [ocp-schedules-19,cd3b235e85c33b45,d078ec210ff316a4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:36:32.318 INFO 21062 --- [ocp-schedules-18,5f8ebb0c131ba206,04782f993450b8e2] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:36:32.319 INFO 21062 --- [ocp-schedules-23,c241cc3b56000280,f2c96673e902727f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:36:32.319+08:00, clusters=[1] 2024-11-04 14:36:32.322 INFO 21062 --- [ocp-schedules-18,5f8ebb0c131ba206,04782f993450b8e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:32.322 INFO 21062 --- [ocp-schedules-23,c241cc3b56000280,f2c96673e902727f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:36:32.380 ERROR 21062 --- [ocp-schedules-18,5f8ebb0c131ba206,04782f993450b8e2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:36:32.380 INFO 21062 --- [ocp-schedules-18,5f8ebb0c131ba206,04782f993450b8e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:36:32.983 INFO 21062 --- [prometheus-minute-schedule-1,00c3ac00e6865a25,4b13b4c1af047be7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:33.270 WARN 21062 --- [ocp-async-6,4859b53a1d040844,0745f4f64e724a0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:33.302 INFO 21062 --- [ocp-async-5,c1ba6cc61aedd81f,d0ff0c74ef49cabc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:33.489 INFO 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:36:33.489 INFO 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:36:33.489 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:36:33.489 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:36:33.499 INFO 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:36:33.499 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:36:33.500 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:36:33.501 WARN 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:36:33.501 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:36:33.501 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:36:33.502 INFO 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:36:33.502 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:36:33.502 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:36:33.503 WARN 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:36:33.503 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:36:33.503 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:36:33.510 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:36:33.510 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:36:33.510 INFO 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:36:33.523 ERROR 21062 --- [ocp-schedules-19,77fe564bb18ecc28,0b3c0651eefc6a06] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:36:35.044 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:35.045 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:35.270 WARN 21062 --- [ocp-async-2,c9d60290557040ab,5b29085d4cac70cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:35.303 INFO 21062 --- [ocp-async-10,c4b8239baeda8532,63726dcec2764e8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:35.966 INFO 21062 --- [ocp-schedules-2,14d3036bdf9f442f,86c7fe4c6c59e0b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:36:35.983 INFO 21062 --- [prometheus-minute-schedule-1,8014d535cae49d2b,3c772f33acf233c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:37.018 INFO 21062 --- [ocp-schedules-12,367064f5ffcdc78d,21a555c94e59f9f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:36:37.018 INFO 21062 --- [ocp-schedules-12,d59fe1303ad67d61,c2ce73061fa287d0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:36:37.022 INFO 21062 --- [ocp-schedules-12,d59fe1303ad67d61,c2ce73061fa287d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:37.050 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:36:37.050 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:36:37.270 WARN 21062 --- [ocp-async-3,5c0f971d12046ddf,ba2ed369b9b05c3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:37.303 INFO 21062 --- [ocp-async-1,c55704d8aca1bab6,d1e3dd04997eef08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:38.045 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:38.046 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:38.505 INFO 21062 --- [ocp-schedules-8,9fba40aa3c79421a,0c279862775ac48b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:36:38.506 INFO 21062 --- [ocp-schedules-6,51e002d0ac02f126,09408bb2894db339] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:36:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:36:38.590792+08:00, lastProcessTime=2024-11-04T14:36:28.592249+08:00 2024-11-04 14:36:38.673 INFO 21062 --- [ocp-async-8,af26ae6e600311dd,99f7ee2284714445] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:36:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:36:38.590792+08:00 2024-11-04 14:36:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c79060183f105723,6c6bd79f13804ad6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:39.270 WARN 21062 --- [ocp-async-5,ae90407e54454ecb,627c018dc891bc07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:39.303 INFO 21062 --- [ocp-async-2,1c9ac84199cb8901,bb8f027b71ebb495] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:39.965 INFO 21062 --- [ocp-schedules-26,18b78bc8c6f465fb,0d4153672de35720] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:36:39.969 INFO 21062 --- [ocp-schedules-26,18b78bc8c6f465fb,0d4153672de35720] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:40.002 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:36:40.002 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:36:40.056 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:36:40.064 ERROR 21062 --- [ocp-schedules-26,18b78bc8c6f465fb,0d4153672de35720] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:36:40.064 INFO 21062 --- [ocp-schedules-26,18b78bc8c6f465fb,0d4153672de35720] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:36:40.740 INFO 21062 --- [ocp-schedules-19,90a84ef2988fc67e,fe4bbfbcfeb583cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:36:41.046 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:41.047 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:41.270 WARN 21062 --- [ocp-async-8,cce08e9acd15928d,f223241a182b5d30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:41.303 INFO 21062 --- [ocp-async-3,a2686dc1dc03f2a7,36f68dbac5466b9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:36:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:36:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:36:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6fa4f6d972b06187,9969b9897a0e2edc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:43.270 WARN 21062 --- [ocp-async-1,6ea2d9c93bb18086,ba953e08650e1e38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:43.303 INFO 21062 --- [ocp-async-4,82f8a008e34097a2,1eb8c1ee6fd48aa9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:44.047 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:44.048 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a3f6091d2cf8ac04,ad47c44a8931b991] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:45.271 WARN 21062 --- [ocp-async-6,49734fcb5dc18ff7,17eb60f94c5395dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:45.303 INFO 21062 --- [ocp-async-7,16ad6e8c053c7460,1c6050e5569bfa2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:46.009 INFO 21062 --- [ocp-schedules-27,8f720b186a4fa6ed,6c0efc9da76e4ad4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:36:47.048 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:47.049 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:47.271 WARN 21062 --- [ocp-async-2,daa4c5a7c84f10c0,65b4f2434660a714] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:47.304 INFO 21062 --- [ocp-async-10,63b2e0093212e029,e3781724cc77f6ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:47.983 INFO 21062 --- [prometheus-minute-schedule-1,d75feccd2f403c93,4ae9526f22a877af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:36:48.590798+08:00, lastProcessTime=2024-11-04T14:36:38.590792+08:00 2024-11-04 14:36:48.673 INFO 21062 --- [ocp-async-6,89b67e24afcd40fb,8c628f9b169fb738] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:36:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:36:48.590798+08:00 2024-11-04 14:36:49.271 WARN 21062 --- [ocp-async-1,1cedd2ef1ae8b0c9,ce26860a76e688fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:49.304 INFO 21062 --- [ocp-async-5,122e6b2a0f8f5498,05cec98874949c06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:50.049 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:50.050 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:50.743 INFO 21062 --- [ocp-schedules-13,8428d1cb4561dafd,2733b4d8e31375b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:36:50.983 INFO 21062 --- [prometheus-minute-schedule-1,73c046d58163528b,5b6d1ce7aaddf528] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:51.169 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:36:51.169 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:36:51.177 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:36:51.177 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:36:51.177 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:36:51.177 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:36:51.185 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:36:51.185 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:36:51.185 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:36:51.185 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:36:51.221 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:36:51.229 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:36:51.229 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:36:51.229 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:36:51.229 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:36:51.229 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:36:51.237 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:36:51.271 WARN 21062 --- [ocp-async-6,7cb48a51e9c6a666,72fb4b30d090a02e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:51.304 INFO 21062 --- [ocp-async-2,1c0b170684764792,71aaabc084a4e898] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:51.748 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:36:51.832 INFO 21062 --- [ocp-schedules-9,4cea074ed3739083,91cdd3e2ac755257] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:36:51.837 INFO 21062 --- [ocp-schedules-9,4cea074ed3739083,91cdd3e2ac755257] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:36:51.838 INFO 21062 --- [ocp-schedules-9,4cea074ed3739083,91cdd3e2ac755257] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:36:52.340 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.155567198S 2024-11-04 14:36:52.341 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:36:52.439 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:36:52.439 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:36:52.466 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:36:52.524 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:36:52.524 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:36:52.524 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:36:52.524 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:36:52.524 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:36:52.524 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:36:52.535 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:36:52.707 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] o.s.boot.SpringApplication : Started application in 1.53 seconds (JVM running for 7626.386) 2024-11-04 14:36:52.707 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.530505329S 2024-11-04 14:36:53.050 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:53.051 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:53.137 INFO 21062 --- [ocp-schedules-17,d3a49b63fbe8a5da,814ad3c31ee4e32c] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:36:53.271 WARN 21062 --- [ocp-async-2,5f938724c5c9b42b,21d44958d825dc98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:53.304 INFO 21062 --- [ocp-async-10,0df6d50280eba060,e5d9ae460e242162] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:53.983 INFO 21062 --- [prometheus-minute-schedule-1,b8ba48fb34dbcd98,cefe101a69341c33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:55.271 WARN 21062 --- [ocp-async-10,b1d5ad8aba590e27,7e60afd89afafbe6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:55.304 INFO 21062 --- [ocp-async-9,6c8ac0a42a5875af,3d18a60f1ca8eae1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:56.034 INFO 21062 --- [ocp-schedules-9,814184a6fb68ee4f,96dd96220b1cfcf9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:36:56.051 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:56.051 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7a6d4a31583485b5,f339eab5829c62a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:36:57.271 WARN 21062 --- [ocp-async-9,c969ba55bf244a8f,ca3b8a1dd34d3248] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:57.304 INFO 21062 --- [ocp-async-4,7f49e7de974ab3d2,d75a1439da02863a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:58.500 INFO 21062 --- [ocp-schedules-13,9a02822f2266b98e,da2babddcc3d91a2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:36:58.500 INFO 21062 --- [ocp-schedules-5,6fa0eeccd4ace354,3af86e49f4e7234d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:36:58.502 INFO 21062 --- [ocp-schedules-19,d0b8f4c437948ec4,4d665bbc48429cb6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:36:58.503 INFO 21062 --- [ocp-schedules-13,9a02822f2266b98e,da2babddcc3d91a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:36:58.504 INFO 21062 --- [ocp-schedules-24,ab0ccd66642b2bf5,aa7a9189fe4f8b6f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:36:58.507 INFO 21062 --- [ocp-schedules-5,6fa0eeccd4ace354,3af86e49f4e7234d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:36:58.510 INFO 21062 --- [ocp-schedules-28,7565c91669559a03,92bb48e67596d791] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:36:58.510 INFO 21062 --- [ocp-schedules-20,053ae616ab0514ec,4e1d05e0b82a36d4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:36:58.510 INFO 21062 --- [ocp-schedules-20,053ae616ab0514ec,4e1d05e0b82a36d4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:36:58.510 INFO 21062 --- [ocp-schedules-28,7565c91669559a03,92bb48e67596d791] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:36:58.511 INFO 21062 --- [ocp-schedules-29,84c075afe4a36924,a30c5a7acc39875d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:36:58.511 INFO 21062 --- [ocp-schedules-29,84c075afe4a36924,a30c5a7acc39875d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:36:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:36:58.590566+08:00, lastProcessTime=2024-11-04T14:36:48.590798+08:00 2024-11-04 14:36:58.673 INFO 21062 --- [ocp-async-10,4b2579dd8b3c6122,241f5061a6c7e24a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:36:58.748 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:36:58.590566+08:00 2024-11-04 14:36:59.052 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:36:59.052 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:36:59.241 INFO 21062 --- [ocp-schedules-10,4d636f19dabbd8f5,8a8e53f08d602258] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:36:59.272 WARN 21062 --- [ocp-async-2,32fb1c0c8a96b128,3dc312b85682d65b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:36:59.305 INFO 21062 --- [ocp-async-10,4f179f88e01f3425,cf1dd60e2d119c64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:36:59.983 INFO 21062 --- [prometheus-minute-schedule-1,4d8f03c6102bc319,47589944fe7bbe8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702220, interval=60 2024-11-04 14:37:00.747 INFO 21062 --- [ocp-schedules-14,f4c6156fd4739620,155b08322de00af3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:37:01.272 WARN 21062 --- [ocp-async-10,97139b9a66cc80ad,1e047f9fb56a18a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:01.305 INFO 21062 --- [ocp-async-9,aab0cd43a8d505d6,886a1305922efd2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:02.053 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:02.053 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:02.983 INFO 21062 --- [prometheus-minute-schedule-1,0fc65a3a23bdc3b9,d7c63bf488b7d6bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:03.272 WARN 21062 --- [ocp-async-9,567bdcfaa9c2eeec,f6a36af961f46236] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:03.305 INFO 21062 --- [ocp-async-5,28df8bd2147312a0,a3e239d96065590e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:05.054 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:05.054 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:05.272 WARN 21062 --- [ocp-async-5,6c73b06439b43ced,b4a4f45d57d5334d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:05.305 INFO 21062 --- [ocp-async-2,94aae97859b0d13b,25f12cf834e04727] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:05.983 INFO 21062 --- [prometheus-minute-schedule-1,4c659e2b82777591,3e70325b47f86083] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:06.059 INFO 21062 --- [ocp-schedules-23,ee12d29f8459b691,7ac535b20891e06f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:37:07.051 INFO 21062 --- [ocp-schedules-12,d59fe1303ad67d61,c2ce73061fa287d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:37:07.051 INFO 21062 --- [ocp-schedules-12,5c0d410dc348ecb9,8819a07dd47b9544] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:37:07.055 INFO 21062 --- [ocp-schedules-12,5c0d410dc348ecb9,8819a07dd47b9544] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:07.084 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:37:07.084 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:37:07.272 WARN 21062 --- [ocp-async-2,f8af5310b677f470,63699831f40df830] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:07.305 INFO 21062 --- [ocp-async-10,f54870b1d9930c2f,9fb8b27e4bff9e56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:08.054 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:08.055 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:08.504 INFO 21062 --- [ocp-schedules-13,9a02822f2266b98e,da2babddcc3d91a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:37:08.507 INFO 21062 --- [ocp-schedules-5,6fa0eeccd4ace354,3af86e49f4e7234d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:37:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:37:08.590529+08:00, lastProcessTime=2024-11-04T14:36:58.590566+08:00 2024-11-04 14:37:08.673 INFO 21062 --- [ocp-async-5,b79c9bab29f6a15d,a8e6c466a0eaf65c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:37:08.738 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:37:08.590529+08:00 2024-11-04 14:37:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ad76fe05b0ceb890,9a2525d1b4584c3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:09.079 INFO 21062 --- [ocp-schedules-28,403b7833d7490d31,7482de7e00f5b600] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:37:09.083 INFO 21062 --- [ocp-schedules-28,403b7833d7490d31,7482de7e00f5b600] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:09.084 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:37:09.272 WARN 21062 --- [ocp-async-9,63b1edd7c36d73e6,4c12f12480fe0d96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:09.305 INFO 21062 --- [ocp-async-4,71543df9bce978c0,e182e2d1114fbd33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:37:10.007 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:37:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:37:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:37:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:37:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:37:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:37:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:37:10.750 INFO 21062 --- [ocp-schedules-8,8c7f9dd15e2e070a,4636c98b112c01b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:37:11.055 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:11.056 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:11.273 WARN 21062 --- [ocp-async-4,f34b7c2415f9c7a6,63bbe4f86456821c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:11.306 INFO 21062 --- [ocp-async-7,a818932f7f7b39c4,743688c644504142] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:11.615 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702160, nodeCount=60, affectRows=60 2024-11-04 14:37:11.615 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702160, nodeCount=60, validCount=60 2024-11-04 14:37:11.615 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702160, nodeCount=68, affectRows=68 2024-11-04 14:37:11.615 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702160, nodeCount=75, affectRows=75 2024-11-04 14:37:11.615 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702160, nodeCount=72, affectRows=72 2024-11-04 14:37:11.615 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702160, nodeCount=49, affectRows=49 2024-11-04 14:37:11.616 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702160, nodeCount=67, validCount=67 2024-11-04 14:37:11.616 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702160, nodeCount=71, validCount=71 2024-11-04 14:37:11.616 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702160, nodeCount=49, validCount=49 2024-11-04 14:37:11.616 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702160, nodeCount=75, validCount=75 2024-11-04 14:37:11.616 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702160, nodeCount=72, affectRows=72 2024-11-04 14:37:11.616 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702160, nodeCount=71, validCount=71 2024-11-04 14:37:11.840 INFO 21062 --- [ocp-schedules-20,932c588d1e4ab512,193dd6faa8294d73] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:37:11.846 INFO 21062 --- [ocp-schedules-20,932c588d1e4ab512,193dd6faa8294d73] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:37:11.846 INFO 21062 --- [ocp-schedules-20,932c588d1e4ab512,193dd6faa8294d73] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:37:11.983 INFO 21062 --- [prometheus-minute-schedule-1,57ac98ddea652157,ad0073bd0cf86279] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:13.273 WARN 21062 --- [ocp-async-7,0aa460974f6f402d,ab2f456223b1fe6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:13.306 INFO 21062 --- [ocp-async-3,009271b75bec46e1,217d489e9c9ceaa6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:13.984 INFO 21062 --- [prometheus-cache-monitor1,3cdb4133ba546246,7c69be4179f38ae5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:37:13.984 INFO 21062 --- [prometheus-cache-monitor1,3cdb4133ba546246,7c69be4179f38ae5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:37:14.056 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:14.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:14.983 INFO 21062 --- [prometheus-minute-schedule-1,81cd8b7a5dd77008,a7a9cfe3580bda91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:15.167 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:33:15.144, end=2024-11-04T14:35:15.164} 2024-11-04 14:37:15.176 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:37:15.176 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:37:15.273 WARN 21062 --- [ocp-async-10,2ceb7faae859c01f,84d0508400e968d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:15.306 INFO 21062 --- [ocp-async-6,352c94cf44099a98,61a0c17852e7133f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:16.083 INFO 21062 --- [ocp-schedules-9,10ec144e55e7533b,08e92beafe450b84] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:37:17.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:17.058 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:17.273 WARN 21062 --- [ocp-async-5,3d7e7a9b681ad80d,156719184825a9c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:17.306 INFO 21062 --- [ocp-async-8,f4e3eb4ea6730f49,b2d4e0cdb7483e93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:17.983 INFO 21062 --- [prometheus-minute-schedule-1,56121bce8a16a434,b747af76b879724a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:37:18.590769+08:00, lastProcessTime=2024-11-04T14:37:08.590529+08:00 2024-11-04 14:37:18.674 INFO 21062 --- [ocp-async-10,ec8171a3bc7c1b74,f89bea99ce0b3653] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:37:18.796 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:37:18.590769+08:00 2024-11-04 14:37:19.084 INFO 21062 --- [ocp-schedules-28,403b7833d7490d31,7482de7e00f5b600] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:37:19.273 WARN 21062 --- [ocp-async-7,a3d5df79f038911d,2b0bf7b4cfe49505] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:19.306 INFO 21062 --- [ocp-async-3,c5f4d0f218864187,28dc49f26b7910a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:20.058 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:20.059 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:20.753 INFO 21062 --- [ocp-schedules-14,dec986b8c3b69b21,16422144f4ed6708] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:37:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8c2a35f07da1fcc0,f0d11cbfd1052364] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:21.273 WARN 21062 --- [ocp-async-10,da81669720791164,3b64557bd54ae04a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:21.306 INFO 21062 --- [ocp-async-6,44eac7f5b8feb1df,a17635cc6170e606] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:23.059 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:23.060 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:23.274 WARN 21062 --- [ocp-async-5,8b862d4536dd1100,d26584a145a0f408] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:23.307 INFO 21062 --- [ocp-async-8,30672097868e0340,3a292dd1cb9ac254] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:23.983 INFO 21062 --- [prometheus-minute-schedule-1,9fae49b41286895d,2cf98d5e8653adf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:25.274 WARN 21062 --- [ocp-async-2,e74d114b6c1a7042,bfb6097612836c35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:25.307 INFO 21062 --- [ocp-async-1,3024bdd8c9bd6d3c,82b25c8f93bd0de6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:26.060 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:26.061 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:26.123 INFO 21062 --- [ocp-schedules-11,4eba4bb7b41bf129,012439b2fdd8dde9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:37:26.983 INFO 21062 --- [prometheus-minute-schedule-1,c5d4c9ba3b055a73,ac94bf4aeeef0d74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b1d4f2820b4d3497,97d38746ed650de6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:37:27.274 WARN 21062 --- [ocp-async-3,50b57139ebf74c14,2d2be8eee8e1c99e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:27.307 INFO 21062 --- [ocp-async-9,b8f23e01c1272bde,e0cb482a3628572c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:28.498 INFO 21062 --- [ocp-schedules-27,fe652a9bebd0da82,12ff313a5164789c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:37:28.500 INFO 21062 --- [ocp-schedules-10,5b05745bfe8dddd1,90f532d0548174f7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:37:28.500 INFO 21062 --- [ocp-schedules-25,839f4ca956cb8c66,b00f513cc530b983] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:37:28.500 INFO 21062 --- [ocp-schedules-16,34b52fd7684aa40d,67ff2e78add76f38] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:37:28.502 INFO 21062 --- [ocp-schedules-11,ceb972c4b7449960,f4230530fba335a0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:37:28.503 INFO 21062 --- [ocp-schedules-27,fe652a9bebd0da82,12ff313a5164789c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:37:28.504 INFO 21062 --- [ocp-schedules-10,5b05745bfe8dddd1,90f532d0548174f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:28.504 INFO 21062 --- [ocp-schedules-16,34b52fd7684aa40d,67ff2e78add76f38] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:37:28.505 INFO 21062 --- [ocp-schedules-8,4bae2e76edfc11d6,dd4db5d32b590bed] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:37:28.505 INFO 21062 --- [ocp-schedules-29,c670a915f8546a3a,195684e08167d14f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:37:28.506 INFO 21062 --- [ocp-schedules-25,839f4ca956cb8c66,b00f513cc530b983] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:37:28.508 INFO 21062 --- [ocp-schedules-29,c670a915f8546a3a,195684e08167d14f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:28.511 INFO 21062 --- [ocp-schedules-7,c27c5a1e7c7876a5,08ceb5dfba2210d6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:37:28.511 INFO 21062 --- [ocp-schedules-14,8e58105a8b2f5efe,531486d8879b2a7f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:37:28.511 INFO 21062 --- [ocp-schedules-7,c27c5a1e7c7876a5,08ceb5dfba2210d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:37:28.511 INFO 21062 --- [ocp-schedules-14,8e58105a8b2f5efe,531486d8879b2a7f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:37:28.511 INFO 21062 --- [ocp-schedules-22,81442f4b98269b41,7b5cd5d0abd90205] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:37:28.511 INFO 21062 --- [ocp-schedules-22,81442f4b98269b41,7b5cd5d0abd90205] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:37:28.538 INFO 21062 --- [ocp-schedules-24,46f25a6e74d4a9ce,64812d7fb928eebe] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:37:28.538 INFO 21062 --- [ocp-schedules-24,46f25a6e74d4a9ce,64812d7fb928eebe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:37:28.579 INFO 21062 --- [pool-12-thread-8,0dc82926f8f237fa,f1f23e101f4c06b5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:37:28.579 INFO 21062 --- [pool-12-thread-6,0dc82926f8f237fa,927f72a83d6ff040] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:37:28.579 INFO 21062 --- [pool-12-thread-7,0dc82926f8f237fa,7dc5d9c71e4d512b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:37:28.580 INFO 21062 --- [pool-12-thread-5,0dc82926f8f237fa,84afe0f7c8798f80] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:37:28.584 INFO 21062 --- [pool-12-thread-6,0dc82926f8f237fa,927f72a83d6ff040] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:37:28.584 INFO 21062 --- [pool-12-thread-6,0dc82926f8f237fa,927f72a83d6ff040] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:37:28.585 INFO 21062 --- [ForkJoinPool-11-worker-6,0dc82926f8f237fa,d407c3d1dde4e0a5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:37:28.585 INFO 21062 --- [ForkJoinPool-11-worker-6,0dc82926f8f237fa,d407c3d1dde4e0a5] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:37:28.585 INFO 21062 --- [ForkJoinPool-11-worker-6,0dc82926f8f237fa,d407c3d1dde4e0a5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:37:28.585 INFO 21062 --- [pool-12-thread-8,0dc82926f8f237fa,f1f23e101f4c06b5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:37:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:37:28.589643+08:00, lastProcessTime=2024-11-04T14:37:18.590769+08:00 2024-11-04 14:37:28.607 INFO 21062 --- [pool-12-thread-5,0dc82926f8f237fa,84afe0f7c8798f80] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:37:28.608 INFO 21062 --- [pool-12-thread-5,0dc82926f8f237fa,84afe0f7c8798f80] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:37:28.609 INFO 21062 --- [pool-12-thread-5,0dc82926f8f237fa,84afe0f7c8798f80] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:37:28.612 INFO 21062 --- [ForkJoinPool-11-worker-6,0dc82926f8f237fa,a13043b46b9edb46] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:37:28.612 INFO 21062 --- [pool-12-thread-7,0dc82926f8f237fa,7dc5d9c71e4d512b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:37:28.674 INFO 21062 --- [ocp-async-1,7047fcf5f61c282e,024267a6390cd22f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:37:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:37:28.589643+08:00 2024-11-04 14:37:28.867 INFO 21062 --- [ocp-schedules-11,48100c6a9d2ade8f,af23e9461bb81fb5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:37:29.047 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:37:29.061 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:29.062 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:29.064 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:37:29.066 WARN 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:37:29.066 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:37:29.067 WARN 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:37:29.074 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:37:29.075 WARN 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:37:29.076 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:37:29.077 WARN 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:37:29.084 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:37:29.085 WARN 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:37:29.086 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:37:29.087 WARN 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:37:29.093 INFO 21062 --- [ocp-schedules-7,b4e50db6ba57d91c,9cd8f64fde307ff2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:37:29.162 INFO 21062 --- [ocp-schedules-27,9451d4a9dcb57d52,238ff2b1fbfd28ea] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:37:29.162 INFO 21062 --- [ocp-schedules-27,9451d4a9dcb57d52,238ff2b1fbfd28ea] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:37:29.243 INFO 21062 --- [ocp-schedules-21,56243c27e81f3eb9,56e8a261028d6290] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:37:29.274 WARN 21062 --- [ocp-async-4,ad72f6a612e18fdd,8f5756b89a5f365d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:29.276 INFO 21062 --- [ocp-schedules-6,943ef1b60f8c7497,aa5b8a2c2be84d2a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:37:29.281 INFO 21062 --- [ocp-schedules-6,943ef1b60f8c7497,aa5b8a2c2be84d2a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:37:29.281 INFO 21062 --- [ocp-schedules-6,943ef1b60f8c7497,aa5b8a2c2be84d2a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:37:29.307 INFO 21062 --- [ocp-async-7,4487ca0a6cf3b700,7e3f7181a7b58b92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:29.457 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:37:29.457 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:37:29.465 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:37:29.465 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:37:29.465 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:37:29.476 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:37:29.476 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:37:29.476 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:37:29.480 INFO 21062 --- [ocp-schedules-4,c92ef7f8de632f69,fe8410de86d60338] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:37:29.633 INFO 21062 --- [ocp-schedules-17,83f43e0e821883cc,99cdfdc2e28c8304] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:37:29.633 INFO 21062 --- [ocp-schedules-17,83f43e0e821883cc,99cdfdc2e28c8304] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:37:29.640 INFO 21062 --- [ocp-schedules-17,83f43e0e821883cc,99cdfdc2e28c8304] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:37:29.640 INFO 21062 --- [ocp-schedules-17,83f43e0e821883cc,99cdfdc2e28c8304] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:37:29.644 INFO 21062 --- [ocp-schedules-17,83f43e0e821883cc,99cdfdc2e28c8304] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:37:29.644 INFO 21062 --- [ocp-schedules-17,83f43e0e821883cc,99cdfdc2e28c8304] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:37:29.916 INFO 21062 --- [ocp-schedules-18,b3c492c29c0807e1,55961638d20f3d4f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:37:29.916 INFO 21062 --- [ocp-schedules-18,b3c492c29c0807e1,55961638d20f3d4f] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:37:29.947 INFO 21062 --- [pool-44-thread-1,b3c492c29c0807e1,8b793becdb8f468a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:37:29.975 WARN 21062 --- [ocp-schedules-18,b3c492c29c0807e1,55961638d20f3d4f] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:37:29.975 INFO 21062 --- [ocp-schedules-18,b3c492c29c0807e1,55961638d20f3d4f] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:37:29.983 INFO 21062 --- [prometheus-minute-schedule-1,a1910a271ba0b07f,aa8337b6a4af182c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:37:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:37:30.756 INFO 21062 --- [ocp-schedules-4,0674c71e892c868d,2b4c7a5e70946a9b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:37:31.274 WARN 21062 --- [ocp-async-1,9edd0e954b20b5a1,a97619c3d852c0f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:31.307 INFO 21062 --- [ocp-async-10,0f5b162d8ff5deaf,517a3f0b36c4ba64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:31.848 INFO 21062 --- [ocp-schedules-2,080680e7bef8df7d,b0a8bcbd03148e93] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:37:31.854 INFO 21062 --- [ocp-schedules-2,080680e7bef8df7d,b0a8bcbd03148e93] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:37:31.854 INFO 21062 --- [ocp-schedules-2,080680e7bef8df7d,b0a8bcbd03148e93] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:37:31.940 INFO 21062 --- [ocp-schedules-25,5462f125b868c02c,bc8fe1bde1a5eeb8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:37:31.940 INFO 21062 --- [ocp-schedules-25,5462f125b868c02c,bc8fe1bde1a5eeb8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:37:31.967 INFO 21062 --- [ocp-schedules-25,5462f125b868c02c,bc8fe1bde1a5eeb8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:37:31.967 INFO 21062 --- [ocp-schedules-25,5462f125b868c02c,bc8fe1bde1a5eeb8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:37:31.968 INFO 21062 --- [ocp-schedules-3,ea47a13f477afa3b,961563f7aece22c3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:37:32.062 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:32.063 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:32.157 INFO 21062 --- [ocp-schedules-8,402522048ce4d86b,ebf535c3f46c356c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:37:32.157+08:00, clusters=[1] 2024-11-04 14:37:32.158 INFO 21062 --- [ocp-schedules-8,402522048ce4d86b,ebf535c3f46c356c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:37:32.348 INFO 21062 --- [ocp-schedules-28,72c16308c667e985,99c181fffca019d6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:37:32.348+08:00, clusters=[1] 2024-11-04 14:37:32.352 INFO 21062 --- [ocp-schedules-28,72c16308c667e985,99c181fffca019d6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:37:32.983 INFO 21062 --- [prometheus-minute-schedule-1,dfd4c6c80a31446e,df7aebb159bd63e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:33.275 WARN 21062 --- [ocp-async-9,5a3f78153767c4d8,085a5e458884db91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:33.307 INFO 21062 --- [ocp-async-5,882e2a13cf74526d,d669120c0163afb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:33.532 INFO 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:37:33.532 INFO 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:37:33.532 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:37:33.532 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:37:33.542 INFO 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:37:33.542 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:37:33.542 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:37:33.543 WARN 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:37:33.543 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:37:33.544 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:37:33.544 INFO 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:37:33.544 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:37:33.545 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:37:33.545 WARN 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:37:33.546 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:37:33.546 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:37:33.552 INFO 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:37:33.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:37:33.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:37:33.563 ERROR 21062 --- [ocp-schedules-27,9a72394d0b921d0b,06f11eb3d8305ab4] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:37:35.063 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:35.064 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:35.275 WARN 21062 --- [ocp-async-8,3eefa607d5e41a67,151b9e43687d4e29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:35.308 INFO 21062 --- [ocp-async-2,497de4b50b63fb7b,89430bb84f423aae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:35.983 INFO 21062 --- [prometheus-minute-schedule-1,05f972925a0cbaec,1e989ee1e5cc03e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:36.150 INFO 21062 --- [ocp-schedules-5,d68b6b6ac3a90b85,4c3204ae6252d224] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:37:37.085 INFO 21062 --- [ocp-schedules-12,5c0d410dc348ecb9,8819a07dd47b9544] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:37:37.085 INFO 21062 --- [ocp-schedules-12,a37c07669220467c,918e2ef4726ad6a5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:37:37.089 INFO 21062 --- [ocp-schedules-12,a37c07669220467c,918e2ef4726ad6a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:37.117 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:37:37.117 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:37:37.275 WARN 21062 --- [ocp-async-7,1b8dc37011d55f7f,969f38367323aafe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:37.308 INFO 21062 --- [ocp-async-3,c4fd0377b9614307,272c158e54f26323] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:38.064 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:38.065 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:38.504 INFO 21062 --- [ocp-schedules-16,34b52fd7684aa40d,67ff2e78add76f38] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:37:38.505 INFO 21062 --- [ocp-schedules-10,5b05745bfe8dddd1,90f532d0548174f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:37:38.509 INFO 21062 --- [ocp-schedules-29,c670a915f8546a3a,195684e08167d14f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:37:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:37:38.589953+08:00, lastProcessTime=2024-11-04T14:37:28.589643+08:00 2024-11-04 14:37:38.674 INFO 21062 --- [ocp-async-4,48ea0744e35e7834,57281e4b6a15e774] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:37:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:37:38.589953+08:00 2024-11-04 14:37:38.983 INFO 21062 --- [prometheus-minute-schedule-1,bc0da1ac1f465751,e895f416907ba720] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:39.275 WARN 21062 --- [ocp-async-6,a1e5414b8ef2e498,815bd6ea6321bcdf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:39.308 INFO 21062 --- [ocp-async-8,bef4c81e1d08f4e2,d7af686a9be7f431] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:37:40.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:37:40.064 INFO 21062 --- [ocp-schedules-29,ddfa613e682dff6e,38af40812d42bc53] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:37:40.068 INFO 21062 --- [ocp-schedules-29,ddfa613e682dff6e,38af40812d42bc53] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:40.154 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:37:40.156 ERROR 21062 --- [ocp-schedules-29,ddfa613e682dff6e,38af40812d42bc53] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:37:40.156 INFO 21062 --- [ocp-schedules-29,ddfa613e682dff6e,38af40812d42bc53] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:37:40.759 INFO 21062 --- [ocp-schedules-4,c3c81ab9566c4a3a,685e4140f5a8b5ef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:37:41.065 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:41.066 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:41.275 WARN 21062 --- [ocp-async-4,906d2e178071e819,ff746f012fabcc21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:41.308 INFO 21062 --- [ocp-async-7,21c3a642cddc6de3,857c1d6eb4c2c16e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:41.983 INFO 21062 --- [prometheus-minute-schedule-1,46bde2fffd82e639,a6233094e2c0e29a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:43.275 WARN 21062 --- [ocp-async-7,4a03b8ce4c7b9100,6dbff6b1f8576718] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:43.308 INFO 21062 --- [ocp-async-10,5e15f15f5c92a59a,3acb4a49212157b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:44.066 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:44.067 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5b932b64b721b6b3,602191794e4b110b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:45.276 WARN 21062 --- [ocp-async-10,880fdb99889be4c7,0fdf951ff2d682e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:45.308 INFO 21062 --- [ocp-async-5,d49fb18695f4d5c7,82e979b704e4b4d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:46.191 INFO 21062 --- [ocp-schedules-4,94528efaebc014f8,f703cdfd02f31cb6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:37:47.067 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:47.068 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:47.276 WARN 21062 --- [ocp-async-5,56731de423b2e967,bc743052596b3189] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:47.309 INFO 21062 --- [ocp-async-2,f0407c97d64138bf,7a5f7d9fb5bd6a7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2f80651dcc6b0b72,d9d19e59b89ad23e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:37:48.591396+08:00, lastProcessTime=2024-11-04T14:37:38.589953+08:00 2024-11-04 14:37:48.674 INFO 21062 --- [ocp-async-10,c48f8a7ef99b26b4,a9098edc61d533d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:37:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:37:48.591396+08:00 2024-11-04 14:37:49.276 WARN 21062 --- [ocp-async-7,e98080189a29b687,be92fc09c5d08b3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:49.309 INFO 21062 --- [ocp-async-6,c5c204f3d0252903,be74f099c078fb39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:50.068 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:50.070 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:50.761 INFO 21062 --- [ocp-schedules-6,41fb34488715fe09,5905028d231efc63] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:37:50.983 INFO 21062 --- [prometheus-minute-schedule-1,650497aeb5a689b9,2ee7677e6496f439] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:51.276 WARN 21062 --- [ocp-async-6,941fa48e76d19186,12c2002c4a25f73e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:51.309 INFO 21062 --- [ocp-async-5,48ea685e01ebbb0f,f2a4dbc76ccce5fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:51.857 INFO 21062 --- [ocp-schedules-17,73d53518758730f5,b14c98556823c637] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:37:51.862 INFO 21062 --- [ocp-schedules-17,73d53518758730f5,b14c98556823c637] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:37:51.862 INFO 21062 --- [ocp-schedules-17,73d53518758730f5,b14c98556823c637] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:37:53.070 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:53.071 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:53.276 WARN 21062 --- [ocp-async-5,110696b1892445b6,22a199aa2813d96c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:53.309 INFO 21062 --- [ocp-async-2,1d51bfb14fda23cd,ef33f5bb9bd032d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9d469ab6d2185ae6,527bd6d93a848514] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:55.277 WARN 21062 --- [ocp-async-2,6ffd1d48f7ed9382,785e7735beddfd1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:55.309 INFO 21062 --- [ocp-async-1,bf392a065e2404a3,b094c3cff9098d9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:56.071 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:56.072 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:56.217 INFO 21062 --- [ocp-schedules-24,686af4078193aacd,6183d0231dcbd537] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:37:56.983 INFO 21062 --- [prometheus-minute-schedule-1,a174e31180cff877,fd1a1e935c3e090f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:37:57.277 WARN 21062 --- [ocp-async-1,dc833b8b4d6d7230,a95eeeb2c1b34db5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:57.310 INFO 21062 --- [ocp-async-9,906e553f7826e3e7,e6e504401ad65ae1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:58.500 INFO 21062 --- [ocp-schedules-4,40d2db031bdac2a3,d88205c1a379bbe8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:37:58.500 INFO 21062 --- [ocp-schedules-27,019b28a774a7f52d,5345a96e2b1d75d3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:37:58.503 INFO 21062 --- [ocp-schedules-29,4939f5b4797eccdb,12952fe14db78903] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:37:58.504 INFO 21062 --- [ocp-schedules-4,40d2db031bdac2a3,d88205c1a379bbe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:58.504 INFO 21062 --- [ocp-schedules-27,019b28a774a7f52d,5345a96e2b1d75d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:37:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:37:58.505 INFO 21062 --- [ocp-schedules-14,727ef62682e0bb04,b637c5a95cfc0c07] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:37:58.510 INFO 21062 --- [ocp-schedules-13,f371b2dc4c09fcfb,f9f8bf0cf3f774d5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:37:58.510 INFO 21062 --- [ocp-schedules-11,44005bdb5a7b9fac,9acf720b9bb70cc3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:37:58.510 INFO 21062 --- [ocp-schedules-28,3d6276c76581412d,178a8a6c3ff0f49a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:37:58.511 INFO 21062 --- [ocp-schedules-11,44005bdb5a7b9fac,9acf720b9bb70cc3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:37:58.511 INFO 21062 --- [ocp-schedules-13,f371b2dc4c09fcfb,f9f8bf0cf3f774d5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:37:58.511 INFO 21062 --- [ocp-schedules-28,3d6276c76581412d,178a8a6c3ff0f49a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:37:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:37:58.589512+08:00, lastProcessTime=2024-11-04T14:37:48.591396+08:00 2024-11-04 14:37:58.674 INFO 21062 --- [ocp-async-2,c9f92b39bae32d90,19b7fd2dabc670b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:37:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:37:58.589512+08:00 2024-11-04 14:37:59.072 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:37:59.073 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:37:59.245 INFO 21062 --- [ocp-schedules-18,1f32070f7dd07581,5c54e9e31420614a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:37:59.277 WARN 21062 --- [ocp-async-5,7dd2a3e618390fff,eb7035bf255bebe7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:37:59.310 INFO 21062 --- [ocp-async-2,0ab057a13c7bd47b,0363f4d4e0051287] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:37:59.983 INFO 21062 --- [prometheus-minute-schedule-1,8f9c9b598fe57f04,d8d72cce09e2fc16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702280, interval=60 2024-11-04 14:38:00.765 INFO 21062 --- [ocp-schedules-2,1f465d1a9dbc2356,94ea15c51e64af72] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:38:01.277 WARN 21062 --- [ocp-async-2,176ed4f0a0954a2f,a27578b9b2666c44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:01.310 INFO 21062 --- [ocp-async-1,2c50b8a01119b8da,faf12f22c5f02d79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:02.073 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:02.074 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ee26a947ee5d459b,edbc84bca4e38ffa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:03.277 WARN 21062 --- [ocp-async-1,8a8363976896e229,be8d8c673eb11f68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:03.310 INFO 21062 --- [ocp-async-10,0afd522fcf22426a,801439c7a2fc19d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:05.074 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:05.076 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:05.277 WARN 21062 --- [ocp-async-10,af66d137630a8c22,c29fc062396c47db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:05.310 INFO 21062 --- [ocp-async-8,7755fa7bf6e7194d,27b548e7ebfe4f99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:05.983 INFO 21062 --- [prometheus-minute-schedule-1,bdcb1f1a6ee37b79,876ef9c734b004c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:06.259 INFO 21062 --- [ocp-schedules-10,a998f1c1891bcadb,6e644f85e9377a00] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:38:07.117 INFO 21062 --- [ocp-schedules-12,a37c07669220467c,918e2ef4726ad6a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:38:07.118 INFO 21062 --- [ocp-schedules-12,502f7fc7e04e1289,e91c4c79db02238b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:38:07.122 INFO 21062 --- [ocp-schedules-12,502f7fc7e04e1289,e91c4c79db02238b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:07.154 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:38:07.154 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:38:07.278 WARN 21062 --- [ocp-async-8,60f07781bc5db61e,05a59d14ea938231] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:07.310 INFO 21062 --- [ocp-async-7,e3bb4b999590504d,7af709392d1001ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:08.076 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:08.077 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:08.504 INFO 21062 --- [ocp-schedules-27,019b28a774a7f52d,5345a96e2b1d75d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:38:08.505 INFO 21062 --- [ocp-schedules-4,40d2db031bdac2a3,d88205c1a379bbe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:38:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:38:08.590339+08:00, lastProcessTime=2024-11-04T14:37:58.589512+08:00 2024-11-04 14:38:08.675 INFO 21062 --- [ocp-async-10,e74f7309e97dd5bf,15e2a2ca4e424692] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:38:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:38:08.590339+08:00 2024-11-04 14:38:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d969b53cbedc0dac,e6a21a3318dfdf88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:09.278 WARN 21062 --- [ocp-async-1,12fdde8743b41bb5,a5938a9d5e09977e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:09.311 INFO 21062 --- [ocp-async-9,abf21ea5a232904f,42545c4f3b582978] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:38:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:38:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:38:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:38:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:38:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:38:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:38:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:38:10.768 INFO 21062 --- [ocp-schedules-7,cd0f72111c1258eb,def6d86b0019c930] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:38:11.077 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:11.078 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:11.278 WARN 21062 --- [ocp-async-10,c8598fc2f2309f48,3a8766ffc6547eb8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:11.311 INFO 21062 --- [ocp-async-4,05def2ac8e5e8822,9eead44f690696e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:11.619 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=2, affectRows=2 2024-11-04 14:38:11.619 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=2, validCount=2 2024-11-04 14:38:11.620 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=11, affectRows=11 2024-11-04 14:38:11.620 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=20, affectRows=20 2024-11-04 14:38:11.620 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=11, validCount=11 2024-11-04 14:38:11.620 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=20, validCount=20 2024-11-04 14:38:11.620 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=45, affectRows=45 2024-11-04 14:38:11.620 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=49, affectRows=49 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=50, affectRows=50 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=34, affectRows=34 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=60, affectRows=60 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=49, validCount=49 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=45, validCount=45 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=49, validCount=49 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=34, validCount=34 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=60, validCount=60 2024-11-04 14:38:11.621 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=59, affectRows=59 2024-11-04 14:38:11.622 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=57, validCount=57 2024-11-04 14:38:11.622 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702220, nodeCount=66, affectRows=66 2024-11-04 14:38:11.622 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702220, nodeCount=66, validCount=66 2024-11-04 14:38:11.865 INFO 21062 --- [ocp-schedules-25,8d85a729752018cd,650ad7f29979af3f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:38:11.870 INFO 21062 --- [ocp-schedules-25,8d85a729752018cd,650ad7f29979af3f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:38:11.870 INFO 21062 --- [ocp-schedules-25,8d85a729752018cd,650ad7f29979af3f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:38:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6a503964cff610c6,bf15adfa5370199f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:13.278 WARN 21062 --- [ocp-async-8,1097a0ed2cbc3cf3,b36ed63ac373b915] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:13.311 INFO 21062 --- [ocp-async-3,30403d76307916ef,eb8a0ae85a4e8854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:13.984 INFO 21062 --- [prometheus-cache-monitor1,a7b5f6171e05d0bc,15de73b34a56ec2e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:38:13.984 INFO 21062 --- [prometheus-cache-monitor1,a7b5f6171e05d0bc,15de73b34a56ec2e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:38:14.078 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:14.079 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:14.308 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:32:14.289, end=2024-11-04T14:37:14.305} 2024-11-04 14:38:14.308 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:32:14.289, end=2024-11-04T14:37:14.305} 2024-11-04 14:38:14.312 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:38:14.312 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:38:14.312 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:38:14.312 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:38:14.983 INFO 21062 --- [prometheus-minute-schedule-1,536b0a5358a03493,80bb5e4379daf84f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:15.278 WARN 21062 --- [ocp-async-7,5281ad9fe76673c7,1eeb1a136fc6fc5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:15.311 INFO 21062 --- [ocp-async-6,bda262df2e35ffc7,da6acb2a3482abc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:16.283 INFO 21062 --- [ocp-schedules-23,13dfdbf55ff4ad00,278fb68b9339cbb8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:38:17.079 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:17.080 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:17.278 WARN 21062 --- [ocp-async-9,93b9d21c42c3a217,bb9038a25504cbae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:17.311 INFO 21062 --- [ocp-async-5,38fb42718fe328d2,eac601fd6366d6ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:17.983 INFO 21062 --- [prometheus-minute-schedule-1,379b33592b698950,663e7c179218c881] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:38:18.589824+08:00, lastProcessTime=2024-11-04T14:38:08.590339+08:00 2024-11-04 14:38:18.675 INFO 21062 --- [ocp-async-7,ffe818521fdea033,901df5867d388212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:38:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:38:18.589824+08:00 2024-11-04 14:38:19.084 INFO 21062 --- [ocp-schedules-6,84aa55ed93f2dbba,64eeb9690c6c208e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:38:19.089 INFO 21062 --- [ocp-schedules-6,84aa55ed93f2dbba,64eeb9690c6c208e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:19.089 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:38:19.279 WARN 21062 --- [ocp-async-8,2cc33ad34b65c29e,31c4f791af0fa2ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:19.311 INFO 21062 --- [ocp-async-3,2676da54d12b7541,a26956b5706c3794] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:20.080 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:20.081 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:20.772 INFO 21062 --- [ocp-schedules-24,4aa4f1aafee6bb9d,338952a7eccfb0e4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:38:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a089831cdd817186,ef2dcb0a82e94e55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:21.279 WARN 21062 --- [ocp-async-7,7de8959f0f1d4223,4d6755e1447c88b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:21.311 INFO 21062 --- [ocp-async-6,640f7389a4d2bee8,4573296fce99f92f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:23.081 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:23.082 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:23.279 WARN 21062 --- [ocp-async-9,be8683bb4f841cda,3974a0404c604726] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:23.312 INFO 21062 --- [ocp-async-5,832bc60f3d6902a8,a32bd89a2c0d0661] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:23.983 INFO 21062 --- [prometheus-minute-schedule-1,f1e2121f70818830,924156454d957b7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:25.279 WARN 21062 --- [ocp-async-4,89f4452f74c4d0e2,fcc2af2b0547a3a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:25.312 INFO 21062 --- [ocp-async-2,9b9a5019183e3f97,04936d2c5784388d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:26.082 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:26.083 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:26.326 INFO 21062 --- [ocp-schedules-13,510489318f57b157,9d2171d8b8dc6ce0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:38:26.705 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,584fc7f915234ebe,434ebbd135921905] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=584fc7f915234ebe, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 14:38:26.706 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,584fc7f915234ebe,434ebbd135921905] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=584fc7f915234ebe, duration=15 ms] 2024-11-04 14:38:26.983 INFO 21062 --- [prometheus-minute-schedule-1,750fd098bc65bae6,bc8f2aad5ae2e6f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ab40558f92f97334,d74b05becceb8297] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:38:27.279 WARN 21062 --- [ocp-async-3,c0120d3928e23154,3af020a9e436e785] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:27.312 INFO 21062 --- [ocp-async-1,8d162aa69c250e13,543194795eeb6e6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:28.498 INFO 21062 --- [ocp-schedules-3,200b8d161bf52093,1bc489f749d1fa55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:38:28.499 INFO 21062 --- [ocp-schedules-29,663a729c93b6ff35,8e8e3c61c5378afe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:38:28.500 INFO 21062 --- [ocp-schedules-19,47a79d571cfab4f8,472af8c1f52697af] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:38:28.500 INFO 21062 --- [ocp-schedules-28,0ff4fb3d51af780c,b4253b868a8bfd65] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:38:28.502 INFO 21062 --- [ocp-schedules-16,ea16d2aaf59a76ed,2554fdc259814123] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:38:28.503 INFO 21062 --- [ocp-schedules-3,200b8d161bf52093,1bc489f749d1fa55] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:38:28.504 INFO 21062 --- [ocp-schedules-19,47a79d571cfab4f8,472af8c1f52697af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:28.504 INFO 21062 --- [ocp-schedules-28,0ff4fb3d51af780c,b4253b868a8bfd65] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:38:28.504 INFO 21062 --- [ocp-schedules-30,3102ea307abe77a4,f946c9febac26839] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:38:28.505 INFO 21062 --- [ocp-schedules-29,663a729c93b6ff35,8e8e3c61c5378afe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:38:28.510 INFO 21062 --- [ocp-schedules-23,4bcdee789afa0500,562dbb07830edb0a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:38:28.510 INFO 21062 --- [ocp-schedules-23,4bcdee789afa0500,562dbb07830edb0a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:38:28.510 INFO 21062 --- [ocp-schedules-25,af743ef359975817,f5e69cc0b223df78] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:38:28.510 INFO 21062 --- [ocp-schedules-25,af743ef359975817,f5e69cc0b223df78] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:38:28.511 INFO 21062 --- [ocp-schedules-2,8ed7e65143f58779,4afb702d152ce93b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:38:28.511 INFO 21062 --- [ocp-schedules-2,8ed7e65143f58779,4afb702d152ce93b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:38:28.579 INFO 21062 --- [ocp-schedules-17,dfb63bc451421357,1938f2dd10d05cbb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1584,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":395,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":393} 2024-11-04 14:38:28.579 INFO 21062 --- [ocp-schedules-17,dfb63bc451421357,1938f2dd10d05cbb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=53, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=53, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:38:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:38:28.590300+08:00, lastProcessTime=2024-11-04T14:38:18.589824+08:00 2024-11-04 14:38:28.675 INFO 21062 --- [ocp-async-2,4d5e98d533b27f6b,eab653b71ba04e54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:38:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:38:28.590300+08:00 2024-11-04 14:38:28.840 INFO 21062 --- [ocp-schedules-21,aaebaf891671124d,a4d9470e986761a0] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:38:28.840 INFO 21062 --- [ocp-schedules-21,aaebaf891671124d,a4d9470e986761a0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:38:28.870 INFO 21062 --- [ocp-schedules-4,c221fcd8f0a5cd9f,9d96b57737146511] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:38:29.084 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:29.084 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:29.090 INFO 21062 --- [ocp-schedules-6,84aa55ed93f2dbba,64eeb9690c6c208e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:38:29.166 INFO 21062 --- [ocp-schedules-29,851657e264b7f460,81c93b8ff4c6ed7f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:38:29.166 INFO 21062 --- [ocp-schedules-29,851657e264b7f460,81c93b8ff4c6ed7f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:38:29.248 INFO 21062 --- [ocp-schedules-3,c3705ea003f6abc8,0bc1414e9776932e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:38:29.279 WARN 21062 --- [ocp-async-10,f706c54e27dbaa2f,99ae3df9657ae29a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:29.281 INFO 21062 --- [ocp-schedules-9,9f6296b72dc25ea1,9ef3faeb7e886057] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:38:29.287 INFO 21062 --- [ocp-schedules-9,9f6296b72dc25ea1,9ef3faeb7e886057] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:38:29.287 INFO 21062 --- [ocp-schedules-9,9f6296b72dc25ea1,9ef3faeb7e886057] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:38:29.312 INFO 21062 --- [ocp-async-8,8b32e479a6c6e26c,1fc135159c2ece0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:29.649 INFO 21062 --- [ocp-schedules-8,1c2e3c365fa09d07,797e287bffb0a8eb] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:38:29.649 INFO 21062 --- [ocp-schedules-8,1c2e3c365fa09d07,797e287bffb0a8eb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:38:29.650 INFO 21062 --- [ocp-schedules-8,1c2e3c365fa09d07,797e287bffb0a8eb] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:38:29.650 INFO 21062 --- [ocp-schedules-8,1c2e3c365fa09d07,797e287bffb0a8eb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:38:29.651 INFO 21062 --- [ocp-schedules-8,1c2e3c365fa09d07,797e287bffb0a8eb] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:38:29.651 INFO 21062 --- [ocp-schedules-8,1c2e3c365fa09d07,797e287bffb0a8eb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:38:29.983 INFO 21062 --- [prometheus-minute-schedule-1,78b08a1f64a2e85f,e00396f6e1e837c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:38:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:38:30.774 INFO 21062 --- [ocp-schedules-20,c5bee4bec351c74b,7ed1b6a6d7a6145d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:38:31.279 WARN 21062 --- [ocp-async-2,82cda1047e3bf0f1,21cdaf9e7191fbea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:31.312 INFO 21062 --- [ocp-async-7,64222d44a0e8bfa1,fd796f642ece03ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:31.872 INFO 21062 --- [ocp-schedules-7,264b5fc9bbb492ac,6241fe69be1bed3e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:38:31.877 INFO 21062 --- [ocp-schedules-7,264b5fc9bbb492ac,6241fe69be1bed3e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:38:31.877 INFO 21062 --- [ocp-schedules-7,264b5fc9bbb492ac,6241fe69be1bed3e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:38:31.968 INFO 21062 --- [ocp-schedules-2,e4bc30b85510b1d8,bd62bc8b7d97ddb8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:38:31.968 INFO 21062 --- [ocp-schedules-2,e4bc30b85510b1d8,bd62bc8b7d97ddb8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:38:31.996 INFO 21062 --- [ocp-schedules-16,3b856b72d525c5d0,94b490e4a724e2a1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:38:31.996 INFO 21062 --- [ocp-schedules-2,e4bc30b85510b1d8,bd62bc8b7d97ddb8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:38:31.996 INFO 21062 --- [ocp-schedules-2,e4bc30b85510b1d8,bd62bc8b7d97ddb8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:38:32.084 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:32.085 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:32.096 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7bcff5d01edb0085,70c3b56f26fde758] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/creators?null, client=10.130.47.17, traceId=7bcff5d01edb0085, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.listCreatorInfo(TaskType), args=MANUAL,] 2024-11-04 14:38:32.101 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7bcff5d01edb0085,70c3b56f26fde758] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/creators client=10.130.47.17, traceId=7bcff5d01edb0085, duration=13 ms] 2024-11-04 14:38:32.107 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,024b13c9fab7a892,bcc0d4b7d6a9ea8b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/manual-instances?page=1&size=10, client=10.130.47.17, traceId=024b13c9fab7a892, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listManualTaskInstances(String,Pageable,List,String), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:38:32.128 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,024b13c9fab7a892,bcc0d4b7d6a9ea8b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/manual-instances client=10.130.47.17, traceId=024b13c9fab7a892, duration=28 ms] 2024-11-04 14:38:32.184 INFO 21062 --- [ocp-schedules-15,5ca6a2ce9370608c,abc5f8da8329e256] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:38:32.184+08:00, clusters=[1] 2024-11-04 14:38:32.185 INFO 21062 --- [ocp-schedules-15,5ca6a2ce9370608c,abc5f8da8329e256] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:38:32.378 INFO 21062 --- [ocp-schedules-25,cf7b292ddd1aed40,7ce91a7ea4842e1d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:38:32.378+08:00, clusters=[1] 2024-11-04 14:38:32.380 INFO 21062 --- [ocp-schedules-14,ab8390cd3c4a06ae,a28a0e1710389d0d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:38:32.381 INFO 21062 --- [ocp-schedules-25,cf7b292ddd1aed40,7ce91a7ea4842e1d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:38:32.384 INFO 21062 --- [ocp-schedules-14,ab8390cd3c4a06ae,a28a0e1710389d0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:32.444 ERROR 21062 --- [ocp-schedules-14,ab8390cd3c4a06ae,a28a0e1710389d0d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:38:32.445 INFO 21062 --- [ocp-schedules-14,ab8390cd3c4a06ae,a28a0e1710389d0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:38:32.983 INFO 21062 --- [prometheus-minute-schedule-1,783e258d74548938,0d40d28d4629b580] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:33.280 WARN 21062 --- [ocp-async-1,5705f9aa514f74da,092d3162311772aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:33.313 INFO 21062 --- [ocp-async-9,39c8e64e7f72e5ae,08f4405f801607ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:33.572 INFO 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:38:33.572 INFO 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:38:33.572 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:38:33.572 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:38:33.582 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:38:33.583 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:38:33.583 INFO 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:38:33.584 WARN 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:38:33.584 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:38:33.584 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:38:33.585 INFO 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:38:33.585 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:38:33.585 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:38:33.586 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:38:33.586 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:38:33.586 WARN 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:38:33.593 INFO 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:38:33.593 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:38:33.593 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:38:33.604 ERROR 21062 --- [ocp-schedules-1,4ffe133050c7cda0,ce968679e0ed8a18] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:38:34.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:38:34.621 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/cluster/1, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:38:34.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:38:35.085 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:35.086 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:35.280 WARN 21062 --- [ocp-async-5,c4bc47f1938f324c,772456acf9bcc44c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:35.313 INFO 21062 --- [ocp-async-4,3b9b2b5c836193e0,04e37582338919a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:35.983 INFO 21062 --- [prometheus-minute-schedule-1,2a60fe2ff7fc9352,0897537ba2c2f1c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:36.351 INFO 21062 --- [ocp-schedules-14,2f56c97b3f4b554f,8ca429bc80a5fedd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:38:36.385 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,05715fbae2724f16,776c9811aa2e4596] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.130.47.17, traceId=05715fbae2724f16, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2024-11-04 14:38:36.387 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,05715fbae2724f16,776c9811aa2e4596] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.130.47.17, traceId=05715fbae2724f16, duration=7 ms] 2024-11-04 14:38:37.055 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,e9b32272c74bd909,7df6c99d425115d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.130.47.17, traceId=e9b32272c74bd909, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 14:38:37.072 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,203ee6c261623df5,c4998ece2d524571] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=203ee6c261623df5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 14:38:37.076 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,e9b32272c74bd909,7df6c99d425115d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=e9b32272c74bd909, duration=26 ms] 2024-11-04 14:38:37.080 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,203ee6c261623df5,c4998ece2d524571] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=203ee6c261623df5, duration=12 ms] 2024-11-04 14:38:37.131 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,430f99ec9bfefcdf,d8eae12948df7af5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=430f99ec9bfefcdf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 14:38:37.131 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,430f99ec9bfefcdf,d8eae12948df7af5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=430f99ec9bfefcdf, duration=4 ms] 2024-11-04 14:38:37.155 INFO 21062 --- [ocp-schedules-12,502f7fc7e04e1289,e91c4c79db02238b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:38:37.155 INFO 21062 --- [ocp-schedules-12,cac3927578efbf90,8c5fe42890b87f48] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:38:37.158 INFO 21062 --- [ocp-schedules-12,cac3927578efbf90,8c5fe42890b87f48] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:37.185 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:38:37.185 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:38:37.280 WARN 21062 --- [ocp-async-8,cf755227d0f746d1,952cca08f38104cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:37.313 INFO 21062 --- [ocp-async-3,c10ab24374fe25b1,e49cf1dceadfed18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:37.319 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,0da529a9c40e1cd1,5f2e15883685c716] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=0da529a9c40e1cd1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:38:37.322 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f7fde17fda283f4e,96c89628acd4a48f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=f7fde17fda283f4e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 14:38:37.324 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,fd96050423d5ca97,74ac11652b622bfa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1&status=RUNNING%2CFAILED, client=10.130.47.17, traceId=fd96050423d5ca97, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1,] 2024-11-04 14:38:37.338 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,fd96050423d5ca97,74ac11652b622bfa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.130.47.17, traceId=fd96050423d5ca97, duration=22 ms] 2024-11-04 14:38:37.363 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f7fde17fda283f4e,96c89628acd4a48f] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:37.458 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f7fde17fda283f4e,96c89628acd4a48f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=f7fde17fda283f4e, duration=142 ms] 2024-11-04 14:38:37.536 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,0da529a9c40e1cd1,5f2e15883685c716] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=0da529a9c40e1cd1, duration=224 ms] 2024-11-04 14:38:37.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,d02e7d4cb08e1845,7f410b7ad4f61d76] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/eventMeta?id=1, client=10.130.47.17, traceId=d02e7d4cb08e1845, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1,] 2024-11-04 14:38:37.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,808695bae77e8d30,0c6d2866516ec6c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/stats?id=1, client=10.130.47.17, traceId=808695bae77e8d30, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1,] 2024-11-04 14:38:37.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f92b420b75d3e971,68f9789c7b5a2060] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=f92b420b75d3e971, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 14:38:37.620 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,d02e7d4cb08e1845,7f410b7ad4f61d76] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/eventMeta client=10.130.47.17, traceId=d02e7d4cb08e1845, duration=6 ms] 2024-11-04 14:38:37.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d59f77498ff586b2,3921ca1ab475312b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topSql/switch?id=1, client=10.130.47.17, traceId=d59f77498ff586b2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1,] 2024-11-04 14:38:37.623 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,fe3b89896160a53b,d463fdc12566edbf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/events?startTime=2024-10-28T14%3A38%3A37%2B08%3A00&endTime=2024-11-04T14%3A38%3A37%2B08%3A00&page=1&size=5, client=10.130.47.17, traceId=fe3b89896160a53b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1,2024-10-28T14:38:37+08:00,2024-11-04T14:38:37+08:00,Page request [number: 0, size 5, sort: timestamp: DESC],] 2024-11-04 14:38:37.629 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,fe3b89896160a53b,d463fdc12566edbf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/events client=10.130.47.17, traceId=fe3b89896160a53b, duration=11 ms] 2024-11-04 14:38:37.646 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,808695bae77e8d30,0c6d2866516ec6c9] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:37.649 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,808695bae77e8d30,0c6d2866516ec6c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/stats client=10.130.47.17, traceId=808695bae77e8d30, duration=34 ms] 2024-11-04 14:38:37.662 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f92b420b75d3e971,68f9789c7b5a2060] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:37.662 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d59f77498ff586b2,3921ca1ab475312b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topSql/switch client=10.130.47.17, traceId=d59f77498ff586b2, duration=46 ms] 2024-11-04 14:38:37.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f92b420b75d3e971,68f9789c7b5a2060] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=f92b420b75d3e971, duration=99 ms] 2024-11-04 14:38:38.086 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:38.087 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:38.196 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b4f4c59848dd2332,3686bdf4fc7c2a6e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=b4f4c59848dd2332, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 14:38:38.201 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b4f4c59848dd2332,3686bdf4fc7c2a6e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=b4f4c59848dd2332, duration=11 ms] 2024-11-04 14:38:38.504 INFO 21062 --- [ocp-schedules-28,0ff4fb3d51af780c,b4253b868a8bfd65] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:38:38.505 INFO 21062 --- [ocp-schedules-19,47a79d571cfab4f8,472af8c1f52697af] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:38:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:38:38.589517+08:00, lastProcessTime=2024-11-04T14:38:28.590300+08:00 2024-11-04 14:38:38.647 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,054f00f3d196d1b7,20873a54e65cb6dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.130.47.17, traceId=054f00f3d196d1b7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 14:38:38.660 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,d85cdfaee50296c7,512885f12d34fe5e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/alarmSummaries?id=1, client=10.130.47.17, traceId=d85cdfaee50296c7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1,] 2024-11-04 14:38:38.661 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,5438fcf85f5b8978,1e7fbf41e540855a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/serverStats?id=1, client=10.130.47.17, traceId=5438fcf85f5b8978, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1,] 2024-11-04 14:38:38.665 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,054f00f3d196d1b7,20873a54e65cb6dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=054f00f3d196d1b7, duration=24 ms] 2024-11-04 14:38:38.675 INFO 21062 --- [ocp-async-10,eb366fd10dc9cf6a,e71e618c2a8f9a45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:38:38.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,5438fcf85f5b8978,1e7fbf41e540855a] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:38.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,5438fcf85f5b8978,1e7fbf41e540855a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/serverStats client=10.130.47.17, traceId=5438fcf85f5b8978, duration=32 ms] 2024-11-04 14:38:38.698 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,d85cdfaee50296c7,512885f12d34fe5e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/alarmSummaries client=10.130.47.17, traceId=d85cdfaee50296c7, duration=43 ms] 2024-11-04 14:38:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:38:38.589517+08:00 2024-11-04 14:38:38.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,625a31093959da59,a65cd1eaf38e92cc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=625a31093959da59, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 14:38:38.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d52dc3dd570c7438,ea02b5976eaedf41] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=d52dc3dd570c7438, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 14:38:38.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d52dc3dd570c7438,ea02b5976eaedf41] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=d52dc3dd570c7438, duration=4 ms] 2024-11-04 14:38:38.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,625a31093959da59,a65cd1eaf38e92cc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=625a31093959da59, duration=4 ms] 2024-11-04 14:38:38.895 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,aebb0869508c70bc,fe9a457081ec6a8f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=aebb0869508c70bc, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 14:38:38.895 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,aebb0869508c70bc,fe9a457081ec6a8f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=aebb0869508c70bc, duration=4 ms] 2024-11-04 14:38:38.907 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:38:38.907 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:38:38.907 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:38:38.907 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:38:38.907 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:38:38.907 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:38:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ebd3e5e9dbbfb0bb,27cd3c611ab15cea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:39.280 WARN 21062 --- [ocp-async-6,cf35f25d0780ef05,88b5b25d8133e602] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:39.313 INFO 21062 --- [ocp-async-5,1529b103927d00e2,e975d77e67cf2f19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:38:40.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:38:40.156 INFO 21062 --- [ocp-schedules-7,2bd6917a693a528e,84958f52e9bacbce] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:38:40.160 INFO 21062 --- [ocp-schedules-7,2bd6917a693a528e,84958f52e9bacbce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:40.244 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:38:40.246 ERROR 21062 --- [ocp-schedules-7,2bd6917a693a528e,84958f52e9bacbce] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:38:40.246 INFO 21062 --- [ocp-schedules-7,2bd6917a693a528e,84958f52e9bacbce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:38:40.777 INFO 21062 --- [ocp-schedules-9,f6411681ee2e9d83,3daad7885ab80976] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:38:41.087 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:41.088 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:41.280 WARN 21062 --- [ocp-async-10,bbd72a28504506a1,dad980aa7f837d62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:41.313 INFO 21062 --- [ocp-async-8,da0f7e6694157639,a513188056107187] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:41.387 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,26934e28845eb4f6,20499700fb29dd8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=26934e28845eb4f6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 14:38:41.387 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,26934e28845eb4f6,20499700fb29dd8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=26934e28845eb4f6, duration=6 ms] 2024-11-04 14:38:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:38:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:38:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:38:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:38:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:38:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:38:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:38:41.983 INFO 21062 --- [prometheus-minute-schedule-1,3e878e3447052c59,5e16de2db7ea00a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:43.280 WARN 21062 --- [ocp-async-2,268c8c9410b8a6e0,bfe1fb3b3d4ba6dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:43.313 INFO 21062 --- [ocp-async-7,0889f418526d1752,7abc7ec4218c4582] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:44.088 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:44.089 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:44.983 INFO 21062 --- [prometheus-minute-schedule-1,efac4058c92ed91b,889173ffae52307b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:45.281 WARN 21062 --- [ocp-async-7,4ce55a87a18037c9,395ae892b84a0d3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:45.314 INFO 21062 --- [ocp-async-9,aa9f710cb155a1d0,eff6bc47ed79358b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:46.395 INFO 21062 --- [ocp-schedules-9,30a36d074948d503,d40902aae18bf331] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:38:47.089 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:47.089 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:47.281 WARN 21062 --- [ocp-async-9,725ba9e58491bb63,5640a79107ad5d02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:47.314 INFO 21062 --- [ocp-async-4,e61a83781fa44858,bbee4acd5f08b113] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:47.604 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8c0a2257165ffc47,b1a4044d90e95678] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topo?id=1, client=10.130.47.17, traceId=8c0a2257165ffc47, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterTopo(Long), args=1,] 2024-11-04 14:38:47.662 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,8c0a2257165ffc47,b1a4044d90e95678] c.o.o.o.i.cluster.ClusterServiceImpl : failed to fix unit_num, clusterId=1, reason=CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor 2024-11-04 14:38:47.663 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8c0a2257165ffc47,b1a4044d90e95678] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topo client=10.130.47.17, traceId=8c0a2257165ffc47, duration=64 ms] 2024-11-04 14:38:47.983 INFO 21062 --- [prometheus-minute-schedule-1,97d0732aaca12461,054016cab7930cac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:38:48.590220+08:00, lastProcessTime=2024-11-04T14:38:38.589517+08:00 2024-11-04 14:38:48.676 INFO 21062 --- [ocp-async-7,42d6f0af80c91354,9768c9c4f600b423] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:38:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:38:48.590220+08:00 2024-11-04 14:38:49.281 WARN 21062 --- [ocp-async-8,4199eebf8a13468b,79b58e691ad32780] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:49.314 INFO 21062 --- [ocp-async-6,a04fd74c7177fdf2,9d2db36f5d636d16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:50.090 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:50.090 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:50.780 INFO 21062 --- [ocp-schedules-13,103f815cf4dbc771,4f00e1325ffd0ebc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:38:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8e4b9455f904af2a,a34171fd747a8a22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:51.281 WARN 21062 --- [ocp-async-6,0036208f861fcb7c,914bdbc6292dac1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:51.314 INFO 21062 --- [ocp-async-9,2c3f1480b0065813,646d3f727049cc09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:51.399 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,22cfdddad2b3fae8,17f9c685525aca9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/alarmSummaries?id=1, client=10.130.47.17, traceId=22cfdddad2b3fae8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1,] 2024-11-04 14:38:51.399 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,e2045953a7ee1d69,768093c9e5963476] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=e2045953a7ee1d69, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 14:38:51.400 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,1dd6ba25a618af46,4f4489c74f0fb0a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/stats?id=1, client=10.130.47.17, traceId=1dd6ba25a618af46, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1,] 2024-11-04 14:38:51.400 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,c8d8d4df6436761b,5801523a3f57f656] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/serverStats?id=1, client=10.130.47.17, traceId=c8d8d4df6436761b, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1,] 2024-11-04 14:38:51.400 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,4ee7a8740ca7a823,0549bbde59ee7a78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/eventMeta?id=1, client=10.130.47.17, traceId=4ee7a8740ca7a823, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1,] 2024-11-04 14:38:51.400 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,3fd858332053dfc2,2f24a26a1a409643] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topSql/switch?id=1, client=10.130.47.17, traceId=3fd858332053dfc2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1,] 2024-11-04 14:38:51.400 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,4ee7a8740ca7a823,0549bbde59ee7a78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/eventMeta client=10.130.47.17, traceId=4ee7a8740ca7a823, duration=6 ms] 2024-11-04 14:38:51.427 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,1dd6ba25a618af46,4f4489c74f0fb0a0] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:51.427 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,c8d8d4df6436761b,5801523a3f57f656] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:51.427 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,c8d8d4df6436761b,5801523a3f57f656] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/serverStats client=10.130.47.17, traceId=c8d8d4df6436761b, duration=32 ms] 2024-11-04 14:38:51.428 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,22cfdddad2b3fae8,17f9c685525aca9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/alarmSummaries client=10.130.47.17, traceId=22cfdddad2b3fae8, duration=37 ms] 2024-11-04 14:38:51.430 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,1dd6ba25a618af46,4f4489c74f0fb0a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/stats client=10.130.47.17, traceId=1dd6ba25a618af46, duration=35 ms] 2024-11-04 14:38:51.442 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,e2045953a7ee1d69,768093c9e5963476] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:51.450 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,3fd858332053dfc2,2f24a26a1a409643] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topSql/switch client=10.130.47.17, traceId=3fd858332053dfc2, duration=55 ms] 2024-11-04 14:38:51.483 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,fb86db8d316a0c2a,1902bf953205b369] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/events?startTime=2024-10-28T14%3A38%3A50%2B08%3A00&endTime=2024-11-04T14%3A38%3A50%2B08%3A00&page=1&size=7, client=10.130.47.17, traceId=fb86db8d316a0c2a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1,2024-10-28T14:38:50+08:00,2024-11-04T14:38:50+08:00,Page request [number: 0, size 7, sort: timestamp: DESC],] 2024-11-04 14:38:51.489 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,fb86db8d316a0c2a,1902bf953205b369] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/events client=10.130.47.17, traceId=fb86db8d316a0c2a, duration=12 ms] 2024-11-04 14:38:51.498 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,e2045953a7ee1d69,768093c9e5963476] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=e2045953a7ee1d69, duration=103 ms] 2024-11-04 14:38:51.880 INFO 21062 --- [ocp-schedules-8,2077f123dbaebc4f,43b81b9dce588f68] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:38:51.885 INFO 21062 --- [ocp-schedules-8,2077f123dbaebc4f,43b81b9dce588f68] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:38:51.885 INFO 21062 --- [ocp-schedules-8,2077f123dbaebc4f,43b81b9dce588f68] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:38:52.608 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,86d1e5bf85ae9ed7,df6dbf4da06d3ee5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topo?id=1, client=10.130.47.17, traceId=86d1e5bf85ae9ed7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterTopo(Long), args=1,] 2024-11-04 14:38:52.662 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-3,86d1e5bf85ae9ed7,df6dbf4da06d3ee5] c.o.o.o.i.cluster.ClusterServiceImpl : failed to fix unit_num, clusterId=1, reason=CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor 2024-11-04 14:38:52.662 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,86d1e5bf85ae9ed7,df6dbf4da06d3ee5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topo client=10.130.47.17, traceId=86d1e5bf85ae9ed7, duration=60 ms] 2024-11-04 14:38:53.091 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:53.091 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:53.281 WARN 21062 --- [ocp-async-9,4883e8a877d55ed5,26c34690dcf84121] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:53.314 INFO 21062 --- [ocp-async-4,8a922f6d81ee26f5,fcd10770d5eafe19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:53.983 INFO 21062 --- [prometheus-minute-schedule-1,88aac172a1605dca,f6812fe116d3c03b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:54.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,21b8fee72e1a51ab,beb0f50b4722b245] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/alarmSummaries?id=1, client=10.130.47.17, traceId=21b8fee72e1a51ab, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1,] 2024-11-04 14:38:54.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f24b19013086a8fe,1e85054cc34d8376] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=f24b19013086a8fe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 14:38:54.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,90b27d3c0e07a187,f38de56827b27b73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/stats?id=1, client=10.130.47.17, traceId=90b27d3c0e07a187, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1,] 2024-11-04 14:38:54.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,8e628f7725ae6a1a,e7faa3b0c6903671] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/serverStats?id=1, client=10.130.47.17, traceId=8e628f7725ae6a1a, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1,] 2024-11-04 14:38:54.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,fd615e6c9402c739,1e610361594a645e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/eventMeta?id=1, client=10.130.47.17, traceId=fd615e6c9402c739, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1,] 2024-11-04 14:38:54.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,7105098efee6767f,dfedd92a70aa6d4b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topSql/switch?id=1, client=10.130.47.17, traceId=7105098efee6767f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1,] 2024-11-04 14:38:54.894 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,fd615e6c9402c739,1e610361594a645e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/eventMeta client=10.130.47.17, traceId=fd615e6c9402c739, duration=6 ms] 2024-11-04 14:38:54.921 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,8e628f7725ae6a1a,e7faa3b0c6903671] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:54.921 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,90b27d3c0e07a187,f38de56827b27b73] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:54.921 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,8e628f7725ae6a1a,e7faa3b0c6903671] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/serverStats client=10.130.47.17, traceId=8e628f7725ae6a1a, duration=33 ms] 2024-11-04 14:38:54.921 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,21b8fee72e1a51ab,beb0f50b4722b245] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/alarmSummaries client=10.130.47.17, traceId=21b8fee72e1a51ab, duration=37 ms] 2024-11-04 14:38:54.924 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,90b27d3c0e07a187,f38de56827b27b73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/stats client=10.130.47.17, traceId=90b27d3c0e07a187, duration=35 ms] 2024-11-04 14:38:54.936 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,7105098efee6767f,dfedd92a70aa6d4b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topSql/switch client=10.130.47.17, traceId=7105098efee6767f, duration=47 ms] 2024-11-04 14:38:54.937 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f24b19013086a8fe,1e85054cc34d8376] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:38:54.979 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,7b7fc648aa6f1176,02df3e74a31f6771] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/events?startTime=2024-10-28T14%3A38%3A54%2B08%3A00&endTime=2024-11-04T14%3A38%3A54%2B08%3A00&page=1&size=7, client=10.130.47.17, traceId=7b7fc648aa6f1176, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1,2024-10-28T14:38:54+08:00,2024-11-04T14:38:54+08:00,Page request [number: 0, size 7, sort: timestamp: DESC],] 2024-11-04 14:38:54.983 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,7b7fc648aa6f1176,02df3e74a31f6771] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/events client=10.130.47.17, traceId=7b7fc648aa6f1176, duration=10 ms] 2024-11-04 14:38:54.992 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f24b19013086a8fe,1e85054cc34d8376] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=f24b19013086a8fe, duration=104 ms] 2024-11-04 14:38:55.281 WARN 21062 --- [ocp-async-4,215b32891edeb031,331988dc8b923083] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:55.315 INFO 21062 --- [ocp-async-3,995a22bb8fbe45ec,1eaf0080b5f9e124] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:56.091 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:56.092 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:56.421 INFO 21062 --- [ocp-schedules-24,5a1432e98e79bc02,14a04ed8b46b71b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:38:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c69ba1919adb8fcd,4a6e0fce9bbed033] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:38:57.281 WARN 21062 --- [ocp-async-3,d3452cb517b4603c,5faff94b1cc6f964] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:57.315 INFO 21062 --- [ocp-async-1,c264935973eeab48,8bdb890fba0e908d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:58.500 INFO 21062 --- [ocp-schedules-20,536341e897eb2661,7d88f14ab1ea2c1d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:38:58.500 INFO 21062 --- [ocp-schedules-23,a11a5b0180df63e6,619f96cabb35633b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:38:58.502 INFO 21062 --- [ocp-schedules-3,b3196c70dc28a58a,a6963b0351f0b388] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:38:58.504 INFO 21062 --- [ocp-schedules-23,a11a5b0180df63e6,619f96cabb35633b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:58.504 INFO 21062 --- [ocp-schedules-20,536341e897eb2661,7d88f14ab1ea2c1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:38:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:38:58.505 INFO 21062 --- [ocp-schedules-26,5c0f9e0b53321d5f,db6aeec3db477fe0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:38:58.510 INFO 21062 --- [ocp-schedules-9,5056414660bf16bb,e5a4d4fed5c73016] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:38:58.510 INFO 21062 --- [ocp-schedules-18,98bf5cb32c4ee997,de2e13ad41db8cf6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:38:58.510 INFO 21062 --- [ocp-schedules-9,5056414660bf16bb,e5a4d4fed5c73016] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:38:58.510 INFO 21062 --- [ocp-schedules-18,98bf5cb32c4ee997,de2e13ad41db8cf6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:38:58.510 INFO 21062 --- [ocp-schedules-10,5f8bc8990e246fa9,96b735dff7755a9f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:38:58.510 INFO 21062 --- [ocp-schedules-10,5f8bc8990e246fa9,96b735dff7755a9f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:38:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:38:58.589974+08:00, lastProcessTime=2024-11-04T14:38:48.590220+08:00 2024-11-04 14:38:58.675 INFO 21062 --- [ocp-async-4,e3fb3442b9b54422,fb11be55c737db76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:38:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:38:58.589974+08:00 2024-11-04 14:38:59.092 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:38:59.093 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:38:59.251 INFO 21062 --- [ocp-schedules-30,a571d44d4e641e8b,26e59ccf6fcd449f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:38:59.282 WARN 21062 --- [ocp-async-9,9355d5091fbe323c,fd3f25dda6233f96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:38:59.315 INFO 21062 --- [ocp-async-4,6a72b3c262a12501,1869ba51c74a8c84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:38:59.983 INFO 21062 --- [prometheus-minute-schedule-1,44032ce4bea7d5e8,4392bb94aab873d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702340, interval=60 2024-11-04 14:39:00.782 INFO 21062 --- [ocp-schedules-4,d4c6f561c7d4b46d,da8953a99d147f65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:39:01.282 WARN 21062 --- [ocp-async-4,5e85adaef952a4ec,eef0427da74f525b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:01.315 INFO 21062 --- [ocp-async-3,535a47ce7b5633bd,d48bc3423c961036] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:02.093 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:02.094 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2753334659fa3d6f,a94dc711da84840c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:03.282 WARN 21062 --- [ocp-async-3,722952dc4ebf45a7,37db8a50e0fbf2fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:03.315 INFO 21062 --- [ocp-async-1,a183be417ef56047,68e82760791596f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:05.094 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:05.095 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:05.282 WARN 21062 --- [ocp-async-1,d24bc8e424f3667e,510d879ceea2c238] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:05.315 INFO 21062 --- [ocp-async-10,6f138cf497c7faf8,6e56483550bb768d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:05.983 INFO 21062 --- [prometheus-minute-schedule-1,59dfc87a035f02d5,8f14b70dd582ad6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:06.463 INFO 21062 --- [ocp-schedules-21,88832249fe8b0c72,44401b87667873a7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:39:07.185 INFO 21062 --- [ocp-schedules-12,cac3927578efbf90,8c5fe42890b87f48] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:39:07.185 INFO 21062 --- [ocp-schedules-12,604b622190afeda2,2cf509869d242126] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:39:07.189 INFO 21062 --- [ocp-schedules-12,604b622190afeda2,2cf509869d242126] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:07.219 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:39:07.219 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:39:07.282 WARN 21062 --- [ocp-async-10,fe8a9436f331232d,34b37c81f2c0203f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:07.316 INFO 21062 --- [ocp-async-8,59e793deb7382bde,5126e36ab535a854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:08.095 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:08.096 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:08.505 INFO 21062 --- [ocp-schedules-23,a11a5b0180df63e6,619f96cabb35633b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:39:08.505 INFO 21062 --- [ocp-schedules-20,536341e897eb2661,7d88f14ab1ea2c1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:39:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:39:08.589945+08:00, lastProcessTime=2024-11-04T14:38:58.589974+08:00 2024-11-04 14:39:08.676 INFO 21062 --- [ocp-async-1,a1aebb286b700ea2,11bd2eec32318d33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:39:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:39:08.589945+08:00 2024-11-04 14:39:08.983 INFO 21062 --- [prometheus-minute-schedule-1,fb47929104bd3426,1aaa822e8404169b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:09.283 WARN 21062 --- [ocp-async-3,6783ac11c236b30d,09dce3e3127138c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:09.316 INFO 21062 --- [ocp-async-7,be79511e456b0603,ebe5d0ebde6dc5e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:10.002 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:39:10.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:39:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:39:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:39:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:39:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:39:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:39:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:39:10.785 INFO 21062 --- [ocp-schedules-19,754aaceecaf63194,6b976724c028986a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:39:10.965 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1244c44624e16260,d397ddf2c45d02cc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topo?id=1, client=10.130.47.17, traceId=1244c44624e16260, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterTopo(Long), args=1,] 2024-11-04 14:39:11.020 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-7,1244c44624e16260,d397ddf2c45d02cc] c.o.o.o.i.cluster.ClusterServiceImpl : failed to fix unit_num, clusterId=1, reason=CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor 2024-11-04 14:39:11.020 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1244c44624e16260,d397ddf2c45d02cc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topo client=10.130.47.17, traceId=1244c44624e16260, duration=65 ms] 2024-11-04 14:39:11.096 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:11.097 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:11.283 WARN 21062 --- [ocp-async-7,aae7b29dae059a2b,00889942c9ff74ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:11.316 INFO 21062 --- [ocp-async-5,a91107980c944d2e,0d056d326256e8ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=7, affectRows=7 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=6, validCount=6 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=23, affectRows=23 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=15, affectRows=15 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=15, affectRows=15 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=23, validCount=23 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=15, validCount=15 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=41, affectRows=41 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=15, validCount=15 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=28, affectRows=28 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=35, affectRows=35 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=28, validCount=28 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=41, validCount=41 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=43, affectRows=43 2024-11-04 14:39:11.624 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=69, affectRows=69 2024-11-04 14:39:11.625 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=35, validCount=35 2024-11-04 14:39:11.625 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=61, affectRows=61 2024-11-04 14:39:11.625 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=43, validCount=43 2024-11-04 14:39:11.625 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=68, validCount=68 2024-11-04 14:39:11.625 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=61, validCount=61 2024-11-04 14:39:11.625 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702280, nodeCount=67, affectRows=67 2024-11-04 14:39:11.625 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702280, nodeCount=66, validCount=66 2024-11-04 14:39:11.889 INFO 21062 --- [ocp-schedules-18,9f52e9402d2e9bc0,aafa5a53282c9922] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:39:11.894 INFO 21062 --- [ocp-schedules-18,9f52e9402d2e9bc0,aafa5a53282c9922] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:39:11.894 INFO 21062 --- [ocp-schedules-18,9f52e9402d2e9bc0,aafa5a53282c9922] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:39:11.983 INFO 21062 --- [prometheus-minute-schedule-1,f4ebbf582f0cff28,9f9f232b597fc4fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:13.283 WARN 21062 --- [ocp-async-5,cdee6b77e64586ae,230b5be82e96682e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:13.316 INFO 21062 --- [ocp-async-2,de00deee78416aec,b4db5ba601f326a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,1da39ec0f45595bc,db4177abad940375] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1592,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":404,"secondWriteQueue":0,"minuteWriteCacheSize":401} 2024-11-04 14:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,1da39ec0f45595bc,db4177abad940375] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=57, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=57, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:39:14.097 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:14.098 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:14.983 INFO 21062 --- [prometheus-minute-schedule-1,b61ab16a78c0b06f,8dc4764276bb464b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:15.188 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:35:15.164, end=2024-11-04T14:37:15.185} 2024-11-04 14:39:15.192 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:39:15.192 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:39:15.283 WARN 21062 --- [ocp-async-8,cf0997b1ac2f28a8,9a214056f4247a53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:15.316 INFO 21062 --- [ocp-async-6,3cf61c578f415195,8bd3ef883dd1c1d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:16.490 INFO 21062 --- [ocp-schedules-19,aa56337bcea9b445,a034eb7b8cedb408] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:39:17.098 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:17.099 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:17.283 WARN 21062 --- [ocp-async-1,b6c9008142f41a18,01d44e740e448549] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:17.317 INFO 21062 --- [ocp-async-9,33712eb1d0a71ae1,49441e27b681fe5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:17.983 INFO 21062 --- [prometheus-minute-schedule-1,2af25fb8d502e67b,d759dfa4c3aba874] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:18.355 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d646dfda968c5eb4,cd33a96065d7c6f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=d646dfda968c5eb4, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 14:39:18.356 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d646dfda968c5eb4,cd33a96065d7c6f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=d646dfda968c5eb4, duration=9 ms] 2024-11-04 14:39:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:39:18.590362+08:00, lastProcessTime=2024-11-04T14:39:08.589945+08:00 2024-11-04 14:39:18.676 INFO 21062 --- [ocp-async-8,af0ceb4a480e313b,9ae2f5db7f58d10d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:39:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:39:18.590362+08:00 2024-11-04 14:39:18.787 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,75ee383062a81bc4,f6d5ce49ce4fa585] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=75ee383062a81bc4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 14:39:18.789 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,2d73500ed4de508a,f11f809ba27ec9d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=2d73500ed4de508a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 14:39:18.789 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,2d73500ed4de508a,f11f809ba27ec9d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=2d73500ed4de508a, duration=6 ms] 2024-11-04 14:39:18.789 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,12335019d7ef33ad,eb32fa0affb67783] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=12335019d7ef33ad, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 14:39:18.790 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,12335019d7ef33ad,eb32fa0affb67783] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=12335019d7ef33ad, duration=7 ms] 2024-11-04 14:39:18.792 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,6b80da949ce8615d,554bff60c49ef07d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=6b80da949ce8615d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:18.793 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,75ee383062a81bc4,f6d5ce49ce4fa585] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=75ee383062a81bc4, duration=11 ms] 2024-11-04 14:39:18.863 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,615accba4711af78,68d6e4889f234160] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/creators?null, client=10.130.47.17, traceId=615accba4711af78, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.listCreatorInfo(TaskType), args=MANUAL,] 2024-11-04 14:39:18.864 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,615accba4711af78,68d6e4889f234160] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/creators client=10.130.47.17, traceId=615accba4711af78, duration=7 ms] 2024-11-04 14:39:18.877 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,7f40a40055aaf7ff,8e0becb978438166] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/manual-instances?page=1&size=10, client=10.130.47.17, traceId=7f40a40055aaf7ff, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listManualTaskInstances(String,Pageable,List,String), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:18.889 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,7f40a40055aaf7ff,8e0becb978438166] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/manual-instances client=10.130.47.17, traceId=7f40a40055aaf7ff, duration=17 ms] 2024-11-04 14:39:18.941 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,14e15352572410c7,83465aa06ded640b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=14e15352572410c7, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 14:39:18.945 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,14e15352572410c7,83465aa06ded640b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=14e15352572410c7, duration=9 ms] 2024-11-04 14:39:18.974 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,6b80da949ce8615d,554bff60c49ef07d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=6b80da949ce8615d, duration=190 ms] 2024-11-04 14:39:19.283 WARN 21062 --- [ocp-async-5,2669d80493ecc295,ac7414d2ac030a33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:19.317 INFO 21062 --- [ocp-async-2,1113084ff074fe48,dcfeac3dec9c533b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:20.099 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:20.100 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:20.787 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,4c5b7bb5cbeb6e76,3e7709f85853164c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=4c5b7bb5cbeb6e76, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 14:39:20.787 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,4c5b7bb5cbeb6e76,3e7709f85853164c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=4c5b7bb5cbeb6e76, duration=4 ms] 2024-11-04 14:39:20.788 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,4dbbdad0394656a6,740114a8c6557c06] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=4dbbdad0394656a6, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 14:39:20.788 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,4dbbdad0394656a6,740114a8c6557c06] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=4dbbdad0394656a6, duration=5 ms] 2024-11-04 14:39:20.788 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,eeb67c703f4b03f9,aa2c538255793b53] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=eeb67c703f4b03f9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 14:39:20.788 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,eeb67c703f4b03f9,aa2c538255793b53] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=eeb67c703f4b03f9, duration=6 ms] 2024-11-04 14:39:20.789 INFO 21062 --- [ocp-schedules-23,e725e4e92e9a6a24,390eb72c35eb0f0a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:39:20.954 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:20.954 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:20.954 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:20.954 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:20.954 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:20.954 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:20.983 INFO 21062 --- [prometheus-minute-schedule-1,7494d97f0ce654e6,f52d54d16dbf48f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:21.284 WARN 21062 --- [ocp-async-8,080ae477d28b824d,fafcdb3fa07b1303] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:21.317 INFO 21062 --- [ocp-async-6,1914df60dfe0c4f1,64cfab118efdb0af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:21.370 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:21.370 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/cluster/1, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:21.370 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:22.202 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,06aa94e86cce7f6e,37376714903c09bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.130.47.17, traceId=06aa94e86cce7f6e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2024-11-04 14:39:22.205 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,06aa94e86cce7f6e,37376714903c09bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.130.47.17, traceId=06aa94e86cce7f6e, duration=9 ms] 2024-11-04 14:39:22.337 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,fb307d0303f8b3b1,7c3c87466036a57a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.130.47.17, traceId=fb307d0303f8b3b1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 14:39:22.338 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8975112eaea8bbd9,f0b971883670dbb4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=8975112eaea8bbd9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 14:39:22.339 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9999c82dec135999,0e502b6da99ace39] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=9999c82dec135999, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 14:39:22.339 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9999c82dec135999,0e502b6da99ace39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=9999c82dec135999, duration=5 ms] 2024-11-04 14:39:22.344 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8975112eaea8bbd9,f0b971883670dbb4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=8975112eaea8bbd9, duration=10 ms] 2024-11-04 14:39:22.356 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,fb307d0303f8b3b1,7c3c87466036a57a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=fb307d0303f8b3b1, duration=26 ms] 2024-11-04 14:39:22.635 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,d4723947c06fefd2,e47a80a3035edaa3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/eventMeta?id=1, client=10.130.47.17, traceId=d4723947c06fefd2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1,] 2024-11-04 14:39:22.636 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,d4723947c06fefd2,e47a80a3035edaa3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/eventMeta client=10.130.47.17, traceId=d4723947c06fefd2, duration=9 ms] 2024-11-04 14:39:22.637 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,c7c7b34985f7f79f,d492bcdc3d25f888] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=c7c7b34985f7f79f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 14:39:22.637 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,575c953a531a1c9b,21910a65ec4065dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/stats?id=1, client=10.130.47.17, traceId=575c953a531a1c9b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1,] 2024-11-04 14:39:22.638 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,31ee3ac2138ad618,7efcbfd2f2a6c0ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topSql/switch?id=1, client=10.130.47.17, traceId=31ee3ac2138ad618, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1,] 2024-11-04 14:39:22.664 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,575c953a531a1c9b,21910a65ec4065dc] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:39:22.667 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,575c953a531a1c9b,21910a65ec4065dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/stats client=10.130.47.17, traceId=575c953a531a1c9b, duration=36 ms] 2024-11-04 14:39:22.678 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,c7c7b34985f7f79f,d492bcdc3d25f888] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:39:22.679 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,31ee3ac2138ad618,7efcbfd2f2a6c0ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/topSql/switch client=10.130.47.17, traceId=31ee3ac2138ad618, duration=47 ms] 2024-11-04 14:39:22.683 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,26c4bdb864421f95,84f1ad4e6cb85178] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/events?startTime=2024-10-28T14%3A39%3A22%2B08%3A00&endTime=2024-11-04T14%3A39%3A22%2B08%3A00&page=1&size=5, client=10.130.47.17, traceId=26c4bdb864421f95, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1,2024-10-28T14:39:22+08:00,2024-11-04T14:39:22+08:00,Page request [number: 0, size 5, sort: timestamp: DESC],] 2024-11-04 14:39:22.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,26c4bdb864421f95,84f1ad4e6cb85178] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/events client=10.130.47.17, traceId=26c4bdb864421f95, duration=11 ms] 2024-11-04 14:39:22.735 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,c7c7b34985f7f79f,d492bcdc3d25f888] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=c7c7b34985f7f79f, duration=104 ms] 2024-11-04 14:39:22.853 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,037c9dd765a278aa,cc0736e076f84bae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=037c9dd765a278aa, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 14:39:22.858 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,037c9dd765a278aa,cc0736e076f84bae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=037c9dd765a278aa, duration=10 ms] 2024-11-04 14:39:23.100 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:23.101 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:23.134 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,961572db0e44084f,06d4695d0ba7648c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=961572db0e44084f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:23.136 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,1d5666c734ee44e3,5f840001e4359bcf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.130.47.17, traceId=1d5666c734ee44e3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2024-11-04 14:39:23.137 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,38070e1a45b55293,f3988a2838c8e965] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1&status=RUNNING%2CFAILED, client=10.130.47.17, traceId=38070e1a45b55293, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1,] 2024-11-04 14:39:23.150 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,38070e1a45b55293,f3988a2838c8e965] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.130.47.17, traceId=38070e1a45b55293, duration=20 ms] 2024-11-04 14:39:23.181 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,1d5666c734ee44e3,5f840001e4359bcf] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:39:23.233 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,1d5666c734ee44e3,5f840001e4359bcf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/summary client=10.130.47.17, traceId=1d5666c734ee44e3, duration=103 ms] 2024-11-04 14:39:23.284 WARN 21062 --- [ocp-async-1,a6ada71f37427e2e,8eac284d9dcacb03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:23.310 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,961572db0e44084f,06d4695d0ba7648c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=961572db0e44084f, duration=184 ms] 2024-11-04 14:39:23.317 INFO 21062 --- [ocp-async-9,05ddca680b3366f5,be889c03da0e0d8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:23.753 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,ac881e76191856b9,e261f3c35632737d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.130.47.17, traceId=ac881e76191856b9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 14:39:23.755 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,855cc150e32a9a85,01759873ef8d6cac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/serverStats?id=1, client=10.130.47.17, traceId=855cc150e32a9a85, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1,] 2024-11-04 14:39:23.755 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3eacd41ae744c1e5,0ee56312f0554aa8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/alarmSummaries?id=1, client=10.130.47.17, traceId=3eacd41ae744c1e5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1,] 2024-11-04 14:39:23.772 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,ac881e76191856b9,e261f3c35632737d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=ac881e76191856b9, duration=27 ms] 2024-11-04 14:39:23.782 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,855cc150e32a9a85,01759873ef8d6cac] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster dba_test_ob:1730694047 is not connectable, skip to collect stats 2024-11-04 14:39:23.782 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,855cc150e32a9a85,01759873ef8d6cac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/serverStats client=10.130.47.17, traceId=855cc150e32a9a85, duration=33 ms] 2024-11-04 14:39:23.784 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3eacd41ae744c1e5,0ee56312f0554aa8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/alarmSummaries client=10.130.47.17, traceId=3eacd41ae744c1e5, duration=35 ms] 2024-11-04 14:39:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4f8e728be743f028,d59564fa994eac16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:24.471 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,0f3623058ceeea63,0eedafffdd2ab89c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=0f3623058ceeea63, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 14:39:24.471 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,0f3623058ceeea63,0eedafffdd2ab89c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=0f3623058ceeea63, duration=5 ms] 2024-11-04 14:39:24.471 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,d55c8250c21beee1,335db63fb730c0f2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=d55c8250c21beee1, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 14:39:24.471 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,f4a64c9910c3d771,6f8024dff5f708b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=f4a64c9910c3d771, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 14:39:24.471 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,d55c8250c21beee1,335db63fb730c0f2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=d55c8250c21beee1, duration=5 ms] 2024-11-04 14:39:24.471 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,f4a64c9910c3d771,6f8024dff5f708b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=f4a64c9910c3d771, duration=6 ms] 2024-11-04 14:39:24.483 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:24.483 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:24.483 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:24.485 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:24.485 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:24.485 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:25.284 WARN 21062 --- [ocp-async-10,79cf4d350281c467,6e8b995decb7983a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:25.317 INFO 21062 --- [ocp-async-4,7909355ed8e5aa2e,274e82609de393f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:26.101 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:26.102 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:26.529 INFO 21062 --- [ocp-schedules-8,a3c2f69c4bd958e9,55e78cfdfcccacdc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:39:26.983 INFO 21062 --- [prometheus-minute-schedule-1,df6b33d7518100b2,79265e9a2e632407] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:26.984 INFO 21062 --- [prometheus-hour-schedule-1,d3a21665a1f78fa7,0935e2b840e1bda8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:39:27.206 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,37c2479f3eed1d47,26c0a8c66a43deb5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=37c2479f3eed1d47, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 14:39:27.206 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,37c2479f3eed1d47,26c0a8c66a43deb5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=37c2479f3eed1d47, duration=7 ms] 2024-11-04 14:39:27.284 WARN 21062 --- [ocp-async-2,212dc550f3e7d1e6,79c7a70e1d4612ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:27.317 INFO 21062 --- [ocp-async-3,b9ee9137fc01018a,cdebb2ae1bfa78f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:28.498 INFO 21062 --- [ocp-schedules-30,d9fdd877ad915414,1c6ee205b747c201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:39:28.499 INFO 21062 --- [ocp-schedules-13,5675de32e32ecdaa,d35b513f210b26e2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:39:28.500 INFO 21062 --- [ocp-schedules-16,65272996e69766ba,f275228f9b84ab8d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:39:28.500 INFO 21062 --- [ocp-schedules-17,01db7b5ae536881d,2f63b7ee71cb8258] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:39:28.502 INFO 21062 --- [ocp-schedules-22,d89b89b07aa1d7d0,0da207471758dd90] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:39:28.503 INFO 21062 --- [ocp-schedules-30,d9fdd877ad915414,1c6ee205b747c201] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:39:28.504 INFO 21062 --- [ocp-schedules-16,65272996e69766ba,f275228f9b84ab8d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:28.504 INFO 21062 --- [ocp-schedules-17,01db7b5ae536881d,2f63b7ee71cb8258] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:28.504 INFO 21062 --- [ocp-schedules-13,5675de32e32ecdaa,d35b513f210b26e2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:39:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:39:28.505 INFO 21062 --- [ocp-schedules-24,129a39172700715c,5daa1598963fd692] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:39:28.510 INFO 21062 --- [ocp-schedules-4,0f7e27cb1bd283a2,680cc898e6edb499] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:39:28.510 INFO 21062 --- [ocp-schedules-18,0c3f87cb89289f4c,991bcdcbc7b9fb9a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:39:28.510 INFO 21062 --- [ocp-schedules-18,0c3f87cb89289f4c,991bcdcbc7b9fb9a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:39:28.510 INFO 21062 --- [ocp-schedules-4,0f7e27cb1bd283a2,680cc898e6edb499] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:39:28.510 INFO 21062 --- [ocp-schedules-28,6d976725f83dc9d8,d2bdfc7a4c487ec3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:39:28.510 INFO 21062 --- [ocp-schedules-28,6d976725f83dc9d8,d2bdfc7a4c487ec3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:39:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:39:28.590131+08:00, lastProcessTime=2024-11-04T14:39:18.590362+08:00 2024-11-04 14:39:28.598 INFO 21062 --- [ocp-schedules-27,e4804f7ea7358b21,a71af7f2f39b0988] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:39:28.598+08:00, tenantSize=0 2024-11-04 14:39:28.638 INFO 21062 --- [ocp-schedules-26,69c08b0820c3f3b2,f23e019b0286e2c9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:39:28.638 INFO 21062 --- [ocp-schedules-26,69c08b0820c3f3b2,f23e019b0286e2c9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:39:28.676 INFO 21062 --- [ocp-async-4,dc2181669bc216d8,e7b6a867ee5c0877] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:39:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:39:28.590131+08:00 2024-11-04 14:39:28.872 INFO 21062 --- [ocp-schedules-25,4fe23643463cad94,9ddc73c65a9df022] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:39:29.090 INFO 21062 --- [ocp-schedules-4,48409ce3de0c80f3,15a677e39b004057] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:39:29.095 INFO 21062 --- [ocp-schedules-4,48409ce3de0c80f3,15a677e39b004057] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:29.095 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:39:29.095 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,df2aa9ee0fc34cba,8c8e80f56e4845b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=df2aa9ee0fc34cba, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 14:39:29.095 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,58f0c14c61571502,aabc40c6632af64a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=58f0c14c61571502, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 14:39:29.095 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,df2aa9ee0fc34cba,8c8e80f56e4845b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=df2aa9ee0fc34cba, duration=5 ms] 2024-11-04 14:39:29.096 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,67aed957be4dbda7,320e9720b2aa6f1f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=67aed957be4dbda7, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 14:39:29.097 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,67aed957be4dbda7,320e9720b2aa6f1f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=67aed957be4dbda7, duration=7 ms] 2024-11-04 14:39:29.099 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,50728db3f94d4d15,90e901a2a094b454] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=50728db3f94d4d15, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:29.102 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,58f0c14c61571502,aabc40c6632af64a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=58f0c14c61571502, duration=12 ms] 2024-11-04 14:39:29.102 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:29.103 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:29.171 INFO 21062 --- [ocp-schedules-28,d9b9bfe935f0e772,2337e2475d0981e2] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:39:29.171 INFO 21062 --- [ocp-schedules-28,d9b9bfe935f0e772,2337e2475d0981e2] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:39:29.234 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,844552d215dd35e4,74889b2290fb4ac4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/regions?null, client=10.130.47.17, traceId=844552d215dd35e4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listRegions(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-11-04 14:39:29.237 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,ed31ef6c6fe83f07,fe13989d498fde00] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.130.47.17, traceId=ed31ef6c6fe83f07, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-11-04 14:39:29.238 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,e4abea6020438a86,02f49a1ce8520ccd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.130.47.17, traceId=e4abea6020438a86, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-11-04 14:39:29.240 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,ed31ef6c6fe83f07,fe13989d498fde00] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.130.47.17, traceId=ed31ef6c6fe83f07, duration=7 ms] 2024-11-04 14:39:29.244 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,844552d215dd35e4,74889b2290fb4ac4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/regions client=10.130.47.17, traceId=844552d215dd35e4, duration=16 ms] 2024-11-04 14:39:29.249 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,e4abea6020438a86,02f49a1ce8520ccd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.130.47.17, traceId=e4abea6020438a86, duration=16 ms] 2024-11-04 14:39:29.253 INFO 21062 --- [ocp-schedules-30,49b5581a9d74700c,4275ca43d21ee879] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:39:29.259 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,5a6a9acc24531cad,0274076bcd3827e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.130.47.17, traceId=5a6a9acc24531cad, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:29.263 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-8,5a6a9acc24531cad,0274076bcd3827e0] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-11-04 14:39:29.278 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,50728db3f94d4d15,90e901a2a094b454] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=50728db3f94d4d15, duration=187 ms] 2024-11-04 14:39:29.284 WARN 21062 --- [ocp-async-7,59bece4556b32d60,80e6a3803de6f0f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:29.287 INFO 21062 --- [ocp-schedules-8,9bcadf28e54aebbd,18b6f7b7e624e802] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:39:29.292 INFO 21062 --- [ocp-schedules-8,9bcadf28e54aebbd,18b6f7b7e624e802] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:39:29.292 INFO 21062 --- [ocp-schedules-8,9bcadf28e54aebbd,18b6f7b7e624e802] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:39:29.318 INFO 21062 --- [ocp-async-5,2fa519feaae11734,6a2c2043bfb5965c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:29.340 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,9570746632ef6ec0,dea045e378cb3cfc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=9570746632ef6ec0, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 14:39:29.344 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,9570746632ef6ec0,dea045e378cb3cfc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=9570746632ef6ec0, duration=9 ms] 2024-11-04 14:39:29.402 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,5a6a9acc24531cad,0274076bcd3827e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.130.47.17, traceId=5a6a9acc24531cad, duration=148 ms] 2024-11-04 14:39:29.652 INFO 21062 --- [ocp-schedules-20,d88095e84e1b3ee9,033cbbe1e74d0206] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:39:29.652 INFO 21062 --- [ocp-schedules-20,d88095e84e1b3ee9,033cbbe1e74d0206] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:39:29.653 INFO 21062 --- [ocp-schedules-20,d88095e84e1b3ee9,033cbbe1e74d0206] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:39:29.653 INFO 21062 --- [ocp-schedules-20,d88095e84e1b3ee9,033cbbe1e74d0206] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:39:29.654 INFO 21062 --- [ocp-schedules-20,d88095e84e1b3ee9,033cbbe1e74d0206] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:39:29.654 INFO 21062 --- [ocp-schedules-20,d88095e84e1b3ee9,033cbbe1e74d0206] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709932806753767424 2024-11-04 14:39:29.655 INFO 21062 --- [ocp-schedules-20,d88095e84e1b3ee9,033cbbe1e74d0206] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=8, maxSyncedRecordId=709939727355367424 2024-11-04 14:39:29.983 INFO 21062 --- [prometheus-minute-schedule-1,058358667f279dc8,16f0cc1fe83e3db2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:39:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:39:30.792 INFO 21062 --- [ocp-schedules-8,399423ae496fba28,f8ec329c57cc692d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:39:31.082 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,807d9bacbad02caf,24ef58062dba7ecc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=807d9bacbad02caf, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 14:39:31.082 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f6f7d6a48b2e0fa9,ad64c861b2c24394] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=f6f7d6a48b2e0fa9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 14:39:31.082 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,807d9bacbad02caf,24ef58062dba7ecc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=807d9bacbad02caf, duration=4 ms] 2024-11-04 14:39:31.082 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f6f7d6a48b2e0fa9,ad64c861b2c24394] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=f6f7d6a48b2e0fa9, duration=4 ms] 2024-11-04 14:39:31.082 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d5ceda1ecdc7b679,52d8c52096bf78fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=d5ceda1ecdc7b679, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 14:39:31.083 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d5ceda1ecdc7b679,52d8c52096bf78fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=d5ceda1ecdc7b679, duration=5 ms] 2024-11-04 14:39:31.261 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:31.261 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:31.261 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:31.262 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:31.262 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:31.262 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:31.284 WARN 21062 --- [ocp-async-4,a9d76f07a8ea4234,431fa8f59dd93890] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:31.318 INFO 21062 --- [ocp-async-8,5de61ba1aab6b3e4,176ce6c433af5c8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:31.897 INFO 21062 --- [ocp-schedules-7,305ed438c5d86863,88b7118f3a0cfaa3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:39:31.904 INFO 21062 --- [ocp-schedules-7,305ed438c5d86863,88b7118f3a0cfaa3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:39:31.904 INFO 21062 --- [ocp-schedules-7,305ed438c5d86863,88b7118f3a0cfaa3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:39:31.996 INFO 21062 --- [ocp-schedules-5,67f5552760e7ed21,8c6a2409f623e59a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:39:31.996 INFO 21062 --- [ocp-schedules-5,67f5552760e7ed21,8c6a2409f623e59a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:39:32.024 INFO 21062 --- [ocp-schedules-8,8df4332cfa8b21de,51b4e75b4495c00e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:39:32.024 INFO 21062 --- [ocp-schedules-5,67f5552760e7ed21,8c6a2409f623e59a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:39:32.024 INFO 21062 --- [ocp-schedules-5,67f5552760e7ed21,8c6a2409f623e59a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:39:32.103 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:32.103 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:32.213 INFO 21062 --- [ocp-schedules-19,998d60fffdbfc4f0,d58a9c099a05a6fc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:39:32.212+08:00, clusters=[1] 2024-11-04 14:39:32.214 INFO 21062 --- [ocp-schedules-19,998d60fffdbfc4f0,d58a9c099a05a6fc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:39:32.407 INFO 21062 --- [ocp-schedules-13,3d53ab8d7e2d547d,e6bda8c88c5b2e21] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:39:32.407+08:00, clusters=[1] 2024-11-04 14:39:32.411 INFO 21062 --- [ocp-schedules-13,3d53ab8d7e2d547d,e6bda8c88c5b2e21] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:39:32.983 INFO 21062 --- [prometheus-minute-schedule-1,5788eb4a6c0a964e,a1fd2df2d23c2bd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:33.285 WARN 21062 --- [ocp-async-3,cd0a365c98a5bb4b,c797f1f78a14d9dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:33.318 INFO 21062 --- [ocp-async-1,138260cae910454a,9208853bdc91777b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:33.616 INFO 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:39:33.616 INFO 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:39:33.616 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:39:33.616 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:39:33.626 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:39:33.626 INFO 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:39:33.627 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:39:33.628 WARN 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:39:33.628 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:39:33.629 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:39:33.629 INFO 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:39:33.629 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:39:33.629 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:39:33.630 WARN 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:39:33.630 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:39:33.630 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:39:33.638 INFO 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:39:33.638 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:39:33.638 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:39:33.648 ERROR 21062 --- [ocp-schedules-6,bd25f193147a8234,242397152cf535e3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:39:34.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,19b80a518f23bab8,cb187a233ba00489] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.130.47.17, traceId=19b80a518f23bab8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:34.688 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-7,19b80a518f23bab8,cb187a233ba00489] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-11-04 14:39:34.727 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,19b80a518f23bab8,cb187a233ba00489] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.130.47.17, traceId=19b80a518f23bab8, duration=49 ms] 2024-11-04 14:39:35.104 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:35.104 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:35.285 WARN 21062 --- [ocp-async-9,bdf885c612342cf1,8e6d0c2591c1e868] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:35.318 INFO 21062 --- [ocp-async-10,b5efd4ed08765603,f4fda3589cf80267] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:35.983 INFO 21062 --- [prometheus-minute-schedule-1,13b75cbd8cdc0dd8,a731442a32e4b7a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:36.556 INFO 21062 --- [ocp-schedules-22,fa3d570762d7a7fe,ac293ed2e49be69b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:39:37.219 INFO 21062 --- [ocp-schedules-12,604b622190afeda2,2cf509869d242126] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:39:37.219 INFO 21062 --- [ocp-schedules-12,538af8724a1e5133,e3077b18cb052e34] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:39:37.223 INFO 21062 --- [ocp-schedules-12,538af8724a1e5133,e3077b18cb052e34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:37.255 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:39:37.255 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:39:37.285 WARN 21062 --- [ocp-async-5,5b311e218a097598,cdea6342321c4297] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:37.318 INFO 21062 --- [ocp-async-2,07f9540e30a93769,349d178f33720f27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:37.614 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d55c3c9d38910edd,89b9f72ea07536f9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=10.130.47.17, traceId=d55c3c9d38910edd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,] 2024-11-04 14:39:37.623 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d55c3c9d38910edd,89b9f72ea07536f9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.130.47.17, traceId=d55c3c9d38910edd, duration=14 ms] 2024-11-04 14:39:37.640 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,66b66fff91309fab,f232deff4f1974ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.130.47.17, traceId=66b66fff91309fab, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:37.694 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,66b66fff91309fab,f232deff4f1974ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.130.47.17, traceId=66b66fff91309fab, duration=62 ms] 2024-11-04 14:39:37.921 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,740d65c433ec305a,b3615e74f23c4f98] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-11-04T13%3A39%3A37%2B08%3A00&endTime=2024-11-04T14%3A39%3A37%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.130.47.17, traceId=740d65c433ec305a, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-11-04T13:39:37+08:00,2024-11-04T14:39:37+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-11-04 14:39:37.921 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-6,740d65c433ec305a,b3615e74f23c4f98] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_count, labels = app="OB", groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2024-11-04 14:39:37.923 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,740d65c433ec305a,b3615e74f23c4f98] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.130.47.17, traceId=740d65c433ec305a, duration=9 ms] 2024-11-04 14:39:37.939 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5ac9dd7859c822ba,1204f55b2a51d2b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-11-04T13%3A39%3A37%2B08%3A00&endTime=2024-11-04T14%3A39%3A37%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.130.47.17, traceId=5ac9dd7859c822ba, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-11-04T13:39:37+08:00,2024-11-04T14:39:37+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-11-04 14:39:37.939 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-3,5ac9dd7859c822ba,1204f55b2a51d2b5] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = tps, labels = app="OB", groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2024-11-04 14:39:37.941 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5ac9dd7859c822ba,1204f55b2a51d2b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.130.47.17, traceId=5ac9dd7859c822ba, duration=6 ms] 2024-11-04 14:39:37.963 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,93d510b7564c2144,f65fa60013b44c40] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-11-04T13%3A39%3A37%2B08%3A00&endTime=2024-11-04T14%3A39%3A37%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.130.47.17, traceId=93d510b7564c2144, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-11-04T13:39:37+08:00,2024-11-04T14:39:37+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-11-04 14:39:37.963 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-2,93d510b7564c2144,f65fa60013b44c40] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_rt, labels = app="OB", groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2024-11-04 14:39:37.966 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,93d510b7564c2144,f65fa60013b44c40] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.130.47.17, traceId=93d510b7564c2144, duration=7 ms] 2024-11-04 14:39:37.983 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ab2f0818f481dbb7,30bf1b7250541a2b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-11-04T13%3A39%3A37%2B08%3A00&endTime=2024-11-04T14%3A39%3A37%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.130.47.17, traceId=ab2f0818f481dbb7, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-11-04T13:39:37+08:00,2024-11-04T14:39:37+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-11-04 14:39:37.983 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,ab2f0818f481dbb7,30bf1b7250541a2b] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = tps_rt, labels = app="OB", groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2024-11-04 14:39:37.984 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,ab2f0818f481dbb7,30bf1b7250541a2b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.130.47.17, traceId=ab2f0818f481dbb7, duration=7 ms] 2024-11-04 14:39:38.104 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:38.105 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:38.504 INFO 21062 --- [ocp-schedules-17,01db7b5ae536881d,2f63b7ee71cb8258] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:39:38.504 INFO 21062 --- [ocp-schedules-16,65272996e69766ba,f275228f9b84ab8d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:39:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:39:38.590335+08:00, lastProcessTime=2024-11-04T14:39:28.590131+08:00 2024-11-04 14:39:38.676 INFO 21062 --- [ocp-async-9,5c5b4b7223dbb99b,5b8c8b444852c137] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:39:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:39:38.590335+08:00 2024-11-04 14:39:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e8a0cb244d3869d8,531ff790f2634e38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:39.095 INFO 21062 --- [ocp-schedules-4,48409ce3de0c80f3,15a677e39b004057] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:39:39.285 WARN 21062 --- [ocp-async-6,8f2424d90541e7fc,fdd41b6b7f4150a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:39.318 INFO 21062 --- [ocp-async-7,e046325096bf863b,dc3654c6b54a2453] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:39:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:39:40.247 INFO 21062 --- [ocp-schedules-10,c6c0c470245ab52c,baaa110e282eec91] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:39:40.250 INFO 21062 --- [ocp-schedules-10,c6c0c470245ab52c,baaa110e282eec91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:40.336 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:39:40.338 ERROR 21062 --- [ocp-schedules-10,c6c0c470245ab52c,baaa110e282eec91] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:39:40.338 INFO 21062 --- [ocp-schedules-10,c6c0c470245ab52c,baaa110e282eec91] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:39:40.795 INFO 21062 --- [ocp-schedules-13,b361c99b18db157b,d243e37073c871c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:39:41.105 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:41.106 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:41.285 WARN 21062 --- [ocp-async-9,4859fd748e10b2db,efa59b49246efa57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:41.319 INFO 21062 --- [ocp-async-5,1bbecfb14d1b57dc,5dfe6fc8299b2df4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:41.983 INFO 21062 --- [prometheus-minute-schedule-1,024b7e8b411869ed,fcb3b02cf8b76b85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:42.629 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,0b9ef69bdcb9a96c,abb70c1500e96fc0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/dashboards?null, client=10.130.47.17, traceId=0b9ef69bdcb9a96c, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorDashboardsController.queryMonitorDashboards(Pageable), args=Page request [number: 0, size 5000, sort: order: ASC],] 2024-11-04 14:39:42.688 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,0b9ef69bdcb9a96c,abb70c1500e96fc0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/dashboards client=10.130.47.17, traceId=0b9ef69bdcb9a96c, duration=65 ms] 2024-11-04 14:39:43.285 WARN 21062 --- [ocp-async-4,79395c991de89f83,8d3c48df85ca748f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:43.319 INFO 21062 --- [ocp-async-8,cce2b1fc566738c5,c330c09dbb41d8e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:43.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,3943bc6b896c1474,4c4f55f150017a9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.130.47.17, traceId=3943bc6b896c1474, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-11-04 14:39:43.763 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4f113a69eb34c49d,211139fad401723e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/obproxy/clusters?null, client=10.130.47.17, traceId=4f113a69eb34c49d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObproxyController.listObproxyClusters(Pageable,List,String,List,Boolean), args=Page request [number: 0, size 5000, sort: id: ASC],false,] 2024-11-04 14:39:43.764 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9fdcfa1707ffca7a,9abd1d3a5d293cc8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=10.130.47.17, traceId=9fdcfa1707ffca7a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List,WithRemoteTenant,List), args=Page request [number: 0, size 5000, sort: id: DESC],NONE,] 2024-11-04 14:39:43.772 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9fdcfa1707ffca7a,9abd1d3a5d293cc8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=10.130.47.17, traceId=9fdcfa1707ffca7a, duration=16 ms] 2024-11-04 14:39:43.782 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,3943bc6b896c1474,4c4f55f150017a9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.130.47.17, traceId=3943bc6b896c1474, duration=26 ms] 2024-11-04 14:39:43.800 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4f113a69eb34c49d,211139fad401723e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/obproxy/clusters client=10.130.47.17, traceId=4f113a69eb34c49d, duration=44 ms] 2024-11-04 14:39:43.823 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,99634e8c2fcc61c8,bf92e3a4315dcca4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.130.47.17, traceId=99634e8c2fcc61c8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-11-04 14:39:43.825 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-4,99634e8c2fcc61c8,bf92e3a4315dcca4] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-11-04 14:39:43.860 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,99634e8c2fcc61c8,bf92e3a4315dcca4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.130.47.17, traceId=99634e8c2fcc61c8, duration=42 ms] 2024-11-04 14:39:44.106 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:44.107 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9a25bf8733ed58d1,af77d61071a194e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:45.286 WARN 21062 --- [ocp-async-8,feb4192ccadf1ac8,64b9050b039fd959] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:45.319 INFO 21062 --- [ocp-async-1,82f5eb443f1f9c38,c1dca40c75056a8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:45.322 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,47104c4cae88cd1f,da9b0a9acbfae42f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=10.130.47.17, traceId=47104c4cae88cd1f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List,WithRemoteTenant,List), args=Page request [number: 0, size 5000, sort: id: DESC],NONE,] 2024-11-04 14:39:45.324 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,2cc692c2353bc932,24cd012914d1e336] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/groups?null, client=10.130.47.17, traceId=2cc692c2353bc932, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarmGroups(Integer,Integer), args=] 2024-11-04 14:39:45.324 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,159721f7f6025017,0c2d77cb66964b5b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/templates?null, client=10.130.47.17, traceId=159721f7f6025017, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.getAlarmTemplates(String,String,Boolean,Boolean,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-11-04 14:39:45.328 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,47104c4cae88cd1f,da9b0a9acbfae42f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=10.130.47.17, traceId=47104c4cae88cd1f, duration=14 ms] 2024-11-04 14:39:45.346 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,159721f7f6025017,0c2d77cb66964b5b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/templates client=10.130.47.17, traceId=159721f7f6025017, duration=29 ms] 2024-11-04 14:39:45.369 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,2cc692c2353bc932,24cd012914d1e336] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/groups client=10.130.47.17, traceId=2cc692c2353bc932, duration=50 ms] 2024-11-04 14:39:45.501 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1b732906329b1066,6aece3d909bb2d05] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?page=1&size=10&collapsed=false, client=10.130.47.17, traceId=1b732906329b1066, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:45.509 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 14:39:45.509 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/arbitration/services, method=10.130.47.17, client=GET, traceId=] 2024-11-04 14:39:45.509 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 14:39:45.631 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,906dcaff75771c1d,e92847e2d57c7231] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/regions?null, client=10.130.47.17, traceId=906dcaff75771c1d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listRegions(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-11-04 14:39:45.631 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,916285eb257b8f2c,49fdf9011922366c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.130.47.17, traceId=916285eb257b8f2c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-11-04 14:39:45.631 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c1b6a504ad1ca3ac,7941def4acb15034] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.130.47.17, traceId=c1b6a504ad1ca3ac, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-11-04 14:39:45.634 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,906dcaff75771c1d,e92847e2d57c7231] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/regions client=10.130.47.17, traceId=906dcaff75771c1d, duration=8 ms] 2024-11-04 14:39:45.634 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c1b6a504ad1ca3ac,7941def4acb15034] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.130.47.17, traceId=c1b6a504ad1ca3ac, duration=8 ms] 2024-11-04 14:39:45.635 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,916285eb257b8f2c,49fdf9011922366c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.130.47.17, traceId=916285eb257b8f2c, duration=9 ms] 2024-11-04 14:39:45.660 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,775cf26f40f0f34a,bd044d38429716c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.130.47.17, traceId=775cf26f40f0f34a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:45.663 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-4,775cf26f40f0f34a,bd044d38429716c4] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-11-04 14:39:45.681 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1b732906329b1066,6aece3d909bb2d05] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=1b732906329b1066, duration=187 ms] 2024-11-04 14:39:45.699 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,775cf26f40f0f34a,bd044d38429716c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.130.47.17, traceId=775cf26f40f0f34a, duration=44 ms] 2024-11-04 14:39:46.630 INFO 21062 --- [ocp-schedules-19,0f41a79d98d4ffca,9ba447555a0b185b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:39:47.107 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:47.108 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:47.286 WARN 21062 --- [ocp-async-1,834d15a140324d99,dbb38eab9a713248] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:47.319 INFO 21062 --- [ocp-async-10,602cf81f3ef5d9c3,c95bf049beb8018e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:47.983 INFO 21062 --- [prometheus-minute-schedule-1,33e79c1921c49f17,18ae0624ff5fdfaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:39:48.591139+08:00, lastProcessTime=2024-11-04T14:39:38.590335+08:00 2024-11-04 14:39:48.676 INFO 21062 --- [ocp-async-8,4313022f7a755116,5699574667152bf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:39:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:39:48.591139+08:00 2024-11-04 14:39:49.286 WARN 21062 --- [ocp-async-5,0383258d8bbc7e64,3902290edc1f93ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:49.319 INFO 21062 --- [ocp-async-8,93cf88a1db941143,f31102efea45da20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:50.108 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:50.109 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:50.799 INFO 21062 --- [ocp-schedules-26,eb0ebe3eb2362d2c,2dd6b2770c464f47] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:39:50.909 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8a787f808bef3581,845ce0b381d54593] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.130.47.17, traceId=8a787f808bef3581, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:50.911 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-9,8a787f808bef3581,845ce0b381d54593] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-11-04 14:39:50.949 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,8a787f808bef3581,845ce0b381d54593] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.130.47.17, traceId=8a787f808bef3581, duration=49 ms] 2024-11-04 14:39:50.983 INFO 21062 --- [prometheus-minute-schedule-1,124fb429be3975ae,4f05c1da2a404b0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:51.286 WARN 21062 --- [ocp-async-8,c8b8bae22e4ea969,2e3d9f6e1df8bcdb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:51.320 INFO 21062 --- [ocp-async-1,f365498d4582257c,97f09a94bf05a401] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:51.907 INFO 21062 --- [ocp-schedules-20,a38cba73bc76e98c,11efb5a755daa0a9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:39:51.912 INFO 21062 --- [ocp-schedules-20,a38cba73bc76e98c,11efb5a755daa0a9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:39:51.912 INFO 21062 --- [ocp-schedules-20,a38cba73bc76e98c,11efb5a755daa0a9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:39:52.222 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,acb5d62715d80d6f,33dc7336a6b55456] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=acb5d62715d80d6f, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 14:39:52.223 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,acb5d62715d80d6f,33dc7336a6b55456] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=acb5d62715d80d6f, duration=6 ms] 2024-11-04 14:39:52.494 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2b6870ed1a050285,69467744fdd5f294] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/creators?null, client=10.130.47.17, traceId=2b6870ed1a050285, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.listCreatorInfo(TaskType), args=MANUAL,] 2024-11-04 14:39:52.495 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,2b6870ed1a050285,69467744fdd5f294] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/creators client=10.130.47.17, traceId=2b6870ed1a050285, duration=6 ms] 2024-11-04 14:39:52.512 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,8acb512cd10d80c2,72becba32d7dba7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/manual-instances?page=1&size=10, client=10.130.47.17, traceId=8acb512cd10d80c2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listManualTaskInstances(String,Pageable,List,String), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:39:52.522 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,8acb512cd10d80c2,72becba32d7dba7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/manual-instances client=10.130.47.17, traceId=8acb512cd10d80c2, duration=16 ms] 2024-11-04 14:39:53.109 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:53.109 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:53.137 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:39:53.138 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:39:53.143 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:39:53.143 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:39:53.143 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:39:53.143 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:39:53.149 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:39:53.149 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:39:53.149 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:39:53.149 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:39:53.184 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:39:53.192 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:39:53.192 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:39:53.192 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:39:53.192 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:39:53.192 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:39:53.200 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:39:53.286 WARN 21062 --- [ocp-async-1,2ca6c10bb0fadd45,13aef1c1d03e0c4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:53.320 INFO 21062 --- [ocp-async-10,698066af42678712,93000a265880ddd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:53.718 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:39:53.983 INFO 21062 --- [prometheus-minute-schedule-1,eefcef6cf044b5ee,19f13585ffffe4b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:54.278 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.129082173S 2024-11-04 14:39:54.278 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:39:54.378 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:39:54.378 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:39:54.401 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:39:54.492 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:39:54.492 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:39:54.493 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:39:54.493 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:39:54.493 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:39:54.493 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:39:54.503 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:39:54.523 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,65c41e917f4a1c46,deb308134093a4a6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=65c41e917f4a1c46, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:39:54.552 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,65c41e917f4a1c46,deb308134093a4a6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=65c41e917f4a1c46, duration=37 ms] 2024-11-04 14:39:54.679 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] o.s.boot.SpringApplication : Started application in 1.536 seconds (JVM running for 7808.358) 2024-11-04 14:39:54.679 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.536274557S 2024-11-04 14:39:55.096 INFO 21062 --- [ocp-schedules-30,ad3678e02c2f6668,d73b540b30b6933a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:39:55.286 WARN 21062 --- [ocp-async-10,38ace0ebd627ff37,1f464119c49d8d64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:55.320 INFO 21062 --- [ocp-async-2,5886720834f94394,263e1abbb1dc7653] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:55.794 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,0483a6a36de032f7,f6123ffe4d2cb369] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=0483a6a36de032f7, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:39:55.800 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,0483a6a36de032f7,f6123ffe4d2cb369] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=0483a6a36de032f7, duration=14 ms] 2024-11-04 14:39:56.110 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:56.110 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:56.658 INFO 21062 --- [ocp-schedules-24,900f17eb87122f6c,b5af50e6d36e4c56] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:39:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c55c154896020c24,8536109e3f4c0c0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:39:57.287 WARN 21062 --- [ocp-async-2,df7ea72934a58267,0557e4be4de3b27f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:57.320 INFO 21062 --- [ocp-async-3,deeb17a4a2b3e1b5,9f8c363af15afdee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:58.500 INFO 21062 --- [ocp-schedules-28,c3540a14089822a1,38f4cfa94599689a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:39:58.500 INFO 21062 --- [ocp-schedules-19,7c7582cce6a7809b,f80cf07597e13a5d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:39:58.502 INFO 21062 --- [ocp-schedules-2,8139ae32748e8630,52af897878616154] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:39:58.503 INFO 21062 --- [ocp-schedules-28,c3540a14089822a1,38f4cfa94599689a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:39:58.503 INFO 21062 --- [ocp-schedules-19,7c7582cce6a7809b,f80cf07597e13a5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:39:58.505 INFO 21062 --- [ocp-schedules-23,08a9c58ea5226394,57d0aac96da79fd4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:39:58.509 INFO 21062 --- [ocp-schedules-4,0e47b498fc5071ee,a6bc705adc91a9e0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:39:58.509 INFO 21062 --- [ocp-schedules-4,0e47b498fc5071ee,a6bc705adc91a9e0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:39:58.509 INFO 21062 --- [ocp-schedules-9,ac85c7e928bea4d3,f2941c34f3432d7d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:39:58.509 INFO 21062 --- [ocp-schedules-9,ac85c7e928bea4d3,f2941c34f3432d7d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:39:58.509 INFO 21062 --- [ocp-schedules-18,2e6d7450b8756cd5,7c7932480381e698] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:39:58.509 INFO 21062 --- [ocp-schedules-18,2e6d7450b8756cd5,7c7932480381e698] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:39:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:39:58.590346+08:00, lastProcessTime=2024-11-04T14:39:48.591139+08:00 2024-11-04 14:39:58.677 INFO 21062 --- [ocp-async-10,a8e6992a13449f52,51cc2ede80ce72a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:39:58.744 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:39:58.590346+08:00 2024-11-04 14:39:59.110 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:39:59.111 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:39:59.256 INFO 21062 --- [ocp-schedules-30,12e7ea6995ab46b5,e858709e76e490dd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:39:59.287 WARN 21062 --- [ocp-async-1,1121da13b3825177,e5049ace225b1820] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:39:59.320 INFO 21062 --- [ocp-async-10,3e3fd257f33f37e2,a3e8d91ecc6495d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:39:59.983 INFO 21062 --- [prometheus-minute-schedule-1,41ff37e1ae10e44f,abca0fc183d9cba4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702400, interval=60 2024-11-04 14:40:00.117 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,86a970ab48032f37] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:35:00.0, nextRun=2024-11-04 14:40:00.0, currentDate=Mon Nov 04 14:40:00 CST 2024 2024-11-04 14:40:00.117 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,86a970ab48032f37] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:35:00.0, nextRunTime=2024-11-04 14:40:00.0} 2024-11-04 14:40:00.141 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,86a970ab48032f37] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:40:00.167 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,86a970ab48032f37] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:40:00.167 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,86a970ab48032f37] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:40:00.802 INFO 21062 --- [ocp-schedules-18,ba91f0d5c4c1053e,e0d3cd7ccb93bd2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:40:01.287 WARN 21062 --- [ocp-async-10,2007e49eabfcbcd9,e7b9d876826c7ab7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:01.321 INFO 21062 --- [ocp-async-2,4c4380255fb896e2,e56b313770530297] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:02.111 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:02.112 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1a210a38ceafa0b5,d4b1e0370819de78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:03.287 WARN 21062 --- [ocp-async-2,a57cc16e33f5de31,32b4fea5488afbe4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:03.321 INFO 21062 --- [ocp-async-3,f4ee417f67332f84,37c8435a64d7a222] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:04.240 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b1aef303257e8f80] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:30:04.0, nextRun=2024-11-04 14:40:04.0, currentDate=Mon Nov 04 14:40:04 CST 2024 2024-11-04 14:40:04.240 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b1aef303257e8f80] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 14:30:04.0, nextRunTime=2024-11-04 14:40:04.0} 2024-11-04 14:40:04.246 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b1aef303257e8f80] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 14:40:04.246 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b1aef303257e8f80] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T14:40:04.246+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 14:40:04.995 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,9318fe88ca35de76] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 82, operation:EXECUTE, state: RUNNING 2024-11-04 14:40:05.013 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=82, context=Context{parallelIdx=-1, stringMap={task_instance_id=18, task_operation=execute, latest_execution_start_time=2024-11-04T14:40:04.989+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=82}, listMap={}}, executor=10.46.128.24 2024-11-04 14:40:05.018 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:40:05.018 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:40:05.020 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.020 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:40:05.021 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.022 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.022 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.023 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:40:05.038 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.038 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.039 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.039 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:40:05.053 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.054 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.054 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.054 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:40:05.068 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.068 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.069 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.069 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:40:05.081 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.082 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.082 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.083 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:40:05.088 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.088 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.089 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.090 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:40:05.095 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.095 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.096 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.097 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:40:05.101 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.102 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.103 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.103 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:40:05.109 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.109 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.110 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.110 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:40:05.112 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:05.113 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:05.115 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.116 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.116 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.116 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:40:05.122 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.122 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.123 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.123 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:40:05.132 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.133 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.133 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.134 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:40:05.148 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.149 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.149 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.150 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:40:05.154 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.155 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.156 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.156 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:40:05.161 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.162 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.162 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.163 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:40:05.168 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.168 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.169 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.169 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:40:05.174 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.174 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.175 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.175 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:40:05.179 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.180 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.180 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.180 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:40:05.184 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.185 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.186 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.186 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:40:05.190 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.191 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.191 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.192 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:40:05.195 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.196 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.196 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.197 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:40:05.201 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.202 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.202 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.203 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:40:05.207 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.208 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.208 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.208 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:40:05.212 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.213 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.213 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.214 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:40:05.218 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.218 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.218 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.219 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:40:05.223 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.223 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.224 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.224 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:40:05.229 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.229 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.229 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.230 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:40:05.234 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.235 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.235 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.235 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:40:05.240 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.240 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.241 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.241 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:40:05.246 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.246 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.247 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.247 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:40:05.251 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.251 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.252 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.252 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:40:05.257 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.257 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.258 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.258 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:40:05.262 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.263 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.263 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.264 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:40:05.268 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.268 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.269 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.269 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:40:05.276 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.277 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.277 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.278 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:40:05.284 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.284 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.285 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.285 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:40:05.287 WARN 21062 --- [ocp-async-3,ce9a7c1ee70a99f1,9e28ad42f96cc1d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:05.290 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.290 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.291 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.291 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:40:05.296 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.297 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.297 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.298 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:40:05.301 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.301 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.302 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.302 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:40:05.306 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.306 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.307 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.308 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:40:05.311 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.312 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.312 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.313 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:40:05.317 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.317 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.318 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.318 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:40:05.321 INFO 21062 --- [ocp-async-9,214e17d19f02bcbf,b43fb4f678a2353f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:05.327 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.328 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.328 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.329 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:40:05.350 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.350 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.351 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.351 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:40:05.357 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.357 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.358 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.358 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:40:05.364 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.365 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.365 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.366 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:40:05.371 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.372 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.372 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.373 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:40:05.377 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.377 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.378 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.378 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:40:05.382 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.382 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.383 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.384 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:40:05.388 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:40:05.388 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:40:05.389 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.389 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:40:05.390 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.391 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.391 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.392 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:40:05.397 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.397 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:40:05.398 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:05.398 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:40:05.402 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:40:05.409 INFO 21062 --- [subtask-executor15,dd193f8f67a5a423,1ce8a19904c7780d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 82, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:40:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f44a0eb351c59aea,5ee55020e5e0d8cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:06.190 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:18, status:SUCCESS 2024-11-04 14:40:06.191 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:18, status:SUCCESS 2024-11-04 14:40:06.684 INFO 21062 --- [ocp-schedules-15,4b4fc5240931b9ca,c1997de4f5ce72fe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:40:07.255 INFO 21062 --- [ocp-schedules-12,538af8724a1e5133,e3077b18cb052e34] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:40:07.256 INFO 21062 --- [ocp-schedules-12,e17cf04a9240a39f,cf8e2eebef8bb6f6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:40:07.260 INFO 21062 --- [ocp-schedules-12,e17cf04a9240a39f,cf8e2eebef8bb6f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:07.287 WARN 21062 --- [ocp-async-9,bfc6af72c93a56c8,c79472ad022bfa13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:07.289 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:40:07.289 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:40:07.321 INFO 21062 --- [ocp-async-5,d6fdaef29875d772,432572a1571de9ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:08.113 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:08.114 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:08.504 INFO 21062 --- [ocp-schedules-28,c3540a14089822a1,38f4cfa94599689a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:40:08.504 INFO 21062 --- [ocp-schedules-19,7c7582cce6a7809b,f80cf07597e13a5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:40:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:40:08.590352+08:00, lastProcessTime=2024-11-04T14:39:58.590346+08:00 2024-11-04 14:40:08.677 INFO 21062 --- [ocp-async-3,58efebc1b87152c6,763bd24511bb80fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:40:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:40:08.590352+08:00 2024-11-04 14:40:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ca1a32e1234be757,d45fb3a2ff56f5b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:09.287 WARN 21062 --- [ocp-async-2,6797a51e829d5356,aa367b1cff9e6656] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:09.321 INFO 21062 --- [ocp-async-6,744e8d84bc31e9d2,d6982ae739c68478] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:40:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:40:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:40:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:40:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:40:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:40:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:40:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:40:10.805 INFO 21062 --- [ocp-schedules-27,6fc9c87cb515132b,65a895fa3d540711] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:40:11.114 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:11.115 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:11.288 WARN 21062 --- [ocp-async-6,6beb9ff47fa8ad4d,9891315c4d9f83f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:11.321 INFO 21062 --- [ocp-async-7,bae12162cce84092,e16fc6e60f3c477c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:11.629 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=1, affectRows=1 2024-11-04 14:40:11.629 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=1, validCount=1 2024-11-04 14:40:11.629 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=12, affectRows=12 2024-11-04 14:40:11.629 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=12, validCount=12 2024-11-04 14:40:11.629 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=22, affectRows=22 2024-11-04 14:40:11.629 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=16, affectRows=16 2024-11-04 14:40:11.630 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=22, validCount=22 2024-11-04 14:40:11.630 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=18, affectRows=18 2024-11-04 14:40:11.630 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=16, validCount=16 2024-11-04 14:40:11.630 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=18, validCount=18 2024-11-04 14:40:11.630 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=57, affectRows=57 2024-11-04 14:40:11.630 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=56, validCount=56 2024-11-04 14:40:11.630 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=71, affectRows=71 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=71, validCount=71 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=9, affectRows=9 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=9, validCount=9 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=20, affectRows=20 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=19, validCount=19 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=25, affectRows=25 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=33, affectRows=33 2024-11-04 14:40:11.631 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=25, validCount=25 2024-11-04 14:40:11.632 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=33, validCount=33 2024-11-04 14:40:11.632 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=25, affectRows=25 2024-11-04 14:40:11.632 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=25, validCount=25 2024-11-04 14:40:11.633 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=68, affectRows=68 2024-11-04 14:40:11.633 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=67, validCount=67 2024-11-04 14:40:11.633 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702340, nodeCount=45, affectRows=45 2024-11-04 14:40:11.634 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702340, nodeCount=45, validCount=45 2024-11-04 14:40:11.914 INFO 21062 --- [ocp-schedules-10,fc72f09f36997dae,5306397115dd1a62] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:40:11.920 INFO 21062 --- [ocp-schedules-10,fc72f09f36997dae,5306397115dd1a62] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:40:11.920 INFO 21062 --- [ocp-schedules-10,fc72f09f36997dae,5306397115dd1a62] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:40:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6e6e592d2e06f24e,8f93ba510dfe6f4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:13.288 WARN 21062 --- [ocp-async-7,0fac41d20c73efb3,c999feb82f62806f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:13.322 INFO 21062 --- [ocp-async-4,4e4a4f2e0a500cae,f4e25097943e9609] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,0340e44174d90dc6,6f8cbec620127007] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1618,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,0340e44174d90dc6,6f8cbec620127007] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:40:14.115 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:14.116 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9c4ea10e123f6599,bf8faf5d9f669c6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:15.288 WARN 21062 --- [ocp-async-5,5dddfb3565764627,27d80e730f6bd23c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:15.322 INFO 21062 --- [ocp-async-8,c812373413acd4cf,a2c49d05ce83c967] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:16.709 INFO 21062 --- [ocp-schedules-11,640bf5e7dc2ee185,cf9829c7411e4544] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:40:17.116 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:17.117 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:17.288 WARN 21062 --- [ocp-async-3,1dfc7730a189f258,2ace54d81f378021] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:17.322 INFO 21062 --- [ocp-async-1,face693de9dbbb83,9209c5f3e591b84d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9861ac2c2a38b7c4,337a8b2db14ea570] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:40:18.591383+08:00, lastProcessTime=2024-11-04T14:40:08.590352+08:00 2024-11-04 14:40:18.677 INFO 21062 --- [ocp-async-5,960af75aafce442f,3a4f8bad73582f55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:40:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:40:18.591383+08:00 2024-11-04 14:40:19.289 WARN 21062 --- [ocp-async-7,64751959611371a2,f55066f291b01232] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:19.322 INFO 21062 --- [ocp-async-4,66b81453b425bd96,d516f73dc141892e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:20.117 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:20.117 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:20.809 INFO 21062 --- [ocp-schedules-14,d24d1a9c8d5e56a6,f034c955001891ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:40:20.983 INFO 21062 --- [prometheus-minute-schedule-1,537a3b367d262775,884329137a1e8edc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:21.289 WARN 21062 --- [ocp-async-5,10767ece83a02d00,35ce5b8c3d818062] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:21.322 INFO 21062 --- [ocp-async-8,ea24875ede41e892,6b5914a853f703e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:23.118 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:23.119 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:23.289 WARN 21062 --- [ocp-async-3,1ea2044662132e6a,f4f3c1cc2511d3e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:23.322 INFO 21062 --- [ocp-async-1,54efbad537f8c2ea,4db76e7b9ea67187] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8a7bed8f5c604cf1,76ea10c6e82ab8e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:25.289 WARN 21062 --- [ocp-async-9,6fae7b99583516d9,c669f75b971e7965] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:25.323 INFO 21062 --- [ocp-async-10,74ec137be8f5df12,02b25d92b3620276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:26.119 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:26.119 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:26.750 INFO 21062 --- [ocp-schedules-4,941031f261966fca,8a6048c8e3cd2488] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:40:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3127d276a6102966,18058b4d76203f4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8afb98b7247a6fa7,509012d62d54028f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:40:27.289 WARN 21062 --- [ocp-async-4,c0a1d90781e2bab6,a3eda2b0b8e10f5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:27.323 INFO 21062 --- [ocp-async-2,57da0573c5434bda,70f9aa1b6cf8f0f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:28.498 INFO 21062 --- [ocp-schedules-24,a03ad0ec935863ac,9a240cd1307b6842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:40:28.499 INFO 21062 --- [ocp-schedules-9,933ac8b265729c5c,6f85e072a136ad07] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:40:28.500 INFO 21062 --- [ocp-schedules-13,13c3c0e6a7e8f9ea,1d30e8cf51391c98] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:40:28.500 INFO 21062 --- [ocp-schedules-19,c06df9741fb3da38,bfeef881c92e544c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:40:28.502 INFO 21062 --- [ocp-schedules-24,a03ad0ec935863ac,9a240cd1307b6842] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:40:28.502 INFO 21062 --- [ocp-schedules-16,f57663ec90c2ec33,7224b1f2a4f248fb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:40:28.503 INFO 21062 --- [ocp-schedules-13,13c3c0e6a7e8f9ea,1d30e8cf51391c98] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:40:28.504 INFO 21062 --- [ocp-schedules-20,3e9ebabb8183acd2,bda9fc838395161a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:40:28.505 INFO 21062 --- [ocp-schedules-9,933ac8b265729c5c,6f85e072a136ad07] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:40:28.507 INFO 21062 --- [ocp-schedules-19,c06df9741fb3da38,bfeef881c92e544c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:28.509 INFO 21062 --- [ocp-schedules-21,bada63e3c1727db6,69ce3a2932d86819] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:40:28.509 INFO 21062 --- [ocp-schedules-30,a1bdaac5790c0090,9074f19983040264] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:40:28.510 INFO 21062 --- [ocp-schedules-30,a1bdaac5790c0090,9074f19983040264] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:40:28.510 INFO 21062 --- [ocp-schedules-21,bada63e3c1727db6,69ce3a2932d86819] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:40:28.510 INFO 21062 --- [ocp-schedules-3,0a29e01efe07566f,dfef4cb6ff321202] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:40:28.510 INFO 21062 --- [ocp-schedules-3,0a29e01efe07566f,dfef4cb6ff321202] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:40:28.579 INFO 21062 --- [ocp-schedules-25,ca43cc5fb10e14ff,34b50e078de101f7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1618,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:40:28.579 INFO 21062 --- [ocp-schedules-25,ca43cc5fb10e14ff,34b50e078de101f7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:40:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:40:28.589830+08:00, lastProcessTime=2024-11-04T14:40:18.591383+08:00 2024-11-04 14:40:28.677 INFO 21062 --- [ocp-async-10,fd19a9c9c390d035,a77387cbfc473c2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:40:28.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:40:28.589830+08:00 2024-11-04 14:40:28.875 INFO 21062 --- [ocp-schedules-17,eac8ac9fba4f4a33,99d30185f3d6a6a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:40:29.120 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:29.120 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:29.175 INFO 21062 --- [ocp-schedules-6,a6e4c226395eb51d,b671229835d9c844] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:40:29.175 INFO 21062 --- [ocp-schedules-6,a6e4c226395eb51d,b671229835d9c844] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:40:29.258 INFO 21062 --- [ocp-schedules-21,a7b54f7eda520e48,c6014466044c2a20] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:40:29.289 WARN 21062 --- [ocp-async-6,66dd41377e68337c,cd6d1b7791a1057c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:29.292 INFO 21062 --- [ocp-schedules-24,eef2a10a86bc1a99,a9d6ad3245cf0f62] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:40:29.297 INFO 21062 --- [ocp-schedules-24,eef2a10a86bc1a99,a9d6ad3245cf0f62] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:40:29.297 INFO 21062 --- [ocp-schedules-24,eef2a10a86bc1a99,a9d6ad3245cf0f62] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:40:29.323 INFO 21062 --- [ocp-async-7,d05b6956ae19cd68,cd5fe15688ec6779] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:29.480 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:40:29.480 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:40:29.488 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:40:29.488 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:40:29.488 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:40:29.499 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:40:29.499 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:40:29.499 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:40:29.502 INFO 21062 --- [ocp-schedules-22,546b1b9a17419ebf,8b60abf7a8bba5d4] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:40:29.656 INFO 21062 --- [ocp-schedules-15,3056e65f1ba5badc,d7c1f702b3ef5ef5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:40:29.656 INFO 21062 --- [ocp-schedules-15,3056e65f1ba5badc,d7c1f702b3ef5ef5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:40:29.663 INFO 21062 --- [ocp-schedules-15,3056e65f1ba5badc,d7c1f702b3ef5ef5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:40:29.663 INFO 21062 --- [ocp-schedules-15,3056e65f1ba5badc,d7c1f702b3ef5ef5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:40:29.669 INFO 21062 --- [ocp-schedules-15,3056e65f1ba5badc,d7c1f702b3ef5ef5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:40:29.669 INFO 21062 --- [ocp-schedules-15,3056e65f1ba5badc,d7c1f702b3ef5ef5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939727355367424 2024-11-04 14:40:29.674 INFO 21062 --- [ocp-schedules-15,3056e65f1ba5badc,d7c1f702b3ef5ef5] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=18, maxSyncedRecordId=709939979026190336 2024-11-04 14:40:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3d6c2012c7b0ee2f,51a51380ed97e00e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:40:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:40:30.812 INFO 21062 --- [ocp-schedules-2,2354c818a54fa647,3ddfb9f2d481d301] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:40:31.290 WARN 21062 --- [ocp-async-10,85213572f31c5059,0d6cc45168bb1d7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:31.323 INFO 21062 --- [ocp-async-5,a3d713349ef255d0,974500ef6dfa0c73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:31.923 INFO 21062 --- [ocp-schedules-20,b20bb7ac3f5cdf04,574f69a9c9237170] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:40:31.929 INFO 21062 --- [ocp-schedules-20,b20bb7ac3f5cdf04,574f69a9c9237170] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:40:31.929 INFO 21062 --- [ocp-schedules-20,b20bb7ac3f5cdf04,574f69a9c9237170] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:40:32.024 INFO 21062 --- [ocp-schedules-22,30a8475e918d01aa,25bd9528f65105ae] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:40:32.024 INFO 21062 --- [ocp-schedules-22,30a8475e918d01aa,25bd9528f65105ae] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:40:32.052 INFO 21062 --- [ocp-schedules-22,30a8475e918d01aa,25bd9528f65105ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:40:32.052 INFO 21062 --- [ocp-schedules-22,30a8475e918d01aa,25bd9528f65105ae] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:40:32.054 INFO 21062 --- [ocp-schedules-2,46fb2fd89e9b2d1a,564805a5fc921717] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:40:32.120 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:32.121 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:32.240 INFO 21062 --- [ocp-schedules-9,c946afc342982937,42613fa929144d9a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:40:32.240+08:00, clusters=[1] 2024-11-04 14:40:32.242 INFO 21062 --- [ocp-schedules-9,c946afc342982937,42613fa929144d9a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:40:32.437 INFO 21062 --- [ocp-schedules-8,52e05831c00466a4,6e5947a2becba54f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:40:32.437+08:00, clusters=[1] 2024-11-04 14:40:32.440 INFO 21062 --- [ocp-schedules-8,52e05831c00466a4,6e5947a2becba54f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:40:32.445 INFO 21062 --- [ocp-schedules-6,8cb3e1883b1aa644,001d93d2682bbc39] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:40:32.448 INFO 21062 --- [ocp-schedules-6,8cb3e1883b1aa644,001d93d2682bbc39] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:32.507 ERROR 21062 --- [ocp-schedules-6,8cb3e1883b1aa644,001d93d2682bbc39] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:40:32.507 INFO 21062 --- [ocp-schedules-6,8cb3e1883b1aa644,001d93d2682bbc39] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:40:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8e01e0503c6c9b1a,e90c1d4b1f899c5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:33.290 WARN 21062 --- [ocp-async-2,90c796f9b3ec0ed0,c70c7df9d8567254] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:33.324 INFO 21062 --- [ocp-async-3,19f1396323c722e8,979224b148301942] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:33.658 INFO 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:40:33.658 INFO 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:40:33.658 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:40:33.658 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:40:33.667 INFO 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:40:33.668 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:40:33.668 WARN 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:40:33.669 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:40:33.669 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:40:33.669 INFO 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:40:33.669 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:40:33.670 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:40:33.670 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:40:33.671 WARN 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:40:33.671 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:40:33.672 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:40:33.678 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:40:33.678 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:40:33.678 INFO 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:40:33.689 ERROR 21062 --- [ocp-schedules-7,1906fda7191165ae,30ea14346ddd4dc8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:40:35.121 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:35.122 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:35.290 WARN 21062 --- [ocp-async-1,166e6c061071cb4f,fc5abdbf45f7f6d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:35.324 INFO 21062 --- [ocp-async-9,58ccac12916b294f,d4839f129732b316] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e550c0fc11ce43f2,4aec26ab9ae5fd8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:36.775 INFO 21062 --- [ocp-schedules-22,ee9e8d8f53d8c79e,50f601413fea3bc1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:40:37.289 INFO 21062 --- [ocp-schedules-12,e17cf04a9240a39f,cf8e2eebef8bb6f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:40:37.289 INFO 21062 --- [ocp-schedules-12,7f4495b815ab5848,8eb55c17778c081e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:40:37.290 WARN 21062 --- [ocp-async-7,0ac73a918861c714,8b879d22d55f05f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:37.293 INFO 21062 --- [ocp-schedules-12,7f4495b815ab5848,8eb55c17778c081e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:37.323 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:40:37.323 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:40:37.324 INFO 21062 --- [ocp-async-4,7a2147b07b31a6b7,5770a1dee3a5bc3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:38.122 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:38.123 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:38.504 INFO 21062 --- [ocp-schedules-13,13c3c0e6a7e8f9ea,1d30e8cf51391c98] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:40:38.508 INFO 21062 --- [ocp-schedules-19,c06df9741fb3da38,bfeef881c92e544c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:40:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:40:38.590097+08:00, lastProcessTime=2024-11-04T14:40:28.589830+08:00 2024-11-04 14:40:38.677 INFO 21062 --- [ocp-async-1,4f2cc7fca83b1d25,a709795b5191b8b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:40:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:40:38.590097+08:00 2024-11-04 14:40:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3d575be4d35bcd84,b988626d18b39e73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:39.096 INFO 21062 --- [ocp-schedules-21,e119d570a6b4dea8,133ca2e6a33c902d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:40:39.099 INFO 21062 --- [ocp-schedules-21,e119d570a6b4dea8,133ca2e6a33c902d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:39.100 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:40:39.290 WARN 21062 --- [ocp-async-8,33e49171980c322c,0e0beb1fbf84bcf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:39.324 INFO 21062 --- [ocp-async-6,dd93b27d9e24ead1,19ca34f367197c93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:40:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:40:40.338 INFO 21062 --- [ocp-schedules-24,9002146e706c29cd,553d1613879b0465] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:40:40.343 INFO 21062 --- [ocp-schedules-24,9002146e706c29cd,553d1613879b0465] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:40.430 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:40:40.432 ERROR 21062 --- [ocp-schedules-24,9002146e706c29cd,553d1613879b0465] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:40:40.432 INFO 21062 --- [ocp-schedules-24,9002146e706c29cd,553d1613879b0465] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:40:40.815 INFO 21062 --- [ocp-schedules-25,143d1d24b24bad52,39d116c2df92658d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:40:41.124 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:41.124 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:41.290 WARN 21062 --- [ocp-async-1,fc046eb3ac14201a,a77bebde7207e8c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:41.324 INFO 21062 --- [ocp-async-7,63c007823f208bb2,f77dcf0ec64afb76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:40:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:40:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:40:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:40:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:40:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:40:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2cc53c38fb07b810,4c56766ff2971573] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:43.291 WARN 21062 --- [ocp-async-7,0f4e815c9dd1c3c3,29a12330aefabc50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:43.324 INFO 21062 --- [ocp-async-5,e2f149d9c761e1e0,6776d3eeb77ec597] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:44.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:44.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:44.983 INFO 21062 --- [prometheus-minute-schedule-1,51bee4461425cc84,04516e45b8022edb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:45.291 WARN 21062 --- [ocp-async-5,364fa841c6f36c34,e74879d0de3b93b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:45.325 INFO 21062 --- [ocp-async-3,67d9de041fd78065,c8d7b02540e21022] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:46.819 INFO 21062 --- [ocp-schedules-30,69a02b90de1f1d27,dfc25d9589a15b19] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:40:47.125 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:47.126 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:47.291 WARN 21062 --- [ocp-async-3,8f8b7fc1ba9e198b,d0bf331c3ba159a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:47.325 INFO 21062 --- [ocp-async-9,a23a454dfe14cf75,3236c1799dba0819] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:47.983 INFO 21062 --- [prometheus-minute-schedule-1,50e9af7cc0fe85f5,aa784b68ebdd3e75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:40:48.591200+08:00, lastProcessTime=2024-11-04T14:40:38.590097+08:00 2024-11-04 14:40:48.677 INFO 21062 --- [ocp-async-5,81d844b60bb81e35,36916bea911dc816] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:40:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:40:48.591200+08:00 2024-11-04 14:40:49.100 INFO 21062 --- [ocp-schedules-21,e119d570a6b4dea8,133ca2e6a33c902d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:40:49.291 WARN 21062 --- [ocp-async-7,524c8c626239f02a,6966f2784ddfdda4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:49.325 INFO 21062 --- [ocp-async-5,3c70daa1fa40b860,7bd8e720b8a11cea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:50.126 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:50.127 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:50.819 INFO 21062 --- [ocp-schedules-29,cf5cde37b51f85d4,6571010afe6b8ccf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:40:50.983 INFO 21062 --- [prometheus-minute-schedule-1,3ebd34bae0f6c3e0,44009e26a435cfe0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:51.292 WARN 21062 --- [ocp-async-5,b619edb0a2557615,00455257faf80d9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:51.325 INFO 21062 --- [ocp-async-3,70c47358276a9e1c,efd597bce7e6ec70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:51.932 INFO 21062 --- [ocp-schedules-22,ac93cfb4817cbeaf,fdf6178b6be78f72] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:40:51.937 INFO 21062 --- [ocp-schedules-22,ac93cfb4817cbeaf,fdf6178b6be78f72] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:40:51.938 INFO 21062 --- [ocp-schedules-22,ac93cfb4817cbeaf,fdf6178b6be78f72] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:40:53.127 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:53.128 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:53.292 WARN 21062 --- [ocp-async-3,2261fa76bb93569c,031716157bf2efb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:53.325 INFO 21062 --- [ocp-async-9,0d731ba902ec41ba,8507de11464ad247] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:53.983 INFO 21062 --- [prometheus-minute-schedule-1,aa65208342a90066,4d883b6148ebe254] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:55.292 WARN 21062 --- [ocp-async-9,91bc9fe0736b4eb1,546453f5a756dfd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:55.325 INFO 21062 --- [ocp-async-4,79199057e99efebe,cc8b3a8a4d660b4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:56.128 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:56.129 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:56.843 INFO 21062 --- [ocp-schedules-16,751fae7057969758,6a175f4133338341] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:40:56.983 INFO 21062 --- [prometheus-minute-schedule-1,87272f48fa686fb6,548dc8d0e21d4a0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:40:57.292 WARN 21062 --- [ocp-async-4,a10badf0a4edb4b4,1ed710c366036abb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:57.326 INFO 21062 --- [ocp-async-2,6bb76b4c1c2be1fb,ecd9e747a212cf96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:58.499 INFO 21062 --- [ocp-schedules-4,cc55d5e2e5f23979,9e1fdcf1951ec02d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:40:58.500 INFO 21062 --- [ocp-schedules-5,650040468f973ca7,09a306f0ce2b6af7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:40:58.502 INFO 21062 --- [ocp-schedules-26,1e89ac5d46cdce37,52becbe002574f7d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:40:58.504 INFO 21062 --- [ocp-schedules-4,cc55d5e2e5f23979,9e1fdcf1951ec02d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:58.504 INFO 21062 --- [ocp-schedules-5,650040468f973ca7,09a306f0ce2b6af7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:40:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:40:58.505 INFO 21062 --- [ocp-schedules-15,f9b1cd8e61ee2f28,744ad0aa9129089c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:40:58.509 INFO 21062 --- [ocp-schedules-29,a934c4dc38d84eee,67c5482d336ed18d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:40:58.510 INFO 21062 --- [ocp-schedules-29,a934c4dc38d84eee,67c5482d336ed18d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:40:58.510 INFO 21062 --- [ocp-schedules-25,25353636600eaff2,ed46a894623dcfba] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:40:58.510 INFO 21062 --- [ocp-schedules-25,25353636600eaff2,ed46a894623dcfba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:40:58.510 INFO 21062 --- [ocp-schedules-17,a1b8ca8d7f641c5b,7fd053631a990e0c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:40:58.510 INFO 21062 --- [ocp-schedules-17,a1b8ca8d7f641c5b,7fd053631a990e0c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:40:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:40:58.590868+08:00, lastProcessTime=2024-11-04T14:40:48.591200+08:00 2024-11-04 14:40:58.678 INFO 21062 --- [ocp-async-9,5b28a0f45d5fbd36,fb1066d6d1835683] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:40:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:40:58.590868+08:00 2024-11-04 14:40:59.129 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:40:59.130 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:40:59.261 INFO 21062 --- [ocp-schedules-16,7883b0616b200535,4d1706c88e97eea7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:40:59.292 WARN 21062 --- [ocp-async-3,52df4683c0c1d9d7,e0666e3b3f4a6c46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:40:59.326 INFO 21062 --- [ocp-async-9,9c8551538b14d1fe,068b633e2e6b6b3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:40:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a9c8fdf5dd90a0ff,3171f2f334cfca82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702460, interval=60 2024-11-04 14:41:00.823 INFO 21062 --- [ocp-schedules-21,62634b772e23b5d4,232f88b23a4a3751] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:41:01.293 WARN 21062 --- [ocp-async-9,a36bea1a41bdde32,a228a5e29ccafc73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:01.326 INFO 21062 --- [ocp-async-4,dec2fb0db67ab388,97b99bca9680a736] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:02.130 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:02.131 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:02.983 INFO 21062 --- [prometheus-minute-schedule-1,aeeb8966408783d7,ae2b4af977a5dba8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:03.293 WARN 21062 --- [ocp-async-4,fde64f4dafa02c38,83c754e886201e5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:03.326 INFO 21062 --- [ocp-async-2,23d34fe33ba2358e,486663bf47b0fb22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:05.131 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:05.132 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:05.293 WARN 21062 --- [ocp-async-2,89236326f741d755,edc1e1b22a34e7d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:05.326 INFO 21062 --- [ocp-async-1,049ea87e72d7210d,bee2dd6e98283c86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f03d0537212d3ebf,04fe61bccf108b6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:06.889 INFO 21062 --- [ocp-schedules-30,d2c19679daa6c4b4,6d8e7773c04df67d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:41:07.293 WARN 21062 --- [ocp-async-1,1c47f748d3451ef7,d2053e5ffaaed12a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:07.323 INFO 21062 --- [ocp-schedules-12,7f4495b815ab5848,8eb55c17778c081e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:41:07.323 INFO 21062 --- [ocp-schedules-12,0dbf3da6ade36c0c,504c6d4ee15c0ef6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:41:07.327 INFO 21062 --- [ocp-async-7,2f76d9cd3ed4b1ae,e11049ac6ba07131] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:07.328 INFO 21062 --- [ocp-schedules-12,0dbf3da6ade36c0c,504c6d4ee15c0ef6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:07.358 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:41:07.358 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:41:08.132 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:08.133 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:08.505 INFO 21062 --- [ocp-schedules-5,650040468f973ca7,09a306f0ce2b6af7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:41:08.505 INFO 21062 --- [ocp-schedules-4,cc55d5e2e5f23979,9e1fdcf1951ec02d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:41:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:41:08.591395+08:00, lastProcessTime=2024-11-04T14:40:58.590868+08:00 2024-11-04 14:41:08.678 INFO 21062 --- [ocp-async-2,460e5cf272203d9d,244ad2cf5a0a289a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:41:08.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:41:08.591395+08:00 2024-11-04 14:41:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b5b7e687d6b8e920,767887944633bea1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:09.293 WARN 21062 --- [ocp-async-4,8a3a05aa7ce42ef8,14e01da54aca029f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:09.327 INFO 21062 --- [ocp-async-8,566c9a5ab947ec82,a4989c8ba27b0f89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:41:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:41:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:41:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:41:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:41:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:41:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:41:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:41:10.825 INFO 21062 --- [ocp-schedules-26,4e60c28decf48987,2052cd611840bb5c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:41:11.133 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:11.133 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:11.293 WARN 21062 --- [ocp-async-8,7d0ff68e4afd1832,caeb96b0ae0c9a36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:11.327 INFO 21062 --- [ocp-async-6,20026ad8df5ae200,df86ec62a90a5968] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:11.629 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=3, affectRows=3 2024-11-04 14:41:11.629 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=3, validCount=3 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=25, affectRows=25 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=34, affectRows=34 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=4, affectRows=4 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=39, affectRows=39 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=40, affectRows=40 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=12, affectRows=12 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=4, validCount=4 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=25, validCount=25 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=12, validCount=12 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=34, validCount=34 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=39, validCount=39 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=40, validCount=40 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=47, affectRows=47 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=50, affectRows=50 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=16, affectRows=16 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=35, affectRows=35 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=16, validCount=16 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=47, validCount=47 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=25, affectRows=25 2024-11-04 14:41:11.630 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=50, validCount=50 2024-11-04 14:41:11.631 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=35, validCount=35 2024-11-04 14:41:11.631 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=24, validCount=24 2024-11-04 14:41:11.631 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702400, nodeCount=92, affectRows=92 2024-11-04 14:41:11.632 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702400, nodeCount=90, validCount=90 2024-11-04 14:41:11.940 INFO 21062 --- [ocp-schedules-28,e713875d5318b07d,11f05a68d2db6f14] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:41:11.946 INFO 21062 --- [ocp-schedules-28,e713875d5318b07d,11f05a68d2db6f14] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:41:11.946 INFO 21062 --- [ocp-schedules-28,e713875d5318b07d,11f05a68d2db6f14] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:41:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9137c37afd7ad2ff,874321311a8379b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:13.293 WARN 21062 --- [ocp-async-1,96dbdd9de389083b,a5a5b59121f3b75d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:13.327 INFO 21062 --- [ocp-async-10,efa1498ae2a12edd,f21e25223006e3db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:13.984 INFO 21062 --- [prometheus-cache-monitor1,e37586c412c55984,5aa441efbb87c67d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1644,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:41:13.984 INFO 21062 --- [prometheus-cache-monitor1,e37586c412c55984,5aa441efbb87c67d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:41:14.134 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:14.134 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:14.983 INFO 21062 --- [prometheus-minute-schedule-1,70ba694a2f47a932,05725e667097991e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:15.203 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:37:15.185, end=2024-11-04T14:39:15.2} 2024-11-04 14:41:15.215 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:41:15.215 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:41:15.294 WARN 21062 --- [ocp-async-7,bdad73611d4ac014,131c775d9b6c2418] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:15.327 INFO 21062 --- [ocp-async-5,f6d16df5a886e895,56f2dd6ed643486f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:16.916 INFO 21062 --- [ocp-schedules-7,f3eed569da866d6f,9405f2037ec7d974] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:41:17.134 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:17.135 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:17.294 WARN 21062 --- [ocp-async-2,29ed4f9f8dd5313a,2ba5455f7aff685b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:17.328 INFO 21062 --- [ocp-async-3,0627dc7f59363ebc,9fade35a00c1e0f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:17.983 INFO 21062 --- [prometheus-minute-schedule-1,39cf72b32d98f0c2,eb9aa8cb2269fa6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:41:18.590352+08:00, lastProcessTime=2024-11-04T14:41:08.591395+08:00 2024-11-04 14:41:18.678 INFO 21062 --- [ocp-async-7,7cfe3a6b7dc36487,5c0451f25443ca7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:41:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:41:18.590352+08:00 2024-11-04 14:41:19.294 WARN 21062 --- [ocp-async-1,804d005a5945dc26,e81999d4d2bf8972] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:19.328 INFO 21062 --- [ocp-async-10,7005c85d3d9c3c91,76eedcbebb03651d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:20.135 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:20.136 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:20.828 INFO 21062 --- [ocp-schedules-1,ed5a46e4177b87da,b44c83ce7b17de8c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:41:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ee8799fdc1896880,0af62b32e1f6aac5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:21.294 WARN 21062 --- [ocp-async-7,90576d17d6e99d69,064f5ab3b1ef8b1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:21.328 INFO 21062 --- [ocp-async-5,88d4e5966856ec40,6391196daccac245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:23.136 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:23.137 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:23.294 WARN 21062 --- [ocp-async-2,27c822dba257d8b4,3279c10330d793a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:23.328 INFO 21062 --- [ocp-async-3,b140dcfe8b8e0a56,700bf8231cd5408b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:23.983 INFO 21062 --- [prometheus-minute-schedule-1,debdbb3185c05bcf,bdc1ce337af64c5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:25.294 WARN 21062 --- [ocp-async-6,f9a3aca1cca91647,5899fd595e60513c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:25.328 INFO 21062 --- [ocp-async-9,0260becbdf5c2cda,cec6e3e1b543ce1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:26.137 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:26.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:26.955 INFO 21062 --- [ocp-schedules-5,a7f8383a24bc2082,3317c7c89923ce0d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:41:26.983 INFO 21062 --- [prometheus-minute-schedule-1,4b71b14fdc24c0e8,189da42017d710c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:26.984 INFO 21062 --- [prometheus-hour-schedule-1,2f63aa542f5137a5,684e948bf726221e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:41:27.294 WARN 21062 --- [ocp-async-10,6c0abef0b04c975b,d5e9c45b2c35626c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:27.329 INFO 21062 --- [ocp-async-4,32c49fd586088657,fa44eda1ead94b32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:28.498 INFO 21062 --- [ocp-schedules-7,6231bef4162d3a9c,257a88e0c6297f51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:41:28.500 INFO 21062 --- [ocp-schedules-26,e25c4cf990c79e4a,ed22845c3875d7dc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:41:28.500 INFO 21062 --- [ocp-schedules-2,ee66f49f4fc18563,100090e5bbdc00df] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:41:28.500 INFO 21062 --- [ocp-schedules-1,a877cf76641c7b63,6bb1c90baea8f4ee] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:41:28.503 INFO 21062 --- [ocp-schedules-7,6231bef4162d3a9c,257a88e0c6297f51] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:41:28.503 INFO 21062 --- [ocp-schedules-4,ab4361b2f243524b,0a46009e7bda9ca8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:41:28.503 INFO 21062 --- [ocp-schedules-2,ee66f49f4fc18563,100090e5bbdc00df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:28.504 INFO 21062 --- [ocp-schedules-1,a877cf76641c7b63,6bb1c90baea8f4ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:41:28.504 INFO 21062 --- [ocp-schedules-28,369a007eab77f2c2,0328b6705b2ad99a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:41:28.507 INFO 21062 --- [ocp-schedules-26,e25c4cf990c79e4a,ed22845c3875d7dc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:41:28.511 INFO 21062 --- [ocp-schedules-14,5fed4eea81f37dbe,9b6763f5448d3cfd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:41:28.511 INFO 21062 --- [ocp-schedules-14,5fed4eea81f37dbe,9b6763f5448d3cfd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:41:28.512 INFO 21062 --- [ocp-schedules-18,3f79dd5429dd652f,0f6bb0185a6e2dd0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:41:28.512 INFO 21062 --- [ocp-schedules-8,43308f689a259c14,9f62f2a7c6ba638a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:41:28.512 INFO 21062 --- [ocp-schedules-18,3f79dd5429dd652f,0f6bb0185a6e2dd0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:41:28.512 INFO 21062 --- [ocp-schedules-8,43308f689a259c14,9f62f2a7c6ba638a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:41:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:41:28.589078+08:00, lastProcessTime=2024-11-04T14:41:18.590352+08:00 2024-11-04 14:41:28.678 INFO 21062 --- [ocp-async-9,ded348dcb30bba9d,d7c6d65b257ecfbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:41:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:41:28.589078+08:00 2024-11-04 14:41:28.877 INFO 21062 --- [ocp-schedules-27,146e758dba38ef00,53adafe922a2cbf2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:41:29.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:29.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:29.180 INFO 21062 --- [ocp-schedules-4,455928be697441a0,cdfde76ae2e54614] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:41:29.180 INFO 21062 --- [ocp-schedules-4,455928be697441a0,cdfde76ae2e54614] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:41:29.264 INFO 21062 --- [ocp-schedules-13,a2b235507c834a67,996637b6404ea160] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:41:29.295 WARN 21062 --- [ocp-async-8,5bb0ea66c3447c46,19696acba4bdabc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:29.298 INFO 21062 --- [ocp-schedules-24,79a561844a4ac3aa,c0f06c4646cad2d0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:41:29.303 INFO 21062 --- [ocp-schedules-24,79a561844a4ac3aa,c0f06c4646cad2d0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:41:29.303 INFO 21062 --- [ocp-schedules-24,79a561844a4ac3aa,c0f06c4646cad2d0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:41:29.329 INFO 21062 --- [ocp-async-1,0140871d91a9145c,b742a2404bc50022] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:29.675 INFO 21062 --- [ocp-schedules-22,3df0ebc0c0d22bfc,25cbba3b391f4c0c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:41:29.675 INFO 21062 --- [ocp-schedules-22,3df0ebc0c0d22bfc,25cbba3b391f4c0c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:41:29.678 INFO 21062 --- [ocp-schedules-22,3df0ebc0c0d22bfc,25cbba3b391f4c0c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:41:29.678 INFO 21062 --- [ocp-schedules-22,3df0ebc0c0d22bfc,25cbba3b391f4c0c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:41:29.679 INFO 21062 --- [ocp-schedules-22,3df0ebc0c0d22bfc,25cbba3b391f4c0c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:41:29.679 INFO 21062 --- [ocp-schedules-22,3df0ebc0c0d22bfc,25cbba3b391f4c0c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:41:29.983 INFO 21062 --- [prometheus-minute-schedule-1,36b306e41d4fa6e4,c017587ac54fe9b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:41:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:41:30.832 INFO 21062 --- [ocp-schedules-26,a860efefd12ea123,13fb8665d0ef3cc0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:41:31.295 WARN 21062 --- [ocp-async-9,e9140c78de23f72d,393a7329fe78075d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:31.329 INFO 21062 --- [ocp-async-7,e1b9f81707d07b27,f03318a8c22f2be4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:31.949 INFO 21062 --- [ocp-schedules-11,a975f532f73ed38d,b602410ab0ee4c69] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:41:31.955 INFO 21062 --- [ocp-schedules-11,a975f532f73ed38d,b602410ab0ee4c69] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:41:31.955 INFO 21062 --- [ocp-schedules-11,a975f532f73ed38d,b602410ab0ee4c69] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:41:32.052 INFO 21062 --- [ocp-schedules-7,2bfb2901f2c8366b,d383bad759b43f71] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:41:32.052 INFO 21062 --- [ocp-schedules-7,2bfb2901f2c8366b,d383bad759b43f71] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:41:32.081 INFO 21062 --- [ocp-schedules-4,713b01e5b0f49b5f,d7fd725facb4adcc] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:41:32.081 INFO 21062 --- [ocp-schedules-7,2bfb2901f2c8366b,d383bad759b43f71] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:41:32.081 INFO 21062 --- [ocp-schedules-7,2bfb2901f2c8366b,d383bad759b43f71] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:41:32.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:32.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:32.269 INFO 21062 --- [ocp-schedules-22,8ccf5c044bcc4e5c,346b57774d2e75c3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:41:32.269+08:00, clusters=[1] 2024-11-04 14:41:32.270 INFO 21062 --- [ocp-schedules-22,8ccf5c044bcc4e5c,346b57774d2e75c3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:41:32.468 INFO 21062 --- [ocp-schedules-6,f64d27a3f9f1c31f,9d619974fa527657] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:41:32.468+08:00, clusters=[1] 2024-11-04 14:41:32.472 INFO 21062 --- [ocp-schedules-6,f64d27a3f9f1c31f,9d619974fa527657] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:41:32.983 INFO 21062 --- [prometheus-minute-schedule-1,ed4a15ae85235817,46fc488bbd224c1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:33.295 WARN 21062 --- [ocp-async-4,f97eb071212da3f2,31873b17c8840b3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:33.329 INFO 21062 --- [ocp-async-2,9d54f6447b383bee,2f2f68bfc123ef91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:33.700 INFO 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:41:33.700 INFO 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:41:33.700 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:41:33.700 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:41:33.710 INFO 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:41:33.711 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:41:33.711 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:41:33.712 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:41:33.712 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:41:33.712 WARN 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:41:33.713 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:41:33.713 INFO 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:41:33.713 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:41:33.714 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:41:33.714 WARN 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:41:33.714 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:41:33.722 INFO 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:41:33.722 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:41:33.722 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:41:33.733 ERROR 21062 --- [ocp-schedules-6,873d9002cff5b26d,82519a1964ddc337] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:41:35.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:35.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:35.295 WARN 21062 --- [ocp-async-3,9047672cc6b1997c,75b9657dde09653d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:35.329 INFO 21062 --- [ocp-async-6,96742ca183b1ff96,a4fbf92abaf77806] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:35.983 INFO 21062 --- [prometheus-minute-schedule-1,894441114fbab83d,7901eb2adcd8cc48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:36.981 INFO 21062 --- [ocp-schedules-22,134bb3d00995ef09,1ee64017721a86ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:41:37.295 WARN 21062 --- [ocp-async-1,392abea461c6b21c,82094e6e1b8ff700] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:37.329 INFO 21062 --- [ocp-async-10,28256aebd5f1bb67,25d3f33030c44a53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:37.358 INFO 21062 --- [ocp-schedules-12,0dbf3da6ade36c0c,504c6d4ee15c0ef6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:41:37.359 INFO 21062 --- [ocp-schedules-12,2086220fa120b423,73c04c46928c2b5c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:41:37.362 INFO 21062 --- [ocp-schedules-12,2086220fa120b423,73c04c46928c2b5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:37.391 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:41:37.391 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:41:38.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:38.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:38.505 INFO 21062 --- [ocp-schedules-2,ee66f49f4fc18563,100090e5bbdc00df] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:41:38.507 INFO 21062 --- [ocp-schedules-1,a877cf76641c7b63,6bb1c90baea8f4ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:41:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:41:38.590785+08:00, lastProcessTime=2024-11-04T14:41:28.589078+08:00 2024-11-04 14:41:38.679 INFO 21062 --- [ocp-async-3,12204b886882833e,e5ce0a3430de016d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:41:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:41:38.590785+08:00 2024-11-04 14:41:38.983 INFO 21062 --- [prometheus-minute-schedule-1,276edcfc9bc10a80,e809d8c257d24dfd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:39.296 WARN 21062 --- [ocp-async-5,db833c842293b074,5fb9f39553d0a5ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:39.329 INFO 21062 --- [ocp-async-8,0dd291e84bba017b,ae70f26b79d53732] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:41:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:41:40.433 INFO 21062 --- [ocp-schedules-14,6119f8c6f14a3de8,9777e2c0ebd21dbc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:41:40.436 INFO 21062 --- [ocp-schedules-14,6119f8c6f14a3de8,9777e2c0ebd21dbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:40.523 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:41:40.532 ERROR 21062 --- [ocp-schedules-14,6119f8c6f14a3de8,9777e2c0ebd21dbc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:41:40.532 INFO 21062 --- [ocp-schedules-14,6119f8c6f14a3de8,9777e2c0ebd21dbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:41:40.835 INFO 21062 --- [ocp-schedules-1,353cf01891f51eef,8bef96e1fe29ea8b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:41:41.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:41.143 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:41.296 WARN 21062 --- [ocp-async-3,779de481763e4ce8,bb4fd815e037d2e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:41.330 INFO 21062 --- [ocp-async-1,dfc30a9b7e0702fa,fdf4bb7927a7ea71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:41.983 INFO 21062 --- [prometheus-minute-schedule-1,ef6b3076cc227dfb,33ef519223a465f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:43.296 WARN 21062 --- [ocp-async-1,547e9200b664316e,23d758803756099c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:43.330 INFO 21062 --- [ocp-async-7,1e41b502fa3f3130,2515a4a4f0f3137a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:44.143 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:44.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:44.983 INFO 21062 --- [prometheus-minute-schedule-1,335ead9da2f4b332,aeb47edd0597ebf9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:45.296 WARN 21062 --- [ocp-async-7,45289792784fe4d5,843bf8b0d22be8ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:45.330 INFO 21062 --- [ocp-async-2,a64629bcaeefd8c2,2252b43e6698bb63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:47.024 INFO 21062 --- [ocp-schedules-1,184443dd4c27895d,93f3ce344de3ab42] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:41:47.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:47.145 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:47.296 WARN 21062 --- [ocp-async-2,401b2690452bf762,0dcbcac4c8ac88b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:47.330 INFO 21062 --- [ocp-async-6,8c854d07f600c626,3437265281c131cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:47.983 INFO 21062 --- [prometheus-minute-schedule-1,85b607023179fd23,679be43e10e60abb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:41:48.590567+08:00, lastProcessTime=2024-11-04T14:41:38.590785+08:00 2024-11-04 14:41:48.679 INFO 21062 --- [ocp-async-7,467d711a0047de72,482db758bdac8d96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:41:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:41:48.590567+08:00 2024-11-04 14:41:49.100 INFO 21062 --- [ocp-schedules-6,41ecb4cfbc3f1d38,ac07adf181f4fbe9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:41:49.104 INFO 21062 --- [ocp-schedules-6,41ecb4cfbc3f1d38,ac07adf181f4fbe9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:49.105 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:41:49.297 WARN 21062 --- [ocp-async-1,6688ccd49059bfba,3e8efe894c94da06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:49.330 INFO 21062 --- [ocp-async-7,58627ca4f6bfebde,2200a4455230c57e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:50.145 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:50.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:50.838 INFO 21062 --- [ocp-schedules-1,b9a2e9bf086ad061,ed644e4907880466] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:41:50.983 INFO 21062 --- [prometheus-minute-schedule-1,432e80419e88d07f,7c995f1e6ccbf1dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:51.297 WARN 21062 --- [ocp-async-7,15fdaadd7af7163c,704cf282e145fccf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:51.331 INFO 21062 --- [ocp-async-2,4cd7fe4c1bc1658b,c036a04835e88ddb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:51.957 INFO 21062 --- [ocp-schedules-22,c0e4b81bf05fc71d,d76b6d229255837a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:41:51.962 INFO 21062 --- [ocp-schedules-22,c0e4b81bf05fc71d,d76b6d229255837a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:41:51.962 INFO 21062 --- [ocp-schedules-22,c0e4b81bf05fc71d,d76b6d229255837a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:41:53.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:53.147 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:53.297 WARN 21062 --- [ocp-async-2,9678d7856e8ea62e,300209fcd812ffc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:53.331 INFO 21062 --- [ocp-async-6,ca5e2cb0f978fb5a,8a148684e800ee4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:53.983 INFO 21062 --- [prometheus-minute-schedule-1,abf7fcd398481c7f,33ca2cc5481b9a87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:55.297 WARN 21062 --- [ocp-async-6,26b2d0973c4c3da4,4ea4f7909e8c3673] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:55.331 INFO 21062 --- [ocp-async-10,e6a8211c27599434,34cf369c1d094def] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:56.147 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:56.148 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:56.983 INFO 21062 --- [prometheus-minute-schedule-1,23eb82b199b0a03c,49fc86dedecb4b97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:41:57.050 INFO 21062 --- [ocp-schedules-11,d51f3c0e478c350d,c71c06ae58ba16ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:41:57.297 WARN 21062 --- [ocp-async-10,e757e1ad8bf41344,b87bf56401da5599] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:57.331 INFO 21062 --- [ocp-async-4,7583f93dd130e3af,b2a0dd31ae80bd48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:58.500 INFO 21062 --- [ocp-schedules-27,532a592d5d6e997b,463b4ca01a8e69ec] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:41:58.500 INFO 21062 --- [ocp-schedules-30,16a15ba5e8f9e825,fbce3cc7692572c2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:41:58.503 INFO 21062 --- [ocp-schedules-25,5eaa0581ec52bed5,fcc264446d23590d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:41:58.503 INFO 21062 --- [ocp-schedules-27,532a592d5d6e997b,463b4ca01a8e69ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:41:58.504 INFO 21062 --- [ocp-schedules-30,16a15ba5e8f9e825,fbce3cc7692572c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:41:58.506 INFO 21062 --- [ocp-schedules-28,16a45fc4fbf79b1c,c87b2bb1a258499b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:41:58.510 INFO 21062 --- [ocp-schedules-24,7857b2a7dcc0b25a,e7beac46a4223d67] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:41:58.510 INFO 21062 --- [ocp-schedules-24,7857b2a7dcc0b25a,e7beac46a4223d67] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:41:58.510 INFO 21062 --- [ocp-schedules-16,741dade738bcc139,5ebc59c85898d247] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:41:58.510 INFO 21062 --- [ocp-schedules-16,741dade738bcc139,5ebc59c85898d247] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:41:58.510 INFO 21062 --- [ocp-schedules-11,c6e8c69b01d4d482,707082be07f58b9c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:41:58.510 INFO 21062 --- [ocp-schedules-11,c6e8c69b01d4d482,707082be07f58b9c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:41:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:41:58.590208+08:00, lastProcessTime=2024-11-04T14:41:48.590567+08:00 2024-11-04 14:41:58.679 INFO 21062 --- [ocp-async-6,28d768a23612c234,08ffca95dfb1b0b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:41:58.807 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:41:58.590208+08:00 2024-11-04 14:41:59.105 INFO 21062 --- [ocp-schedules-6,41ecb4cfbc3f1d38,ac07adf181f4fbe9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:41:59.148 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:41:59.149 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:41:59.267 INFO 21062 --- [ocp-schedules-20,95cf0b7758735b33,1bb4f7e8870189cd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:41:59.298 WARN 21062 --- [ocp-async-2,f9c2b3960c6926b4,91cf632ec36b4a53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:41:59.331 INFO 21062 --- [ocp-async-6,5a9dec63cb874fe4,4d5b4dcd782343f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:41:59.983 INFO 21062 --- [prometheus-minute-schedule-1,75637c65b99ec5d3,cd7f377ac52899e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702520, interval=60 2024-11-04 14:42:00.841 INFO 21062 --- [ocp-schedules-17,47c7069ccfd08675,37841d15d07d7490] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:42:01.298 WARN 21062 --- [ocp-async-6,3ab7cc89a1d0c90e,6f6eac2da3b1189e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:01.331 INFO 21062 --- [ocp-async-10,83e3ed10bfc4b791,352c9b7beb2cd154] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:02.149 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:02.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:02.983 INFO 21062 --- [prometheus-minute-schedule-1,665a7dcb84f4f562,357743ad71b853c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:03.298 WARN 21062 --- [ocp-async-10,db7015933bf0388a,d2638c7722c48cce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:03.331 INFO 21062 --- [ocp-async-5,d32f003ecf55d6d7,ec7227cf9ff59d90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:05.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:05.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:05.298 WARN 21062 --- [ocp-async-5,76f71a3f5acd0247,f82c9cbbf1832a81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:05.332 INFO 21062 --- [ocp-async-8,4528a05196d1ac32,b5eb6accc9f53888] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:05.983 INFO 21062 --- [prometheus-minute-schedule-1,4c695c9b4d072b47,d7125ab0cf8952db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:07.097 INFO 21062 --- [ocp-schedules-22,1d1ffe8098bec8d4,01b5d3d76347954a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:42:07.298 WARN 21062 --- [ocp-async-8,4383533602a429b4,043740fbe463da06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:07.332 INFO 21062 --- [ocp-async-1,ab400118ae3bd542,8add4e447aa09f57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:07.392 INFO 21062 --- [ocp-schedules-12,2086220fa120b423,73c04c46928c2b5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:42:07.392 INFO 21062 --- [ocp-schedules-12,00ebedd9f3503f3a,31f291d55183808b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:42:07.396 INFO 21062 --- [ocp-schedules-12,00ebedd9f3503f3a,31f291d55183808b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:07.426 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:42:07.426 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:42:08.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:08.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:08.504 INFO 21062 --- [ocp-schedules-27,532a592d5d6e997b,463b4ca01a8e69ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:42:08.505 INFO 21062 --- [ocp-schedules-30,16a15ba5e8f9e825,fbce3cc7692572c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:42:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:42:08.590967+08:00, lastProcessTime=2024-11-04T14:41:58.590208+08:00 2024-11-04 14:42:08.679 INFO 21062 --- [ocp-async-5,f99bae51dec879de,84e6fe2255f11c05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:42:08.778 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:42:08.590967+08:00 2024-11-04 14:42:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8112204dfd2fbf4b,bceecd15644d2fe0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:09.299 WARN 21062 --- [ocp-async-10,7accff4d8190cec7,0ebb23cb1a7453f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:09.332 INFO 21062 --- [ocp-async-4,c22b266ed6c64b21,b8ec08c78677f03d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:42:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:42:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:42:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:42:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:42:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:42:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:42:10.327 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:42:10.844 INFO 21062 --- [ocp-schedules-1,75f5ab2ae1232260,1be6c0cd4bf6e640] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:42:11.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:11.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:11.299 WARN 21062 --- [ocp-async-4,de75b3cf5fb9198f,7cdb170bd09404e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:11.332 INFO 21062 --- [ocp-async-3,e3f93ead9ebc5483,477b1bd23844f604] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:11.643 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702460, nodeCount=422, affectRows=422 2024-11-04 14:42:11.645 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702460, nodeCount=419, validCount=419 2024-11-04 14:42:11.964 INFO 21062 --- [ocp-schedules-14,28711d624fc05bf4,02bc842a230e891f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:42:11.970 INFO 21062 --- [ocp-schedules-14,28711d624fc05bf4,02bc842a230e891f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:42:11.970 INFO 21062 --- [ocp-schedules-14,28711d624fc05bf4,02bc842a230e891f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:42:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8afd0fd75b373588,552f19ad64d21779] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:13.299 WARN 21062 --- [ocp-async-8,fd002a3fe63859a3,0ff017d59f42ea26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:13.333 INFO 21062 --- [ocp-async-9,1e0cd3ccb5bdc5d6,d4310608e8b6ea1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:13.984 INFO 21062 --- [prometheus-cache-monitor1,cd9b1e22d8189861,885ae06091f73399] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1670,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:42:13.984 INFO 21062 --- [prometheus-cache-monitor1,cd9b1e22d8189861,885ae06091f73399] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:42:14.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:14.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:14.983 INFO 21062 --- [prometheus-minute-schedule-1,6023e54b0c38b4f7,bc9203cf7322c847] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:15.299 WARN 21062 --- [ocp-async-1,37011e41390a980d,9ca28ba1e88052fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:15.333 INFO 21062 --- [ocp-async-7,eab003d1e0279dd4,a4867f3c6cfa6b87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:17.127 INFO 21062 --- [ocp-schedules-30,0308bc366e0bab6c,574893f7c5804988] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:42:17.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:17.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:17.299 WARN 21062 --- [ocp-async-5,d8612bf2e947622d,f9d6e3dc3438b48c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:17.333 INFO 21062 --- [ocp-async-2,47cd1d03d2fc251c,e23cb5694867c702] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:17.983 INFO 21062 --- [prometheus-minute-schedule-1,f2640d8129ab0ddc,ae20811518cb89ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:42:18.591187+08:00, lastProcessTime=2024-11-04T14:42:08.590967+08:00 2024-11-04 14:42:18.679 INFO 21062 --- [ocp-async-1,f5a257c22a1907f0,8e503ddacf73b81d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:42:18.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:42:18.591187+08:00 2024-11-04 14:42:19.300 WARN 21062 --- [ocp-async-8,70ff01b20a554502,db3a46eb149372d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:19.333 INFO 21062 --- [ocp-async-9,83761d3a38a17f10,15df020f0f4eb1c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:20.156 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:20.156 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:20.847 INFO 21062 --- [ocp-schedules-16,5165a24c4aa3e9fe,df91f511454ee537] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:42:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e2a4e41965b9a23d,2f6e885bb0c0a20f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:21.300 WARN 21062 --- [ocp-async-1,26b53ffa5b75a055,990b4384675c7956] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:21.333 INFO 21062 --- [ocp-async-7,845d1ebd6c5aa461,ac7ec174c3f7e2fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:23.156 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:23.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:23.300 WARN 21062 --- [ocp-async-5,8c15d987dbdbab8e,64facb21b591f25d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:23.333 INFO 21062 --- [ocp-async-2,76485309a02930e1,ab876401dcbfd0e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:23.983 INFO 21062 --- [prometheus-minute-schedule-1,2d578f26bc3f257d,a4a4a3866aff7a2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:25.300 WARN 21062 --- [ocp-async-3,a4a81abc02cee56d,bb424a0091cd6804] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:25.334 INFO 21062 --- [ocp-async-6,125d0f7dc09ae041,f68164d76dc14f97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:26.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:26.159 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9ebcf3eb1f87c8d2,f75a8b9637368e2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e61b0d18239984b2,78124c80ac5bba45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:42:27.170 INFO 21062 --- [ocp-schedules-10,9186ca67e0fc661f,6c7d2e4cf268b9aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:42:27.300 WARN 21062 --- [ocp-async-9,61eebc42c9a06813,37a3493f723dbbdc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:27.334 INFO 21062 --- [ocp-async-10,2c3b63633bd00135,1d841b6530a857f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:28.498 INFO 21062 --- [ocp-schedules-10,783ee30d06c3c2c7,3aa57f9c1a8c1547] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:42:28.499 INFO 21062 --- [ocp-schedules-25,e1b3de5b853db48b,63a73a7c501af27c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:42:28.499 INFO 21062 --- [ocp-schedules-21,ca10a0b222d845ca,d788ae8a8bb8f52c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:42:28.500 INFO 21062 --- [ocp-schedules-1,ea5ad55c3541bcc6,e70f3261ff5e7a6f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:42:28.502 INFO 21062 --- [ocp-schedules-10,783ee30d06c3c2c7,3aa57f9c1a8c1547] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:42:28.502 INFO 21062 --- [ocp-schedules-14,414b4171c8175a6b,2331caed4ced74b3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:42:28.503 INFO 21062 --- [ocp-schedules-25,e1b3de5b853db48b,63a73a7c501af27c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:42:28.504 INFO 21062 --- [ocp-schedules-1,ea5ad55c3541bcc6,e70f3261ff5e7a6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:28.505 INFO 21062 --- [ocp-schedules-11,f7976f08c4beb2a3,9600272a59467bf3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:42:28.505 INFO 21062 --- [ocp-schedules-8,21af2435a222e88f,3a783f2071439b42] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:42:28.505 INFO 21062 --- [ocp-schedules-21,ca10a0b222d845ca,d788ae8a8bb8f52c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:42:28.508 INFO 21062 --- [ocp-schedules-8,21af2435a222e88f,3a783f2071439b42] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:28.510 INFO 21062 --- [ocp-schedules-20,317934de56809ce3,0c0f96a3c47ca959] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:42:28.510 INFO 21062 --- [ocp-schedules-29,c0b154b75442ccd8,99b48a2bf7968f2e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:42:28.510 INFO 21062 --- [ocp-schedules-20,317934de56809ce3,0c0f96a3c47ca959] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:42:28.510 INFO 21062 --- [ocp-schedules-29,c0b154b75442ccd8,99b48a2bf7968f2e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:42:28.512 INFO 21062 --- [ocp-schedules-6,d6cf93c6151da9a4,b953629968653e90] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:42:28.512 INFO 21062 --- [ocp-schedules-6,d6cf93c6151da9a4,b953629968653e90] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:42:28.538 INFO 21062 --- [ocp-schedules-22,10d571d502642cd2,db863ea0ae8bf8df] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:42:28.538 INFO 21062 --- [ocp-schedules-22,10d571d502642cd2,db863ea0ae8bf8df] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:42:28.580 INFO 21062 --- [ocp-schedules-18,0e805724f930026a,185d43d21cddb1fc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1670,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:42:28.580 INFO 21062 --- [ocp-schedules-18,0e805724f930026a,185d43d21cddb1fc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:42:28.580 INFO 21062 --- [pool-12-thread-9,d7254a3202ca8495,c2950e4e8a859cb5] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:42:28.580 INFO 21062 --- [pool-12-thread-11,d7254a3202ca8495,f610d37fe16af843] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:42:28.580 INFO 21062 --- [pool-12-thread-11,d7254a3202ca8495,f610d37fe16af843] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:42:28.581 INFO 21062 --- [pool-12-thread-12,d7254a3202ca8495,d2130144799f71c2] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:42:28.581 INFO 21062 --- [pool-12-thread-10,d7254a3202ca8495,892b05508c9ac0c4] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:42:28.583 INFO 21062 --- [pool-12-thread-11,d7254a3202ca8495,f610d37fe16af843] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:42:28.584 INFO 21062 --- [pool-12-thread-9,d7254a3202ca8495,c2950e4e8a859cb5] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:42:28.584 INFO 21062 --- [pool-12-thread-9,d7254a3202ca8495,c2950e4e8a859cb5] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:42:28.586 INFO 21062 --- [pool-12-thread-11,d7254a3202ca8495,f610d37fe16af843] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:42:28.586 INFO 21062 --- [ForkJoinPool-11-worker-8,d7254a3202ca8495,8f616baf0bde1ec8] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:42:28.586 INFO 21062 --- [ForkJoinPool-11-worker-8,d7254a3202ca8495,8f616baf0bde1ec8] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:42:28.586 INFO 21062 --- [ForkJoinPool-11-worker-8,d7254a3202ca8495,8f616baf0bde1ec8] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:42:28.586 INFO 21062 --- [ForkJoinPool-11-worker-15,d7254a3202ca8495,fd61e03bb8fc4e2c] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:42:28.586 INFO 21062 --- [pool-12-thread-12,d7254a3202ca8495,d2130144799f71c2] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:42:28.586 INFO 21062 --- [pool-12-thread-10,d7254a3202ca8495,892b05508c9ac0c4] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:42:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:42:28.588582+08:00, lastProcessTime=2024-11-04T14:42:18.591187+08:00 2024-11-04 14:42:28.639 INFO 21062 --- [ocp-schedules-4,5d37298f91ce9488,b34ad7016a506764] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:42:28.640 INFO 21062 --- [ocp-schedules-4,5d37298f91ce9488,b34ad7016a506764] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:42:28.680 INFO 21062 --- [ocp-async-6,9d80bd47843f33c7,ba1dcb54620404d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:42:28.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:42:28.588582+08:00 2024-11-04 14:42:28.851 INFO 21062 --- [ocp-schedules-21,0e6ff7dc6c7974f5,80cba7584fdf8b05] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:42:28.851 INFO 21062 --- [ocp-schedules-21,0e6ff7dc6c7974f5,80cba7584fdf8b05] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:42:28.879 INFO 21062 --- [ocp-schedules-14,096ca42ceacc471f,39545c3a537e7642] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 14:42:28.879 INFO 21062 --- [ocp-schedules-14,096ca42ceacc471f,39545c3a537e7642] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 14:42:28.879 INFO 21062 --- [ocp-schedules-14,096ca42ceacc471f,39545c3a537e7642] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=422 2024-11-04 14:42:28.880 INFO 21062 --- [ocp-schedules-29,ffd26326b8b200f2,bf225a77f41f9368] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:42:28.907 INFO 21062 --- [ocp-schedules-14,096ca42ceacc471f,39545c3a537e7642] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=422 2024-11-04 14:42:29.093 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:42:29.112 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:42:29.113 WARN 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:42:29.114 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:42:29.115 WARN 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:42:29.122 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:42:29.123 WARN 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:42:29.124 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:42:29.125 WARN 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:42:29.131 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:42:29.132 WARN 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:42:29.133 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:42:29.134 WARN 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:42:29.140 INFO 21062 --- [ocp-schedules-30,b5cdb7a5151cd758,084bc0a8a1f4440b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 14:42:29.140 INFO 21062 --- [ocp-schedules-22,affa903fa57a9d96,1945d6c091266aa9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:42:29.143 INFO 21062 --- [ocp-schedules-30,b5cdb7a5151cd758,084bc0a8a1f4440b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:29.159 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:29.160 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:29.178 ERROR 21062 --- [ocp-schedules-30,b5cdb7a5151cd758,084bc0a8a1f4440b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:42:29.178 INFO 21062 --- [ocp-schedules-30,b5cdb7a5151cd758,084bc0a8a1f4440b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 14:42:29.184 INFO 21062 --- [ocp-schedules-3,a02a4c2a2db0d798,653c909a7157c60f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:42:29.184 INFO 21062 --- [ocp-schedules-3,a02a4c2a2db0d798,653c909a7157c60f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:42:29.270 INFO 21062 --- [ocp-schedules-17,9b087aabbedde40e,df8e4deaad8c5d09] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:42:29.300 WARN 21062 --- [ocp-async-4,02e75bc315e9df8e,d2df73e3da67ad79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:29.303 INFO 21062 --- [ocp-schedules-23,13b4bad6772e2893,ebdfcb1485bd0f8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:42:29.309 INFO 21062 --- [ocp-schedules-23,13b4bad6772e2893,ebdfcb1485bd0f8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:42:29.309 INFO 21062 --- [ocp-schedules-23,13b4bad6772e2893,ebdfcb1485bd0f8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:42:29.334 INFO 21062 --- [ocp-async-8,b09bbf28bbb536ff,0d0914bf7543abf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:29.681 INFO 21062 --- [ocp-schedules-27,a88244918a83ea1e,a64e7f628b6a4c10] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:42:29.681 INFO 21062 --- [ocp-schedules-27,a88244918a83ea1e,a64e7f628b6a4c10] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:42:29.682 INFO 21062 --- [ocp-schedules-27,a88244918a83ea1e,a64e7f628b6a4c10] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:42:29.682 INFO 21062 --- [ocp-schedules-27,a88244918a83ea1e,a64e7f628b6a4c10] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:42:29.683 INFO 21062 --- [ocp-schedules-27,a88244918a83ea1e,a64e7f628b6a4c10] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:42:29.683 INFO 21062 --- [ocp-schedules-27,a88244918a83ea1e,a64e7f628b6a4c10] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:42:29.975 INFO 21062 --- [ocp-schedules-28,1d587fdffec63b75,615049faf9a4f234] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:42:29.975 INFO 21062 --- [ocp-schedules-28,1d587fdffec63b75,615049faf9a4f234] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:42:29.983 INFO 21062 --- [prometheus-minute-schedule-1,f0dd4c9eea85930e,c009ea608247e567] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:30.003 INFO 21062 --- [pool-45-thread-1,1d587fdffec63b75,c4dd8c6851c96d44] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:42:30.031 WARN 21062 --- [ocp-schedules-28,1d587fdffec63b75,615049faf9a4f234] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:42:30.031 INFO 21062 --- [ocp-schedules-28,1d587fdffec63b75,615049faf9a4f234] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:42:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:42:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:42:30.851 INFO 21062 --- [ocp-schedules-18,3ab94135ff797175,ebee8b22d9ba4bfd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:42:31.300 WARN 21062 --- [ocp-async-6,77873949c3a4fb8a,2a5bf6bba6420427] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:31.334 INFO 21062 --- [ocp-async-1,04e5103baf1b0ccd,f68ed3087d8cc3ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:31.972 INFO 21062 --- [ocp-schedules-22,f111d3f9a4a5a115,86d613b5ec6c4043] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:42:31.978 INFO 21062 --- [ocp-schedules-22,f111d3f9a4a5a115,86d613b5ec6c4043] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:42:31.978 INFO 21062 --- [ocp-schedules-22,f111d3f9a4a5a115,86d613b5ec6c4043] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:42:32.082 INFO 21062 --- [ocp-schedules-7,87e76b977380fa15,ef157aa17e621bb5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:42:32.082 INFO 21062 --- [ocp-schedules-7,87e76b977380fa15,ef157aa17e621bb5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:42:32.110 INFO 21062 --- [ocp-schedules-7,87e76b977380fa15,ef157aa17e621bb5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:42:32.110 INFO 21062 --- [ocp-schedules-7,87e76b977380fa15,ef157aa17e621bb5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:42:32.110 INFO 21062 --- [ocp-schedules-10,3481e25e6a776674,b453ce0cb7ad675a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:42:32.160 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:32.161 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:32.296 INFO 21062 --- [ocp-schedules-5,5ff591d3815151d5,5f2698624851f679] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:42:32.296+08:00, clusters=[1] 2024-11-04 14:42:32.298 INFO 21062 --- [ocp-schedules-5,5ff591d3815151d5,5f2698624851f679] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:42:32.498 INFO 21062 --- [ocp-schedules-3,9190ecab6eab8f0d,fa0d4c8c34d555db] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:42:32.498+08:00, clusters=[1] 2024-11-04 14:42:32.502 INFO 21062 --- [ocp-schedules-3,9190ecab6eab8f0d,fa0d4c8c34d555db] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:42:32.507 INFO 21062 --- [ocp-schedules-21,40c205ad6bd3d46f,b80e77e01e81aa0e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:42:32.511 INFO 21062 --- [ocp-schedules-21,40c205ad6bd3d46f,b80e77e01e81aa0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:32.566 ERROR 21062 --- [ocp-schedules-21,40c205ad6bd3d46f,b80e77e01e81aa0e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:42:32.566 INFO 21062 --- [ocp-schedules-21,40c205ad6bd3d46f,b80e77e01e81aa0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:42:32.983 INFO 21062 --- [prometheus-minute-schedule-1,61e905c5beb12f01,6edc2eccf615bc5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:33.301 WARN 21062 --- [ocp-async-10,173e26370d124414,e17599386ea35bb8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:33.334 INFO 21062 --- [ocp-async-5,4ad0ac69908aaa76,b841da78e55118bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:33.741 INFO 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:42:33.741 INFO 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:42:33.741 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:42:33.741 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:42:33.750 INFO 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:42:33.751 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:42:33.751 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:42:33.752 WARN 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:42:33.752 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:42:33.752 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:42:33.753 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:42:33.753 INFO 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:42:33.753 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:42:33.754 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:42:33.754 WARN 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:42:33.754 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:42:33.760 INFO 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:42:33.760 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:42:33.761 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:42:33.772 ERROR 21062 --- [ocp-schedules-5,117f466ae1d5c3cf,51ed78634290cb2c] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:42:35.162 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:35.163 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:35.301 WARN 21062 --- [ocp-async-2,a593831a86561e44,e9825679564745f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:35.335 INFO 21062 --- [ocp-async-3,63b61377db2f444b,b3d1fc0637ffbb89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:35.983 INFO 21062 --- [prometheus-minute-schedule-1,55023dc7557b4f0c,e68b8bd7c894879c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:37.197 INFO 21062 --- [ocp-schedules-7,118d37c64569c6ab,d9b4fa15767b75a5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:42:37.301 WARN 21062 --- [ocp-async-8,8346721ab1b7a588,4b351ed2896663e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:37.335 INFO 21062 --- [ocp-async-9,845d2abd0ed194a0,06f7f224d5c91624] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:37.426 INFO 21062 --- [ocp-schedules-12,00ebedd9f3503f3a,31f291d55183808b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:42:37.427 INFO 21062 --- [ocp-schedules-12,2647108534131497,ba14ab92c390c8bc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:42:37.430 INFO 21062 --- [ocp-schedules-12,2647108534131497,ba14ab92c390c8bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:37.460 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:42:37.460 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:42:38.163 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:38.164 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:38.504 INFO 21062 --- [ocp-schedules-25,e1b3de5b853db48b,63a73a7c501af27c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:42:38.504 INFO 21062 --- [ocp-schedules-1,ea5ad55c3541bcc6,e70f3261ff5e7a6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:42:38.508 INFO 21062 --- [ocp-schedules-8,21af2435a222e88f,3a783f2071439b42] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:42:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:42:38.590095+08:00, lastProcessTime=2024-11-04T14:42:28.588582+08:00 2024-11-04 14:42:38.680 INFO 21062 --- [ocp-async-2,f86b436c9d4c35d0,6507b6ab84a45783] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:42:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:42:38.590095+08:00 2024-11-04 14:42:38.983 INFO 21062 --- [prometheus-minute-schedule-1,58904ef5f0b07d3b,e148beade051331a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:39.301 WARN 21062 --- [ocp-async-7,74c716f8fe6b5256,9238a611445cc544] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:39.335 INFO 21062 --- [ocp-async-2,ef59a9fa2d2879b5,4d396d730911992f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:40.002 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:42:40.002 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:42:40.533 INFO 21062 --- [ocp-schedules-17,a3bbbb57d3090047,c1b6081f4f910a65] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:42:40.537 INFO 21062 --- [ocp-schedules-17,a3bbbb57d3090047,c1b6081f4f910a65] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:40.621 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:42:40.624 ERROR 21062 --- [ocp-schedules-17,a3bbbb57d3090047,c1b6081f4f910a65] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:42:40.624 INFO 21062 --- [ocp-schedules-17,a3bbbb57d3090047,c1b6081f4f910a65] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:42:40.854 INFO 21062 --- [ocp-schedules-2,cd8f64a4d654e441,a989caad31042554] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:42:41.164 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:41.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:41.301 WARN 21062 --- [ocp-async-4,cbfff5efc87e0a18,9e49db35f209c759] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:41.335 INFO 21062 --- [ocp-async-8,4a7bdd376bdaa7fb,8728d1d92789828b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:42:41.823 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:42:41.823 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:42:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a1f14f3de0b30bf2,e56552de119c7527] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:43.302 WARN 21062 --- [ocp-async-8,469b00235232b237,add0c24383d3d0bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:43.335 INFO 21062 --- [ocp-async-1,90671d729413b751,5528f08d6dfd5850] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:44.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:44.166 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:44.983 INFO 21062 --- [prometheus-minute-schedule-1,62380474fed512f3,1080043ab539b227] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:45.302 WARN 21062 --- [ocp-async-1,24f2d4563118705c,26df7940eab947d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:45.335 INFO 21062 --- [ocp-async-5,ca86705e4ea5c512,f3545729b2ae7bbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:47.166 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:47.167 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:47.238 INFO 21062 --- [ocp-schedules-10,14103f1342228d6d,0af2072549b71e09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:42:47.302 WARN 21062 --- [ocp-async-5,a975f5a02ec4883b,3eeb4bde662ef2ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:47.336 INFO 21062 --- [ocp-async-3,48dc6fb5aeb0a4e5,f4bdd5f9b3777188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:47.983 INFO 21062 --- [prometheus-minute-schedule-1,92bd55113f01d9bc,e1a4e7fdad4f679c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:42:48.591673+08:00, lastProcessTime=2024-11-04T14:42:38.590095+08:00 2024-11-04 14:42:48.680 INFO 21062 --- [ocp-async-1,429494af5b7ab44d,a4e06e5e448cf205] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:42:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:42:48.591673+08:00 2024-11-04 14:42:49.302 WARN 21062 --- [ocp-async-8,84fea3985d0b7bcc,48c93e21ed95f9b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:49.336 INFO 21062 --- [ocp-async-1,622cd83e2a0c7671,e6e2e43341f2aab1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:50.167 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:50.167 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:50.857 INFO 21062 --- [ocp-schedules-28,f192b97641ac6095,7b95c1c14db39b89] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:42:50.983 INFO 21062 --- [prometheus-minute-schedule-1,21f06aa64d7c3162,2acacf05ea4f1bfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:51.302 WARN 21062 --- [ocp-async-1,31248e82225e134c,b43800fe59d4b0a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:51.336 INFO 21062 --- [ocp-async-5,171e4c1002989b50,c3a2fdd62aaf3867] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:51.980 INFO 21062 --- [ocp-schedules-21,23a9b60504248c82,ab4d6cf138a367e0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:42:51.986 INFO 21062 --- [ocp-schedules-21,23a9b60504248c82,ab4d6cf138a367e0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:42:51.986 INFO 21062 --- [ocp-schedules-21,23a9b60504248c82,ab4d6cf138a367e0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:42:53.168 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:53.168 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:53.302 WARN 21062 --- [ocp-async-5,7659ca41d9010d90,5441e245b2d43cae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:53.336 INFO 21062 --- [ocp-async-3,68f6edf208833c2c,a9a5f76d82c29687] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e293244ec0feff7c,16b6d531715ed424] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:55.097 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:42:55.097 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:42:55.104 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:42:55.104 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:42:55.104 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:42:55.104 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:42:55.110 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:42:55.110 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:42:55.110 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:42:55.110 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:42:55.145 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:42:55.153 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:42:55.153 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:42:55.154 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:42:55.154 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:42:55.154 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:42:55.162 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:42:55.303 WARN 21062 --- [ocp-async-3,a9f1331ab05b31f8,a9329e95effcc85e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:55.336 INFO 21062 --- [ocp-async-9,00c8c29b78afad64,be7f849953d9b57c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:55.774 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:42:56.169 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:56.170 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:56.336 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.225851048S 2024-11-04 14:42:56.336 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:42:56.437 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:42:56.437 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:42:56.465 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:42:56.524 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:42:56.524 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:42:56.524 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:42:56.524 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:42:56.524 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:42:56.524 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:42:56.536 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:42:56.710 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] o.s.boot.SpringApplication : Started application in 1.606 seconds (JVM running for 7990.39) 2024-11-04 14:42:56.711 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.606653988S 2024-11-04 14:42:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7620b245a4f5c21c,69d6d3dc7176aa2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:42:57.182 INFO 21062 --- [ocp-schedules-18,da15fd6aaf0b9f49,b97cce330a12b38c] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:42:57.267 INFO 21062 --- [ocp-schedules-7,a6f8aba05312a48a,dddd3e655506447f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:42:57.303 WARN 21062 --- [ocp-async-9,2417ff18f224a0e2,ef4e64a851088e38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:57.337 INFO 21062 --- [ocp-async-7,3364828dea4ba779,429432f9902c72e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:58.500 INFO 21062 --- [ocp-schedules-22,6fda1520d5bb967f,5101114f0c4937ca] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:42:58.500 INFO 21062 --- [ocp-schedules-7,2b728629c626585f,f6928e109f15b3db] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:42:58.503 INFO 21062 --- [ocp-schedules-2,a472982c628344df,862bf174b7dc1de2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:42:58.503 INFO 21062 --- [ocp-schedules-22,6fda1520d5bb967f,5101114f0c4937ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:58.504 INFO 21062 --- [ocp-schedules-7,2b728629c626585f,f6928e109f15b3db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:42:58.505 INFO 21062 --- [ocp-schedules-30,fd21f5dfddc285ae,62e7ab1b5e00b084] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:42:58.511 INFO 21062 --- [ocp-schedules-9,ec24b2b0f7c64ab4,88f4ed85cfb98f68] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:42:58.511 INFO 21062 --- [ocp-schedules-8,f1a1c4f895790473,6295b0898fc107df] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:42:58.511 INFO 21062 --- [ocp-schedules-9,ec24b2b0f7c64ab4,88f4ed85cfb98f68] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:42:58.511 INFO 21062 --- [ocp-schedules-8,f1a1c4f895790473,6295b0898fc107df] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:42:58.512 INFO 21062 --- [ocp-schedules-15,bbb05e5916b6ecc9,ff1303cc8090d674] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:42:58.512 INFO 21062 --- [ocp-schedules-15,bbb05e5916b6ecc9,ff1303cc8090d674] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:42:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:42:58.590571+08:00, lastProcessTime=2024-11-04T14:42:48.591673+08:00 2024-11-04 14:42:58.680 INFO 21062 --- [ocp-async-3,10d2db1a26d4ce84,0d434374569072e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:42:58.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:42:58.590571+08:00 2024-11-04 14:42:59.106 INFO 21062 --- [ocp-schedules-26,2a2058f0ef40ceb1,409fd6971ebbfebd] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:42:59.110 INFO 21062 --- [ocp-schedules-26,2a2058f0ef40ceb1,409fd6971ebbfebd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:42:59.110 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:42:59.170 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:42:59.171 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:42:59.273 INFO 21062 --- [ocp-schedules-6,fb56f5fd4315e07d,799317fd341f3cc1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:42:59.303 WARN 21062 --- [ocp-async-5,a641d5227934a6ec,fabc4a0e9b2ecccb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:42:59.337 INFO 21062 --- [ocp-async-2,44b68f2173578242,199aa5347b26f494] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:42:59.983 INFO 21062 --- [prometheus-minute-schedule-1,b1d87ce7f6664bce,c3300968e4ff461f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702580, interval=60 2024-11-04 14:43:00.861 INFO 21062 --- [ocp-schedules-21,973ca79fefe70226,860f2397f1d2690f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:43:01.303 WARN 21062 --- [ocp-async-3,a096a378b073bba6,5741112f7f6e48b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:01.337 INFO 21062 --- [ocp-async-6,717946bb57905a16,f684f633e916d141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:02.171 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:02.172 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:02.983 INFO 21062 --- [prometheus-minute-schedule-1,01c30040afa340aa,c5b06a3815d8014a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:03.303 WARN 21062 --- [ocp-async-9,95d14f4dc3dc98c2,8ea1f167fb02a771] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:03.337 INFO 21062 --- [ocp-async-7,caed7b8d522ea7c8,ac2a63f7d8bc5f26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:05.172 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:05.173 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:05.303 WARN 21062 --- [ocp-async-10,521e420d2e0a9e4c,fe96f5fd8e361fd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:05.337 INFO 21062 --- [ocp-async-4,7b3498592139487f,daa363b1a5cf4280] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c2ee6d000b4f2a0d,296b1548526c5584] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:07.303 WARN 21062 --- [ocp-async-2,b98e32ba1f800ec0,4ae21ea6d2e8a649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:07.334 INFO 21062 --- [ocp-schedules-8,19fd8c332a9be0c3,eef65fe2ce570fa4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:43:07.337 INFO 21062 --- [ocp-async-8,60403da8b2ee7496,f25ac272570296a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:07.461 INFO 21062 --- [ocp-schedules-12,2647108534131497,ba14ab92c390c8bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:43:07.461 INFO 21062 --- [ocp-schedules-12,3a1eafcc62203ada,1461386a5d240916] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:43:07.466 INFO 21062 --- [ocp-schedules-12,3a1eafcc62203ada,1461386a5d240916] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:43:07.497 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:43:07.497 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:43:08.173 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:08.174 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:08.504 INFO 21062 --- [ocp-schedules-7,2b728629c626585f,f6928e109f15b3db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:43:08.504 INFO 21062 --- [ocp-schedules-22,6fda1520d5bb967f,5101114f0c4937ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:43:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:43:08.590554+08:00, lastProcessTime=2024-11-04T14:42:58.590571+08:00 2024-11-04 14:43:08.680 INFO 21062 --- [ocp-async-10,dee3a7d93b230ace,2dcb949068d6f262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:43:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:43:08.590554+08:00 2024-11-04 14:43:08.983 INFO 21062 --- [prometheus-minute-schedule-1,4b7cf612a53e0ce9,13f52550a4913acd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:09.111 INFO 21062 --- [ocp-schedules-26,2a2058f0ef40ceb1,409fd6971ebbfebd] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:43:09.304 WARN 21062 --- [ocp-async-9,94aafe68144afb3c,a776e7026ad6be47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:09.338 INFO 21062 --- [ocp-async-7,1a31c2a8c944abe2,622f1a9ed9494591] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:10.002 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:43:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:43:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:43:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:43:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:43:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:43:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:43:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:43:10.863 INFO 21062 --- [ocp-schedules-16,86acf975b724309f,6ecc3d5c2c224190] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:43:11.174 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:11.174 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:11.304 WARN 21062 --- [ocp-async-10,c07984e5ae9faa15,a0fef28ec7b50a3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:11.338 INFO 21062 --- [ocp-async-4,067f09dce90b2ad5,24d9d66b534eb2b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=3, affectRows=3 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=3, validCount=3 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=12, affectRows=12 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=14, affectRows=14 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=12, validCount=12 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=25, affectRows=25 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=14, validCount=14 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=6, affectRows=6 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=6, validCount=6 2024-11-04 14:43:11.639 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=25, validCount=25 2024-11-04 14:43:11.640 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=9, affectRows=9 2024-11-04 14:43:11.640 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=55, affectRows=55 2024-11-04 14:43:11.640 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=9, validCount=9 2024-11-04 14:43:11.640 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=55, validCount=55 2024-11-04 14:43:11.640 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=65, affectRows=65 2024-11-04 14:43:11.640 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=34, affectRows=34 2024-11-04 14:43:11.640 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=79, affectRows=79 2024-11-04 14:43:11.641 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=34, validCount=34 2024-11-04 14:43:11.641 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=65, validCount=65 2024-11-04 14:43:11.641 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=90, affectRows=90 2024-11-04 14:43:11.641 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702520, nodeCount=30, affectRows=30 2024-11-04 14:43:11.641 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=77, validCount=77 2024-11-04 14:43:11.641 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=30, validCount=30 2024-11-04 14:43:11.641 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702520, nodeCount=89, validCount=89 2024-11-04 14:43:11.983 INFO 21062 --- [prometheus-minute-schedule-1,45be4e99d15a6f9d,ec7cc54e1f81d3bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:11.987 INFO 21062 --- [ocp-schedules-15,598ecfefee7648bf,c4f4bd6775490cf6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:43:11.992 INFO 21062 --- [ocp-schedules-15,598ecfefee7648bf,c4f4bd6775490cf6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:43:11.992 INFO 21062 --- [ocp-schedules-15,598ecfefee7648bf,c4f4bd6775490cf6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:43:13.304 WARN 21062 --- [ocp-async-2,635e3a8ba5bcd4d3,96f55d0197bacd7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:13.338 INFO 21062 --- [ocp-async-8,d6c9bcc71d4f277c,60955079f2f8a1fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:13.984 INFO 21062 --- [prometheus-cache-monitor1,b3023798282d9b6c,4ccad9cef50db0ec] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:43:13.984 INFO 21062 --- [prometheus-cache-monitor1,b3023798282d9b6c,4ccad9cef50db0ec] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:43:14.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:14.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:14.324 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:37:14.305, end=2024-11-04T14:42:14.321} 2024-11-04 14:43:14.324 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:37:14.305, end=2024-11-04T14:42:14.321} 2024-11-04 14:43:14.331 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:43:14.331 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:43:14.331 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:43:14.331 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:43:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f57cf32a27db5d19,11e4314b2b76ee19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:15.226 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:39:15.2, end=2024-11-04T14:41:15.223} 2024-11-04 14:43:15.229 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:43:15.229 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:43:15.304 WARN 21062 --- [ocp-async-6,861eda332a650fee,b9cedaba0ae9e596] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:15.338 INFO 21062 --- [ocp-async-1,3379a90ecf1b7c20,787b8efb34f80e8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:17.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:17.176 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:17.304 WARN 21062 --- [ocp-async-7,3d93a6323d5d10cd,107773df3625c592] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:17.338 INFO 21062 --- [ocp-async-5,ac88f49fc1d94981,21549f84b70de484] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:17.359 INFO 21062 --- [ocp-schedules-8,ee5104bba9a69c0f,0614c60eac95f069] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:43:17.983 INFO 21062 --- [prometheus-minute-schedule-1,32e4edfbda88829f,aebea2ec644ddfb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:43:18.590986+08:00, lastProcessTime=2024-11-04T14:43:08.590554+08:00 2024-11-04 14:43:18.680 INFO 21062 --- [ocp-async-8,3bafb286cccb37db,c88a43bc1cc188bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:43:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:43:18.590986+08:00 2024-11-04 14:43:19.304 WARN 21062 --- [ocp-async-2,94d1bad2b05d75d4,f90f9a328a7e0c68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:19.338 INFO 21062 --- [ocp-async-6,5ee2503cead2aedd,d870d6147338c221] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:20.176 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:20.177 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:20.866 INFO 21062 --- [ocp-schedules-21,e95274483610af98,f9c468672ced72c9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:43:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8be268350e0b0b3d,6c9948d0b0858d23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:21.304 WARN 21062 --- [ocp-async-8,e08f0a46d9e0a854,ca20672bd8b5b41a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:21.339 INFO 21062 --- [ocp-async-1,3e2df27920256ba2,d34a1ecab1e2206c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:23.177 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:23.178 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:23.305 WARN 21062 --- [ocp-async-7,7b696326aeb9866d,8200c71235d6a0a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:23.339 INFO 21062 --- [ocp-async-5,c5fbc5858ec58e7c,5c1d7b12db8ff2dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8ca3719e4f08f002,f5a544c6124fac1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:25.305 WARN 21062 --- [ocp-async-5,8bf150379fff9d9e,8d14be6bd506b44f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:25.339 INFO 21062 --- [ocp-async-3,bc9eb3177a74c938,bb14a01b74489346] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:26.179 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:26.180 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:26.983 INFO 21062 --- [prometheus-minute-schedule-1,207fb62a55873f66,9851c757d1d87c9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f7e541bc430ee56f,d7e040493f68e842] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:43:27.305 WARN 21062 --- [ocp-async-3,e4d2e5ff1d0b6c9b,7decee4b7154ce8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:27.339 INFO 21062 --- [ocp-async-9,c13b9a357bb99ab0,f60c5c77a017280c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:27.401 INFO 21062 --- [ocp-schedules-7,572e07897d537b84,aed4b2f42957083b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:43:28.498 INFO 21062 --- [ocp-schedules-2,a44342ede51db409,0ddf0a293734b127] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:43:28.499 INFO 21062 --- [ocp-schedules-9,b73279f3ebe25b2f,06cbf601834d06b4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:43:28.500 INFO 21062 --- [ocp-schedules-27,0092aa5da9f8e1ee,60ef5e9c1719880d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:43:28.500 INFO 21062 --- [ocp-schedules-21,9c4ee2d385c6f3d5,bcbc0bb071dc3266] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:43:28.502 INFO 21062 --- [ocp-schedules-8,c578c96937dd392d,2aa2f09f27d86ec1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:43:28.503 INFO 21062 --- [ocp-schedules-2,a44342ede51db409,0ddf0a293734b127] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:43:28.503 INFO 21062 --- [ocp-schedules-27,0092aa5da9f8e1ee,60ef5e9c1719880d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:43:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:43:28.504 INFO 21062 --- [ocp-schedules-21,9c4ee2d385c6f3d5,bcbc0bb071dc3266] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:43:28.505 INFO 21062 --- [ocp-schedules-4,d0ed9eae39656864,2e7aae1e3b29b3b0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:43:28.505 INFO 21062 --- [ocp-schedules-9,b73279f3ebe25b2f,06cbf601834d06b4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:43:28.510 INFO 21062 --- [ocp-schedules-29,ce6af16635b26f58,2fc45f7558612bd1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:43:28.510 INFO 21062 --- [ocp-schedules-29,ce6af16635b26f58,2fc45f7558612bd1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:43:28.510 INFO 21062 --- [ocp-schedules-25,90075ba9aeba1091,2da739757452c03a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:43:28.510 INFO 21062 --- [ocp-schedules-1,f9b1e0167220fd20,078e3db8d1b3da82] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:43:28.510 INFO 21062 --- [ocp-schedules-25,90075ba9aeba1091,2da739757452c03a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:43:28.510 INFO 21062 --- [ocp-schedules-1,f9b1e0167220fd20,078e3db8d1b3da82] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:43:28.584 INFO 21062 --- [ocp-schedules-11,0794a761c12751b5,23d3d5392baa28c1] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 14:43:28.584 INFO 21062 --- [ocp-schedules-11,0794a761c12751b5,23d3d5392baa28c1] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 14:43:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:43:28.590084+08:00, lastProcessTime=2024-11-04T14:43:18.590986+08:00 2024-11-04 14:43:28.680 INFO 21062 --- [ocp-async-2,75fee802317bdc10,2a85f15bb4cce6aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:43:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:43:28.590084+08:00 2024-11-04 14:43:28.882 INFO 21062 --- [ocp-schedules-13,e7f1c95fb80808e9,1ead9e85f8f2566d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:43:29.180 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:29.181 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:29.189 INFO 21062 --- [ocp-schedules-25,5cd005f0d2073b7f,654dad1326949f9c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:43:29.189 INFO 21062 --- [ocp-schedules-25,5cd005f0d2073b7f,654dad1326949f9c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:43:29.276 INFO 21062 --- [ocp-schedules-2,4eb5a4b89a82403f,9465bd24d8423f61] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:43:29.305 WARN 21062 --- [ocp-async-7,ebcb870cca1b41e2,0c1fd071fe30ff27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:29.309 INFO 21062 --- [ocp-schedules-19,84f8bfae10508658,3bbd9ae7f92a6938] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:43:29.314 INFO 21062 --- [ocp-schedules-19,84f8bfae10508658,3bbd9ae7f92a6938] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:43:29.314 INFO 21062 --- [ocp-schedules-19,84f8bfae10508658,3bbd9ae7f92a6938] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:43:29.339 INFO 21062 --- [ocp-async-2,9917d7d4608623d0,ae920801cdedc2b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:29.502 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:43:29.502 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:43:29.511 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:43:29.511 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:43:29.512 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:43:29.522 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:43:29.522 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:43:29.522 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:43:29.526 INFO 21062 --- [ocp-schedules-3,5e1ac345cb360d37,34c3e30232b22ae4] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:43:29.684 INFO 21062 --- [ocp-schedules-30,f162629dd1c1e613,873f87d04b838f43] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:43:29.684 INFO 21062 --- [ocp-schedules-30,f162629dd1c1e613,873f87d04b838f43] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:43:29.692 INFO 21062 --- [ocp-schedules-30,f162629dd1c1e613,873f87d04b838f43] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:43:29.692 INFO 21062 --- [ocp-schedules-30,f162629dd1c1e613,873f87d04b838f43] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:43:29.696 INFO 21062 --- [ocp-schedules-30,f162629dd1c1e613,873f87d04b838f43] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:43:29.696 INFO 21062 --- [ocp-schedules-30,f162629dd1c1e613,873f87d04b838f43] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:43:29.983 INFO 21062 --- [prometheus-minute-schedule-1,a1e9f662dd320a12,0f476805c84f97d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:43:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:43:30.869 INFO 21062 --- [ocp-schedules-14,071f715b2421e5a9,bb3fc9340a378073] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:43:31.305 WARN 21062 --- [ocp-async-2,1b9ef85399a36ee0,f85a7470066cc938] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:31.340 INFO 21062 --- [ocp-async-8,a5a80bb55bb718fd,b6672ac710bb08f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:31.995 INFO 21062 --- [ocp-schedules-2,0e789d90e6171309,820a2642fd4efc5f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:43:32.000 INFO 21062 --- [ocp-schedules-2,0e789d90e6171309,820a2642fd4efc5f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:43:32.000 INFO 21062 --- [ocp-schedules-2,0e789d90e6171309,820a2642fd4efc5f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:43:32.110 INFO 21062 --- [ocp-schedules-28,d7cf6316d621daed,c853770a01884bcf] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:43:32.110 INFO 21062 --- [ocp-schedules-28,d7cf6316d621daed,c853770a01884bcf] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:43:32.138 INFO 21062 --- [ocp-schedules-28,d7cf6316d621daed,c853770a01884bcf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:43:32.138 INFO 21062 --- [ocp-schedules-28,d7cf6316d621daed,c853770a01884bcf] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:43:32.139 INFO 21062 --- [ocp-schedules-25,807572b8542340c1,9d3ed670cec32fe7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:43:32.181 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:32.181 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:32.324 INFO 21062 --- [ocp-schedules-24,24e3997e73cae543,02a257e58bfd9288] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:43:32.324+08:00, clusters=[1] 2024-11-04 14:43:32.326 INFO 21062 --- [ocp-schedules-24,24e3997e73cae543,02a257e58bfd9288] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:43:32.527 INFO 21062 --- [ocp-schedules-30,87f3f0a13ffbbfa5,fef1af01ea60c050] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:43:32.527+08:00, clusters=[1] 2024-11-04 14:43:32.531 INFO 21062 --- [ocp-schedules-30,87f3f0a13ffbbfa5,fef1af01ea60c050] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:43:32.983 INFO 21062 --- [prometheus-minute-schedule-1,c63761139247b37a,fdf5535b97b497e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:33.305 WARN 21062 --- [ocp-async-8,811ffb46b55b71b7,13ea54c355d7d493] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:33.340 INFO 21062 --- [ocp-async-10,3887f2f8fbbde50a,aba52eeef9beed07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:33.782 INFO 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:43:33.782 INFO 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:43:33.782 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:43:33.782 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:43:33.792 INFO 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:43:33.793 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:43:33.793 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:43:33.794 WARN 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:43:33.794 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:43:33.795 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:43:33.795 INFO 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:43:33.795 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:43:33.796 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:43:33.796 WARN 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:43:33.796 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:43:33.797 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:43:33.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:43:33.804 INFO 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:43:33.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:43:33.814 ERROR 21062 --- [ocp-schedules-20,defb008fcdec570f,649953505c4df209] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:43:35.182 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:35.182 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:35.306 WARN 21062 --- [ocp-async-10,a52ba4c1905950a6,c90b9c699b1b249d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:35.340 INFO 21062 --- [ocp-async-4,c3e80b9119c69a26,7f082f3023926af8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:35.983 INFO 21062 --- [prometheus-minute-schedule-1,dd2d8da2301f321a,6c8c872cafa3c405] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:37.306 WARN 21062 --- [ocp-async-4,86e31f07189ed787,7b4051428febbef1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:37.340 INFO 21062 --- [ocp-async-3,771dea762fc0f376,5c6a2a08a3d3c29b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:37.427 INFO 21062 --- [ocp-schedules-10,29b592ce93babe70,4a286af0471b3958] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:43:37.498 INFO 21062 --- [ocp-schedules-12,3a1eafcc62203ada,1461386a5d240916] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:43:37.498 INFO 21062 --- [ocp-schedules-12,c3b6717422ebc767,a3345e3e1bf4c13e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:43:37.501 INFO 21062 --- [ocp-schedules-12,c3b6717422ebc767,a3345e3e1bf4c13e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:43:37.531 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:43:37.531 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:43:38.182 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:38.183 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:38.504 INFO 21062 --- [ocp-schedules-27,0092aa5da9f8e1ee,60ef5e9c1719880d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:43:38.504 INFO 21062 --- [ocp-schedules-21,9c4ee2d385c6f3d5,bcbc0bb071dc3266] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:43:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:43:38.590571+08:00, lastProcessTime=2024-11-04T14:43:28.590084+08:00 2024-11-04 14:43:38.681 INFO 21062 --- [ocp-async-7,a74d8426dfdb9688,af77035f4b3e7f64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:43:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:43:38.590571+08:00 2024-11-04 14:43:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a8a110bff5e4981a,c3d331290ecfa2b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:39.306 WARN 21062 --- [ocp-async-1,c3c98de9dc79d767,2bf596763ed22721] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:39.340 INFO 21062 --- [ocp-async-10,1b5aef7ab64b631f,1aa306700483742f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:43:40.006 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:43:40.624 INFO 21062 --- [ocp-schedules-16,ac9e7f30758f3757,20ee4abdfc5166a5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:43:40.628 INFO 21062 --- [ocp-schedules-16,ac9e7f30758f3757,20ee4abdfc5166a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:43:40.716 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:43:40.719 ERROR 21062 --- [ocp-schedules-16,ac9e7f30758f3757,20ee4abdfc5166a5] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:43:40.719 INFO 21062 --- [ocp-schedules-16,ac9e7f30758f3757,20ee4abdfc5166a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:43:40.871 INFO 21062 --- [ocp-schedules-1,dc30449048c31770,cbdc10f65d7b6a85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:43:41.183 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:41.184 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:41.306 WARN 21062 --- [ocp-async-10,312b9d72a1caeafd,4c8002efeca325c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:41.340 INFO 21062 --- [ocp-async-4,a843c9d410f6e8e7,96da7d612a623723] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:41.983 INFO 21062 --- [prometheus-minute-schedule-1,aa1c7a3218b36600,2f252f7f0fefdea7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:43.306 WARN 21062 --- [ocp-async-4,9dbadb9eaa1b6a41,b39e7f227285b907] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:43.341 INFO 21062 --- [ocp-async-3,56df44e64d762fe1,8c798b6021c55ceb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:44.184 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:44.185 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b85b24c9326bca33,c4429523a6c90ec4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:45.306 WARN 21062 --- [ocp-async-3,68ee4a576eb122bd,db5b2167f5c3aa06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:45.341 INFO 21062 --- [ocp-async-9,a0dc91008ff58dc1,82861c324313a5c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:47.185 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:47.186 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:47.307 WARN 21062 --- [ocp-async-7,672092b020accf78,290fabc15ba4f0cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:47.341 INFO 21062 --- [ocp-async-5,e723997d699eac6f,bb53c64a83e25731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:47.466 INFO 21062 --- [ocp-schedules-4,4c7165e45f6c36ba,203caee9908f4fe9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:43:47.983 INFO 21062 --- [prometheus-minute-schedule-1,75b2bf4f7ad5f891,d907a71d58267f42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:43:48.593606+08:00, lastProcessTime=2024-11-04T14:43:38.590571+08:00 2024-11-04 14:43:48.681 INFO 21062 --- [ocp-async-3,9739d75c60751e4e,f49f92728dc39d5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:43:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:43:48.593606+08:00 2024-11-04 14:43:49.307 WARN 21062 --- [ocp-async-4,f018c6f254ccf707,847912d269658b28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:49.341 INFO 21062 --- [ocp-async-8,8a7296f7da9da135,6b2d63d78c87dd33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:50.186 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:50.187 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:50.875 INFO 21062 --- [ocp-schedules-18,a4436d86ec91e8fd,fe8010486630e564] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:43:50.983 INFO 21062 --- [prometheus-minute-schedule-1,2eeaa1a73dc43648,293a2e381c9f6229] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:51.307 WARN 21062 --- [ocp-async-3,da14eb2f8c064ec1,1e0a9a9e35c2fa0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:51.341 INFO 21062 --- [ocp-async-9,63ae348d3aebde00,e47f616db505e4b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:52.002 INFO 21062 --- [ocp-schedules-17,8aa8fc75423ec5d3,532b35768af3b216] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:43:52.007 INFO 21062 --- [ocp-schedules-17,8aa8fc75423ec5d3,532b35768af3b216] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:43:52.007 INFO 21062 --- [ocp-schedules-17,8aa8fc75423ec5d3,532b35768af3b216] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:43:53.187 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:53.188 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:53.307 WARN 21062 --- [ocp-async-7,c30d54a33390a306,c885293fae37c024] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:53.342 INFO 21062 --- [ocp-async-5,5cc5a5ef01e0e885,720234e817c15949] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:53.983 INFO 21062 --- [prometheus-minute-schedule-1,38cdcc8e99aef026,64903fd46c7c631e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:55.307 WARN 21062 --- [ocp-async-2,0c0dd4f0cae4f3a2,80051cf9a46d7287] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:55.342 INFO 21062 --- [ocp-async-6,65e125ec8760ff4f,cfb0201ae0697f24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:56.188 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:56.189 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:56.983 INFO 21062 --- [prometheus-minute-schedule-1,93ea8ce488e6dd4f,c21f5b87b1748d20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:43:57.308 WARN 21062 --- [ocp-async-8,ecbbe6434af03357,48dd990936bf4e07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:57.342 INFO 21062 --- [ocp-async-1,fd4997e47c719ef4,f06e8cb21fb2e218] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:57.491 INFO 21062 --- [ocp-schedules-24,d119171b3ace063b,984a88bda47a2eb5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:43:58.500 INFO 21062 --- [ocp-schedules-10,9d102e270717d642,3f20680adeabef07] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:43:58.500 INFO 21062 --- [ocp-schedules-6,b25451e475482034,4746bf97f0c5e4cb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:43:58.502 INFO 21062 --- [ocp-schedules-3,ec7dc1330052a3a3,e6d3997603fb4eb7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:43:58.504 INFO 21062 --- [ocp-schedules-10,9d102e270717d642,3f20680adeabef07] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:43:58.504 INFO 21062 --- [ocp-schedules-6,b25451e475482034,4746bf97f0c5e4cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:43:58.508 INFO 21062 --- [ocp-schedules-29,906d4eeefc36cf08,f0a42ce8370e5cf5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:43:58.508 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:43:58.510 INFO 21062 --- [ocp-schedules-5,d5eb309217380275,5e34939856e622b1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:43:58.510 INFO 21062 --- [ocp-schedules-18,a13160ea768e7a39,309f85075d0b6dbb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:43:58.510 INFO 21062 --- [ocp-schedules-16,19a0c45c82a3e9a5,b88fdab01f0373d7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:43:58.510 INFO 21062 --- [ocp-schedules-5,d5eb309217380275,5e34939856e622b1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:43:58.510 INFO 21062 --- [ocp-schedules-16,19a0c45c82a3e9a5,b88fdab01f0373d7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:43:58.511 INFO 21062 --- [ocp-schedules-18,a13160ea768e7a39,309f85075d0b6dbb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:43:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:43:58.589247+08:00, lastProcessTime=2024-11-04T14:43:48.593606+08:00 2024-11-04 14:43:58.681 INFO 21062 --- [ocp-async-2,83900f85d9d6e89e,903b09bdfa599e32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:43:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:43:58.589247+08:00 2024-11-04 14:43:59.189 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:43:59.190 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:43:59.279 INFO 21062 --- [ocp-schedules-15,e98c25e2575fac30,f301b79a32cbb0e8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:43:59.308 WARN 21062 --- [ocp-async-7,b8313b3fd3ffa637,a96ecb0d06d56508] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:43:59.342 INFO 21062 --- [ocp-async-5,dde3df3a35459ee9,7e72e0d54aa435f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:43:59.983 INFO 21062 --- [prometheus-minute-schedule-1,bf8d28be74efdcd1,fff3e22d989380d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702640, interval=60 2024-11-04 14:44:00.878 INFO 21062 --- [ocp-schedules-1,9a6a777a26a2b734,c034cabab5e08226] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:44:01.308 WARN 21062 --- [ocp-async-2,2b1974d152bd9de4,e809ffe5938d202e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:01.342 INFO 21062 --- [ocp-async-6,8aa2d043f8fea99c,0e3118a09cdabeaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:02.190 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:02.191 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:02.983 INFO 21062 --- [prometheus-minute-schedule-1,8a24fc9c6ed87396,970ecf7176d111aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:03.308 WARN 21062 --- [ocp-async-8,0b33fed28c12b74c,8de4be8a10f895b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:03.343 INFO 21062 --- [ocp-async-1,13462d4a06b019ab,67e59f246b83a94e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:05.191 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:05.192 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:05.308 WARN 21062 --- [ocp-async-9,4368d77e649d8260,0f31be59b23ee7d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:05.343 INFO 21062 --- [ocp-async-10,a5388d3df326bb7a,7ef3aeac4218190f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8c5ac822db19148b,9a4313b14a8a58c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:07.308 WARN 21062 --- [ocp-async-5,6943521c18600a9c,ba6d5f56eaef897d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:07.343 INFO 21062 --- [ocp-async-4,31a90396c1cb454c,e0971a48a1b3f762] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:07.531 INFO 21062 --- [ocp-schedules-12,c3b6717422ebc767,a3345e3e1bf4c13e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:44:07.531 INFO 21062 --- [ocp-schedules-12,f130e120d1d94704,11d64a8354a88a69] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:44:07.536 INFO 21062 --- [ocp-schedules-14,845a1eaf6d4cd450,4196036ac1ea8da4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:44:07.536 INFO 21062 --- [ocp-schedules-12,f130e120d1d94704,11d64a8354a88a69] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:07.582 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:44:07.582 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:44:08.192 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:08.193 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:08.509 INFO 21062 --- [ocp-schedules-6,b25451e475482034,4746bf97f0c5e4cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:44:08.509 INFO 21062 --- [ocp-schedules-10,9d102e270717d642,3f20680adeabef07] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:44:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:44:08.589522+08:00, lastProcessTime=2024-11-04T14:43:58.589247+08:00 2024-11-04 14:44:08.681 INFO 21062 --- [ocp-async-1,0cae5318e8434196,300fca04aa6ad9bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:44:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:44:08.589522+08:00 2024-11-04 14:44:08.983 INFO 21062 --- [prometheus-minute-schedule-1,1cf9a81396dfa18b,61cf723a14e3f590] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:09.111 INFO 21062 --- [ocp-schedules-2,8528c50ef05841d2,d9bafc11c78f8c12] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:44:09.114 INFO 21062 --- [ocp-schedules-2,8528c50ef05841d2,d9bafc11c78f8c12] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:09.115 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:44:09.309 WARN 21062 --- [ocp-async-8,d1a8c0b9d13352ad,99d63fbca41b0700] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:09.343 INFO 21062 --- [ocp-async-1,32795b0ba27a2d3d,7007c0ef2b52b5ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:44:10.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:44:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:44:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:44:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:44:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:44:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:44:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:44:10.453 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4cae28405fcd76ba,442d98201e3d8591] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.130.47.17, traceId 4cae28405fcd76ba] 2024-11-04 14:44:10.459 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4cae28405fcd76ba,442d98201e3d8591] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.130.47.17, traceId=4cae28405fcd76ba, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2024-11-04 14:44:10.631 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4cae28405fcd76ba,442d98201e3d8591] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 42, current state: FAILED, new state: READY 2024-11-04 14:44:10.631 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4cae28405fcd76ba,442d98201e3d8591] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 13, current state: FAILED, new state: READY 2024-11-04 14:44:10.661 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4cae28405fcd76ba,442d98201e3d8591] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.130.47.17, serverIp=null, startTime=2024-11-04T14:44:10.461+08:00, endTime=2024-11-04T14:44:10.660+08:00, role=null, aliUid=null) 2024-11-04 14:44:10.673 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4cae28405fcd76ba,442d98201e3d8591] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.130.47.17, traceId=4cae28405fcd76ba, duration=229 ms] 2024-11-04 14:44:10.881 INFO 21062 --- [ocp-schedules-29,e431ab35fbe105a0,eb60fe7663bc1dcb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:44:11.117 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,51a878ec27b34967,33783cc9402d452f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=51a878ec27b34967, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:11.144 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,51a878ec27b34967,33783cc9402d452f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=51a878ec27b34967, duration=34 ms] 2024-11-04 14:44:11.175 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d69d085450543921,abebf46b0400130c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=d69d085450543921, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 14:44:11.176 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d69d085450543921,abebf46b0400130c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=d69d085450543921, duration=6 ms] 2024-11-04 14:44:11.193 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:11.194 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:11.309 WARN 21062 --- [ocp-async-9,7ce55909e87552e6,22fbf9420de54982] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:11.343 INFO 21062 --- [ocp-async-10,4cc56a1b757f9817,c44b2d01094caa8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:11.597 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,dd1feafd185f7c75] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: RUNNING 2024-11-04 14:44:11.605 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,dd1feafd185f7c75] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: RUNNING 2024-11-04 14:44:11.618 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=13, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:44:11.589+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=13}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:44:11.621 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:44:11.622 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=42, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:44:11.601+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=42}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:44:11.628 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:44:11.625 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:44:11.630 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:44:11.633 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:44:11.635 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=15, affectRows=15 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=15, validCount=15 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=30, affectRows=30 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=14, affectRows=14 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=14, validCount=14 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=15, affectRows=15 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=30, validCount=30 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=11, affectRows=11 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=15, validCount=15 2024-11-04 14:44:11.644 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=11, validCount=11 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=5, affectRows=5 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=5, validCount=5 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=69, affectRows=69 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=3, affectRows=3 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=3, validCount=3 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=57, affectRows=57 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=66, validCount=66 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=29, affectRows=29 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=23, affectRows=23 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=23, validCount=23 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=29, validCount=29 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=57, validCount=57 2024-11-04 14:44:11.645 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=40, affectRows=40 2024-11-04 14:44:11.646 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=40, validCount=40 2024-11-04 14:44:11.647 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=41, affectRows=41 2024-11-04 14:44:11.647 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=41, validCount=41 2024-11-04 14:44:11.647 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702580, nodeCount=70, affectRows=70 2024-11-04 14:44:11.647 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702580, nodeCount=70, validCount=70 2024-11-04 14:44:11.657 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:11.658 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:11.662 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.665 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.707 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:11.710 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:11.713 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.716 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.759 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:44:11.762 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:44:11.785 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.788 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.793 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.796 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.831 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:11.837 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.838 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:11.844 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.883 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:44:11.890 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:44:11.908 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.911 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.921 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.923 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.955 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:11.961 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.966 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:11.971 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0df3f03e0d7b481f,73070133eba3b809] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:12.010 INFO 21062 --- [ocp-schedules-19,94ecbc1b98b4da36,514990a7893e5639] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:44:12.007 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:44:12.016 INFO 21062 --- [ocp-schedules-19,94ecbc1b98b4da36,514990a7893e5639] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:44:12.016 INFO 21062 --- [ocp-schedules-19,94ecbc1b98b4da36,514990a7893e5639] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:44:12.018 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:44:12.033 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.036 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.038 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:44:12.049 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.052 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.055 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:44:12.075 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:12.081 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.090 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:12.095 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.127 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 14:44:12.142 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 14:44:12.173 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.176 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.179 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:44:12.181 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:44:12.184 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:44:12.190 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.192 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.195 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:44:12.197 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:44:12.199 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:44:12.215 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:12.221 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.234 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:12.239 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.267 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 14:44:12.286 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 14:44:12.313 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.316 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.318 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 14:44:12.321 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 14:44:12.323 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:44:12.325 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:44:12.329 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:44:12.332 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.334 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:44:12.335 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.338 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:44:12.344 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:44:12.347 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:44:12.350 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:44:12.368 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:12.373 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.374 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:12.381 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.419 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:12.425 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.426 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/7d42d4ad63424c31 on 10.46.128.11 2024-11-04 14:44:12.434 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/7d42d4ad63424c31 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/7d42d4ad63424c31, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.437 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/7d42d4ad63424c31 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/7d42d4ad63424c31, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.471 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:44:12.482 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:12.488 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.494 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.497 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.513 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,26aa417cdb9b0d37,af5888abc269e3e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=26aa417cdb9b0d37, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:44:12.522 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,26aa417cdb9b0d37,af5888abc269e3e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=26aa417cdb9b0d37, duration=17 ms] 2024-11-04 14:44:12.534 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/7d42d4ad63424c31/uninstall_package.sh && chmod +x /tmp/7d42d4ad63424c31/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/7d42d4ad63424c31/uninstall_package.sh on 10.46.128.11 2024-11-04 14:44:12.539 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:12.542 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/7d42d4ad63424c31/uninstall_package.sh && chmod +x /tmp/7d42d4ad63424c31/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/7d42d4ad63424c31/uninstall_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/7d42d4ad63424c31/uninstall_package.sh && chmod +x /tmp/7d42d4ad63424c31/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/7d42d4ad63424c31/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.545 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/7d42d4ad63424c31/uninstall_package.sh && chmod +x /tmp/7d42d4ad63424c31/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/7d42d4ad63424c31/uninstall_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/7d42d4ad63424c31/uninstall_package.sh && chmod +x /tmp/7d42d4ad63424c31/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/7d42d4ad63424c31/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.545 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.586 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:12.587 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:44:12.591 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.612 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.615 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.634 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/7d42d4ad63424c31/uninstall_package.sh on 10.46.128.11 2024-11-04 14:44:12.642 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:44:12.655 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:12.660 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.707 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12 2024-11-04 14:44:12.730 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,af411f5a547b2f1a,952130ce2f9b95be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.12, traceId=af411f5a547b2f1a, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:44:12.735 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,af411f5a547b2f1a,952130ce2f9b95be] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:44:12.736 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,af411f5a547b2f1a,952130ce2f9b95be] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:44:12.736 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,af411f5a547b2f1a,952130ce2f9b95be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.12, traceId=af411f5a547b2f1a, duration=9 ms] 2024-11-04 14:44:12.824 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.829 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.843 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/7d42d4ad63424c31/uninstall_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/7d42d4ad63424c31/uninstall_package.sh, out=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 14:44:12.846 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/7d42d4ad63424c31/uninstall_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/7d42d4ad63424c31/uninstall_package.sh, out=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 14:44:12.867 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:12.873 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.890 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:12.896 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.915 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12 2024-11-04 14:44:12.938 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: rm -rf /tmp/7d42d4ad63424c31 on 10.46.128.11 2024-11-04 14:44:12.944 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: rm -rf /tmp/7d42d4ad63424c31 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=rm -rf /tmp/7d42d4ad63424c31, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.946 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: rm -rf /tmp/7d42d4ad63424c31 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=rm -rf /tmp/7d42d4ad63424c31, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:12.949 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.ocp.executor.executor.SshExecutor : uninstall complete, rpmName=ocp-agent-ce, result=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ... 2024-11-04 14:44:12.951 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:44:12.954 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:44:12.956 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:44:12.960 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:44:12.964 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:44:12.968 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:44:12.971 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:44:12.974 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:44:12.993 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:12.997 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.042 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:13.047 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.064 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.067 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.070 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:44:13.094 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:44:13.111 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:13.117 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.119 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.122 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.163 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/0161648f95f94a6f on 10.46.128.12 2024-11-04 14:44:13.166 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:13.169 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/0161648f95f94a6f on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/0161648f95f94a6f, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.171 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.172 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/0161648f95f94a6f on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/0161648f95f94a6f, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.211 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:13.217 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.218 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:44:13.243 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.246 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.259 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/0161648f95f94a6f/cpio && chmod +x /tmp/0161648f95f94a6f/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/0161648f95f94a6f/cpio on 10.46.128.12 2024-11-04 14:44:13.264 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:44:13.266 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/0161648f95f94a6f/cpio && chmod +x /tmp/0161648f95f94a6f/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/0161648f95f94a6f/cpio on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/0161648f95f94a6f/cpio && chmod +x /tmp/0161648f95f94a6f/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/0161648f95f94a6f/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.269 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/0161648f95f94a6f/cpio && chmod +x /tmp/0161648f95f94a6f/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/0161648f95f94a6f/cpio on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/0161648f95f94a6f/cpio && chmod +x /tmp/0161648f95f94a6f/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/0161648f95f94a6f/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.290 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:13.295 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.309 WARN 21062 --- [ocp-async-5,5f07806ebb3c96eb,b1a76de93b962461] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:13.311 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:13.315 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5d2a81e15c496081,4abbe38b3dc393f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=5d2a81e15c496081, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:13.316 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.339 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,5d2a81e15c496081,4abbe38b3dc393f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=5d2a81e15c496081, duration=30 ms] 2024-11-04 14:44:13.343 INFO 21062 --- [ocp-async-4,fe6da44b419cb37c,9373c73e678557cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:13.342 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11 2024-11-04 14:44:13.359 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/0161648f95f94a6f on 10.46.128.12 2024-11-04 14:44:13.365 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,795fad9cc021f605,cc428f5437da214d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.11, traceId=795fad9cc021f605, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:44:13.365 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/0161648f95f94a6f on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/0161648f95f94a6f, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.368 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,795fad9cc021f605,cc428f5437da214d] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:44:13.369 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,795fad9cc021f605,cc428f5437da214d] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:44:13.369 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,795fad9cc021f605,cc428f5437da214d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.11, traceId=795fad9cc021f605, duration=6 ms] 2024-11-04 14:44:13.368 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/0161648f95f94a6f on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/0161648f95f94a6f, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.411 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:13.417 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.456 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.459 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/0161648f95f94a6f/install_package.sh && chmod +x /tmp/0161648f95f94a6f/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/0161648f95f94a6f/install_package.sh on 10.46.128.12 2024-11-04 14:44:13.460 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.467 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/0161648f95f94a6f/install_package.sh && chmod +x /tmp/0161648f95f94a6f/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/0161648f95f94a6f/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/0161648f95f94a6f/install_package.sh && chmod +x /tmp/0161648f95f94a6f/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/0161648f95f94a6f/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.470 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/0161648f95f94a6f/install_package.sh && chmod +x /tmp/0161648f95f94a6f/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/0161648f95f94a6f/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/0161648f95f94a6f/install_package.sh && chmod +x /tmp/0161648f95f94a6f/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/0161648f95f94a6f/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.502 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:13.507 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.515 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:44:13.520 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.554 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11 2024-11-04 14:44:13.567 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/0161648f95f94a6f/install_package.sh on 10.46.128.12 2024-11-04 14:44:13.704 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.707 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.710 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:44:13.750 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:13.756 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.798 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/a0798a4fe7a246e4 on 10.46.128.11 2024-11-04 14:44:13.804 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/a0798a4fe7a246e4 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a0798a4fe7a246e4, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.807 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/a0798a4fe7a246e4 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a0798a4fe7a246e4, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.846 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:13.851 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.894 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/a0798a4fe7a246e4/cpio && chmod +x /tmp/a0798a4fe7a246e4/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a0798a4fe7a246e4/cpio on 10.46.128.11 2024-11-04 14:44:13.901 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/a0798a4fe7a246e4/cpio && chmod +x /tmp/a0798a4fe7a246e4/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a0798a4fe7a246e4/cpio on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a0798a4fe7a246e4/cpio && chmod +x /tmp/a0798a4fe7a246e4/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a0798a4fe7a246e4/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.904 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/a0798a4fe7a246e4/cpio && chmod +x /tmp/a0798a4fe7a246e4/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a0798a4fe7a246e4/cpio on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a0798a4fe7a246e4/cpio && chmod +x /tmp/a0798a4fe7a246e4/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/a0798a4fe7a246e4/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.942 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:13.947 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.984 INFO 21062 --- [prometheus-cache-monitor1,22452852724adcfa,29537f793097e959] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:44:13.984 INFO 21062 --- [prometheus-cache-monitor1,22452852724adcfa,29537f793097e959] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:44:13.990 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/a0798a4fe7a246e4 on 10.46.128.11 2024-11-04 14:44:13.996 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/a0798a4fe7a246e4 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a0798a4fe7a246e4, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:13.999 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/a0798a4fe7a246e4 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a0798a4fe7a246e4, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:14.038 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:14.043 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:14.090 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/a0798a4fe7a246e4/install_package.sh && chmod +x /tmp/a0798a4fe7a246e4/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a0798a4fe7a246e4/install_package.sh on 10.46.128.11 2024-11-04 14:44:14.098 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/a0798a4fe7a246e4/install_package.sh && chmod +x /tmp/a0798a4fe7a246e4/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a0798a4fe7a246e4/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a0798a4fe7a246e4/install_package.sh && chmod +x /tmp/a0798a4fe7a246e4/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a0798a4fe7a246e4/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:14.101 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/a0798a4fe7a246e4/install_package.sh && chmod +x /tmp/a0798a4fe7a246e4/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a0798a4fe7a246e4/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a0798a4fe7a246e4/install_package.sh && chmod +x /tmp/a0798a4fe7a246e4/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/a0798a4fe7a246e4/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:14.142 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:44:14.147 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:44:14.194 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:14.195 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:14.194 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/a0798a4fe7a246e4/install_package.sh on 10.46.128.11 2024-11-04 14:44:14.222 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ec37c0e48c0297c4,b42bec4f890144d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=ec37c0e48c0297c4, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:44:14.229 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ec37c0e48c0297c4,b42bec4f890144d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=ec37c0e48c0297c4, duration=15 ms] 2024-11-04 14:44:14.983 INFO 21062 --- [prometheus-minute-schedule-1,63ec1448e6a995b6,058cd27191325b1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:15.309 WARN 21062 --- [ocp-async-6,9e3d838a8048ea47,64509977bc56f077] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:15.312 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,2dac1d6185ba3725,29adda2dcedb0872] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=2dac1d6185ba3725, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:15.341 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,2dac1d6185ba3725,29adda2dcedb0872] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=2dac1d6185ba3725, duration=36 ms] 2024-11-04 14:44:15.344 INFO 21062 --- [ocp-async-3,91859d81e1e20000,0d8fcb12e5ada8a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:16.973 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,b361f2a2c23140d2,6465e0ec54995069] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=b361f2a2c23140d2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:17.000 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,b361f2a2c23140d2,6465e0ec54995069] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=b361f2a2c23140d2, duration=34 ms] 2024-11-04 14:44:17.195 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:17.196 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:17.309 WARN 21062 --- [ocp-async-3,a640e85992a32c0f,823e4ce6a8b4c638] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:17.344 INFO 21062 --- [ocp-async-7,89fa5f4f7b2b3cee,e8d7c60a7eaa793a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:17.580 INFO 21062 --- [ocp-schedules-16,3859167384842cb4,d7740e0326ea7504] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:44:17.983 INFO 21062 --- [prometheus-minute-schedule-1,e8ea04acda2c4fd4,8efea4a25914f990] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:44:18.591180+08:00, lastProcessTime=2024-11-04T14:44:08.589522+08:00 2024-11-04 14:44:18.681 INFO 21062 --- [ocp-async-4,203301e860c78d74,2295347dcc9b3ffb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:44:18.746 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d2ffd9e6b70b774b,18f96719bf094d3e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=d2ffd9e6b70b774b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:44:18.591180+08:00 2024-11-04 14:44:18.770 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d2ffd9e6b70b774b,18f96719bf094d3e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=d2ffd9e6b70b774b, duration=30 ms] 2024-11-04 14:44:19.115 INFO 21062 --- [ocp-schedules-2,8528c50ef05841d2,d9bafc11c78f8c12] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:44:19.309 WARN 21062 --- [ocp-async-10,aa6c7898a5950cca,94a4c33c5c8b5f28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:19.344 INFO 21062 --- [ocp-async-4,b9b6530cfac722b5,a35636adc234cf15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:20.196 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:20.197 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:20.312 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,80ca6d3d47ed7c11,c5345c08205093fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=80ca6d3d47ed7c11, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:20.339 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,80ca6d3d47ed7c11,c5345c08205093fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=80ca6d3d47ed7c11, duration=32 ms] 2024-11-04 14:44:20.884 INFO 21062 --- [ocp-schedules-3,5ad80a2f35b16b7a,dbb9602e8ac86a06] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:44:20.983 INFO 21062 --- [prometheus-minute-schedule-1,3fb5d075ba00b6e4,74ca99d0e3d194a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:21.310 WARN 21062 --- [ocp-async-4,ece4b300473d5915,6d49a2f763321c9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:21.344 INFO 21062 --- [ocp-async-3,034e82216bee2431,ea3264f4c4407a5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:21.861 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,798f18896cde1dbb,0ba00e3ca41741a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=798f18896cde1dbb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:21.887 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,798f18896cde1dbb,0ba00e3ca41741a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=798f18896cde1dbb, duration=32 ms] 2024-11-04 14:44:23.197 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:23.198 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:23.310 WARN 21062 --- [ocp-async-3,9bfedfb5b330e12f,9b0f6eb8225d1b58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:23.344 INFO 21062 --- [ocp-async-7,61317d419445fefc,e28c8bf13257bf85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:23.410 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b26781bb0c8aa375,7e3c8dcb0952a682] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=b26781bb0c8aa375, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:23.436 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,b26781bb0c8aa375,7e3c8dcb0952a682] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=b26781bb0c8aa375, duration=33 ms] 2024-11-04 14:44:23.983 INFO 21062 --- [prometheus-minute-schedule-1,041fc5370a5e598b,fda1d39028b85f7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:24.966 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,39cd0f9ebbe405f1,5ec6949bbea78c4b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=39cd0f9ebbe405f1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:24.992 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,39cd0f9ebbe405f1,5ec6949bbea78c4b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=39cd0f9ebbe405f1, duration=34 ms] 2024-11-04 14:44:25.310 WARN 21062 --- [ocp-async-7,ccea6ca5ea5fbfcc,c1320ecbcfbb92d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:25.344 INFO 21062 --- [ocp-async-5,8098bcb160e52d7f,0f008eaa26a05f51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:26.198 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:26.198 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:26.550 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,25645d05a69690a8,bd374eed72e58242] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=25645d05a69690a8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:26.575 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,25645d05a69690a8,bd374eed72e58242] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=25645d05a69690a8, duration=32 ms] 2024-11-04 14:44:26.983 INFO 21062 --- [prometheus-minute-schedule-1,501aefd0d3385f3c,0f1ecc2fd1f7adbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9844f2e6780c9cf5,4b8c31748eb5345a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:44:27.310 WARN 21062 --- [ocp-async-5,01494c824bad6704,020d7880ebcddfc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:27.345 INFO 21062 --- [ocp-async-6,d93c993172e796bd,9346435d1c191ae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:27.624 INFO 21062 --- [ocp-schedules-18,ab6fa7cabc219a5d,f1dc32bdf5d6e85a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:44:28.114 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,ec31d37de4e5dac3,528e7aca81f8f82a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=ec31d37de4e5dac3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:28.145 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,ec31d37de4e5dac3,528e7aca81f8f82a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=ec31d37de4e5dac3, duration=40 ms] 2024-11-04 14:44:28.498 INFO 21062 --- [ocp-schedules-28,bebd12f79ca99e09,0be6fe39d3085853] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:44:28.500 INFO 21062 --- [ocp-schedules-25,e05a9b9567d20cb0,b51bb0b79511d016] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:44:28.500 INFO 21062 --- [ocp-schedules-5,2638f2c3b3a65241,2764630ec646b611] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:44:28.500 INFO 21062 --- [ocp-schedules-17,f9b30b7323d0dca4,bb9580ca9a9bfcd5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:44:28.502 INFO 21062 --- [ocp-schedules-28,bebd12f79ca99e09,0be6fe39d3085853] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:44:28.502 INFO 21062 --- [ocp-schedules-15,199efcda744becb4,b78eb0aaf4f36280] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:44:28.504 INFO 21062 --- [ocp-schedules-5,2638f2c3b3a65241,2764630ec646b611] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:28.504 INFO 21062 --- [ocp-schedules-17,f9b30b7323d0dca4,bb9580ca9a9bfcd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:44:28.505 INFO 21062 --- [ocp-schedules-25,e05a9b9567d20cb0,b51bb0b79511d016] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:44:28.505 INFO 21062 --- [ocp-schedules-27,44241f3ef47cd2e0,8052653d5d33dee7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:44:28.509 INFO 21062 --- [ocp-schedules-24,9fb3162733315457,e7346542f104f02c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:44:28.509 INFO 21062 --- [ocp-schedules-24,9fb3162733315457,e7346542f104f02c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:44:28.509 INFO 21062 --- [ocp-schedules-30,2927408c905dca43,e6204887c53e2061] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:44:28.510 INFO 21062 --- [ocp-schedules-30,2927408c905dca43,e6204887c53e2061] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:44:28.510 INFO 21062 --- [ocp-schedules-7,82249d7006d2d16e,4cc02a3e349b06b2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:44:28.510 INFO 21062 --- [ocp-schedules-7,82249d7006d2d16e,4cc02a3e349b06b2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:44:28.580 INFO 21062 --- [ocp-schedules-16,9bac9c4fd64264ee,87a84087d86e467f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:44:28.580 INFO 21062 --- [ocp-schedules-16,9bac9c4fd64264ee,87a84087d86e467f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:44:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:44:28.589295+08:00, lastProcessTime=2024-11-04T14:44:18.591180+08:00 2024-11-04 14:44:28.600 INFO 21062 --- [ocp-schedules-1,cf86f4f380d1c4f9,d6e1bc6ba0e63513] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:44:28.600+08:00, tenantSize=0 2024-11-04 14:44:28.682 INFO 21062 --- [ocp-async-2,1035f85cb16a7afa,731025a7026f2856] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:44:28.683 INFO 21062 --- [ocp-schedules-21,c6bc22a67c1dca3a,5208997bde3a2111] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 14:44:28.683 INFO 21062 --- [ocp-schedules-21,c6bc22a67c1dca3a,5208997bde3a2111] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 14:44:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:44:28.589295+08:00 2024-11-04 14:44:28.884 INFO 21062 --- [ocp-schedules-14,bbfc209ecc939e9a,0fc00ff1c62fcab0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:44:29.193 INFO 21062 --- [ocp-schedules-24,b67e3b777db86623,d95b1d6becb70f0d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:44:29.193 INFO 21062 --- [ocp-schedules-24,b67e3b777db86623,d95b1d6becb70f0d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:44:29.199 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:29.199 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:29.282 INFO 21062 --- [ocp-schedules-28,55f775c379412f9c,33139b4a06307e0d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:44:29.310 WARN 21062 --- [ocp-async-1,9fadcc1e2c8050e3,81f15033b449320a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:29.315 INFO 21062 --- [ocp-schedules-1,2a6109fa786854e0,c373939988795570] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:44:29.320 INFO 21062 --- [ocp-schedules-1,2a6109fa786854e0,c373939988795570] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:44:29.320 INFO 21062 --- [ocp-schedules-1,2a6109fa786854e0,c373939988795570] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:44:29.345 INFO 21062 --- [ocp-async-2,a187c8d0bc10165b,1238efdd8bfd8da4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:29.701 INFO 21062 --- [ocp-schedules-9,d3b91cbdc9941da0,71d4153f7db8f10a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:44:29.701 INFO 21062 --- [ocp-schedules-9,d3b91cbdc9941da0,71d4153f7db8f10a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:44:29.702 INFO 21062 --- [ocp-schedules-9,d3b91cbdc9941da0,71d4153f7db8f10a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:44:29.702 INFO 21062 --- [ocp-schedules-9,d3b91cbdc9941da0,71d4153f7db8f10a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:44:29.703 INFO 21062 --- [ocp-schedules-9,d3b91cbdc9941da0,71d4153f7db8f10a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:44:29.703 INFO 21062 --- [ocp-schedules-9,d3b91cbdc9941da0,71d4153f7db8f10a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:44:29.747 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,9cff4f2a121515de,a01ad31c5603aae5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9cff4f2a121515de, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:29.774 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,9cff4f2a121515de,a01ad31c5603aae5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9cff4f2a121515de, duration=36 ms] 2024-11-04 14:44:29.983 INFO 21062 --- [prometheus-minute-schedule-1,d011174a5d55f617,bc3b97bdc61147eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:44:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:44:30.887 INFO 21062 --- [ocp-schedules-19,a942b725e0ebd642,e1973c7dce1a45c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:44:31.306 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4d2e0164198da8c6,99537baa1fcca05c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=4d2e0164198da8c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:31.310 WARN 21062 --- [ocp-async-2,734ac7b758e00132,07d0871c40255562] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:31.332 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,4d2e0164198da8c6,99537baa1fcca05c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=4d2e0164198da8c6, duration=33 ms] 2024-11-04 14:44:31.345 INFO 21062 --- [ocp-async-8,7c1bc176044f78a0,acd52496b4d31dbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:32.018 INFO 21062 --- [ocp-schedules-3,443d1c05f0ed9cb4,d84baa7f5e22aedb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:44:32.024 INFO 21062 --- [ocp-schedules-3,443d1c05f0ed9cb4,d84baa7f5e22aedb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:44:32.024 INFO 21062 --- [ocp-schedules-3,443d1c05f0ed9cb4,d84baa7f5e22aedb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:44:32.138 INFO 21062 --- [ocp-schedules-15,9eb4141a79e15633,769e484e8dd522ea] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:44:32.138 INFO 21062 --- [ocp-schedules-15,9eb4141a79e15633,769e484e8dd522ea] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:44:32.165 INFO 21062 --- [ocp-schedules-15,9eb4141a79e15633,769e484e8dd522ea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:44:32.165 INFO 21062 --- [ocp-schedules-15,9eb4141a79e15633,769e484e8dd522ea] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:44:32.165 INFO 21062 --- [ocp-schedules-11,404ef8b20622bd9a,746d7dcff6ea5a50] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:44:32.200 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:32.201 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:32.352 INFO 21062 --- [ocp-schedules-21,40cbf2bec29c4bde,7a0daf6b54fadb02] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:44:32.352+08:00, clusters=[1] 2024-11-04 14:44:32.353 INFO 21062 --- [ocp-schedules-21,40cbf2bec29c4bde,7a0daf6b54fadb02] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:44:32.558 INFO 21062 --- [ocp-schedules-22,ed70593f54c3dc97,5cfe0c27ee318423] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:44:32.558+08:00, clusters=[1] 2024-11-04 14:44:32.561 INFO 21062 --- [ocp-schedules-22,ed70593f54c3dc97,5cfe0c27ee318423] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:44:32.566 INFO 21062 --- [ocp-schedules-2,f97cb4b650b3b03c,be6264d5d325aa73] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:44:32.569 INFO 21062 --- [ocp-schedules-2,f97cb4b650b3b03c,be6264d5d325aa73] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:32.626 ERROR 21062 --- [ocp-schedules-2,f97cb4b650b3b03c,be6264d5d325aa73] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:44:32.626 INFO 21062 --- [ocp-schedules-2,f97cb4b650b3b03c,be6264d5d325aa73] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:44:32.849 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,62d9f324ab3e5349,c8022dcada3d2934] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=62d9f324ab3e5349, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:32.874 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,62d9f324ab3e5349,c8022dcada3d2934] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=62d9f324ab3e5349, duration=31 ms] 2024-11-04 14:44:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3f8f72f2ea0b2fe7,a3684707bbea30f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:33.311 WARN 21062 --- [ocp-async-8,e305c1f8d0bd9550,d624c18a66e04184] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:33.345 INFO 21062 --- [ocp-async-9,a4aacaaa254a87f2,519c0939aaf1b3f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:33.824 INFO 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:44:33.824 INFO 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:44:33.824 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:44:33.824 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:44:33.833 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:44:33.833 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:44:33.834 INFO 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:44:33.835 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:44:33.835 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:44:33.835 WARN 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:44:33.835 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:44:33.835 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:44:33.836 INFO 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:44:33.836 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:44:33.836 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:44:33.837 WARN 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:44:33.843 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:44:33.844 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:44:33.844 INFO 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:44:33.854 ERROR 21062 --- [ocp-schedules-19,241d5b518a42da86,c0d89cb1c421ff81] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:44:34.408 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,84c4151abd948394,479af3876a784ec7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=84c4151abd948394, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:34.434 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,84c4151abd948394,479af3876a784ec7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=84c4151abd948394, duration=32 ms] 2024-11-04 14:44:35.201 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:35.201 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:35.311 WARN 21062 --- [ocp-async-9,d0d0f0e9bc6fb37b,761e8c471e7ba01c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:35.345 INFO 21062 --- [ocp-async-10,c12573efd273d35e,dde947fdd995acdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1bc1296a581978b3,48a50890328f553f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:35.993 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,fdcdfbcbce7d75b7,f97448a129f5e2b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=fdcdfbcbce7d75b7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:36.018 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,fdcdfbcbce7d75b7,f97448a129f5e2b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=fdcdfbcbce7d75b7, duration=32 ms] 2024-11-04 14:44:37.311 WARN 21062 --- [ocp-async-10,aed2f74805a86a17,b296663405b2c531] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:37.346 INFO 21062 --- [ocp-async-5,78dc67346927bf2e,929293b9476ae4e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:37.575 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,5e51d62d129ccefd,f62ce4df2e0b07e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=5e51d62d129ccefd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:37.583 INFO 21062 --- [ocp-schedules-12,f130e120d1d94704,11d64a8354a88a69] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:44:37.583 INFO 21062 --- [ocp-schedules-12,11fa0fed5283ac7f,f394955d93d8ad2e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:44:37.586 INFO 21062 --- [ocp-schedules-12,11fa0fed5283ac7f,f394955d93d8ad2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:37.601 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,5e51d62d129ccefd,f62ce4df2e0b07e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=5e51d62d129ccefd, duration=32 ms] 2024-11-04 14:44:37.614 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:44:37.614 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:44:37.650 INFO 21062 --- [ocp-schedules-4,9d81a8a03c66744d,198f6b3ebd518264] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:44:38.201 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:38.202 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:38.504 INFO 21062 --- [ocp-schedules-17,f9b30b7323d0dca4,bb9580ca9a9bfcd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:44:38.505 INFO 21062 --- [ocp-schedules-5,2638f2c3b3a65241,2764630ec646b611] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:44:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:44:38.590348+08:00, lastProcessTime=2024-11-04T14:44:28.589295+08:00 2024-11-04 14:44:38.682 INFO 21062 --- [ocp-async-1,6674c501d0837e4f,a05fc33f2ac9ae6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:44:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:44:38.590348+08:00 2024-11-04 14:44:38.983 INFO 21062 --- [prometheus-minute-schedule-1,28e381e7759ea09d,0431e90356759d9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:39.142 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,4d82cb6bc7dad0f4,5b2bbd4d96ef23ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=4d82cb6bc7dad0f4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:39.168 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,4d82cb6bc7dad0f4,5b2bbd4d96ef23ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=4d82cb6bc7dad0f4, duration=33 ms] 2024-11-04 14:44:39.311 WARN 21062 --- [ocp-async-3,279f155cbd1f731a,ac4a96d8cdec1575] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:39.346 INFO 21062 --- [ocp-async-9,f3f214e96d3945f1,63eb9b16ecb5e72a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:44:40.004 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:44:40.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,f9bea93adee4687b,5d4eb028d0e7cd3b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=f9bea93adee4687b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:40.713 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,f9bea93adee4687b,5d4eb028d0e7cd3b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=f9bea93adee4687b, duration=34 ms] 2024-11-04 14:44:40.719 INFO 21062 --- [ocp-schedules-13,743fc1cf07ae8d39,c6245a389532f13a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:44:40.723 INFO 21062 --- [ocp-schedules-13,743fc1cf07ae8d39,c6245a389532f13a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:40.810 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:44:40.812 ERROR 21062 --- [ocp-schedules-13,743fc1cf07ae8d39,c6245a389532f13a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:44:40.812 INFO 21062 --- [ocp-schedules-13,743fc1cf07ae8d39,c6245a389532f13a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:44:40.890 INFO 21062 --- [ocp-schedules-30,8d83a2fd1cea287f,459e46c67d3cd464] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:44:41.202 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:41.204 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:41.311 WARN 21062 --- [ocp-async-9,01a844a23aa26fc9,9cd555c435318d33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:41.346 INFO 21062 --- [ocp-async-10,10bfdb9752e71295,f53cfd0f2485870f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:44:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:44:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:44:41.854 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:44:41.855 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:44:41.983 INFO 21062 --- [prometheus-minute-schedule-1,885519688f7446fb,8843a5c01f990c3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:42.266 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,336912ad3bba8cbe,88e13a1c408ac1d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=336912ad3bba8cbe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:42.293 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,336912ad3bba8cbe,88e13a1c408ac1d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=336912ad3bba8cbe, duration=34 ms] 2024-11-04 14:44:43.298 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:44:43.298 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:44:43.299 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.300 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:44:43.301 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.301 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.302 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.303 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:44:43.312 WARN 21062 --- [ocp-async-10,470f0d9425193b65,46ebfa024b9d528a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:43.318 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.319 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.320 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.320 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:44:43.334 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.334 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.335 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.335 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:44:43.346 INFO 21062 --- [ocp-async-5,c26745886581a7f8,893ddc4c28344b32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:43.350 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.350 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.351 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.351 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:44:43.365 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.365 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.366 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.366 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:44:43.371 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.372 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.372 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.372 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:44:43.377 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.377 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.378 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.378 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:44:43.382 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.383 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.383 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.383 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:44:43.390 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.391 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.391 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.392 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:44:43.397 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.398 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.398 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.399 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:44:43.405 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.405 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.406 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.406 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:44:43.415 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.415 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.416 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.416 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:44:43.430 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.430 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.431 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.431 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:44:43.437 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.437 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.438 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.438 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:44:43.445 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.445 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.446 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.446 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:44:43.452 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.453 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.453 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.454 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:44:43.458 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.459 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.459 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.460 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:44:43.464 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.465 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.465 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.466 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:44:43.471 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.471 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.471 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.472 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:44:43.477 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.477 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.478 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.478 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:44:43.482 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.482 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.483 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.483 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:44:43.488 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.489 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.489 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.490 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:44:43.494 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.495 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.496 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.496 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:44:43.501 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.501 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.502 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.502 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:44:43.507 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.507 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.507 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.508 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:44:43.512 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.512 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.513 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.513 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:44:43.519 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.519 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.520 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.520 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:44:43.525 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.525 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.525 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.526 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:44:43.531 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.531 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.532 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.532 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:44:43.537 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.537 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.538 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.538 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:44:43.542 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.543 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.543 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.543 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:44:43.548 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.548 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.549 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.549 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:44:43.554 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.555 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.555 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.556 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:44:43.560 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.561 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.561 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.562 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:44:43.569 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.569 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.569 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.570 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:44:43.576 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.576 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.577 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.577 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:44:43.583 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.583 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.584 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.584 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:44:43.590 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.590 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.591 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.591 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:44:43.595 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.595 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.596 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.597 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:44:43.600 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.601 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.601 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.601 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:44:43.606 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.606 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.607 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.607 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:44:43.611 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.611 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.612 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.612 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:44:43.621 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.621 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.622 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.622 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:44:43.643 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.644 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.644 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.644 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:44:43.650 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.651 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.651 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.651 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:44:43.657 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.658 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.658 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.658 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:44:43.665 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.665 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.665 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.666 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:44:43.670 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.671 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.671 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.672 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:44:43.676 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.676 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.677 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.677 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:44:43.682 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:44:43.682 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:44:43.683 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.684 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:44:43.685 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.685 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.686 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.686 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:44:43.692 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.692 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:44:43.693 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:44:43.693 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:44:43.697 INFO 21062 --- [ocp-schedules-9,83c66ee49a28233c,adfd2e5b013e21b7] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:44:43.807 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,9a3c56637d7cceb2,bfbad17d57e07ce2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9a3c56637d7cceb2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:43.834 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,9a3c56637d7cceb2,bfbad17d57e07ce2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9a3c56637d7cceb2, duration=33 ms] 2024-11-04 14:44:44.204 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:44.205 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:44.983 INFO 21062 --- [prometheus-minute-schedule-1,8731a99aec0fa5a8,3a6f13ad46e7f4ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:45.312 WARN 21062 --- [ocp-async-5,0c7411d8de9162d1,038f9c6bc29dcc64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:45.346 INFO 21062 --- [ocp-async-6,ee4632771d851697,2d472880cc635306] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:45.366 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,c01d4b2ada28592d,5ebfd03a2fa2f5db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c01d4b2ada28592d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:45.391 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,c01d4b2ada28592d,5ebfd03a2fa2f5db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c01d4b2ada28592d, duration=32 ms] 2024-11-04 14:44:46.997 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,aa1f1869009866bf,30a1a22df6c3056c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=aa1f1869009866bf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:47.025 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,aa1f1869009866bf,30a1a22df6c3056c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=aa1f1869009866bf, duration=37 ms] 2024-11-04 14:44:47.205 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:47.206 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:47.312 WARN 21062 --- [ocp-async-1,f23c8ed415922ea5,e03f1643335478ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:47.346 INFO 21062 --- [ocp-async-7,2fb8cefac24a699e,71694275fce88ba8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:47.691 INFO 21062 --- [ocp-schedules-9,05f3fbf32da7b340,fb613fcafae63fe7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:44:47.983 INFO 21062 --- [prometheus-minute-schedule-1,79cbc11f2c44cbbc,c42c24691f992412] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:44:48.591014+08:00, lastProcessTime=2024-11-04T14:44:38.590348+08:00 2024-11-04 14:44:48.594 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,85e5364fa4ad12ea,86d415cf5f7b5185] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=85e5364fa4ad12ea, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:48.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,85e5364fa4ad12ea,86d415cf5f7b5185] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=85e5364fa4ad12ea, duration=37 ms] 2024-11-04 14:44:48.682 INFO 21062 --- [ocp-async-5,913746ef7d2217e6,bbac14154a2f6dfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:44:48.777 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:44:48.591014+08:00 2024-11-04 14:44:49.312 WARN 21062 --- [ocp-async-10,2e16a6abc271d061,3c3f014b18022f08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:49.347 INFO 21062 --- [ocp-async-8,3a78ed1aed3c753d,65baeb1476353b69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:49.358 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6bc9451731bfd8ff,144e6efd92fe217c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=6bc9451731bfd8ff, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 14:44:49.534 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,6bc9451731bfd8ff,144e6efd92fe217c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=6bc9451731bfd8ff, duration=188 ms] 2024-11-04 14:44:50.147 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,e464ccd0439b3718,3bec0ae31482ef31] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e464ccd0439b3718, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:50.174 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,e464ccd0439b3718,3bec0ae31482ef31] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e464ccd0439b3718, duration=35 ms] 2024-11-04 14:44:50.206 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:50.207 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:50.894 INFO 21062 --- [ocp-schedules-23,c626b318daacb9ba,d9166657256bf458] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:44:50.983 INFO 21062 --- [prometheus-minute-schedule-1,f222c943fd89b7d2,1ee2cf99230e86d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:51.312 WARN 21062 --- [ocp-async-5,f4f976adfdc6395c,6fb0f49dec44c613] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:51.347 INFO 21062 --- [ocp-async-6,349ca5953f402f29,aa96f5144bc12a93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:51.830 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,fa8e7945ddb86c41,0b16c5fce0258d3b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=fa8e7945ddb86c41, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:51.856 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,fa8e7945ddb86c41,0b16c5fce0258d3b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=fa8e7945ddb86c41, duration=33 ms] 2024-11-04 14:44:52.026 INFO 21062 --- [ocp-schedules-20,a7850caf4921c61f,cbd5ab2e79f80edd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:44:52.031 INFO 21062 --- [ocp-schedules-20,a7850caf4921c61f,cbd5ab2e79f80edd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:44:52.031 INFO 21062 --- [ocp-schedules-20,a7850caf4921c61f,cbd5ab2e79f80edd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:44:53.059 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/0161648f95f94a6f/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/0161648f95f94a6f/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:44:53.063 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/0161648f95f94a6f/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/0161648f95f94a6f/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:44:53.066 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/0161648f95f94a6f/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:44:53.068 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/0161648f95f94a6f/install_package.sh, exit code: 2, cause:null 2024-11-04 14:44:53.072 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/0161648f95f94a6f/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/0161648f95f94a6f/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:44:53.082 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,95be01f87dba14a3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: FAILED 2024-11-04 14:44:53.207 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:53.208 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:53.312 WARN 21062 --- [ocp-async-1,e4d4d18e6aa9e2ca,88e47a3d8afa72ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:53.347 INFO 21062 --- [ocp-async-7,f33f11f01ae2d464,fabc918cf4b03166] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:53.551 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,9954ba52c207b381,7c0e81ef54625969] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9954ba52c207b381, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:53.577 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,9954ba52c207b381,7c0e81ef54625969] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9954ba52c207b381, duration=31 ms] 2024-11-04 14:44:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ae1c8dc5dbcf6a36,3b6ff6b007bae22d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:55.219 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,83860378f98e6ff9,88280fd4b3712aad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=83860378f98e6ff9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:55.244 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,83860378f98e6ff9,88280fd4b3712aad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=83860378f98e6ff9, duration=33 ms] 2024-11-04 14:44:55.312 WARN 21062 --- [ocp-async-2,e3582c7fe7e44c23,f4b560f7495b3acf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:55.347 INFO 21062 --- [ocp-async-4,184d2b0802590423,9b1d761d5e0d6042] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:56.208 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:56.209 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:56.251 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/a0798a4fe7a246e4/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/a0798a4fe7a246e4/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:44:56.255 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/a0798a4fe7a246e4/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/a0798a4fe7a246e4/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:44:56.258 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/a0798a4fe7a246e4/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:44:56.260 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/a0798a4fe7a246e4/install_package.sh, exit code: 2, cause:null 2024-11-04 14:44:56.263 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/a0798a4fe7a246e4/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/a0798a4fe7a246e4/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:44:56.273 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,04f3bdae39dcf43f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: FAILED 2024-11-04 14:44:56.528 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=2024-11-04T14:24:27.335+08:00, subtasks=[SubtaskInstanceEntity{id=68, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=33, startTime=2024-11-04T14:11:16.789+08:00, endTime=2024-11-04T14:11:16.854+08:00}, SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busin 2024-11-04 14:44:56.538 WARN 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:44:56.539 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 14:44:56.540 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:2, status:FAILED 2024-11-04 14:44:56.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,41a475ead7fa048b,d816776abc1cc121] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=41a475ead7fa048b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:44:56.942 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,41a475ead7fa048b,d816776abc1cc121] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=41a475ead7fa048b, duration=32 ms] 2024-11-04 14:44:56.983 INFO 21062 --- [prometheus-minute-schedule-1,467a8f42a600dd47,213d1af58a54f478] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:44:57.313 WARN 21062 --- [ocp-async-8,ccf4c73761f36e8a,be811fbbf08eb840] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:57.347 INFO 21062 --- [ocp-async-3,bc331a3ae9cc97f5,53e5cd163004aaa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:57.718 INFO 21062 --- [ocp-schedules-10,3cd13bd340635587,13cd85be3b1c6bec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:44:57.772 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,86877e30c98a3b40,b1b382eba5cdfe86] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=86877e30c98a3b40, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:44:57.780 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,86877e30c98a3b40,b1b382eba5cdfe86] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=86877e30c98a3b40, duration=15 ms] 2024-11-04 14:44:58.500 INFO 21062 --- [ocp-schedules-16,b96f8f288357b03c,2957780a7e5ad4e0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:44:58.500 INFO 21062 --- [ocp-schedules-14,ae95d3918fa9a717,10d51e84f169eb86] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:44:58.502 INFO 21062 --- [ocp-schedules-13,bcb7e354b98b16f4,d6cd8decf47aa8e1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:44:58.503 INFO 21062 --- [ocp-schedules-16,b96f8f288357b03c,2957780a7e5ad4e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:58.503 INFO 21062 --- [ocp-schedules-14,ae95d3918fa9a717,10d51e84f169eb86] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:44:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:44:58.505 INFO 21062 --- [ocp-schedules-2,08302f52826b4eb7,1c7d8265f4fca3e1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:44:58.510 INFO 21062 --- [ocp-schedules-8,9f66ad08761104a1,f2294c325dde4d90] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:44:58.510 INFO 21062 --- [ocp-schedules-8,9f66ad08761104a1,f2294c325dde4d90] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:44:58.510 INFO 21062 --- [ocp-schedules-18,00f9c5e7480a6e77,abd6a2b993b80b4d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:44:58.510 INFO 21062 --- [ocp-schedules-18,00f9c5e7480a6e77,abd6a2b993b80b4d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:44:58.511 INFO 21062 --- [ocp-schedules-19,f1cbe6a27c0697f4,19874ac51758d7e8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:44:58.511 INFO 21062 --- [ocp-schedules-19,f1cbe6a27c0697f4,19874ac51758d7e8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:44:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:44:58.589747+08:00, lastProcessTime=2024-11-04T14:44:48.591014+08:00 2024-11-04 14:44:58.682 INFO 21062 --- [ocp-async-7,8734e7fc7f783c7a,0700adfa5d7be287] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:44:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:44:58.589747+08:00 2024-11-04 14:44:59.209 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:44:59.210 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:44:59.285 INFO 21062 --- [ocp-schedules-26,1e829287d6faa6e4,912ecfbd28eaedd9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:44:59.313 WARN 21062 --- [ocp-async-1,47b77b66ee3ac01f,421d787cdbaf9631] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:44:59.348 INFO 21062 --- [ocp-async-2,5f97551490dbb3a5,ff51c8869046c6af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:44:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ba3987e1fea737d4,c3366fc3a1938c66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702700, interval=60 2024-11-04 14:45:00.050 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b572739bfabba987] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:40:00.0, nextRun=2024-11-04 14:45:00.0, currentDate=Mon Nov 04 14:45:00 CST 2024 2024-11-04 14:45:00.050 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b572739bfabba987] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:40:00.0, nextRunTime=2024-11-04 14:45:00.0} 2024-11-04 14:45:00.076 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b572739bfabba987] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:45:00.080 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b572739bfabba987] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:45:00.080 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b572739bfabba987] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:45:00.897 INFO 21062 --- [ocp-schedules-9,4e7ffdf9ca9088f6,ca95cfe9b2940196] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:45:01.313 WARN 21062 --- [ocp-async-7,e0d97296e1bee01b,d281945272648027] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:01.348 INFO 21062 --- [ocp-async-4,c6d18d81a654a515,8faf774d9760fce4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:02.210 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:02.211 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:02.983 INFO 21062 --- [prometheus-minute-schedule-1,904aff5690e1e4cd,d5b3ad16355ba24e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:03.313 WARN 21062 --- [ocp-async-8,bcff4be20323f261,7a664570897ff216] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:03.348 INFO 21062 --- [ocp-async-3,eedc97f32018ec5f,4eebee1f97ab6391] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:05.211 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:05.212 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:05.313 WARN 21062 --- [ocp-async-6,e2c065e20921886d,9772c321cd60e3b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:05.348 INFO 21062 --- [ocp-async-9,0e9c01e251bd1602,0b5f7cab86904ef4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:05.983 INFO 21062 --- [prometheus-minute-schedule-1,5a0eb278dabd32f7,5ea31c8413e73f0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:07.313 WARN 21062 --- [ocp-async-2,b08a3d11cb884e00,b319f0389a28f732] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:07.349 INFO 21062 --- [ocp-async-10,61666b3b1dd3db3a,9d192cfdab438e24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:07.615 INFO 21062 --- [ocp-schedules-12,11fa0fed5283ac7f,f394955d93d8ad2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:45:07.615 INFO 21062 --- [ocp-schedules-12,078cfe4eedaf9493,eaa350b82c0a87c8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:45:07.619 INFO 21062 --- [ocp-schedules-12,078cfe4eedaf9493,eaa350b82c0a87c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:07.648 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:45:07.648 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:45:07.762 INFO 21062 --- [ocp-schedules-4,e3638e952050b9f2,aed1c3637e8bfbed] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:45:08.212 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:08.213 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:08.504 INFO 21062 --- [ocp-schedules-14,ae95d3918fa9a717,10d51e84f169eb86] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:45:08.504 INFO 21062 --- [ocp-schedules-16,b96f8f288357b03c,2957780a7e5ad4e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:45:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:45:08.590117+08:00, lastProcessTime=2024-11-04T14:44:58.589747+08:00 2024-11-04 14:45:08.682 INFO 21062 --- [ocp-async-3,8e3394ccc41fe914,e673dfa7a48067f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:45:08.805 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:45:08.590117+08:00 2024-11-04 14:45:08.983 INFO 21062 --- [prometheus-minute-schedule-1,1af9ed29860ea985,58634d024e066e46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:09.314 WARN 21062 --- [ocp-async-8,3e7514dd3c6745ee,5409ecd1ef2f69c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:09.349 INFO 21062 --- [ocp-async-3,9f5d56b2ca4784e0,b968677ec737820c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:10.004 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:45:10.004 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:45:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:45:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:45:10.296 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:45:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:45:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:45:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:45:10.900 INFO 21062 --- [ocp-schedules-16,62c86df6e864cfc3,8613ff3366aeceb8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:45:11.213 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:11.214 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:11.314 WARN 21062 --- [ocp-async-6,a8083f85b79f3e11,4da4f837ac449ead] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:11.349 INFO 21062 --- [ocp-async-9,5bd9f4b7febbf3ce,4ff86bff4d9db1fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:11.647 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=7, affectRows=7 2024-11-04 14:45:11.647 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=11, affectRows=11 2024-11-04 14:45:11.647 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=7, validCount=7 2024-11-04 14:45:11.647 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=11, validCount=11 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=26, affectRows=26 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=15, affectRows=15 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=22, affectRows=22 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=19, affectRows=19 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=38, affectRows=38 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=30, affectRows=30 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=15, validCount=15 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=19, validCount=19 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=22, validCount=22 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=29, validCount=29 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=26, validCount=26 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=38, validCount=38 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=58, affectRows=58 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=59, affectRows=59 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=34, affectRows=34 2024-11-04 14:45:11.648 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=62, affectRows=62 2024-11-04 14:45:11.649 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702640, nodeCount=41, affectRows=41 2024-11-04 14:45:11.649 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=34, validCount=34 2024-11-04 14:45:11.649 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=58, validCount=58 2024-11-04 14:45:11.649 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=61, validCount=61 2024-11-04 14:45:11.649 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=58, validCount=58 2024-11-04 14:45:11.649 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702640, nodeCount=41, validCount=41 2024-11-04 14:45:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0c1cd4914b66a5c7,8ae839c6e9cd1624] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:12.034 INFO 21062 --- [ocp-schedules-26,29a21ff874daa588,824682d3736bd654] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:45:12.039 INFO 21062 --- [ocp-schedules-26,29a21ff874daa588,824682d3736bd654] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:45:12.039 INFO 21062 --- [ocp-schedules-26,29a21ff874daa588,824682d3736bd654] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:45:13.314 WARN 21062 --- [ocp-async-2,94d4952c60a44cd5,9651345e75960fad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:13.349 INFO 21062 --- [ocp-async-10,f066f5909a88ff26,3835cfb7071b6987] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:13.984 INFO 21062 --- [prometheus-cache-monitor1,d7b73f643b860a7d,6b7553cc6695c1b8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:45:13.984 INFO 21062 --- [prometheus-cache-monitor1,d7b73f643b860a7d,6b7553cc6695c1b8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:45:14.214 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:14.215 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:14.983 INFO 21062 --- [prometheus-minute-schedule-1,07b5db4b77f76b89,db364ecd8dc2a7dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:15.241 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:41:15.223, end=2024-11-04T14:43:15.238} 2024-11-04 14:45:15.245 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:45:15.245 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:45:15.314 WARN 21062 --- [ocp-async-4,e6f9afad299a2e04,799dca0528b6feb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:15.349 INFO 21062 --- [ocp-async-5,59b21dbe4d0581c1,25b92486b1f08716] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:17.215 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:17.216 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:17.314 WARN 21062 --- [ocp-async-5,d86bfbe931a8b570,4c5ce6105e76298d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:17.349 INFO 21062 --- [ocp-async-1,842fd437f72c9a90,d119202cff4a7edf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:17.787 INFO 21062 --- [ocp-schedules-21,f593d63a6274ebc4,f9033949995a6ae8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:45:17.983 INFO 21062 --- [prometheus-minute-schedule-1,354668e72f3928f8,2d79e8b50052e629] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:45:18.591036+08:00, lastProcessTime=2024-11-04T14:45:08.590117+08:00 2024-11-04 14:45:18.683 INFO 21062 --- [ocp-async-10,a411be781bbfe2eb,67c8fae136c4f68e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:45:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:45:18.591036+08:00 2024-11-04 14:45:19.115 INFO 21062 --- [ocp-schedules-14,1f17ce912b1ba6ec,6888d9dfc7758d4d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:45:19.119 INFO 21062 --- [ocp-schedules-14,1f17ce912b1ba6ec,6888d9dfc7758d4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:19.119 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:45:19.315 WARN 21062 --- [ocp-async-9,146cb5eba18e5e69,0df79a8c623f7864] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:19.349 INFO 21062 --- [ocp-async-10,245c61c316a4e4b5,5ae63f254a07685c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:20.216 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:20.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:20.902 INFO 21062 --- [ocp-schedules-26,46fadb8a83fa0741,a41915f314e70616] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:45:20.983 INFO 21062 --- [prometheus-minute-schedule-1,0d0a5387848e0b30,4b84a40e3125c669] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:21.315 WARN 21062 --- [ocp-async-10,73d4967109dde5b9,f46875ebe6bd8583] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:21.350 INFO 21062 --- [ocp-async-5,60d0dcccd2ebd7ab,d34a1f801b826c4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:23.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:23.217 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:23.315 WARN 21062 --- [ocp-async-5,367511d56561543f,efe3302c3a878335] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:23.350 INFO 21062 --- [ocp-async-1,03d91ee9c3a62dbf,54c99ccb8e9f212b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c7a1739b5d1def90,8c196ec1f71b3e32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:25.315 WARN 21062 --- [ocp-async-1,2af9a41dad92da21,3767b4142e28e641] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:25.350 INFO 21062 --- [ocp-async-2,93d5d2df21c00947,f71ea768010e234c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:26.218 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:26.218 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:26.983 INFO 21062 --- [prometheus-minute-schedule-1,d83168aebb2a7426,5dc969b73e71e21c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:26.984 INFO 21062 --- [prometheus-hour-schedule-1,0fedf02da2b892bd,91cb1c634b9fce17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:45:27.315 WARN 21062 --- [ocp-async-2,cbaf2b3d2156cd41,d7f6ebd70761450a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:27.350 INFO 21062 --- [ocp-async-8,55284671a145fc43,ca1208502a2a13af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:27.826 INFO 21062 --- [ocp-schedules-18,29d5c28d044f20c7,1c7573eb4a00bae8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:45:28.498 INFO 21062 --- [ocp-schedules-19,0feec5888293a808,b6b924c8ca4ef367] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:45:28.499 INFO 21062 --- [ocp-schedules-3,d4faad758cc7fe30,ef013024e5826fcd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:45:28.499 INFO 21062 --- [ocp-schedules-6,f82397877d15b7c0,49fbb10ab3430aa2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:45:28.500 INFO 21062 --- [ocp-schedules-26,4da23e77bd3f16b3,80cfe6041a26696d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:45:28.502 INFO 21062 --- [ocp-schedules-16,085d491102ab037c,abc21eaa3456df1a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:45:28.503 INFO 21062 --- [ocp-schedules-19,0feec5888293a808,b6b924c8ca4ef367] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:45:28.503 INFO 21062 --- [ocp-schedules-6,f82397877d15b7c0,49fbb10ab3430aa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:45:28.504 INFO 21062 --- [ocp-schedules-26,4da23e77bd3f16b3,80cfe6041a26696d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:28.504 INFO 21062 --- [ocp-schedules-7,7dfa2bed0877ec46,01e837d6c6d2ee1e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:45:28.505 INFO 21062 --- [ocp-schedules-3,d4faad758cc7fe30,ef013024e5826fcd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:45:28.509 INFO 21062 --- [ocp-schedules-24,967336d71a5ae1b8,0961512bb868f262] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:45:28.509 INFO 21062 --- [ocp-schedules-30,6be2364270cea092,9b2bf0bfbdcd0dbc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:45:28.509 INFO 21062 --- [ocp-schedules-24,967336d71a5ae1b8,0961512bb868f262] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:45:28.509 INFO 21062 --- [ocp-schedules-30,6be2364270cea092,9b2bf0bfbdcd0dbc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:45:28.510 INFO 21062 --- [ocp-schedules-20,74389c348f50c2a8,a9e2631e451d1bcd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:45:28.510 INFO 21062 --- [ocp-schedules-20,74389c348f50c2a8,a9e2631e451d1bcd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:45:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:45:28.589080+08:00, lastProcessTime=2024-11-04T14:45:18.591036+08:00 2024-11-04 14:45:28.641 INFO 21062 --- [ocp-schedules-9,7f604a526073e26b,445fb380325a2e0e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:45:28.642 INFO 21062 --- [ocp-schedules-9,7f604a526073e26b,445fb380325a2e0e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:45:28.683 INFO 21062 --- [ocp-async-7,5d60c0110891d035,0edfcc22a89edd44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:45:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:45:28.589080+08:00 2024-11-04 14:45:28.887 INFO 21062 --- [ocp-schedules-25,3eecf60faba3b1e4,4ffa6c1a3e2c6304] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:45:29.120 INFO 21062 --- [ocp-schedules-14,1f17ce912b1ba6ec,6888d9dfc7758d4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:45:29.198 INFO 21062 --- [ocp-schedules-24,c9ee3334361bca1c,ba7d148180b6f53c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:45:29.198 INFO 21062 --- [ocp-schedules-24,c9ee3334361bca1c,ba7d148180b6f53c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:45:29.218 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:29.219 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:29.288 INFO 21062 --- [ocp-schedules-20,2ffbff13c9633e0c,8e7c4700d4ef89a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:45:29.315 WARN 21062 --- [ocp-async-3,d9fe06f49b120045,cff7b89628b604d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:29.320 INFO 21062 --- [ocp-schedules-23,0a3de3d040939378,7bbb4b81cdc1017e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:45:29.325 INFO 21062 --- [ocp-schedules-23,0a3de3d040939378,7bbb4b81cdc1017e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:45:29.325 INFO 21062 --- [ocp-schedules-23,0a3de3d040939378,7bbb4b81cdc1017e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:45:29.350 INFO 21062 --- [ocp-async-7,67855080f98ab4a4,b7f1774a75983174] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:29.704 INFO 21062 --- [ocp-schedules-11,dcf9350424826ef7,e9450feca6b40679] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:45:29.704 INFO 21062 --- [ocp-schedules-11,dcf9350424826ef7,e9450feca6b40679] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:45:29.706 INFO 21062 --- [ocp-schedules-11,dcf9350424826ef7,e9450feca6b40679] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:45:29.706 INFO 21062 --- [ocp-schedules-11,dcf9350424826ef7,e9450feca6b40679] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:45:29.706 INFO 21062 --- [ocp-schedules-11,dcf9350424826ef7,e9450feca6b40679] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:45:29.706 INFO 21062 --- [ocp-schedules-11,dcf9350424826ef7,e9450feca6b40679] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:45:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3bbad7c34cb0b0bc,af440fd3fe553981] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:45:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:45:30.906 INFO 21062 --- [ocp-schedules-21,ad064cb8e4b0acaa,76f6984c1b3dbc28] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:45:31.316 WARN 21062 --- [ocp-async-7,03ab48260f54a276,43b2cd763d626887] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:31.350 INFO 21062 --- [ocp-async-4,6257477bc29eb80d,bcad1714285304d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:32.041 INFO 21062 --- [ocp-schedules-28,ff3bc0dd99aa5c4a,a33e5909d4ad5735] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:45:32.046 INFO 21062 --- [ocp-schedules-28,ff3bc0dd99aa5c4a,a33e5909d4ad5735] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:45:32.046 INFO 21062 --- [ocp-schedules-28,ff3bc0dd99aa5c4a,a33e5909d4ad5735] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:45:32.165 INFO 21062 --- [ocp-schedules-2,8ed3cfa7e0906d90,87b78261b5ac7e4a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:45:32.165 INFO 21062 --- [ocp-schedules-2,8ed3cfa7e0906d90,87b78261b5ac7e4a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:45:32.192 INFO 21062 --- [ocp-schedules-2,8ed3cfa7e0906d90,87b78261b5ac7e4a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:45:32.192 INFO 21062 --- [ocp-schedules-2,8ed3cfa7e0906d90,87b78261b5ac7e4a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:45:32.192 INFO 21062 --- [ocp-schedules-21,b36fbedfaa710a0a,a01ef55fae78ad57] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:45:32.220 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:32.220 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:32.380 INFO 21062 --- [ocp-schedules-17,0cd1f596dcea1c2c,777e40ce98d427aa] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:45:32.380+08:00, clusters=[1] 2024-11-04 14:45:32.382 INFO 21062 --- [ocp-schedules-17,0cd1f596dcea1c2c,777e40ce98d427aa] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:45:32.587 INFO 21062 --- [ocp-schedules-11,3969bfbc77af949f,ad3d4bd7d2b93e5b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:45:32.587+08:00, clusters=[1] 2024-11-04 14:45:32.591 INFO 21062 --- [ocp-schedules-11,3969bfbc77af949f,ad3d4bd7d2b93e5b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:45:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b288cf8ad5952294,43c53614c34cfa4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:33.316 WARN 21062 --- [ocp-async-4,39e3ad4b898f3396,a6d2c3a121d469d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:33.351 INFO 21062 --- [ocp-async-6,0134c8c6720e7671,6e2d702273e1291b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:33.864 INFO 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:45:33.864 INFO 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:45:33.864 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:45:33.864 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:45:33.874 INFO 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:45:33.874 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:45:33.874 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:45:33.875 WARN 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:45:33.876 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:45:33.876 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:45:33.876 INFO 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:45:33.876 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:45:33.877 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:45:33.877 WARN 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:45:33.877 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:45:33.878 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:45:33.884 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:45:33.884 INFO 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:45:33.884 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:45:33.895 ERROR 21062 --- [ocp-schedules-10,a73801f815e66c11,24627013853b7389] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:45:35.220 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:35.221 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:35.316 WARN 21062 --- [ocp-async-6,7ae32acc7ae98ae8,cef1ef99f662b803] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:35.351 INFO 21062 --- [ocp-async-9,ef0587c6efb0c197,00fefc719b8980e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0541f046bcab1b63,d533923f56d983df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:37.316 WARN 21062 --- [ocp-async-9,057bc3742ace79d8,28b30c4ffa1db06d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:37.351 INFO 21062 --- [ocp-async-2,62414f6a9bd5a817,cac3601c5f3616cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:37.648 INFO 21062 --- [ocp-schedules-12,078cfe4eedaf9493,eaa350b82c0a87c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:45:37.648 INFO 21062 --- [ocp-schedules-12,46c81fdbbdeb7b11,b1eb2799eca90b71] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:45:37.652 INFO 21062 --- [ocp-schedules-12,46c81fdbbdeb7b11,b1eb2799eca90b71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:37.680 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:45:37.680 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:45:37.859 INFO 21062 --- [ocp-schedules-19,e49c4767073681bd,ef31e3df2420dee5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:45:38.221 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:38.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:38.504 INFO 21062 --- [ocp-schedules-6,f82397877d15b7c0,49fbb10ab3430aa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:45:38.504 INFO 21062 --- [ocp-schedules-26,4da23e77bd3f16b3,80cfe6041a26696d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:45:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:45:38.589355+08:00, lastProcessTime=2024-11-04T14:45:28.589080+08:00 2024-11-04 14:45:38.683 INFO 21062 --- [ocp-async-3,ee5bbb31acbf938b,fc829585029f365a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:45:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:45:38.589355+08:00 2024-11-04 14:45:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3353ea2f28c1a9fb,ff2bdfddbf11b845] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:39.316 WARN 21062 --- [ocp-async-8,b957009c221ece27,60a49e2094774336] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:39.351 INFO 21062 --- [ocp-async-6,2e72987037374398,2498534b43eddd4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:45:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:45:40.812 INFO 21062 --- [ocp-schedules-11,87bf91600f416e46,e2293fda54635315] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:45:40.816 INFO 21062 --- [ocp-schedules-11,87bf91600f416e46,e2293fda54635315] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:40.903 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:45:40.906 ERROR 21062 --- [ocp-schedules-11,87bf91600f416e46,e2293fda54635315] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:45:40.906 INFO 21062 --- [ocp-schedules-11,87bf91600f416e46,e2293fda54635315] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:45:40.909 INFO 21062 --- [ocp-schedules-22,260d21928c85e538,ed09b6e06eede950] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:45:41.222 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:41.223 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:41.316 WARN 21062 --- [ocp-async-6,0d2c78f361389e1f,8a4f14c6b6c01784] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:41.351 INFO 21062 --- [ocp-async-9,b15c420bdee5cbe4,14c4ad29645b9b14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:41.983 INFO 21062 --- [prometheus-minute-schedule-1,f2926b63a13fd3e6,f72be47ed927d94c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:43.317 WARN 21062 --- [ocp-async-9,7a11c67673508403,2f07370e002a41be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:43.351 INFO 21062 --- [ocp-async-2,ad3abefc9d87c757,a04d6e863e2dd05b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:44.223 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:44.224 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:44.983 INFO 21062 --- [prometheus-minute-schedule-1,64fdb0430f804b0b,62c6ca6a6c1076f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:45.317 WARN 21062 --- [ocp-async-4,26f34e4feb73821b,bdb46c6599a49b0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:45.352 INFO 21062 --- [ocp-async-5,aebf2f94f4e0cb3f,5eb5a9095f8a849e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:47.224 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:47.225 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:47.317 WARN 21062 --- [ocp-async-3,03df04c9e271989f,fb9cd573e39eaca3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:47.352 INFO 21062 --- [ocp-async-1,8a96470f29ab1223,1232dced07041f62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:47.902 INFO 21062 --- [ocp-schedules-2,31f4ef407bc2d367,7bc8627408f456b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:45:47.983 INFO 21062 --- [prometheus-minute-schedule-1,71c28c4f3991f8f5,8faa7a564b987a25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:45:48.592257+08:00, lastProcessTime=2024-11-04T14:45:38.589355+08:00 2024-11-04 14:45:48.683 INFO 21062 --- [ocp-async-4,bcc1573268ebb325,afe42f12004b2358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:45:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:45:48.592257+08:00 2024-11-04 14:45:49.317 WARN 21062 --- [ocp-async-9,11a65e1edc391732,7f2cf6ae09e89442] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:49.352 INFO 21062 --- [ocp-async-2,ee61dd48bb02dcca,945ffaebb0d8f6c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:50.225 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:50.226 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:50.912 INFO 21062 --- [ocp-schedules-29,2fcfd908575c3a93,11cacc3c84b7927a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:45:50.983 INFO 21062 --- [prometheus-minute-schedule-1,f16299805b7b2971,956b15f2607ae6cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:51.317 WARN 21062 --- [ocp-async-4,5c584ecf1ad43916,79806432e7a0b268] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:51.352 INFO 21062 --- [ocp-async-5,295cf13db2fe6bf5,b73e6297cc657fdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:52.048 INFO 21062 --- [ocp-schedules-27,6d967391dc9f68f3,0b3c9dd56dbd2611] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:45:52.053 INFO 21062 --- [ocp-schedules-27,6d967391dc9f68f3,0b3c9dd56dbd2611] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:45:52.053 INFO 21062 --- [ocp-schedules-27,6d967391dc9f68f3,0b3c9dd56dbd2611] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:45:53.226 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:53.227 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:53.317 WARN 21062 --- [ocp-async-3,75963b23a479fd41,67617ac70474a246] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:53.352 INFO 21062 --- [ocp-async-1,37d738b48bfc66d1,4a100a234a56d0ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:53.983 INFO 21062 --- [prometheus-minute-schedule-1,12ae28074ab1e6c2,bd61c6c281038888] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:55.318 WARN 21062 --- [ocp-async-7,6a8ec47f20935fad,3079b3639126b55a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:55.353 INFO 21062 --- [ocp-async-10,32f9814e3fead66e,1734e3754666adb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:56.227 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:56.228 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d15d93e30773b804,b66a51b4135340bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:45:57.182 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:45:57.182 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:45:57.191 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:45:57.191 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:45:57.191 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:45:57.191 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:45:57.196 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:45:57.196 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:45:57.196 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:45:57.197 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:45:57.233 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:45:57.241 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:45:57.241 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:45:57.242 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:45:57.242 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:45:57.242 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:45:57.250 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:45:57.318 WARN 21062 --- [ocp-async-2,8b71ada601d30a60,fbea6625ba192d96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:57.353 INFO 21062 --- [ocp-async-8,70d6ec4d42984612,830826d4274d75ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:57.761 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:45:57.964 INFO 21062 --- [ocp-schedules-3,4be19617358ad9da,8b4f9c8696f82794] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:45:58.331 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.134989833S 2024-11-04 14:45:58.332 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:45:58.400 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:45:58.400 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:45:58.430 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:45:58.477 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:45:58.477 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:45:58.477 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:45:58.477 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:45:58.477 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:45:58.477 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:45:58.485 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:45:58.500 INFO 21062 --- [ocp-schedules-17,9a411b89c9454902,681ede6bbc875a40] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:45:58.500 INFO 21062 --- [ocp-schedules-15,68ebdd64d2b924f6,0c0abe266a5f5454] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:45:58.502 INFO 21062 --- [ocp-schedules-19,7078b76a2ef98e01,caecc086a3c08268] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:45:58.503 INFO 21062 --- [ocp-schedules-17,9a411b89c9454902,681ede6bbc875a40] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:58.503 INFO 21062 --- [ocp-schedules-15,68ebdd64d2b924f6,0c0abe266a5f5454] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:45:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:45:58.504 INFO 21062 --- [ocp-schedules-20,c88eb4a04777d9ef,e2387874a02f40c6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:45:58.508 INFO 21062 --- [ocp-schedules-25,570245accea15782,5fdf6f1b0e5abf4d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:45:58.508 INFO 21062 --- [ocp-schedules-25,570245accea15782,5fdf6f1b0e5abf4d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:45:58.509 INFO 21062 --- [ocp-schedules-18,219418b0eb93a328,080da8b1e0462775] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:45:58.509 INFO 21062 --- [ocp-schedules-28,9f2e7dbe91302fb1,a662964705d584d8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:45:58.509 INFO 21062 --- [ocp-schedules-18,219418b0eb93a328,080da8b1e0462775] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:45:58.509 INFO 21062 --- [ocp-schedules-28,9f2e7dbe91302fb1,a662964705d584d8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:45:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:45:58.589623+08:00, lastProcessTime=2024-11-04T14:45:48.592257+08:00 2024-11-04 14:45:58.602 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] o.s.boot.SpringApplication : Started application in 1.41 seconds (JVM running for 8172.281) 2024-11-04 14:45:58.602 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.411141002S 2024-11-04 14:45:58.683 INFO 21062 --- [ocp-async-1,ead6c382ddb22a84,ad1f6621a47dd9a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:45:58.743 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:45:58.589623+08:00 2024-11-04 14:45:58.964 INFO 21062 --- [ocp-schedules-9,9576f7dbb877f82c,5c5a9f9a4166ccdb] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:45:59.228 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:45:59.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:45:59.291 INFO 21062 --- [ocp-schedules-2,48dc9bbec47d2218,59abfa8d1d2ba79a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:45:59.318 WARN 21062 --- [ocp-async-3,41c07bb4358b9c55,1f7ca6b6515b83f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:45:59.353 INFO 21062 --- [ocp-async-7,4e6592cefbfd438d,9f486d82313e74f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:45:59.983 INFO 21062 --- [prometheus-minute-schedule-1,1d77c8575d82083b,c41ec8e7b8ed6cfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702760, interval=60 2024-11-04 14:46:00.914 INFO 21062 --- [ocp-schedules-11,94091a709a9b3186,abbe51abefba68d0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:46:01.318 WARN 21062 --- [ocp-async-1,bb3dfd1250f96150,8d73a2518eaf5301] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:01.353 INFO 21062 --- [ocp-async-10,398d2ecd202bb50f,2d8c43bfb396aad2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:02.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:02.229 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:02.983 INFO 21062 --- [prometheus-minute-schedule-1,3da3181b2d378910,d2994154f1632613] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:03.318 WARN 21062 --- [ocp-async-2,997639a6e857b374,c3ba82b04473a5d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:03.353 INFO 21062 --- [ocp-async-8,43dedf75636d7bfc,427106b4a027b64d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:03.739 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,75869f69786db3f6,a5441d88a3a9d9b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=75869f69786db3f6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:46:03.745 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,75869f69786db3f6,a5441d88a3a9d9b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=75869f69786db3f6, duration=18 ms] 2024-11-04 14:46:04.708 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,686b908dc0cee522,9dbc9b30a25c241c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=686b908dc0cee522, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:46:04.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,686b908dc0cee522,9dbc9b30a25c241c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=686b908dc0cee522, duration=13 ms] 2024-11-04 14:46:05.230 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:05.230 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:05.318 WARN 21062 --- [ocp-async-5,e72e4b7a073fd207,4aeb0d72a87de8f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:05.354 INFO 21062 --- [ocp-async-6,5d5262e0efedd17d,c2b1698c18170701] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0deacdeac8d327d2,b1b1b4979ea519ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:07.319 WARN 21062 --- [ocp-async-7,42b44980b2bf4464,5f98f7fb0e9698be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:07.354 INFO 21062 --- [ocp-async-9,67a824cc4a738d48,8be5048f6a21b4d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:07.681 INFO 21062 --- [ocp-schedules-12,46c81fdbbdeb7b11,b1eb2799eca90b71] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:46:07.681 INFO 21062 --- [ocp-schedules-12,78a8b11ea24df38e,a25adb84b901ea8f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:46:07.685 INFO 21062 --- [ocp-schedules-12,78a8b11ea24df38e,a25adb84b901ea8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:07.714 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:46:07.714 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:46:08.004 INFO 21062 --- [ocp-schedules-25,c5d7dec0552b6865,f52fde72907b8c9f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:46:08.231 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:08.231 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:08.508 INFO 21062 --- [ocp-schedules-17,9a411b89c9454902,681ede6bbc875a40] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:46:08.508 INFO 21062 --- [ocp-schedules-15,68ebdd64d2b924f6,0c0abe266a5f5454] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:46:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:46:08.590117+08:00, lastProcessTime=2024-11-04T14:45:58.589623+08:00 2024-11-04 14:46:08.683 INFO 21062 --- [ocp-async-8,a3bd09f6b9834def,08c418f2f636c197] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:46:08.746 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:46:08.590117+08:00 2024-11-04 14:46:08.983 INFO 21062 --- [prometheus-minute-schedule-1,398bfa82820fa3ba,a254175d9b4346f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:09.319 WARN 21062 --- [ocp-async-2,ddcfb93fbf40196c,a5f1d9124fb57128] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:09.354 INFO 21062 --- [ocp-async-8,93efa4e3850d0c8a,91f15c54e1c0bf41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:46:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:46:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:46:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:46:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:46:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:46:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:46:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:46:10.920 INFO 21062 --- [ocp-schedules-14,34f0dd6bcc378544,69cc5ec21b10be0a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:46:11.231 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:11.232 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:11.319 WARN 21062 --- [ocp-async-5,de200be0e1bef909,98969fac04584502] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:11.354 INFO 21062 --- [ocp-async-6,dbcd4de7f7c01c5a,dcd13ba6a7fcaf6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:11.650 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702700, nodeCount=15, affectRows=15 2024-11-04 14:46:11.650 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702700, nodeCount=2, affectRows=2 2024-11-04 14:46:11.650 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702700, nodeCount=2, validCount=2 2024-11-04 14:46:11.650 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702700, nodeCount=15, validCount=15 2024-11-04 14:46:11.654 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702700, nodeCount=202, affectRows=202 2024-11-04 14:46:11.654 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702700, nodeCount=203, affectRows=203 2024-11-04 14:46:11.655 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702700, nodeCount=200, validCount=200 2024-11-04 14:46:11.655 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702700, nodeCount=202, validCount=202 2024-11-04 14:46:11.983 INFO 21062 --- [prometheus-minute-schedule-1,60c25da8d1c184b0,b5a7fc9855bf8ac0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:12.056 INFO 21062 --- [ocp-schedules-6,1d950f4e36c0b63d,1d427e750a77c301] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:46:12.061 INFO 21062 --- [ocp-schedules-6,1d950f4e36c0b63d,1d427e750a77c301] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:46:12.061 INFO 21062 --- [ocp-schedules-6,1d950f4e36c0b63d,1d427e750a77c301] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:46:13.319 WARN 21062 --- [ocp-async-7,aaa274f0c798c111,c0b22f86728b36d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:13.354 INFO 21062 --- [ocp-async-9,afa7653dfeff1c29,2e35e709b5ab0328] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,c2f08beaecd849aa,4e016ca320d69695] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,c2f08beaecd849aa,4e016ca320d69695] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:46:14.232 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:14.233 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:14.983 INFO 21062 --- [prometheus-minute-schedule-1,29f701793cd87384,686bd85bef4a9b83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:15.319 WARN 21062 --- [ocp-async-9,f57668979388ecb2,ca2daf8aed3d7843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:15.354 INFO 21062 --- [ocp-async-4,49daf61102d09196,d215f05a4971d46f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:17.233 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:17.234 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:17.320 WARN 21062 --- [ocp-async-4,0020343eb1cd9fec,2ee1b5bede0d907d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:17.355 INFO 21062 --- [ocp-async-3,3e064aedbd94a0f5,751102d2842b40e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:17.983 INFO 21062 --- [prometheus-minute-schedule-1,567a86b92442b693,f76cff0e049e3646] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:18.030 INFO 21062 --- [ocp-schedules-21,6f727b4d9be3d067,6051b545a32c3708] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:46:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:46:18.591113+08:00, lastProcessTime=2024-11-04T14:46:08.590117+08:00 2024-11-04 14:46:18.684 INFO 21062 --- [ocp-async-9,61476dbde195158d,2c185b6396bb05e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:46:18.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:46:18.591113+08:00 2024-11-04 14:46:19.320 WARN 21062 --- [ocp-async-6,dffbc44c5c2aabe5,6ff03443b1205862] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:19.355 INFO 21062 --- [ocp-async-9,e70a30696be24632,d19c4043b6bfa664] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:20.234 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:20.235 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:20.925 INFO 21062 --- [ocp-schedules-25,511ba0639c094f04,6b62ce36be0cec78] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:46:20.983 INFO 21062 --- [prometheus-minute-schedule-1,c592fe3714a6e28a,42e9b4cdcc2ef735] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:21.320 WARN 21062 --- [ocp-async-9,95c235abcd2b8342,ac421776efddc8f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:21.355 INFO 21062 --- [ocp-async-4,4f400c5363e7f382,63d4ad8ebdae70d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:23.235 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:23.236 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:23.320 WARN 21062 --- [ocp-async-4,0fe759d721d91f60,92e86ee96ec758e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:23.355 INFO 21062 --- [ocp-async-3,2868a28fd3f08b82,d4a9777b29e72b3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4cce1dd31860446a,4c581ba5ae17a2eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:25.320 WARN 21062 --- [ocp-async-3,dd8a0eeb9458ec66,2699a3e7b917535d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:25.355 INFO 21062 --- [ocp-async-7,d4f6150c6f4eb30c,00653644309b17be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:26.236 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:26.237 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:26.983 INFO 21062 --- [prometheus-minute-schedule-1,0a84cf3aba4e5919,2f3bb3459cda68b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5561ba357bd12aa9,cf7e1cfbcd68aaf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:46:27.321 WARN 21062 --- [ocp-async-7,4d40a00001505a39,4e745771ac40631a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:27.356 INFO 21062 --- [ocp-async-10,15e0088ba14e6771,bad15f83be7469c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:28.072 INFO 21062 --- [ocp-schedules-6,1034119c3993bb56,b9e7705b7de5fb2f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:46:28.498 INFO 21062 --- [ocp-schedules-10,ff992a89ef15d0a4,f90af2e456af3a4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:46:28.500 INFO 21062 --- [ocp-schedules-8,66beec8ccc43b6a3,819328edeefd93f0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:46:28.500 INFO 21062 --- [ocp-schedules-24,0fcc34526ca232ee,289fdc5d5f9f4441] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:46:28.500 INFO 21062 --- [ocp-schedules-29,cd901afbf31c7a68,5fa2c1d7040e2cc3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:46:28.502 INFO 21062 --- [ocp-schedules-10,ff992a89ef15d0a4,f90af2e456af3a4f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:46:28.502 INFO 21062 --- [ocp-schedules-22,2076ce18d96f84cc,657bf21cd3066c63] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:46:28.503 INFO 21062 --- [ocp-schedules-8,66beec8ccc43b6a3,819328edeefd93f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:28.503 INFO 21062 --- [ocp-schedules-29,cd901afbf31c7a68,5fa2c1d7040e2cc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:46:28.504 INFO 21062 --- [ocp-schedules-25,f4caa4129bdb43a4,1d63ca7395d41811] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:46:28.505 INFO 21062 --- [ocp-schedules-24,0fcc34526ca232ee,289fdc5d5f9f4441] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:46:28.509 INFO 21062 --- [ocp-schedules-14,a49fa7911b1d7cde,753ccd85b05b38f3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:46:28.509 INFO 21062 --- [ocp-schedules-14,a49fa7911b1d7cde,753ccd85b05b38f3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:46:28.509 INFO 21062 --- [ocp-schedules-11,2d59e4a9011cd054,a3451635052370e9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:46:28.510 INFO 21062 --- [ocp-schedules-11,2d59e4a9011cd054,a3451635052370e9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:46:28.510 INFO 21062 --- [ocp-schedules-23,024c148cfada70d8,c641e9ce7cabe64f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:46:28.510 INFO 21062 --- [ocp-schedules-23,024c148cfada70d8,c641e9ce7cabe64f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:46:28.581 INFO 21062 --- [ocp-schedules-5,bae8d8be3c105f8f,4398a0776e8e5227] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:46:28.581 INFO 21062 --- [ocp-schedules-5,bae8d8be3c105f8f,4398a0776e8e5227] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:46:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:46:28.590473+08:00, lastProcessTime=2024-11-04T14:46:18.591113+08:00 2024-11-04 14:46:28.684 INFO 21062 --- [ocp-async-1,a75b1d70103a08c2,249c94e686de0b7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:46:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:46:28.590473+08:00 2024-11-04 14:46:28.863 INFO 21062 --- [ocp-schedules-20,c317d13366e0967b,66f5e1d321d57914] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:46:28.863 INFO 21062 --- [ocp-schedules-20,c317d13366e0967b,66f5e1d321d57914] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:46:28.889 INFO 21062 --- [ocp-schedules-21,abd5e46950b00204,72420774d9791c29] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:46:29.120 INFO 21062 --- [ocp-schedules-14,715e697c785d17b4,56fd1ecda61146eb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:46:29.123 INFO 21062 --- [ocp-schedules-14,715e697c785d17b4,56fd1ecda61146eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:29.124 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:46:29.202 INFO 21062 --- [ocp-schedules-23,9d35bec75ec9c846,f0bf51baa6cb39ab] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:46:29.202 INFO 21062 --- [ocp-schedules-23,9d35bec75ec9c846,f0bf51baa6cb39ab] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:46:29.237 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:29.237 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:29.293 INFO 21062 --- [ocp-schedules-19,0e806c4571c9a201,1e85c472f14cff20] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:46:29.321 WARN 21062 --- [ocp-async-8,21726f5f8436fbfc,c92aa8b4bbb88e11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:29.325 INFO 21062 --- [ocp-schedules-28,260b4e88e02c5fa1,8911e3dc51d49887] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:46:29.331 INFO 21062 --- [ocp-schedules-28,260b4e88e02c5fa1,8911e3dc51d49887] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:46:29.331 INFO 21062 --- [ocp-schedules-28,260b4e88e02c5fa1,8911e3dc51d49887] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:46:29.356 INFO 21062 --- [ocp-async-1,fed6cd77246eecda,fdf03c2ec3803948] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:29.526 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:46:29.526 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:46:29.535 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:46:29.535 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:46:29.535 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:46:29.546 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:46:29.546 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:46:29.546 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:46:29.550 INFO 21062 --- [ocp-schedules-4,dad3a27f44251a90,660f4ef5cd0204b9] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:46:29.707 INFO 21062 --- [ocp-schedules-13,3f980543cde5b42d,8e835387ba72f04f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:46:29.707 INFO 21062 --- [ocp-schedules-13,3f980543cde5b42d,8e835387ba72f04f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:46:29.714 INFO 21062 --- [ocp-schedules-13,3f980543cde5b42d,8e835387ba72f04f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:46:29.714 INFO 21062 --- [ocp-schedules-13,3f980543cde5b42d,8e835387ba72f04f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:46:29.719 INFO 21062 --- [ocp-schedules-13,3f980543cde5b42d,8e835387ba72f04f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:46:29.719 INFO 21062 --- [ocp-schedules-13,3f980543cde5b42d,8e835387ba72f04f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:46:29.983 INFO 21062 --- [prometheus-minute-schedule-1,15cc1e14cfda48fb,0b3e2f44f07f7098] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:46:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:46:30.927 INFO 21062 --- [ocp-schedules-4,b8d00ed3f5fa65b6,0a4895adcf1c7821] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:46:31.321 WARN 21062 --- [ocp-async-1,1d5f04ef862eecb1,9d9bf548251554fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:31.356 INFO 21062 --- [ocp-async-2,bb198e52ecf7651d,d61a2351d82ee466] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:32.063 INFO 21062 --- [ocp-schedules-22,10c157558ccde21d,338b65fb7d695bc7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:46:32.070 INFO 21062 --- [ocp-schedules-22,10c157558ccde21d,338b65fb7d695bc7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:46:32.070 INFO 21062 --- [ocp-schedules-22,10c157558ccde21d,338b65fb7d695bc7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:46:32.193 INFO 21062 --- [ocp-schedules-4,05f6d17eae4bbb30,14894c8e790fa55e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:46:32.193 INFO 21062 --- [ocp-schedules-4,05f6d17eae4bbb30,14894c8e790fa55e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:46:32.221 INFO 21062 --- [ocp-schedules-4,05f6d17eae4bbb30,14894c8e790fa55e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:46:32.221 INFO 21062 --- [ocp-schedules-4,05f6d17eae4bbb30,14894c8e790fa55e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:46:32.222 INFO 21062 --- [ocp-schedules-3,9ed2badbd4f8e9b4,69c03ed848fe6980] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:46:32.238 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:32.238 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:32.406 INFO 21062 --- [ocp-schedules-19,3543100d8520677e,a6fa6303f1862990] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:46:32.406+08:00, clusters=[1] 2024-11-04 14:46:32.410 INFO 21062 --- [ocp-schedules-19,3543100d8520677e,a6fa6303f1862990] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:46:32.617 INFO 21062 --- [ocp-schedules-25,ded4740390977758,b158ab7547caa392] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:46:32.617+08:00, clusters=[1] 2024-11-04 14:46:32.621 INFO 21062 --- [ocp-schedules-25,ded4740390977758,b158ab7547caa392] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:46:32.626 INFO 21062 --- [ocp-schedules-26,43ff752ceaa14011,8121e29f1a9b6934] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:46:32.630 INFO 21062 --- [ocp-schedules-26,43ff752ceaa14011,8121e29f1a9b6934] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:32.690 ERROR 21062 --- [ocp-schedules-26,43ff752ceaa14011,8121e29f1a9b6934] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:46:32.690 INFO 21062 --- [ocp-schedules-26,43ff752ceaa14011,8121e29f1a9b6934] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:46:32.983 INFO 21062 --- [prometheus-minute-schedule-1,da03b77bdc746403,a2dd96a23c4e68a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:33.321 WARN 21062 --- [ocp-async-2,86a50060247323de,cc5a6f5b12156f94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:33.356 INFO 21062 --- [ocp-async-5,3e00ef2ca69cd63e,5f12c6ec2c0af33c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:33.904 INFO 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:46:33.904 INFO 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:46:33.904 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:46:33.904 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:46:33.913 INFO 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:46:33.914 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:46:33.914 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:46:33.915 WARN 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:46:33.915 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:46:33.916 INFO 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:46:33.916 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:46:33.916 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:46:33.917 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:46:33.917 WARN 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:46:33.917 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:46:33.918 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:46:33.925 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:46:33.925 INFO 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:46:33.925 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:46:33.934 ERROR 21062 --- [ocp-schedules-19,830e4fd508e6ccfd,9f8a0f67bb8f8988] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:46:35.239 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:35.240 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:35.321 WARN 21062 --- [ocp-async-5,fc86643c3b17bf89,cd9f26ab86b1f1ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:35.356 INFO 21062 --- [ocp-async-6,d4f9321b21e3cbb8,d8ddc71333845309] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0f6062cc55949ccd,ff73786e115453ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:37.321 WARN 21062 --- [ocp-async-6,40772539e53281e3,8bcc620997579e0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:37.356 INFO 21062 --- [ocp-async-7,07da671050987539,d0c678e91bd6cb05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:37.715 INFO 21062 --- [ocp-schedules-12,78a8b11ea24df38e,a25adb84b901ea8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:46:37.715 INFO 21062 --- [ocp-schedules-12,0f92739966a78255,351ef6114d66b109] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:46:37.718 INFO 21062 --- [ocp-schedules-12,0f92739966a78255,351ef6114d66b109] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:37.747 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:46:37.747 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:46:38.097 INFO 21062 --- [ocp-schedules-20,bce02a713a4cfe5e,4edfa4f8487414e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:46:38.240 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:38.241 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:38.504 INFO 21062 --- [ocp-schedules-29,cd901afbf31c7a68,5fa2c1d7040e2cc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:46:38.504 INFO 21062 --- [ocp-schedules-8,66beec8ccc43b6a3,819328edeefd93f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:46:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:46:38.589733+08:00, lastProcessTime=2024-11-04T14:46:28.590473+08:00 2024-11-04 14:46:38.684 INFO 21062 --- [ocp-async-8,b4a12f1fdedcbd6b,212a418da814ad1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:46:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:46:38.589733+08:00 2024-11-04 14:46:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0430b960607f6067,50a5bb69db94ed9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:39.125 INFO 21062 --- [ocp-schedules-14,715e697c785d17b4,56fd1ecda61146eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:46:39.321 WARN 21062 --- [ocp-async-10,5a7044640ddeae2e,650f2dd08d55b17c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:39.357 INFO 21062 --- [ocp-async-5,4644306cd7980492,1939c1221c65cf94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:40.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:46:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:46:40.906 INFO 21062 --- [ocp-schedules-23,51a88638a6020fbc,7f1c4b538ba82b02] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:46:40.910 INFO 21062 --- [ocp-schedules-23,51a88638a6020fbc,7f1c4b538ba82b02] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:40.930 INFO 21062 --- [ocp-schedules-26,032ac2258d769d83,c56fc0a4229662e5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:46:40.995 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:46:40.996 ERROR 21062 --- [ocp-schedules-23,51a88638a6020fbc,7f1c4b538ba82b02] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:46:40.996 INFO 21062 --- [ocp-schedules-23,51a88638a6020fbc,7f1c4b538ba82b02] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:46:41.241 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:41.242 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:41.322 WARN 21062 --- [ocp-async-5,f2cc6f892a9dd427,40aa98ad856a4215] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:41.357 INFO 21062 --- [ocp-async-6,7e818449d2213dbf,69685c020c2e4f19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:46:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:46:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:46:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:46:41.830 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:46:41.856 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:46:41.856 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:46:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2d731c75bf7b6de1,f2dc21b8e50d37c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:43.322 WARN 21062 --- [ocp-async-6,f45df46dfd75681d,82ee86ba450830ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:43.357 INFO 21062 --- [ocp-async-7,4220043097e7c289,031d789fac7171a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:44.242 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:44.243 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:44.983 INFO 21062 --- [prometheus-minute-schedule-1,0661d5eb2b885198,72bec46e9a350592] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:45.322 WARN 21062 --- [ocp-async-7,805956fc7bca8a07,24533c9785d4e7c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:45.357 INFO 21062 --- [ocp-async-4,c35b0a506b6466f7,ce9fb8d0801551d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:47.243 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:47.244 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:47.322 WARN 21062 --- [ocp-async-8,c440796b4d8a8902,123e5dfe517fc249] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:47.357 INFO 21062 --- [ocp-async-3,2fa2473ac797a60f,61ad335ddf3cfe57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:47.983 INFO 21062 --- [prometheus-minute-schedule-1,9180446469788000,c9cc65c1dce368b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:48.140 INFO 21062 --- [ocp-schedules-6,8712640f9d56173a,15de2114e455f238] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:46:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:46:48.591815+08:00, lastProcessTime=2024-11-04T14:46:38.589733+08:00 2024-11-04 14:46:48.684 INFO 21062 --- [ocp-async-7,d4b6ff469c899e85,38a0604fb8df66ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:46:48.779 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:46:48.591815+08:00 2024-11-04 14:46:49.322 WARN 21062 --- [ocp-async-6,191f43f60b311779,1c5de24848bb56e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:49.358 INFO 21062 --- [ocp-async-2,cf72ac3b56e318fc,1bedcc6e03c29f40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:50.244 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:50.245 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:50.933 INFO 21062 --- [ocp-schedules-11,3b0bd7c718cd3b40,df57ccf9aea953f1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:46:50.983 INFO 21062 --- [prometheus-minute-schedule-1,5c668483b7d56d28,d226f2b470178650] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:51.323 WARN 21062 --- [ocp-async-7,35679b188706c464,c7caf93e2389c34d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:51.358 INFO 21062 --- [ocp-async-4,a2c9dab1e94658ac,9f0d3c92a1c17632] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:52.072 INFO 21062 --- [ocp-schedules-16,50c1a25d501e8c4c,1eebea776c5b0dc8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:46:52.078 INFO 21062 --- [ocp-schedules-16,50c1a25d501e8c4c,1eebea776c5b0dc8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:46:52.078 INFO 21062 --- [ocp-schedules-16,50c1a25d501e8c4c,1eebea776c5b0dc8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:46:53.245 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:53.246 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:53.323 WARN 21062 --- [ocp-async-8,492883cae2fab524,e5fd8ad6ffbdb3b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:53.358 INFO 21062 --- [ocp-async-3,109e419231fca9d4,fab2dd017583ea5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1ec2bd4f8f6e3156,4901af5044b33f99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:55.323 WARN 21062 --- [ocp-async-1,8ff8f9fd3d588f1a,fd49d66da0789bd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:55.358 INFO 21062 --- [ocp-async-9,85df1c014ccf81da,5ca01b97c5c1c0fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:56.246 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:56.247 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e010f760c3e35e4b,31721ab648c8fac5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:46:57.323 WARN 21062 --- [ocp-async-2,90ecb59281b040a5,39b5dab212f37551] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:57.358 INFO 21062 --- [ocp-async-10,a512ae3b0f23c79b,7020f74db342641a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:58.166 INFO 21062 --- [ocp-schedules-25,25d7be573d9d9e8a,db86f3dd8fc035d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:46:58.500 INFO 21062 --- [ocp-schedules-19,5b6a505890c6dcf3,30139edc9bd77463] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:46:58.500 INFO 21062 --- [ocp-schedules-20,a096ae705ffe82de,fad2890b6e6768b6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:46:58.502 INFO 21062 --- [ocp-schedules-2,f7abddcc9392bb3e,9926c5bc312583d1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:46:58.503 INFO 21062 --- [ocp-schedules-19,5b6a505890c6dcf3,30139edc9bd77463] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:58.503 INFO 21062 --- [ocp-schedules-20,a096ae705ffe82de,fad2890b6e6768b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:46:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:46:58.504 INFO 21062 --- [ocp-schedules-1,81711605f1cdc700,6d8b89debb6848bb] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:46:58.509 INFO 21062 --- [ocp-schedules-29,e7566e35219890ec,bf095db4b4c39057] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:46:58.509 INFO 21062 --- [ocp-schedules-14,9f85e516a9ea4fc4,f7d0b30d6bf942ea] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:46:58.510 INFO 21062 --- [ocp-schedules-29,e7566e35219890ec,bf095db4b4c39057] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:46:58.510 INFO 21062 --- [ocp-schedules-14,9f85e516a9ea4fc4,f7d0b30d6bf942ea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:46:58.510 INFO 21062 --- [ocp-schedules-5,d82be9d52a4403cc,050643c146e84f19] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:46:58.510 INFO 21062 --- [ocp-schedules-5,d82be9d52a4403cc,050643c146e84f19] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:46:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:46:58.589726+08:00, lastProcessTime=2024-11-04T14:46:48.591815+08:00 2024-11-04 14:46:58.684 INFO 21062 --- [ocp-async-3,c2c2db4cb29aa3a2,e65931304db62647] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:46:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:46:58.589726+08:00 2024-11-04 14:46:59.247 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:46:59.247 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:46:59.296 INFO 21062 --- [ocp-schedules-28,9ce5cf0fe0f928fb,01293f71069df3a3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:46:59.323 WARN 21062 --- [ocp-async-8,e6c45d629123684f,158caffd6ffa7da4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:46:59.358 INFO 21062 --- [ocp-async-1,a7040e1001700f5c,7e546253ce6177f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:46:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ae2f2454764e6545,be8d0734eb64c85a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702820, interval=60 2024-11-04 14:47:00.936 INFO 21062 --- [ocp-schedules-26,71d3c9d44862c1db,a3688d5d08c2db52] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:47:01.323 WARN 21062 --- [ocp-async-3,00cb678b4d5f7065,07e7dd0b87552c67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:01.359 INFO 21062 --- [ocp-async-9,d79c51b3d61c3419,0583a79eb77410fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:02.248 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:02.248 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4ec7e559fc4816a0,cc6db656158a7ac5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:03.324 WARN 21062 --- [ocp-async-2,74659c0087ccd882,1efd571b137fdd65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:03.359 INFO 21062 --- [ocp-async-10,319dd4b75cfe3202,a4608feb7b8bfef1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:05.248 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:05.249 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:05.324 WARN 21062 --- [ocp-async-4,e35f2e658e727303,d15b333b5717bcb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:05.359 INFO 21062 --- [ocp-async-5,2a14b6669c03c87b,0a08071d228e7b6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0d140084dd4a6973,4dc1c0823b657b38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:07.324 WARN 21062 --- [ocp-async-1,66d7738449552bc2,a00f33919d449b2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:07.359 INFO 21062 --- [ocp-async-6,43a33eec2d3506e4,fd640c01f4aff081] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:07.748 INFO 21062 --- [ocp-schedules-12,0f92739966a78255,351ef6114d66b109] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:47:07.748 INFO 21062 --- [ocp-schedules-12,b8fc69dd22eccd98,1366d97d2cc83f3c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:47:07.752 INFO 21062 --- [ocp-schedules-12,b8fc69dd22eccd98,1366d97d2cc83f3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:07.782 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:47:07.782 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:47:08.214 INFO 21062 --- [ocp-schedules-17,65a55f2e80418f36,dadcc1a9c91a8273] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:47:08.249 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:08.250 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:08.504 INFO 21062 --- [ocp-schedules-20,a096ae705ffe82de,fad2890b6e6768b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:47:08.504 INFO 21062 --- [ocp-schedules-19,5b6a505890c6dcf3,30139edc9bd77463] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:47:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:47:08.591622+08:00, lastProcessTime=2024-11-04T14:46:58.589726+08:00 2024-11-04 14:47:08.684 INFO 21062 --- [ocp-async-10,df527702de4d4e22,92ccb83d4f22bf60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:47:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:47:08.591622+08:00 2024-11-04 14:47:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ab4954c188800f40,3b268a47ee17a43c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:09.324 WARN 21062 --- [ocp-async-2,fcb6394f66c068b8,ecfc0f5648353c0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:09.359 INFO 21062 --- [ocp-async-10,cea504dafa31cc09,375172ecf34e63c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:47:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:47:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:47:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:47:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:47:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:47:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:47:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:47:10.938 INFO 21062 --- [ocp-schedules-7,2a668ee80b251372,c5fe65a6b9a04ff9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:47:11.250 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:11.251 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:11.324 WARN 21062 --- [ocp-async-4,11712f1de1587950,4f77819d6fe24b3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:11.359 INFO 21062 --- [ocp-async-5,be6de3803b397e33,2f46ef5b68a83354] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:11.657 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=2, affectRows=2 2024-11-04 14:47:11.657 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=1, affectRows=1 2024-11-04 14:47:11.657 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=2, validCount=2 2024-11-04 14:47:11.657 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=1, validCount=1 2024-11-04 14:47:11.658 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=6, affectRows=6 2024-11-04 14:47:11.658 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=6, validCount=6 2024-11-04 14:47:11.658 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=8, affectRows=8 2024-11-04 14:47:11.658 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=8, validCount=8 2024-11-04 14:47:11.658 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=46, affectRows=46 2024-11-04 14:47:11.658 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=36, affectRows=36 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=17, affectRows=17 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=35, validCount=35 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=46, validCount=46 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=17, validCount=17 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=36, affectRows=36 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=66, affectRows=66 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=36, validCount=36 2024-11-04 14:47:11.659 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=65, validCount=65 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=21, affectRows=21 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=26, affectRows=26 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=21, validCount=21 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=26, validCount=26 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=30, affectRows=30 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=56, affectRows=56 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=30, validCount=30 2024-11-04 14:47:11.660 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=56, validCount=56 2024-11-04 14:47:11.661 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702760, nodeCount=71, affectRows=71 2024-11-04 14:47:11.661 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702760, nodeCount=70, validCount=70 2024-11-04 14:47:11.983 INFO 21062 --- [prometheus-minute-schedule-1,556a518c5ecd3abf,11e0952969c5b59b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:12.080 INFO 21062 --- [ocp-schedules-26,dcbef9c396b22bcc,05a1c91b211b5b8d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:47:12.086 INFO 21062 --- [ocp-schedules-26,dcbef9c396b22bcc,05a1c91b211b5b8d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:47:12.086 INFO 21062 --- [ocp-schedules-26,dcbef9c396b22bcc,05a1c91b211b5b8d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:47:13.324 WARN 21062 --- [ocp-async-1,8259b33b981bcb48,4ab867942c9bf932] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:13.360 INFO 21062 --- [ocp-async-6,83cfd6ea19f555b8,a5742136a13c0c28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:13.984 INFO 21062 --- [prometheus-cache-monitor1,f1a08c105a35671a,5c529aebb9699252] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:47:13.984 INFO 21062 --- [prometheus-cache-monitor1,f1a08c105a35671a,5c529aebb9699252] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:47:14.251 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:14.252 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:14.983 INFO 21062 --- [prometheus-minute-schedule-1,162fa29d840de3a0,0c3409f083622f13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:15.257 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:43:15.238, end=2024-11-04T14:45:15.254} 2024-11-04 14:47:15.266 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:47:15.266 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:47:15.325 WARN 21062 --- [ocp-async-6,15f01a5b5368f0e8,1e099eb0a603c9d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:15.360 INFO 21062 --- [ocp-async-7,6595246963d5980d,ae08cc48d8353a24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:17.252 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:17.254 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:17.325 WARN 21062 --- [ocp-async-7,4c96608d37d19737,8998e528f165f97d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:17.360 INFO 21062 --- [ocp-async-8,3273d39ca7a17c6a,0e4898628f206432] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:17.983 INFO 21062 --- [prometheus-minute-schedule-1,0af7e62415095f7d,19868a91e209086b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:18.241 INFO 21062 --- [ocp-schedules-9,fae5a1a3e76b1fc6,88f8fb8a6c6157d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:47:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:47:18.591001+08:00, lastProcessTime=2024-11-04T14:47:08.591622+08:00 2024-11-04 14:47:18.685 INFO 21062 --- [ocp-async-6,1ab01acb0fbe8d7b,974a11e2e998bfaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:47:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:47:18.591001+08:00 2024-11-04 14:47:19.325 WARN 21062 --- [ocp-async-5,38fd7b4f07c84e34,f12881c4e22c082c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:19.360 INFO 21062 --- [ocp-async-6,8af7dd16a1711898,40732004614eff15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:20.254 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:20.255 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:20.941 INFO 21062 --- [ocp-schedules-15,4b5cb5b15bc0cea3,63f288f7c97598c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:47:20.983 INFO 21062 --- [prometheus-minute-schedule-1,2ee89e7feb32be8f,b79a19c38443906a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:21.325 WARN 21062 --- [ocp-async-6,35662f02a0b961f6,f1e403ed2b6eb63e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:21.360 INFO 21062 --- [ocp-async-7,66e943a6ceda016e,d1671c3c166511a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:23.255 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:23.257 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:23.326 WARN 21062 --- [ocp-async-7,b9d5a5b047fa05e7,a5b17989e04e44a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:23.360 INFO 21062 --- [ocp-async-8,208511e8dbc14b56,ceb30c42c9be9442] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:23.983 INFO 21062 --- [prometheus-minute-schedule-1,9559e2975d73c3fb,0f324ef712c168a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:25.326 WARN 21062 --- [ocp-async-8,0338782a6c7506a5,9b6628ba7952760e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:25.361 INFO 21062 --- [ocp-async-1,d757adf76a0aa53a,3fa0fec59e996369] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:26.257 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:26.258 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f35380e08595be6f,c866553cabb0a971] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:26.984 INFO 21062 --- [prometheus-hour-schedule-1,0d4642d9cae15c9d,247da4141decbc55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:47:27.326 WARN 21062 --- [ocp-async-1,d428c2e0f62551ee,ce49df30036e55fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:27.361 INFO 21062 --- [ocp-async-2,a9d0b406e2282553,9d7bbf8bfbd553d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:28.283 INFO 21062 --- [ocp-schedules-6,3c5bf5f8b1cb7430,8c75b7e5e0739115] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:47:28.498 INFO 21062 --- [ocp-schedules-28,b16ecb8ad8d5c09b,2e6b9845cc7e7797] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:47:28.499 INFO 21062 --- [ocp-schedules-22,c32cca3363d9b269,099695601270e9dd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:47:28.500 INFO 21062 --- [ocp-schedules-15,d08f6758ad1d6cab,2db195737799ca54] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:47:28.500 INFO 21062 --- [ocp-schedules-16,0efadcb45f8a748b,d1c2c938ebdd786e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:47:28.502 INFO 21062 --- [ocp-schedules-28,b16ecb8ad8d5c09b,2e6b9845cc7e7797] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:47:28.502 INFO 21062 --- [ocp-schedules-30,494e4bcff59739d7,44281c9679f48e10] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:47:28.503 INFO 21062 --- [ocp-schedules-15,d08f6758ad1d6cab,2db195737799ca54] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:28.504 INFO 21062 --- [ocp-schedules-16,0efadcb45f8a748b,d1c2c938ebdd786e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:47:28.504 INFO 21062 --- [ocp-schedules-11,c47387e3520cdc88,998e4b81b8def9b4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:47:28.505 INFO 21062 --- [ocp-schedules-10,c51681448c18cafb,bc2e46ea02cd3e0e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:47:28.505 INFO 21062 --- [ocp-schedules-22,c32cca3363d9b269,099695601270e9dd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:47:28.508 INFO 21062 --- [ocp-schedules-10,c51681448c18cafb,bc2e46ea02cd3e0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:28.510 INFO 21062 --- [ocp-schedules-21,453e14e170f74c3e,153c3db11f135232] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:47:28.510 INFO 21062 --- [ocp-schedules-25,1e667c5f28f55e1a,ceff15ff6c4b46ef] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:47:28.510 INFO 21062 --- [ocp-schedules-21,453e14e170f74c3e,153c3db11f135232] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:47:28.510 INFO 21062 --- [ocp-schedules-25,1e667c5f28f55e1a,ceff15ff6c4b46ef] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:47:28.511 INFO 21062 --- [ocp-schedules-5,b50de0f5075808c4,ffb621c01438999c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:47:28.511 INFO 21062 --- [ocp-schedules-5,b50de0f5075808c4,ffb621c01438999c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:47:28.538 INFO 21062 --- [ocp-schedules-8,e9afd3bfed233e18,b5cc4f9b8445d65b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:47:28.538 INFO 21062 --- [ocp-schedules-8,e9afd3bfed233e18,b5cc4f9b8445d65b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:47:28.580 INFO 21062 --- [pool-12-thread-13,0b6d67a33e0efd68,3a0d4fae697b9e45] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:47:28.580 INFO 21062 --- [pool-12-thread-14,0b6d67a33e0efd68,e906c7d5c107f63a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:47:28.580 INFO 21062 --- [pool-12-thread-15,0b6d67a33e0efd68,d70bb314636b12ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:47:28.580 INFO 21062 --- [pool-12-thread-13,0b6d67a33e0efd68,3a0d4fae697b9e45] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:47:28.580 INFO 21062 --- [pool-12-thread-16,0b6d67a33e0efd68,782d405409b34209] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:47:28.583 INFO 21062 --- [pool-12-thread-13,0b6d67a33e0efd68,3a0d4fae697b9e45] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:47:28.585 INFO 21062 --- [pool-12-thread-15,0b6d67a33e0efd68,d70bb314636b12ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:47:28.585 INFO 21062 --- [pool-12-thread-15,0b6d67a33e0efd68,d70bb314636b12ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:47:28.585 INFO 21062 --- [ForkJoinPool-11-worker-1,0b6d67a33e0efd68,dab893bae73ff9c2] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:47:28.585 INFO 21062 --- [ForkJoinPool-11-worker-1,0b6d67a33e0efd68,dab893bae73ff9c2] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:47:28.585 INFO 21062 --- [ForkJoinPool-11-worker-1,0b6d67a33e0efd68,dab893bae73ff9c2] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:47:28.585 INFO 21062 --- [pool-12-thread-14,0b6d67a33e0efd68,e906c7d5c107f63a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:47:28.585 INFO 21062 --- [pool-12-thread-13,0b6d67a33e0efd68,3a0d4fae697b9e45] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:47:28.585 INFO 21062 --- [ForkJoinPool-11-worker-10,0b6d67a33e0efd68,1eb0ea4383842da1] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:47:28.585 INFO 21062 --- [pool-12-thread-16,0b6d67a33e0efd68,782d405409b34209] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:47:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:47:28.589512+08:00, lastProcessTime=2024-11-04T14:47:18.591001+08:00 2024-11-04 14:47:28.685 INFO 21062 --- [ocp-async-3,a09c9b0a3deca289,b9a91e96b8fd049b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:47:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:47:28.589512+08:00 2024-11-04 14:47:28.892 INFO 21062 --- [ocp-schedules-11,75cf00dd67445c49,dfca7e85573d6e46] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:47:29.140 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:47:29.157 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:47:29.159 WARN 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:47:29.160 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:47:29.161 WARN 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:47:29.168 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:47:29.170 WARN 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:47:29.170 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:47:29.171 WARN 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:47:29.178 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:47:29.179 WARN 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:47:29.180 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:47:29.181 WARN 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:47:29.187 INFO 21062 --- [ocp-schedules-5,f8ed4184613a63ed,05fb6d944a93bba5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:47:29.207 INFO 21062 --- [ocp-schedules-8,cb06c4d144a5e626,73daefc1085e4b65] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:47:29.207 INFO 21062 --- [ocp-schedules-8,cb06c4d144a5e626,73daefc1085e4b65] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:47:29.258 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:29.259 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:29.298 INFO 21062 --- [ocp-schedules-17,0775ede30cc16ec5,7bdca90775398983] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:47:29.326 WARN 21062 --- [ocp-async-4,e26b25928087a774,5c73a6424c22bd05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:29.331 INFO 21062 --- [ocp-schedules-3,cdace2cb89337af1,e4b8cbe3171e6aa8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:47:29.336 INFO 21062 --- [ocp-schedules-3,cdace2cb89337af1,e4b8cbe3171e6aa8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:47:29.336 INFO 21062 --- [ocp-schedules-3,cdace2cb89337af1,e4b8cbe3171e6aa8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:47:29.361 INFO 21062 --- [ocp-async-3,86fdacb1fbf759ef,24468b07c2672d21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:29.724 INFO 21062 --- [ocp-schedules-19,bb49b1dea9b6d4c4,e1cc4af8921896a6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:47:29.724 INFO 21062 --- [ocp-schedules-19,bb49b1dea9b6d4c4,e1cc4af8921896a6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:47:29.725 INFO 21062 --- [ocp-schedules-19,bb49b1dea9b6d4c4,e1cc4af8921896a6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:47:29.725 INFO 21062 --- [ocp-schedules-19,bb49b1dea9b6d4c4,e1cc4af8921896a6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:47:29.726 INFO 21062 --- [ocp-schedules-19,bb49b1dea9b6d4c4,e1cc4af8921896a6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:47:29.726 INFO 21062 --- [ocp-schedules-19,bb49b1dea9b6d4c4,e1cc4af8921896a6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:47:29.983 INFO 21062 --- [prometheus-minute-schedule-1,64bfb423749f3b9e,3631cd59f5d35690] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:30.031 INFO 21062 --- [ocp-schedules-27,0b8716d056140c4a,4a6951de73dba0b2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:47:30.031 INFO 21062 --- [ocp-schedules-27,0b8716d056140c4a,4a6951de73dba0b2] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:47:30.059 INFO 21062 --- [pool-46-thread-1,0b8716d056140c4a,0c326b1cdefaa347] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:47:30.087 WARN 21062 --- [ocp-schedules-27,0b8716d056140c4a,4a6951de73dba0b2] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:47:30.087 INFO 21062 --- [ocp-schedules-27,0b8716d056140c4a,4a6951de73dba0b2] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:47:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:47:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:47:30.944 INFO 21062 --- [ocp-schedules-29,95cdb969a5d130a4,04d9175788b66993] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:47:31.326 WARN 21062 --- [ocp-async-3,a11fd5284cfa934a,5df834c8fd63848f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:31.361 INFO 21062 --- [ocp-async-9,d5cee959b76dd637,8246b7ea78710477] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:32.088 INFO 21062 --- [ocp-schedules-26,fe6569f7096a7732,e3e3da17d7462864] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:47:32.094 INFO 21062 --- [ocp-schedules-26,fe6569f7096a7732,e3e3da17d7462864] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:47:32.094 INFO 21062 --- [ocp-schedules-26,fe6569f7096a7732,e3e3da17d7462864] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:47:32.221 INFO 21062 --- [ocp-schedules-3,45e6ade266158b11,9dcaf9cd382b2439] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:47:32.221 INFO 21062 --- [ocp-schedules-3,45e6ade266158b11,9dcaf9cd382b2439] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:47:32.248 INFO 21062 --- [ocp-schedules-3,45e6ade266158b11,9dcaf9cd382b2439] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:47:32.248 INFO 21062 --- [ocp-schedules-3,45e6ade266158b11,9dcaf9cd382b2439] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:47:32.249 INFO 21062 --- [ocp-schedules-19,38e96eef0894a0ac,c86b184436384b3e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:47:32.259 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:32.261 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:32.437 INFO 21062 --- [ocp-schedules-22,b82b4a88c50c0cee,0179a42a92fd55da] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:47:32.437+08:00, clusters=[1] 2024-11-04 14:47:32.438 INFO 21062 --- [ocp-schedules-22,b82b4a88c50c0cee,0179a42a92fd55da] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:47:32.647 INFO 21062 --- [ocp-schedules-27,5f76303cb3ce7726,c53d9b9543be3a26] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:47:32.647+08:00, clusters=[1] 2024-11-04 14:47:32.651 INFO 21062 --- [ocp-schedules-27,5f76303cb3ce7726,c53d9b9543be3a26] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:47:32.983 INFO 21062 --- [prometheus-minute-schedule-1,eb95fcc703d8c87c,885699b8f0c8e8be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:33.326 WARN 21062 --- [ocp-async-9,36be3612018245ac,53e2a45781ed7f5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:33.361 INFO 21062 --- [ocp-async-10,352e2fc5115480e9,c2ad36ab123987b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:33.945 INFO 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:47:33.945 INFO 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:47:33.945 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:47:33.945 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:47:33.955 INFO 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:47:33.956 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:47:33.956 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:47:33.957 WARN 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:47:33.957 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:47:33.957 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:47:33.958 INFO 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:47:33.958 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:47:33.958 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:47:33.959 WARN 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:47:33.959 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:47:33.959 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:47:33.966 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:47:33.966 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:47:33.966 INFO 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:47:33.978 ERROR 21062 --- [ocp-schedules-27,c7390509b946645e,f511c5429d010852] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:47:35.261 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:35.262 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:35.326 WARN 21062 --- [ocp-async-10,9d06139c005deb47,125be2d9280e5ec2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:35.361 INFO 21062 --- [ocp-async-8,e81cb4f8ccbb995d,2cd3f35cef0346ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:35.983 INFO 21062 --- [prometheus-minute-schedule-1,540df973279818d7,f5fa5b6d65ad9018] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:37.327 WARN 21062 --- [ocp-async-8,552783b94717b0c4,6026719ad0e0258c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:37.362 INFO 21062 --- [ocp-async-1,e39bfbe46baa888a,f38a52562b8abc96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:37.783 INFO 21062 --- [ocp-schedules-12,b8fc69dd22eccd98,1366d97d2cc83f3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:47:37.783 INFO 21062 --- [ocp-schedules-12,e4864aae674276e3,15aad2f47ab47190] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:47:37.787 INFO 21062 --- [ocp-schedules-12,e4864aae674276e3,15aad2f47ab47190] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:37.815 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:47:37.815 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:47:38.262 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:38.263 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:38.308 INFO 21062 --- [ocp-schedules-8,c7ca6574c8b149a6,9c3c32aabdac1672] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:47:38.504 INFO 21062 --- [ocp-schedules-15,d08f6758ad1d6cab,2db195737799ca54] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:47:38.504 INFO 21062 --- [ocp-schedules-16,0efadcb45f8a748b,d1c2c938ebdd786e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:47:38.508 INFO 21062 --- [ocp-schedules-10,c51681448c18cafb,bc2e46ea02cd3e0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:47:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:47:38.590346+08:00, lastProcessTime=2024-11-04T14:47:28.589512+08:00 2024-11-04 14:47:38.685 INFO 21062 --- [ocp-async-4,359dd820004a3e39,076e8f711a732538] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:47:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:47:38.590346+08:00 2024-11-04 14:47:38.983 INFO 21062 --- [prometheus-minute-schedule-1,68ff287c6417cba7,810fdc678c50692a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:39.125 INFO 21062 --- [ocp-schedules-11,76e3ec7aa6d2a185,d82a174d3a0e9707] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:47:39.128 INFO 21062 --- [ocp-schedules-11,76e3ec7aa6d2a185,d82a174d3a0e9707] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:39.129 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:47:39.327 WARN 21062 --- [ocp-async-2,7ef41c2b646ac318,d7f7ee41998eadc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:39.362 INFO 21062 --- [ocp-async-10,07c94bd29a888cde,57de43cb8ee0051e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:47:40.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:47:40.949 INFO 21062 --- [ocp-schedules-26,a0c0ae507fc4f494,e2fafb7ee7a8c83c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:47:40.997 INFO 21062 --- [ocp-schedules-17,0d24d6a144d5752f,23510dbcbe044c62] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:47:41.000 INFO 21062 --- [ocp-schedules-17,0d24d6a144d5752f,23510dbcbe044c62] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:41.087 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:47:41.089 ERROR 21062 --- [ocp-schedules-17,0d24d6a144d5752f,23510dbcbe044c62] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:47:41.089 INFO 21062 --- [ocp-schedules-17,0d24d6a144d5752f,23510dbcbe044c62] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:47:41.264 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:41.265 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:41.327 WARN 21062 --- [ocp-async-10,a9ba42c53ae04a07,e64a044ad7e5a97b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:41.362 INFO 21062 --- [ocp-async-8,53e0a9dc9412eb5e,85da36c7e64c1443] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:41.983 INFO 21062 --- [prometheus-minute-schedule-1,26e9f9be62bee19d,80e67c64a5661acc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:43.327 WARN 21062 --- [ocp-async-8,f2d89fd148726031,2551c72da47e44c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:43.362 INFO 21062 --- [ocp-async-1,a776e2536eb021e0,145a2b98ae4db303] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:44.265 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:44.266 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9a324c5f3510197b,8daaf91ee3c0390a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:45.327 WARN 21062 --- [ocp-async-9,d7c2e2f1915bda3d,5e96834dab71a915] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:45.363 INFO 21062 --- [ocp-async-7,8c1c373e4854734a,86172da57b633a3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:47.266 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:47.268 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:47.327 WARN 21062 --- [ocp-async-4,750a756070328b18,13e3d622713000b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:47.363 INFO 21062 --- [ocp-async-5,5d7f42d7654d9116,77e46a2450893f6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:47.983 INFO 21062 --- [prometheus-minute-schedule-1,81690d1b12bb5a53,a36c8bcca7c8fd93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:48.349 INFO 21062 --- [ocp-schedules-29,d81a2a76123b4810,10f9f2786d99a78c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:47:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:47:48.590738+08:00, lastProcessTime=2024-11-04T14:47:38.590346+08:00 2024-11-04 14:47:48.685 INFO 21062 --- [ocp-async-1,61353dee626c139f,ca973110a6a5c741] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:47:48.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:47:48.590738+08:00 2024-11-04 14:47:49.153 INFO 21062 --- [ocp-schedules-11,76e3ec7aa6d2a185,d82a174d3a0e9707] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:47:49.328 WARN 21062 --- [ocp-async-8,e41f96dca38ab009,bdacdef7aa295728] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:49.363 INFO 21062 --- [ocp-async-9,b0a9bcb0b79f6ea2,e68efb4e384b471f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:50.268 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:50.269 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:50.951 INFO 21062 --- [ocp-schedules-15,df464d5411c80d59,16f66c5e360b8b2d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:47:50.983 INFO 21062 --- [prometheus-minute-schedule-1,9ed4b179ae4126c2,e630fd1836bb9d04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:51.328 WARN 21062 --- [ocp-async-1,d5197750f768790a,b468ae7d18b33f6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:51.363 INFO 21062 --- [ocp-async-7,82c6574bbf12167b,6d2eed3186566008] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:52.097 INFO 21062 --- [ocp-schedules-16,ac78ba63ed080a0c,49721e0eec8639cd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:47:52.101 INFO 21062 --- [ocp-schedules-16,ac78ba63ed080a0c,49721e0eec8639cd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:47:52.101 INFO 21062 --- [ocp-schedules-16,ac78ba63ed080a0c,49721e0eec8639cd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:47:53.269 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:53.270 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:53.328 WARN 21062 --- [ocp-async-4,6af34c70928ed874,62bf2cffe1c5f36f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:53.363 INFO 21062 --- [ocp-async-5,57b9eec225e29922,6667d0efbe380c21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:53.983 INFO 21062 --- [prometheus-minute-schedule-1,3093c983a8554754,6f79cf843a091574] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:55.328 WARN 21062 --- [ocp-async-3,05e17295c9b1cb6b,f1b5924fcde646c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:55.363 INFO 21062 --- [ocp-async-6,8dbe28c912cc3873,fc3ab28e87e85174] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:56.270 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:56.271 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ee13456f5feca4db,81778e38aefa09bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:47:57.328 WARN 21062 --- [ocp-async-9,9da319ca67eb9d6f,301fb9eef74686e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:57.363 INFO 21062 --- [ocp-async-2,34e61551f1911b05,cd041d91a2c9f158] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:58.374 INFO 21062 --- [ocp-schedules-4,0d6757ce2eed8381,93308a69a0cd9418] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:47:58.500 INFO 21062 --- [ocp-schedules-27,6e929dd4fd50f5e6,22997a876d2f88bd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:47:58.500 INFO 21062 --- [ocp-schedules-20,a1fd6e03abdb0bb2,49e8eebc65183b5f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:47:58.502 INFO 21062 --- [ocp-schedules-17,3632be9c144abdb6,e5d10e5c5ce8948f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:47:58.503 INFO 21062 --- [ocp-schedules-27,6e929dd4fd50f5e6,22997a876d2f88bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:47:58.504 INFO 21062 --- [ocp-schedules-20,a1fd6e03abdb0bb2,49e8eebc65183b5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:47:58.505 INFO 21062 --- [ocp-schedules-13,e557c9f7cd3c993c,5bff49871dc17bbc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:47:58.510 INFO 21062 --- [ocp-schedules-28,cf11530d5425c4c1,2465a48aa0fe8ba6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:47:58.510 INFO 21062 --- [ocp-schedules-10,48bc95ac4cf56bc1,5dcde9b85ec14e13] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:47:58.510 INFO 21062 --- [ocp-schedules-28,cf11530d5425c4c1,2465a48aa0fe8ba6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:47:58.510 INFO 21062 --- [ocp-schedules-10,48bc95ac4cf56bc1,5dcde9b85ec14e13] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:47:58.510 INFO 21062 --- [ocp-schedules-25,f84fbb226463b795,c509df4a3c57c629] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:47:58.510 INFO 21062 --- [ocp-schedules-25,f84fbb226463b795,c509df4a3c57c629] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:47:58.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:47:58.592898+08:00, lastProcessTime=2024-11-04T14:47:48.590738+08:00 2024-11-04 14:47:58.685 INFO 21062 --- [ocp-async-5,3e568feec4e6afe5,aa8c14f4c441a27a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:47:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:47:58.592898+08:00 2024-11-04 14:47:59.271 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:47:59.272 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:47:59.301 INFO 21062 --- [ocp-schedules-29,29b215d09d5e8c70,a80d966c9650aad3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:47:59.328 WARN 21062 --- [ocp-async-4,124ab13214cc1b06,068077c9073fb9ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:47:59.364 INFO 21062 --- [ocp-async-5,c6cadb51d61b4878,7dee5f941a9a7e3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:47:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e12591ff35c98e9a,9a5ecdefdde22d6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702880, interval=60 2024-11-04 14:48:00.955 INFO 21062 --- [ocp-schedules-2,275a3dc446f3e27c,c5c355a576275143] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:48:01.329 WARN 21062 --- [ocp-async-3,e28148e7778ab9b8,ab9ab9a9037861cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:01.364 INFO 21062 --- [ocp-async-6,e516743b78888090,45caac2f8df10c17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:02.272 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:02.272 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5f6a9fdca6ef0a0f,37910c6690c9225a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:03.329 WARN 21062 --- [ocp-async-9,d2fbdf9aa3d5a9a7,bfa26c2b6f603cea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:03.364 INFO 21062 --- [ocp-async-2,f865c97db3a5c9a0,a2b5546767e42c07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:05.273 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:05.273 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:05.329 WARN 21062 --- [ocp-async-7,4aa922c448f44620,10fba612dabf3c97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:05.364 INFO 21062 --- [ocp-async-10,14f96c52768272e9,cb959886a4c46e2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f6bc4284fe925b83,e615d38f7fa8400e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:07.329 WARN 21062 --- [ocp-async-5,843285993e578217,e61072759f483621] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:07.364 INFO 21062 --- [ocp-async-8,f66ee84bfe290a6b,a9946d9e22a4ffe1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:07.816 INFO 21062 --- [ocp-schedules-12,e4864aae674276e3,15aad2f47ab47190] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:48:07.816 INFO 21062 --- [ocp-schedules-12,2954deb14e6d8117,642db5ed4d90b4eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:48:07.820 INFO 21062 --- [ocp-schedules-12,2954deb14e6d8117,642db5ed4d90b4eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:07.849 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:48:07.849 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:48:08.274 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:08.275 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:08.419 INFO 21062 --- [ocp-schedules-13,8c79b0ac69ae3d49,f69915c6c14dce6a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:48:08.504 INFO 21062 --- [ocp-schedules-27,6e929dd4fd50f5e6,22997a876d2f88bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:48:08.505 INFO 21062 --- [ocp-schedules-20,a1fd6e03abdb0bb2,49e8eebc65183b5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:48:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:48:08.591004+08:00, lastProcessTime=2024-11-04T14:47:58.592898+08:00 2024-11-04 14:48:08.685 INFO 21062 --- [ocp-async-2,d969cb5eef0884cb,14ac710c33319086] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:48:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:48:08.591004+08:00 2024-11-04 14:48:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2f278e9705be95eb,1fb51b2414fa8963] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:09.329 WARN 21062 --- [ocp-async-9,1e314c9ac635abe4,41bec331f58d6932] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:09.364 INFO 21062 --- [ocp-async-2,81c866091f5bf3db,3f64b5b1aebaaae7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:48:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:48:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:48:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:48:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:48:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:48:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:48:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:48:10.958 INFO 21062 --- [ocp-schedules-17,218242d1a9270e8f,b753fa199677ebf9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:48:11.275 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:11.276 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:11.329 WARN 21062 --- [ocp-async-7,1f8e0dd194e9f834,91a54f1ef4b52d01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:11.365 INFO 21062 --- [ocp-async-10,1e5436a4cfc122bb,97258bd144992f11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:11.658 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=23, affectRows=23 2024-11-04 14:48:11.658 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=51, affectRows=51 2024-11-04 14:48:11.658 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=58, affectRows=58 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=23, validCount=23 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=52, affectRows=52 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=47, affectRows=47 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=35, affectRows=35 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=50, affectRows=50 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=45, affectRows=45 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=51, validCount=51 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=58, validCount=58 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=52, validCount=52 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=45, validCount=45 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=47, validCount=47 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=35, validCount=35 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=49, validCount=49 2024-11-04 14:48:11.659 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702820, nodeCount=61, affectRows=61 2024-11-04 14:48:11.660 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702820, nodeCount=59, validCount=59 2024-11-04 14:48:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0b871791214a3b7c,dda8832d4c7b8fb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:12.104 INFO 21062 --- [ocp-schedules-3,844b6284fe2be9ad,ca3251f4be49126c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:48:12.111 INFO 21062 --- [ocp-schedules-3,844b6284fe2be9ad,ca3251f4be49126c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:48:12.111 INFO 21062 --- [ocp-schedules-3,844b6284fe2be9ad,ca3251f4be49126c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:48:13.330 WARN 21062 --- [ocp-async-5,dba8fd42adb70f50,9a61221bb7298f53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:13.365 INFO 21062 --- [ocp-async-8,8cc10ab497110a4a,c74ec8f2ed208348] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:13.984 INFO 21062 --- [prometheus-cache-monitor1,7e65455780ffb8aa,672bc5b231602991] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:48:13.985 INFO 21062 --- [prometheus-cache-monitor1,7e65455780ffb8aa,672bc5b231602991] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:48:14.276 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:14.277 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:14.342 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:42:14.321, end=2024-11-04T14:47:14.339} 2024-11-04 14:48:14.342 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:42:14.321, end=2024-11-04T14:47:14.339} 2024-11-04 14:48:14.346 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:48:14.346 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:48:14.346 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:48:14.346 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:48:14.983 INFO 21062 --- [prometheus-minute-schedule-1,224ee84d68dd6172,6d8a9e21e74482f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:15.330 WARN 21062 --- [ocp-async-8,b665da6130220275,1bcdeaa119539899] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:15.365 INFO 21062 --- [ocp-async-1,cb3226fb785f6dc0,f5ff6335d20883f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:17.277 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:17.278 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:17.330 WARN 21062 --- [ocp-async-1,d4b3a31f90c59d5e,27cab0e173dc37b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:17.365 INFO 21062 --- [ocp-async-4,71e27e74b2927342,448cb9639f05a94d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:17.983 INFO 21062 --- [prometheus-minute-schedule-1,0fea17d09f9842d4,346be451243503e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:18.446 INFO 21062 --- [ocp-schedules-25,71602b27ce9f0655,0d68d45662a79af7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:48:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:48:18.590204+08:00, lastProcessTime=2024-11-04T14:48:08.591004+08:00 2024-11-04 14:48:18.686 INFO 21062 --- [ocp-async-8,0f8f857dd38c7d4e,54dd25f875ae0c62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:48:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:48:18.590204+08:00 2024-11-04 14:48:19.330 WARN 21062 --- [ocp-async-10,517543de9dec9c50,0189061832f0391e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:19.365 INFO 21062 --- [ocp-async-8,2b66d150772cbfc1,976709f1f2e9066d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:20.278 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:20.279 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:20.961 INFO 21062 --- [ocp-schedules-25,5691e6f94f975092,7c3aedf4d545d158] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:48:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9e52cb9f28a37430,18a32fee6c3d43d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:21.330 WARN 21062 --- [ocp-async-8,dd9f5a362a85f030,e5b08a174c729c08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:21.366 INFO 21062 --- [ocp-async-1,559ab27516d57944,7a4caa345cc9a228] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:23.279 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:23.280 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:23.330 WARN 21062 --- [ocp-async-1,11c78ea7c6b30940,8db7b310f7352c30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:23.366 INFO 21062 --- [ocp-async-4,9a656b6d05be8b91,fd155a8e8f8d2fb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:23.983 INFO 21062 --- [prometheus-minute-schedule-1,1b50a232e7fc022a,e3190f8fdded61c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:25.330 WARN 21062 --- [ocp-async-4,687bbd562daa34e2,4e6da4c6348ca711] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:25.366 INFO 21062 --- [ocp-async-5,188b2f8ef2bc2f94,a4b3375b2d00f02c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:26.280 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:26.281 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:26.983 INFO 21062 --- [prometheus-minute-schedule-1,de083685e8aa333d,15d6f10ac640e1ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:26.984 INFO 21062 --- [prometheus-hour-schedule-1,edb7f6c6e1a8b4fb,fb8d476f7e53425d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:48:27.331 WARN 21062 --- [ocp-async-5,36636b248e3ad260,b26c744e9e0363fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:27.366 INFO 21062 --- [ocp-async-9,35a5dbff052b552a,6498a1d45c986a3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:28.498 INFO 21062 --- [ocp-schedules-26,5374f95d5c015ea6,2f3477edb7dcd4b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:48:28.500 INFO 21062 --- [ocp-schedules-29,142ce35cfb059855,c24df3ea9f36a417] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:48:28.500 INFO 21062 --- [ocp-schedules-24,bc2e9cdbb9492bf6,7b638ab8b2fc0c9b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:48:28.500 INFO 21062 --- [ocp-schedules-10,deb045bc3585a4f6,bfc334b1cab9f673] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:48:28.502 INFO 21062 --- [ocp-schedules-16,749c6ea9d6bfe13c,ff5f181c42b94076] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:48:28.503 INFO 21062 --- [ocp-schedules-26,5374f95d5c015ea6,2f3477edb7dcd4b5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:48:28.504 INFO 21062 --- [ocp-schedules-29,142ce35cfb059855,c24df3ea9f36a417] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:48:28.504 INFO 21062 --- [ocp-schedules-24,bc2e9cdbb9492bf6,7b638ab8b2fc0c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:28.505 INFO 21062 --- [ocp-schedules-9,d12f29e5a6edc85d,d24dca54744dd1a2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:48:28.506 INFO 21062 --- [ocp-schedules-14,ed9cba32c301eb53,4a80ec8dc0144a37] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:48:28.507 INFO 21062 --- [ocp-schedules-10,deb045bc3585a4f6,bfc334b1cab9f673] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:48:28.510 INFO 21062 --- [ocp-schedules-21,4d2f1941e783efb4,713355c4be7f5a42] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:48:28.510 INFO 21062 --- [ocp-schedules-27,8c802c3ed434844c,e36e52b2f2f089a5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:48:28.510 INFO 21062 --- [ocp-schedules-21,4d2f1941e783efb4,713355c4be7f5a42] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:48:28.510 INFO 21062 --- [ocp-schedules-27,8c802c3ed434844c,e36e52b2f2f089a5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:48:28.510 INFO 21062 --- [ocp-schedules-23,3b567dff181f8f13,618c68a593a93faa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:48:28.510 INFO 21062 --- [ocp-schedules-23,3b567dff181f8f13,618c68a593a93faa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:48:28.581 INFO 21062 --- [ocp-schedules-17,a5b7761c2ac4c18a,7209fce117c6a100] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:48:28.581 INFO 21062 --- [ocp-schedules-17,a5b7761c2ac4c18a,7209fce117c6a100] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:48:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:48:28.589584+08:00, lastProcessTime=2024-11-04T14:48:18.590204+08:00 2024-11-04 14:48:28.643 INFO 21062 --- [ocp-schedules-18,d054ca5b63a1c74e,803b848a8df4ecbd] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:48:28.644 INFO 21062 --- [ocp-schedules-18,d054ca5b63a1c74e,803b848a8df4ecbd] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:48:28.686 INFO 21062 --- [ocp-async-3,884c78c60074dc30,96b7a379e8fc3652] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:48:28.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:48:28.589584+08:00 2024-11-04 14:48:28.894 INFO 21062 --- [ocp-schedules-16,ef6fd353e285011e,ddfd038d824a235b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:48:29.210 INFO 21062 --- [ocp-schedules-26,3396f9b980bc4357,d7016e0a62b67be4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:48:29.210 INFO 21062 --- [ocp-schedules-26,3396f9b980bc4357,d7016e0a62b67be4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:48:29.281 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:29.282 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:29.304 INFO 21062 --- [ocp-schedules-19,d0ed6ad4e4871c62,6efd0194039a0a07] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:48:29.331 WARN 21062 --- [ocp-async-7,804c4c4ac0405b47,b80213ec2d4bb661] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:29.336 INFO 21062 --- [ocp-schedules-13,19d8572e47b5a55d,78189da705c964c7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:48:29.341 INFO 21062 --- [ocp-schedules-13,19d8572e47b5a55d,78189da705c964c7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:48:29.341 INFO 21062 --- [ocp-schedules-13,19d8572e47b5a55d,78189da705c964c7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:48:29.366 INFO 21062 --- [ocp-async-3,a3d19b56d21f54a3,9bf3f8cd5f293f2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:29.729 INFO 21062 --- [ocp-schedules-8,af52f863cd285e9e,d5236ba294fc388d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:48:29.729 INFO 21062 --- [ocp-schedules-8,af52f863cd285e9e,d5236ba294fc388d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:48:29.730 INFO 21062 --- [ocp-schedules-8,af52f863cd285e9e,d5236ba294fc388d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:48:29.730 INFO 21062 --- [ocp-schedules-8,af52f863cd285e9e,d5236ba294fc388d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:48:29.730 INFO 21062 --- [ocp-schedules-8,af52f863cd285e9e,d5236ba294fc388d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:48:29.730 INFO 21062 --- [ocp-schedules-8,af52f863cd285e9e,d5236ba294fc388d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:48:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4e6c4ef412f3b32d,c9ab9b5278f4bd3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:48:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:48:30.964 INFO 21062 --- [ocp-schedules-22,50743e92f200a1b7,05575689d3a4226e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:48:31.331 WARN 21062 --- [ocp-async-3,4848e85bb6a18e5d,9a46ecec0f9f2a24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:31.366 INFO 21062 --- [ocp-async-6,42c6d4fa04318801,077c46d3e8c7451e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:32.114 INFO 21062 --- [ocp-schedules-3,b24bd261ae3eeda7,0a270ef1994fb349] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:48:32.121 INFO 21062 --- [ocp-schedules-3,b24bd261ae3eeda7,0a270ef1994fb349] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:48:32.121 INFO 21062 --- [ocp-schedules-3,b24bd261ae3eeda7,0a270ef1994fb349] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:48:32.248 INFO 21062 --- [ocp-schedules-6,66cc8922d02715ed,75bb24d0e28728a4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:48:32.248 INFO 21062 --- [ocp-schedules-6,66cc8922d02715ed,75bb24d0e28728a4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:48:32.276 INFO 21062 --- [ocp-schedules-6,66cc8922d02715ed,75bb24d0e28728a4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:48:32.276 INFO 21062 --- [ocp-schedules-6,66cc8922d02715ed,75bb24d0e28728a4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:48:32.277 INFO 21062 --- [ocp-schedules-9,71550d7eeb639ebc,56ac825a5fe240a4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:48:32.282 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:32.283 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:32.465 INFO 21062 --- [ocp-schedules-20,34fa8981e0d14611,a4e9cded02cd1cb1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:48:32.465+08:00, clusters=[1] 2024-11-04 14:48:32.466 INFO 21062 --- [ocp-schedules-20,34fa8981e0d14611,a4e9cded02cd1cb1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:48:32.678 INFO 21062 --- [ocp-schedules-8,c0b54180505b6ac0,f4bcf177bc3ebe38] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:48:32.678+08:00, clusters=[1] 2024-11-04 14:48:32.681 INFO 21062 --- [ocp-schedules-8,c0b54180505b6ac0,f4bcf177bc3ebe38] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:48:32.690 INFO 21062 --- [ocp-schedules-14,dda9d162feec5084,9361b850db82d8f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:48:32.694 INFO 21062 --- [ocp-schedules-14,dda9d162feec5084,9361b850db82d8f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:32.752 ERROR 21062 --- [ocp-schedules-14,dda9d162feec5084,9361b850db82d8f2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:48:32.752 INFO 21062 --- [ocp-schedules-14,dda9d162feec5084,9361b850db82d8f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:48:32.983 INFO 21062 --- [prometheus-minute-schedule-1,e289c5f090c0488b,79601fa574a051e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:33.331 WARN 21062 --- [ocp-async-6,111c41af37449307,31c1d1b68af3ef16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:33.367 INFO 21062 --- [ocp-async-2,730185abb8987766,7783449bc47237cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:33.990 INFO 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:48:33.990 INFO 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:48:33.990 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:48:33.990 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:48:34.000 INFO 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:48:34.000 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:48:34.001 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:48:34.003 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:48:34.003 WARN 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:48:34.003 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:48:34.003 INFO 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:48:34.004 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:48:34.004 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:48:34.005 WARN 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:48:34.005 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:48:34.005 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:48:34.012 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:48:34.012 INFO 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:48:34.012 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:48:34.026 ERROR 21062 --- [ocp-schedules-25,b9d0f13dae16fb72,0d7a43966e14640e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:48:35.283 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:35.284 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:35.331 WARN 21062 --- [ocp-async-2,0b242370ba79cc60,216fff08c0b1560d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:35.367 INFO 21062 --- [ocp-async-4,8ef01cd9970f9c52,8f8d6264fd804c97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:35.983 INFO 21062 --- [prometheus-minute-schedule-1,40d424a6a46034ad,79d07ba152ceb2b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:37.332 WARN 21062 --- [ocp-async-4,2d79e4c67dab9ad4,f528411045fb5c45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:37.367 INFO 21062 --- [ocp-async-5,380247a0db3286aa,375fa4d0f3c333b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:37.849 INFO 21062 --- [ocp-schedules-12,2954deb14e6d8117,642db5ed4d90b4eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:48:37.850 INFO 21062 --- [ocp-schedules-12,63f5b4508529a300,8d6f8bd28d1f45f5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:48:37.853 INFO 21062 --- [ocp-schedules-12,63f5b4508529a300,8d6f8bd28d1f45f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:37.882 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:48:37.882 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:48:38.284 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:38.285 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:38.504 INFO 21062 --- [ocp-schedules-29,142ce35cfb059855,c24df3ea9f36a417] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:48:38.505 INFO 21062 --- [ocp-schedules-24,bc2e9cdbb9492bf6,7b638ab8b2fc0c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:48:38.531 INFO 21062 --- [ocp-schedules-3,fa92c644d9b9712a,39edc2bf75dfde25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:48:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:48:38.589598+08:00, lastProcessTime=2024-11-04T14:48:28.589584+08:00 2024-11-04 14:48:38.686 INFO 21062 --- [ocp-async-7,cbba4d24ea03ca54,e31c7bc10c12250c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:48:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:48:38.589598+08:00 2024-11-04 14:48:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a987a055b3aa3353,00277750cf7bf8df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:39.332 WARN 21062 --- [ocp-async-9,7806335745ef77d7,3b9d5b08c853f169] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:39.367 INFO 21062 --- [ocp-async-2,eff7368824d9e20e,3624c9f2f7f3053a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:40.004 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:48:40.004 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:48:40.968 INFO 21062 --- [ocp-schedules-4,168428bb52ff66d9,1ee3f9b54538b360] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:48:41.089 INFO 21062 --- [ocp-schedules-23,132416b783cdf054,7188da76ece03d73] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:48:41.093 INFO 21062 --- [ocp-schedules-23,132416b783cdf054,7188da76ece03d73] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:41.180 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:48:41.182 ERROR 21062 --- [ocp-schedules-23,132416b783cdf054,7188da76ece03d73] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:48:41.182 INFO 21062 --- [ocp-schedules-23,132416b783cdf054,7188da76ece03d73] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:48:41.285 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:41.286 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:41.332 WARN 21062 --- [ocp-async-2,080433bf0964734e,ba54a1878678da4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:41.367 INFO 21062 --- [ocp-async-4,88a4704c7c8de104,8692917f64ace139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:48:41.822 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:48:41.822 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=25ms, period=120000ms 2024-11-04 14:48:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6c220fc7746d9e17,33422751b893acbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:43.332 WARN 21062 --- [ocp-async-3,7fe9d1bb6727f1b1,081ea4e7c1abcc33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:43.368 INFO 21062 --- [ocp-async-5,9e7faba1570beae8,a06e561c8cede517] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:44.286 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:44.288 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5089d9a87f6f87f3,9c8d9d9e903c538a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:45.332 WARN 21062 --- [ocp-async-6,024df6c48be369c8,498f313dc06bc78d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:45.368 INFO 21062 --- [ocp-async-1,6fb40c73d84f0cb0,6f6a87d431f2c370] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:47.288 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:47.289 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:47.332 WARN 21062 --- [ocp-async-7,8caefaeb91a8703e,4c16a2304153ad98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:47.368 INFO 21062 --- [ocp-async-10,ac0135ed43957ae5,1ed96c4132f043dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2c908a3eb2a9ae2a,bff0b15a92938693] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:48.571 INFO 21062 --- [ocp-schedules-4,c84ff00392ca7af6,de84e0ad29650969] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:48:48.612 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:48:48.606740+08:00, lastProcessTime=2024-11-04T14:48:38.589598+08:00 2024-11-04 14:48:48.686 INFO 21062 --- [ocp-async-5,b9f3a263eefe45a2,96ae23db3f15971b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:48:48.792 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:48:48.606740+08:00 2024-11-04 14:48:49.153 INFO 21062 --- [ocp-schedules-29,a483b315098c1294,54db6fbd2de3f953] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:48:49.157 INFO 21062 --- [ocp-schedules-29,a483b315098c1294,54db6fbd2de3f953] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:49.158 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:48:49.333 WARN 21062 --- [ocp-async-3,baa2914e56405ae6,5e72ad5b38f4da53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:49.368 INFO 21062 --- [ocp-async-6,bc2670aa7198401e,aa90fe50569593ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:50.289 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:50.291 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:50.970 INFO 21062 --- [ocp-schedules-27,a4873254c865c0f2,8cb23936d17a4079] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:48:50.983 INFO 21062 --- [prometheus-minute-schedule-1,6523e7a9545f2312,31001547c6da4a22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:51.333 WARN 21062 --- [ocp-async-5,c411e78754b5db9d,753c818e80fdf5cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:51.368 INFO 21062 --- [ocp-async-1,7c7562fc570df1d8,f5ea8aaf61b09731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:52.124 INFO 21062 --- [ocp-schedules-13,16442d96c2d09a93,83a71fc6626e81fa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:48:52.129 INFO 21062 --- [ocp-schedules-13,16442d96c2d09a93,83a71fc6626e81fa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:48:52.129 INFO 21062 --- [ocp-schedules-13,16442d96c2d09a93,83a71fc6626e81fa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:48:53.291 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:53.292 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:53.333 WARN 21062 --- [ocp-async-7,3882d7aaa3f167da,f06b2f0fa3c98f33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:53.369 INFO 21062 --- [ocp-async-10,e0dcd0deb08723ac,28816687cc961606] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:53.983 INFO 21062 --- [prometheus-minute-schedule-1,15281b0723dd79fc,a26aaaa0e5ec65d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:55.333 WARN 21062 --- [ocp-async-4,c44fe9a941c07b51,f5a70c72c6c37bc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:55.369 INFO 21062 --- [ocp-async-8,b9d3dc3e5afe170c,cc3b0d9e51a2d5a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:56.292 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:56.294 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:56.983 INFO 21062 --- [prometheus-minute-schedule-1,90022a4b3dee74aa,64eae0895fd849bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:48:57.334 WARN 21062 --- [ocp-async-6,196a641720503d24,3163316887b9c99b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:57.369 INFO 21062 --- [ocp-async-9,107bfbae28e3e020,6d07c534e117c235] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:58.500 INFO 21062 --- [ocp-schedules-9,2cc511a827ad37e5,0726844ba8fd61b8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:48:58.500 INFO 21062 --- [ocp-schedules-30,19300193aacf0a55,e7d0d0849430c842] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:48:58.502 INFO 21062 --- [ocp-schedules-17,fada393ae2607327,5032f4b462b91c9f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:48:58.503 INFO 21062 --- [ocp-schedules-9,2cc511a827ad37e5,0726844ba8fd61b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:48:58.504 INFO 21062 --- [ocp-schedules-30,19300193aacf0a55,e7d0d0849430c842] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:48:58.505 INFO 21062 --- [ocp-schedules-1,693696a5b29a022b,8b10b4f27f1a0d4a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:48:58.509 INFO 21062 --- [ocp-schedules-6,9048597d2a976564,4637049a273b76fc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:48:58.509 INFO 21062 --- [ocp-schedules-4,cb502a648bca677e,75b7dae8c835c86d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:48:58.510 INFO 21062 --- [ocp-schedules-6,9048597d2a976564,4637049a273b76fc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:48:58.510 INFO 21062 --- [ocp-schedules-4,cb502a648bca677e,75b7dae8c835c86d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:48:58.510 INFO 21062 --- [ocp-schedules-23,ca2a49645eb4d4ee,7431db8c221d1047] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:48:58.510 INFO 21062 --- [ocp-schedules-23,ca2a49645eb4d4ee,7431db8c221d1047] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:48:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:48:58.589295+08:00, lastProcessTime=2024-11-04T14:48:48.606740+08:00 2024-11-04 14:48:58.641 INFO 21062 --- [ocp-schedules-11,1e175bab7bdc4b12,0616620186eb6a7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:48:58.686 INFO 21062 --- [ocp-async-10,d679b6063de8e265,ccb30e47aaf1eb1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:48:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:48:58.589295+08:00 2024-11-04 14:48:58.964 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:48:58.964 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:48:58.972 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:48:58.972 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:48:58.972 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:48:58.972 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:48:58.978 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:48:58.978 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:48:58.978 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:48:58.978 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:48:59.016 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:48:59.025 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:48:59.025 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:48:59.025 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:48:59.025 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:48:59.025 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:48:59.076 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:48:59.158 INFO 21062 --- [ocp-schedules-29,a483b315098c1294,54db6fbd2de3f953] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:48:59.294 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:48:59.295 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:48:59.308 INFO 21062 --- [ocp-schedules-15,ea14a20a068cb0f5,bbce6c637ecaabb5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:48:59.334 WARN 21062 --- [ocp-async-7,146719c85676ea61,285d6d234f40a56d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:48:59.369 INFO 21062 --- [ocp-async-4,c17d7b83eb446c29,ba230f193dbab240] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:48:59.622 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:48:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ceeba93387199812,783d89a0c18abac6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730702940, interval=60 2024-11-04 14:49:00.182 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.203748482S 2024-11-04 14:49:00.182 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:49:00.283 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:49:00.284 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:49:00.311 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:49:00.372 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:49:00.372 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:49:00.372 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:49:00.372 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:49:00.372 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:49:00.373 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:49:00.384 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:49:00.560 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] o.s.boot.SpringApplication : Started application in 1.588 seconds (JVM running for 8354.239) 2024-11-04 14:49:00.560 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.588312528S 2024-11-04 14:49:00.973 INFO 21062 --- [ocp-schedules-20,d510ed13987b2f95,0c50b0975a740612] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:49:01.026 INFO 21062 --- [ocp-schedules-5,e4dcd1dd86b78987,6b5ac2c393ea7d91] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:49:01.334 WARN 21062 --- [ocp-async-10,6ffa39c1cf20dfc1,e159afa6b5175d93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:01.369 INFO 21062 --- [ocp-async-8,fa0d727a3923bf7f,011047115292351b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:02.295 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:02.297 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:02.983 INFO 21062 --- [prometheus-minute-schedule-1,46acd91c02f881d1,3a2d5c325ad7febf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:03.334 WARN 21062 --- [ocp-async-6,a529fee638e7a65e,633e88b7caa440ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:03.369 INFO 21062 --- [ocp-async-9,c7ba786a9fc69779,6d16f1d73e74a7b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:05.297 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:05.299 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:05.334 WARN 21062 --- [ocp-async-1,313f9eee603072de,9401296696e64773] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:05.369 INFO 21062 --- [ocp-async-2,c0eb619204b3468c,a2221cd0e8de5de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f0222d483afb3cff,33dd5e1530eae33c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:07.335 WARN 21062 --- [ocp-async-4,902717c3e28035d3,2b8668dced573b5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:07.370 INFO 21062 --- [ocp-async-3,62e0b378dff15434,2c1cd507fb77b28a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:07.883 INFO 21062 --- [ocp-schedules-12,63f5b4508529a300,8d6f8bd28d1f45f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:49:07.883 INFO 21062 --- [ocp-schedules-12,f7a15d9a9bba1b71,deafe17617791cff] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:49:07.887 INFO 21062 --- [ocp-schedules-12,f7a15d9a9bba1b71,deafe17617791cff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:07.917 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:49:07.917 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:49:08.299 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:08.301 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:08.504 INFO 21062 --- [ocp-schedules-9,2cc511a827ad37e5,0726844ba8fd61b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:49:08.504 INFO 21062 --- [ocp-schedules-30,19300193aacf0a55,e7d0d0849430c842] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:49:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:49:08.591841+08:00, lastProcessTime=2024-11-04T14:48:58.589295+08:00 2024-11-04 14:49:08.685 INFO 21062 --- [ocp-schedules-7,c47ce21e749b1966,f8004ce820b5fda8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:49:08.686 INFO 21062 --- [ocp-async-9,9d8662ee66133c52,f44023999b735158] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:49:08.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:49:08.591841+08:00 2024-11-04 14:49:08.992 INFO 21062 --- [prometheus-minute-schedule-1,df59cd26e85af350,a44374675364f653] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:09.335 WARN 21062 --- [ocp-async-6,098d8a249b25181f,4973b97ddf535139] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:09.370 INFO 21062 --- [ocp-async-9,3a378a57d2c64636,319042d7cc27bd1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:10.002 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:49:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:49:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:49:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:49:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:49:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:49:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:49:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:49:10.976 INFO 21062 --- [ocp-schedules-4,ed169e7af8895032,ea1bd918231f9284] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:49:11.301 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:11.304 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:11.335 WARN 21062 --- [ocp-async-9,8d9cef28d6138018,fe9adad60db35170] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:11.370 INFO 21062 --- [ocp-async-2,b95cafee61124df2,99f571d66325ac6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:11.678 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=2, affectRows=2 2024-11-04 14:49:11.678 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=2, validCount=2 2024-11-04 14:49:11.678 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=6, affectRows=6 2024-11-04 14:49:11.678 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=5, validCount=5 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=18, affectRows=18 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=30, affectRows=30 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=18, validCount=18 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=30, validCount=30 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=14, affectRows=14 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=14, validCount=14 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=31, affectRows=31 2024-11-04 14:49:11.679 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=31, validCount=31 2024-11-04 14:49:11.680 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=31, affectRows=31 2024-11-04 14:49:11.680 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=43, affectRows=43 2024-11-04 14:49:11.680 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=31, validCount=31 2024-11-04 14:49:11.680 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=43, validCount=43 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=44, affectRows=44 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=10, affectRows=10 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=43, validCount=43 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=10, validCount=10 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=6, affectRows=6 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=6, validCount=6 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=18, affectRows=18 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=24, affectRows=24 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=58, affectRows=58 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=31, affectRows=31 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702880, nodeCount=56, affectRows=56 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=24, validCount=24 2024-11-04 14:49:11.681 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=18, validCount=18 2024-11-04 14:49:11.682 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=31, validCount=31 2024-11-04 14:49:11.682 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=58, validCount=58 2024-11-04 14:49:11.682 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702880, nodeCount=55, validCount=55 2024-11-04 14:49:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e005110e8962db60,496a54308d2dc07e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:12.132 INFO 21062 --- [ocp-schedules-21,fba07157fa5d0e6d,e42547ddf32747e2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:49:12.139 INFO 21062 --- [ocp-schedules-21,fba07157fa5d0e6d,e42547ddf32747e2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:49:12.139 INFO 21062 --- [ocp-schedules-21,fba07157fa5d0e6d,e42547ddf32747e2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:49:13.335 WARN 21062 --- [ocp-async-2,1342a2f6c968440d,56402cff06157b5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:13.370 INFO 21062 --- [ocp-async-3,d754ceb7073fe059,007baf59374f96c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:13.984 INFO 21062 --- [prometheus-cache-monitor1,28622324ad93a515,fb77c7677c978089] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:49:13.984 INFO 21062 --- [prometheus-cache-monitor1,28622324ad93a515,fb77c7677c978089] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:49:14.305 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:14.306 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a21f09a9d41df2fd,21e3f7292f1c1e24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:15.277 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:45:15.254, end=2024-11-04T14:47:15.274} 2024-11-04 14:49:15.286 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:49:15.286 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:49:15.335 WARN 21062 --- [ocp-async-3,34f039fab4a4af60,f56ec8a56d0d41a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:15.370 INFO 21062 --- [ocp-async-5,e75a46eec26c542f,72ce98f6d59e7e6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:17.306 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:17.307 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:17.335 WARN 21062 --- [ocp-async-5,63b8e31a518251a0,66977e0067c4f90b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:17.370 INFO 21062 --- [ocp-async-7,0e9a255805ec9aca,b489e4afb7f61c56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7d063a4ffbcbc4cf,803744e9a8dd9506] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:49:18.590792+08:00, lastProcessTime=2024-11-04T14:49:08.591841+08:00 2024-11-04 14:49:18.687 INFO 21062 --- [ocp-async-3,ce8bdb6849d49cb4,e657d99c9b12e433] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:49:18.712 INFO 21062 --- [ocp-schedules-14,a65b19909659a261,a2d558fb9baa7007] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:49:18.775 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:49:18.590792+08:00 2024-11-04 14:49:19.336 WARN 21062 --- [ocp-async-2,b9e5c0a2336811ce,6fbd47d2e6991dca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:19.371 INFO 21062 --- [ocp-async-3,8382d18fbaf09147,da21497025b691e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:20.307 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:20.309 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:20.979 INFO 21062 --- [ocp-schedules-14,188a1df61089f728,03c413503b693123] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:49:20.983 INFO 21062 --- [prometheus-minute-schedule-1,822fb08f1085ce55,64afd5e7c86209ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:21.336 WARN 21062 --- [ocp-async-3,542a9817d3ee98cb,6edf22d85a3bb440] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:21.371 INFO 21062 --- [ocp-async-5,fb16277d768ee79f,df9e4f7e86ee5f68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:23.309 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:23.310 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:23.336 WARN 21062 --- [ocp-async-5,f3c8ca304aa3e777,a99314f4cf57a936] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:23.371 INFO 21062 --- [ocp-async-7,776739bf98eb4443,6a4b8939dc9810a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:23.983 INFO 21062 --- [prometheus-minute-schedule-1,247207da7d1fc1fb,8d5e7dcfa6abc633] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:25.336 WARN 21062 --- [ocp-async-7,1b69c28faf997453,89f68017f3e9a676] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:25.371 INFO 21062 --- [ocp-async-4,9dba599f11403258,186439a2144bf6a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:25.514 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3f8ccc2d324555e0,acf563e2a2ddb986] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=3f8ccc2d324555e0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:49:25.523 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,3f8ccc2d324555e0,acf563e2a2ddb986] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=3f8ccc2d324555e0, duration=23 ms] 2024-11-04 14:49:26.310 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:26.312 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:26.983 INFO 21062 --- [prometheus-minute-schedule-1,27b2a7e0bac6a910,bcaec69c8270f306] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:26.984 INFO 21062 --- [prometheus-hour-schedule-1,15d6b14971f5395b,8ad24238d7a57445] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:49:27.336 WARN 21062 --- [ocp-async-4,14380288379a0a27,dd3130538a2da118] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:27.371 INFO 21062 --- [ocp-async-6,adf84738c1b9b7d0,c22fe47780a97699] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:28.498 INFO 21062 --- [ocp-schedules-23,6f5bf0f6d92a2a2b,179251379233d72c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:49:28.499 INFO 21062 --- [ocp-schedules-13,d76539e4697b7131,a83505542232c7d3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:49:28.500 INFO 21062 --- [ocp-schedules-21,205066ec2e663faf,d5943b51330d599b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:49:28.500 INFO 21062 --- [ocp-schedules-7,d6fc6b3f20f0d240,876867dd8abb21d0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:49:28.502 INFO 21062 --- [ocp-schedules-23,6f5bf0f6d92a2a2b,179251379233d72c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:49:28.502 INFO 21062 --- [ocp-schedules-19,9452be7616626747,80809f7ab40b0039] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:49:28.503 INFO 21062 --- [ocp-schedules-13,d76539e4697b7131,a83505542232c7d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:28.503 INFO 21062 --- [ocp-schedules-7,d6fc6b3f20f0d240,876867dd8abb21d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:49:28.504 INFO 21062 --- [ocp-schedules-24,42219dd9ac9de5f6,362cca3951fbb1da] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:49:28.505 INFO 21062 --- [ocp-schedules-21,205066ec2e663faf,d5943b51330d599b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:49:28.510 INFO 21062 --- [ocp-schedules-29,74faea750a8f4e02,82231230ffadbb8a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:49:28.510 INFO 21062 --- [ocp-schedules-6,b1db35a818028be2,b9b3505b3c8335b4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:49:28.510 INFO 21062 --- [ocp-schedules-6,b1db35a818028be2,b9b3505b3c8335b4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:49:28.510 INFO 21062 --- [ocp-schedules-29,74faea750a8f4e02,82231230ffadbb8a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:49:28.510 INFO 21062 --- [ocp-schedules-1,4139cd97be9128ba,802306f94a0d52d2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:49:28.510 INFO 21062 --- [ocp-schedules-1,4139cd97be9128ba,802306f94a0d52d2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:49:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:49:28.590132+08:00, lastProcessTime=2024-11-04T14:49:18.590792+08:00 2024-11-04 14:49:28.602 INFO 21062 --- [ocp-schedules-5,aedfd898bb90793d,e6b5976c98ea46b4] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:49:28.602+08:00, tenantSize=0 2024-11-04 14:49:28.690 INFO 21062 --- [ocp-async-10,5c2549074551320d,b0f4c20a113a117d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:49:28.755 INFO 21062 --- [ocp-schedules-27,d3d2950a8ddda7c6,5722d48617f96bfe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:49:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:49:28.590132+08:00 2024-11-04 14:49:28.896 INFO 21062 --- [ocp-schedules-15,3fdfa38620a0b29b,ded9990409e0616c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:49:29.214 INFO 21062 --- [ocp-schedules-29,ce2dc10d87e05ad0,24cbaeda3f97b676] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:49:29.214 INFO 21062 --- [ocp-schedules-29,ce2dc10d87e05ad0,24cbaeda3f97b676] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:49:29.310 INFO 21062 --- [ocp-schedules-5,13240087fe7389be,9f2e58cf46b3200e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:49:29.312 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:29.313 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:29.337 WARN 21062 --- [ocp-async-9,fbae79a7a161a9a2,4e1cba4e281c4fe6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:29.341 INFO 21062 --- [ocp-schedules-25,a09b2f49412f4117,01be2f677efb96a7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:49:29.346 INFO 21062 --- [ocp-schedules-25,a09b2f49412f4117,01be2f677efb96a7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:49:29.346 INFO 21062 --- [ocp-schedules-25,a09b2f49412f4117,01be2f677efb96a7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:49:29.371 INFO 21062 --- [ocp-async-10,77363d2a7f3bd8be,488f57da36e4e4ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:29.550 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:49:29.550 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:49:29.559 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:49:29.559 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:49:29.559 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:49:29.570 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:49:29.570 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:49:29.570 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:49:29.574 INFO 21062 --- [ocp-schedules-17,893e6fe530be512c,586f9919534a5a88] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:49:29.731 INFO 21062 --- [ocp-schedules-9,a6c1a8f334d57e24,a15fa1fb94fd426f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:49:29.731 INFO 21062 --- [ocp-schedules-9,a6c1a8f334d57e24,a15fa1fb94fd426f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:49:29.738 INFO 21062 --- [ocp-schedules-9,a6c1a8f334d57e24,a15fa1fb94fd426f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:49:29.738 INFO 21062 --- [ocp-schedules-9,a6c1a8f334d57e24,a15fa1fb94fd426f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:49:29.743 INFO 21062 --- [ocp-schedules-9,a6c1a8f334d57e24,a15fa1fb94fd426f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:49:29.743 INFO 21062 --- [ocp-schedules-9,a6c1a8f334d57e24,a15fa1fb94fd426f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:49:29.983 INFO 21062 --- [prometheus-minute-schedule-1,1ee1661e68673578,94724e783ef65b7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:49:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:49:30.982 INFO 21062 --- [ocp-schedules-10,ac40d4357a9aa108,0e4183b801624fe6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:49:31.337 WARN 21062 --- [ocp-async-10,f8c62d07ac02a7a5,7522a9667d51ec19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:31.372 INFO 21062 --- [ocp-async-3,252a9ac164b33473,bf325b27002a823b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:32.141 INFO 21062 --- [ocp-schedules-22,ff4514ad8e086a87,d90ee0ed3a92f853] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:49:32.147 INFO 21062 --- [ocp-schedules-22,ff4514ad8e086a87,d90ee0ed3a92f853] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:49:32.147 INFO 21062 --- [ocp-schedules-22,ff4514ad8e086a87,d90ee0ed3a92f853] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:49:32.276 INFO 21062 --- [ocp-schedules-8,237c90a928155270,9e05130e79ab6b3f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:49:32.276 INFO 21062 --- [ocp-schedules-8,237c90a928155270,9e05130e79ab6b3f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:49:32.303 INFO 21062 --- [ocp-schedules-8,237c90a928155270,9e05130e79ab6b3f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:49:32.303 INFO 21062 --- [ocp-schedules-8,237c90a928155270,9e05130e79ab6b3f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:49:32.304 INFO 21062 --- [ocp-schedules-10,faff5c6541ff3688,af4a492c76a27788] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:49:32.313 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:32.314 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:32.492 INFO 21062 --- [ocp-schedules-19,2bdac294e275aa9f,ae4149d6bbdc3ffc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:49:32.492+08:00, clusters=[1] 2024-11-04 14:49:32.493 INFO 21062 --- [ocp-schedules-19,2bdac294e275aa9f,ae4149d6bbdc3ffc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:49:32.709 INFO 21062 --- [ocp-schedules-25,ec332f3df8fc4339,be1bea2ea1d2b10b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:49:32.709+08:00, clusters=[1] 2024-11-04 14:49:32.712 INFO 21062 --- [ocp-schedules-25,ec332f3df8fc4339,be1bea2ea1d2b10b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:49:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3085f4027e5121b7,8175af6d96e252b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:33.337 WARN 21062 --- [ocp-async-3,dbd42a459404d7b7,9daeb45566079777] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:33.372 INFO 21062 --- [ocp-async-5,1d9503827ef265ab,dc2e7d977c6b1a4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:34.036 INFO 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:49:34.036 INFO 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:49:34.036 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:49:34.036 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:49:34.046 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:49:34.046 INFO 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:49:34.047 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:49:34.048 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:49:34.048 WARN 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:49:34.048 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:49:34.049 INFO 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:49:34.049 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:49:34.049 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:49:34.050 WARN 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:49:34.050 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:49:34.050 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:49:34.058 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:49:34.058 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:49:34.058 INFO 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:49:34.069 ERROR 21062 --- [ocp-schedules-23,f2ac585a3b40025c,9bce331c549c50ce] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:49:35.314 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:35.316 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:35.337 WARN 21062 --- [ocp-async-5,f8b009f5c094c95d,55735b655a2c262b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:35.372 INFO 21062 --- [ocp-async-7,3bc817a887fd1963,cf14a6204ee4af5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:35.983 INFO 21062 --- [prometheus-minute-schedule-1,70ea1f5dfbafa4c7,db3de77a6c08ddd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:37.337 WARN 21062 --- [ocp-async-7,63ec5605a674055f,99f8ad4412f72e3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:37.372 INFO 21062 --- [ocp-async-4,43f1d6db0260343e,7c59ca0287b29d76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:37.917 INFO 21062 --- [ocp-schedules-12,f7a15d9a9bba1b71,deafe17617791cff] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:49:37.917 INFO 21062 --- [ocp-schedules-12,d9ce779dbad1b176,de59c11751e85249] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:49:37.921 INFO 21062 --- [ocp-schedules-12,d9ce779dbad1b176,de59c11751e85249] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:37.949 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:49:37.949 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:49:38.316 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:38.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:38.504 INFO 21062 --- [ocp-schedules-7,d6fc6b3f20f0d240,876867dd8abb21d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:49:38.504 INFO 21062 --- [ocp-schedules-13,d76539e4697b7131,a83505542232c7d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:49:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:49:38.590134+08:00, lastProcessTime=2024-11-04T14:49:28.590132+08:00 2024-11-04 14:49:38.687 INFO 21062 --- [ocp-async-9,07f98721237bbea9,86b5d49b45861c94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:49:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:49:38.590134+08:00 2024-11-04 14:49:38.779 INFO 21062 --- [ocp-schedules-23,7186cc143d9a0da4,bd809312fba8ef50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:49:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0c59c201d1aa65d9,37506cc4f06b580b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:39.338 WARN 21062 --- [ocp-async-6,791871ac283cf243,fbffb1df3f78fa75] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:39.373 INFO 21062 --- [ocp-async-5,d128fd5065d1dfa6,f1de2d42b14a306f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:49:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:49:40.986 INFO 21062 --- [ocp-schedules-25,2393c19303680902,1df6d2528bc7e2c3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:49:41.182 INFO 21062 --- [ocp-schedules-26,f3f1c4d6e884d8df,839a5de4dbfb8826] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:49:41.189 INFO 21062 --- [ocp-schedules-26,f3f1c4d6e884d8df,839a5de4dbfb8826] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:41.276 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:49:41.281 ERROR 21062 --- [ocp-schedules-26,f3f1c4d6e884d8df,839a5de4dbfb8826] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:49:41.281 INFO 21062 --- [ocp-schedules-26,f3f1c4d6e884d8df,839a5de4dbfb8826] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:49:41.318 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:41.319 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:41.338 WARN 21062 --- [ocp-async-9,13d95728716ac7f9,eb14aabdd3066657] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:41.373 INFO 21062 --- [ocp-async-7,4cbe17390b2d78af,243cd2842b928bde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:41.983 INFO 21062 --- [prometheus-minute-schedule-1,e77465a33d1ff89c,ee52980f57ad4d4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:43.338 WARN 21062 --- [ocp-async-10,f1b15eb8f29fddef,8f2dc13e90fab6cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:43.373 INFO 21062 --- [ocp-async-4,15bc8a70f856c5cc,39992ef2fa5ae53d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:44.319 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:44.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:44.983 INFO 21062 --- [prometheus-minute-schedule-1,65e428f433e7e892,df483d001eb1763b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:45.338 WARN 21062 --- [ocp-async-3,8a7bd0a4534cf1a8,8dfdc71da51db79a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:45.373 INFO 21062 --- [ocp-async-1,08092bda519ce98c,0fa5df7b33ce441a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:47.321 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:47.322 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:47.338 WARN 21062 --- [ocp-async-5,b5cf19ee7ef29cf8,c8ca32630d601218] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:47.373 INFO 21062 --- [ocp-async-2,650c58045f919393,a620160173977577] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:47.983 INFO 21062 --- [prometheus-minute-schedule-1,79f04fd17b058c91,61c182cf543496d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:49:48.591419+08:00, lastProcessTime=2024-11-04T14:49:38.590134+08:00 2024-11-04 14:49:48.687 INFO 21062 --- [ocp-async-4,45a207a15eec8c15,9c404d8264b5e381] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:49:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:49:48.591419+08:00 2024-11-04 14:49:48.821 INFO 21062 --- [ocp-schedules-29,ef623ca72f8d9319,4089a24c266f8c68] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:49:49.338 WARN 21062 --- [ocp-async-10,2a19106d311bcd7d,ed4abbad89a1144d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:49.374 INFO 21062 --- [ocp-async-3,c6b39f045401822e,72fb701ea1237586] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:50.323 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:50.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:50.983 INFO 21062 --- [prometheus-minute-schedule-1,6eb0ae9deea5c105,d9c1a7097e545f1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:50.989 INFO 21062 --- [ocp-schedules-26,84c1bd0697a69131,dfc3845dd68800a4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:49:51.339 WARN 21062 --- [ocp-async-4,de44911d0ade9c3b,f48f523770ddbe63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:51.374 INFO 21062 --- [ocp-async-1,862ff398b15ff2df,49dea1168be76569] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:52.150 INFO 21062 --- [ocp-schedules-13,635c9476dc4c58e2,ce0ab137999d6958] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:49:52.155 INFO 21062 --- [ocp-schedules-13,635c9476dc4c58e2,ce0ab137999d6958] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:49:52.155 INFO 21062 --- [ocp-schedules-13,635c9476dc4c58e2,ce0ab137999d6958] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:49:53.324 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:53.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:53.339 WARN 21062 --- [ocp-async-5,49d12a113f4635df,9697d23657627142] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:53.374 INFO 21062 --- [ocp-async-2,f5387bddd9354886,bbf88266031a07a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:53.983 INFO 21062 --- [prometheus-minute-schedule-1,871dfaa4513e2301,4bb7a31b4d5e0cef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:55.339 WARN 21062 --- [ocp-async-7,a1719babdaddc29c,11e06beb2c5f5ffb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:55.374 INFO 21062 --- [ocp-async-8,d42196da83ac2484,146f3da207382257] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:56.327 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:56.328 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ba6701ec4604404b,e7739162bf69da74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:49:57.339 WARN 21062 --- [ocp-async-3,7617dcba3e32ed2a,906b530739a01e82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:57.374 INFO 21062 --- [ocp-async-6,314d5eecdff903b0,d58ada16f6e03c64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:58.500 INFO 21062 --- [ocp-schedules-7,d3fa29a2fcd51843,1772d184ebe163ea] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:49:58.500 INFO 21062 --- [ocp-schedules-21,e1a1273c489d1d33,33a02db87433aa34] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:49:58.502 INFO 21062 --- [ocp-schedules-24,cc37faf7e454dac1,3ae1a7095267bbe4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:49:58.503 INFO 21062 --- [ocp-schedules-7,d3fa29a2fcd51843,1772d184ebe163ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:49:58.504 INFO 21062 --- [ocp-schedules-21,e1a1273c489d1d33,33a02db87433aa34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:58.505 INFO 21062 --- [ocp-schedules-19,a15e5cd51dd2b21b,03dd63629eea8e21] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:49:58.510 INFO 21062 --- [ocp-schedules-17,7daf76442f4206b9,ea4016f3af1e9c75] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:49:58.510 INFO 21062 --- [ocp-schedules-27,c86d29515d294c54,3116506488c5ce93] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:49:58.510 INFO 21062 --- [ocp-schedules-1,e23f3225cad24a2c,e94650ab043b54fa] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:49:58.510 INFO 21062 --- [ocp-schedules-1,e23f3225cad24a2c,e94650ab043b54fa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:49:58.510 INFO 21062 --- [ocp-schedules-17,7daf76442f4206b9,ea4016f3af1e9c75] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:49:58.510 INFO 21062 --- [ocp-schedules-27,c86d29515d294c54,3116506488c5ce93] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:49:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:49:58.590038+08:00, lastProcessTime=2024-11-04T14:49:48.591419+08:00 2024-11-04 14:49:58.687 INFO 21062 --- [ocp-async-2,0ed09c833a3a9b24,87b660d03c4792e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:49:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:49:58.590038+08:00 2024-11-04 14:49:58.846 INFO 21062 --- [ocp-schedules-29,9e1bc33dc0017bd3,981c8f4e646b834e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:49:59.158 INFO 21062 --- [ocp-schedules-4,eb7ecdb1907c30ec,877b57ac52eb1b80] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:49:59.162 INFO 21062 --- [ocp-schedules-4,eb7ecdb1907c30ec,877b57ac52eb1b80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:49:59.163 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:49:59.313 INFO 21062 --- [ocp-schedules-11,11f9ba43233a222a,cd406942166fc468] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:49:59.328 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:49:59.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:49:59.339 WARN 21062 --- [ocp-async-5,dcaf0c172be54c17,6b037738730b9e36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:49:59.374 INFO 21062 --- [ocp-async-2,d3c699ab13d7f1ee,d29e8cdc0dbe8487] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:49:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2ba88070fa8fdedc,5eb900d2997b2f8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703000, interval=60 2024-11-04 14:50:00.979 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e00eef7610cb2051] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:45:00.0, nextRun=2024-11-04 14:50:00.0, currentDate=Mon Nov 04 14:50:00 CST 2024 2024-11-04 14:50:00.979 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e00eef7610cb2051] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:45:00.0, nextRunTime=2024-11-04 14:50:00.0} 2024-11-04 14:50:00.991 INFO 21062 --- [ocp-schedules-2,9041a1f40d1de55f,51b0accabd93a06f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:50:01.006 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e00eef7610cb2051] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:50:01.038 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e00eef7610cb2051] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:50:01.038 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,e00eef7610cb2051] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:50:01.339 WARN 21062 --- [ocp-async-7,3badfab68d307485,ab6d0d19f838a3a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:01.375 INFO 21062 --- [ocp-async-8,8eb121edca1cba31,4823ec2733b43eb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:02.329 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:02.331 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:02.983 INFO 21062 --- [prometheus-minute-schedule-1,775a941182035a40,17de7c394338146e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:03.340 WARN 21062 --- [ocp-async-3,85b5e77cfdff0e08,975c1779f91d1332] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:03.375 INFO 21062 --- [ocp-async-6,2504b87ab6a63b03,58d1c38c7870be6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:04.095 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,775e67d55886a0c5] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:40:04.0, nextRun=2024-11-04 14:50:04.0, currentDate=Mon Nov 04 14:50:04 CST 2024 2024-11-04 14:50:04.096 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,775e67d55886a0c5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 14:40:04.0, nextRunTime=2024-11-04 14:50:04.0} 2024-11-04 14:50:04.105 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,775e67d55886a0c5] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 14:50:04.105 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,775e67d55886a0c5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T14:50:04.105+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 14:50:04.525 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,98dadf1f52c57882] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 83, operation:EXECUTE, state: RUNNING 2024-11-04 14:50:04.544 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=83, context=Context{parallelIdx=-1, stringMap={task_instance_id=19, task_operation=execute, latest_execution_start_time=2024-11-04T14:50:04.517+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=83}, listMap={}}, executor=10.46.128.24 2024-11-04 14:50:04.550 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:50:04.550 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:50:04.552 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.553 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:50:04.554 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.554 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.555 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.555 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:50:04.573 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.573 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.574 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.574 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:50:04.588 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.589 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.589 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.590 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:50:04.604 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.604 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.605 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.606 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:50:04.619 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.620 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.620 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.621 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:50:04.626 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.627 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.627 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.628 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:50:04.632 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.633 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.633 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.633 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:50:04.638 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.638 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.639 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.639 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:50:04.645 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.645 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.646 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.646 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:50:04.652 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.653 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.653 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.654 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:50:04.659 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.660 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.660 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.660 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:50:04.669 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.669 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.670 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.670 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:50:04.684 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.684 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.685 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.685 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:50:04.691 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.691 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.692 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.692 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:50:04.698 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.698 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.699 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.699 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:50:04.705 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.705 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.706 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.706 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:50:04.711 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.711 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.712 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.712 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:50:04.717 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.718 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.718 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.719 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:50:04.723 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.723 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.724 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.724 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:50:04.729 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.729 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.730 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.730 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:50:04.734 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.734 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.734 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.735 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:50:04.739 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.740 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.740 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.741 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:50:04.745 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.745 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.745 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.746 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:50:04.750 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.751 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.751 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.752 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:50:04.756 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.756 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.757 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.757 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:50:04.761 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.762 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.762 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.763 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:50:04.767 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.767 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.768 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.768 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:50:04.773 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.773 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.773 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.774 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:50:04.778 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.779 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.779 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.780 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:50:04.784 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.784 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.785 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.785 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:50:04.789 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.790 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.790 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.791 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:50:04.795 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.795 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.796 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.796 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:50:04.801 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.801 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.802 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.802 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:50:04.806 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.807 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.807 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.808 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:50:04.815 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.815 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.815 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.816 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:50:04.822 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.822 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.823 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.823 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:50:04.829 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.829 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.830 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.830 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:50:04.836 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.836 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.837 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.837 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:50:04.841 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.841 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.842 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.842 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:50:04.845 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.846 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.846 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.846 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:50:04.850 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.850 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.851 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.851 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:50:04.854 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.855 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.855 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.856 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:50:04.865 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.865 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.866 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.866 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:50:04.887 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.887 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.888 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.889 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:50:04.895 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.895 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.896 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.896 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:50:04.902 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.902 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.903 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.903 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:50:04.909 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.909 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.910 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.910 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:50:04.914 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.914 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.915 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.915 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:50:04.919 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.919 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.920 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.921 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:50:04.925 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:50:04.925 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:50:04.927 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.927 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:50:04.928 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.928 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.929 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.929 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:50:04.934 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.935 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:50:04.935 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:04.936 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:50:04.940 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:50:04.949 INFO 21062 --- [subtask-executor16,dd193f8f67a5a423,0b8e44b26aa0255e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 83, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 14:50:05.007 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:19, status:SUCCESS 2024-11-04 14:50:05.008 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:19, status:SUCCESS 2024-11-04 14:50:05.331 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:05.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:05.340 WARN 21062 --- [ocp-async-1,6f1d0ca38ea1bcf6,c4a57f28db4bfddc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:05.375 INFO 21062 --- [ocp-async-9,2c19b6bc2910f346,6af286b206392ee3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c5888934dd774d72,e10b661a55488f71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:07.340 WARN 21062 --- [ocp-async-2,941461f8a5bab565,6562fee013297e63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:07.375 INFO 21062 --- [ocp-async-10,ce07e40f42cb3277,1e87b70a0c9e23a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:07.950 INFO 21062 --- [ocp-schedules-12,d9ce779dbad1b176,de59c11751e85249] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:50:07.950 INFO 21062 --- [ocp-schedules-12,180c6cbe3d1f4c76,a151f967d6c81ce2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:50:07.955 INFO 21062 --- [ocp-schedules-12,180c6cbe3d1f4c76,a151f967d6c81ce2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:07.986 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:50:07.986 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:50:08.332 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:08.334 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:08.504 INFO 21062 --- [ocp-schedules-7,d3fa29a2fcd51843,1772d184ebe163ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:50:08.505 INFO 21062 --- [ocp-schedules-21,e1a1273c489d1d33,33a02db87433aa34] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:50:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:50:08.589677+08:00, lastProcessTime=2024-11-04T14:49:58.590038+08:00 2024-11-04 14:50:08.688 INFO 21062 --- [ocp-async-6,ff91c8a962fa25d4,7a1430ebac9d2b4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:50:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:50:08.589677+08:00 2024-11-04 14:50:08.889 INFO 21062 --- [ocp-schedules-28,a572e5394f4eb0ff,6c409398e7f658f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:50:08.983 INFO 21062 --- [prometheus-minute-schedule-1,c8546a813d366b4e,0cbbb3905c82a2fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:09.163 INFO 21062 --- [ocp-schedules-4,eb7ecdb1907c30ec,877b57ac52eb1b80] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:50:09.340 WARN 21062 --- [ocp-async-3,4af5d3471487e2ff,7740cb073de8c53c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:09.375 INFO 21062 --- [ocp-async-6,14961e591197c90e,1585b9997aff6509] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:50:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:50:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:50:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:50:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:50:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:50:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:50:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:50:10.994 INFO 21062 --- [ocp-schedules-22,b2b0096357c1a7cd,76000de3e5c5e6a7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:50:11.334 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:11.335 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:11.340 WARN 21062 --- [ocp-async-6,762248363b47dc55,76df5095852f71d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:11.376 INFO 21062 --- [ocp-async-9,92b96f81beffad17,f3d88da837158c2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:11.669 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=3, affectRows=3 2024-11-04 14:50:11.669 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=3, validCount=3 2024-11-04 14:50:11.669 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=5, affectRows=5 2024-11-04 14:50:11.669 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=5, validCount=5 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=26, affectRows=26 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=11, affectRows=11 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=28, affectRows=28 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=11, validCount=11 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=26, validCount=26 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=17, affectRows=17 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=28, validCount=28 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=17, validCount=17 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=33, affectRows=33 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=35, affectRows=35 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=33, validCount=33 2024-11-04 14:50:11.670 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=34, validCount=34 2024-11-04 14:50:11.671 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=54, affectRows=54 2024-11-04 14:50:11.671 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=62, affectRows=62 2024-11-04 14:50:11.671 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=75, affectRows=75 2024-11-04 14:50:11.671 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=54, validCount=54 2024-11-04 14:50:11.671 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=62, validCount=62 2024-11-04 14:50:11.671 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730702940, nodeCount=73, affectRows=73 2024-11-04 14:50:11.671 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=75, validCount=75 2024-11-04 14:50:11.672 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730702940, nodeCount=71, validCount=71 2024-11-04 14:50:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a860d5e518c24c53,91f3ec86976bfd6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:12.158 INFO 21062 --- [ocp-schedules-27,b1c1fcba76ead568,3081b492da7c3d92] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:50:12.164 INFO 21062 --- [ocp-schedules-27,b1c1fcba76ead568,3081b492da7c3d92] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:50:12.164 INFO 21062 --- [ocp-schedules-27,b1c1fcba76ead568,3081b492da7c3d92] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:50:13.341 WARN 21062 --- [ocp-async-9,5dcadc7c75e9b9cb,e2501977c109eb7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:13.376 INFO 21062 --- [ocp-async-10,b94a46e759b0b10b,507862d1264cd188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:13.984 INFO 21062 --- [prometheus-cache-monitor1,7f8b7c757450eff6,787da0d4e5a482c0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:50:13.984 INFO 21062 --- [prometheus-cache-monitor1,7f8b7c757450eff6,787da0d4e5a482c0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:50:14.335 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:14.336 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:14.983 INFO 21062 --- [prometheus-minute-schedule-1,66f4340dea70b5c3,ed9d217765ee881b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:15.341 WARN 21062 --- [ocp-async-10,d99f3c4224195fdf,fb41cf47a8868a30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:15.376 INFO 21062 --- [ocp-async-4,0b79b4a0c40a4fe9,1a6364560681858e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:17.336 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:17.338 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:17.341 WARN 21062 --- [ocp-async-4,84f021211e37c986,48e1773658648e93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:17.376 INFO 21062 --- [ocp-async-5,f8ee26441b562bb8,6005ea929ae26d0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:17.983 INFO 21062 --- [prometheus-minute-schedule-1,dbede43ede9189ce,f1381f11dd215489] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:50:18.590263+08:00, lastProcessTime=2024-11-04T14:50:08.589677+08:00 2024-11-04 14:50:18.688 INFO 21062 --- [ocp-async-10,ceabf93365702053,d04b35bdaa3eabd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:50:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:50:18.590263+08:00 2024-11-04 14:50:18.913 INFO 21062 --- [ocp-schedules-30,1efc8de986035a45,b61afe5f8ba33749] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:50:19.341 WARN 21062 --- [ocp-async-9,f0d9860d5f2fc388,ce7f4a2b0c2c839e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:19.376 INFO 21062 --- [ocp-async-10,028ed60cd5134ea6,adde2227e36bbd61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:20.338 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:20.339 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:20.983 INFO 21062 --- [prometheus-minute-schedule-1,1d168ba3a307565d,0664a7a1ad3cab3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:20.996 INFO 21062 --- [ocp-schedules-13,1f3cfa3acbd3e381,95d5745d9f99842f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:50:21.341 WARN 21062 --- [ocp-async-10,4baf781df63e9302,1322191200178789] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:21.376 INFO 21062 --- [ocp-async-4,447afa310f58c61b,c5ad390032ed7f42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:23.339 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:23.340 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:23.342 WARN 21062 --- [ocp-async-4,11d742fa96282c2c,7c00a567b03d5ac5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:23.377 INFO 21062 --- [ocp-async-5,153bc04ad79d2429,08af701cb3fdf71b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4c7fc83f12beb4bc,11a2a65bc5ec1f87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:25.342 WARN 21062 --- [ocp-async-5,2779f9e594e6edc3,1537a92336a32c6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:25.377 INFO 21062 --- [ocp-async-2,aa13f44eba3f4693,a8586f8f55e60b45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:26.340 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:26.341 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:26.983 INFO 21062 --- [prometheus-minute-schedule-1,a48c768d66ec314e,505e72360a5051d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:26.984 INFO 21062 --- [prometheus-hour-schedule-1,67941c3dbc1d62b0,d14121fbe37904ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:50:27.345 WARN 21062 --- [ocp-async-2,7ccfd685a7887f04,531d514729ef8776] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:27.377 INFO 21062 --- [ocp-async-8,e061ac33e8634fd9,a83877904f4934ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:28.498 INFO 21062 --- [ocp-schedules-7,be1da53d45bc9e9b,5d44691dd5d3dc4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:50:28.500 INFO 21062 --- [ocp-schedules-22,8e652d48df92c7f8,19a844bee6403d3a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:50:28.500 INFO 21062 --- [ocp-schedules-5,b857781091ffdb28,8d1b52d544de87bd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:50:28.502 INFO 21062 --- [ocp-schedules-25,e287d1b1630739b5,a350e3781f7d6618] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:50:28.502 INFO 21062 --- [ocp-schedules-21,4e0e6736858cf971,f8ee73d94eced4ae] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:50:28.504 INFO 21062 --- [ocp-schedules-7,be1da53d45bc9e9b,5d44691dd5d3dc4e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:50:28.504 INFO 21062 --- [ocp-schedules-22,8e652d48df92c7f8,19a844bee6403d3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:28.504 INFO 21062 --- [ocp-schedules-5,b857781091ffdb28,8d1b52d544de87bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:50:28.505 INFO 21062 --- [ocp-schedules-23,7c55e275d71aab95,0116833d41a34380] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:50:28.508 INFO 21062 --- [ocp-schedules-25,e287d1b1630739b5,a350e3781f7d6618] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:50:28.510 INFO 21062 --- [ocp-schedules-30,8a0a9755db0c9c99,c80fe1c676218767] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:50:28.510 INFO 21062 --- [ocp-schedules-9,ca7535fc18a82aa1,95eca08ea7e51590] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:50:28.510 INFO 21062 --- [ocp-schedules-30,8a0a9755db0c9c99,c80fe1c676218767] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:50:28.510 INFO 21062 --- [ocp-schedules-9,ca7535fc18a82aa1,95eca08ea7e51590] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:50:28.510 INFO 21062 --- [ocp-schedules-18,07069765489ca07e,6eff66bb74a96e8a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:50:28.510 INFO 21062 --- [ocp-schedules-18,07069765489ca07e,6eff66bb74a96e8a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:50:28.581 INFO 21062 --- [ocp-schedules-1,22eb873143bc83bf,9a06e4113d97e8fd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:50:28.581 INFO 21062 --- [ocp-schedules-1,22eb873143bc83bf,9a06e4113d97e8fd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:50:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:50:28.589699+08:00, lastProcessTime=2024-11-04T14:50:18.590263+08:00 2024-11-04 14:50:28.688 INFO 21062 --- [ocp-async-7,5bb52a39ae04ddf8,624bb3d8698317cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:50:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:50:28.589699+08:00 2024-11-04 14:50:28.874 INFO 21062 --- [ocp-schedules-24,497f033b7c078a9f,f6bee433ee6220c4] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:50:28.874 INFO 21062 --- [ocp-schedules-24,497f033b7c078a9f,f6bee433ee6220c4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:50:28.899 INFO 21062 --- [ocp-schedules-17,37150bd2c45cf9fd,64022afa90de36f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:50:28.955 INFO 21062 --- [ocp-schedules-10,2f04059a599263b8,ef8e63ab2e4f702b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:50:29.218 INFO 21062 --- [ocp-schedules-23,540f385b516451ed,3ed45680caa60fde] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:50:29.218 INFO 21062 --- [ocp-schedules-23,540f385b516451ed,3ed45680caa60fde] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:50:29.316 INFO 21062 --- [ocp-schedules-30,6ed0ef5c7ad9b032,74ff2808ca3b5531] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:50:29.341 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:29.342 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:29.345 WARN 21062 --- [ocp-async-6,5217179a1b3d9b6c,c52178e52307b970] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:29.346 INFO 21062 --- [ocp-schedules-14,cd84c36a3efd344b,2bf82a30177fa1ce] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:50:29.352 INFO 21062 --- [ocp-schedules-14,cd84c36a3efd344b,2bf82a30177fa1ce] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:50:29.352 INFO 21062 --- [ocp-schedules-14,cd84c36a3efd344b,2bf82a30177fa1ce] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:50:29.377 INFO 21062 --- [ocp-async-9,98065b9d06754b94,0e61b70ae3c184bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:29.747 INFO 21062 --- [ocp-schedules-13,137cbd0871c06de5,4ec29268fb4af1f4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:50:29.747 INFO 21062 --- [ocp-schedules-13,137cbd0871c06de5,4ec29268fb4af1f4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:50:29.751 INFO 21062 --- [ocp-schedules-13,137cbd0871c06de5,4ec29268fb4af1f4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:50:29.751 INFO 21062 --- [ocp-schedules-13,137cbd0871c06de5,4ec29268fb4af1f4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:50:29.752 INFO 21062 --- [ocp-schedules-13,137cbd0871c06de5,4ec29268fb4af1f4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:50:29.752 INFO 21062 --- [ocp-schedules-13,137cbd0871c06de5,4ec29268fb4af1f4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:50:29.983 INFO 21062 --- [prometheus-minute-schedule-1,bd923d8cdf72d0a0,290b20d4f995fc7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:50:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:50:30.999 INFO 21062 --- [ocp-schedules-14,cb17898c9a97c331,8f10e2635bfa0405] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:50:31.345 WARN 21062 --- [ocp-async-7,b7fef073c9f25ac8,98bcda3628277a2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:31.378 INFO 21062 --- [ocp-async-10,14cc750b68bfdad8,745d8aeb8c552393] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:32.166 INFO 21062 --- [ocp-schedules-7,b91ce834a46a2a33,e8629c7d27305fdb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:50:32.172 INFO 21062 --- [ocp-schedules-7,b91ce834a46a2a33,e8629c7d27305fdb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:50:32.172 INFO 21062 --- [ocp-schedules-7,b91ce834a46a2a33,e8629c7d27305fdb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:50:32.303 INFO 21062 --- [ocp-schedules-4,fbd8d0b6a4673f3d,bde5c34dc196e5b6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:50:32.303 INFO 21062 --- [ocp-schedules-4,fbd8d0b6a4673f3d,bde5c34dc196e5b6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:50:32.330 INFO 21062 --- [ocp-schedules-4,fbd8d0b6a4673f3d,bde5c34dc196e5b6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:50:32.331 INFO 21062 --- [ocp-schedules-4,fbd8d0b6a4673f3d,bde5c34dc196e5b6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:50:32.331 INFO 21062 --- [ocp-schedules-13,a7ed09f035b4c278,1c0ddfbb69860be0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:50:32.342 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:32.343 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:32.523 INFO 21062 --- [ocp-schedules-10,6a79e65aecc02a69,36ae12f048e878bf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:50:32.523+08:00, clusters=[1] 2024-11-04 14:50:32.524 INFO 21062 --- [ocp-schedules-10,6a79e65aecc02a69,36ae12f048e878bf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:50:32.739 INFO 21062 --- [ocp-schedules-19,2081f4867a469ff6,84e85a5218fe1bf1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:50:32.739+08:00, clusters=[1] 2024-11-04 14:50:32.744 INFO 21062 --- [ocp-schedules-19,2081f4867a469ff6,84e85a5218fe1bf1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:50:32.753 INFO 21062 --- [ocp-schedules-23,87b74c725cd73adf,3f5d53c38a64dabc] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:50:32.757 INFO 21062 --- [ocp-schedules-23,87b74c725cd73adf,3f5d53c38a64dabc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:32.814 ERROR 21062 --- [ocp-schedules-23,87b74c725cd73adf,3f5d53c38a64dabc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:50:32.814 INFO 21062 --- [ocp-schedules-23,87b74c725cd73adf,3f5d53c38a64dabc] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:50:32.983 INFO 21062 --- [prometheus-minute-schedule-1,782ff146422a354e,8014c1f975c9d57b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:33.345 WARN 21062 --- [ocp-async-3,73554ebaf026f088,ed19a23570b9af71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:33.378 INFO 21062 --- [ocp-async-4,8ede680c5fc85c3b,63dda0249acbbb00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:34.081 INFO 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:50:34.081 INFO 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:50:34.081 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:50:34.081 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:50:34.091 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:50:34.092 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:50:34.094 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:50:34.094 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:50:34.094 INFO 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:50:34.095 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:50:34.095 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:50:34.095 WARN 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:50:34.096 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:50:34.096 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:50:34.097 INFO 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:50:34.098 WARN 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:50:34.104 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:50:34.104 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:50:34.104 INFO 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:50:34.129 ERROR 21062 --- [ocp-schedules-27,9a6d64d2182023bc,8602f8fc3a05bd43] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:50:35.343 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:35.344 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:35.345 WARN 21062 --- [ocp-async-1,e97da7752f4d4db3,ff9b2dba6dbd65c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:35.378 INFO 21062 --- [ocp-async-5,32951d138a9aa71e,461e9f98fc116090] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:35.983 INFO 21062 --- [prometheus-minute-schedule-1,bdb1a37f5b24c68f,7d458e7b357c6233] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:37.345 WARN 21062 --- [ocp-async-9,5581aa191654650a,eda80e0ac55e2143] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:37.378 INFO 21062 --- [ocp-async-2,6baa463bf9bf335f,4d4aab74cf8a4c77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:37.986 INFO 21062 --- [ocp-schedules-12,180c6cbe3d1f4c76,a151f967d6c81ce2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:50:37.987 INFO 21062 --- [ocp-schedules-12,b9512dea7ae17692,5984135a1b8093d6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:50:37.990 INFO 21062 --- [ocp-schedules-12,b9512dea7ae17692,5984135a1b8093d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:38.019 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:50:38.019 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:50:38.345 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:38.346 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:38.505 INFO 21062 --- [ocp-schedules-5,b857781091ffdb28,8d1b52d544de87bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:50:38.505 INFO 21062 --- [ocp-schedules-22,8e652d48df92c7f8,19a844bee6403d3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:50:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:50:38.589513+08:00, lastProcessTime=2024-11-04T14:50:28.589699+08:00 2024-11-04 14:50:38.688 INFO 21062 --- [ocp-async-1,f004a3c47708b0b0,ff42080701e960d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:50:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:50:38.589513+08:00 2024-11-04 14:50:38.983 INFO 21062 --- [prometheus-minute-schedule-1,28b55345cbc61364,73aad4020b511273] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:39.042 INFO 21062 --- [ocp-schedules-3,0b8fd753a21fa921,c4a77a6bde81af60] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:50:39.346 WARN 21062 --- [ocp-async-8,604a8f7b16213650,fa810cbeaf434f8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:39.378 INFO 21062 --- [ocp-async-1,98ef33eebb85df8c,ae97aa89fee9d8d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:50:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:50:41.002 INFO 21062 --- [ocp-schedules-22,aa9da5f8e692cd63,f2cb0c335cd1124b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:50:41.281 INFO 21062 --- [ocp-schedules-17,9c1d156013978c75,421eb85ba1033911] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:50:41.285 INFO 21062 --- [ocp-schedules-17,9c1d156013978c75,421eb85ba1033911] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:41.346 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:41.346 WARN 21062 --- [ocp-async-6,017b37af40d873e9,f87d21833db1c805] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:41.346 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:41.370 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:50:41.372 ERROR 21062 --- [ocp-schedules-17,9c1d156013978c75,421eb85ba1033911] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:50:41.372 INFO 21062 --- [ocp-schedules-17,9c1d156013978c75,421eb85ba1033911] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:50:41.378 INFO 21062 --- [ocp-async-9,d4e01c541afd3478,7d84f222042e8885] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:50:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:50:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:50:41.800 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:50:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:50:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:50:41.983 INFO 21062 --- [prometheus-minute-schedule-1,f16c5ae99f1389d7,c34dbcb31c7018a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:43.346 WARN 21062 --- [ocp-async-7,45b57f2496c70a79,a3cdec858332121b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:43.379 INFO 21062 --- [ocp-async-10,85511c58673be898,3327e6d48e6b94df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:44.346 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:44.347 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:44.983 INFO 21062 --- [prometheus-minute-schedule-1,71a0aaa5e479ef99,eae607c966e6b0e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:45.346 WARN 21062 --- [ocp-async-3,583c1bf46fda8383,30dc10a9e0625cf6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:45.379 INFO 21062 --- [ocp-async-4,617e5e8644e3e545,b94da059c52208aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:47.346 WARN 21062 --- [ocp-async-4,4ca7ae905a120583,d2796381ef5520a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:47.348 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:47.348 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:47.379 INFO 21062 --- [ocp-async-5,2c58dc0bce0754b5,55bdd696063076ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fb28bfba6babe65b,9a135a34269f316e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:50:48.591867+08:00, lastProcessTime=2024-11-04T14:50:38.589513+08:00 2024-11-04 14:50:48.688 INFO 21062 --- [ocp-async-10,907d7bd806732057,3d2ca97617926ee5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:50:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:50:48.591867+08:00 2024-11-04 14:50:49.087 INFO 21062 --- [ocp-schedules-29,fc4c7a09a5c470bb,cdad0fa2be8d9b7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:50:49.346 WARN 21062 --- [ocp-async-9,1dc0aaa4b9879bd3,6f4a528505d9d9d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:49.379 INFO 21062 --- [ocp-async-10,f416cb0ad11748bf,66a021b4baece332] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:50.349 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:50.350 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b6f293666e394818,0fe099c799f74553] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:51.005 INFO 21062 --- [ocp-schedules-27,f1226d15c799ff8b,630bfd660d3c3c31] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:50:51.347 WARN 21062 --- [ocp-async-10,09b93a21100d66a3,e2bf67ed1d2ef23f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:51.379 INFO 21062 --- [ocp-async-4,dea0f61f8d6c78a6,88b6ff7698a4b1d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:52.174 INFO 21062 --- [ocp-schedules-24,2a72f441f319f2bf,0f8e1cc6a1f4a0ea] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:50:52.180 INFO 21062 --- [ocp-schedules-24,2a72f441f319f2bf,0f8e1cc6a1f4a0ea] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:50:52.180 INFO 21062 --- [ocp-schedules-24,2a72f441f319f2bf,0f8e1cc6a1f4a0ea] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:50:53.347 WARN 21062 --- [ocp-async-4,0f067616a4b0c05c,2094b8bbf4d13f46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:53.350 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:53.351 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:53.380 INFO 21062 --- [ocp-async-5,0b7f878445a0bc73,ef5420a7126ea400] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9d4d4d3216147ad9,fa349be0c40ae898] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:55.347 WARN 21062 --- [ocp-async-5,ea020673e20dbb28,aa3088b1d8dd1f37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:55.380 INFO 21062 --- [ocp-async-7,a6da2b6cda5dbc5a,199099fdf3f30895] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:56.351 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:56.352 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:56.983 INFO 21062 --- [prometheus-minute-schedule-1,0b366d0403a7bc72,34974104df095d4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:50:57.347 WARN 21062 --- [ocp-async-7,e9135a18932b5f02,5fcab299c3db4f5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:57.380 INFO 21062 --- [ocp-async-3,ceca792e3dc46243,3d706acfa0088aa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:58.500 INFO 21062 --- [ocp-schedules-2,1891f584eda3d85c,b6d0b7edbedc0140] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:50:58.500 INFO 21062 --- [ocp-schedules-21,1a5053b0b6a09f45,d635856b7e36a672] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:50:58.502 INFO 21062 --- [ocp-schedules-9,1c2d07831ab696ea,5e4c5632e32a5c13] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:50:58.504 INFO 21062 --- [ocp-schedules-2,1891f584eda3d85c,b6d0b7edbedc0140] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:58.504 INFO 21062 --- [ocp-schedules-21,1a5053b0b6a09f45,d635856b7e36a672] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:50:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:50:58.505 INFO 21062 --- [ocp-schedules-3,9988ce113854a354,9effd8912298f998] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:50:58.512 INFO 21062 --- [ocp-schedules-22,1868bb08d96d3304,db2a836aded94909] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:50:58.512 INFO 21062 --- [ocp-schedules-22,1868bb08d96d3304,db2a836aded94909] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:50:58.512 INFO 21062 --- [ocp-schedules-28,075fdb3e4326352e,db54d0434a5d4bfc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:50:58.513 INFO 21062 --- [ocp-schedules-28,075fdb3e4326352e,db54d0434a5d4bfc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:50:58.513 INFO 21062 --- [ocp-schedules-18,4bb5a6cc1a160d5f,bd944aca5c4491c9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:50:58.513 INFO 21062 --- [ocp-schedules-18,4bb5a6cc1a160d5f,bd944aca5c4491c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:50:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:50:58.590354+08:00, lastProcessTime=2024-11-04T14:50:48.591867+08:00 2024-11-04 14:50:58.688 INFO 21062 --- [ocp-async-5,bbc2148300518978,f4be3b1c874f6057] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:50:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:50:58.590354+08:00 2024-11-04 14:50:59.114 INFO 21062 --- [ocp-schedules-19,4ec5b819e06ca3af,8ed2e78388393587] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:50:59.319 INFO 21062 --- [ocp-schedules-14,0c68c4bf145fadd1,4eb4393c3002d801] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:50:59.347 WARN 21062 --- [ocp-async-4,8730b50e0ae8c058,a86120a17c74dc57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:50:59.352 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:50:59.353 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:50:59.380 INFO 21062 --- [ocp-async-5,df306c9dc61a61ef,66774ba337a844b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:50:59.983 INFO 21062 --- [prometheus-minute-schedule-1,38ff96bcfb0400d4,0a598be682705a64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703060, interval=60 2024-11-04 14:51:01.008 INFO 21062 --- [ocp-schedules-17,73ffcc730c434875,c8d39a301f17238f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:51:01.347 WARN 21062 --- [ocp-async-5,3830ce1fc32be4b4,da9644a398fa082c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:01.380 INFO 21062 --- [ocp-async-7,97a2248633e14ac4,729111fb9036adf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:02.353 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:02.354 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a42d78ef96c671ae,7d50b13fb5dd8a37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:03.348 WARN 21062 --- [ocp-async-7,a60f76c4c847ce1f,b0790824f73c3837] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:03.381 INFO 21062 --- [ocp-async-3,17c4006bcaf99990,59895ddbad5876e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:05.348 WARN 21062 --- [ocp-async-3,56b08e8e67daf428,362dc251f0b7808c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:05.354 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:05.355 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:05.381 INFO 21062 --- [ocp-async-6,7e5093bf28c067c4,d43de159d6ae5b00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e3b9eae63f313708,6f8f43a94d3e4694] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:07.348 WARN 21062 --- [ocp-async-6,8327fb7257892854,0e8c9b8c7892887b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:07.381 INFO 21062 --- [ocp-async-2,890f2a8dc39742e6,c9c70f3a8f72a2e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:08.020 INFO 21062 --- [ocp-schedules-12,b9512dea7ae17692,5984135a1b8093d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:51:08.020 INFO 21062 --- [ocp-schedules-12,a71675dba6a26f7c,29981c91fdf73f71] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:51:08.024 INFO 21062 --- [ocp-schedules-12,a71675dba6a26f7c,29981c91fdf73f71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:08.053 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:51:08.053 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:51:08.355 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:08.356 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:08.505 INFO 21062 --- [ocp-schedules-21,1a5053b0b6a09f45,d635856b7e36a672] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:51:08.505 INFO 21062 --- [ocp-schedules-2,1891f584eda3d85c,b6d0b7edbedc0140] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:51:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:51:08.590305+08:00, lastProcessTime=2024-11-04T14:50:58.590354+08:00 2024-11-04 14:51:08.689 INFO 21062 --- [ocp-async-3,b2323e0337640e3d,f90db83e8e419e28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:51:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:51:08.590305+08:00 2024-11-04 14:51:08.983 INFO 21062 --- [prometheus-minute-schedule-1,10b4bb909618d83b,58a1cb6ad09f2caf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:09.157 INFO 21062 --- [ocp-schedules-21,4c205940d692130a,113ae436b70042b2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:51:09.163 INFO 21062 --- [ocp-schedules-2,190e91a0fb414ed2,d56385d1d5935cfc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:51:09.167 INFO 21062 --- [ocp-schedules-2,190e91a0fb414ed2,d56385d1d5935cfc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:09.168 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:51:09.348 WARN 21062 --- [ocp-async-7,aa8ea4f55d3eedd4,c7010e8d3e794f9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:09.381 INFO 21062 --- [ocp-async-8,26d3838a3af424d2,ed0f20e305e51077] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:10.002 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:51:10.002 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:51:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:51:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:51:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:51:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:51:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:51:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:51:11.011 INFO 21062 --- [ocp-schedules-24,145b9821d3622e89,59924cdd5f9c9d62] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:51:11.349 WARN 21062 --- [ocp-async-8,456a67616e96af6f,69a08b785ca327d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:11.356 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:11.357 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:11.381 INFO 21062 --- [ocp-async-1,d652699091d9c7c3,b6aefefba23b817b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:11.677 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=1, affectRows=1 2024-11-04 14:51:11.677 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=1, validCount=1 2024-11-04 14:51:11.677 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=5, affectRows=5 2024-11-04 14:51:11.677 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=5, validCount=5 2024-11-04 14:51:11.677 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=3, affectRows=3 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=3, validCount=3 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=17, affectRows=17 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=16, affectRows=16 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=17, validCount=17 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=16, validCount=16 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=9, affectRows=9 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=9, validCount=9 2024-11-04 14:51:11.678 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=26, affectRows=26 2024-11-04 14:51:11.679 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=26, validCount=26 2024-11-04 14:51:11.679 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=14, affectRows=14 2024-11-04 14:51:11.679 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=14, validCount=14 2024-11-04 14:51:11.679 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=36, affectRows=36 2024-11-04 14:51:11.679 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=26, affectRows=26 2024-11-04 14:51:11.679 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=26, validCount=26 2024-11-04 14:51:11.679 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=36, validCount=36 2024-11-04 14:51:11.680 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=33, affectRows=33 2024-11-04 14:51:11.680 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=32, validCount=32 2024-11-04 14:51:11.680 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=53, affectRows=53 2024-11-04 14:51:11.680 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=47, affectRows=47 2024-11-04 14:51:11.681 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=47, validCount=47 2024-11-04 14:51:11.681 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=53, validCount=53 2024-11-04 14:51:11.681 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=64, affectRows=64 2024-11-04 14:51:11.681 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703000, nodeCount=72, affectRows=72 2024-11-04 14:51:11.681 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=64, validCount=64 2024-11-04 14:51:11.681 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703000, nodeCount=70, validCount=70 2024-11-04 14:51:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6c57182fe3a733d4,42675c72ebaf14c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:12.182 INFO 21062 --- [ocp-schedules-30,ce9611951d5331c0,1d13dfdc9335fbd5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:51:12.189 INFO 21062 --- [ocp-schedules-30,ce9611951d5331c0,1d13dfdc9335fbd5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:51:12.189 INFO 21062 --- [ocp-schedules-30,ce9611951d5331c0,1d13dfdc9335fbd5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:51:13.349 WARN 21062 --- [ocp-async-1,1bca08e6f3921eb1,c3bce98ea419ca66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:13.382 INFO 21062 --- [ocp-async-9,03019aac8751a0d7,9ac359cfa964a6dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:13.984 INFO 21062 --- [prometheus-cache-monitor1,0d625592ccc196f3,db7f4ee080eef9b9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:51:13.984 INFO 21062 --- [prometheus-cache-monitor1,0d625592ccc196f3,db7f4ee080eef9b9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:51:14.357 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:14.358 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:14.983 INFO 21062 --- [prometheus-minute-schedule-1,333eac56d5db03df,032e5c478295d3fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:15.299 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:47:15.274, end=2024-11-04T14:49:15.296} 2024-11-04 14:51:15.303 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:51:15.303 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:51:15.349 WARN 21062 --- [ocp-async-9,2b14ef2fcd44d9bd,abadee15027fc780] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:15.382 INFO 21062 --- [ocp-async-10,f919cce389701054,1709a945086d16ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:17.349 WARN 21062 --- [ocp-async-3,27e267c7f0fa3ff7,77b76994c0f1ec19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:17.358 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:17.359 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:17.382 INFO 21062 --- [ocp-async-4,c7d6fd8bbf661b6d,e1a2178ed1dbb79f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:17.983 INFO 21062 --- [prometheus-minute-schedule-1,de2a2254565ecd38,5751975ec1f74d33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:51:18.590983+08:00, lastProcessTime=2024-11-04T14:51:08.590305+08:00 2024-11-04 14:51:18.689 INFO 21062 --- [ocp-async-2,c666a78a489b1b50,9b07fff9d64b94a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:51:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:51:18.590983+08:00 2024-11-04 14:51:19.168 INFO 21062 --- [ocp-schedules-2,190e91a0fb414ed2,d56385d1d5935cfc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:51:19.182 INFO 21062 --- [ocp-schedules-26,7e3039f7a13c0bb5,3e809aaa271fa834] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:51:19.349 WARN 21062 --- [ocp-async-1,bf9662c75cc2e941,ca40eb201dd5ce10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:19.382 INFO 21062 --- [ocp-async-9,2ff0d34590d303db,49fcc31408778c18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:20.359 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:20.360 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:20.983 INFO 21062 --- [prometheus-minute-schedule-1,84ce9dab05768344,c88602d12e8cedae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:21.014 INFO 21062 --- [ocp-schedules-3,ecd80b13d1dffc05,0a4f0193846455d8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:51:21.349 WARN 21062 --- [ocp-async-2,6d8dedb49efb7952,0568410cbed049ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:21.382 INFO 21062 --- [ocp-async-10,8a3b106ba3a7a633,24f0a705ca371039] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:23.349 WARN 21062 --- [ocp-async-3,636bef30cc612e08,58833714a04c92ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:23.360 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:23.361 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:23.382 INFO 21062 --- [ocp-async-4,124bbf642d048dfe,cde825a53212f864] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4e6d9cec13e3ba86,8e4baf45994ed337] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:25.350 WARN 21062 --- [ocp-async-6,5e2595e9b47de16b,f29466d4e4793373] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:25.382 INFO 21062 --- [ocp-async-5,afb315ca41486d03,f45c0a2c15c30f83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:26.361 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:26.362 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:26.983 INFO 21062 --- [prometheus-minute-schedule-1,664ed75c3e103bd8,f993f905cf9efe1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:26.984 INFO 21062 --- [prometheus-hour-schedule-1,449685afcc027da7,0e485ee5cf632c14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:51:27.350 WARN 21062 --- [ocp-async-9,568207d451a1d468,53500081dd0894c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:27.383 INFO 21062 --- [ocp-async-7,6b2c5220058530cf,6536a8dbd574716b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:28.498 INFO 21062 --- [ocp-schedules-29,76851130d16b496c,a9df274d8325a4de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:51:28.500 INFO 21062 --- [ocp-schedules-26,0379247286c95f66,5e15de5c500d9d99] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:51:28.500 INFO 21062 --- [ocp-schedules-22,a810bd5c3395c37d,e66885201386c15f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:51:28.500 INFO 21062 --- [ocp-schedules-30,ab3bb5deb7bfedda,f847847f5d167925] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:51:28.502 INFO 21062 --- [ocp-schedules-16,929471ec8ce77576,3dfd3ad139557890] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:51:28.503 INFO 21062 --- [ocp-schedules-29,76851130d16b496c,a9df274d8325a4de] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:51:28.503 INFO 21062 --- [ocp-schedules-26,0379247286c95f66,5e15de5c500d9d99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:28.503 INFO 21062 --- [ocp-schedules-30,ab3bb5deb7bfedda,f847847f5d167925] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:51:28.504 INFO 21062 --- [ocp-schedules-11,b2f6346250083c54,3fdc961a926b3f78] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:51:28.506 INFO 21062 --- [ocp-schedules-22,a810bd5c3395c37d,e66885201386c15f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:51:28.510 INFO 21062 --- [ocp-schedules-13,6f864f89b38d950e,63e5b24199dbd428] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:51:28.510 INFO 21062 --- [ocp-schedules-6,21db7945bbf9d902,792da68832017fc4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:51:28.510 INFO 21062 --- [ocp-schedules-13,6f864f89b38d950e,63e5b24199dbd428] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:51:28.511 INFO 21062 --- [ocp-schedules-8,1c6b385cfedb8176,42023e2d50e29d70] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:51:28.511 INFO 21062 --- [ocp-schedules-6,21db7945bbf9d902,792da68832017fc4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:51:28.511 INFO 21062 --- [ocp-schedules-8,1c6b385cfedb8176,42023e2d50e29d70] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:51:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:51:28.590061+08:00, lastProcessTime=2024-11-04T14:51:18.590983+08:00 2024-11-04 14:51:28.645 INFO 21062 --- [ocp-schedules-15,4671f9b4e15b6515,685f034552f9ab16] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:51:28.646 INFO 21062 --- [ocp-schedules-15,4671f9b4e15b6515,685f034552f9ab16] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:51:28.689 INFO 21062 --- [ocp-async-1,22def5034ced9bb6,4f36c8fda596ce82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:51:28.777 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:51:28.590061+08:00 2024-11-04 14:51:28.902 INFO 21062 --- [ocp-schedules-4,f452754adc3b008a,b91e4ad158dd7cce] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:51:29.222 INFO 21062 --- [ocp-schedules-8,522db7d4d94029c3,512fab543c9d93db] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:51:29.223 INFO 21062 --- [ocp-schedules-29,9207e2e6673c39f9,c8f4c317362d5e4f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:51:29.223 INFO 21062 --- [ocp-schedules-29,9207e2e6673c39f9,c8f4c317362d5e4f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:51:29.321 INFO 21062 --- [ocp-schedules-21,76d3557580568b83,c8aa99a6288bd6bf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:51:29.350 WARN 21062 --- [ocp-async-4,2f1562a5d35b4648,17dd97290c20b40f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:29.352 INFO 21062 --- [ocp-schedules-1,fa6489897d0be342,600861441356f3f6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:51:29.358 INFO 21062 --- [ocp-schedules-1,fa6489897d0be342,600861441356f3f6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:51:29.358 INFO 21062 --- [ocp-schedules-1,fa6489897d0be342,600861441356f3f6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:51:29.362 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:29.363 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:29.383 INFO 21062 --- [ocp-async-1,fcdc995af788b28c,6b3ca8831b6c5e30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:29.755 INFO 21062 --- [ocp-schedules-14,08fe34db5c357a8a,796d75aa60de87f8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:51:29.755 INFO 21062 --- [ocp-schedules-14,08fe34db5c357a8a,796d75aa60de87f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:51:29.757 INFO 21062 --- [ocp-schedules-14,08fe34db5c357a8a,796d75aa60de87f8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:51:29.757 INFO 21062 --- [ocp-schedules-14,08fe34db5c357a8a,796d75aa60de87f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:51:29.757 INFO 21062 --- [ocp-schedules-14,08fe34db5c357a8a,796d75aa60de87f8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:51:29.757 INFO 21062 --- [ocp-schedules-14,08fe34db5c357a8a,796d75aa60de87f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:51:29.983 INFO 21062 --- [prometheus-minute-schedule-1,67ff049c80cb3849,ba6be53569b227f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:51:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:51:31.018 INFO 21062 --- [ocp-schedules-18,64b41b7abdd00ef4,a21b347226705d47] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:51:31.350 WARN 21062 --- [ocp-async-5,b9ea7b049435d9f5,8e6123143f410489] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:31.383 INFO 21062 --- [ocp-async-2,69d8ec77c6b02d01,b4734efb9de631d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:32.191 INFO 21062 --- [ocp-schedules-16,23f5083fbe01731c,46929040179c39ee] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:51:32.196 INFO 21062 --- [ocp-schedules-16,23f5083fbe01731c,46929040179c39ee] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:51:32.196 INFO 21062 --- [ocp-schedules-16,23f5083fbe01731c,46929040179c39ee] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:51:32.331 INFO 21062 --- [ocp-schedules-11,7c68298239ad4e18,46e846654d8a8b20] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:51:32.331 INFO 21062 --- [ocp-schedules-11,7c68298239ad4e18,46e846654d8a8b20] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:51:32.357 INFO 21062 --- [ocp-schedules-23,dac68954c8df21c4,fa11919812ae0770] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:51:32.357 INFO 21062 --- [ocp-schedules-11,7c68298239ad4e18,46e846654d8a8b20] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:51:32.357 INFO 21062 --- [ocp-schedules-11,7c68298239ad4e18,46e846654d8a8b20] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:51:32.363 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:32.364 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:32.551 INFO 21062 --- [ocp-schedules-6,32def0ada15027ba,4edd5920a2dc2d65] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:51:32.551+08:00, clusters=[1] 2024-11-04 14:51:32.553 INFO 21062 --- [ocp-schedules-6,32def0ada15027ba,4edd5920a2dc2d65] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:51:32.770 INFO 21062 --- [ocp-schedules-28,90e29d70309c8da1,4ed30573b1620cec] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:51:32.770+08:00, clusters=[1] 2024-11-04 14:51:32.774 INFO 21062 --- [ocp-schedules-28,90e29d70309c8da1,4ed30573b1620cec] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:51:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7425f67a950a2a13,73731336219d32d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:33.350 WARN 21062 --- [ocp-async-7,b8837a9deca4ef17,e004529fafd3899e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:33.383 INFO 21062 --- [ocp-async-8,3b75fb9e908b2241,b101211c660bbe10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:34.139 INFO 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:51:34.139 INFO 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:51:34.139 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:51:34.139 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:51:34.149 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:51:34.149 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:51:34.150 INFO 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:51:34.151 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:51:34.151 WARN 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:51:34.151 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:51:34.151 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:51:34.152 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:51:34.152 INFO 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:51:34.152 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:51:34.153 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:51:34.153 WARN 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:51:34.160 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:51:34.160 INFO 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:51:34.160 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:51:34.171 ERROR 21062 --- [ocp-schedules-4,0d944233db351960,15d9158baaa1bdcc] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:51:35.351 WARN 21062 --- [ocp-async-3,f905bb5fe74b1ec2,7fb139e4267c5d79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:35.364 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:35.365 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:35.383 INFO 21062 --- [ocp-async-6,ec46ff1919fa191b,57b1e0ef30056119] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c3d3180de005bdee,ac2e76f41b33d143] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:36.538 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4e3eed67fdadd4f7,88e55187d0405db3] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.130.47.17, traceId 4e3eed67fdadd4f7] 2024-11-04 14:51:36.545 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4e3eed67fdadd4f7,88e55187d0405db3] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.130.47.17, traceId=4e3eed67fdadd4f7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2024-11-04 14:51:36.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4e3eed67fdadd4f7,88e55187d0405db3] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 42, current state: FAILED, new state: READY 2024-11-04 14:51:36.714 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4e3eed67fdadd4f7,88e55187d0405db3] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 13, current state: FAILED, new state: READY 2024-11-04 14:51:36.729 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4e3eed67fdadd4f7,88e55187d0405db3] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.130.47.17, serverIp=null, startTime=2024-11-04T14:51:36.548+08:00, endTime=2024-11-04T14:51:36.728+08:00, role=null, aliUid=null) 2024-11-04 14:51:36.743 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4e3eed67fdadd4f7,88e55187d0405db3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.130.47.17, traceId=4e3eed67fdadd4f7, duration=212 ms] 2024-11-04 14:51:36.769 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,87096e04fd4254e3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: RUNNING 2024-11-04 14:51:36.775 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,87096e04fd4254e3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: RUNNING 2024-11-04 14:51:36.789 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=13, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:51:36.767+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=13}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:51:36.793 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:51:36.794 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=42, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T14:51:36.773+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=42}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 14:51:36.798 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 14:51:36.796 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:51:36.801 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:51:36.803 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:51:36.806 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:51:36.823 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:36.825 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:36.829 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.831 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.874 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:36.879 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:36.879 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.884 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.922 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:51:36.931 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:51:36.951 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.954 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.957 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.959 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:36.998 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:37.003 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:37.003 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.009 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.050 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:51:37.051 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:51:37.077 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.079 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.078 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.085 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.119 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:37.125 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.126 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:37.133 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.171 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 14:51:37.174 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 14:51:37.189 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,40ccba87a689afef,2be38aeb5ca93880] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=40ccba87a689afef, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:37.196 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.199 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.201 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:51:37.203 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.207 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.209 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 14:51:37.216 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,40ccba87a689afef,2be38aeb5ca93880] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=40ccba87a689afef, duration=33 ms] 2024-11-04 14:51:37.243 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:37.248 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.250 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:37.256 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.291 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 14:51:37.298 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 14:51:37.335 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.338 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.341 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:51:37.343 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:51:37.343 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.346 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:51:37.351 WARN 21062 --- [ocp-async-1,96d85b065e6225d2,66524cb5cc23c87a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:37.348 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.353 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 14:51:37.355 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 14:51:37.358 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 14:51:37.383 INFO 21062 --- [ocp-async-9,7c1d47e6144b7362,5b4d3bb0d055b7ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:37.383 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:37.388 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.390 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:37.396 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.431 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 14:51:37.438 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 14:51:37.476 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.479 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.484 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.487 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.523 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:37.529 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.530 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:37.537 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.575 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/076446a479724c29 on 10.46.128.12 2024-11-04 14:51:37.578 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/610b338304e44d0c on 10.46.128.11 2024-11-04 14:51:37.591 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/076446a479724c29 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/076446a479724c29, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.593 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/076446a479724c29 on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/076446a479724c29, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.597 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/610b338304e44d0c on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/610b338304e44d0c, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.599 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/610b338304e44d0c on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/610b338304e44d0c, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.635 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:37.641 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.642 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:37.649 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.683 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/076446a479724c29/uninstall_package.sh && chmod +x /tmp/076446a479724c29/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/076446a479724c29/uninstall_package.sh on 10.46.128.12 2024-11-04 14:51:37.690 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/610b338304e44d0c/uninstall_package.sh && chmod +x /tmp/610b338304e44d0c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/610b338304e44d0c/uninstall_package.sh on 10.46.128.11 2024-11-04 14:51:37.690 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/076446a479724c29/uninstall_package.sh && chmod +x /tmp/076446a479724c29/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/076446a479724c29/uninstall_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/076446a479724c29/uninstall_package.sh && chmod +x /tmp/076446a479724c29/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/076446a479724c29/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.695 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/076446a479724c29/uninstall_package.sh && chmod +x /tmp/076446a479724c29/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/076446a479724c29/uninstall_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/076446a479724c29/uninstall_package.sh && chmod +x /tmp/076446a479724c29/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/076446a479724c29/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.697 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/610b338304e44d0c/uninstall_package.sh && chmod +x /tmp/610b338304e44d0c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/610b338304e44d0c/uninstall_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/610b338304e44d0c/uninstall_package.sh && chmod +x /tmp/610b338304e44d0c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/610b338304e44d0c/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.700 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/610b338304e44d0c/uninstall_package.sh && chmod +x /tmp/610b338304e44d0c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/610b338304e44d0c/uninstall_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/610b338304e44d0c/uninstall_package.sh && chmod +x /tmp/610b338304e44d0c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/610b338304e44d0c/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.735 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:37.740 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.742 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:37.748 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:37.787 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/076446a479724c29/uninstall_package.sh on 10.46.128.12 2024-11-04 14:51:37.794 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/610b338304e44d0c/uninstall_package.sh on 10.46.128.11 2024-11-04 14:51:37.993 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/610b338304e44d0c/uninstall_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/610b338304e44d0c/uninstall_package.sh, out=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 14:51:37.996 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/610b338304e44d0c/uninstall_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/610b338304e44d0c/uninstall_package.sh, out=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 14:51:38.030 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/076446a479724c29/uninstall_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/076446a479724c29/uninstall_package.sh, out=(Reading database ... 94029 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 14:51:38.033 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/076446a479724c29/uninstall_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/076446a479724c29/uninstall_package.sh, out=(Reading database ... 94029 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 14:51:38.034 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:38.042 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.054 INFO 21062 --- [ocp-schedules-12,a71675dba6a26f7c,29981c91fdf73f71] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:51:38.054 INFO 21062 --- [ocp-schedules-12,e8e581004252b18f,27a4a1c373913f9a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:51:38.059 INFO 21062 --- [ocp-schedules-12,e8e581004252b18f,27a4a1c373913f9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:38.075 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:38.080 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.088 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:51:38.088 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:51:38.086 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: rm -rf /tmp/610b338304e44d0c on 10.46.128.11 2024-11-04 14:51:38.092 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: rm -rf /tmp/610b338304e44d0c on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=rm -rf /tmp/610b338304e44d0c, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.095 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: rm -rf /tmp/610b338304e44d0c on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=rm -rf /tmp/610b338304e44d0c, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.097 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.ocp.executor.executor.SshExecutor : uninstall complete, rpmName=ocp-agent-ce, result=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ... 2024-11-04 14:51:38.099 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:51:38.102 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:51:38.105 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:51:38.108 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:51:38.112 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:51:38.116 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:51:38.119 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:51:38.121 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:51:38.127 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: rm -rf /tmp/076446a479724c29 on 10.46.128.12 2024-11-04 14:51:38.133 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: rm -rf /tmp/076446a479724c29 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=rm -rf /tmp/076446a479724c29, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.136 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: rm -rf /tmp/076446a479724c29 on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=rm -rf /tmp/076446a479724c29, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.138 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.ocp.executor.executor.SshExecutor : uninstall complete, rpmName=ocp-agent-ce, result=(Reading database ... 94029 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ... 2024-11-04 14:51:38.140 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 14:51:38.143 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 14:51:38.139 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:38.145 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:51:38.150 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.151 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:51:38.156 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:51:38.159 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:51:38.162 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 14:51:38.165 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 14:51:38.180 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:38.185 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.198 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:38.203 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.231 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:38.236 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.250 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:51:38.274 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.277 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.279 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:51:38.302 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.305 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.314 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:38.319 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.347 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:38.352 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.365 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:38.366 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:38.366 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 14:51:38.389 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.392 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.399 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 14:51:38.412 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:51:38.421 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.424 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.430 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:38.435 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.441 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 14:51:38.467 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:38.473 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.482 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11 2024-11-04 14:51:38.504 INFO 21062 --- [ocp-schedules-30,ab3bb5deb7bfedda,f847847f5d167925] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:51:38.504 INFO 21062 --- [ocp-schedules-26,0379247286c95f66,5e15de5c500d9d99] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:51:38.505 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,168c141823121f7e,07e304f76b2a8cd4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.11, traceId=168c141823121f7e, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:51:38.509 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,168c141823121f7e,07e304f76b2a8cd4] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:51:38.510 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,168c141823121f7e,07e304f76b2a8cd4] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:51:38.510 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,168c141823121f7e,07e304f76b2a8cd4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.11, traceId=168c141823121f7e, duration=8 ms] 2024-11-04 14:51:38.515 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12 2024-11-04 14:51:38.536 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,88fb3442976ad074,7128db5df3251781] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.12, traceId=88fb3442976ad074, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 14:51:38.539 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,88fb3442976ad074,7128db5df3251781] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 14:51:38.540 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,88fb3442976ad074,7128db5df3251781] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 14:51:38.540 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,88fb3442976ad074,7128db5df3251781] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.12, traceId=88fb3442976ad074, duration=6 ms] 2024-11-04 14:51:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:51:38.591485+08:00, lastProcessTime=2024-11-04T14:51:28.590061+08:00 2024-11-04 14:51:38.595 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,023d444b4590810d,c84a7908b6a0e52a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=023d444b4590810d, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 14:51:38.603 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,023d444b4590810d,c84a7908b6a0e52a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=023d444b4590810d, duration=17 ms] 2024-11-04 14:51:38.602 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.606 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.633 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.636 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.650 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:38.656 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.679 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:38.685 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.689 INFO 21062 --- [ocp-async-3,c246a8e81767c98e,75bd241f8a9d6a78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:51:38.702 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11 2024-11-04 14:51:38.731 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12 2024-11-04 14:51:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:51:38.591485+08:00 2024-11-04 14:51:38.853 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.856 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.859 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:51:38.879 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.882 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.885 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 14:51:38.898 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:38.904 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.923 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:38.929 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.950 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d997e146ce2e4c02 on 10.46.128.11 2024-11-04 14:51:38.956 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d997e146ce2e4c02 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d997e146ce2e4c02, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.959 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d997e146ce2e4c02 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d997e146ce2e4c02, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.971 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/7c055c56402f4c0b on 10.46.128.12 2024-11-04 14:51:38.977 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/7c055c56402f4c0b on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/7c055c56402f4c0b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.980 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/7c055c56402f4c0b on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/7c055c56402f4c0b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3058c77946015fd3,c984b3302b6e5915] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:39.002 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:39.008 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.023 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:39.028 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.054 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d997e146ce2e4c02/cpio && chmod +x /tmp/d997e146ce2e4c02/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d997e146ce2e4c02/cpio on 10.46.128.11 2024-11-04 14:51:39.061 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d997e146ce2e4c02/cpio && chmod +x /tmp/d997e146ce2e4c02/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d997e146ce2e4c02/cpio on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d997e146ce2e4c02/cpio && chmod +x /tmp/d997e146ce2e4c02/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d997e146ce2e4c02/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.064 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d997e146ce2e4c02/cpio && chmod +x /tmp/d997e146ce2e4c02/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d997e146ce2e4c02/cpio on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d997e146ce2e4c02/cpio && chmod +x /tmp/d997e146ce2e4c02/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/d997e146ce2e4c02/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.075 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/7c055c56402f4c0b/cpio && chmod +x /tmp/7c055c56402f4c0b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7c055c56402f4c0b/cpio on 10.46.128.12 2024-11-04 14:51:39.083 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/7c055c56402f4c0b/cpio && chmod +x /tmp/7c055c56402f4c0b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7c055c56402f4c0b/cpio on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/7c055c56402f4c0b/cpio && chmod +x /tmp/7c055c56402f4c0b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7c055c56402f4c0b/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.085 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/7c055c56402f4c0b/cpio && chmod +x /tmp/7c055c56402f4c0b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7c055c56402f4c0b/cpio on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/7c055c56402f4c0b/cpio && chmod +x /tmp/7c055c56402f4c0b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7c055c56402f4c0b/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.106 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:39.111 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.127 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:39.132 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.158 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/d997e146ce2e4c02 on 10.46.128.11 2024-11-04 14:51:39.164 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/d997e146ce2e4c02 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d997e146ce2e4c02, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.167 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/d997e146ce2e4c02 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/d997e146ce2e4c02, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.175 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/7c055c56402f4c0b on 10.46.128.12 2024-11-04 14:51:39.181 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/7c055c56402f4c0b on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/7c055c56402f4c0b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.183 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/7c055c56402f4c0b on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/7c055c56402f4c0b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.210 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:39.216 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.227 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:39.232 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.246 INFO 21062 --- [ocp-schedules-10,807594849cc585bf,530f031858bcfcf3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:51:39.262 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/d997e146ce2e4c02/install_package.sh && chmod +x /tmp/d997e146ce2e4c02/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d997e146ce2e4c02/install_package.sh on 10.46.128.11 2024-11-04 14:51:39.269 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/d997e146ce2e4c02/install_package.sh && chmod +x /tmp/d997e146ce2e4c02/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d997e146ce2e4c02/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d997e146ce2e4c02/install_package.sh && chmod +x /tmp/d997e146ce2e4c02/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d997e146ce2e4c02/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.272 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/d997e146ce2e4c02/install_package.sh && chmod +x /tmp/d997e146ce2e4c02/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d997e146ce2e4c02/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/d997e146ce2e4c02/install_package.sh && chmod +x /tmp/d997e146ce2e4c02/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/d997e146ce2e4c02/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.279 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/7c055c56402f4c0b/install_package.sh && chmod +x /tmp/7c055c56402f4c0b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7c055c56402f4c0b/install_package.sh on 10.46.128.12 2024-11-04 14:51:39.287 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/7c055c56402f4c0b/install_package.sh && chmod +x /tmp/7c055c56402f4c0b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7c055c56402f4c0b/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/7c055c56402f4c0b/install_package.sh && chmod +x /tmp/7c055c56402f4c0b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7c055c56402f4c0b/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.289 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/7c055c56402f4c0b/install_package.sh && chmod +x /tmp/7c055c56402f4c0b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7c055c56402f4c0b/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/7c055c56402f4c0b/install_package.sh && chmod +x /tmp/7c055c56402f4c0b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7c055c56402f4c0b/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.314 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 14:51:39.319 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.331 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 14:51:39.336 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 14:51:39.351 WARN 21062 --- [ocp-async-10,ac164b130c90ac24,a9ea6c6b8144b6d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:39.366 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/d997e146ce2e4c02/install_package.sh on 10.46.128.11 2024-11-04 14:51:39.384 INFO 21062 --- [ocp-async-3,d3b1cc79e938cb6f,04ac4f30a18f5b69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:39.383 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/7c055c56402f4c0b/install_package.sh on 10.46.128.12 2024-11-04 14:51:39.755 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,da64ee806cd1f4db,3001cf2364704d23] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=da64ee806cd1f4db, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:39.781 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,da64ee806cd1f4db,3001cf2364704d23] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=da64ee806cd1f4db, duration=34 ms] 2024-11-04 14:51:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:51:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:51:41.020 INFO 21062 --- [ocp-schedules-5,7a02981ef7471491,bd830556b7086f9a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:51:41.351 WARN 21062 --- [ocp-async-4,8831384f58c24d65,6d86be2e72088638] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:41.366 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:41.366 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:41.372 INFO 21062 --- [ocp-schedules-8,42f7f4c57c0deea5,3aac7f3b6bdadc3b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:51:41.376 INFO 21062 --- [ocp-schedules-8,42f7f4c57c0deea5,3aac7f3b6bdadc3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:41.384 INFO 21062 --- [ocp-async-1,49d057ca417df9f1,d0a850c9884baa65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:41.463 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:51:41.467 ERROR 21062 --- [ocp-schedules-8,42f7f4c57c0deea5,3aac7f3b6bdadc3b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:51:41.467 INFO 21062 --- [ocp-schedules-8,42f7f4c57c0deea5,3aac7f3b6bdadc3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:51:41.811 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,a9d0d16a0814eb52,d3776f4e139383a6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=a9d0d16a0814eb52, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:41.838 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,a9d0d16a0814eb52,d3776f4e139383a6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=a9d0d16a0814eb52, duration=34 ms] 2024-11-04 14:51:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a298765060bc76d0,f0922b2b567f54af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:43.351 WARN 21062 --- [ocp-async-5,abdac94dd2de8e3c,4aae722517ca36f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:43.384 INFO 21062 --- [ocp-async-2,b7e5ff3946e486e1,fc6b7d8accad977a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:43.439 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,924ee087a26b10dc,2e7ec385623d86ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=924ee087a26b10dc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:43.466 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,924ee087a26b10dc,2e7ec385623d86ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=924ee087a26b10dc, duration=36 ms] 2024-11-04 14:51:44.367 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:44.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b6e6b4405f262495,c7d76ac8a448531c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:45.064 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,0522155eebcd93ef,dd6d144a4cdfd436] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=0522155eebcd93ef, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:45.093 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,0522155eebcd93ef,dd6d144a4cdfd436] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=0522155eebcd93ef, duration=38 ms] 2024-11-04 14:51:45.352 WARN 21062 --- [ocp-async-2,2c93436bad2b01bf,63c86a4c7a3b8ee5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:45.384 INFO 21062 --- [ocp-async-8,6f364be0134b2101,73411015edd61a1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:46.733 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,c5f947e224d294a8,8fcbae1944afa3ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c5f947e224d294a8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:46.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,c5f947e224d294a8,8fcbae1944afa3ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c5f947e224d294a8, duration=36 ms] 2024-11-04 14:51:47.182 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c4b97f09fd29681c,e5eb4007ca90bb18] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=c4b97f09fd29681c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:51:47.191 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,c4b97f09fd29681c,e5eb4007ca90bb18] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=c4b97f09fd29681c, duration=16 ms] 2024-11-04 14:51:47.352 WARN 21062 --- [ocp-async-8,8b87469f5c246164,7462d625e3066106] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:47.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:47.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:47.384 INFO 21062 --- [ocp-async-6,7f5e5aa1f05a86f2,9a5c500320e7834b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:47.983 INFO 21062 --- [prometheus-minute-schedule-1,4fa5599b190ea99d,dea29662cce78ec9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:51:48.591178+08:00, lastProcessTime=2024-11-04T14:51:38.591485+08:00 2024-11-04 14:51:48.689 INFO 21062 --- [ocp-async-2,b9d15b43c2075889,9cc86c2722637d1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:51:48.703 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,2639ce7822b7aedb,de1e06fda17a9f44] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=2639ce7822b7aedb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:48.727 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,2639ce7822b7aedb,de1e06fda17a9f44] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=2639ce7822b7aedb, duration=31 ms] 2024-11-04 14:51:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:51:48.591178+08:00 2024-11-04 14:51:49.289 INFO 21062 --- [ocp-schedules-17,04af0c723be9f529,0be12e8af08538a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:51:49.352 WARN 21062 --- [ocp-async-1,a60d64a8d3762f0c,aa8b1477cf4b96a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:49.384 INFO 21062 --- [ocp-async-2,ddff656b45e14996,05497041b6d0937e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:50.351 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ba3dd34d268d659d,88b28f8553510191] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=ba3dd34d268d659d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:50.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:50.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:50.378 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ba3dd34d268d659d,88b28f8553510191] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=ba3dd34d268d659d, duration=34 ms] 2024-11-04 14:51:50.983 INFO 21062 --- [prometheus-minute-schedule-1,3178b0cd84093aa0,8587344065bf9e3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:51.024 INFO 21062 --- [ocp-schedules-13,d3b9baa813550630,03982f52b78489f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:51:51.352 WARN 21062 --- [ocp-async-2,f644efd4be142136,a9ffeb7eda21bb67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:51.385 INFO 21062 --- [ocp-async-8,93ceddff0957d725,2dcfe6d8d47c20c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:51.973 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f450a577dd250016,f05934fbbba560c7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=f450a577dd250016, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:51.998 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f450a577dd250016,f05934fbbba560c7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=f450a577dd250016, duration=32 ms] 2024-11-04 14:51:52.199 INFO 21062 --- [ocp-schedules-26,0af2935ff507656b,e5867f082c870152] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:51:52.204 INFO 21062 --- [ocp-schedules-26,0af2935ff507656b,e5867f082c870152] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:51:52.204 INFO 21062 --- [ocp-schedules-26,0af2935ff507656b,e5867f082c870152] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:51:53.352 WARN 21062 --- [ocp-async-8,232f9d28533fe5c4,4146fea4089efc31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:53.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:53.370 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:53.385 INFO 21062 --- [ocp-async-6,e0b9a8c7b9f0ad23,866d3a9e715e29b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:53.639 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,76f12d22c6a1f072,9daea80cb4c45c6a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=76f12d22c6a1f072, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:53.664 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,76f12d22c6a1f072,9daea80cb4c45c6a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=76f12d22c6a1f072, duration=32 ms] 2024-11-04 14:51:53.983 INFO 21062 --- [prometheus-minute-schedule-1,db65845036414d68,cd7fd7f4e9d94bd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:55.321 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b782054eea683cd9,fad8d3734e6e65e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=b782054eea683cd9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:55.348 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,b782054eea683cd9,fad8d3734e6e65e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=b782054eea683cd9, duration=34 ms] 2024-11-04 14:51:55.352 WARN 21062 --- [ocp-async-6,0f9f57816f2e163a,b95923eed63249ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:55.385 INFO 21062 --- [ocp-async-9,70b70ca520959f3f,7f88a4ce9fd3ca8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:56.370 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:56.371 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:56.896 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,99617e78f96fe426,5e99b52902e7a85f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=99617e78f96fe426, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:56.922 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,99617e78f96fe426,5e99b52902e7a85f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=99617e78f96fe426, duration=32 ms] 2024-11-04 14:51:56.983 INFO 21062 --- [prometheus-minute-schedule-1,533acc9fa989bf38,882e683657508238] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:51:57.353 WARN 21062 --- [ocp-async-9,83de447132c302d6,0e71f23a477113a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:57.385 INFO 21062 --- [ocp-async-10,8564d1b953f52048,eac8baecc1834881] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:58.489 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,48227e9856d3971e,9b50f637cdf6167b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=48227e9856d3971e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:51:58.500 INFO 21062 --- [ocp-schedules-21,0c7c869749917e89,70b36c26b18bb34e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:51:58.500 INFO 21062 --- [ocp-schedules-19,f7b456458a7fa1c7,8691a73a35c18b33] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:51:58.502 INFO 21062 --- [ocp-schedules-11,cbb3b8b338990ff4,fba5f1b95bff3a87] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:51:58.504 INFO 21062 --- [ocp-schedules-21,0c7c869749917e89,70b36c26b18bb34e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:51:58.504 INFO 21062 --- [ocp-schedules-19,f7b456458a7fa1c7,8691a73a35c18b33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:51:58.506 INFO 21062 --- [ocp-schedules-4,8b7062ef16cf295e,5a16161ec82b0881] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:51:58.509 INFO 21062 --- [ocp-schedules-14,2b3deed5aed4463d,5942ceaed3984465] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:51:58.509 INFO 21062 --- [ocp-schedules-14,2b3deed5aed4463d,5942ceaed3984465] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:51:58.509 INFO 21062 --- [ocp-schedules-22,c63cebb74bdc564d,a6a50fb59148c872] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:51:58.509 INFO 21062 --- [ocp-schedules-3,4c8777f8a9efb32c,bb49f4daf8f654a9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:51:58.509 INFO 21062 --- [ocp-schedules-22,c63cebb74bdc564d,a6a50fb59148c872] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:51:58.509 INFO 21062 --- [ocp-schedules-3,4c8777f8a9efb32c,bb49f4daf8f654a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:51:58.514 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,48227e9856d3971e,9b50f637cdf6167b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=48227e9856d3971e, duration=33 ms] 2024-11-04 14:51:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:51:58.589666+08:00, lastProcessTime=2024-11-04T14:51:48.591178+08:00 2024-11-04 14:51:58.690 INFO 21062 --- [ocp-async-6,f90ff63ca62c4ea0,665975991419c546] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:51:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:51:58.589666+08:00 2024-11-04 14:51:59.318 INFO 21062 --- [ocp-schedules-26,1c65229167abc83f,1fc6e0d22a17b66e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:51:59.325 INFO 21062 --- [ocp-schedules-18,fa850a8ff6a672c7,439f3bfeea309eae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:51:59.353 WARN 21062 --- [ocp-async-8,b5807fd8aa6277ec,aaec402ff06dfd5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:51:59.371 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:51:59.372 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:51:59.385 INFO 21062 --- [ocp-async-6,aeda1aa27b7daa23,50f0e6d941538d85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:51:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2c7bc10c51291535,32fc3b06f5e746c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703120, interval=60 2024-11-04 14:52:00.323 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4ce90d4c44734858,c1ff0fa84a7991a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=4ce90d4c44734858, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:00.351 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4ce90d4c44734858,c1ff0fa84a7991a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=4ce90d4c44734858, duration=37 ms] 2024-11-04 14:52:01.027 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:52:01.027 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:52:01.027 INFO 21062 --- [ocp-schedules-20,e3ee277c5246be0a,16b8fbfd8d85ba7e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:52:01.035 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:52:01.035 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:52:01.036 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:52:01.036 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:52:01.042 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:52:01.042 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:52:01.042 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:52:01.042 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:52:01.079 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:52:01.088 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:52:01.088 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:52:01.088 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:52:01.088 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:52:01.088 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:52:01.098 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:52:01.353 WARN 21062 --- [ocp-async-6,7054262e57861fcf,aae6551c54f816cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:01.386 INFO 21062 --- [ocp-async-9,c4c8e7b87f301d3a,ba3a66148cb39f85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:01.645 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:52:01.984 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,3f94c5016c835778,a94f5e3a2b80e6ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=3f94c5016c835778, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:02.012 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,3f94c5016c835778,a94f5e3a2b80e6ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=3f94c5016c835778, duration=35 ms] 2024-11-04 14:52:02.252 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.209825388S 2024-11-04 14:52:02.252 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:52:02.350 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:52:02.350 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:52:02.372 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:02.373 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:02.381 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:52:02.443 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:52:02.443 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:52:02.443 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:52:02.443 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:52:02.443 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:52:02.443 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:52:02.454 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:52:02.629 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] o.s.boot.SpringApplication : Started application in 1.593 seconds (JVM running for 8536.308) 2024-11-04 14:52:02.629 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.5939337S 2024-11-04 14:52:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1cf2016b8eeaf93c,82b5ccdb82ac382f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:03.083 INFO 21062 --- [ocp-schedules-7,46ab840d2cb41ec0,7196096173d61300] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:52:03.353 WARN 21062 --- [ocp-async-9,0696d654875f6c5c,36877661804ae47f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:03.386 INFO 21062 --- [ocp-async-7,2645be1e0490e371,a4cf0cf766893047] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:03.641 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,59aae91d75cb5ae7,ede6defefda97e91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=59aae91d75cb5ae7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:03.670 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,59aae91d75cb5ae7,ede6defefda97e91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=59aae91d75cb5ae7, duration=38 ms] 2024-11-04 14:52:05.352 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,53e916d65d1ec752,7f11099b90e90ce3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=53e916d65d1ec752, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:05.353 WARN 21062 --- [ocp-async-7,85c3c87558af982c,fc6dd5c34a5e7a1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:05.373 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:05.374 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:05.378 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,53e916d65d1ec752,7f11099b90e90ce3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=53e916d65d1ec752, duration=32 ms] 2024-11-04 14:52:05.386 INFO 21062 --- [ocp-async-4,01ab81d6e73aa54a,2702edc8064f9001] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c22428cb352a8840,b107b9a844c501e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:07.010 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,cf4756b8a8f16435,d8f34ffa99992196] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=cf4756b8a8f16435, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:07.037 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,cf4756b8a8f16435,d8f34ffa99992196] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=cf4756b8a8f16435, duration=34 ms] 2024-11-04 14:52:07.354 WARN 21062 --- [ocp-async-4,ce66509f9dfe4d55,9dc5a9ee8d11d35c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:07.386 INFO 21062 --- [ocp-async-1,1678d323d96a8c5e,8e6a9891d008c8d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:08.088 INFO 21062 --- [ocp-schedules-12,e8e581004252b18f,27a4a1c373913f9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:52:08.089 INFO 21062 --- [ocp-schedules-12,bca7581115eeb35f,c9a667141ab80b6a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:52:08.093 INFO 21062 --- [ocp-schedules-12,bca7581115eeb35f,c9a667141ab80b6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:08.127 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:52:08.127 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:52:08.374 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:08.375 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:08.505 INFO 21062 --- [ocp-schedules-21,0c7c869749917e89,70b36c26b18bb34e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:52:08.505 INFO 21062 --- [ocp-schedules-19,f7b456458a7fa1c7,8691a73a35c18b33] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:52:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:52:08.589913+08:00, lastProcessTime=2024-11-04T14:51:58.589666+08:00 2024-11-04 14:52:08.685 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c46e1b012d8185af,eeffb50fa8e894c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=c46e1b012d8185af, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:08.690 INFO 21062 --- [ocp-async-10,eddda1cd6b78ecb6,2ea5a125930b35bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:52:08.713 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,c46e1b012d8185af,eeffb50fa8e894c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=c46e1b012d8185af, duration=36 ms] 2024-11-04 14:52:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:52:08.589913+08:00 2024-11-04 14:52:08.983 INFO 21062 --- [prometheus-minute-schedule-1,e50df46a8b367d6f,73b7ae900ad77617] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:09.354 WARN 21062 --- [ocp-async-9,6efc3c27be1aeeec,91bf0a107c5e49ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:09.366 INFO 21062 --- [ocp-schedules-5,bc58d2f0ab02b5f6,2f147b54d5b3337c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:52:09.386 INFO 21062 --- [ocp-async-7,7ff34c744c4b5e5f,7b51f05002c152c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:52:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:52:10.255 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,9397416008575641,533b9b971bca0232] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9397416008575641, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:10.283 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,9397416008575641,533b9b971bca0232] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9397416008575641, duration=34 ms] 2024-11-04 14:52:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:52:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:52:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:52:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:52:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:52:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:52:11.030 INFO 21062 --- [ocp-schedules-3,8989f8417043bab7,96319ebbcb9679ae] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:52:11.354 WARN 21062 --- [ocp-async-7,0467b74e12425823,c29ec3ce814ba7f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:11.375 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:11.376 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:11.386 INFO 21062 --- [ocp-async-3,cb8354132826e63b,0b035abbd53cad0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:11.687 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703060, nodeCount=422, affectRows=422 2024-11-04 14:52:11.689 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703060, nodeCount=419, validCount=419 2024-11-04 14:52:11.853 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9f9cda1f3118ad5b,904b115b35be6692] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9f9cda1f3118ad5b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:11.882 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,9f9cda1f3118ad5b,904b115b35be6692] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9f9cda1f3118ad5b, duration=40 ms] 2024-11-04 14:52:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9456b9b943d896b0,337c301206b15bea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:12.207 INFO 21062 --- [ocp-schedules-14,3221aa2c6ec9cc5d,691f82125b687b7e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:52:12.213 INFO 21062 --- [ocp-schedules-14,3221aa2c6ec9cc5d,691f82125b687b7e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:52:12.213 INFO 21062 --- [ocp-schedules-14,3221aa2c6ec9cc5d,691f82125b687b7e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:52:13.354 WARN 21062 --- [ocp-async-4,2fec4f9f0ffa95ce,c3f40772aeef816f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:13.387 INFO 21062 --- [ocp-async-5,6f39c904b034f587,e9b475af29dcc918] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:13.435 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,e0c63778199df80a,e29f75fed3810e06] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=e0c63778199df80a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:13.462 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,e0c63778199df80a,e29f75fed3810e06] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=e0c63778199df80a, duration=34 ms] 2024-11-04 14:52:13.984 INFO 21062 --- [prometheus-cache-monitor1,ce5df3afd5dbdbd6,b06902d8298c21d8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:52:13.985 INFO 21062 --- [prometheus-cache-monitor1,ce5df3afd5dbdbd6,b06902d8298c21d8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:52:14.376 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:14.377 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d511bd7d6e25a781,fe193e05e4d8f0f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:15.058 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,79a44c64a0ef5549,13a380ad2221688f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=79a44c64a0ef5549, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:15.084 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,79a44c64a0ef5549,13a380ad2221688f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=79a44c64a0ef5549, duration=33 ms] 2024-11-04 14:52:15.354 WARN 21062 --- [ocp-async-1,9d06c203b301306f,b0e6434451529745] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:15.387 INFO 21062 --- [ocp-async-2,f271b55d1254a21a,06e55c4ef5ff1ff2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:16.693 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,cb6d4668a7c16117,50cc1588aafa007a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=cb6d4668a7c16117, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:16.721 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,cb6d4668a7c16117,50cc1588aafa007a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=cb6d4668a7c16117, duration=35 ms] 2024-11-04 14:52:17.355 WARN 21062 --- [ocp-async-10,fc203f40e9326902,8809b45fe5f60a0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:17.377 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:17.378 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:17.389 INFO 21062 --- [ocp-async-8,8f0a7dd62b4b5125,fe426c957d38539c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:17.983 INFO 21062 --- [prometheus-minute-schedule-1,500145d531b0c2fd,776dbe559aa762b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:18.396 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,13f769397550548c,9c7f179224ca21ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=13f769397550548c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:18.422 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,13f769397550548c,9c7f179224ca21ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=13f769397550548c, duration=32 ms] 2024-11-04 14:52:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:52:18.590348+08:00, lastProcessTime=2024-11-04T14:52:08.589913+08:00 2024-11-04 14:52:18.690 INFO 21062 --- [ocp-async-5,33a33303d80b0d86,5873210f70a8ffde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:52:18.723 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/d997e146ce2e4c02/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/d997e146ce2e4c02/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:52:18.727 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/d997e146ce2e4c02/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/d997e146ce2e4c02/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:52:18.730 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/d997e146ce2e4c02/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:52:18.733 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/d997e146ce2e4c02/install_package.sh, exit code: 2, cause:null 2024-11-04 14:52:18.736 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/d997e146ce2e4c02/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/d997e146ce2e4c02/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:52:18.753 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,06c03a60fad98b26] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: FAILED 2024-11-04 14:52:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:52:18.590348+08:00 2024-11-04 14:52:19.102 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/7c055c56402f4c0b/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/7c055c56402f4c0b/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:52:19.108 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/7c055c56402f4c0b/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/7c055c56402f4c0b/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 14:52:19.111 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/7c055c56402f4c0b/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 14:52:19.113 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/7c055c56402f4c0b/install_package.sh, exit code: 2, cause:null 2024-11-04 14:52:19.116 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/7c055c56402f4c0b/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/7c055c56402f4c0b/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:52:19.127 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,06289aec79139802] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: FAILED 2024-11-04 14:52:19.169 INFO 21062 --- [ocp-schedules-29,05b43b560de54af8,da7015e1a82c5a88] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:52:19.173 INFO 21062 --- [ocp-schedules-29,05b43b560de54af8,da7015e1a82c5a88] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:19.174 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:52:19.315 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=2024-11-04T14:44:56.529+08:00, subtasks=[SubtaskInstanceEntity{id=68, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=33, startTime=2024-11-04T14:11:16.789+08:00, endTime=2024-11-04T14:11:16.854+08:00}, SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busin 2024-11-04 14:52:19.324 WARN 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:52:19.325 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 14:52:19.326 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:2, status:FAILED 2024-11-04 14:52:19.355 WARN 21062 --- [ocp-async-4,4a30d7851539f96d,c05c20ce0a5fa52c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:19.389 INFO 21062 --- [ocp-async-5,e09ea23053cb6cbf,84bb49ac611b4f49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:19.393 INFO 21062 --- [ocp-schedules-9,63d84f7fe28deb6c,d9771fd934af276b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:52:19.948 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,bd1e363852db5add,02e5a436b8143c26] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=bd1e363852db5add, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 14:52:19.975 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,bd1e363852db5add,02e5a436b8143c26] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=bd1e363852db5add, duration=35 ms] 2024-11-04 14:52:20.378 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:20.379 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:20.548 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,35df06704edb93bb,14b94271e5db8da1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=35df06704edb93bb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 14:52:20.556 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,35df06704edb93bb,14b94271e5db8da1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=35df06704edb93bb, duration=15 ms] 2024-11-04 14:52:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9b1786be63dd9d59,8fb63ec5ee227376] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:21.033 INFO 21062 --- [ocp-schedules-26,43847474acfca2e6,8f00c6a3b0466400] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:52:21.355 WARN 21062 --- [ocp-async-1,3a3b162459effcdc,9eb69b5af8d82c0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:21.389 INFO 21062 --- [ocp-async-2,87fd3cc85db31086,37aac26ec0acb45c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:23.355 WARN 21062 --- [ocp-async-10,c3ba511fd9e7fb58,e1c9a566c756ac29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:23.379 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:23.380 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:23.389 INFO 21062 --- [ocp-async-8,ddabcef531498be7,7c02899ebb61df2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:23.983 INFO 21062 --- [prometheus-minute-schedule-1,68d52582930a4634,e166d777cd5f29a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:25.355 WARN 21062 --- [ocp-async-3,f9b35db26b471130,9751696db1cf60da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:25.390 INFO 21062 --- [ocp-async-6,14c6db94aabaf668,93ba0fc699ff1f00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:26.380 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:26.381 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:26.983 INFO 21062 --- [prometheus-minute-schedule-1,143fc0e7c059567b,3a1c5331f3a57d29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8507a1ecaf3e43c2,9b70b118728c4d0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:52:27.355 WARN 21062 --- [ocp-async-5,fdc0842d181df382,0af5e186b13e689f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:27.390 INFO 21062 --- [ocp-async-9,eada3063cd7dc1ed,5ede583eb29f3638] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:28.498 INFO 21062 --- [ocp-schedules-10,c9550ba01ad153f7,d46402846e1f56be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:52:28.499 INFO 21062 --- [ocp-schedules-26,8b7123de035c6824,43c6902d8dab7506] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:52:28.500 INFO 21062 --- [ocp-schedules-15,9fdcfdf65c0d45dd,d9607376c1520808] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:52:28.500 INFO 21062 --- [ocp-schedules-17,884953e09a394aa5,7fccc94b215b28b6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:52:28.502 INFO 21062 --- [ocp-schedules-10,c9550ba01ad153f7,d46402846e1f56be] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:52:28.502 INFO 21062 --- [ocp-schedules-25,f8a36880ec4eafc6,6f90f17c7b84b8d3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:52:28.504 INFO 21062 --- [ocp-schedules-15,9fdcfdf65c0d45dd,d9607376c1520808] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:28.504 INFO 21062 --- [ocp-schedules-17,884953e09a394aa5,7fccc94b215b28b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:28.504 INFO 21062 --- [ocp-schedules-19,2bd97d7a4291256e,65a4aba2da91cd36] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:52:28.504 INFO 21062 --- [ocp-schedules-5,5dcc2f09febc87c2,1bdb9024699ddce5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:52:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:52:28.506 INFO 21062 --- [ocp-schedules-26,8b7123de035c6824,43c6902d8dab7506] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:52:28.507 INFO 21062 --- [ocp-schedules-19,2bd97d7a4291256e,65a4aba2da91cd36] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:28.509 INFO 21062 --- [ocp-schedules-6,5fa6cbdc86bca74a,95d1e5f2701f3b3c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:52:28.510 INFO 21062 --- [ocp-schedules-6,5fa6cbdc86bca74a,95d1e5f2701f3b3c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:52:28.510 INFO 21062 --- [ocp-schedules-16,f05e6c16d784ddeb,fd80bce1c7a93141] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:52:28.510 INFO 21062 --- [ocp-schedules-16,f05e6c16d784ddeb,fd80bce1c7a93141] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:52:28.510 INFO 21062 --- [ocp-schedules-30,c49efe91ac6eb3c0,1f014c7258508378] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:52:28.510 INFO 21062 --- [ocp-schedules-30,c49efe91ac6eb3c0,1f014c7258508378] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:52:28.539 INFO 21062 --- [ocp-schedules-18,9bea94207fc74a29,b6b9bd06cb38e83e] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:52:28.539 INFO 21062 --- [ocp-schedules-18,9bea94207fc74a29,b6b9bd06cb38e83e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:52:28.579 INFO 21062 --- [pool-12-thread-3,f786525861395776,1ab7d11545e7b84a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:52:28.579 INFO 21062 --- [pool-12-thread-1,f786525861395776,95c53f14230cc980] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:52:28.579 INFO 21062 --- [pool-12-thread-4,f786525861395776,7e18095c0ed11c22] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:52:28.580 INFO 21062 --- [pool-12-thread-2,f786525861395776,379cd487f3223694] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:52:28.582 INFO 21062 --- [ocp-schedules-20,87e1a411e97b9f55,ebb40d3bb6070101] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:52:28.582 INFO 21062 --- [ocp-schedules-20,87e1a411e97b9f55,ebb40d3bb6070101] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:52:28.584 INFO 21062 --- [pool-12-thread-3,f786525861395776,1ab7d11545e7b84a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:52:28.584 INFO 21062 --- [pool-12-thread-3,f786525861395776,1ab7d11545e7b84a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:52:28.585 INFO 21062 --- [ForkJoinPool-11-worker-3,f786525861395776,f426e956f6673a22] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:52:28.585 INFO 21062 --- [ForkJoinPool-11-worker-3,f786525861395776,f426e956f6673a22] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:52:28.585 INFO 21062 --- [ForkJoinPool-11-worker-3,f786525861395776,f426e956f6673a22] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:52:28.585 INFO 21062 --- [pool-12-thread-2,f786525861395776,379cd487f3223694] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:52:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:52:28.589145+08:00, lastProcessTime=2024-11-04T14:52:18.590348+08:00 2024-11-04 14:52:28.607 INFO 21062 --- [pool-12-thread-4,f786525861395776,7e18095c0ed11c22] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:52:28.608 INFO 21062 --- [pool-12-thread-4,f786525861395776,7e18095c0ed11c22] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:52:28.609 INFO 21062 --- [pool-12-thread-4,f786525861395776,7e18095c0ed11c22] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:52:28.612 INFO 21062 --- [ForkJoinPool-11-worker-3,f786525861395776,4ea975bdf8e99dd8] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:52:28.612 INFO 21062 --- [pool-12-thread-1,f786525861395776,95c53f14230cc980] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:52:28.690 INFO 21062 --- [ocp-async-4,eba3af0ef19ae4a7,bc5d4be735063d23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:52:28.800 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:52:28.589145+08:00 2024-11-04 14:52:28.905 INFO 21062 --- [ocp-schedules-5,d6d86e562de64f97,67349eb47aaf16c4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:52:28.912 INFO 21062 --- [ocp-schedules-16,480618964a3b6913,7815046ff28a6958] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 14:52:28.913 INFO 21062 --- [ocp-schedules-16,480618964a3b6913,7815046ff28a6958] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 14:52:28.913 INFO 21062 --- [ocp-schedules-16,480618964a3b6913,7815046ff28a6958] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=422 2024-11-04 14:52:28.938 INFO 21062 --- [ocp-schedules-16,480618964a3b6913,7815046ff28a6958] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=422 2024-11-04 14:52:29.174 INFO 21062 --- [ocp-schedules-29,05b43b560de54af8,da7015e1a82c5a88] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:52:29.178 INFO 21062 --- [ocp-schedules-10,69656c0fc51d01e3,c66b954af9faf74e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 14:52:29.182 INFO 21062 --- [ocp-schedules-10,69656c0fc51d01e3,c66b954af9faf74e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:29.187 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:52:29.203 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:52:29.205 WARN 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:52:29.206 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:52:29.207 WARN 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:52:29.214 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:52:29.215 WARN 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:52:29.216 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:52:29.218 ERROR 21062 --- [ocp-schedules-10,69656c0fc51d01e3,c66b954af9faf74e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:52:29.218 INFO 21062 --- [ocp-schedules-10,69656c0fc51d01e3,c66b954af9faf74e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 14:52:29.219 WARN 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:52:29.225 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:52:29.226 WARN 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:52:29.227 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:52:29.227 INFO 21062 --- [ocp-schedules-27,425386566e9bf76f,c3ce9a63ea8851ca] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:52:29.227 INFO 21062 --- [ocp-schedules-27,425386566e9bf76f,c3ce9a63ea8851ca] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:52:29.228 WARN 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:52:29.234 INFO 21062 --- [ocp-schedules-28,33a7317b22ba4d66,6cf72ba3428757a5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:52:29.329 INFO 21062 --- [ocp-schedules-23,5b303e4399874965,d9be5cede923c235] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:52:29.356 WARN 21062 --- [ocp-async-8,dff18afbaff028d8,e638360a097c75b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:29.358 INFO 21062 --- [ocp-schedules-2,4f7e5c0080c73a5c,89ca9b600aada600] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:52:29.363 INFO 21062 --- [ocp-schedules-2,4f7e5c0080c73a5c,89ca9b600aada600] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:52:29.363 INFO 21062 --- [ocp-schedules-2,4f7e5c0080c73a5c,89ca9b600aada600] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:52:29.381 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:29.382 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:29.390 INFO 21062 --- [ocp-async-4,704dc472f0a50900,009343461ef05711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:29.434 INFO 21062 --- [ocp-schedules-24,44c37238d47aa960,423cbf86968816da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:52:29.574 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:52:29.574 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:52:29.582 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:52:29.582 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:52:29.582 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:52:29.593 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:52:29.593 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:52:29.593 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:52:29.596 INFO 21062 --- [ocp-schedules-13,eef5f32123506c41,00622d0f0e27c4b6] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:52:29.760 INFO 21062 --- [ocp-schedules-11,f30dbca647b0074b,e84cfa9734f10523] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:52:29.760 INFO 21062 --- [ocp-schedules-11,f30dbca647b0074b,e84cfa9734f10523] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:52:29.766 INFO 21062 --- [ocp-schedules-11,f30dbca647b0074b,e84cfa9734f10523] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:52:29.767 INFO 21062 --- [ocp-schedules-11,f30dbca647b0074b,e84cfa9734f10523] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:52:29.771 INFO 21062 --- [ocp-schedules-11,f30dbca647b0074b,e84cfa9734f10523] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:52:29.771 INFO 21062 --- [ocp-schedules-11,f30dbca647b0074b,e84cfa9734f10523] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:52:29.983 INFO 21062 --- [prometheus-minute-schedule-1,0c94c42afe8aa401,fd375d3b0ad68aa6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:30.087 INFO 21062 --- [ocp-schedules-21,6ce658734d789611,cec64485da16048f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:52:30.087 INFO 21062 --- [ocp-schedules-21,6ce658734d789611,cec64485da16048f] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:52:30.116 INFO 21062 --- [pool-47-thread-1,6ce658734d789611,584a98c45a5f05e3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:52:30.144 WARN 21062 --- [ocp-schedules-21,6ce658734d789611,cec64485da16048f] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:52:30.144 INFO 21062 --- [ocp-schedules-21,6ce658734d789611,cec64485da16048f] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:52:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:52:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:52:31.036 INFO 21062 --- [ocp-schedules-20,4ac0a6e50ec68c31,5a9a0a78c6068fd4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:52:31.356 WARN 21062 --- [ocp-async-4,44a4e8211f014dbb,058981c39474fdb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:31.390 INFO 21062 --- [ocp-async-1,75db5445f97a4abf,8bca0fd959651466] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:32.216 INFO 21062 --- [ocp-schedules-27,958d51b60fcfd69f,77cbb3bf6386bbde] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:52:32.221 INFO 21062 --- [ocp-schedules-27,958d51b60fcfd69f,77cbb3bf6386bbde] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:52:32.221 INFO 21062 --- [ocp-schedules-27,958d51b60fcfd69f,77cbb3bf6386bbde] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:52:32.358 INFO 21062 --- [ocp-schedules-25,ea705daddbe7b714,81eac3a798c71072] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:52:32.358 INFO 21062 --- [ocp-schedules-25,ea705daddbe7b714,81eac3a798c71072] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:52:32.382 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:32.383 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:32.385 INFO 21062 --- [ocp-schedules-25,ea705daddbe7b714,81eac3a798c71072] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:52:32.385 INFO 21062 --- [ocp-schedules-25,ea705daddbe7b714,81eac3a798c71072] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:52:32.386 INFO 21062 --- [ocp-schedules-20,8dcade56e7608dc2,6e583762bf888c44] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:52:32.580 INFO 21062 --- [ocp-schedules-30,d383a6e6b71585ac,25345a8c01261a0e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:52:32.580+08:00, clusters=[1] 2024-11-04 14:52:32.582 INFO 21062 --- [ocp-schedules-30,d383a6e6b71585ac,25345a8c01261a0e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:52:32.800 INFO 21062 --- [ocp-schedules-22,fcdf7ab7d73b8a31,699ef1997790b233] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:52:32.800+08:00, clusters=[1] 2024-11-04 14:52:32.804 INFO 21062 --- [ocp-schedules-22,fcdf7ab7d73b8a31,699ef1997790b233] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:52:32.815 INFO 21062 --- [ocp-schedules-7,f6ae7e6cf82af88c,5db71fee2d4a1e74] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:52:32.818 INFO 21062 --- [ocp-schedules-7,f6ae7e6cf82af88c,5db71fee2d4a1e74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:32.877 ERROR 21062 --- [ocp-schedules-7,f6ae7e6cf82af88c,5db71fee2d4a1e74] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:52:32.877 INFO 21062 --- [ocp-schedules-7,f6ae7e6cf82af88c,5db71fee2d4a1e74] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:52:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b2b581278537ba57,01021c44ef126b02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:33.356 WARN 21062 --- [ocp-async-1,91bf8758b793d400,c664e9371e002e60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:33.390 INFO 21062 --- [ocp-async-7,917f76c36707445e,b9c43d95c1a23fed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:34.182 INFO 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:52:34.182 INFO 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:52:34.182 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:52:34.182 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:52:34.191 INFO 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:52:34.192 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:52:34.192 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:52:34.193 WARN 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:52:34.194 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:52:34.194 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:52:34.194 INFO 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:52:34.195 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:52:34.195 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:52:34.195 WARN 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:52:34.196 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:52:34.196 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:52:34.203 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:52:34.203 INFO 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:52:34.203 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:52:34.217 ERROR 21062 --- [ocp-schedules-6,618acffad26cb710,0c1fd2c6829df85a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:52:35.356 WARN 21062 --- [ocp-async-7,79111454387bc1c0,39154c0b38b9ff45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:35.383 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:35.383 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:35.391 INFO 21062 --- [ocp-async-3,6ee0c52304186bc6,da52bde7285c2bb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b012f9259aaf9dc6,298888dfb45b15d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:37.357 WARN 21062 --- [ocp-async-3,5ee4146b0a78d9dc,c13ede727781bef0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:37.391 INFO 21062 --- [ocp-async-5,75c8d866d2622ec5,0923d273bb24057c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:38.127 INFO 21062 --- [ocp-schedules-12,bca7581115eeb35f,c9a667141ab80b6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:52:38.127 INFO 21062 --- [ocp-schedules-12,2a005b78672cebe1,a32a220181ce2586] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:52:38.132 INFO 21062 --- [ocp-schedules-12,2a005b78672cebe1,a32a220181ce2586] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:38.161 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:52:38.161 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:52:38.384 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:38.384 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:38.505 INFO 21062 --- [ocp-schedules-17,884953e09a394aa5,7fccc94b215b28b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:52:38.505 INFO 21062 --- [ocp-schedules-15,9fdcfdf65c0d45dd,d9607376c1520808] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:52:38.507 INFO 21062 --- [ocp-schedules-19,2bd97d7a4291256e,65a4aba2da91cd36] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:52:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:52:38.589954+08:00, lastProcessTime=2024-11-04T14:52:28.589145+08:00 2024-11-04 14:52:38.690 INFO 21062 --- [ocp-async-10,0a1b25afba0f78a9,333378278d7901c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:52:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:52:38.589954+08:00 2024-11-04 14:52:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a42af883e570ba48,0287fff624702efd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:39.357 WARN 21062 --- [ocp-async-2,7b6c94dbf0b80976,e49c67561b041d46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:39.391 INFO 21062 --- [ocp-async-10,d848e446b77dc105,819a8ab0004445dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:39.458 INFO 21062 --- [ocp-schedules-3,b4e75802f20d14d3,a6c5666625db02e4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:52:40.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:52:40.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:52:41.039 INFO 21062 --- [ocp-schedules-17,4d43af290c696793,e952a7aca8e018a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:52:41.357 WARN 21062 --- [ocp-async-10,ca4069cb101902c5,b0aab51773472c6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:41.385 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:41.385 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:41.391 INFO 21062 --- [ocp-async-4,4069bea69950ac1f,588dad7bba97fd50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:41.467 INFO 21062 --- [ocp-schedules-16,a109d0304892da83,5289d92bd1762839] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:52:41.471 INFO 21062 --- [ocp-schedules-16,a109d0304892da83,5289d92bd1762839] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:41.553 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:52:41.555 ERROR 21062 --- [ocp-schedules-16,a109d0304892da83,5289d92bd1762839] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:52:41.555 INFO 21062 --- [ocp-schedules-16,a109d0304892da83,5289d92bd1762839] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:52:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:52:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:52:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:52:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:52:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:52:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:52:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:52:41.983 INFO 21062 --- [prometheus-minute-schedule-1,e35cfacc98fbec7a,bae92b0a2629873c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:43.357 WARN 21062 --- [ocp-async-4,c4f3f12766647c45,d6fa1fe672ec1a4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:43.391 INFO 21062 --- [ocp-async-1,ab1798d68bc2d7a5,3126c0c4c35b40bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:44.385 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:44.386 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5739c50ae9a2835c,5ca354e796f5754a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:45.357 WARN 21062 --- [ocp-async-1,161f5df6fb1c9b58,73ed638fdeb09fe8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:45.392 INFO 21062 --- [ocp-async-7,e2c371a85f478dff,4b53a71f3ac07362] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:47.357 WARN 21062 --- [ocp-async-7,7693beae6a77e537,c9a504e06262c92a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:47.386 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:47.387 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:47.392 INFO 21062 --- [ocp-async-3,a97594842f52d4fc,0fb9f669ca920f68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:47.983 INFO 21062 --- [prometheus-minute-schedule-1,15a83486a6bb2c9e,0e2e0c115ce10362] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:52:48.590786+08:00, lastProcessTime=2024-11-04T14:52:38.589954+08:00 2024-11-04 14:52:48.690 INFO 21062 --- [ocp-async-1,5eb1eb427c2d45c5,e9a8807d9525d7ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:52:48.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:52:48.590786+08:00 2024-11-04 14:52:49.358 WARN 21062 --- [ocp-async-4,e3de57761a455623,e12ec8732532e399] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:49.392 INFO 21062 --- [ocp-async-1,f132b46d0365eaa7,a7fee2d65e82b1c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:49.504 INFO 21062 --- [ocp-schedules-29,d91e18454a3da92d,3da496f9752d06a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:52:50.387 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:50.388 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8c86c4ba7de3900b,f3bf8680a5835ebd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:51.042 INFO 21062 --- [ocp-schedules-20,e2aa9650d3a23931,e58e1860fdb01f8a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:52:51.358 WARN 21062 --- [ocp-async-1,e52319e2b15a7a9c,d1fc1053c79863b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:51.392 INFO 21062 --- [ocp-async-9,f5562e31b63b16f3,2ae637df96a1fb83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:52.224 INFO 21062 --- [ocp-schedules-30,9539ed45d50810f0,0022c053b7d12926] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:52:52.229 INFO 21062 --- [ocp-schedules-30,9539ed45d50810f0,0022c053b7d12926] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:52:52.229 INFO 21062 --- [ocp-schedules-30,9539ed45d50810f0,0022c053b7d12926] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:52:53.358 WARN 21062 --- [ocp-async-9,fa96a95c58c1cd36,e20c5eddec9064fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:53.388 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:53.389 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:53.392 INFO 21062 --- [ocp-async-8,81a0aa2d3b0c9cdd,2b768e16a7d41027] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e332872148945b48,ee31fb62e316945d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:55.358 WARN 21062 --- [ocp-async-8,533046011cd82595,162aceb0f1ecfe4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:55.392 INFO 21062 --- [ocp-async-6,cac6a472fafb950d,dca37710736ba53b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:56.389 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:56.390 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:56.983 INFO 21062 --- [prometheus-minute-schedule-1,965e67c87d8d5ec1,6c13fcc558d7b9eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:52:57.358 WARN 21062 --- [ocp-async-6,a490f4deae5afc10,c230b69b4476465a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:57.393 INFO 21062 --- [ocp-async-2,90a22261d2962567,6e278564ab574cbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:58.500 INFO 21062 --- [ocp-schedules-1,4274883005837b3a,0fd20b85933efbb2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:52:58.500 INFO 21062 --- [ocp-schedules-28,f45824b2c215d5ad,f31689fa2f4f66f5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:52:58.503 INFO 21062 --- [ocp-schedules-18,fdd42ee41d3eb5c3,974e9b2d839743df] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:52:58.504 INFO 21062 --- [ocp-schedules-1,4274883005837b3a,0fd20b85933efbb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:58.504 INFO 21062 --- [ocp-schedules-28,f45824b2c215d5ad,f31689fa2f4f66f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:52:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:52:58.505 INFO 21062 --- [ocp-schedules-15,03e1cd375c7b34bd,553a56a4dcb27f74] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:52:58.511 INFO 21062 --- [ocp-schedules-9,d24d88ff96c680ff,e134fef573aebcd0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:52:58.511 INFO 21062 --- [ocp-schedules-19,84e00ba015bd3c30,1c916131cc5318d7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:52:58.511 INFO 21062 --- [ocp-schedules-22,06dcf99648f04a7e,d7bcbb20e5089518] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:52:58.511 INFO 21062 --- [ocp-schedules-19,84e00ba015bd3c30,1c916131cc5318d7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:52:58.511 INFO 21062 --- [ocp-schedules-22,06dcf99648f04a7e,d7bcbb20e5089518] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:52:58.511 INFO 21062 --- [ocp-schedules-9,d24d88ff96c680ff,e134fef573aebcd0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:52:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:52:58.589595+08:00, lastProcessTime=2024-11-04T14:52:48.590786+08:00 2024-11-04 14:52:58.691 INFO 21062 --- [ocp-async-3,f367cdd331977646,409ac8730b4a56d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:52:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:52:58.589595+08:00 2024-11-04 14:52:59.331 INFO 21062 --- [ocp-schedules-8,60e6676b3a870bf2,8bc6c2d0df4f7308] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:52:59.358 WARN 21062 --- [ocp-async-9,7e26970e16ac3fc9,8038198a7c7abcd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:52:59.390 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:52:59.391 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:52:59.393 INFO 21062 --- [ocp-async-3,a474d57f3784eb43,21c6c68665708e80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:52:59.530 INFO 21062 --- [ocp-schedules-19,150469d6cb0f5dee,1cd2bacbd62be4b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:52:59.983 INFO 21062 --- [prometheus-minute-schedule-1,cb515e61d10d6f30,3d39c37860bb2c8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703180, interval=60 2024-11-04 14:53:01.046 INFO 21062 --- [ocp-schedules-4,7ac4734d2ecc5fac,341f386df229e410] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:53:01.359 WARN 21062 --- [ocp-async-8,bcc6d68d59805ee3,43a5f770d20be6ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:01.393 INFO 21062 --- [ocp-async-5,e8aa8d544a9d39f7,500b2cd0d82ddd06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:02.391 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:02.391 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7da913c1ab3d3694,8dc0c678dde9eb0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:03.359 WARN 21062 --- [ocp-async-6,f52932270b0375e9,aa7d2076cb94d230] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:03.393 INFO 21062 --- [ocp-async-7,8ea5bd576a756981,29f6483626997c4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:05.359 WARN 21062 --- [ocp-async-2,54393e7ee398371d,61a9a65a8a0f3313] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:05.392 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:05.392 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:05.393 INFO 21062 --- [ocp-async-10,a1957564e6411d82,aabf7c62939bdd96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:05.983 INFO 21062 --- [prometheus-minute-schedule-1,5e64b40484415d9f,a33a6eeae34f38dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:07.359 WARN 21062 --- [ocp-async-3,1943ea539f8d1fc6,cd2d03fbda199a27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:07.393 INFO 21062 --- [ocp-async-4,412a9402f0a2cf73,75b16e9120f129df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:08.162 INFO 21062 --- [ocp-schedules-12,2a005b78672cebe1,a32a220181ce2586] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:53:08.162 INFO 21062 --- [ocp-schedules-12,7dc727b9d860912b,9d0f04d4bdeb8cbd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:53:08.166 INFO 21062 --- [ocp-schedules-12,7dc727b9d860912b,9d0f04d4bdeb8cbd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:08.193 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:53:08.193 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:53:08.392 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:08.393 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:08.505 INFO 21062 --- [ocp-schedules-28,f45824b2c215d5ad,f31689fa2f4f66f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:53:08.505 INFO 21062 --- [ocp-schedules-1,4274883005837b3a,0fd20b85933efbb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:53:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:53:08.589672+08:00, lastProcessTime=2024-11-04T14:52:58.589595+08:00 2024-11-04 14:53:08.691 INFO 21062 --- [ocp-async-7,49f8721e8ab1b6cc,5a6ff2333b99a492] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:53:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:53:08.589672+08:00 2024-11-04 14:53:08.983 INFO 21062 --- [prometheus-minute-schedule-1,e733da6f583908b4,3a0174cd995b4b30] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:09.359 WARN 21062 --- [ocp-async-6,6885034d8cd94949,8cf57577cfe5e477] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:09.394 INFO 21062 --- [ocp-async-7,a941974de6092aca,83b3ed9010adb86a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:09.576 INFO 21062 --- [ocp-schedules-6,cf768240a498252e,8c4bc06b24ae6f7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:53:10.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:53:10.002 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:53:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:53:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:53:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:53:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:53:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:53:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:53:11.051 INFO 21062 --- [ocp-schedules-27,1ebd72d63955cd4f,f9fc4694651c672c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:53:11.359 WARN 21062 --- [ocp-async-2,a4639bba6d07cb90,ce91e6943bcbd3ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:11.393 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:11.394 INFO 21062 --- [ocp-async-10,a77e2ac42cba151b,9290619ad3527dbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:11.394 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:11.689 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703120, nodeCount=215, affectRows=215 2024-11-04 14:53:11.689 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703120, nodeCount=207, affectRows=207 2024-11-04 14:53:11.690 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703120, nodeCount=214, validCount=214 2024-11-04 14:53:11.690 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703120, nodeCount=205, validCount=205 2024-11-04 14:53:11.983 INFO 21062 --- [prometheus-minute-schedule-1,bb79b398ebe04283,714e58522f5ae108] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:12.231 INFO 21062 --- [ocp-schedules-13,71e4e009dba02360,02fb9dae5e112c22] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:53:12.236 INFO 21062 --- [ocp-schedules-13,71e4e009dba02360,02fb9dae5e112c22] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:53:12.236 INFO 21062 --- [ocp-schedules-13,71e4e009dba02360,02fb9dae5e112c22] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:53:13.360 WARN 21062 --- [ocp-async-3,4ad7c0b47b6ebaf7,b3762bb3e555d9a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:13.394 INFO 21062 --- [ocp-async-4,8be2a198aef247f0,7a55fc14edbd3b00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:13.985 INFO 21062 --- [prometheus-cache-monitor1,01c1b68263343d65,6be1313962c74039] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:53:13.985 INFO 21062 --- [prometheus-cache-monitor1,01c1b68263343d65,6be1313962c74039] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:53:14.358 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:47:14.339, end=2024-11-04T14:52:14.356} 2024-11-04 14:53:14.358 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:47:14.339, end=2024-11-04T14:52:14.356} 2024-11-04 14:53:14.368 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:53:14.368 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:53:14.368 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:53:14.368 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:53:14.394 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:14.395 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:14.983 INFO 21062 --- [prometheus-minute-schedule-1,167f9b07d801e023,9310ca13a9036810] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:15.313 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:49:15.296, end=2024-11-04T14:51:15.311} 2024-11-04 14:53:15.318 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:53:15.318 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:53:15.360 WARN 21062 --- [ocp-async-5,f5f8858b3a312d1a,d70474b7187a7da8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:15.394 INFO 21062 --- [ocp-async-1,a629bff33ef521ac,6a7474626ef31825] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:17.360 WARN 21062 --- [ocp-async-7,f9e09fd65f1a5e04,4f37b8d25fa728e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:17.394 INFO 21062 --- [ocp-async-9,c3231ccbdd78aa8f,c51e13a458b9c4bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:17.395 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:17.396 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ff40e7ed45e28158,6a5b36f739d1ef74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:53:18.591140+08:00, lastProcessTime=2024-11-04T14:53:08.589672+08:00 2024-11-04 14:53:18.691 INFO 21062 --- [ocp-async-4,f8677c33be973188,e372824c38bf4583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:53:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:53:18.591140+08:00 2024-11-04 14:53:19.360 WARN 21062 --- [ocp-async-3,dabee918fdaae3d8,f94010d157c82b22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:19.394 INFO 21062 --- [ocp-async-4,66f10d471b85dd0f,a6663f2e7447d030] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:19.602 INFO 21062 --- [ocp-schedules-27,e26ef26f558225aa,51910f521cabf643] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:53:20.396 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:20.397 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:20.983 INFO 21062 --- [prometheus-minute-schedule-1,bc41ecba2ecd18be,ec73fb677c102448] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:21.053 INFO 21062 --- [ocp-schedules-27,df78e9b80221df81,cad9695191169cd5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:53:21.360 WARN 21062 --- [ocp-async-5,e7afccf350a8dea3,ed43c8aad0322806] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:21.395 INFO 21062 --- [ocp-async-1,0e1aeb9757160de1,ad1c768688d4affd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:23.361 WARN 21062 --- [ocp-async-7,c18bcf451fda21b0,28a3de23a64a9706] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:23.395 INFO 21062 --- [ocp-async-9,e2dcc1f8aa729979,7275d6214256fc27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:23.397 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:23.398 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8d560503c0ff860d,3004b0f4e2b4fef6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:25.361 WARN 21062 --- [ocp-async-10,7096065880b988b3,1a13efd8b6d4f1a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:25.395 INFO 21062 --- [ocp-async-8,b679fd248c62791d,dd879e74998cd5bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:26.398 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:26.399 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:26.983 INFO 21062 --- [prometheus-minute-schedule-1,b0247ee2ba1b8d1f,d22cb27d75682ebc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9d842c4fd5c7491b,99b4bd86333abc9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:53:27.361 WARN 21062 --- [ocp-async-4,6959593f0252ece6,1aa830f49eecb534] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:27.395 INFO 21062 --- [ocp-async-6,b156904e07804425,686c702dff76a0b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:28.498 INFO 21062 --- [ocp-schedules-1,9a5545a43605ba07,11cf770901a37a53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:53:28.499 INFO 21062 --- [ocp-schedules-6,b9c7366f8e9fffa2,8fc878b8b27a3a66] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:53:28.500 INFO 21062 --- [ocp-schedules-23,13944c57416bf67c,fe152355d09a0634] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:53:28.500 INFO 21062 --- [ocp-schedules-26,e145b3b45ddde2aa,87dfa818c9e2afff] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:53:28.502 INFO 21062 --- [ocp-schedules-1,9a5545a43605ba07,11cf770901a37a53] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:53:28.503 INFO 21062 --- [ocp-schedules-9,f19b95a89c66e7de,67a807f9d6b7ddbd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:53:28.504 INFO 21062 --- [ocp-schedules-23,13944c57416bf67c,fe152355d09a0634] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:28.504 INFO 21062 --- [ocp-schedules-26,e145b3b45ddde2aa,87dfa818c9e2afff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:53:28.505 INFO 21062 --- [ocp-schedules-3,ce46aa1b533b9a8f,210a33b0526fddee] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:53:28.505 INFO 21062 --- [ocp-schedules-6,b9c7366f8e9fffa2,8fc878b8b27a3a66] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:53:28.510 INFO 21062 --- [ocp-schedules-15,c77e0aa000aa0eea,04d0cc8eb5c458fe] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:53:28.510 INFO 21062 --- [ocp-schedules-17,70ba67425a1374e7,8c76ee92d1c3fc87] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:53:28.510 INFO 21062 --- [ocp-schedules-15,c77e0aa000aa0eea,04d0cc8eb5c458fe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:53:28.510 INFO 21062 --- [ocp-schedules-17,70ba67425a1374e7,8c76ee92d1c3fc87] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:53:28.510 INFO 21062 --- [ocp-schedules-24,c979d2e312e94acd,9832cb5bf6e9339f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:53:28.510 INFO 21062 --- [ocp-schedules-24,c979d2e312e94acd,9832cb5bf6e9339f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:53:28.587 INFO 21062 --- [ocp-schedules-16,ebbacba1955dad3a,949724eb1553335a] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 14:53:28.587 INFO 21062 --- [ocp-schedules-16,ebbacba1955dad3a,949724eb1553335a] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 14:53:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:53:28.588719+08:00, lastProcessTime=2024-11-04T14:53:18.591140+08:00 2024-11-04 14:53:28.691 INFO 21062 --- [ocp-async-3,bf938180aff2e951,1b41a2a7c4586885] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:53:28.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:53:28.588719+08:00 2024-11-04 14:53:28.908 INFO 21062 --- [ocp-schedules-5,513f429fea63da56,39ae1707b9676883] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:53:29.174 INFO 21062 --- [ocp-schedules-6,0af852fa565eb6d2,39e78244fa7773d1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:53:29.179 INFO 21062 --- [ocp-schedules-6,0af852fa565eb6d2,39e78244fa7773d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:29.179 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:53:29.231 INFO 21062 --- [ocp-schedules-17,ac5651540ed90b38,078839b053e4bf7a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:53:29.231 INFO 21062 --- [ocp-schedules-17,ac5651540ed90b38,078839b053e4bf7a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:53:29.334 INFO 21062 --- [ocp-schedules-24,b03a2ee621e8c3ab,3af1b9d9e24d4029] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:53:29.361 WARN 21062 --- [ocp-async-9,dc8735a5b6f41645,64e279ffe5867f3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:29.363 INFO 21062 --- [ocp-schedules-2,7f8dc2f4a2731c44,d7a27b96e337f94c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:53:29.368 INFO 21062 --- [ocp-schedules-2,7f8dc2f4a2731c44,d7a27b96e337f94c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:53:29.368 INFO 21062 --- [ocp-schedules-2,7f8dc2f4a2731c44,d7a27b96e337f94c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:53:29.395 INFO 21062 --- [ocp-async-3,1e754992dc44d3eb,99be95de010e8d4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:29.399 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:29.400 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:29.642 INFO 21062 --- [ocp-schedules-7,eaf9d7cc913c140e,16cb48cabdda655b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:53:29.772 INFO 21062 --- [ocp-schedules-22,2d65ce1aa1b2aebd,546c1a87a18cd7da] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:53:29.772 INFO 21062 --- [ocp-schedules-22,2d65ce1aa1b2aebd,546c1a87a18cd7da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:53:29.775 INFO 21062 --- [ocp-schedules-22,2d65ce1aa1b2aebd,546c1a87a18cd7da] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:53:29.775 INFO 21062 --- [ocp-schedules-22,2d65ce1aa1b2aebd,546c1a87a18cd7da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:53:29.775 INFO 21062 --- [ocp-schedules-22,2d65ce1aa1b2aebd,546c1a87a18cd7da] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:53:29.775 INFO 21062 --- [ocp-schedules-22,2d65ce1aa1b2aebd,546c1a87a18cd7da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:53:29.983 INFO 21062 --- [prometheus-minute-schedule-1,a343be098f3774e9,5332612c59d1383e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:53:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:53:31.056 INFO 21062 --- [ocp-schedules-8,5f6e0ecf1c500c42,11b61bfda04cac7a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:53:31.361 WARN 21062 --- [ocp-async-3,94c5efb94bf021d2,a2ac28a3f84da716] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:31.396 INFO 21062 --- [ocp-async-5,6e9900d3fb9441e6,f112106e3cc3b785] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:32.239 INFO 21062 --- [ocp-schedules-2,89605c10dccd1320,4f2ae71602ca5265] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:53:32.245 INFO 21062 --- [ocp-schedules-2,89605c10dccd1320,4f2ae71602ca5265] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:53:32.245 INFO 21062 --- [ocp-schedules-2,89605c10dccd1320,4f2ae71602ca5265] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:53:32.385 INFO 21062 --- [ocp-schedules-1,8b516014149171fc,3d40e5a0111c779e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:53:32.385 INFO 21062 --- [ocp-schedules-1,8b516014149171fc,3d40e5a0111c779e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:53:32.400 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:32.401 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:32.412 INFO 21062 --- [ocp-schedules-1,8b516014149171fc,3d40e5a0111c779e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:53:32.412 INFO 21062 --- [ocp-schedules-1,8b516014149171fc,3d40e5a0111c779e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:53:32.413 INFO 21062 --- [ocp-schedules-18,6e0a7d224de5d170,c5a8785549cdb4a4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:53:32.608 INFO 21062 --- [ocp-schedules-29,cea3452f527683ce,fc0a03f2133422a3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:53:32.608+08:00, clusters=[1] 2024-11-04 14:53:32.610 INFO 21062 --- [ocp-schedules-29,cea3452f527683ce,fc0a03f2133422a3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:53:32.830 INFO 21062 --- [ocp-schedules-24,b8d5efc2859e913f,431b3b02712aa4a3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:53:32.830+08:00, clusters=[1] 2024-11-04 14:53:32.833 INFO 21062 --- [ocp-schedules-24,b8d5efc2859e913f,431b3b02712aa4a3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:53:32.983 INFO 21062 --- [prometheus-minute-schedule-1,dc4e5055deb8a10e,5bdc5895900adb32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:33.361 WARN 21062 --- [ocp-async-5,03b3e1b801644035,04d52795b9c00383] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:33.396 INFO 21062 --- [ocp-async-2,b985b242e5d217f7,c05bc0d309efdb9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:34.227 INFO 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:53:34.227 INFO 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:53:34.227 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:53:34.227 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:53:34.236 INFO 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:53:34.237 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:53:34.237 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:53:34.238 WARN 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:53:34.238 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:53:34.239 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:53:34.239 INFO 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:53:34.239 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:53:34.239 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:53:34.240 WARN 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:53:34.240 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:53:34.240 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:53:34.247 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:53:34.247 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:53:34.248 INFO 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:53:34.263 ERROR 21062 --- [ocp-schedules-17,7719964041179e0c,94a36afb7e9ed825] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:53:35.362 WARN 21062 --- [ocp-async-2,23b171cf75c9820f,14ba251b046bbee6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:35.396 INFO 21062 --- [ocp-async-10,6caf14d1b3ce12c2,873d15b6fadb89f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:35.401 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:35.402 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:35.983 INFO 21062 --- [prometheus-minute-schedule-1,072dd452bd58117b,8a34d4359038243c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:37.362 WARN 21062 --- [ocp-async-10,7459f3702d4f9065,bc665ee30c54090e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:37.396 INFO 21062 --- [ocp-async-4,1e57b279905f9043,4024c638ff078cd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:38.194 INFO 21062 --- [ocp-schedules-12,7dc727b9d860912b,9d0f04d4bdeb8cbd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:53:38.194 INFO 21062 --- [ocp-schedules-12,0a18622613b254cf,a7f2b6b3e2b824ad] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:53:38.199 INFO 21062 --- [ocp-schedules-12,0a18622613b254cf,a7f2b6b3e2b824ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:38.228 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:53:38.228 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:53:38.402 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:38.402 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:38.504 INFO 21062 --- [ocp-schedules-26,e145b3b45ddde2aa,87dfa818c9e2afff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:53:38.504 INFO 21062 --- [ocp-schedules-23,13944c57416bf67c,fe152355d09a0634] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:53:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:53:38.589452+08:00, lastProcessTime=2024-11-04T14:53:28.588719+08:00 2024-11-04 14:53:38.691 INFO 21062 --- [ocp-async-9,5d6a606e0f4d33a6,8f826b3382d36a08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:53:38.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:53:38.589452+08:00 2024-11-04 14:53:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c4ce7b88954fd08c,7e66299087f3ee58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:39.180 INFO 21062 --- [ocp-schedules-6,0af852fa565eb6d2,39e78244fa7773d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:53:39.362 WARN 21062 --- [ocp-async-1,2dce1fb36711d035,9356ca9c2bb6a16e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:39.396 INFO 21062 --- [ocp-async-9,2efe7332ead27d49,40125bf85b8a247b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:39.668 INFO 21062 --- [ocp-schedules-16,a11493c9c0af27c1,b1b524945fa49de6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:53:40.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:53:40.004 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:53:41.059 INFO 21062 --- [ocp-schedules-11,93a9e1322770d92c,419e236578f43fd1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:53:41.362 WARN 21062 --- [ocp-async-9,56ea4e4612040163,8eb8ca2a1ea896ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:41.397 INFO 21062 --- [ocp-async-3,5945e97175aca8e6,6c232b9125599907] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:41.403 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:41.403 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:41.556 INFO 21062 --- [ocp-schedules-29,77f1d08678570027,68ac80befb39c571] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:53:41.560 INFO 21062 --- [ocp-schedules-29,77f1d08678570027,68ac80befb39c571] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:41.648 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:53:41.657 ERROR 21062 --- [ocp-schedules-29,77f1d08678570027,68ac80befb39c571] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:53:41.657 INFO 21062 --- [ocp-schedules-29,77f1d08678570027,68ac80befb39c571] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:53:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d2d1c0c08992eb11,3d9e248e7bca8189] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:43.362 WARN 21062 --- [ocp-async-3,d70af693b968306c,e5105ddcdfd9862a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:43.397 INFO 21062 --- [ocp-async-4,cfedfaf946155fbb,6383953e1a324730] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:44.403 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:44.406 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:44.983 INFO 21062 --- [prometheus-minute-schedule-1,df289ba040ae224c,a82945037096be56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:45.362 WARN 21062 --- [ocp-async-5,b2e5df3e452ae9c5,0286b88eac63298c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:45.397 INFO 21062 --- [ocp-async-6,019edc7ca6d5c579,b38ae4354cd1852d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:47.363 WARN 21062 --- [ocp-async-2,72890ab84dbc0509,bd1e502b2abe0def] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:47.397 INFO 21062 --- [ocp-async-7,15f27f22ed52c26e,2990a0b3aae2e08c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:47.407 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:47.407 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:47.983 INFO 21062 --- [prometheus-minute-schedule-1,1b21274a013b3a60,f5bbf4f47413ee6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:48.594 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:53:48.594329+08:00, lastProcessTime=2024-11-04T14:53:38.589452+08:00 2024-11-04 14:53:48.691 INFO 21062 --- [ocp-async-4,b826e7c26e95a99c,0063971aee9c6a4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:53:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:53:48.594329+08:00 2024-11-04 14:53:49.363 WARN 21062 --- [ocp-async-3,75f1653291ec86ba,ff3ab22d1511b95d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:49.397 INFO 21062 --- [ocp-async-4,9018542ca73e6dc8,70bea80f27db6fee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:49.711 INFO 21062 --- [ocp-schedules-30,6335500835a465df,1de5d8ce07ada6f9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:53:50.407 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:50.408 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b88827efec491b54,e2cb0645f100d417] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:51.062 INFO 21062 --- [ocp-schedules-15,9fe0c8c519061b71,8f916977860363f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:53:51.363 WARN 21062 --- [ocp-async-5,8c6acd0798e00f45,f0985d8dc461f5d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:51.398 INFO 21062 --- [ocp-async-6,acf7403fef502776,bbfb30bf2b90cedd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:52.248 INFO 21062 --- [ocp-schedules-5,e9987ec734d91c5d,0f3bd1c136e8ed50] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:53:52.253 INFO 21062 --- [ocp-schedules-5,e9987ec734d91c5d,0f3bd1c136e8ed50] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:53:52.253 INFO 21062 --- [ocp-schedules-5,e9987ec734d91c5d,0f3bd1c136e8ed50] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:53:53.363 WARN 21062 --- [ocp-async-2,03bb4f4691af43ac,8a1a897ddbebcfa2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:53.398 INFO 21062 --- [ocp-async-7,3281978380d65e70,a6c344b870889f36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:53.408 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:53.409 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:53.983 INFO 21062 --- [prometheus-minute-schedule-1,777468cd7cfa6fd4,e4ed8ac51c5ded0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:55.363 WARN 21062 --- [ocp-async-10,b399f25d560f3d2d,0891de0793c9dc2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:55.398 INFO 21062 --- [ocp-async-8,cfdcc3db1c38c702,8101b1f1f2276557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:56.409 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:56.410 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:56.983 INFO 21062 --- [prometheus-minute-schedule-1,26820cc840458775,60038c39aedadf94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:53:57.363 WARN 21062 --- [ocp-async-4,297ba78da9311ae0,ab58eb3221175a43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:57.398 INFO 21062 --- [ocp-async-1,b11d565b962a6179,0cd85c98aed03d4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:58.500 INFO 21062 --- [ocp-schedules-7,2682566d61184dda,62286fa9e09bbf0d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:53:58.500 INFO 21062 --- [ocp-schedules-19,712e1678543c6659,5df26ca4c1446a5d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:53:58.503 INFO 21062 --- [ocp-schedules-26,039d56ac31d7607e,36fb6ddd759301e4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:53:58.503 INFO 21062 --- [ocp-schedules-7,2682566d61184dda,62286fa9e09bbf0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:53:58.504 INFO 21062 --- [ocp-schedules-19,712e1678543c6659,5df26ca4c1446a5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:53:58.504 INFO 21062 --- [ocp-schedules-23,61064e2255a81767,f8adffbd758906e3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:53:58.511 INFO 21062 --- [ocp-schedules-5,e6d60e175c11bc02,d23af8af1feaec92] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:53:58.511 INFO 21062 --- [ocp-schedules-29,361c9635eefd8863,ee523e287c15f153] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:53:58.511 INFO 21062 --- [ocp-schedules-5,e6d60e175c11bc02,d23af8af1feaec92] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:53:58.511 INFO 21062 --- [ocp-schedules-29,361c9635eefd8863,ee523e287c15f153] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:53:58.511 INFO 21062 --- [ocp-schedules-3,99f71c51b7bd5e38,8451c14c7595a4c2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:53:58.511 INFO 21062 --- [ocp-schedules-3,99f71c51b7bd5e38,8451c14c7595a4c2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:53:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:53:58.590135+08:00, lastProcessTime=2024-11-04T14:53:48.594329+08:00 2024-11-04 14:53:58.692 INFO 21062 --- [ocp-async-7,bfdff2e300f0ba4c,6ac94476092ed2fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:53:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:53:58.590135+08:00 2024-11-04 14:53:59.337 INFO 21062 --- [ocp-schedules-13,1e58c71c3518e933,49dc73df946207ed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:53:59.364 WARN 21062 --- [ocp-async-2,6a703705c1b32daa,7dcad13e279947da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:53:59.398 INFO 21062 --- [ocp-async-7,5c84020cb2138011,b498c92138ad3883] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:53:59.410 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:53:59.411 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:53:59.739 INFO 21062 --- [ocp-schedules-3,c29da0f0eeded50c,5f405d676b718460] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:53:59.983 INFO 21062 --- [prometheus-minute-schedule-1,bb62738fa8d4ffc1,c0ffcca32395c8a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703240, interval=60 2024-11-04 14:54:01.064 INFO 21062 --- [ocp-schedules-27,7ecd9c27335469a4,2c78a462f6a2f8bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:54:01.364 WARN 21062 --- [ocp-async-10,9ab4e3710117d5c5,9608eb04e5b47c1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:01.399 INFO 21062 --- [ocp-async-8,c62d7cb237fb4f82,8d89d83d0a9010d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:02.411 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:02.412 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4cffd901adf69d5d,f5fa5f43a9d1d9bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:03.364 WARN 21062 --- [ocp-async-4,eaf61b1c1498c893,6b1f3bdd959e944e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:03.399 INFO 21062 --- [ocp-async-1,bb89d1a89e3bd3fa,f8a99df464344134] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:05.364 WARN 21062 --- [ocp-async-6,a56dd2e5c2033d7e,277cf94f0da12219] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:05.399 INFO 21062 --- [ocp-async-9,cc7a07815a370c41,a971030b543ae50e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:05.412 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:05.413 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:05.983 INFO 21062 --- [prometheus-minute-schedule-1,9f563a6b0e95025a,0c68ea02b175fda9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:07.364 WARN 21062 --- [ocp-async-7,31fc282da93f283d,5ed169cb87baf14e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:07.399 INFO 21062 --- [ocp-async-3,8fd8df80d0284faf,4bb100a02c4b35f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:08.229 INFO 21062 --- [ocp-schedules-12,0a18622613b254cf,a7f2b6b3e2b824ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:54:08.229 INFO 21062 --- [ocp-schedules-12,fb33601cbfcbdad8,ea51e45721841110] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:54:08.234 INFO 21062 --- [ocp-schedules-12,fb33601cbfcbdad8,ea51e45721841110] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:08.263 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:54:08.263 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:54:08.413 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:08.413 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:08.504 INFO 21062 --- [ocp-schedules-7,2682566d61184dda,62286fa9e09bbf0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:54:08.505 INFO 21062 --- [ocp-schedules-19,712e1678543c6659,5df26ca4c1446a5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:54:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:54:08.590782+08:00, lastProcessTime=2024-11-04T14:53:58.590135+08:00 2024-11-04 14:54:08.692 INFO 21062 --- [ocp-async-1,687ce711debe6d87,0c759a50eaff7545] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:54:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:54:08.590782+08:00 2024-11-04 14:54:08.983 INFO 21062 --- [prometheus-minute-schedule-1,69c9d6e0270e4246,6b184d20a6a09e0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:09.364 WARN 21062 --- [ocp-async-4,a0c6aa87ba49df1f,944abf0704e2075f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:09.399 INFO 21062 --- [ocp-async-1,0d4a651cebbebd58,19e82d5c3605c7b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:09.785 INFO 21062 --- [ocp-schedules-30,2f10b3cb9466119f,736ab88c473c3d6e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:54:10.002 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:54:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:54:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:54:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:54:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:54:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:54:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:54:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:54:11.067 INFO 21062 --- [ocp-schedules-1,60c6658a32fb99d2,52789dab6578175b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:54:11.364 WARN 21062 --- [ocp-async-6,ccf1f4d9cef19744,8c8299107e01cc4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:11.399 INFO 21062 --- [ocp-async-9,8e926af0a7f3aed1,e51f8b8fcd4dbbe9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:11.414 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:11.414 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:11.690 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=1, affectRows=1 2024-11-04 14:54:11.690 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=1, validCount=1 2024-11-04 14:54:11.690 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=7, affectRows=7 2024-11-04 14:54:11.690 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=7, validCount=7 2024-11-04 14:54:11.690 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=7, affectRows=7 2024-11-04 14:54:11.690 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=16, affectRows=16 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=6, validCount=6 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=16, validCount=16 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=11, affectRows=11 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=21, affectRows=21 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=11, validCount=11 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=34, affectRows=34 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=21, validCount=21 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=34, validCount=34 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=14, affectRows=14 2024-11-04 14:54:11.691 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=14, validCount=14 2024-11-04 14:54:11.692 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=50, affectRows=50 2024-11-04 14:54:11.692 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=50, validCount=50 2024-11-04 14:54:11.692 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=51, affectRows=51 2024-11-04 14:54:11.692 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=64, affectRows=64 2024-11-04 14:54:11.692 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=71, affectRows=71 2024-11-04 14:54:11.692 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=51, validCount=51 2024-11-04 14:54:11.693 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703180, nodeCount=75, affectRows=75 2024-11-04 14:54:11.693 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=64, validCount=64 2024-11-04 14:54:11.693 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=70, validCount=70 2024-11-04 14:54:11.693 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703180, nodeCount=74, validCount=74 2024-11-04 14:54:11.983 INFO 21062 --- [prometheus-minute-schedule-1,477aaaf4e1bfb239,31cc3c0b3258c0c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:12.255 INFO 21062 --- [ocp-schedules-4,ffc4102561e1d6b5,47927e63122127b6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:54:12.260 INFO 21062 --- [ocp-schedules-4,ffc4102561e1d6b5,47927e63122127b6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:54:12.260 INFO 21062 --- [ocp-schedules-4,ffc4102561e1d6b5,47927e63122127b6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:54:13.365 WARN 21062 --- [ocp-async-9,fde9ae4aed976769,e2d2d5664f39d83d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:13.400 INFO 21062 --- [ocp-async-3,0844774d58add1a0,834793ea38081776] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:13.984 INFO 21062 --- [prometheus-cache-monitor1,5329979aa6fcd463,d3f26b94190825f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:54:13.984 INFO 21062 --- [prometheus-cache-monitor1,5329979aa6fcd463,d3f26b94190825f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:54:14.414 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:14.415 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d896a75dcb6c6b7c,6ea8191b3ef98c9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:15.365 WARN 21062 --- [ocp-async-3,3cac945a1fc1ea80,021c8e24f2badb06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:15.400 INFO 21062 --- [ocp-async-5,556597c211230c45,4b7c2bf62192d7de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:17.365 WARN 21062 --- [ocp-async-5,f6212e9b32b9f363,b18b569fcf6da6e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:17.400 INFO 21062 --- [ocp-async-2,249f61fb1e5726f7,f856fd3502c77464] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:17.415 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:17.416 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:17.983 INFO 21062 --- [prometheus-minute-schedule-1,35e9cdc6bc0049be,386f6a0ba902b4bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:54:18.591191+08:00, lastProcessTime=2024-11-04T14:54:08.590782+08:00 2024-11-04 14:54:18.692 INFO 21062 --- [ocp-async-3,fcd14b5c18d7b329,fe066fb323be53a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:54:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:54:18.591191+08:00 2024-11-04 14:54:19.365 WARN 21062 --- [ocp-async-9,89360796427424e4,2fa6f720492f9f07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:19.400 INFO 21062 --- [ocp-async-3,d1c313132b4d3296,fabcf445d409065f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:19.816 INFO 21062 --- [ocp-schedules-24,9fd7173c259b2017,9636d350b4071df0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:54:20.416 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:20.416 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4eea38edf303239f,d835a478b6bfe4cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:21.070 INFO 21062 --- [ocp-schedules-28,15e07e1dc9cf9046,8278a9a0d066e450] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:54:21.365 WARN 21062 --- [ocp-async-3,e0002a82fb865703,459aa5c66b97e26c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:21.400 INFO 21062 --- [ocp-async-5,39278db9947ec494,2bfc5be36478176e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:23.365 WARN 21062 --- [ocp-async-5,e7d62a021280aa7a,75733150a32ac487] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:23.400 INFO 21062 --- [ocp-async-2,93d8314837a8d7f4,758ac63fae9b4aa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:23.417 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:23.417 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:23.983 INFO 21062 --- [prometheus-minute-schedule-1,0be65b5b818645ab,1092d35d5d3fac60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:25.366 WARN 21062 --- [ocp-async-2,8878e36ac7f9b09e,a773d560ce876b3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:25.400 INFO 21062 --- [ocp-async-7,15fc52d8f293a152,63f9ce9076cf4f29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:26.417 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:26.418 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:26.983 INFO 21062 --- [prometheus-minute-schedule-1,b2fba9923e601bd3,876e128999ae013e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:26.984 INFO 21062 --- [prometheus-hour-schedule-1,1a50c2aac7947646,4080701e1b24fc5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:54:27.366 WARN 21062 --- [ocp-async-7,5a1d46efc7bd8f36,2a49f13a4715c52b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:27.401 INFO 21062 --- [ocp-async-8,162d9eac11223cf7,dfb283508d84b9ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:28.498 INFO 21062 --- [ocp-schedules-14,4a69c7dfad126839,4ed6c43088ba96b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:54:28.499 INFO 21062 --- [ocp-schedules-22,5c3acab08fb0d40c,7d85a268701b09dc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:54:28.500 INFO 21062 --- [ocp-schedules-15,6f987f9c07aa11e8,37de75a42b819c1c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:54:28.500 INFO 21062 --- [ocp-schedules-9,7b7a963c88833580,af31edb543aab1d9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:54:28.502 INFO 21062 --- [ocp-schedules-14,4a69c7dfad126839,4ed6c43088ba96b8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:54:28.502 INFO 21062 --- [ocp-schedules-23,741194e9dda2a572,0539c564d9e9b985] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:54:28.504 INFO 21062 --- [ocp-schedules-15,6f987f9c07aa11e8,37de75a42b819c1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:28.504 INFO 21062 --- [ocp-schedules-11,0d1f43f086437942,68623f9f584f9fef] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:54:28.504 INFO 21062 --- [ocp-schedules-9,7b7a963c88833580,af31edb543aab1d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:54:28.505 INFO 21062 --- [ocp-schedules-22,5c3acab08fb0d40c,7d85a268701b09dc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:54:28.510 INFO 21062 --- [ocp-schedules-4,004501a5f888235f,69743b1f99533d17] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:54:28.510 INFO 21062 --- [ocp-schedules-3,3ee5ad6333def72b,fbe857cf33fe2486] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:54:28.510 INFO 21062 --- [ocp-schedules-4,004501a5f888235f,69743b1f99533d17] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:54:28.510 INFO 21062 --- [ocp-schedules-3,3ee5ad6333def72b,fbe857cf33fe2486] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:54:28.510 INFO 21062 --- [ocp-schedules-1,2c33ff1237f0cfd1,334397627a4b19d6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:54:28.510 INFO 21062 --- [ocp-schedules-1,2c33ff1237f0cfd1,334397627a4b19d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:54:28.582 INFO 21062 --- [ocp-schedules-29,112bf8d4174dc4b2,e8e7c569e3c186ed] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:54:28.582 INFO 21062 --- [ocp-schedules-29,112bf8d4174dc4b2,e8e7c569e3c186ed] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:54:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:54:28.588852+08:00, lastProcessTime=2024-11-04T14:54:18.591191+08:00 2024-11-04 14:54:28.605 INFO 21062 --- [ocp-schedules-2,119e1e72d8c76399,0ce2075e3b5a1eb1] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:54:28.605+08:00, tenantSize=0 2024-11-04 14:54:28.647 INFO 21062 --- [ocp-schedules-16,48bdbcf85d4f0a7a,1cc82d6e67bd469e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:54:28.647 INFO 21062 --- [ocp-schedules-16,48bdbcf85d4f0a7a,1cc82d6e67bd469e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:54:28.692 INFO 21062 --- [ocp-async-9,67ac62b181024e92,532689cc20c2daf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:54:28.692 INFO 21062 --- [ocp-schedules-28,42ea43679cffa9ac,f8ca0b289da65ef2] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 14:54:28.692 INFO 21062 --- [ocp-schedules-28,42ea43679cffa9ac,f8ca0b289da65ef2] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 14:54:28.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:54:28.588852+08:00 2024-11-04 14:54:28.885 INFO 21062 --- [ocp-schedules-7,cd3431a516e843ae,ecb83f3f00484b02] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:54:28.885 INFO 21062 --- [ocp-schedules-7,cd3431a516e843ae,ecb83f3f00484b02] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:54:28.910 INFO 21062 --- [ocp-schedules-23,e9c1aab7a33e5c37,a4d2c03ade292f26] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:54:29.235 INFO 21062 --- [ocp-schedules-1,5cfe5d712a8c5e8d,096ecbdd2c998025] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:54:29.235 INFO 21062 --- [ocp-schedules-1,5cfe5d712a8c5e8d,096ecbdd2c998025] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:54:29.339 INFO 21062 --- [ocp-schedules-20,740d2bd27ca7cb37,e06efbe7e1c57a96] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:54:29.366 WARN 21062 --- [ocp-async-1,727db6599b26d1e4,9b5b7853442fdba2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:29.368 INFO 21062 --- [ocp-schedules-27,889ac0eb5f2ca872,9b59acecb9556a8d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:54:29.374 INFO 21062 --- [ocp-schedules-27,889ac0eb5f2ca872,9b59acecb9556a8d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:54:29.374 INFO 21062 --- [ocp-schedules-27,889ac0eb5f2ca872,9b59acecb9556a8d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:54:29.401 INFO 21062 --- [ocp-async-9,1a0c643d5e27b210,cd73087a89e46e5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:29.418 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:29.419 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:29.776 INFO 21062 --- [ocp-schedules-6,34450cc7f515cc3e,b9b0e5b163c347f5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:54:29.776 INFO 21062 --- [ocp-schedules-6,34450cc7f515cc3e,b9b0e5b163c347f5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:54:29.779 INFO 21062 --- [ocp-schedules-6,34450cc7f515cc3e,b9b0e5b163c347f5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:54:29.779 INFO 21062 --- [ocp-schedules-6,34450cc7f515cc3e,b9b0e5b163c347f5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:54:29.780 INFO 21062 --- [ocp-schedules-6,34450cc7f515cc3e,b9b0e5b163c347f5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:54:29.780 INFO 21062 --- [ocp-schedules-6,34450cc7f515cc3e,b9b0e5b163c347f5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:54:29.859 INFO 21062 --- [ocp-schedules-25,5e1f11f4596533df,533c7e101c2a6f4f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:54:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b469b5365f0c5181,cb387005a9c7821c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:54:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:54:31.073 INFO 21062 --- [ocp-schedules-5,595d11b85309d287,b5c7765708089f59] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:54:31.366 WARN 21062 --- [ocp-async-9,cc6affd915044bb6,f76b12e7b088f2b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:31.401 INFO 21062 --- [ocp-async-4,10816616d4020fa2,cfd532a9824fbd47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:32.263 INFO 21062 --- [ocp-schedules-29,c77b9687fc92e060,7bb2b0eb85bf92eb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:54:32.269 INFO 21062 --- [ocp-schedules-29,c77b9687fc92e060,7bb2b0eb85bf92eb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:54:32.269 INFO 21062 --- [ocp-schedules-29,c77b9687fc92e060,7bb2b0eb85bf92eb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:54:32.412 INFO 21062 --- [ocp-schedules-25,51593f2f5c86bc32,16108f3746863f03] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:54:32.412 INFO 21062 --- [ocp-schedules-25,51593f2f5c86bc32,16108f3746863f03] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:54:32.419 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:32.420 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:32.439 INFO 21062 --- [ocp-schedules-25,51593f2f5c86bc32,16108f3746863f03] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:54:32.439 INFO 21062 --- [ocp-schedules-25,51593f2f5c86bc32,16108f3746863f03] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:54:32.439 INFO 21062 --- [ocp-schedules-22,6329165c878c545d,4cd369558738e7c6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:54:32.634 INFO 21062 --- [ocp-schedules-23,50e18b49aa5f69b3,06990c5ee6da5f79] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:54:32.634+08:00, clusters=[1] 2024-11-04 14:54:32.636 INFO 21062 --- [ocp-schedules-23,50e18b49aa5f69b3,06990c5ee6da5f79] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:54:32.860 INFO 21062 --- [ocp-schedules-16,0ff6aa00d8f8d1bb,a706342690026bc8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:54:32.860+08:00, clusters=[1] 2024-11-04 14:54:32.863 INFO 21062 --- [ocp-schedules-16,0ff6aa00d8f8d1bb,a706342690026bc8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:54:32.877 INFO 21062 --- [ocp-schedules-8,4ec7af89249db4ad,f02ec33d9534c7b2] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:54:32.881 INFO 21062 --- [ocp-schedules-8,4ec7af89249db4ad,f02ec33d9534c7b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:32.937 ERROR 21062 --- [ocp-schedules-8,4ec7af89249db4ad,f02ec33d9534c7b2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:54:32.937 INFO 21062 --- [ocp-schedules-8,4ec7af89249db4ad,f02ec33d9534c7b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:54:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a5eabf8c07d7d7c5,99039196180754b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:33.367 WARN 21062 --- [ocp-async-4,2990597a2ed8eaf5,ab3344aad843f721] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:33.401 INFO 21062 --- [ocp-async-5,7dae957ea0355201,d24be1afee35fd76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:34.273 INFO 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:54:34.273 INFO 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:54:34.273 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:54:34.273 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:54:34.282 INFO 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:54:34.282 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:54:34.283 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:54:34.283 WARN 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:54:34.284 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:54:34.284 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:54:34.284 INFO 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:54:34.285 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:54:34.285 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:54:34.285 WARN 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:54:34.286 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:54:34.286 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:54:34.293 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:54:34.293 INFO 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:54:34.293 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:54:34.304 ERROR 21062 --- [ocp-schedules-24,f6fb42e25252576f,9716c052edea8415] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:54:35.367 WARN 21062 --- [ocp-async-5,e9e674dca9d0e25b,627772ae67b10d7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:35.401 INFO 21062 --- [ocp-async-2,6a94552a008ab74d,85730ffd146fe024] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:35.420 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:35.420 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:35.983 INFO 21062 --- [prometheus-minute-schedule-1,35c29b052b49a299,14751beed212f5f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:37.367 WARN 21062 --- [ocp-async-2,afc7a705858692fb,a14831ce7df5f579] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:37.402 INFO 21062 --- [ocp-async-7,7ea0020ffa0fd952,ce7b1957b252f9b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:38.264 INFO 21062 --- [ocp-schedules-12,fb33601cbfcbdad8,ea51e45721841110] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:54:38.264 INFO 21062 --- [ocp-schedules-12,dfc56cc7d9b78344,7de79aa6531c6f33] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:54:38.269 INFO 21062 --- [ocp-schedules-12,dfc56cc7d9b78344,7de79aa6531c6f33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:38.298 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:54:38.298 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:54:38.420 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:38.421 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:38.505 INFO 21062 --- [ocp-schedules-9,7b7a963c88833580,af31edb543aab1d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:54:38.505 INFO 21062 --- [ocp-schedules-15,6f987f9c07aa11e8,37de75a42b819c1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:54:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:54:38.589495+08:00, lastProcessTime=2024-11-04T14:54:28.588852+08:00 2024-11-04 14:54:38.692 INFO 21062 --- [ocp-async-5,5ec1d1835907afd1,e110fb8f01d0e20d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:54:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:54:38.589495+08:00 2024-11-04 14:54:38.983 INFO 21062 --- [prometheus-minute-schedule-1,690f6e2cb76a0291,9d6d64ce7e3c7896] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:39.180 INFO 21062 --- [ocp-schedules-3,262917b391b77bab,1df7e51abbf99998] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:54:39.184 INFO 21062 --- [ocp-schedules-3,262917b391b77bab,1df7e51abbf99998] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:39.184 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:54:39.367 WARN 21062 --- [ocp-async-8,67d78f8bb1c9c403,6d50d6ec9ba27365] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:39.402 INFO 21062 --- [ocp-async-5,1c03fd61aa41be11,be0876ced0a2edce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:39.882 INFO 21062 --- [ocp-schedules-14,d3324da966c79a82,ab574da30cdd5e84] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:54:40.002 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:54:40.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:54:41.076 INFO 21062 --- [ocp-schedules-11,f378c9781d91074b,29853728ffbed358] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:54:41.367 WARN 21062 --- [ocp-async-5,90137617f5df421d,d1f84f748bc2272a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:41.402 INFO 21062 --- [ocp-async-2,c716eb11091b45ea,a35b4594b66dee7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:41.421 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:41.422 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:41.658 INFO 21062 --- [ocp-schedules-21,efeb09da4dbf8d2c,29861260b90e10f8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:54:41.662 INFO 21062 --- [ocp-schedules-21,efeb09da4dbf8d2c,29861260b90e10f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:41.746 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:54:41.748 ERROR 21062 --- [ocp-schedules-21,efeb09da4dbf8d2c,29861260b90e10f8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:54:41.748 INFO 21062 --- [ocp-schedules-21,efeb09da4dbf8d2c,29861260b90e10f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:54:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:54:41.823 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:54:41.823 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:54:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d79a611190ddc106,0a772b64b144181c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:43.368 WARN 21062 --- [ocp-async-9,e2c95f636a68d425,4468e34fc959bf70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:43.402 INFO 21062 --- [ocp-async-7,e460dc4da0877978,4903ea0fe8034a7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:43.705 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 14:54:43.705 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 14:54:43.707 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.707 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:54:43.708 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.709 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.709 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.710 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 14:54:43.725 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.726 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.727 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.727 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 14:54:43.740 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.741 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.741 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.742 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 14:54:43.756 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.756 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.757 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.757 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 14:54:43.772 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.772 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.773 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.773 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 14:54:43.779 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.779 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.780 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.780 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 14:54:43.785 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.785 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.785 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.786 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 14:54:43.791 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.791 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.792 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.792 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 14:54:43.798 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.799 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.799 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.799 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 14:54:43.805 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.805 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.806 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.806 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 14:54:43.812 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.812 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.813 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.813 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 14:54:43.822 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.822 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.823 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.824 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 14:54:43.837 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.837 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.838 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.838 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 14:54:43.843 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.844 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.844 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.845 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 14:54:43.852 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.852 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.852 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.853 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 14:54:43.859 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.859 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.860 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.860 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 14:54:43.866 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.866 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.867 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.867 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 14:54:43.871 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.872 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.872 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.873 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 14:54:43.877 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.877 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.878 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.878 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 14:54:43.883 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.884 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.884 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.884 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 14:54:43.888 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.888 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.889 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.889 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 14:54:43.893 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.894 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.894 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.895 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 14:54:43.899 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.899 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.900 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.900 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 14:54:43.905 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.906 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.906 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.907 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 14:54:43.911 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.911 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.912 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.912 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 14:54:43.916 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.917 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.917 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.918 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 14:54:43.922 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.922 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.923 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.923 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 14:54:43.927 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.928 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.929 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.929 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 14:54:43.933 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.933 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.934 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.934 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 14:54:43.939 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.939 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.939 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.940 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 14:54:43.944 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.944 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.945 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.945 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 14:54:43.949 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.949 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.950 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.950 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 14:54:43.955 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.955 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.956 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.956 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 14:54:43.960 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.960 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.961 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.961 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 14:54:43.968 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.968 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.969 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.969 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 14:54:43.974 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.974 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.975 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.975 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 14:54:43.980 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.981 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.981 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.982 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 14:54:43.987 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.987 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.988 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.988 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 14:54:43.992 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.992 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.993 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.993 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 14:54:43.996 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.997 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:43.997 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:43.998 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 14:54:44.001 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.002 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.002 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.002 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 14:54:44.007 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.008 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.008 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.009 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 14:54:44.017 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.018 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.018 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.019 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 14:54:44.039 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.039 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.040 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.040 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 14:54:44.046 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.047 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.047 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.047 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 14:54:44.053 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.054 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.054 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.054 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 14:54:44.060 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.061 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.061 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.062 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 14:54:44.065 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.066 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.066 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.067 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 14:54:44.070 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.071 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.071 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.072 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 14:54:44.075 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 14:54:44.075 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 14:54:44.077 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.077 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:54:44.079 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.079 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.080 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.080 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 14:54:44.086 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.086 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 14:54:44.087 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:54:44.087 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 14:54:44.092 INFO 21062 --- [ocp-schedules-22,d2adbc85a93206b6,2d71f0eaab6eb497] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 14:54:44.422 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:44.423 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:44.983 INFO 21062 --- [prometheus-minute-schedule-1,4281f0bda281a92a,4f25b56d20dce58d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:45.368 WARN 21062 --- [ocp-async-4,371b67005dd5e30d,2142ddf4cd60ad2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:45.402 INFO 21062 --- [ocp-async-6,7dd13c8d2b6e7c3c,5de520cf7bc278fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:47.368 WARN 21062 --- [ocp-async-1,4ac4e215491d475a,909d1d2f9f6308e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:47.403 INFO 21062 --- [ocp-async-10,d51a6bf0d5d801ff,9e52502953ae141e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:47.423 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:47.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:47.983 INFO 21062 --- [prometheus-minute-schedule-1,9986a8f9c648189d,9d6b271e82713f21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:54:48.591195+08:00, lastProcessTime=2024-11-04T14:54:38.589495+08:00 2024-11-04 14:54:48.693 INFO 21062 --- [ocp-async-7,917818b64e1852f1,d51b58a767abad97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:54:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:54:48.591195+08:00 2024-11-04 14:54:49.185 INFO 21062 --- [ocp-schedules-3,262917b391b77bab,1df7e51abbf99998] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:54:49.368 WARN 21062 --- [ocp-async-9,0604062b7ba35db6,23e3432fc2db8d69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:49.403 INFO 21062 --- [ocp-async-7,72b6e2d0652af6d3,64a7acc7b7815c59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:49.923 INFO 21062 --- [ocp-schedules-18,fedb2ae12679cb49,b89fd0d9881c126b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:54:50.424 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:50.425 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4b56939e2d96b3ed,c6190abb55325259] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:51.079 INFO 21062 --- [ocp-schedules-24,a5bc64e8deb53935,d6de72fcb34739cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:54:51.368 WARN 21062 --- [ocp-async-4,cbe9ffeeac939244,be2519acf45aafd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:51.403 INFO 21062 --- [ocp-async-6,3397cbe3d049a32b,7a7da8af5e87777e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:52.271 INFO 21062 --- [ocp-schedules-21,e06e85f243ceea62,c8c94e9980472352] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:54:52.277 INFO 21062 --- [ocp-schedules-21,e06e85f243ceea62,c8c94e9980472352] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:54:52.277 INFO 21062 --- [ocp-schedules-21,e06e85f243ceea62,c8c94e9980472352] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:54:53.368 WARN 21062 --- [ocp-async-1,0332c039c4454ba7,f81247633d8e4366] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:53.403 INFO 21062 --- [ocp-async-10,95258e571ec9f236,c1fea16b31bcf428] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:53.425 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:53.425 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:53.983 INFO 21062 --- [prometheus-minute-schedule-1,5485f353cc875560,c8476c665ea9443b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:55.369 WARN 21062 --- [ocp-async-2,7242486a53bc8321,c27b901004645bbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:55.403 INFO 21062 --- [ocp-async-3,b8923d8372ddebdc,cc76b8456861932f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:56.426 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:56.426 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e92d09d9b5ad11fa,4667d7ad5bb72cca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:54:57.369 WARN 21062 --- [ocp-async-7,ae4760ae773e534e,fcec0cbf774ab2df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:57.403 INFO 21062 --- [ocp-async-8,4493025ca815ed42,160271338ce46e88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:58.500 INFO 21062 --- [ocp-schedules-11,7bcf63c9ce407679,a64d2fc842751bcc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:54:58.500 INFO 21062 --- [ocp-schedules-28,42508be106840b0c,aaf4e234089d0e4f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:54:58.502 INFO 21062 --- [ocp-schedules-29,30c019a34af13b28,a4217fdfc6a06073] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:54:58.503 INFO 21062 --- [ocp-schedules-11,7bcf63c9ce407679,a64d2fc842751bcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:54:58.504 INFO 21062 --- [ocp-schedules-28,42508be106840b0c,aaf4e234089d0e4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:54:58.504 INFO 21062 --- [ocp-schedules-8,c60d625233ec73c1,67cfeb5f82c6d32a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:54:58.510 INFO 21062 --- [ocp-schedules-13,1afb2688838268fe,b64a8081e9ba990c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:54:58.510 INFO 21062 --- [ocp-schedules-13,1afb2688838268fe,b64a8081e9ba990c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:54:58.510 INFO 21062 --- [ocp-schedules-26,564b8655b8051206,1079e430edaa52e6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:54:58.510 INFO 21062 --- [ocp-schedules-3,139b4f5f33f600be,8bbe8e92a565fc28] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:54:58.510 INFO 21062 --- [ocp-schedules-26,564b8655b8051206,1079e430edaa52e6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:54:58.510 INFO 21062 --- [ocp-schedules-3,139b4f5f33f600be,8bbe8e92a565fc28] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:54:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:54:58.589925+08:00, lastProcessTime=2024-11-04T14:54:48.591195+08:00 2024-11-04 14:54:58.693 INFO 21062 --- [ocp-async-10,d1c2ceac8513e8ec,6ab3b4ff7e5f9f02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:54:58.796 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:54:58.589925+08:00 2024-11-04 14:54:59.342 INFO 21062 --- [ocp-schedules-1,257cee43a3be9817,e9336d396a68e852] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:54:59.369 WARN 21062 --- [ocp-async-1,55ae76a633cd3b61,1232395a2b78621a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:54:59.404 INFO 21062 --- [ocp-async-10,3afe3888f9e261f8,5901f49548908629] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:54:59.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:54:59.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:54:59.949 INFO 21062 --- [ocp-schedules-19,aac93dfc0ba6b128,95e53b962b3d6f9a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:54:59.983 INFO 21062 --- [prometheus-minute-schedule-1,c55f1d32df3e4fca,8daf19d0671c6c3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703300, interval=60 2024-11-04 14:55:00.952 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d067bb897a49f0b0] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:50:01.0, nextRun=2024-11-04 14:55:00.0, currentDate=Mon Nov 04 14:55:00 CST 2024 2024-11-04 14:55:00.952 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d067bb897a49f0b0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:50:01.0, nextRunTime=2024-11-04 14:55:00.0} 2024-11-04 14:55:00.980 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d067bb897a49f0b0] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 14:55:00.984 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d067bb897a49f0b0] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 14:55:00.984 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d067bb897a49f0b0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 14:55:01.082 INFO 21062 --- [ocp-schedules-30,54f30c724bc96062,8006e85580e28a66] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:55:01.369 WARN 21062 --- [ocp-async-2,3e376851871b3304,d1fa7a3f2c72ab1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:01.404 INFO 21062 --- [ocp-async-3,c99b4b53a83f0dfa,ce4778fe2a51d3fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:02.427 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:02.428 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:02.983 INFO 21062 --- [prometheus-minute-schedule-1,76746de15b39ee3d,cbf61877159775fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:03.083 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:55:03.083 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:55:03.091 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:55:03.091 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:55:03.091 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:55:03.091 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:55:03.096 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:55:03.096 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:55:03.096 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:55:03.096 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:55:03.133 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:55:03.142 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:55:03.142 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:55:03.142 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:55:03.142 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:55:03.142 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:55:03.151 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:55:03.369 WARN 21062 --- [ocp-async-7,a1f721a72922bd34,93faa4c07fae9bc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:03.404 INFO 21062 --- [ocp-async-8,281aae8540317c8a,4740dde011d08a7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:03.680 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:55:04.238 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.142005901S 2024-11-04 14:55:04.238 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:55:04.339 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:55:04.339 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:55:04.366 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:55:04.428 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:55:04.428 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:55:04.429 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:55:04.429 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:55:04.429 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:55:04.429 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:55:04.439 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:55:04.613 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] o.s.boot.SpringApplication : Started application in 1.522 seconds (JVM running for 8718.292) 2024-11-04 14:55:04.613 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.522845274S 2024-11-04 14:55:05.113 INFO 21062 --- [ocp-schedules-1,a6832b75baf4dbbb,c417ea34a4dce3f1] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:55:05.369 WARN 21062 --- [ocp-async-6,5d9b6f64d2cbf000,80f28e839cab9fbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:05.404 INFO 21062 --- [ocp-async-5,10a478d211bb836e,f75a9d7398b713f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:05.429 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:05.429 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:05.983 INFO 21062 --- [prometheus-minute-schedule-1,db55d48f79acf3e8,c97370d0432e1793] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:07.370 WARN 21062 --- [ocp-async-10,1b399202ed7a21bb,aa2b2a03a5b820ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:07.404 INFO 21062 --- [ocp-async-9,76a5d82846bbeb32,6731e8a5652c8e25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:08.298 INFO 21062 --- [ocp-schedules-12,dfc56cc7d9b78344,7de79aa6531c6f33] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:55:08.299 INFO 21062 --- [ocp-schedules-12,edf57ce049350812,cdcd613b2a124a29] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:55:08.303 INFO 21062 --- [ocp-schedules-12,edf57ce049350812,cdcd613b2a124a29] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:08.335 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:55:08.335 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:55:08.430 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:08.430 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:08.504 INFO 21062 --- [ocp-schedules-11,7bcf63c9ce407679,a64d2fc842751bcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:55:08.504 INFO 21062 --- [ocp-schedules-28,42508be106840b0c,aaf4e234089d0e4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:55:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:55:08.588897+08:00, lastProcessTime=2024-11-04T14:54:58.589925+08:00 2024-11-04 14:55:08.693 INFO 21062 --- [ocp-async-8,72cb081f021ea7ae,8865490f7997d571] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:55:08.784 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:55:08.588897+08:00 2024-11-04 14:55:08.983 INFO 21062 --- [prometheus-minute-schedule-1,dfc786995e832312,afcec8c8d65fd192] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:09.370 WARN 21062 --- [ocp-async-7,3c6188d901214ec9,314093ca3ea92791] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:09.405 INFO 21062 --- [ocp-async-8,303e3fbc0f978ec2,77d1b8882baf04b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:09.993 INFO 21062 --- [ocp-schedules-30,41bc2b60a86a72b8,b309813cca9b7eb9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:55:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:55:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:55:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:55:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:55:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:55:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:55:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:55:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:55:11.085 INFO 21062 --- [ocp-schedules-5,15c914f38466fafe,8bb1c0a1a966b053] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:55:11.370 WARN 21062 --- [ocp-async-8,fb67e6b851e4891e,ce9490979004bbc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:11.405 INFO 21062 --- [ocp-async-5,7d20e5d6b0ce7daf,7d71e60446c6c79f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:11.431 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:11.431 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:11.694 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703240, nodeCount=72, affectRows=72 2024-11-04 14:55:11.694 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703240, nodeCount=97, affectRows=97 2024-11-04 14:55:11.694 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703240, nodeCount=72, validCount=72 2024-11-04 14:55:11.695 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703240, nodeCount=88, affectRows=88 2024-11-04 14:55:11.695 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703240, nodeCount=71, affectRows=71 2024-11-04 14:55:11.695 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703240, nodeCount=96, validCount=96 2024-11-04 14:55:11.695 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703240, nodeCount=71, validCount=71 2024-11-04 14:55:11.695 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703240, nodeCount=94, affectRows=94 2024-11-04 14:55:11.695 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703240, nodeCount=88, validCount=88 2024-11-04 14:55:11.696 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703240, nodeCount=92, validCount=92 2024-11-04 14:55:11.983 INFO 21062 --- [prometheus-minute-schedule-1,10b00babfffa424c,cbcaf06bc8a9f757] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:12.279 INFO 21062 --- [ocp-schedules-3,b87519f8a8378a55,42501d6e091f67dd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:55:12.285 INFO 21062 --- [ocp-schedules-3,b87519f8a8378a55,42501d6e091f67dd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:55:12.285 INFO 21062 --- [ocp-schedules-3,b87519f8a8378a55,42501d6e091f67dd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:55:13.370 WARN 21062 --- [ocp-async-5,45e27a905670d30f,3743a98341849f6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:13.405 INFO 21062 --- [ocp-async-9,ae4cb12de070773f,8cef074e375c5c0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:13.984 INFO 21062 --- [prometheus-cache-monitor1,8344200a96ae8bc3,214764a9367a33ce] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:55:13.985 INFO 21062 --- [prometheus-cache-monitor1,8344200a96ae8bc3,214764a9367a33ce] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:55:14.431 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:14.432 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:14.983 INFO 21062 --- [prometheus-minute-schedule-1,e0cf2ba8181e1259,cbca12889b53710a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:15.329 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:51:15.311, end=2024-11-04T14:53:15.327} 2024-11-04 14:55:15.338 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:55:15.338 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:55:15.370 WARN 21062 --- [ocp-async-9,913ddd0e826f27e3,809ca7930bf47cec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:15.405 INFO 21062 --- [ocp-async-4,5051bca37fa0522a,e7ee9a29bdfb3f5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:17.371 WARN 21062 --- [ocp-async-4,304ef7ff1ae7bb70,d1ff4cce3bca7bd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:17.405 INFO 21062 --- [ocp-async-1,a0e566dc9353bd61,915987f3f48908cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:17.432 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:17.433 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:17.983 INFO 21062 --- [prometheus-minute-schedule-1,12c1ed2672a83e0e,4be456d2c19a1730] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:55:18.591327+08:00, lastProcessTime=2024-11-04T14:55:08.588897+08:00 2024-11-04 14:55:18.693 INFO 21062 --- [ocp-async-9,95e98f5658622c5d,9a9d9e318313c75a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:55:18.748 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:55:18.591327+08:00 2024-11-04 14:55:19.371 WARN 21062 --- [ocp-async-5,5de5263ca58f74cb,64abf203fde062f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:19.405 INFO 21062 --- [ocp-async-9,959f058928e3b3b7,a2535f069989876e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:20.019 INFO 21062 --- [ocp-schedules-24,e17518edf8a38cb8,eca609612e315376] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:55:20.434 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:20.434 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:20.983 INFO 21062 --- [prometheus-minute-schedule-1,aa37d278a7d49b33,1256f657bd028269] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:21.088 INFO 21062 --- [ocp-schedules-23,90510bb6494bd520,b1e759f6f604840d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:55:21.371 WARN 21062 --- [ocp-async-9,c193df12beff4a15,927d375f8e30a549] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:21.406 INFO 21062 --- [ocp-async-4,e0b1c59ada0f21ed,cb2d30a97d17345c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:23.371 WARN 21062 --- [ocp-async-4,a56a6ccc75d5540d,875b2d21dfae92cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:23.406 INFO 21062 --- [ocp-async-1,075de9de983bc79d,cb8add13146d128d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:23.435 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:23.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a94bcd7c269aafbb,40a8b5487b826723] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:25.371 WARN 21062 --- [ocp-async-1,ba805c095ba9a88d,484dd2436b8ae6af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:25.406 INFO 21062 --- [ocp-async-10,3bd377c4e6e475a3,30bc27d9de4e41d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:26.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:26.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:26.983 INFO 21062 --- [prometheus-minute-schedule-1,7ba1f46e65ebfcc5,c7d7ab7445303d1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8916a3053e20b247,f8f1e02d54be5caf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:55:27.371 WARN 21062 --- [ocp-async-10,df0481800938f797,040b99dd2923337e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:27.406 INFO 21062 --- [ocp-async-7,82b6df70513ca86d,cca061dda5fa5e9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:28.498 INFO 21062 --- [ocp-schedules-9,63e4264010307914,79693437bb4a15bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:55:28.499 INFO 21062 --- [ocp-schedules-28,8ebc93f61afd5155,e743b6413ee503fa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:55:28.500 INFO 21062 --- [ocp-schedules-18,f32a7318853b6568,22cecafd00b6d567] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:55:28.500 INFO 21062 --- [ocp-schedules-3,911abf234d426b77,ffbae8040b68b951] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:55:28.502 INFO 21062 --- [ocp-schedules-9,63e4264010307914,79693437bb4a15bc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:55:28.502 INFO 21062 --- [ocp-schedules-15,6599691229409984,ac3088b2558ff639] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:55:28.503 INFO 21062 --- [ocp-schedules-28,8ebc93f61afd5155,e743b6413ee503fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:28.504 INFO 21062 --- [ocp-schedules-3,911abf234d426b77,ffbae8040b68b951] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:55:28.504 INFO 21062 --- [ocp-schedules-11,94513611e61f1c8e,8b3aae5cf8b20674] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:55:28.505 INFO 21062 --- [ocp-schedules-18,f32a7318853b6568,22cecafd00b6d567] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:55:28.511 INFO 21062 --- [ocp-schedules-29,3879d47e4b7ceae8,4a25477eb04ca709] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:55:28.511 INFO 21062 --- [ocp-schedules-6,3e82196b8a901cb0,708cb08a2c1668dc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:55:28.511 INFO 21062 --- [ocp-schedules-6,3e82196b8a901cb0,708cb08a2c1668dc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:55:28.511 INFO 21062 --- [ocp-schedules-29,3879d47e4b7ceae8,4a25477eb04ca709] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:55:28.511 INFO 21062 --- [ocp-schedules-20,71b16ec11b20dff2,ec156ab6bc30c2d6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:55:28.511 INFO 21062 --- [ocp-schedules-20,71b16ec11b20dff2,ec156ab6bc30c2d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:55:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:55:28.589296+08:00, lastProcessTime=2024-11-04T14:55:18.591327+08:00 2024-11-04 14:55:28.693 INFO 21062 --- [ocp-async-5,a8f9edbd3b638490,e13ab15597692ce0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:55:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:55:28.589296+08:00 2024-11-04 14:55:28.913 INFO 21062 --- [ocp-schedules-14,24a69690e34ac48c,9abb6c5eecfbc53c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:55:29.239 INFO 21062 --- [ocp-schedules-15,a4530f954cef16d8,9515f93fe309b0ea] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:55:29.239 INFO 21062 --- [ocp-schedules-15,a4530f954cef16d8,9515f93fe309b0ea] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:55:29.345 INFO 21062 --- [ocp-schedules-29,59073051dc9baeb9,2807c6e27ee5ed17] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:55:29.372 WARN 21062 --- [ocp-async-4,f327f98a08620322,f44a4552957ad60e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:29.374 INFO 21062 --- [ocp-schedules-23,d7951a6aeed06f86,66db4e90ae42c469] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:55:29.379 INFO 21062 --- [ocp-schedules-23,d7951a6aeed06f86,66db4e90ae42c469] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:55:29.379 INFO 21062 --- [ocp-schedules-23,d7951a6aeed06f86,66db4e90ae42c469] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:55:29.406 INFO 21062 --- [ocp-async-5,5a7a329426927bde,53c4c5f10fd2a766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:29.437 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:29.438 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:29.597 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:55:29.597 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:55:29.605 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:55:29.605 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:55:29.605 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:55:29.616 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:55:29.616 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:55:29.616 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:55:29.619 INFO 21062 --- [ocp-schedules-7,b2354884ec0e2b5e,71df05d62350095d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:55:29.783 INFO 21062 --- [ocp-schedules-24,0bff8efc272a1834,25547bd60e43c0da] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:55:29.784 INFO 21062 --- [ocp-schedules-24,0bff8efc272a1834,25547bd60e43c0da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:55:29.789 INFO 21062 --- [ocp-schedules-24,0bff8efc272a1834,25547bd60e43c0da] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:55:29.789 INFO 21062 --- [ocp-schedules-24,0bff8efc272a1834,25547bd60e43c0da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:55:29.793 INFO 21062 --- [ocp-schedules-24,0bff8efc272a1834,25547bd60e43c0da] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:55:29.793 INFO 21062 --- [ocp-schedules-24,0bff8efc272a1834,25547bd60e43c0da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:55:29.983 INFO 21062 --- [prometheus-minute-schedule-1,c05faa42c9b6e4cd,06d08f961bf2c35a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:30.044 INFO 21062 --- [ocp-schedules-5,a9a527f15bf2808a,a1e945e3da496936] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:55:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:55:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:55:31.091 INFO 21062 --- [ocp-schedules-18,e05f335f72c4d58a,cf969cbb967471f3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:55:31.372 WARN 21062 --- [ocp-async-5,f27dfeb4c3205405,f594edd69a1f7924] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:31.407 INFO 21062 --- [ocp-async-3,cee06989c028d2ca,f0ebcb2f53679f50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:32.288 INFO 21062 --- [ocp-schedules-23,cdd53242a42c45d6,345dfaf3c1e457ec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:55:32.293 INFO 21062 --- [ocp-schedules-23,cdd53242a42c45d6,345dfaf3c1e457ec] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:55:32.293 INFO 21062 --- [ocp-schedules-23,cdd53242a42c45d6,345dfaf3c1e457ec] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:55:32.438 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:32.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:32.439 INFO 21062 --- [ocp-schedules-21,0e0363ef423e43ac,8a34f71a025ea9ad] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:55:32.439 INFO 21062 --- [ocp-schedules-21,0e0363ef423e43ac,8a34f71a025ea9ad] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:55:32.467 INFO 21062 --- [ocp-schedules-21,0e0363ef423e43ac,8a34f71a025ea9ad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:55:32.467 INFO 21062 --- [ocp-schedules-21,0e0363ef423e43ac,8a34f71a025ea9ad] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:55:32.469 INFO 21062 --- [ocp-schedules-26,9256999d08a6916f,f21fa03d285e9b67] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:55:32.664 INFO 21062 --- [ocp-schedules-25,1f8653bc7fa011e4,f358aa1f873fcf43] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:55:32.663+08:00, clusters=[1] 2024-11-04 14:55:32.665 INFO 21062 --- [ocp-schedules-25,1f8653bc7fa011e4,f358aa1f873fcf43] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:55:32.889 INFO 21062 --- [ocp-schedules-2,a23aae08b5a905e0,181973d9289eccd7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:55:32.889+08:00, clusters=[1] 2024-11-04 14:55:32.893 INFO 21062 --- [ocp-schedules-2,a23aae08b5a905e0,181973d9289eccd7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:55:32.983 INFO 21062 --- [prometheus-minute-schedule-1,20e50566825defc9,cf022034eda7ac82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:33.372 WARN 21062 --- [ocp-async-3,c18e27675d338617,147409697826bd63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:33.407 INFO 21062 --- [ocp-async-6,5fe2da98996de5ee,b3f9fac5a73f5d85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:34.314 INFO 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:55:34.314 INFO 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:55:34.314 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:55:34.314 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:55:34.324 INFO 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:55:34.324 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:55:34.324 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:55:34.326 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:55:34.326 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:55:34.326 WARN 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:55:34.326 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:55:34.327 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:55:34.327 INFO 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:55:34.328 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:55:34.328 WARN 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:55:34.328 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:55:34.335 INFO 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:55:34.335 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:55:34.335 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:55:34.349 ERROR 21062 --- [ocp-schedules-9,432e7cc8ac5a15b5,af91096e4f327901] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:55:35.372 WARN 21062 --- [ocp-async-6,3055bb98e99bed41,35ae26c648209ea2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:35.407 INFO 21062 --- [ocp-async-1,c2498609435fb6d8,a74b297b812e09ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:35.439 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:35.441 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0eaf90216ccec2d5,940f67b1e95fe148] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:37.372 WARN 21062 --- [ocp-async-1,160635f8f1a81b08,d1f406b44f55b007] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:37.407 INFO 21062 --- [ocp-async-10,0a0b3c3aa05c1a62,e04989c1c6c95a21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:38.335 INFO 21062 --- [ocp-schedules-12,edf57ce049350812,cdcd613b2a124a29] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:55:38.336 INFO 21062 --- [ocp-schedules-12,c7df8756254f4fed,08b94b3197bfc24e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:55:38.340 INFO 21062 --- [ocp-schedules-12,c7df8756254f4fed,08b94b3197bfc24e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:38.369 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:55:38.369 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:55:38.441 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:38.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:38.504 INFO 21062 --- [ocp-schedules-28,8ebc93f61afd5155,e743b6413ee503fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:55:38.507 INFO 21062 --- [ocp-schedules-3,911abf234d426b77,ffbae8040b68b951] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:55:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:55:38.589731+08:00, lastProcessTime=2024-11-04T14:55:28.589296+08:00 2024-11-04 14:55:38.693 INFO 21062 --- [ocp-async-6,d28b1364f02cb394,a0ed074a5a0b4106] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:55:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:55:38.589731+08:00 2024-11-04 14:55:38.983 INFO 21062 --- [prometheus-minute-schedule-1,74d09a700c50bbac,f75f3c4bd271388a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:39.372 WARN 21062 --- [ocp-async-7,1bf2e192270aba72,ece0d638650419fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:39.408 INFO 21062 --- [ocp-async-6,eebc2c952e60b730,a65008eff33fd93a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:55:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:55:40.070 INFO 21062 --- [ocp-schedules-11,477cc1bc0e0eef30,f1e15a64411cd11a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:55:41.094 INFO 21062 --- [ocp-schedules-21,e278d30ab447af63,c8ef0888a3fb7d2d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:55:41.372 WARN 21062 --- [ocp-async-6,487dda4a8641c8ee,e11cce9725007894] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:41.408 INFO 21062 --- [ocp-async-1,519aa27bcc9c8505,c0bf5600d57cbdba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:41.442 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:41.443 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:41.748 INFO 21062 --- [ocp-schedules-28,3e35f01854fbf060,e93059c9336ef1ad] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:55:41.752 INFO 21062 --- [ocp-schedules-28,3e35f01854fbf060,e93059c9336ef1ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:41.837 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:55:41.839 ERROR 21062 --- [ocp-schedules-28,3e35f01854fbf060,e93059c9336ef1ad] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:55:41.839 INFO 21062 --- [ocp-schedules-28,3e35f01854fbf060,e93059c9336ef1ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:55:41.983 INFO 21062 --- [prometheus-minute-schedule-1,23cfe19c691e1040,9372fb1d1e6f6a46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:43.373 WARN 21062 --- [ocp-async-5,d269833b99f2987a,9bea8c3d02ce88c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:43.408 INFO 21062 --- [ocp-async-10,9ea6e4417fe49e86,3e5fe5cf8bdbe672] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:44.443 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:44.444 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3e36ff155cd90d2f,04dbceb31df6ca7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:45.373 WARN 21062 --- [ocp-async-3,abe8d48e96872006,4e947da84ebb0c73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:45.408 INFO 21062 --- [ocp-async-8,e5765c8bc3940965,d1dd5f64fe01d435] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:47.373 WARN 21062 --- [ocp-async-4,ff77b7382595d43c,a08fce645030f465] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:47.408 INFO 21062 --- [ocp-async-2,30f52a434a8e7a55,7275b477854bfcb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:47.444 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:47.445 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:47.983 INFO 21062 --- [prometheus-minute-schedule-1,219630a9975caaba,d43b9633b924ca3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:55:48.590566+08:00, lastProcessTime=2024-11-04T14:55:38.589731+08:00 2024-11-04 14:55:48.694 INFO 21062 --- [ocp-async-10,159d45035e72adf6,4b9ebf8d99bff98d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:55:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:55:48.590566+08:00 2024-11-04 14:55:49.185 INFO 21062 --- [ocp-schedules-8,d1250d6b5dea20d5,0e091944228dceb4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:55:49.188 INFO 21062 --- [ocp-schedules-8,d1250d6b5dea20d5,0e091944228dceb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:49.189 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:55:49.373 WARN 21062 --- [ocp-async-5,32992576bafb381e,9c056b2e0b14c971] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:49.409 INFO 21062 --- [ocp-async-10,e4495fa1e09fd5fb,c0d12d5d1920f2ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:50.109 INFO 21062 --- [ocp-schedules-13,4a3822af7c9b6e3f,b0bdbe1b7f3c71a6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:55:50.445 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:50.446 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:50.983 INFO 21062 --- [prometheus-minute-schedule-1,22a191dae0e5a555,4b387bc144a4f934] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:51.099 INFO 21062 --- [ocp-schedules-15,c1f5127946440d40,0282afcb2cc71d0e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:55:51.373 WARN 21062 --- [ocp-async-3,07f39372ed353c68,b05c3763db94653d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:51.409 INFO 21062 --- [ocp-async-8,c5d2736e6e8defc1,3d3c537ed97e79f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:52.295 INFO 21062 --- [ocp-schedules-10,e8f37709e25f2687,b2ca8b0530a12deb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:55:52.300 INFO 21062 --- [ocp-schedules-10,e8f37709e25f2687,b2ca8b0530a12deb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:55:52.300 INFO 21062 --- [ocp-schedules-10,e8f37709e25f2687,b2ca8b0530a12deb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:55:53.373 WARN 21062 --- [ocp-async-4,f3604915bdc40d08,8bb8896bf07b19d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:53.409 INFO 21062 --- [ocp-async-2,407029b9625d0f91,d57c494c0212709c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:53.446 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:53.446 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9158509533ba4cb5,5c6d0c24d1f626d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:55.374 WARN 21062 --- [ocp-async-1,1bd820f4e127e440,e8c0cf0c5478945d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:55.409 INFO 21062 --- [ocp-async-9,27122de413f6a26a,52bb26aa1ca5dbb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:56.447 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:56.447 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7e2ab1b7de6ef8be,d93749ab6f20a5e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:55:57.374 WARN 21062 --- [ocp-async-10,4f4d2012cfd557e3,f39f35dd864da2ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:57.409 INFO 21062 --- [ocp-async-7,9d2b539d87cb28bb,c90d772469ea73af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:58.500 INFO 21062 --- [ocp-schedules-3,7cefab83ca11cd53,da4dba941dd5c2d6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:55:58.500 INFO 21062 --- [ocp-schedules-15,1403a10be1629789,5f2160581742965e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:55:58.502 INFO 21062 --- [ocp-schedules-10,72febf55db009732,aa32f14b97e65a23] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:55:58.503 INFO 21062 --- [ocp-schedules-3,7cefab83ca11cd53,da4dba941dd5c2d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:58.503 INFO 21062 --- [ocp-schedules-15,1403a10be1629789,5f2160581742965e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:55:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:55:58.504 INFO 21062 --- [ocp-schedules-22,e1671e1761127ece,45c5e4deea24f493] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:55:58.509 INFO 21062 --- [ocp-schedules-9,06ffe820fe4b98b5,48dc318624b693e8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:55:58.509 INFO 21062 --- [ocp-schedules-9,06ffe820fe4b98b5,48dc318624b693e8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:55:58.509 INFO 21062 --- [ocp-schedules-13,d6980b32b50e4c57,e1e799bcec180be5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:55:58.509 INFO 21062 --- [ocp-schedules-1,656f79ce5450f0a0,d1a213fed98f87c2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:55:58.509 INFO 21062 --- [ocp-schedules-1,656f79ce5450f0a0,d1a213fed98f87c2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:55:58.509 INFO 21062 --- [ocp-schedules-13,d6980b32b50e4c57,e1e799bcec180be5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:55:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:55:58.590028+08:00, lastProcessTime=2024-11-04T14:55:48.590566+08:00 2024-11-04 14:55:58.694 INFO 21062 --- [ocp-async-2,cd6bacfc3532d378,062722ba8ee4e377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:55:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:55:58.590028+08:00 2024-11-04 14:55:59.189 INFO 21062 --- [ocp-schedules-8,d1250d6b5dea20d5,0e091944228dceb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:55:59.348 INFO 21062 --- [ocp-schedules-16,410bf84c376ced44,e76bfa7184f703c0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:55:59.374 WARN 21062 --- [ocp-async-4,c641db10b52a7d21,50b9ae28a89b34fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:55:59.409 INFO 21062 --- [ocp-async-2,062ebe26ebd86c78,e832b3b7158727a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:55:59.448 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:55:59.448 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:55:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2e6bc9a867becc96,2fbea0dab8b803fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703360, interval=60 2024-11-04 14:56:00.138 INFO 21062 --- [ocp-schedules-6,7e58cf580307e6ec,fcdb56cf0999d40c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:56:01.102 INFO 21062 --- [ocp-schedules-24,d1d95007330f53e7,85da93b458529c56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:56:01.374 WARN 21062 --- [ocp-async-1,16b6f6a4aeee09b0,2f7c63be8badd6dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:01.410 INFO 21062 --- [ocp-async-9,fc63c8ba92a4e05d,5fb4ebf29d59bb0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:02.449 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:02.450 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:02.983 INFO 21062 --- [prometheus-minute-schedule-1,aeb6b052d8fbba50,4df4cd0ea723574c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:03.374 WARN 21062 --- [ocp-async-10,63a82addc71997f0,3ca2b4e0f6516be1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:03.410 INFO 21062 --- [ocp-async-7,fb73e9cb831247c2,f9505c88ead1a71d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:05.374 WARN 21062 --- [ocp-async-8,a782d99e2b966e67,724e75336d311e3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:05.410 INFO 21062 --- [ocp-async-6,aaab70aed9a09d37,076503259ebde98c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:05.450 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:05.451 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:05.983 INFO 21062 --- [prometheus-minute-schedule-1,f8b8af19cec5117e,bc4f44869c8f9139] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:07.375 WARN 21062 --- [ocp-async-2,10e557292c6c51d8,cfe16c7fe49d1ade] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:07.410 INFO 21062 --- [ocp-async-5,0f50b27d84779425,13aaffabec9d6567] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:08.370 INFO 21062 --- [ocp-schedules-12,c7df8756254f4fed,08b94b3197bfc24e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:56:08.370 INFO 21062 --- [ocp-schedules-12,eb51722a6864b3a4,a59a36e9021968ea] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:56:08.375 INFO 21062 --- [ocp-schedules-12,eb51722a6864b3a4,a59a36e9021968ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:08.405 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:56:08.405 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:56:08.453 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:08.454 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:08.504 INFO 21062 --- [ocp-schedules-15,1403a10be1629789,5f2160581742965e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:56:08.504 INFO 21062 --- [ocp-schedules-3,7cefab83ca11cd53,da4dba941dd5c2d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:56:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:56:08.589262+08:00, lastProcessTime=2024-11-04T14:55:58.590028+08:00 2024-11-04 14:56:08.694 INFO 21062 --- [ocp-async-7,987135a4f3ab07b7,bc61587229b74642] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:56:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:56:08.589262+08:00 2024-11-04 14:56:08.983 INFO 21062 --- [prometheus-minute-schedule-1,34b14b19768d62e2,a22f3bf81621a193] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:09.375 WARN 21062 --- [ocp-async-9,4aaf6de339f199ce,6b82bf494588e67d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:09.410 INFO 21062 --- [ocp-async-7,76c06575ee5537b8,327e9480f47befc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:56:10.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:56:10.182 INFO 21062 --- [ocp-schedules-7,66a2297feda85a6f,0310c0d3ab46b052] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:56:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:56:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:56:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:56:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:56:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:56:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:56:11.105 INFO 21062 --- [ocp-schedules-2,de35f9b8c3a109c3,5a5cc120e39eb8be] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:56:11.375 WARN 21062 --- [ocp-async-7,e9519ea6faec2870,150dea26e72d67c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:11.410 INFO 21062 --- [ocp-async-6,b5e4b3a443fec81f,8d42307d3674fa6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:11.454 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:11.454 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:11.697 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=2, affectRows=2 2024-11-04 14:56:11.697 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=1, validCount=1 2024-11-04 14:56:11.697 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=21, affectRows=21 2024-11-04 14:56:11.697 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=10, affectRows=10 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=20, affectRows=20 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=21, validCount=21 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=10, validCount=10 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=36, affectRows=36 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=20, validCount=20 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=36, validCount=36 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=61, affectRows=61 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=66, affectRows=66 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=49, affectRows=49 2024-11-04 14:56:11.698 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=61, validCount=61 2024-11-04 14:56:11.699 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=49, validCount=49 2024-11-04 14:56:11.699 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=66, validCount=66 2024-11-04 14:56:11.699 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=70, affectRows=70 2024-11-04 14:56:11.699 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703300, nodeCount=87, affectRows=87 2024-11-04 14:56:11.699 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=70, validCount=70 2024-11-04 14:56:11.699 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703300, nodeCount=85, validCount=85 2024-11-04 14:56:11.983 INFO 21062 --- [prometheus-minute-schedule-1,05a2ab4bf0ad1201,b14da4fdded929bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:12.302 INFO 21062 --- [ocp-schedules-24,653cc62304c6e848,57df8bae55158989] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:56:12.307 INFO 21062 --- [ocp-schedules-24,653cc62304c6e848,57df8bae55158989] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:56:12.307 INFO 21062 --- [ocp-schedules-24,653cc62304c6e848,57df8bae55158989] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:56:13.375 WARN 21062 --- [ocp-async-6,8846c999088d94f8,0ea757fb1217dd0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:13.411 INFO 21062 --- [ocp-async-5,02c9041a34ae5319,0c794a0ebd66b34a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:13.984 INFO 21062 --- [prometheus-cache-monitor1,2224212dd0ef4ad7,393923d2ffa41237] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:56:13.984 INFO 21062 --- [prometheus-cache-monitor1,2224212dd0ef4ad7,393923d2ffa41237] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:56:14.455 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:14.455 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5f9e325ac4d111a8,4061467a27b17afe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:15.375 WARN 21062 --- [ocp-async-5,49e0632f1cb2598b,457a0fa926992be8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:15.411 INFO 21062 --- [ocp-async-3,b88ab2161dd7173a,39627c82defa9cad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:17.376 WARN 21062 --- [ocp-async-3,3cf540ccfde72a36,e591c3c0accfa462] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:17.411 INFO 21062 --- [ocp-async-8,68e1a520d4757c6c,e4644f19112ef463] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:17.456 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:17.456 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:17.983 INFO 21062 --- [prometheus-minute-schedule-1,e54904a7aded2350,5673342dc73c5697] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:56:18.590568+08:00, lastProcessTime=2024-11-04T14:56:08.589262+08:00 2024-11-04 14:56:18.694 INFO 21062 --- [ocp-async-5,c9eb66520008a7ac,6d92d829cd9fc3d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:56:18.774 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:56:18.590568+08:00 2024-11-04 14:56:19.376 WARN 21062 --- [ocp-async-6,dacf3d2cfbf14465,af972b5e40372545] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:19.411 INFO 21062 --- [ocp-async-5,b4f6bf6debc34a5f,0011fc9e71ce2843] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:20.206 INFO 21062 --- [ocp-schedules-21,a32562e3e73558da,1c24fdf0f02d2f9c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:56:20.456 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:20.457 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a7c4a6e44cc8dbc6,6d27a5471d6b6969] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:21.108 INFO 21062 --- [ocp-schedules-23,409bb490a5624ab5,80751311d6dddaed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:56:21.376 WARN 21062 --- [ocp-async-5,baed4808d1657502,dbb1ed7fca7d4289] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:21.411 INFO 21062 --- [ocp-async-3,886c955dc711d6ef,c9aca6cfa430d446] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:23.376 WARN 21062 --- [ocp-async-3,a47b1bf0aca204aa,54aa2d9a1012bbcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:23.411 INFO 21062 --- [ocp-async-8,e3bc848e633f854d,de137155c4ae7b90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:23.457 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:23.458 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:23.983 INFO 21062 --- [prometheus-minute-schedule-1,68f8c558533d4c4d,81c2e40ca988dc80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:25.376 WARN 21062 --- [ocp-async-8,f176ca7817bd3768,87ccb807eca6db01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:25.411 INFO 21062 --- [ocp-async-2,032d59021191f384,04fb72d83b929be1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:26.458 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:26.459 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:26.983 INFO 21062 --- [prometheus-minute-schedule-1,18246a6a476c2a64,ddd4cc536284fd51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:26.984 INFO 21062 --- [prometheus-hour-schedule-1,d65f1e8c8db57ad3,56526e258d035c54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:56:27.377 WARN 21062 --- [ocp-async-2,d59db8ee1d6cbc4f,38fb35c924574d68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:27.412 INFO 21062 --- [ocp-async-9,90dc55528a05a918,c1fc63a17b505433] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:28.498 INFO 21062 --- [ocp-schedules-23,dc5eede58540c49e,e0659c755970e1a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:56:28.499 INFO 21062 --- [ocp-schedules-21,648db2f9f501f640,6462c03e42e88df4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:56:28.499 INFO 21062 --- [ocp-schedules-13,824be3149bd0e952,b7e546820810a67b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:56:28.500 INFO 21062 --- [ocp-schedules-14,49ab8b4982d29176,192f65f87702469b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:56:28.502 INFO 21062 --- [ocp-schedules-23,dc5eede58540c49e,e0659c755970e1a5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:56:28.503 INFO 21062 --- [ocp-schedules-22,e84915ffa964540e,6e2b0729d58b35f2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:56:28.503 INFO 21062 --- [ocp-schedules-13,824be3149bd0e952,b7e546820810a67b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:56:28.504 INFO 21062 --- [ocp-schedules-14,49ab8b4982d29176,192f65f87702469b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:28.504 INFO 21062 --- [ocp-schedules-1,c5213aec3b1ad2ef,19cad58a347e80e0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:56:28.505 INFO 21062 --- [ocp-schedules-21,648db2f9f501f640,6462c03e42e88df4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:56:28.509 INFO 21062 --- [ocp-schedules-29,8de99e829448220b,09774a85bc70938d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:56:28.509 INFO 21062 --- [ocp-schedules-4,f80efc25c0077ce3,5bd9c79c442120b1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:56:28.510 INFO 21062 --- [ocp-schedules-4,f80efc25c0077ce3,5bd9c79c442120b1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:56:28.510 INFO 21062 --- [ocp-schedules-29,8de99e829448220b,09774a85bc70938d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:56:28.511 INFO 21062 --- [ocp-schedules-28,3ca66d8fd53195e9,9a36d4f3d7e03b75] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:56:28.511 INFO 21062 --- [ocp-schedules-28,3ca66d8fd53195e9,9a36d4f3d7e03b75] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:56:28.582 INFO 21062 --- [ocp-schedules-20,b0e7d8844bc36280,bcd91dff364c8eb5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:56:28.582 INFO 21062 --- [ocp-schedules-20,b0e7d8844bc36280,bcd91dff364c8eb5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:56:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:56:28.589293+08:00, lastProcessTime=2024-11-04T14:56:18.590568+08:00 2024-11-04 14:56:28.694 INFO 21062 --- [ocp-async-6,a5a5fb6bbc810896,debbb57c44364a3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:56:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:56:28.589293+08:00 2024-11-04 14:56:28.915 INFO 21062 --- [ocp-schedules-10,cf295ac5da18f24e,08c699fd98dd8839] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:56:29.243 INFO 21062 --- [ocp-schedules-5,04b82196150f56a7,c7bf0e050151ce3b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:56:29.243 INFO 21062 --- [ocp-schedules-5,04b82196150f56a7,c7bf0e050151ce3b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:56:29.351 INFO 21062 --- [ocp-schedules-23,30f7275b803b6b17,16c2f111c21c7e5d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:56:29.377 WARN 21062 --- [ocp-async-4,b784ab13b94a404c,fb670249a0bbb13b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:29.380 INFO 21062 --- [ocp-schedules-3,83f417a6ed0d547d,855bcbae41f9ff27] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:56:29.385 INFO 21062 --- [ocp-schedules-3,83f417a6ed0d547d,855bcbae41f9ff27] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:56:29.385 INFO 21062 --- [ocp-schedules-3,83f417a6ed0d547d,855bcbae41f9ff27] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:56:29.412 INFO 21062 --- [ocp-async-6,77bea32961c7a454,eedda63e5de9c106] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:29.459 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:29.460 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:29.800 INFO 21062 --- [ocp-schedules-9,46a067f9b1d05b6b,e1feab4e380d64d6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:56:29.800 INFO 21062 --- [ocp-schedules-9,46a067f9b1d05b6b,e1feab4e380d64d6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:56:29.803 INFO 21062 --- [ocp-schedules-9,46a067f9b1d05b6b,e1feab4e380d64d6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:56:29.803 INFO 21062 --- [ocp-schedules-9,46a067f9b1d05b6b,e1feab4e380d64d6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:56:29.805 INFO 21062 --- [ocp-schedules-9,46a067f9b1d05b6b,e1feab4e380d64d6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:56:29.805 INFO 21062 --- [ocp-schedules-9,46a067f9b1d05b6b,e1feab4e380d64d6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:56:29.983 INFO 21062 --- [prometheus-minute-schedule-1,88b957110faabaf7,d8c5cea0b85676bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:56:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:56:30.322 INFO 21062 --- [ocp-schedules-18,883c70eda02ace5b,6811644ae656d959] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:56:31.110 INFO 21062 --- [ocp-schedules-15,2f8eb2f7583845c1,2c5df1d2c95bd2f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:56:31.377 WARN 21062 --- [ocp-async-6,6a0f118a8c1d61ef,895217760e94b38a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:31.412 INFO 21062 --- [ocp-async-5,57301fd103d630aa,bb3d159fef8eb228] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:32.310 INFO 21062 --- [ocp-schedules-18,d6999a507c7bd12c,1530ad968af6df2d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:56:32.314 INFO 21062 --- [ocp-schedules-18,d6999a507c7bd12c,1530ad968af6df2d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:56:32.314 INFO 21062 --- [ocp-schedules-18,d6999a507c7bd12c,1530ad968af6df2d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:56:32.460 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:32.460 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:32.467 INFO 21062 --- [ocp-schedules-15,45fbd1b75b4f109d,06ee42a53ff0e963] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:56:32.467 INFO 21062 --- [ocp-schedules-15,45fbd1b75b4f109d,06ee42a53ff0e963] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:56:32.494 INFO 21062 --- [ocp-schedules-15,45fbd1b75b4f109d,06ee42a53ff0e963] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:56:32.494 INFO 21062 --- [ocp-schedules-15,45fbd1b75b4f109d,06ee42a53ff0e963] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:56:32.495 INFO 21062 --- [ocp-schedules-27,ff134456f02e9ae2,55f3101e7aeff9f1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:56:32.692 INFO 21062 --- [ocp-schedules-9,03ab1b63e92aadac,359a8160ef63c0c4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:56:32.692+08:00, clusters=[1] 2024-11-04 14:56:32.694 INFO 21062 --- [ocp-schedules-9,03ab1b63e92aadac,359a8160ef63c0c4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:56:32.919 INFO 21062 --- [ocp-schedules-22,0a8bda84126d4cf3,4c70a7da31aeff86] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:56:32.919+08:00, clusters=[1] 2024-11-04 14:56:32.923 INFO 21062 --- [ocp-schedules-22,0a8bda84126d4cf3,4c70a7da31aeff86] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:56:32.938 INFO 21062 --- [ocp-schedules-28,ae974358710aa291,e5f47d7cb89eeea5] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:56:32.941 INFO 21062 --- [ocp-schedules-28,ae974358710aa291,e5f47d7cb89eeea5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:32.983 INFO 21062 --- [prometheus-minute-schedule-1,df3488e431caafda,ba960abdd15eb712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:32.999 ERROR 21062 --- [ocp-schedules-28,ae974358710aa291,e5f47d7cb89eeea5] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:56:32.999 INFO 21062 --- [ocp-schedules-28,ae974358710aa291,e5f47d7cb89eeea5] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:56:33.377 WARN 21062 --- [ocp-async-5,7b8acc5417e6980d,1e015f7f81af8ba3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:33.412 INFO 21062 --- [ocp-async-7,933248b518f7407e,cb879c6d5e8d2e38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:34.359 INFO 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:56:34.360 INFO 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:56:34.360 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:56:34.360 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:56:34.369 INFO 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:56:34.370 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:56:34.370 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:56:34.371 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:56:34.371 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:56:34.371 WARN 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:56:34.372 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:56:34.372 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:56:34.372 INFO 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:56:34.373 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:56:34.373 WARN 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:56:34.373 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:56:34.380 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:56:34.380 INFO 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:56:34.380 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:56:34.391 ERROR 21062 --- [ocp-schedules-1,be842d9a1c247e16,0caab0d8ac7e5bc7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:56:35.377 WARN 21062 --- [ocp-async-7,24931cba9de3eef6,b5d9211ae5b5611d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:35.412 INFO 21062 --- [ocp-async-8,54d3e5a1ffa7b3e1,0493275ed27df2fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:35.461 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:35.461 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e3616a97beccca1e,879ee74d72665d33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:37.377 WARN 21062 --- [ocp-async-1,72364048bae37b5f,1fbeb1b434e658cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:37.413 INFO 21062 --- [ocp-async-2,0c0a91e3e9bfe408,93620c46207292f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:38.406 INFO 21062 --- [ocp-schedules-12,eb51722a6864b3a4,a59a36e9021968ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:56:38.406 INFO 21062 --- [ocp-schedules-12,1f23ae28f586a050,018fc2e259579491] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:56:38.410 INFO 21062 --- [ocp-schedules-12,1f23ae28f586a050,018fc2e259579491] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:38.438 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:56:38.439 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:56:38.461 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:38.462 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:38.504 INFO 21062 --- [ocp-schedules-13,824be3149bd0e952,b7e546820810a67b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:56:38.505 INFO 21062 --- [ocp-schedules-14,49ab8b4982d29176,192f65f87702469b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:56:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:56:38.588774+08:00, lastProcessTime=2024-11-04T14:56:28.589293+08:00 2024-11-04 14:56:38.695 INFO 21062 --- [ocp-async-7,8509e5dd7a60fc80,66cc3708eb5ee05f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:56:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:56:38.588774+08:00 2024-11-04 14:56:38.983 INFO 21062 --- [prometheus-minute-schedule-1,de5e23ce1daf5e38,005aaf99f2c46f3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:39.378 WARN 21062 --- [ocp-async-9,eb6c2edf6b5fe0bf,8cac37099bd95880] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:39.413 INFO 21062 --- [ocp-async-7,8e3cac6c6bda1339,034c47b6ad15a3b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:56:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:56:40.347 INFO 21062 --- [ocp-schedules-28,0f314e58b8dd9399,a17fe9bad5667c7b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:56:41.114 INFO 21062 --- [ocp-schedules-19,7aca695ab1508408,8cfe73a097706249] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:56:41.378 WARN 21062 --- [ocp-async-4,f4c811e512ee9197,cd8200242b4281d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:41.413 INFO 21062 --- [ocp-async-1,546118958bab5fd9,d3d9ca706da32508] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:41.463 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:41.463 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:56:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:56:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:56:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:56:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:56:41.839 INFO 21062 --- [ocp-schedules-6,ee83ffbf7d5601f8,42a4c568851b76c8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:56:41.843 INFO 21062 --- [ocp-schedules-6,ee83ffbf7d5601f8,42a4c568851b76c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:41.854 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:56:41.854 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 14:56:41.925 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:56:41.932 ERROR 21062 --- [ocp-schedules-6,ee83ffbf7d5601f8,42a4c568851b76c8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:56:41.932 INFO 21062 --- [ocp-schedules-6,ee83ffbf7d5601f8,42a4c568851b76c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:56:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6d2fb9bf0f5f858d,d4e487479000294f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:43.378 WARN 21062 --- [ocp-async-6,b6754a9951c8df5a,465037d37f6a7bed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:43.413 INFO 21062 --- [ocp-async-10,8f5d2abed3cfb1cc,d681eee9584f9a8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:44.463 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:44.464 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f98d004debb9fa5f,772e04c1d4d2d8f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:45.378 WARN 21062 --- [ocp-async-5,af8dbfe9ca131df4,e409b1d8a29dbfaf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:45.413 INFO 21062 --- [ocp-async-3,f641b29135440c14,512013a24f0e93a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:47.378 WARN 21062 --- [ocp-async-7,f6e3b3a17a426fa9,68acd851f76cd544] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:47.413 INFO 21062 --- [ocp-async-8,de748e0d83808c73,e49602c69c4fce2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:47.464 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:47.465 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:47.983 INFO 21062 --- [prometheus-minute-schedule-1,14cd2983c4d0eddf,a3eb81640eaedc6c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:56:48.590313+08:00, lastProcessTime=2024-11-04T14:56:38.588774+08:00 2024-11-04 14:56:48.695 INFO 21062 --- [ocp-async-10,5e47f1efe15900da,2d210c639b71f24b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:56:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:56:48.590313+08:00 2024-11-04 14:56:49.379 WARN 21062 --- [ocp-async-6,da0491f06a5553e1,5b5ecc99a2359463] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:49.414 INFO 21062 --- [ocp-async-10,6776bdd805d7ae5b,939bd13ea7350d1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:50.390 INFO 21062 --- [ocp-schedules-14,21d433ffd6bf4f35,5d294be49c20c6b9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:56:50.465 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:50.466 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:50.983 INFO 21062 --- [prometheus-minute-schedule-1,99bfa3031bf61116,b9dbf133832f50f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:51.117 INFO 21062 --- [ocp-schedules-27,5f753d47b70fd74e,61aa58a1379d97b6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:56:51.379 WARN 21062 --- [ocp-async-5,377289c9c4dea504,acac0d3b52e39dd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:51.414 INFO 21062 --- [ocp-async-3,14ebdc06d0ecbf83,aaa1bfd3ecd3a833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:52.318 INFO 21062 --- [ocp-schedules-22,7b29366f92f47556,08300943dc3c23a1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:56:52.324 INFO 21062 --- [ocp-schedules-22,7b29366f92f47556,08300943dc3c23a1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:56:52.324 INFO 21062 --- [ocp-schedules-22,7b29366f92f47556,08300943dc3c23a1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:56:53.379 WARN 21062 --- [ocp-async-7,ab367613ea700a9e,3ba6f05c46007cf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:53.414 INFO 21062 --- [ocp-async-8,4fa3457c8d38a3dc,fcdc41d5c2a4d318] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:53.466 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:53.467 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:53.983 INFO 21062 --- [prometheus-minute-schedule-1,79d8705e97caa49a,06a44e6c58a23e66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:55.379 WARN 21062 --- [ocp-async-1,be1d3a3b5bd45db1,bc644ec2cda60701] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:55.414 INFO 21062 --- [ocp-async-2,04f0c1b75ce0cf07,3f1dd8de12954a56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:56.467 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:56.468 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:56.983 INFO 21062 --- [prometheus-minute-schedule-1,9263cc6a14c8d7da,84ea2f90a15aba08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:56:57.379 WARN 21062 --- [ocp-async-10,68ceeaf22ef92e09,74387304bfec1d6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:57.414 INFO 21062 --- [ocp-async-9,84e833329415ef6d,5610408e3a994b5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:58.500 INFO 21062 --- [ocp-schedules-17,5bce3063793d6d13,8ba72bdb2a203415] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:56:58.500 INFO 21062 --- [ocp-schedules-20,f5da67aeb4e428e6,33d21460f5a19473] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:56:58.502 INFO 21062 --- [ocp-schedules-22,7bc813dece64d635,7a39f0d72fabd1cd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:56:58.503 INFO 21062 --- [ocp-schedules-17,5bce3063793d6d13,8ba72bdb2a203415] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:58.504 INFO 21062 --- [ocp-schedules-20,f5da67aeb4e428e6,33d21460f5a19473] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:56:58.505 INFO 21062 --- [ocp-schedules-15,5a3d172e6c13597e,b951c82b331e0c97] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:56:58.510 INFO 21062 --- [ocp-schedules-11,27adfec139791ca6,79f4c8221c0ef0b2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:56:58.510 INFO 21062 --- [ocp-schedules-11,27adfec139791ca6,79f4c8221c0ef0b2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:56:58.510 INFO 21062 --- [ocp-schedules-2,cff044cc91706135,ecb741687680b745] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:56:58.511 INFO 21062 --- [ocp-schedules-2,cff044cc91706135,ecb741687680b745] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:56:58.511 INFO 21062 --- [ocp-schedules-18,8df6d68694a74cdf,6c88b4fe2a021420] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:56:58.511 INFO 21062 --- [ocp-schedules-18,8df6d68694a74cdf,6c88b4fe2a021420] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:56:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:56:58.590769+08:00, lastProcessTime=2024-11-04T14:56:48.590313+08:00 2024-11-04 14:56:58.695 INFO 21062 --- [ocp-async-8,f08676a59900cbbe,3997aef38efff957] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:56:58.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:56:58.590769+08:00 2024-11-04 14:56:59.189 INFO 21062 --- [ocp-schedules-5,110c1e588cde0c28,0c2605eee0287d3c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:56:59.194 INFO 21062 --- [ocp-schedules-5,110c1e588cde0c28,0c2605eee0287d3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:56:59.196 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:56:59.355 INFO 21062 --- [ocp-schedules-22,7160c0225ae328ea,bd4fd25f9f42207b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:56:59.379 WARN 21062 --- [ocp-async-7,35afbf073136268b,4e54858b5d8c9294] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:56:59.414 INFO 21062 --- [ocp-async-8,d4a78c1d58a86878,b9383b197d7be1a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:56:59.468 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:56:59.469 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:56:59.983 INFO 21062 --- [prometheus-minute-schedule-1,c24b6f75398ea9be,c7ee9af9f991a588] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703420, interval=60 2024-11-04 14:57:00.414 INFO 21062 --- [ocp-schedules-10,aaafda497c0ae6dc,a9c2a9d98bd24365] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:57:01.120 INFO 21062 --- [ocp-schedules-23,3d698f7743bd4948,a57fb4a488d0d466] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:57:01.380 WARN 21062 --- [ocp-async-1,257784d80fcacd57,afb8acaa34d989c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:01.415 INFO 21062 --- [ocp-async-2,cfda3a74b3b3184b,813c557758e6b00c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:02.469 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:02.470 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:02.983 INFO 21062 --- [prometheus-minute-schedule-1,385b83da0ef9f9aa,734741cc40d473b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:03.380 WARN 21062 --- [ocp-async-10,9f36e82bd9f9dbce,df9aef61c302d232] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:03.415 INFO 21062 --- [ocp-async-9,1e45d619de1d9099,dcbd80096e34abcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:05.380 WARN 21062 --- [ocp-async-9,09e0024d01801610,1bfb02ca04ad464d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:05.415 INFO 21062 --- [ocp-async-4,8ecbc89ab8f9e67e,92d35c62416bd6d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:05.470 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:05.471 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:05.983 INFO 21062 --- [prometheus-minute-schedule-1,6aee0e532fb4f732,744c311f01873710] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:07.380 WARN 21062 --- [ocp-async-4,ea9ff7136ada917b,7c92877dce49f606] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:07.415 INFO 21062 --- [ocp-async-6,99c028865ff082aa,240c4967de693657] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:08.439 INFO 21062 --- [ocp-schedules-12,1f23ae28f586a050,018fc2e259579491] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:57:08.439 INFO 21062 --- [ocp-schedules-12,dfa5f7a34ee89d76,51a92acf92fc49ae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:57:08.444 INFO 21062 --- [ocp-schedules-12,dfa5f7a34ee89d76,51a92acf92fc49ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:08.471 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:08.472 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:08.473 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:57:08.473 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:57:08.504 INFO 21062 --- [ocp-schedules-20,f5da67aeb4e428e6,33d21460f5a19473] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:57:08.504 INFO 21062 --- [ocp-schedules-17,5bce3063793d6d13,8ba72bdb2a203415] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:57:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:57:08.590277+08:00, lastProcessTime=2024-11-04T14:56:58.590769+08:00 2024-11-04 14:57:08.695 INFO 21062 --- [ocp-async-9,adba4faad5fe0d97,d45d69cdb56bbe55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:57:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:57:08.590277+08:00 2024-11-04 14:57:08.983 INFO 21062 --- [prometheus-minute-schedule-1,09222bf825ce9c28,848f6259e952f123] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:09.197 INFO 21062 --- [ocp-schedules-5,110c1e588cde0c28,0c2605eee0287d3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:57:09.380 WARN 21062 --- [ocp-async-2,8f91731970ba519a,aee8b33383a840a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:09.416 INFO 21062 --- [ocp-async-9,a36583525cbe8010,880e4c19fb0ad8a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:57:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:57:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:57:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:57:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:57:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:57:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:57:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:57:10.458 INFO 21062 --- [ocp-schedules-17,52007c7ab89e05a8,4d513799f75f2e50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:57:11.123 INFO 21062 --- [ocp-schedules-9,4c2438ee72994af1,2fc70fc3004ae87c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:57:11.380 WARN 21062 --- [ocp-async-9,939e53c21f0ca900,1791c69035b7c8f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:11.416 INFO 21062 --- [ocp-async-4,7f98ed17779b2b2b,f3e7db2af447b864] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:11.472 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:11.473 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=3, affectRows=3 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=3, validCount=3 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=4, affectRows=4 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=4, validCount=4 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=13, affectRows=13 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=12, validCount=12 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=19, affectRows=19 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=15, affectRows=15 2024-11-04 14:57:11.700 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=26, affectRows=26 2024-11-04 14:57:11.701 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=15, validCount=15 2024-11-04 14:57:11.701 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=19, validCount=19 2024-11-04 14:57:11.701 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=26, validCount=26 2024-11-04 14:57:11.701 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=43, affectRows=43 2024-11-04 14:57:11.701 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=43, validCount=43 2024-11-04 14:57:11.701 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=67, affectRows=67 2024-11-04 14:57:11.702 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=67, validCount=67 2024-11-04 14:57:11.702 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=54, affectRows=54 2024-11-04 14:57:11.702 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=54, validCount=54 2024-11-04 14:57:11.702 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=83, affectRows=83 2024-11-04 14:57:11.702 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703360, nodeCount=95, affectRows=95 2024-11-04 14:57:11.703 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=83, validCount=83 2024-11-04 14:57:11.703 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703360, nodeCount=93, validCount=93 2024-11-04 14:57:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8f68270b1c8072f2,eab229204a6d4f94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:12.327 INFO 21062 --- [ocp-schedules-4,ff903c5606b2ac04,8b8bf0f1de302742] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:57:12.333 INFO 21062 --- [ocp-schedules-4,ff903c5606b2ac04,8b8bf0f1de302742] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:57:12.333 INFO 21062 --- [ocp-schedules-4,ff903c5606b2ac04,8b8bf0f1de302742] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:57:13.381 WARN 21062 --- [ocp-async-4,79d290816c1ae328,499ef7bc2ae86d03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:13.416 INFO 21062 --- [ocp-async-6,a171c89eae8ae31a,925563fcedd3aa3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:13.984 INFO 21062 --- [prometheus-cache-monitor1,d6804fde3ac25767,dc0eb4ef1b438ab6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:57:13.984 INFO 21062 --- [prometheus-cache-monitor1,d6804fde3ac25767,dc0eb4ef1b438ab6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:57:14.473 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:14.474 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:14.983 INFO 21062 --- [prometheus-minute-schedule-1,b7c2ba39edaf49f5,f532da139131c316] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:15.351 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:53:15.327, end=2024-11-04T14:55:15.348} 2024-11-04 14:57:15.355 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:57:15.356 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:57:15.381 WARN 21062 --- [ocp-async-6,a1664991392d7c76,918a6d70e2e79d93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:15.416 INFO 21062 --- [ocp-async-5,243f67300745a46a,00f76ea218e5a3fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:17.381 WARN 21062 --- [ocp-async-5,0fc9ffa0806eeaf3,561fbe412fb86999] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:17.416 INFO 21062 --- [ocp-async-7,f21e161e2b124f9e,2fa71ed6b4fbcae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:17.475 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:17.476 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:17.983 INFO 21062 --- [prometheus-minute-schedule-1,1c043384ee5e7c87,091d837b7c423f5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:57:18.590785+08:00, lastProcessTime=2024-11-04T14:57:08.590277+08:00 2024-11-04 14:57:18.695 INFO 21062 --- [ocp-async-8,5e11d241e51967a9,80b252fc037876c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:57:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:57:18.590785+08:00 2024-11-04 14:57:19.381 WARN 21062 --- [ocp-async-4,f9f030164a742fcb,fe1565f8e0a1fa87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:19.416 INFO 21062 --- [ocp-async-8,eac89ec15c1171b5,7c16baef48b5f998] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:20.476 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:20.477 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:20.486 INFO 21062 --- [ocp-schedules-1,c6690104b4060029,265382e65e4517a6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:57:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e4f88c8759aca434,12ce6c4d79897f88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:21.126 INFO 21062 --- [ocp-schedules-21,c18409ab977ac471,439fae8ff696ec6b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:57:21.381 WARN 21062 --- [ocp-async-8,44e917bdb5e3ea5b,b91858d5be1f4635] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:21.417 INFO 21062 --- [ocp-async-5,ba7847738610e1a8,d6ed2805f9bf55ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:23.382 WARN 21062 --- [ocp-async-5,0d52a1a8be097bb0,0642f9158b5f73d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:23.417 INFO 21062 --- [ocp-async-7,ce778a1465774cb2,3b80bf883451367f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:23.477 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:23.478 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:23.983 INFO 21062 --- [prometheus-minute-schedule-1,f03cf9c74671a768,5a55fced4deb6852] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:25.382 WARN 21062 --- [ocp-async-7,2bbc36a779f1bdd3,95d6f64bfd704736] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:25.417 INFO 21062 --- [ocp-async-6,b33360e7f9c31953,16182124512f9bbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:26.479 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:26.480 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:26.983 INFO 21062 --- [prometheus-minute-schedule-1,64541c8ab7ce17e6,462eb4e460f553f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f04496904d54e38b,d4da9cdec3d725c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:57:27.382 WARN 21062 --- [ocp-async-6,2bf08de4bb98d0d3,e4ca05f81d88c09f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:27.417 INFO 21062 --- [ocp-async-2,2a9867eee6c38bb9,d9baa49fb37a51c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:28.498 INFO 21062 --- [ocp-schedules-30,0d02620d60da224b,4c94bfbb6aa911d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:57:28.500 INFO 21062 --- [ocp-schedules-18,ac10c0521717ff94,0699bf33d0851e5c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:57:28.500 INFO 21062 --- [ocp-schedules-26,ef9d81d11ef06296,65655ac34f86b4a0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:57:28.500 INFO 21062 --- [ocp-schedules-11,201505475b2c0590,57afd2b9935c8320] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:57:28.502 INFO 21062 --- [ocp-schedules-30,0d02620d60da224b,4c94bfbb6aa911d5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:57:28.502 INFO 21062 --- [ocp-schedules-1,95902d384eda58d5,91991cb3417f1e8f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:57:28.503 INFO 21062 --- [ocp-schedules-11,201505475b2c0590,57afd2b9935c8320] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:28.503 INFO 21062 --- [ocp-schedules-18,ac10c0521717ff94,0699bf33d0851e5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:57:28.504 INFO 21062 --- [ocp-schedules-5,6009c9ae96976da8,2cc11410e0bb9b85] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 14:57:28.504 INFO 21062 --- [ocp-schedules-10,154b258a1093d300,b6db09c65b95ce1b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:57:28.505 INFO 21062 --- [ocp-schedules-26,ef9d81d11ef06296,65655ac34f86b4a0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:57:28.507 INFO 21062 --- [ocp-schedules-5,6009c9ae96976da8,2cc11410e0bb9b85] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:28.510 INFO 21062 --- [ocp-schedules-7,6afed106d80f1f62,6c401498494c4e88] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:57:28.510 INFO 21062 --- [ocp-schedules-7,6afed106d80f1f62,6c401498494c4e88] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:57:28.510 INFO 21062 --- [ocp-schedules-6,5e5a3c3cdf895c9f,5c64a5943f6092af] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:57:28.510 INFO 21062 --- [ocp-schedules-4,fd6f51d76cf3334b,45112dab146dbfc8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:57:28.510 INFO 21062 --- [ocp-schedules-6,5e5a3c3cdf895c9f,5c64a5943f6092af] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:57:28.510 INFO 21062 --- [ocp-schedules-4,fd6f51d76cf3334b,45112dab146dbfc8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:57:28.539 INFO 21062 --- [ocp-schedules-24,9a95a4f2cc7bb407,4e327bb475202863] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 14:57:28.539 INFO 21062 --- [ocp-schedules-24,9a95a4f2cc7bb407,4e327bb475202863] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 14:57:28.580 INFO 21062 --- [pool-12-thread-7,ca7dfb9c1727eefc,1edf86d3f49e99ec] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 14:57:28.580 INFO 21062 --- [pool-12-thread-8,ca7dfb9c1727eefc,bcda70a7055635f4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 14:57:28.581 INFO 21062 --- [pool-12-thread-6,ca7dfb9c1727eefc,1b857e0b3b39fa1d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 14:57:28.581 INFO 21062 --- [pool-12-thread-6,ca7dfb9c1727eefc,1b857e0b3b39fa1d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 14:57:28.581 INFO 21062 --- [pool-12-thread-5,ca7dfb9c1727eefc,e089d4baed663a58] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 14:57:28.584 INFO 21062 --- [pool-12-thread-6,ca7dfb9c1727eefc,1b857e0b3b39fa1d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 14:57:28.586 INFO 21062 --- [ForkJoinPool-11-worker-12,ca7dfb9c1727eefc,2aa2b0b12411ca54] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 14:57:28.586 INFO 21062 --- [pool-12-thread-7,ca7dfb9c1727eefc,1edf86d3f49e99ec] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 14:57:28.587 INFO 21062 --- [pool-12-thread-6,ca7dfb9c1727eefc,1b857e0b3b39fa1d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 14:57:28.587 INFO 21062 --- [ForkJoinPool-11-worker-5,ca7dfb9c1727eefc,7fa2f2080c3d08fe] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 14:57:28.587 INFO 21062 --- [ForkJoinPool-11-worker-5,ca7dfb9c1727eefc,7fa2f2080c3d08fe] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 14:57:28.587 INFO 21062 --- [ForkJoinPool-11-worker-5,ca7dfb9c1727eefc,7fa2f2080c3d08fe] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 14:57:28.587 INFO 21062 --- [pool-12-thread-5,ca7dfb9c1727eefc,e089d4baed663a58] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 14:57:28.587 INFO 21062 --- [pool-12-thread-8,ca7dfb9c1727eefc,bcda70a7055635f4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 14:57:28.587 INFO 21062 --- [pool-12-thread-8,ca7dfb9c1727eefc,bcda70a7055635f4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 14:57:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:57:28.588666+08:00, lastProcessTime=2024-11-04T14:57:18.590785+08:00 2024-11-04 14:57:28.648 INFO 21062 --- [ocp-schedules-20,ef8bd12368fc6739,91d0bb4a64e22e27] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:57:28.649 INFO 21062 --- [ocp-schedules-20,ef8bd12368fc6739,91d0bb4a64e22e27] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 14:57:28.696 INFO 21062 --- [ocp-async-4,d604dfeddc955ddb,359a2e8fb642f87a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:57:28.753 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:57:28.588666+08:00 2024-11-04 14:57:28.918 INFO 21062 --- [ocp-schedules-10,dbc31185b07c2f39,ccd3a4da298d4f1a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:57:29.234 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 14:57:29.248 INFO 21062 --- [ocp-schedules-24,a09a4bcfa9845953,f68366d3dbb47b70] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:57:29.248 INFO 21062 --- [ocp-schedules-24,a09a4bcfa9845953,f68366d3dbb47b70] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:57:29.251 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:57:29.254 WARN 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:57:29.255 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:57:29.256 WARN 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:57:29.263 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:57:29.264 WARN 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:57:29.265 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:57:29.266 WARN 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:57:29.271 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:57:29.272 WARN 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:57:29.273 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:57:29.274 WARN 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:57:29.280 INFO 21062 --- [ocp-schedules-2,585bddf2c92c2972,a0005e4dd03f00a2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 14:57:29.357 INFO 21062 --- [ocp-schedules-16,a17209365289b5fa,4134d16b6d3867f8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:57:29.382 WARN 21062 --- [ocp-async-3,f94d4e740092235c,8327c1b2676f76fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:29.385 INFO 21062 --- [ocp-schedules-21,8982740abfe80bf5,23aa93737d61d4d7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:57:29.390 INFO 21062 --- [ocp-schedules-21,8982740abfe80bf5,23aa93737d61d4d7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:57:29.390 INFO 21062 --- [ocp-schedules-21,8982740abfe80bf5,23aa93737d61d4d7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:57:29.417 INFO 21062 --- [ocp-async-4,dfffd21f69ed06c1,d2f992f2da38b3ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:29.480 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:29.481 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:29.806 INFO 21062 --- [ocp-schedules-15,1eeadd282d9218cb,c981bbe88b9e5b0d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:57:29.806 INFO 21062 --- [ocp-schedules-15,1eeadd282d9218cb,c981bbe88b9e5b0d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:57:29.807 INFO 21062 --- [ocp-schedules-15,1eeadd282d9218cb,c981bbe88b9e5b0d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:57:29.807 INFO 21062 --- [ocp-schedules-15,1eeadd282d9218cb,c981bbe88b9e5b0d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:57:29.807 INFO 21062 --- [ocp-schedules-15,1eeadd282d9218cb,c981bbe88b9e5b0d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:57:29.807 INFO 21062 --- [ocp-schedules-15,1eeadd282d9218cb,c981bbe88b9e5b0d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:57:29.983 INFO 21062 --- [prometheus-minute-schedule-1,c8a19929280eb768,f56a521540cfd601] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:30.144 INFO 21062 --- [ocp-schedules-4,035beb5551c6c77b,1a1ea2bde0f6d987] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 14:57:30.144 INFO 21062 --- [ocp-schedules-4,035beb5551c6c77b,1a1ea2bde0f6d987] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 14:57:30.172 INFO 21062 --- [pool-48-thread-1,035beb5551c6c77b,71f608d44cf5e331] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 14:57:30.202 WARN 21062 --- [ocp-schedules-4,035beb5551c6c77b,1a1ea2bde0f6d987] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 14:57:30.202 INFO 21062 --- [ocp-schedules-4,035beb5551c6c77b,1a1ea2bde0f6d987] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 14:57:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:57:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:57:30.526 INFO 21062 --- [ocp-schedules-16,887f21bde74d2785,68567a7d4bd0ed83] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:57:31.129 INFO 21062 --- [ocp-schedules-27,b5d77f4fb6d5c749,efa67f505bbe691c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:57:31.382 WARN 21062 --- [ocp-async-4,01d12ddc70355f7e,02c2eab4b872831d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:31.418 INFO 21062 --- [ocp-async-8,46ed80ae80fad650,013b8dfd6523d0e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:32.336 INFO 21062 --- [ocp-schedules-7,6df732ca7e085deb,4bd06a568612a06b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:57:32.341 INFO 21062 --- [ocp-schedules-7,6df732ca7e085deb,4bd06a568612a06b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:57:32.341 INFO 21062 --- [ocp-schedules-7,6df732ca7e085deb,4bd06a568612a06b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:57:32.481 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:32.482 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:32.494 INFO 21062 --- [ocp-schedules-23,4ac766f4e30f617b,9f829f0407989531] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:57:32.494 INFO 21062 --- [ocp-schedules-23,4ac766f4e30f617b,9f829f0407989531] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:57:32.523 INFO 21062 --- [ocp-schedules-23,4ac766f4e30f617b,9f829f0407989531] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:57:32.523 INFO 21062 --- [ocp-schedules-23,4ac766f4e30f617b,9f829f0407989531] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:57:32.523 INFO 21062 --- [ocp-schedules-3,9f645b8f962a73b9,4057feb79705570d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:57:32.720 INFO 21062 --- [ocp-schedules-6,c067bd1c56dcd83a,b6d2b688ba1041aa] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:57:32.720+08:00, clusters=[1] 2024-11-04 14:57:32.721 INFO 21062 --- [ocp-schedules-6,c067bd1c56dcd83a,b6d2b688ba1041aa] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:57:32.950 INFO 21062 --- [ocp-schedules-28,9730acbea57cdfe6,b0d1eeb1bbfd318f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:57:32.950+08:00, clusters=[1] 2024-11-04 14:57:32.953 INFO 21062 --- [ocp-schedules-28,9730acbea57cdfe6,b0d1eeb1bbfd318f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:57:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3cf7b6d496bd8cf1,10aff6457cf9c801] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:33.383 WARN 21062 --- [ocp-async-8,fe1211455960a30d,4141f5a4f72ee8c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:33.418 INFO 21062 --- [ocp-async-9,2ae322852914f608,7f03bd88811e3070] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:34.400 INFO 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:57:34.400 INFO 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:57:34.400 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:57:34.400 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:57:34.410 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:57:34.411 INFO 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:57:34.411 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:57:34.412 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:57:34.412 WARN 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:57:34.412 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:57:34.413 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:57:34.413 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:57:34.413 INFO 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:57:34.414 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:57:34.414 WARN 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:57:34.414 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:57:34.420 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:57:34.420 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:57:34.421 INFO 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:57:34.432 ERROR 21062 --- [ocp-schedules-9,98a2687c8843402d,e6c69b356e2acdec] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:57:35.383 WARN 21062 --- [ocp-async-5,d92301ccaf1f283e,ad16c7363b0d2e52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:35.418 INFO 21062 --- [ocp-async-7,0b75be4d102de6c9,3c2b609a0bb0f108] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:35.482 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:35.483 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4284fa0992cf09ad,893a4badd7fda237] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:37.383 WARN 21062 --- [ocp-async-1,32bdce4c48bde572,93def1dff3628c0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:37.418 INFO 21062 --- [ocp-async-6,1746dd71d57da1f5,bfe2805d7cff254e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:38.474 INFO 21062 --- [ocp-schedules-12,dfa5f7a34ee89d76,51a92acf92fc49ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:57:38.474 INFO 21062 --- [ocp-schedules-12,dbc2fc081ed6fa74,caaf32899408d755] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:57:38.480 INFO 21062 --- [ocp-schedules-12,dbc2fc081ed6fa74,caaf32899408d755] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:38.483 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:38.484 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:38.504 INFO 21062 --- [ocp-schedules-11,201505475b2c0590,57afd2b9935c8320] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:57:38.504 INFO 21062 --- [ocp-schedules-18,ac10c0521717ff94,0699bf33d0851e5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:57:38.508 INFO 21062 --- [ocp-schedules-5,6009c9ae96976da8,2cc11410e0bb9b85] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 14:57:38.508 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:57:38.508 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:57:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:57:38.590298+08:00, lastProcessTime=2024-11-04T14:57:28.588666+08:00 2024-11-04 14:57:38.696 INFO 21062 --- [ocp-async-5,045001a1c644938c,038ddd6864de584b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:57:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:57:38.590298+08:00 2024-11-04 14:57:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5fc506b398a04718,2503808f900b3f85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:39.383 WARN 21062 --- [ocp-async-2,a31c90a439167528,dbac90938fccbabb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:39.418 INFO 21062 --- [ocp-async-5,c38c17822a728602,fdb938149dd7146a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:40.002 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:57:40.002 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:57:40.550 INFO 21062 --- [ocp-schedules-25,4a0bed64e4bb900a,8a1a25fadd606045] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:57:41.132 INFO 21062 --- [ocp-schedules-29,893273867d50d5f1,c37439df63aa42d0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:57:41.383 WARN 21062 --- [ocp-async-3,719b1543eada9aed,70e3408451bd3804] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:41.419 INFO 21062 --- [ocp-async-1,caa3d1e8c68ff4e0,b3a80d73b181a6f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:41.485 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:41.485 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:41.933 INFO 21062 --- [ocp-schedules-25,88b02ca4be5a2fd9,15eb9d95ec745eee] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:57:41.936 INFO 21062 --- [ocp-schedules-25,88b02ca4be5a2fd9,15eb9d95ec745eee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:41.983 INFO 21062 --- [prometheus-minute-schedule-1,5a737b444e4e078e,809975af54d425bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:42.020 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:57:42.022 ERROR 21062 --- [ocp-schedules-25,88b02ca4be5a2fd9,15eb9d95ec745eee] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:57:42.022 INFO 21062 --- [ocp-schedules-25,88b02ca4be5a2fd9,15eb9d95ec745eee] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:57:43.384 WARN 21062 --- [ocp-async-4,3879821be0286d2b,7601e175e285ac72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:43.419 INFO 21062 --- [ocp-async-10,2a479354606bd38f,c4c8192760f7910f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:44.485 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:44.486 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:44.983 INFO 21062 --- [prometheus-minute-schedule-1,fe07940410a7ef23,6b470f1cfcd0253a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:45.384 WARN 21062 --- [ocp-async-8,5ab5b193b8473554,169276f6879acba0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:45.419 INFO 21062 --- [ocp-async-9,5125eac0a1368c69,0b3ce240c139c4bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:47.384 WARN 21062 --- [ocp-async-5,8fd664aeb4db444b,14298ae40cdc3dce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:47.419 INFO 21062 --- [ocp-async-7,9f0c650b4eb76398,5babced8f453045b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:47.487 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:47.487 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8eb69512c6a0e587,91d951e19796a25f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:57:48.592574+08:00, lastProcessTime=2024-11-04T14:57:38.590298+08:00 2024-11-04 14:57:48.696 INFO 21062 --- [ocp-async-10,866f67f5cc1111a2,a72230941b724e34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:57:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:57:48.592574+08:00 2024-11-04 14:57:49.384 WARN 21062 --- [ocp-async-4,69e26555958d39ff,fb2fb85ace07fe43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:49.419 INFO 21062 --- [ocp-async-10,6898ba6b894ea6eb,c17e583bace57cd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:50.487 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:50.489 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:50.633 INFO 21062 --- [ocp-schedules-4,c6c7c9a54b67bae7,1d2d33cf0ff21fd2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:57:50.983 INFO 21062 --- [prometheus-minute-schedule-1,95c1673125c0539b,5c6cb164fa0ca256] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:51.135 INFO 21062 --- [ocp-schedules-30,d91bdb86b7af7101,4721307f3d5d3d11] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:57:51.384 WARN 21062 --- [ocp-async-8,0597a81c4a641beb,0262d75f2431cd30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:51.419 INFO 21062 --- [ocp-async-9,59289651b56b9668,408480c465a56b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:52.344 INFO 21062 --- [ocp-schedules-2,3064e7c6bef149ac,15ce06ccfe561722] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:57:52.349 INFO 21062 --- [ocp-schedules-2,3064e7c6bef149ac,15ce06ccfe561722] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:57:52.349 INFO 21062 --- [ocp-schedules-2,3064e7c6bef149ac,15ce06ccfe561722] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:57:53.385 WARN 21062 --- [ocp-async-5,cacd942d027aeb4f,639251d89ddeaaef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:53.419 INFO 21062 --- [ocp-async-7,8b076a383fcb45de,dcf8662f0ec3e92f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:53.489 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:53.490 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:53.983 INFO 21062 --- [prometheus-minute-schedule-1,881096ef58fd656a,bf1ae589fbbd961e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:55.385 WARN 21062 --- [ocp-async-1,897b2b5ab58fe951,a7dbf757b261dce6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:55.419 INFO 21062 --- [ocp-async-6,93f01a0a29eed8e1,6c5db90381e61bc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:56.490 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:56.491 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:56.983 INFO 21062 --- [prometheus-minute-schedule-1,8ff2a94842329bfd,020ccba734fe6975] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:57:57.385 WARN 21062 --- [ocp-async-10,6da91541b8d080aa,a71237aaa3380e02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:57.420 INFO 21062 --- [ocp-async-2,621c9109ad5c00f4,9dba6bd476d21723] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:58.500 INFO 21062 --- [ocp-schedules-7,f718d4d3cb11ff8f,409578c22b85fb05] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:57:58.500 INFO 21062 --- [ocp-schedules-9,7eb10332465c71d1,44f0a5677e5f90d2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:57:58.502 INFO 21062 --- [ocp-schedules-6,57113d15951ab2c6,2a952b311dea5fb0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:57:58.504 INFO 21062 --- [ocp-schedules-7,f718d4d3cb11ff8f,409578c22b85fb05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:58.504 INFO 21062 --- [ocp-schedules-10,dee421f49e290468,365f27d5a144f6e6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:57:58.504 INFO 21062 --- [ocp-schedules-9,7eb10332465c71d1,44f0a5677e5f90d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:57:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:57:58.510 INFO 21062 --- [ocp-schedules-16,d32df5274d05e068,d39fe8de7d66a780] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:57:58.511 INFO 21062 --- [ocp-schedules-16,d32df5274d05e068,d39fe8de7d66a780] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:57:58.511 INFO 21062 --- [ocp-schedules-26,da0bd975ef328f5e,8f77a73e4ee319ab] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:57:58.511 INFO 21062 --- [ocp-schedules-26,da0bd975ef328f5e,8f77a73e4ee319ab] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:57:58.511 INFO 21062 --- [ocp-schedules-27,aff3537f3c9befd3,eb1cd60bb6cc30bc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:57:58.511 INFO 21062 --- [ocp-schedules-27,aff3537f3c9befd3,eb1cd60bb6cc30bc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:57:58.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:57:58.592387+08:00, lastProcessTime=2024-11-04T14:57:48.592574+08:00 2024-11-04 14:57:58.696 INFO 21062 --- [ocp-async-7,5f754e63276fc997,0c668dcb90737d0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:57:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:57:58.592387+08:00 2024-11-04 14:57:59.360 INFO 21062 --- [ocp-schedules-23,ccb88e9ce8498e21,225a20daabe83737] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:57:59.385 WARN 21062 --- [ocp-async-5,0c37bc19f167ebc3,912b5cfca6b9ca59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:57:59.420 INFO 21062 --- [ocp-async-7,f17154d9cb92d3ab,150dca1eb2892bef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:57:59.491 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:57:59.492 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:57:59.983 INFO 21062 --- [prometheus-minute-schedule-1,64e2cee941420a1b,4c1141bd6a02637f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703480, interval=60 2024-11-04 14:58:00.658 INFO 21062 --- [ocp-schedules-1,4630e7f267390a3d,697925536ade5be9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:58:01.137 INFO 21062 --- [ocp-schedules-28,165e20059f959f78,e71c94a26f18d79f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:58:01.385 WARN 21062 --- [ocp-async-1,08b09e73a0ca886e,2808ef9ff7e61c5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:01.420 INFO 21062 --- [ocp-async-6,1f33336c5cc47eae,411fea62a2af04aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:02.492 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:02.493 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5a7362cb8f05f509,61cff7dd7cd44330] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:03.385 WARN 21062 --- [ocp-async-10,dc4cd748baa42ea6,8ef221099b2dedc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:03.420 INFO 21062 --- [ocp-async-2,bed2b4fa420fa05a,4cacc0d9b2da114d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:05.113 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 14:58:05.113 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 14:58:05.122 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:58:05.122 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:58:05.122 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:58:05.122 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:58:05.127 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 14:58:05.127 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 14:58:05.127 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 14:58:05.127 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 14:58:05.163 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:58:05.171 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:58:05.171 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:58:05.171 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:58:05.172 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:58:05.172 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:58:05.180 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:58:05.386 WARN 21062 --- [ocp-async-2,fd2b3c7817f60824,8083f8070f1ab4c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:05.421 INFO 21062 --- [ocp-async-3,38cb1c83b1dee19b,38064c7544d7940e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:05.493 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:05.494 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:05.722 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 14:58:05.983 INFO 21062 --- [prometheus-minute-schedule-1,08187718b687a61d,10e06c7f4ea17afc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:06.287 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.159924046S 2024-11-04 14:58:06.287 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 14:58:06.423 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 14:58:06.423 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 14:58:06.451 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 14:58:06.505 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 14:58:06.505 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 14:58:06.506 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 14:58:06.506 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 14:58:06.506 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 14:58:06.506 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 14:58:06.517 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 14:58:06.693 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] o.s.boot.SpringApplication : Started application in 1.571 seconds (JVM running for 8900.372) 2024-11-04 14:58:06.693 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.571820154S 2024-11-04 14:58:07.155 INFO 21062 --- [ocp-schedules-5,ee9577b12728c141,024cef9f8af970f8] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 14:58:07.386 WARN 21062 --- [ocp-async-3,ef17f45715af6060,7b69a4b567902d57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:07.421 INFO 21062 --- [ocp-async-1,4b6c8fc546602d57,0e4b246138c762d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:08.494 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:08.496 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:08.505 INFO 21062 --- [ocp-schedules-9,7eb10332465c71d1,44f0a5677e5f90d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:58:08.505 INFO 21062 --- [ocp-schedules-7,f718d4d3cb11ff8f,409578c22b85fb05] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:58:08.509 INFO 21062 --- [ocp-schedules-12,dbc2fc081ed6fa74,caaf32899408d755] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:58:08.509 INFO 21062 --- [ocp-schedules-12,4d50a0a6b3652375,ede7c73e1d7394e3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:58:08.514 INFO 21062 --- [ocp-schedules-12,4d50a0a6b3652375,ede7c73e1d7394e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:08.544 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:58:08.544 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:58:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:58:08.590650+08:00, lastProcessTime=2024-11-04T14:57:58.592387+08:00 2024-11-04 14:58:08.696 INFO 21062 --- [ocp-async-8,6dd3d054fd710be9,9f194bda8eefe8fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:58:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:58:08.590650+08:00 2024-11-04 14:58:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ce816e868417ca3b,71ac65315727335f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:09.197 INFO 21062 --- [ocp-schedules-29,939fd174550469de,fa5f551aacaa867f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:58:09.200 INFO 21062 --- [ocp-schedules-29,939fd174550469de,fa5f551aacaa867f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:09.201 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:58:09.386 WARN 21062 --- [ocp-async-6,373a9e3f9aeda78c,e5c97c496e74edf8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:09.421 INFO 21062 --- [ocp-async-8,41092ac71bbd132f,ea7b1da76e6e2335] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:10.002 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:58:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:58:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:58:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:58:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:58:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:58:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:58:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:58:10.699 INFO 21062 --- [ocp-schedules-23,6bb97a2b01ed2e5b,3a108130db401e37] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:58:11.140 INFO 21062 --- [ocp-schedules-19,d77cbc92bd52bda2,210dfef59a72935e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:58:11.386 WARN 21062 --- [ocp-async-8,7a0cf963de0c9083,ccfc9efb9e24baa0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:11.421 INFO 21062 --- [ocp-async-5,54b784c9b60c3c2f,548d41c7d97dd230] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:11.496 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:11.497 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:11.703 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=1, affectRows=1 2024-11-04 14:58:11.703 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=0, validCount=0 2024-11-04 14:58:11.703 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=3, affectRows=3 2024-11-04 14:58:11.703 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=3, validCount=3 2024-11-04 14:58:11.704 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=9, affectRows=9 2024-11-04 14:58:11.704 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=9, validCount=9 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=41, affectRows=41 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=27, affectRows=27 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=74, affectRows=74 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=41, validCount=41 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=27, validCount=27 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=27, affectRows=27 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=27, validCount=27 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=74, validCount=74 2024-11-04 14:58:11.705 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=84, affectRows=84 2024-11-04 14:58:11.706 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=83, validCount=83 2024-11-04 14:58:11.706 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=103, affectRows=103 2024-11-04 14:58:11.706 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703420, nodeCount=53, affectRows=53 2024-11-04 14:58:11.707 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=53, validCount=53 2024-11-04 14:58:11.707 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703420, nodeCount=102, validCount=102 2024-11-04 14:58:11.983 INFO 21062 --- [prometheus-minute-schedule-1,070aa30ca3798ddc,a3b4f7642950883c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:12.352 INFO 21062 --- [ocp-schedules-15,95acf130fdc34733,42c82854fcccdaca] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:58:12.358 INFO 21062 --- [ocp-schedules-15,95acf130fdc34733,42c82854fcccdaca] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:58:12.358 INFO 21062 --- [ocp-schedules-15,95acf130fdc34733,42c82854fcccdaca] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:58:13.386 WARN 21062 --- [ocp-async-5,44a18e7f718a814c,eb21eb8ee40ce180] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:13.421 INFO 21062 --- [ocp-async-4,a9506aa82b9c943b,f27559c1b95670f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:13.984 INFO 21062 --- [prometheus-cache-monitor1,de19f2f9399ca05a,4dfd466290d4a5d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:58:13.984 INFO 21062 --- [prometheus-cache-monitor1,de19f2f9399ca05a,4dfd466290d4a5d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:58:14.379 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:52:14.356, end=2024-11-04T14:57:14.376} 2024-11-04 14:58:14.379 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:52:14.356, end=2024-11-04T14:57:14.376} 2024-11-04 14:58:14.387 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 14:58:14.387 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:58:14.388 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 14:58:14.388 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 14:58:14.497 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:14.498 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2414ca88b4619d0a,1365255e20d9cb96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:15.387 WARN 21062 --- [ocp-async-1,81b2bcef351ca043,01912d3e56b16c70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:15.421 INFO 21062 --- [ocp-async-10,7008deceedf770aa,1dc16f6a92aa8557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:17.387 WARN 21062 --- [ocp-async-9,c406ab1f934c5805,3103ef8f5827756b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:17.422 INFO 21062 --- [ocp-async-2,b84e793cb47cb680,a6937f3d4d15ffbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:17.498 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:17.499 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:17.983 INFO 21062 --- [prometheus-minute-schedule-1,a8cc1f1d5094401c,01632821f7867642] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:58:18.590331+08:00, lastProcessTime=2024-11-04T14:58:08.590650+08:00 2024-11-04 14:58:18.696 INFO 21062 --- [ocp-async-4,764be7bc8852c48b,fa451a2f3392cf5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:58:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:58:18.590331+08:00 2024-11-04 14:58:19.201 INFO 21062 --- [ocp-schedules-29,939fd174550469de,fa5f551aacaa867f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:58:19.387 WARN 21062 --- [ocp-async-5,f63e0f66092caa52,810f2d35887a1724] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:19.422 INFO 21062 --- [ocp-async-4,8809cc505bc09d48,795de4d4342ebcca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:20.499 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:20.500 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:20.726 INFO 21062 --- [ocp-schedules-18,101b41f23252c9d9,6d5e4207c1236bfc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:58:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f92b33a8f5fd99ca,1b104e886302548a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:21.143 INFO 21062 --- [ocp-schedules-9,d5e4803e852dfc98,f9843dd8b30f3b71] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:58:21.387 WARN 21062 --- [ocp-async-1,1feb1b4deb7bc3bc,136442fc1a1d05c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:21.422 INFO 21062 --- [ocp-async-10,da5baabde56b6dd5,772141787aa7d365] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:23.387 WARN 21062 --- [ocp-async-9,c47500df419f3490,32b1204ea671c302] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:23.422 INFO 21062 --- [ocp-async-2,27bea453d8679882,aae1d6d5202e61a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:23.500 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:23.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c2b32238f42ecc2b,1bc3a6a2e7133763] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:25.388 WARN 21062 --- [ocp-async-3,6be2d3726f620f6f,16db23fe095c30d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:25.422 INFO 21062 --- [ocp-async-7,987aa1c0c7d9866a,21b06ca9ab634c42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:26.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:26.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:26.983 INFO 21062 --- [prometheus-minute-schedule-1,380c99af341b05e8,5e57f3a1f2114eb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e372f29714e9fd47,d04d101ef7613a1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:58:27.388 WARN 21062 --- [ocp-async-4,fd04ecda49094601,9a74d714cc6f21a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:27.422 INFO 21062 --- [ocp-async-6,34f5a05397142190,385e9c312b74330a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:28.498 INFO 21062 --- [ocp-schedules-21,d542dd585adcb99e,92b45f5466f8f9d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:58:28.499 INFO 21062 --- [ocp-schedules-3,0d0bdd7be8c964e5,0a86acd9251d1a8a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:58:28.500 INFO 21062 --- [ocp-schedules-30,2a7d2104f8948e6c,978d6e890804c8d2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:58:28.500 INFO 21062 --- [ocp-schedules-22,b361899df72f5eb0,02d4b6c0b8537a22] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:58:28.502 INFO 21062 --- [ocp-schedules-21,d542dd585adcb99e,92b45f5466f8f9d4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:58:28.502 INFO 21062 --- [ocp-schedules-14,a52af66ba82b5f06,7913c3aabeeb31ba] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:58:28.504 INFO 21062 --- [ocp-schedules-26,9a2e2562190548d3,203d30d601ed5263] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:58:28.504 INFO 21062 --- [ocp-schedules-3,0d0bdd7be8c964e5,0a86acd9251d1a8a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:58:28.504 INFO 21062 --- [ocp-schedules-22,b361899df72f5eb0,02d4b6c0b8537a22] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:28.505 INFO 21062 --- [ocp-schedules-30,2a7d2104f8948e6c,978d6e890804c8d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:58:28.509 INFO 21062 --- [ocp-schedules-28,a5a0ca19011e9734,a4fba8e36771f898] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:58:28.509 INFO 21062 --- [ocp-schedules-28,a5a0ca19011e9734,a4fba8e36771f898] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:58:28.509 INFO 21062 --- [ocp-schedules-23,11c6df0e2b42df75,fafca48d3b5b0ef0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:58:28.509 INFO 21062 --- [ocp-schedules-23,11c6df0e2b42df75,fafca48d3b5b0ef0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:58:28.511 INFO 21062 --- [ocp-schedules-4,da96dd373b69f943,8c6ebc13f1735cf6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:58:28.511 INFO 21062 --- [ocp-schedules-4,da96dd373b69f943,8c6ebc13f1735cf6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:58:28.583 INFO 21062 --- [ocp-schedules-19,0f9d3f69a3bccfa5,a9c2b081aaa97208] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:58:28.583 INFO 21062 --- [ocp-schedules-19,0f9d3f69a3bccfa5,a9c2b081aaa97208] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:58:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:58:28.589443+08:00, lastProcessTime=2024-11-04T14:58:18.590331+08:00 2024-11-04 14:58:28.697 INFO 21062 --- [ocp-async-5,806fca53e786e2e6,a3efa0f691591ede] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:58:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:58:28.589443+08:00 2024-11-04 14:58:28.896 INFO 21062 --- [ocp-schedules-18,0b97c404d3fd67bc,2f6c44b04e5fdc9b] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 14:58:28.896 INFO 21062 --- [ocp-schedules-18,0b97c404d3fd67bc,2f6c44b04e5fdc9b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 14:58:28.920 INFO 21062 --- [ocp-schedules-8,3ea3bfd0128cdeda,37267720d2588f7d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:58:29.255 INFO 21062 --- [ocp-schedules-26,bdff5f953501ec27,80a929451ddc7155] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:58:29.255 INFO 21062 --- [ocp-schedules-26,bdff5f953501ec27,80a929451ddc7155] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:58:29.364 INFO 21062 --- [ocp-schedules-4,74abf84807ad055f,4115ecdb4df79bd2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:58:29.388 WARN 21062 --- [ocp-async-2,8b50d444a60686d4,8f68edbb565019e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:29.390 INFO 21062 --- [ocp-schedules-16,4af84bac40e40a5e,d8361a8f70c118e0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:58:29.395 INFO 21062 --- [ocp-schedules-16,4af84bac40e40a5e,d8361a8f70c118e0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:58:29.395 INFO 21062 --- [ocp-schedules-16,4af84bac40e40a5e,d8361a8f70c118e0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:58:29.423 INFO 21062 --- [ocp-async-5,4cf48e1220162bce,444872e0aa2fe20c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:29.503 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:29.503 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:29.620 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 14:58:29.620 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 14:58:29.628 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 14:58:29.628 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 14:58:29.628 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 14:58:29.639 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 14:58:29.639 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 14:58:29.640 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 14:58:29.643 INFO 21062 --- [ocp-schedules-15,9d0818533db428f3,78cc1eeaed728e8d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 14:58:29.809 INFO 21062 --- [ocp-schedules-7,8bb703dc3982de4d,e686fd4197df689b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:58:29.809 INFO 21062 --- [ocp-schedules-7,8bb703dc3982de4d,e686fd4197df689b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:58:29.816 INFO 21062 --- [ocp-schedules-7,8bb703dc3982de4d,e686fd4197df689b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:58:29.816 INFO 21062 --- [ocp-schedules-7,8bb703dc3982de4d,e686fd4197df689b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:58:29.820 INFO 21062 --- [ocp-schedules-7,8bb703dc3982de4d,e686fd4197df689b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:58:29.820 INFO 21062 --- [ocp-schedules-7,8bb703dc3982de4d,e686fd4197df689b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:58:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3fba5009d2f455a8,7805dbaeb9365978] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:58:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:58:30.766 INFO 21062 --- [ocp-schedules-19,87cd25c0b79af694,6d32720b7593427f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:58:31.146 INFO 21062 --- [ocp-schedules-11,b8f8b42f7b60abf2,20f26ec7d71e8173] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:58:31.388 WARN 21062 --- [ocp-async-7,243923b9e6f35397,fdd0ab30bb590fb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:31.423 INFO 21062 --- [ocp-async-1,15b58bea78dd5d3b,ac3b35f7bb538f1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:32.360 INFO 21062 --- [ocp-schedules-6,ef20f59f31249d35,948a65dfd4757981] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:58:32.366 INFO 21062 --- [ocp-schedules-6,ef20f59f31249d35,948a65dfd4757981] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:58:32.366 INFO 21062 --- [ocp-schedules-6,ef20f59f31249d35,948a65dfd4757981] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:58:32.503 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:32.505 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:32.523 INFO 21062 --- [ocp-schedules-24,50d34a8e199c62d1,cd82769611ee1580] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:58:32.523 INFO 21062 --- [ocp-schedules-24,50d34a8e199c62d1,cd82769611ee1580] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:58:32.550 INFO 21062 --- [ocp-schedules-24,50d34a8e199c62d1,cd82769611ee1580] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:58:32.550 INFO 21062 --- [ocp-schedules-24,50d34a8e199c62d1,cd82769611ee1580] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:58:32.552 INFO 21062 --- [ocp-schedules-14,ac8f8afa8c5395ba,96e4f7a47fdc99ac] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:58:32.748 INFO 21062 --- [ocp-schedules-9,03513bd471bb074e,8facb1054c447513] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:58:32.748+08:00, clusters=[1] 2024-11-04 14:58:32.749 INFO 21062 --- [ocp-schedules-9,03513bd471bb074e,8facb1054c447513] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:58:32.980 INFO 21062 --- [ocp-schedules-23,e2bea59620a8df49,db9d2577a63dc05e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:58:32.980+08:00, clusters=[1] 2024-11-04 14:58:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8d30cb9b8884db44,ee602ec516c1afa1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:32.983 INFO 21062 --- [ocp-schedules-23,e2bea59620a8df49,db9d2577a63dc05e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:58:32.999 INFO 21062 --- [ocp-schedules-2,2b87c092ee426f46,180fcd48320b6276] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 14:58:33.002 INFO 21062 --- [ocp-schedules-2,2b87c092ee426f46,180fcd48320b6276] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:33.059 ERROR 21062 --- [ocp-schedules-2,2b87c092ee426f46,180fcd48320b6276] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:58:33.059 INFO 21062 --- [ocp-schedules-2,2b87c092ee426f46,180fcd48320b6276] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 14:58:33.388 WARN 21062 --- [ocp-async-6,8a97bb0e0d3c1ea1,b787fc440ce1ea43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:33.423 INFO 21062 --- [ocp-async-8,d2e4a68a9849bdd1,3f039eee71c23a1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:34.441 INFO 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:58:34.441 INFO 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:58:34.441 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:58:34.441 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:58:34.451 INFO 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:58:34.451 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:58:34.452 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:58:34.452 WARN 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:58:34.453 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:58:34.453 INFO 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:58:34.453 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:58:34.454 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:58:34.454 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:58:34.454 WARN 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:58:34.455 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:58:34.455 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:58:34.462 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:58:34.462 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:58:34.463 INFO 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:58:34.474 ERROR 21062 --- [ocp-schedules-3,e61406f4d9ba4317,970b3fd9cf18060e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:58:35.389 WARN 21062 --- [ocp-async-9,0c65eb26bbd8526d,d8b56e4e703b67de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:35.423 INFO 21062 --- [ocp-async-3,cf31f0925fc00726,dd0a772b1fc1cd27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:35.505 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:35.506 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b1d98f16b1c56591,7a3fbd8ca9bd9ab5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:37.389 WARN 21062 --- [ocp-async-5,85b4d198c2f2b337,d3d2fe130048a933] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:37.423 INFO 21062 --- [ocp-async-4,c4f98929e0042dd5,2adf26fea007bf4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:38.505 INFO 21062 --- [ocp-schedules-22,b361899df72f5eb0,02d4b6c0b8537a22] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:58:38.506 INFO 21062 --- [ocp-schedules-30,2a7d2104f8948e6c,978d6e890804c8d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:58:38.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:38.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:38.544 INFO 21062 --- [ocp-schedules-12,4d50a0a6b3652375,ede7c73e1d7394e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:58:38.544 INFO 21062 --- [ocp-schedules-12,3c8dd3b378ab1412,bc499305b5c3217a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:58:38.548 INFO 21062 --- [ocp-schedules-12,3c8dd3b378ab1412,bc499305b5c3217a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:38.576 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:58:38.576 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:58:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:58:38.589782+08:00, lastProcessTime=2024-11-04T14:58:28.589443+08:00 2024-11-04 14:58:38.697 INFO 21062 --- [ocp-async-9,c33dbbd2a40cf250,974a993936946a3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:58:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:58:38.589782+08:00 2024-11-04 14:58:38.983 INFO 21062 --- [prometheus-minute-schedule-1,505e070d970393ff,defa8dfa4d717188] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:39.389 WARN 21062 --- [ocp-async-10,82f2adf9fd734b0b,1c74cccd05affd2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:39.424 INFO 21062 --- [ocp-async-9,f286a019caa7c38f,52b1e7002601be7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:58:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:58:40.793 INFO 21062 --- [ocp-schedules-11,93b7d028bfd8e05e,25818e642220cbe7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:58:41.149 INFO 21062 --- [ocp-schedules-13,ebf46c51df3a28b1,06ee7a9a57c6756b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:58:41.389 WARN 21062 --- [ocp-async-2,adae2f390b4316ff,cbb43ef43da52ea0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:41.424 INFO 21062 --- [ocp-async-5,f8504ce951d47848,e3ebeaf9ab08eb20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:41.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:41.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:58:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:58:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 14:58:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 14:58:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 14:58:41.854 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 14:58:41.855 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 14:58:41.983 INFO 21062 --- [prometheus-minute-schedule-1,174d5f99108ececf,d6aa11de4f03201a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:42.022 INFO 21062 --- [ocp-schedules-5,b6b354d9004114fd,c205de6ffa23b164] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:58:42.026 INFO 21062 --- [ocp-schedules-5,b6b354d9004114fd,c205de6ffa23b164] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:42.109 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:58:42.112 ERROR 21062 --- [ocp-schedules-5,b6b354d9004114fd,c205de6ffa23b164] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:58:42.113 INFO 21062 --- [ocp-schedules-5,b6b354d9004114fd,c205de6ffa23b164] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:58:43.389 WARN 21062 --- [ocp-async-7,eb3915ebad7af845,35e5409bd3cda412] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:43.424 INFO 21062 --- [ocp-async-1,c3732cecb874540c,f5ddf5cfaad43964] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:44.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:44.510 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9e77c9f42c203e2b,df4b8cd184f2d935] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:45.389 WARN 21062 --- [ocp-async-1,ce170d26d47be1ac,18734a663ce327dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:45.424 INFO 21062 --- [ocp-async-8,e0b568d2d8d9928f,cbe1865e405ff540] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:47.390 WARN 21062 --- [ocp-async-8,7d7285c7cff4a1c4,1f7770526f356e30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:47.424 INFO 21062 --- [ocp-async-3,e53254f097e6fc70,c54b0bfb0bf6eaf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:47.511 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:47.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b640f041efe66d75,a88d171045fa9350] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:58:48.590549+08:00, lastProcessTime=2024-11-04T14:58:38.589782+08:00 2024-11-04 14:58:48.697 INFO 21062 --- [ocp-async-1,b6d32cf3e4d5b206,47e2cf73059c7002] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:58:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:58:48.590549+08:00 2024-11-04 14:58:49.390 WARN 21062 --- [ocp-async-5,aaf47e6c4e19d1af,ca3f31b972d68208] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:49.424 INFO 21062 --- [ocp-async-1,20f72812e70e0f28,9ac0cdf631f2f8cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:50.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:50.514 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:50.831 INFO 21062 --- [ocp-schedules-18,6ece9c01bb26c80f,71f77c131ef05016] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:58:50.983 INFO 21062 --- [prometheus-minute-schedule-1,cd87b9184b5f8f60,f16e40ec06ae09cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:51.152 INFO 21062 --- [ocp-schedules-26,1bddd3075c1f9e54,4d580bcf85bdc7b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:58:51.390 WARN 21062 --- [ocp-async-1,a7e5c38955882498,aaf3719f632fbafe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:51.425 INFO 21062 --- [ocp-async-8,c49bf599f5c1ad47,71a01250ac0d102e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:52.369 INFO 21062 --- [ocp-schedules-3,a78cbac33ab830eb,b1341a1623d821ea] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:58:52.374 INFO 21062 --- [ocp-schedules-3,a78cbac33ab830eb,b1341a1623d821ea] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:58:52.374 INFO 21062 --- [ocp-schedules-3,a78cbac33ab830eb,b1341a1623d821ea] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:58:53.390 WARN 21062 --- [ocp-async-8,cdc1780011d4943f,2041ae3815dc8aef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:53.425 INFO 21062 --- [ocp-async-3,298f703a108300f8,134beadeb68716d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:53.514 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:53.515 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6e20742710933cd1,bfb8e21b02761a04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:55.390 WARN 21062 --- [ocp-async-3,6be70488a08823dc,9e3cff664c603584] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:55.425 INFO 21062 --- [ocp-async-4,67a8ab5230fbc3b8,45a1d29e9145ddd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:56.515 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:56.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e20c36e2a3324875,0ce5336187b4cbad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:58:57.391 WARN 21062 --- [ocp-async-4,dedb719b98368b58,fafca7c031913ad6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:57.425 INFO 21062 --- [ocp-async-6,dcdc4c8156e2dc09,fbf2291ad8ab199b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:58.500 INFO 21062 --- [ocp-schedules-7,0b9c8bc078f38dee,5291516b92c7a70a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:58:58.500 INFO 21062 --- [ocp-schedules-22,79b0ff2ce5dbd184,1c75bf32cde3da0a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:58:58.502 INFO 21062 --- [ocp-schedules-25,fc2c5cd93bd3cca4,a8016b584626ae1d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:58:58.504 INFO 21062 --- [ocp-schedules-22,79b0ff2ce5dbd184,1c75bf32cde3da0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:58.504 INFO 21062 --- [ocp-schedules-7,0b9c8bc078f38dee,5291516b92c7a70a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:58:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:58:58.506 INFO 21062 --- [ocp-schedules-3,339fa4ae5ddf1e03,b14ad7f0d0074d0b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:58:58.511 INFO 21062 --- [ocp-schedules-29,f34d7a1ff6346360,a855f6dd6996a8d9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:58:58.511 INFO 21062 --- [ocp-schedules-29,f34d7a1ff6346360,a855f6dd6996a8d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:58:58.511 INFO 21062 --- [ocp-schedules-13,16da49ae2a873978,94efd4d0fd54e4b5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:58:58.511 INFO 21062 --- [ocp-schedules-13,16da49ae2a873978,94efd4d0fd54e4b5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:58:58.511 INFO 21062 --- [ocp-schedules-1,056a74da65c62669,f23e450df19ab99b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:58:58.511 INFO 21062 --- [ocp-schedules-1,056a74da65c62669,f23e450df19ab99b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:58:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:58:58.589272+08:00, lastProcessTime=2024-11-04T14:58:48.590549+08:00 2024-11-04 14:58:58.697 INFO 21062 --- [ocp-async-9,e3990d81273c1db3,f6bc113f51638c70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:58:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:58:58.589272+08:00 2024-11-04 14:58:59.366 INFO 21062 --- [ocp-schedules-25,bd108de93dddf96c,20514664c1a7f784] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:58:59.391 WARN 21062 --- [ocp-async-8,4a67ad4cc379980c,52d7a60fae5249ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:58:59.425 INFO 21062 --- [ocp-async-9,3dd14ab16da1aac3,1de632c605b19923] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:58:59.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:58:59.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:58:59.983 INFO 21062 --- [prometheus-minute-schedule-1,aaf7e330a4216f21,6e74a0ef508bdf35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703540, interval=60 2024-11-04 14:59:00.857 INFO 21062 --- [ocp-schedules-29,9c1f6506fa9f58ef,51e41ebc6ca52916] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:59:01.154 INFO 21062 --- [ocp-schedules-17,2e3087c6a0fe2b99,0555b3cf19255288] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:59:01.391 WARN 21062 --- [ocp-async-9,a5ca42356bae900d,c023196d5dd4d667] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:01.426 INFO 21062 --- [ocp-async-7,6747dfe97c2dbef4,a2449c80a255fddd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:02.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:02.518 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5751cd28af54a993,3762189bc5e45300] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:03.391 WARN 21062 --- [ocp-async-7,57b26910e3d4eb58,06a4e735581a9a8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:03.426 INFO 21062 --- [ocp-async-10,49756288d7f20d29,8248b47dcf561860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:05.391 WARN 21062 --- [ocp-async-6,96e1507a57e94d49,0fc7ffa95a3ea683] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:05.426 INFO 21062 --- [ocp-async-2,b14b3af5f624b2a9,570517e6d31f5333] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:05.518 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:05.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a7e926848002f126,a61f306158ae912b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:07.392 WARN 21062 --- [ocp-async-3,3ab152c5c78f7c8f,d8ca02ab64ae14ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:07.426 INFO 21062 --- [ocp-async-5,2e7d4215902c6f65,c386d7e48100e92e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:08.505 INFO 21062 --- [ocp-schedules-22,79b0ff2ce5dbd184,1c75bf32cde3da0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:59:08.505 INFO 21062 --- [ocp-schedules-7,0b9c8bc078f38dee,5291516b92c7a70a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:59:08.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:08.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:08.577 INFO 21062 --- [ocp-schedules-12,3c8dd3b378ab1412,bc499305b5c3217a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:59:08.577 INFO 21062 --- [ocp-schedules-12,4cd9c7e9f92d5452,cb55cb6d58db9d98] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:59:08.581 INFO 21062 --- [ocp-schedules-12,4cd9c7e9f92d5452,cb55cb6d58db9d98] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:59:08.589912+08:00, lastProcessTime=2024-11-04T14:58:58.589272+08:00 2024-11-04 14:59:08.610 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:59:08.610 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:59:08.697 INFO 21062 --- [ocp-async-10,6c59a484239b1833,4a490b2761ab389c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:59:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:59:08.589912+08:00 2024-11-04 14:59:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b18de2af5b22df10,7747df8d0f54f7de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:09.392 WARN 21062 --- [ocp-async-7,061242a16b57cede,4a5c5b898b52dc4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:09.426 INFO 21062 --- [ocp-async-10,23d31a8e23b5f778,74557811e1913994] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:59:10.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:59:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:59:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:59:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 14:59:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 14:59:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 14:59:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 14:59:10.901 INFO 21062 --- [ocp-schedules-9,2bb3fc325af99129,8fdcca4387574431] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:59:11.156 INFO 21062 --- [ocp-schedules-4,671b127b89dfaaa6,db05b7064249d5d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:59:11.392 WARN 21062 --- [ocp-async-6,09da859e1b0010b5,3ce7f62b23ae5ed2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:11.427 INFO 21062 --- [ocp-async-2,e8cb098944e24960,0136587e5ee153a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:11.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:11.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=5, affectRows=5 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=23, affectRows=23 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=5, validCount=5 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=13, affectRows=13 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=10, affectRows=10 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=10, validCount=10 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=13, validCount=13 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=16, affectRows=16 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=23, validCount=23 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=35, affectRows=35 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=24, affectRows=24 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=16, validCount=16 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=35, validCount=35 2024-11-04 14:59:11.711 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=24, validCount=24 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=52, affectRows=52 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=53, affectRows=53 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=73, affectRows=73 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=51, validCount=51 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=53, validCount=53 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=68, affectRows=68 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=72, validCount=72 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703480, nodeCount=50, affectRows=50 2024-11-04 14:59:11.712 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=67, validCount=67 2024-11-04 14:59:11.713 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703480, nodeCount=50, validCount=50 2024-11-04 14:59:11.983 INFO 21062 --- [prometheus-minute-schedule-1,eabc959e24667e19,9e5ce4f9796d3381] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:12.377 INFO 21062 --- [ocp-schedules-3,d4dbc0c8e3fa1875,64c1132ab6d8986a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:59:12.383 INFO 21062 --- [ocp-schedules-3,d4dbc0c8e3fa1875,64c1132ab6d8986a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:59:12.384 INFO 21062 --- [ocp-schedules-3,d4dbc0c8e3fa1875,64c1132ab6d8986a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:59:13.392 WARN 21062 --- [ocp-async-3,50f69fb844b4e62d,6604d1a140e7d7e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:13.427 INFO 21062 --- [ocp-async-5,465ce57273e48070,22207b836318eb8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:13.984 INFO 21062 --- [prometheus-cache-monitor1,4cdc98c439d78b71,5815a2d4c54d3bea] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 14:59:13.984 INFO 21062 --- [prometheus-cache-monitor1,4cdc98c439d78b71,5815a2d4c54d3bea] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 14:59:14.521 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:14.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2279472e351ad1c8,85a7387f09b1d6ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:15.368 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:55:15.348, end=2024-11-04T14:57:15.365} 2024-11-04 14:59:15.372 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 14:59:15.372 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 14:59:15.392 WARN 21062 --- [ocp-async-4,531531493165a599,33d2f46c89d663fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:15.427 INFO 21062 --- [ocp-async-1,aa457ed173653ff5,2b5968bcdca04d98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:17.392 WARN 21062 --- [ocp-async-10,85fcbffed297c353,5b10ddea63f790af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:17.427 INFO 21062 --- [ocp-async-8,2c69f067c1dec824,b4beefec358970a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:17.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:17.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:17.983 INFO 21062 --- [prometheus-minute-schedule-1,a000ff71f0058803,195eaa6fb5a1b246] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:59:18.591840+08:00, lastProcessTime=2024-11-04T14:59:08.589912+08:00 2024-11-04 14:59:18.697 INFO 21062 --- [ocp-async-5,843eaa5632ee3164,106570217466fe81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:59:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:59:18.591840+08:00 2024-11-04 14:59:19.201 INFO 21062 --- [ocp-schedules-9,d915b578ff5986a5,2fa2734b12a3ed00] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 14:59:19.207 INFO 21062 --- [ocp-schedules-9,d915b578ff5986a5,2fa2734b12a3ed00] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:19.208 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 14:59:19.393 WARN 21062 --- [ocp-async-3,078c8f73f74267d4,8907966e8d1605c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:19.427 INFO 21062 --- [ocp-async-5,8fad793f9e235e39,b78c08b568f8ed45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:20.523 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:20.523 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:20.931 INFO 21062 --- [ocp-schedules-13,7a2bb3d9b8aefd30,7aa1e105494758cf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:59:20.983 INFO 21062 --- [prometheus-minute-schedule-1,0865f640e91861dd,fc2e579a097e3755] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:21.159 INFO 21062 --- [ocp-schedules-16,eefc074624c49d7f,76f692a94b901cb8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:59:21.393 WARN 21062 --- [ocp-async-4,a0b82440e92fe9ef,b99ee8064af637ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:21.428 INFO 21062 --- [ocp-async-1,01eae669900e2aa1,8d1381d7d66f6182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:23.393 WARN 21062 --- [ocp-async-10,b24fab43d820f6e9,62e042411af8dc5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:23.428 INFO 21062 --- [ocp-async-8,fdaa317c042a38e0,9361a38d5b4f3dbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:23.523 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:23.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c2ef8d875f9655d3,7cc8f1f4399d33b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:25.393 WARN 21062 --- [ocp-async-2,52b41b0fe1c692e7,f7e659bc384f67b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:25.428 INFO 21062 --- [ocp-async-9,9e27f62e80e7e6d7,597de0ef4833256f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:26.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:26.525 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:26.983 INFO 21062 --- [prometheus-minute-schedule-1,301f99b7478d6045,170de106d26b0185] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e27d94ca3c14693e,4bac8ae4a4b03121] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=3600 2024-11-04 14:59:27.393 WARN 21062 --- [ocp-async-5,1dec83735463987a,b67e429ef512de1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:27.428 INFO 21062 --- [ocp-async-7,2d652d56a821176e,72a7955a24f7eb2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:28.498 INFO 21062 --- [ocp-schedules-24,0149e19b20cbbfdb,9e0a05fc1f8b483c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 14:59:28.500 INFO 21062 --- [ocp-schedules-17,389b2bc5f1250568,6f2c25636401954a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:59:28.500 INFO 21062 --- [ocp-schedules-25,344c9d6cd4ff1389,408b30328347fee5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 14:59:28.500 INFO 21062 --- [ocp-schedules-29,282a5b3d3b66ae23,31a7f69df3c74228] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:59:28.502 INFO 21062 --- [ocp-schedules-24,0149e19b20cbbfdb,9e0a05fc1f8b483c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 14:59:28.502 INFO 21062 --- [ocp-schedules-19,80709e1e90da76c5,edf1a938a6de7ac1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:59:28.503 INFO 21062 --- [ocp-schedules-17,389b2bc5f1250568,6f2c25636401954a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:59:28.504 INFO 21062 --- [ocp-schedules-29,282a5b3d3b66ae23,31a7f69df3c74228] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:28.505 INFO 21062 --- [ocp-schedules-1,a5f7f07a0c868aef,c939c2880262a6d7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:59:28.506 INFO 21062 --- [ocp-schedules-25,344c9d6cd4ff1389,408b30328347fee5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 14:59:28.511 INFO 21062 --- [ocp-schedules-18,d5303bba4756c87d,f4f0626692e528fc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:59:28.511 INFO 21062 --- [ocp-schedules-18,d5303bba4756c87d,f4f0626692e528fc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:59:28.511 INFO 21062 --- [ocp-schedules-5,fbd41cac3bb4dd19,21b3c3d89e282c32] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:59:28.511 INFO 21062 --- [ocp-schedules-5,fbd41cac3bb4dd19,21b3c3d89e282c32] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:59:28.511 INFO 21062 --- [ocp-schedules-6,46ddf35e032137e2,f43b455d4563c61d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:59:28.511 INFO 21062 --- [ocp-schedules-6,46ddf35e032137e2,f43b455d4563c61d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:59:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:59:28.590562+08:00, lastProcessTime=2024-11-04T14:59:18.591840+08:00 2024-11-04 14:59:28.607 INFO 21062 --- [ocp-schedules-20,8fb6e5b9f3d544d4,a5a67dade36be172] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T14:59:28.607+08:00, tenantSize=0 2024-11-04 14:59:28.698 INFO 21062 --- [ocp-async-3,4975552873cf8771,33e7171e716d521c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:59:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:59:28.590562+08:00 2024-11-04 14:59:28.923 INFO 21062 --- [ocp-schedules-26,f7de9acc86e0489f,c41f1818c2138746] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 14:59:29.208 INFO 21062 --- [ocp-schedules-9,d915b578ff5986a5,2fa2734b12a3ed00] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:59:29.260 INFO 21062 --- [ocp-schedules-1,c39ceb8bc04ea6e3,6715a3d3dc3ee2b8] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 14:59:29.260 INFO 21062 --- [ocp-schedules-1,c39ceb8bc04ea6e3,6715a3d3dc3ee2b8] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 14:59:29.369 INFO 21062 --- [ocp-schedules-6,a22a17ceff0efbd0,81ac204f26bf4936] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:59:29.393 WARN 21062 --- [ocp-async-8,dc6a11654584bc39,30f9673ea5beebc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:29.395 INFO 21062 --- [ocp-schedules-14,0f741e167177f3b7,0e834ccc39d2a0c6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 14:59:29.404 INFO 21062 --- [ocp-schedules-14,0f741e167177f3b7,0e834ccc39d2a0c6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 14:59:29.404 INFO 21062 --- [ocp-schedules-14,0f741e167177f3b7,0e834ccc39d2a0c6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 14:59:29.428 INFO 21062 --- [ocp-async-3,d0481b3930e25dc5,a45da46634fa88fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:29.525 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:29.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:29.825 INFO 21062 --- [ocp-schedules-7,3044f7b88176935e,ad9a9885945da605] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 14:59:29.825 INFO 21062 --- [ocp-schedules-7,3044f7b88176935e,ad9a9885945da605] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 14:59:29.826 INFO 21062 --- [ocp-schedules-7,3044f7b88176935e,ad9a9885945da605] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 14:59:29.826 INFO 21062 --- [ocp-schedules-7,3044f7b88176935e,ad9a9885945da605] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 14:59:29.827 INFO 21062 --- [ocp-schedules-7,3044f7b88176935e,ad9a9885945da605] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 14:59:29.827 INFO 21062 --- [ocp-schedules-7,3044f7b88176935e,ad9a9885945da605] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 14:59:29.983 INFO 21062 --- [prometheus-minute-schedule-1,48522db60fd084a8,d2af644d401d27d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:59:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 14:59:30.973 INFO 21062 --- [ocp-schedules-18,5d1dd41cfafcde9f,b7406cf5cac5918b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:59:31.162 INFO 21062 --- [ocp-schedules-30,c571830ebd5d30f6,6eda939bb23300e9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:59:31.394 WARN 21062 --- [ocp-async-9,b0b900bbcdbbf969,a6993d7ddbb26dbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:31.428 INFO 21062 --- [ocp-async-4,85e1157c1d01a85f,82b3139740cc1b6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:32.386 INFO 21062 --- [ocp-schedules-20,b0d129cb24f12546,463dc56cea257116] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:59:32.391 INFO 21062 --- [ocp-schedules-20,b0d129cb24f12546,463dc56cea257116] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:59:32.391 INFO 21062 --- [ocp-schedules-20,b0d129cb24f12546,463dc56cea257116] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:59:32.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:32.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:32.551 INFO 21062 --- [ocp-schedules-13,9f3810ae8b345198,3d2c46bbe6574659] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 14:59:32.551 INFO 21062 --- [ocp-schedules-13,9f3810ae8b345198,3d2c46bbe6574659] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 14:59:32.580 INFO 21062 --- [ocp-schedules-27,d0bf0fdc8c31c297,f3ae8cacf77f5ee5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 14:59:32.583 INFO 21062 --- [ocp-schedules-13,9f3810ae8b345198,3d2c46bbe6574659] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 14:59:32.583 INFO 21062 --- [ocp-schedules-13,9f3810ae8b345198,3d2c46bbe6574659] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 14:59:32.775 INFO 21062 --- [ocp-schedules-8,ef5c6e29969a4a58,3fa0d067cef16558] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T14:59:32.775+08:00, clusters=[1] 2024-11-04 14:59:32.777 INFO 21062 --- [ocp-schedules-8,ef5c6e29969a4a58,3fa0d067cef16558] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 14:59:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f0e024ab3c6454ae,3ec1f7f443cfa34a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:33.011 INFO 21062 --- [ocp-schedules-26,94381ea2b0e63222,7d78c5bf1f962115] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T14:59:33.011+08:00, clusters=[1] 2024-11-04 14:59:33.015 INFO 21062 --- [ocp-schedules-26,94381ea2b0e63222,7d78c5bf1f962115] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 14:59:33.394 WARN 21062 --- [ocp-async-7,c79cb37a6f22d871,b31929c8d6dea5ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:33.429 INFO 21062 --- [ocp-async-6,a1ed236209d5fc35,6ce6b689d1fb0c50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:34.483 INFO 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 14:59:34.483 INFO 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 14:59:34.483 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 14:59:34.483 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 14:59:34.493 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:59:34.493 INFO 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:59:34.494 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:59:34.495 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:59:34.495 WARN 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:59:34.495 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:59:34.496 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 14:59:34.496 INFO 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 14:59:34.496 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 14:59:34.497 WARN 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 14:59:34.497 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 14:59:34.497 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 14:59:34.505 INFO 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 14:59:34.505 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 14:59:34.505 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 14:59:34.518 ERROR 21062 --- [ocp-schedules-16,826d0b1a07b7ea0e,b13205c435330d16] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 14:59:35.394 WARN 21062 --- [ocp-async-6,8f44345a2dc1b52e,f1edcda6184b4cf3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:35.429 INFO 21062 --- [ocp-async-2,47b80577bb438fe5,6c20642706b1e378] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:35.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:35.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1c16caa96b898097,1a8810880fc46277] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:37.394 WARN 21062 --- [ocp-async-2,6874d097b49a6efa,5b94f0d473fdad43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:37.429 INFO 21062 --- [ocp-async-5,0d903b4908b4b990,779556624c1eef8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:38.504 INFO 21062 --- [ocp-schedules-17,389b2bc5f1250568,6f2c25636401954a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:59:38.504 INFO 21062 --- [ocp-schedules-29,282a5b3d3b66ae23,31a7f69df3c74228] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 14:59:38.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:38.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:59:38.590315+08:00, lastProcessTime=2024-11-04T14:59:28.590562+08:00 2024-11-04 14:59:38.611 INFO 21062 --- [ocp-schedules-12,4cd9c7e9f92d5452,cb55cb6d58db9d98] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 14:59:38.611 INFO 21062 --- [ocp-schedules-12,5696c9e0533808da,d8fb368be838f6ab] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 14:59:38.614 INFO 21062 --- [ocp-schedules-12,5696c9e0533808da,d8fb368be838f6ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:38.642 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 14:59:38.642 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:59:38.698 INFO 21062 --- [ocp-async-10,77957e0f17d41962,c51a9e6bd1468fde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:59:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:59:38.590315+08:00 2024-11-04 14:59:38.983 INFO 21062 --- [prometheus-minute-schedule-1,f59669abe357be95,d247dfaeeb09da12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:39.395 WARN 21062 --- [ocp-async-7,07838ba0aa18996f,b178631921259505] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:39.429 INFO 21062 --- [ocp-async-10,774bb00ac50531cf,c46d3d338e2e37ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:40.002 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 14:59:40.002 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 14:59:41.009 INFO 21062 --- [ocp-schedules-9,45f7e8f43b2a4b49,a863c99de1cfd063] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:59:41.165 INFO 21062 --- [ocp-schedules-26,fa505187c5c22428,3e2315c304ccf691] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:59:41.395 WARN 21062 --- [ocp-async-10,b5a9e45f5df61d91,b9cb433738c98f39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:41.430 INFO 21062 --- [ocp-async-9,009639332bd78159,d2ac8771bcc05379] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:41.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:41.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:41.983 INFO 21062 --- [prometheus-minute-schedule-1,1c2accfb2ba67161,04be488a70c19ae5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:42.113 INFO 21062 --- [ocp-schedules-29,e2024ffde7b89aec,9d546a9fbff551ae] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 14:59:42.117 INFO 21062 --- [ocp-schedules-29,e2024ffde7b89aec,9d546a9fbff551ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:42.201 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 14:59:42.203 ERROR 21062 --- [ocp-schedules-29,e2024ffde7b89aec,9d546a9fbff551ae] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 14:59:42.203 INFO 21062 --- [ocp-schedules-29,e2024ffde7b89aec,9d546a9fbff551ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 14:59:43.395 WARN 21062 --- [ocp-async-9,1db073b13e677a75,cfc1274cad6bb996] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:43.430 INFO 21062 --- [ocp-async-4,23123df990fb0a82,273b17265df36ea6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:44.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:44.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:44.983 INFO 21062 --- [prometheus-minute-schedule-1,ef4d5e97a359e92b,eab191eae33ddfa7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:45.395 WARN 21062 --- [ocp-async-4,4a1bea3afd98ea77,85dc00cdc721a7ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:45.430 INFO 21062 --- [ocp-async-1,bbced4d8d889b861,8cb459fb52983d1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:47.395 WARN 21062 --- [ocp-async-1,02100904bb034776,eec321b8d7d4e4c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:47.430 INFO 21062 --- [ocp-async-2,28b725f13caf0024,4432dd280643570a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:47.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:47.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:47.983 INFO 21062 --- [prometheus-minute-schedule-1,36fef43562e482cc,0be2799d6770531a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:59:48.592393+08:00, lastProcessTime=2024-11-04T14:59:38.590315+08:00 2024-11-04 14:59:48.698 INFO 21062 --- [ocp-async-5,704c4ffeb516365c,8538a15133ceb276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:59:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:59:48.592393+08:00 2024-11-04 14:59:49.395 WARN 21062 --- [ocp-async-9,5423778a53572622,b6bad496a0726ac5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:49.430 INFO 21062 --- [ocp-async-4,51b62e96a2da4b68,d36096e3d1ef339a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:50.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:50.532 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ac911b1819f88ee0,1bc9776ec7173362] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:51.055 INFO 21062 --- [ocp-schedules-13,872e353bb5d27cde,43acb24336d09f8d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 14:59:51.168 INFO 21062 --- [ocp-schedules-11,cc98e67f82eea676,2ba9c9f40886fde7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 14:59:51.396 WARN 21062 --- [ocp-async-4,a58926707b1580c1,a0144cdd070a6bc7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:51.430 INFO 21062 --- [ocp-async-1,518e4a7acd783ed4,8143ac4c936230c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:52.395 INFO 21062 --- [ocp-schedules-18,2d552cf2171bb137,803be8bf0b1471e6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 14:59:52.401 INFO 21062 --- [ocp-schedules-18,2d552cf2171bb137,803be8bf0b1471e6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:59:52.401 INFO 21062 --- [ocp-schedules-18,2d552cf2171bb137,803be8bf0b1471e6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 14:59:53.396 WARN 21062 --- [ocp-async-1,34d9af350c0a5c89,e65bec7917b52513] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:53.430 INFO 21062 --- [ocp-async-2,db1c3c1c6e6f6d7c,4b7317d6741db832] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:53.532 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:53.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c293b323ca544041,a0b9bb65d13a1590] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:55.396 WARN 21062 --- [ocp-async-2,2adae8d1bbfc0f68,250b5749d4770e5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:55.431 INFO 21062 --- [ocp-async-3,564f858d2788dfe9,3b6cbca6b5e7b209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:56.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:56.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:56.983 INFO 21062 --- [prometheus-minute-schedule-1,02a01e39f7c9dc86,e72b39f7888f0a83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 14:59:57.396 WARN 21062 --- [ocp-async-3,735cd83333e01f05,569fe5c883c0379a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:57.431 INFO 21062 --- [ocp-async-7,e4da4a704f93de4c,f3a2127b9a9579ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:58.500 INFO 21062 --- [ocp-schedules-14,8a60833a2cc27f68,92806e8f18f60643] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 14:59:58.500 INFO 21062 --- [ocp-schedules-17,4d186e3134da7121,f72a6c49376020a6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 14:59:58.502 INFO 21062 --- [ocp-schedules-4,278914b3059eea66,cfd36b2c0065559e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 14:59:58.504 INFO 21062 --- [ocp-schedules-14,8a60833a2cc27f68,92806e8f18f60643] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:58.504 INFO 21062 --- [ocp-schedules-23,19cf55463787d6e2,8d3cf606ccc0b823] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 14:59:58.504 INFO 21062 --- [ocp-schedules-17,4d186e3134da7121,f72a6c49376020a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 14:59:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 14:59:58.509 INFO 21062 --- [ocp-schedules-2,55b66695cf23c092,f8c12366c1c12e50] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 14:59:58.509 INFO 21062 --- [ocp-schedules-2,55b66695cf23c092,f8c12366c1c12e50] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 14:59:58.509 INFO 21062 --- [ocp-schedules-21,abaf0ce0cf9a68df,bf7947b325eb7053] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 14:59:58.509 INFO 21062 --- [ocp-schedules-21,abaf0ce0cf9a68df,bf7947b325eb7053] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 14:59:58.511 INFO 21062 --- [ocp-schedules-5,48c616d455a41ef6,0409e628dc2133dd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 14:59:58.511 INFO 21062 --- [ocp-schedules-5,48c616d455a41ef6,0409e628dc2133dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 14:59:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T14:59:58.589735+08:00, lastProcessTime=2024-11-04T14:59:48.592393+08:00 2024-11-04 14:59:58.698 INFO 21062 --- [ocp-async-8,d4c5b6493b96fda4,f930d8f03d3068eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 14:59:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T14:59:58.589735+08:00 2024-11-04 14:59:59.371 INFO 21062 --- [ocp-schedules-2,a3bd898db80b9ce7,85f3fec5063e200a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 14:59:59.396 WARN 21062 --- [ocp-async-1,0a51df65ffbd0800,192415b93269acb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 14:59:59.431 INFO 21062 --- [ocp-async-8,bf57ecd98143de8f,61981b6717949e16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 14:59:59.534 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 14:59:59.534 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 14:59:59.983 INFO 21062 --- [prometheus-minute-schedule-1,916858427044a9ab,d34ebcca651bab44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703600, interval=60 2024-11-04 15:00:00.805 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:55:01.0, nextRun=2024-11-04 15:00:00.0, currentDate=Mon Nov 04 15:00:00 CST 2024 2024-11-04 15:00:00.805 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 14:55:01.0, nextRunTime=2024-11-04 15:00:00.0} 2024-11-04 15:00:00.831 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:00:00.834 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:00:00.834 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:00:00.840 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2024-11-04 15:00:00.0, currentDate=Mon Nov 04 15:00:00 CST 2024 2024-11-04 15:00:00.841 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000001, name=Collect backup files size on schedule, templateName=BackupScheduleCollectTemplateFactory, scheduleRule=0 0 */3 * * *, lastRunTime=null, nextRunTime=2024-11-04 15:00:00.0} 2024-11-04 15:00:00.854 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Collect backup files size, nodeCount=1, taskCount=1 2024-11-04 15:00:00.854 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,f554aa523f2fa606] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Collect backup files size', taskDefinitionId=1000001, state=RUNNING, type=SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T15:00:00.854+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Collect backup files size, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.BackupCollectSizeTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 15:00:00.961 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,4d1fd1638cbf764d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 84, operation:EXECUTE, state: RUNNING 2024-11-04 15:00:00.980 INFO 21062 --- [subtask-executor17,dd193f8f67a5a423,0654bbbddebc990d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=84, context=Context{parallelIdx=-1, stringMap={task_instance_id=20, task_operation=execute, latest_execution_start_time=2024-11-04T15:00:00.954+08:00, sub_task_instance_name=Collect backup files size, sub_task_instance_id=84}, listMap={}}, executor=10.46.128.24 2024-11-04 15:00:00.988 INFO 21062 --- [subtask-executor17,dd193f8f67a5a423,0654bbbddebc990d] c.o.o.b.i.m.ProxyBackupStorageMonitor : collect backup storage info for clusters: 2024-11-04 15:00:00.991 INFO 21062 --- [subtask-executor17,dd193f8f67a5a423,0654bbbddebc990d] c.o.o.b.i.m.ProxyBackupStorageMonitor : collectInfoList=[] 2024-11-04 15:00:00.993 INFO 21062 --- [subtask-executor17,dd193f8f67a5a423,0654bbbddebc990d] c.o.o.b.i.m.s.BackupCollectInfoProvider : storageCollectInfoList size:[0] 2024-11-04 15:00:00.996 INFO 21062 --- [subtask-executor17,dd193f8f67a5a423,0654bbbddebc990d] c.o.o.b.i.m.s.StorageCapacityExporter : Start pre hook computers 2024-11-04 15:00:00.999 INFO 21062 --- [subtask-executor17,dd193f8f67a5a423,0654bbbddebc990d] c.o.o.b.i.m.s.StorageCapacityExporter : Start post hook computers 2024-11-04 15:00:01.006 INFO 21062 --- [subtask-executor17,dd193f8f67a5a423,0654bbbddebc990d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 84, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 15:00:01.086 INFO 21062 --- [ocp-schedules-28,104c33fcb7e1869a,3f56e9c2132c004b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:00:01.171 INFO 21062 --- [ocp-schedules-15,c2c659e9a16de131,0356960d2bdf4b8b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:00:01.397 WARN 21062 --- [ocp-async-8,41db116e363c2df8,3d242611a8d9eb2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:01.431 INFO 21062 --- [ocp-async-5,9b2979a0bf4ef276,0ece321033eb57b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:01.851 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:20, status:SUCCESS 2024-11-04 15:00:01.852 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:20, status:SUCCESS 2024-11-04 15:00:02.535 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:02.535 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1afe6d4615d4c5cd,bfba2e5a2bd8b7aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:03.397 WARN 21062 --- [ocp-async-5,7143a84ad0cf7617,1dc6bb02dbd3ec4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:03.431 INFO 21062 --- [ocp-async-6,7c6a39445bbe1a00,7948b50726241a71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:04.935 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,8ff43483b8c016d0] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:50:04.0, nextRun=2024-11-04 15:00:04.0, currentDate=Mon Nov 04 15:00:04 CST 2024 2024-11-04 15:00:04.936 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,8ff43483b8c016d0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 14:50:04.0, nextRunTime=2024-11-04 15:00:04.0} 2024-11-04 15:00:04.942 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,8ff43483b8c016d0] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 15:00:04.943 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,8ff43483b8c016d0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T15:00:04.942+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 15:00:04.981 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,4c25890d2dc2d6fb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 85, operation:EXECUTE, state: RUNNING 2024-11-04 15:00:04.999 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=85, context=Context{parallelIdx=-1, stringMap={task_instance_id=21, task_operation=execute, latest_execution_start_time=2024-11-04T15:00:04.978+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=85}, listMap={}}, executor=10.46.128.24 2024-11-04 15:00:05.007 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:00:05.007 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:00:05.009 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.010 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:00:05.011 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.011 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.012 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.012 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:00:05.028 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.028 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.029 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.029 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:00:05.042 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.042 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.043 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.043 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:00:05.057 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.057 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.058 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.058 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:00:05.071 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.071 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.072 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.072 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:00:05.078 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.078 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.078 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.079 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:00:05.083 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.083 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.084 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.084 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:00:05.089 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.089 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.090 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.090 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:00:05.096 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.096 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.097 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.098 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:00:05.103 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.103 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.104 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.104 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:00:05.110 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.110 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.111 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.111 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:00:05.119 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.120 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.120 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.121 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:00:05.134 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.134 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.135 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.135 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:00:05.140 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.141 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.141 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.142 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:00:05.147 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.148 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.148 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.149 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:00:05.154 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.155 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.155 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.155 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:00:05.160 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.160 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.161 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.161 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:00:05.165 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.165 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.166 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.166 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:00:05.170 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.171 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.171 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.172 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:00:05.176 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.177 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.177 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.177 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:00:05.181 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.182 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.182 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.183 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:00:05.188 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.188 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.189 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.189 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:00:05.193 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.193 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.194 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.194 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:00:05.198 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.199 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.200 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.200 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:00:05.205 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.205 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.205 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.206 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:00:05.210 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.210 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.211 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.212 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:00:05.216 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.216 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.217 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.217 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:00:05.221 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.222 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.222 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.223 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:00:05.227 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.227 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.228 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.228 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:00:05.232 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.233 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.233 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.234 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:00:05.237 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.238 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.238 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.239 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:00:05.243 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.244 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.244 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.244 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:00:05.249 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.249 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.250 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.250 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:00:05.254 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.255 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.255 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.256 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:00:05.263 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.263 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.264 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.264 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:00:05.271 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.271 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.272 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.272 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:00:05.277 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.278 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.278 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.279 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:00:05.284 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.284 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.285 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.285 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:00:05.290 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.291 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.292 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.292 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:00:05.297 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.297 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.298 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.298 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:00:05.302 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.302 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.303 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.303 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:00:05.307 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.307 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.308 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.309 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:00:05.318 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.318 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.319 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.319 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:00:05.340 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.340 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.341 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.341 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:00:05.347 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.347 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.348 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.348 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:00:05.354 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.355 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.355 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.356 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:00:05.362 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.362 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.363 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.363 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:00:05.367 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.368 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.368 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.369 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:00:05.372 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.373 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.374 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.374 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:00:05.378 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:00:05.378 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:00:05.379 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.380 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:00:05.381 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.381 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.382 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.383 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:00:05.389 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.389 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:00:05.390 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:00:05.390 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:00:05.395 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:00:05.397 WARN 21062 --- [ocp-async-7,df125ad586027685,788a321898271be9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:05.403 INFO 21062 --- [subtask-executor18,dd193f8f67a5a423,4f47186ef5cf09b3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 85, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 15:00:05.432 INFO 21062 --- [ocp-async-10,4e91f41357fadc5f,865380b814cb9214] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:05.536 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:05.536 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:05.867 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:21, status:SUCCESS 2024-11-04 15:00:05.868 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:21, status:SUCCESS 2024-11-04 15:00:05.983 INFO 21062 --- [prometheus-minute-schedule-1,b7daf41da5d561cb,7c71e5d2a4c2adfd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:07.397 WARN 21062 --- [ocp-async-2,95c5a63d89938bf2,6c7b37cbaa2aface] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:07.432 INFO 21062 --- [ocp-async-9,4327f0bbf733f376,ca41f53ba5159705] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:08.505 INFO 21062 --- [ocp-schedules-17,4d186e3134da7121,f72a6c49376020a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:00:08.505 INFO 21062 --- [ocp-schedules-14,8a60833a2cc27f68,92806e8f18f60643] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:00:08.537 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:08.537 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:00:08.589255+08:00, lastProcessTime=2024-11-04T14:59:58.589735+08:00 2024-11-04 15:00:08.643 INFO 21062 --- [ocp-schedules-12,5696c9e0533808da,d8fb368be838f6ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:00:08.643 INFO 21062 --- [ocp-schedules-12,4f8cbe26191acc4d,494686df9bd39bc7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:00:08.646 INFO 21062 --- [ocp-schedules-12,4f8cbe26191acc4d,494686df9bd39bc7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:08.674 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:00:08.674 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:00:08.699 INFO 21062 --- [ocp-async-6,cd7486ba55fc1f97,1b5e0d02466b26ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:00:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:00:08.589255+08:00 2024-11-04 15:00:08.983 INFO 21062 --- [prometheus-minute-schedule-1,562fe2da5e932955,f50898208d8016e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:09.397 WARN 21062 --- [ocp-async-5,85eea7d7d381d7bc,bb6044826d618b96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:09.432 INFO 21062 --- [ocp-async-6,df3ee1c79a31153d,2d20897b3155942c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:10.002 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:00:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:00:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:00:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:00:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:00:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:00:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:00:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:00:11.132 INFO 21062 --- [ocp-schedules-17,57c1fc4cc43d5984,cf34c9532296182e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:00:11.175 INFO 21062 --- [ocp-schedules-5,76c82cc57819e7b3,a9625d290844a48a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:00:11.397 WARN 21062 --- [ocp-async-7,2f3d9c843d4effe7,10117f8a241918a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:11.432 INFO 21062 --- [ocp-async-10,b49a9d6c057ed51a,ac58a099d43ec350] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:11.538 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:11.538 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:11.721 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=1, affectRows=1 2024-11-04 15:00:11.721 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=1, validCount=1 2024-11-04 15:00:11.721 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=4, affectRows=4 2024-11-04 15:00:11.721 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=3, validCount=3 2024-11-04 15:00:11.721 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=8, affectRows=8 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=8, validCount=8 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=24, affectRows=24 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=36, affectRows=36 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=24, validCount=24 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=36, validCount=36 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=43, affectRows=43 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=22, affectRows=22 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=15, affectRows=15 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=43, validCount=43 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=22, validCount=22 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=15, validCount=15 2024-11-04 15:00:11.722 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=34, affectRows=34 2024-11-04 15:00:11.723 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=34, validCount=34 2024-11-04 15:00:11.723 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=61, affectRows=61 2024-11-04 15:00:11.723 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=61, validCount=61 2024-11-04 15:00:11.724 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=45, affectRows=45 2024-11-04 15:00:11.724 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=45, validCount=45 2024-11-04 15:00:11.724 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=47, affectRows=47 2024-11-04 15:00:11.724 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=47, validCount=47 2024-11-04 15:00:11.725 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703540, nodeCount=82, affectRows=82 2024-11-04 15:00:11.725 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703540, nodeCount=80, validCount=80 2024-11-04 15:00:11.983 INFO 21062 --- [prometheus-minute-schedule-1,714b235cc53bdeef,6c2cd81be2b26e0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:12.403 INFO 21062 --- [ocp-schedules-16,96cec85db80abe6f,e33f8f9387d6baa1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:00:12.408 INFO 21062 --- [ocp-schedules-16,96cec85db80abe6f,e33f8f9387d6baa1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:00:12.408 INFO 21062 --- [ocp-schedules-16,96cec85db80abe6f,e33f8f9387d6baa1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:00:13.397 WARN 21062 --- [ocp-async-2,03fbe698f2836218,0c10504ab3d85908] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:13.432 INFO 21062 --- [ocp-async-9,c7fba0a339446534,dbcab1a571dcc753] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:13.984 INFO 21062 --- [prometheus-cache-monitor1,b7d50a08d9f43ec0,6e6599d286ec4040] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:00:13.984 INFO 21062 --- [prometheus-cache-monitor1,b7d50a08d9f43ec0,6e6599d286ec4040] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:00:14.538 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:14.539 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:14.983 INFO 21062 --- [prometheus-minute-schedule-1,bf6a2cd78b8aa67e,aa702b727d78eed6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:15.398 WARN 21062 --- [ocp-async-3,39a1e8a9ae3f1c9d,af44d18e974b7c7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:15.432 INFO 21062 --- [ocp-async-4,dc989f93a7f76925,3ce69f15ae91928a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:17.398 WARN 21062 --- [ocp-async-6,6a0b729b19d565aa,8b13d5e91e232e5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:17.432 INFO 21062 --- [ocp-async-1,15e5b21e2e3487b8,dcec2e158b0cbe53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:17.539 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:17.540 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:17.983 INFO 21062 --- [prometheus-minute-schedule-1,70422e8226b1d345,5ffb2220dca10c36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:00:18.590537+08:00, lastProcessTime=2024-11-04T15:00:08.589255+08:00 2024-11-04 15:00:18.699 INFO 21062 --- [ocp-async-9,d422f30f3a23f409,0c83fc2d330b239f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:00:18.805 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:00:18.590537+08:00 2024-11-04 15:00:19.398 WARN 21062 --- [ocp-async-2,3e054c12bb80b73d,75033ecad289b375] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:19.433 INFO 21062 --- [ocp-async-9,bf9fd70f28fa5cf3,334121737f2c10c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:20.540 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:20.541 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:20.983 INFO 21062 --- [prometheus-minute-schedule-1,c9ddbb7e9339d426,b71a7f456974be1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:21.169 INFO 21062 --- [ocp-schedules-10,0d5935a937ef83ea,51fc2840ad42ccb4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:00:21.177 INFO 21062 --- [ocp-schedules-25,a0bc352741b53065,fd0a5abf4ff5c63c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:00:21.398 WARN 21062 --- [ocp-async-3,0939045c4ec27b59,df326e4c1b97ef84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:21.433 INFO 21062 --- [ocp-async-4,c082816ebebd92ac,34fd92c661dc9332] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:23.398 WARN 21062 --- [ocp-async-6,df1f7f88c0c99bcd,988f0eb24be344cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:23.433 INFO 21062 --- [ocp-async-1,411e40d8e2ed69d7,94fa630f710198ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:23.541 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:23.542 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a55fc9c36a5e326e,71807b6f4ee5cdb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:25.398 WARN 21062 --- [ocp-async-10,a98dfa355f421c0d,f532b1d759b9ac3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:25.433 INFO 21062 --- [ocp-async-8,a0f48c1440a6f6c7,592848dffbb949b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:26.542 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:26.543 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e628e4b74437a0ad,577e8dd7bf497da3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:26.984 INFO 21062 --- [prometheus-hour-schedule-1,158c07e4528487cb,2199549c8572e83b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:00:27.399 WARN 21062 --- [ocp-async-9,20383db6871d76b8,b77e3845b8d4c327] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:27.433 INFO 21062 --- [ocp-async-5,0e8b53888156ad2f,2142ee6d5bf53da0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:28.498 INFO 21062 --- [ocp-schedules-3,41c5e387096fe780,48ad4d8974f4baa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:00:28.499 INFO 21062 --- [ocp-schedules-2,4730a5c3763a0979,7fb53ff579127bc5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:00:28.500 INFO 21062 --- [ocp-schedules-6,35d5964cbe4e9728,b0526eb145884c3c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:00:28.500 INFO 21062 --- [ocp-schedules-1,11fdb047e1824353,727f4b50fe6af17d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:00:28.502 INFO 21062 --- [ocp-schedules-3,41c5e387096fe780,48ad4d8974f4baa2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:00:28.502 INFO 21062 --- [ocp-schedules-19,7234b2244b49ca37,5d587cea694ffa8c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:00:28.503 INFO 21062 --- [ocp-schedules-6,35d5964cbe4e9728,b0526eb145884c3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:00:28.504 INFO 21062 --- [ocp-schedules-1,11fdb047e1824353,727f4b50fe6af17d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:28.504 INFO 21062 --- [ocp-schedules-17,a7c175401d697a70,77b18c64832b5e6a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:00:28.505 INFO 21062 --- [ocp-schedules-2,4730a5c3763a0979,7fb53ff579127bc5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:00:28.509 INFO 21062 --- [ocp-schedules-13,619a25506f0da81f,a4db563344dd92ee] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:00:28.509 INFO 21062 --- [ocp-schedules-5,e53ee110e8113d07,e4c787b5e0652db7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:00:28.509 INFO 21062 --- [ocp-schedules-18,7d321236d6fa1ef5,a1bcd2323d560ce7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:00:28.509 INFO 21062 --- [ocp-schedules-18,7d321236d6fa1ef5,a1bcd2323d560ce7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:00:28.509 INFO 21062 --- [ocp-schedules-13,619a25506f0da81f,a4db563344dd92ee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:00:28.509 INFO 21062 --- [ocp-schedules-5,e53ee110e8113d07,e4c787b5e0652db7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:00:28.583 INFO 21062 --- [ocp-schedules-23,b399e36dd3dfd116,5cc20c3f9b77b431] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:00:28.583 INFO 21062 --- [ocp-schedules-23,b399e36dd3dfd116,5cc20c3f9b77b431] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:00:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:00:28.589691+08:00, lastProcessTime=2024-11-04T15:00:18.590537+08:00 2024-11-04 15:00:28.650 INFO 21062 --- [ocp-schedules-10,e4583c2b13ba2bf8,5d7146482e70e9b7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:00:28.651 INFO 21062 --- [ocp-schedules-10,e4583c2b13ba2bf8,5d7146482e70e9b7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:00:28.699 INFO 21062 --- [ocp-async-2,cad94fef83b5bc2b,01a8ddc212971686] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:00:28.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:00:28.589691+08:00 2024-11-04 15:00:28.925 INFO 21062 --- [ocp-schedules-11,21c2e16df05e67a6,be9ff70cf555cdcb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:00:29.208 INFO 21062 --- [ocp-schedules-5,f8672679f678813b,f48a8edfb394f5ee] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:00:29.212 INFO 21062 --- [ocp-schedules-5,f8672679f678813b,f48a8edfb394f5ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:29.213 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:00:29.266 INFO 21062 --- [ocp-schedules-3,9f3bd977cd3e427f,d51f82170543dcb2] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:00:29.266 INFO 21062 --- [ocp-schedules-3,9f3bd977cd3e427f,d51f82170543dcb2] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:00:29.374 INFO 21062 --- [ocp-schedules-20,43024aebce873fd4,420ce83652559426] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:00:29.399 WARN 21062 --- [ocp-async-1,a759ccf4b0997ee2,0901ca37ddcb711d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:29.404 INFO 21062 --- [ocp-schedules-14,aa04740bf120f13f,f8980c1a8da19745] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:00:29.409 INFO 21062 --- [ocp-schedules-14,aa04740bf120f13f,f8980c1a8da19745] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:00:29.409 INFO 21062 --- [ocp-schedules-14,aa04740bf120f13f,f8980c1a8da19745] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:00:29.433 INFO 21062 --- [ocp-async-2,9aa9e61a76d56339,b21d01f0745ceb92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:29.543 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:29.543 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:29.828 INFO 21062 --- [ocp-schedules-21,4e5c3723942a2673,3b43c88f7ed56ff1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:00:29.828 INFO 21062 --- [ocp-schedules-21,4e5c3723942a2673,3b43c88f7ed56ff1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:00:29.829 INFO 21062 --- [ocp-schedules-21,4e5c3723942a2673,3b43c88f7ed56ff1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:00:29.829 INFO 21062 --- [ocp-schedules-21,4e5c3723942a2673,3b43c88f7ed56ff1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:00:29.830 INFO 21062 --- [ocp-schedules-21,4e5c3723942a2673,3b43c88f7ed56ff1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:00:29.830 INFO 21062 --- [ocp-schedules-21,4e5c3723942a2673,3b43c88f7ed56ff1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:00:29.983 INFO 21062 --- [prometheus-minute-schedule-1,0546c383a726108a,dad76a215b027c5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:00:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:00:31.179 INFO 21062 --- [ocp-schedules-4,9c89cbe683b4fc97,5ecc0af08326d4d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:00:31.209 INFO 21062 --- [ocp-schedules-16,1ff8d1442ef7b3f8,b78e7da998a70b25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:00:31.399 WARN 21062 --- [ocp-async-8,33e8c7b230136615,5efe88ac09e0dc47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:31.434 INFO 21062 --- [ocp-async-3,93bce6e05cc0ffa2,45084d1cba571ffe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:32.410 INFO 21062 --- [ocp-schedules-30,c6b257fc9dea7db5,c1ef90843c62730a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:00:32.415 INFO 21062 --- [ocp-schedules-30,c6b257fc9dea7db5,c1ef90843c62730a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:00:32.415 INFO 21062 --- [ocp-schedules-30,c6b257fc9dea7db5,c1ef90843c62730a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:00:32.543 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:32.544 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:32.583 INFO 21062 --- [ocp-schedules-17,59c0c73c87d40ee5,640cb7dc7b1a3c7f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:00:32.583 INFO 21062 --- [ocp-schedules-17,59c0c73c87d40ee5,640cb7dc7b1a3c7f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:00:32.607 INFO 21062 --- [ocp-schedules-18,ace07e7d2a9fae2d,8acbe6c57f879be3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:00:32.609 INFO 21062 --- [ocp-schedules-17,59c0c73c87d40ee5,640cb7dc7b1a3c7f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:00:32.609 INFO 21062 --- [ocp-schedules-17,59c0c73c87d40ee5,640cb7dc7b1a3c7f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:00:32.803 INFO 21062 --- [ocp-schedules-23,b9160c91499ed746,e2cd22543af31728] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:00:32.803+08:00, clusters=[1] 2024-11-04 15:00:32.804 INFO 21062 --- [ocp-schedules-23,b9160c91499ed746,e2cd22543af31728] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:00:32.983 INFO 21062 --- [prometheus-minute-schedule-1,c89de0f1535d2a2e,eb273319efbd198c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:33.042 INFO 21062 --- [ocp-schedules-11,5a600d88ff3a9bb1,d0a7615a4fe928a0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:00:33.042+08:00, clusters=[1] 2024-11-04 15:00:33.045 INFO 21062 --- [ocp-schedules-11,5a600d88ff3a9bb1,d0a7615a4fe928a0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:00:33.059 INFO 21062 --- [ocp-schedules-27,b94bfd5f57c60fe5,a7c4947b756ba68e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:00:33.062 INFO 21062 --- [ocp-schedules-27,b94bfd5f57c60fe5,a7c4947b756ba68e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:33.120 ERROR 21062 --- [ocp-schedules-27,b94bfd5f57c60fe5,a7c4947b756ba68e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:00:33.120 INFO 21062 --- [ocp-schedules-27,b94bfd5f57c60fe5,a7c4947b756ba68e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:00:33.399 WARN 21062 --- [ocp-async-5,eba700a116c33441,d238c35b4379fc61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:33.434 INFO 21062 --- [ocp-async-7,0fdc19858068a0c0,af7dbeabf096a184] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:34.530 INFO 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:00:34.531 INFO 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:00:34.531 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:00:34.531 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:00:34.541 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:00:34.542 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:00:34.542 INFO 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:00:34.543 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:00:34.543 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:00:34.543 WARN 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:00:34.544 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:00:34.544 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:00:34.544 INFO 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:00:34.545 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:00:34.545 WARN 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:00:34.545 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:00:34.552 INFO 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:00:34.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:00:34.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:00:34.564 ERROR 21062 --- [ocp-schedules-4,773c4da814afecea,e35e9d22f05f0497] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:00:35.400 WARN 21062 --- [ocp-async-7,8c76783fde68300e,7dcc3f42e09d0e5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:35.434 INFO 21062 --- [ocp-async-10,de9c32071f0c5976,13fae9ea956c8530] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:35.545 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:35.545 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e15d57515cc389d0,31aca3911595d584] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:37.400 WARN 21062 --- [ocp-async-10,8d6b806bb9837185,3be8dfb4511f13ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:37.434 INFO 21062 --- [ocp-async-9,762c3ca7042a3874,849779bd30aa26ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:38.504 INFO 21062 --- [ocp-schedules-6,35d5964cbe4e9728,b0526eb145884c3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:00:38.504 INFO 21062 --- [ocp-schedules-1,11fdb047e1824353,727f4b50fe6af17d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:00:38.546 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:38.546 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:00:38.590548+08:00, lastProcessTime=2024-11-04T15:00:28.589691+08:00 2024-11-04 15:00:38.675 INFO 21062 --- [ocp-schedules-12,4f8cbe26191acc4d,494686df9bd39bc7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:00:38.675 INFO 21062 --- [ocp-schedules-12,9e6d46c5dc58397d,19e33a429b7fc6bb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:00:38.679 INFO 21062 --- [ocp-schedules-12,9e6d46c5dc58397d,19e33a429b7fc6bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:38.699 INFO 21062 --- [ocp-async-7,c4fd64716d8d2dd1,3e4b6de529a7839d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:00:38.707 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:00:38.707 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:00:38.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:00:38.590548+08:00 2024-11-04 15:00:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e992ed2a1061d3e5,75ecac68c0390d64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:39.213 INFO 21062 --- [ocp-schedules-5,f8672679f678813b,f48a8edfb394f5ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:00:39.400 WARN 21062 --- [ocp-async-5,4ef56cbd44775bab,5eff70bba246e17a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:39.434 INFO 21062 --- [ocp-async-7,ae5dc677fbc128d4,ae3ed1f5d1ee4980] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:00:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:00:41.183 INFO 21062 --- [ocp-schedules-21,56dd80e9cf026494,6b2c7ededc0a3ece] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:00:41.235 INFO 21062 --- [ocp-schedules-14,f9be7826cc4080e3,5d96dd15092c41bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:00:41.400 WARN 21062 --- [ocp-async-7,07b79911fcb41808,6540f449c5050b78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:41.435 INFO 21062 --- [ocp-async-8,ef70fb71d7688ad2,e5d3f1367cd20a24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:41.546 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:41.547 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:00:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:00:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:00:41.983 INFO 21062 --- [prometheus-minute-schedule-1,200d48fe7e3cdab3,951434da9e7c9e06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:42.203 INFO 21062 --- [ocp-schedules-15,4795d51d590895cd,857fa515d01f8662] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:00:42.207 INFO 21062 --- [ocp-schedules-15,4795d51d590895cd,857fa515d01f8662] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:42.290 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:00:42.294 ERROR 21062 --- [ocp-schedules-15,4795d51d590895cd,857fa515d01f8662] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:00:42.294 INFO 21062 --- [ocp-schedules-15,4795d51d590895cd,857fa515d01f8662] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:00:43.400 WARN 21062 --- [ocp-async-8,19c703fadc58b1f7,b50b766229cf5403] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:43.435 INFO 21062 --- [ocp-async-3,9afc46bf0a86d144,401086141a212115] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:44.547 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:44.548 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e4c0b0c1871bee5b,950f9bc306bdc02d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:45.401 WARN 21062 --- [ocp-async-3,8cf0e3770d11677f,13bc0eb2540dd1ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:45.435 INFO 21062 --- [ocp-async-1,e69dab095d30c4b0,f0b18eeede6184e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:47.401 WARN 21062 --- [ocp-async-7,55eb87dd79c24774,337a475e001f62e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:47.435 INFO 21062 --- [ocp-async-10,c57cc2c116f0f3c7,8d864f0154f8c20e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:47.548 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:47.549 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:47.983 INFO 21062 --- [prometheus-minute-schedule-1,741726846c3d0f1a,ba8bfcc748b172ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:00:48.591749+08:00, lastProcessTime=2024-11-04T15:00:38.590548+08:00 2024-11-04 15:00:48.699 INFO 21062 --- [ocp-async-9,903f7a0d2373f682,1e520ceaab86aa13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:00:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:00:48.591749+08:00 2024-11-04 15:00:49.401 WARN 21062 --- [ocp-async-8,0e2a66e543a89274,21c5d3ee4dc0e89f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:49.435 INFO 21062 --- [ocp-async-3,a302531e1ba9eb65,ee016be4717bb768] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:50.549 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:50.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:50.983 INFO 21062 --- [prometheus-minute-schedule-1,3b074f32f96c74d9,552566eac2e803b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:51.186 INFO 21062 --- [ocp-schedules-4,aa689ef321caccde,f3efe1cb7a7788af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:00:51.278 INFO 21062 --- [ocp-schedules-24,c9d6f97c1f924738,63badc17bd0707c0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:00:51.401 WARN 21062 --- [ocp-async-3,d557d144d4d8cf79,c3de12f1986bf51d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:51.436 INFO 21062 --- [ocp-async-7,857ecfb18fb647f0,784465c74f048307] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:52.423 INFO 21062 --- [ocp-schedules-6,f955a124c991d90e,22a85450e1c57676] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:00:52.429 INFO 21062 --- [ocp-schedules-6,f955a124c991d90e,22a85450e1c57676] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:00:52.429 INFO 21062 --- [ocp-schedules-6,f955a124c991d90e,22a85450e1c57676] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:00:53.402 WARN 21062 --- [ocp-async-7,ba10d0f76690ccfe,2587d6f92292e6bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:53.436 INFO 21062 --- [ocp-async-10,e70d0179e2f08a14,618deec531175cc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:53.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:53.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a153f5784d6b63e9,a9025516cce51c69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:55.402 WARN 21062 --- [ocp-async-10,767ee1aa9f932010,504743f17b8c6dfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:55.436 INFO 21062 --- [ocp-async-9,8f9f30f63524b0f3,e9f7cfd21c1e8355] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:56.550 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:56.551 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:56.983 INFO 21062 --- [prometheus-minute-schedule-1,1b5c905e4fd1d88b,b894c322e2c58de7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:00:57.402 WARN 21062 --- [ocp-async-3,9ed2bc449a583f5e,2cb6b9d335eac0f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:57.436 INFO 21062 --- [ocp-async-5,34b172cc3d15687e,17246edb9972b418] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:58.500 INFO 21062 --- [ocp-schedules-28,684baf6cde52e5c0,9205ebbe2c076330] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:00:58.500 INFO 21062 --- [ocp-schedules-25,3be023e4bdeeb7dc,f3570027c3ccc292] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:00:58.502 INFO 21062 --- [ocp-schedules-26,caeb95ba0d668f75,5843be1be482e5fb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:00:58.504 INFO 21062 --- [ocp-schedules-28,684baf6cde52e5c0,9205ebbe2c076330] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:58.504 INFO 21062 --- [ocp-schedules-25,3be023e4bdeeb7dc,f3570027c3ccc292] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:00:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:00:58.506 INFO 21062 --- [ocp-schedules-5,907b116224c14a17,b8a63e534d846493] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:00:58.510 INFO 21062 --- [ocp-schedules-21,631ea9a44160fdc9,f7b938fca0f552be] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:00:58.510 INFO 21062 --- [ocp-schedules-21,631ea9a44160fdc9,f7b938fca0f552be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:00:58.510 INFO 21062 --- [ocp-schedules-14,9c281e5557f54140,407d92d50071d956] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:00:58.510 INFO 21062 --- [ocp-schedules-14,9c281e5557f54140,407d92d50071d956] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:00:58.510 INFO 21062 --- [ocp-schedules-1,adc7b4890eefe517,937baa12a830839a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:00:58.510 INFO 21062 --- [ocp-schedules-1,adc7b4890eefe517,937baa12a830839a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:00:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:00:58.590569+08:00, lastProcessTime=2024-11-04T15:00:48.591749+08:00 2024-11-04 15:00:58.700 INFO 21062 --- [ocp-async-6,7eeec30fdeb4a353,8550ab83eb3494cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:00:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:00:58.590569+08:00 2024-11-04 15:00:59.376 INFO 21062 --- [ocp-schedules-21,701a396ee12a5776,79829fb1d999ceef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:00:59.402 WARN 21062 --- [ocp-async-7,adb07df3bb497c02,fea79a42eab6129b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:00:59.436 INFO 21062 --- [ocp-async-6,0e635227ec71db4d,3f82e6f279930e9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:00:59.551 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:00:59.552 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:00:59.983 INFO 21062 --- [prometheus-minute-schedule-1,04412c8b844cfd10,91c0f732377b8519] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703660, interval=60 2024-11-04 15:01:00.859 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,7b808c527ff516bf] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 14:01:00.0, nextRun=2024-11-04 15:01:00.0, currentDate=Mon Nov 04 15:01:00 CST 2024 2024-11-04 15:01:00.859 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,7b808c527ff516bf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999995, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=2024-11-04 14:01:00.0, nextRunTime=2024-11-04 15:01:00.0} 2024-11-04 15:01:00.861 WARN 21062 --- [task-coordinator1,dd193f8f67a5a423,7b808c527ff516bf] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-11-04 15:01:00.861 WARN 21062 --- [task-coordinator1,dd193f8f67a5a423,7b808c527ff516bf] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-11-04 15:01:00.862 WARN 21062 --- [task-coordinator1,dd193f8f67a5a423,7b808c527ff516bf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : failed to get template for class, action=com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule java.lang.NullPointerException: null at com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule.ready(CollectClusterSnapshotSchedule.java:74) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.getTaskMeta(ReadyTaskWorker.java:366) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.checkAndScheduleCronTask(ReadyTaskWorker.java:173) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.lambda$loopScheduleTask$1(ReadyTaskWorker.java:134) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.task.engine.coordinator.worker.AbstractTaskWorker.transactionWithoutResult(AbstractTaskWorker.java:61) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.loopScheduleTask(ReadyTaskWorker.java:133) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.work(ReadyTaskWorker.java:102) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lockAndExecute(TaskCoordinator.java:137) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$startup$0(TaskCoordinator.java:75) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:01:00.866 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,7b808c527ff516bf] c.o.o.c.t.e.c.worker.SubtaskGenerator : Split list is empty, name=Collect cluster hour metric, splitKey=cluster_ids, context=Argument{parallelIdx=-1, stringMap={}, listMap={}} 2024-11-04 15:01:00.868 WARN 21062 --- [task-coordinator1,dd193f8f67a5a423,7b808c527ff516bf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Subtask is empty, mark task failed, name=Collect cluster hour metric, taskId=22 2024-11-04 15:01:01.188 INFO 21062 --- [ocp-schedules-30,4cd19492abdf6580,846bd1f256116e6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:01:01.305 INFO 21062 --- [ocp-schedules-20,13f1e8ddb58b2565,036790dd62efe0b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:01:01.402 WARN 21062 --- [ocp-async-6,f25307625ebcb0f7,79de7f6eafc51559] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:01.436 INFO 21062 --- [ocp-async-2,281fdf4db534d736,e0cec8cf3d138e38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:02.552 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:02.553 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:02.983 INFO 21062 --- [prometheus-minute-schedule-1,b0da0e9310268ceb,934fee395beb7e19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:03.402 WARN 21062 --- [ocp-async-2,d1ba852854e92498,ff9ba4733800b023] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:03.436 INFO 21062 --- [ocp-async-4,e9bcd2c9918321c4,b0974c10b0b14b36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:05.402 WARN 21062 --- [ocp-async-5,a9a034b4676a9cd0,387f43178e376751] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:05.437 INFO 21062 --- [ocp-async-7,790d500c9f29e1df,26c386166750d923] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:05.553 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:05.554 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a2d61e1485ad6d7b,8c9ed0ba244dccae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:07.156 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:01:07.156 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:01:07.165 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:01:07.165 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:01:07.165 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:01:07.165 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:01:07.170 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:01:07.170 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:01:07.170 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:01:07.170 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:01:07.207 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:01:07.215 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:01:07.215 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:01:07.215 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:01:07.215 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:01:07.215 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:01:07.223 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:01:07.403 WARN 21062 --- [ocp-async-6,968a79ce47ae0368,8962c74f9ede6274] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:07.437 INFO 21062 --- [ocp-async-8,7815e166470c0630,f000bb4cc48afd8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:07.734 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:01:08.311 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.140531942S 2024-11-04 15:01:08.311 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:01:08.407 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:01:08.408 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:01:08.439 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:01:08.494 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:01:08.494 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:01:08.494 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:01:08.494 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:01:08.494 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:01:08.495 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:01:08.505 INFO 21062 --- [ocp-schedules-25,3be023e4bdeeb7dc,f3570027c3ccc292] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:01:08.505 INFO 21062 --- [ocp-schedules-28,684baf6cde52e5c0,9205ebbe2c076330] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:01:08.506 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:01:08.554 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:08.556 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:01:08.590135+08:00, lastProcessTime=2024-11-04T15:00:58.590569+08:00 2024-11-04 15:01:08.686 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] o.s.boot.SpringApplication : Started application in 1.521 seconds (JVM running for 9082.365) 2024-11-04 15:01:08.686 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.521424844S 2024-11-04 15:01:08.700 INFO 21062 --- [ocp-async-4,dcb6e6b1c1b70a3c,e2ea12035ba28023] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:01:08.708 INFO 21062 --- [ocp-schedules-12,9e6d46c5dc58397d,19e33a429b7fc6bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:01:08.708 INFO 21062 --- [ocp-schedules-12,ee14de493798f433,faf4ae57451b7621] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:01:08.712 INFO 21062 --- [ocp-schedules-12,ee14de493798f433,faf4ae57451b7621] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:08.742 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:01:08.742 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:01:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:01:08.590135+08:00 2024-11-04 15:01:08.983 INFO 21062 --- [prometheus-minute-schedule-1,83afdd5e68cf4673,3a0735246bc14eb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:09.141 INFO 21062 --- [ocp-schedules-21,2c96aaed96f3509d,d2345241b78bf828] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:01:09.403 WARN 21062 --- [ocp-async-2,9d8cfb720aaef208,0c5b93df92951505] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:09.439 INFO 21062 --- [ocp-async-4,44faeffd0bd0c028,fa6f71b24151ab6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:10.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:01:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:01:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:01:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:01:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:01:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:01:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:01:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:01:11.191 INFO 21062 --- [ocp-schedules-2,9345ddf531164726,c05fa2dbe1911997] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:01:11.346 INFO 21062 --- [ocp-schedules-23,39fadd61cc7f0db6,af26a27310e4584c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:01:11.403 WARN 21062 --- [ocp-async-5,2f67a54764ecf9e4,6c4ecca38269db37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:11.437 INFO 21062 --- [ocp-async-7,6e43ec877398a84f,1a48ce9e5510e8b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:11.556 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:11.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:11.720 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703600, nodeCount=136, affectRows=136 2024-11-04 15:01:11.720 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703600, nodeCount=137, affectRows=137 2024-11-04 15:01:11.721 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703600, nodeCount=149, affectRows=149 2024-11-04 15:01:11.721 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703600, nodeCount=134, validCount=134 2024-11-04 15:01:11.721 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703600, nodeCount=137, validCount=137 2024-11-04 15:01:11.721 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703600, nodeCount=148, validCount=148 2024-11-04 15:01:11.983 INFO 21062 --- [prometheus-minute-schedule-1,70ff6e5b6bf4411e,bb403e164cc3f051] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:12.432 INFO 21062 --- [ocp-schedules-22,45cf5235ccac59a6,560cabcd1c9b9fc4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:01:12.438 INFO 21062 --- [ocp-schedules-22,45cf5235ccac59a6,560cabcd1c9b9fc4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:01:12.438 INFO 21062 --- [ocp-schedules-22,45cf5235ccac59a6,560cabcd1c9b9fc4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:01:13.403 WARN 21062 --- [ocp-async-6,98b78c065833dc17,14b9f1707ee930f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:13.437 INFO 21062 --- [ocp-async-8,3f1600cae9dcd75d,59452ba4365dc2ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:13.984 INFO 21062 --- [prometheus-cache-monitor1,fd88fe81e9f879ac,7d7d7c093efcf02d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":838} 2024-11-04 15:01:13.984 INFO 21062 --- [prometheus-cache-monitor1,fd88fe81e9f879ac,7d7d7c093efcf02d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:01:14.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:14.558 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9b9f7ab32c70a066,0215d2f918f2eeb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:15.386 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:57:15.365, end=2024-11-04T14:59:15.383} 2024-11-04 15:01:15.394 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:01:15.394 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:01:15.403 WARN 21062 --- [ocp-async-3,006ea5abd6e2268b,bdc271c036a1d192] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:15.437 INFO 21062 --- [ocp-async-9,61d065be2c9627ab,022f631c85eb2530] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:17.403 WARN 21062 --- [ocp-async-5,290e6d78ae4ddd95,7083aeca7d1e3455] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:17.438 INFO 21062 --- [ocp-async-1,1eefc9b31f55fd59,40c5d55ecf13dcd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:17.558 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:17.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:17.983 INFO 21062 --- [prometheus-minute-schedule-1,feac075dc63c7a2b,ae673ebef8b58685] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:01:18.590767+08:00, lastProcessTime=2024-11-04T15:01:08.590135+08:00 2024-11-04 15:01:18.700 INFO 21062 --- [ocp-async-8,72a1814a8bfbac79,1d2aa2799bb02fa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:01:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:01:18.590767+08:00 2024-11-04 15:01:19.404 WARN 21062 --- [ocp-async-10,3f4a65ecb78a23c7,cf47f0a8e68b65d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:19.438 INFO 21062 --- [ocp-async-8,d22e2c130cce5fcc,74c406c6d319243f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:20.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:20.560 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:20.983 INFO 21062 --- [prometheus-minute-schedule-1,a59698791fe8b808,5ac2bbde22db199f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:21.194 INFO 21062 --- [ocp-schedules-29,238061aa8ecff165,54e999bc5f4dc792] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:01:21.373 INFO 21062 --- [ocp-schedules-17,7a358bbbdb13c5a2,3034e74bca9bd077] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:01:21.404 WARN 21062 --- [ocp-async-3,4fd5a8516554e6f2,5d1a6f473b7d8f6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:21.438 INFO 21062 --- [ocp-async-9,f1a4e3109688a92c,be9801b050ed3b94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:23.404 WARN 21062 --- [ocp-async-5,38aa38d38256836f,85e327cbd090e1d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:23.438 INFO 21062 --- [ocp-async-1,eb9a2b9eaeeec434,428d06b8e3efed14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:23.561 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:23.561 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:23.983 INFO 21062 --- [prometheus-minute-schedule-1,d1014f11d123ed8d,9bda16ae608d49da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:25.404 WARN 21062 --- [ocp-async-7,fbae4101a8417db7,05ea6391fdf6220c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:25.438 INFO 21062 --- [ocp-async-6,72de44507fa81aeb,50a4587fb899b0da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:26.561 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:26.562 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:26.983 INFO 21062 --- [prometheus-minute-schedule-1,195b3f86543a7f3a,fc2d56da27573fcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9b20735131b33ebd,2d1c23bfcab84b42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:01:27.404 WARN 21062 --- [ocp-async-8,8aee05476e5e68c3,04d4d2a6c5b9d8a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:27.439 INFO 21062 --- [ocp-async-2,5a34d5565246eb91,b41caeddbdc5b977] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:28.498 INFO 21062 --- [ocp-schedules-30,54cb3a8dc2b271ad,c18093aeb70f0524] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:01:28.499 INFO 21062 --- [ocp-schedules-13,926c1eb8bac35150,09cfe383d89629fa] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:01:28.499 INFO 21062 --- [ocp-schedules-6,ddb6ea0ad3600f9f,d84a6195eb6cf232] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:01:28.500 INFO 21062 --- [ocp-schedules-18,fd5c51f9eb6ca4d4,79e44982ef2c1d27] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:01:28.501 INFO 21062 --- [ocp-schedules-30,54cb3a8dc2b271ad,c18093aeb70f0524] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:01:28.502 INFO 21062 --- [ocp-schedules-20,6bfb523f1b87e378,921ed92bdb78428f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:01:28.504 INFO 21062 --- [ocp-schedules-18,fd5c51f9eb6ca4d4,79e44982ef2c1d27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:28.504 INFO 21062 --- [ocp-schedules-6,ddb6ea0ad3600f9f,d84a6195eb6cf232] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:28.504 INFO 21062 --- [ocp-schedules-19,0f7e93c7f5c3f4d6,c98cdfbaf2ca25b3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:01:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:01:28.505 INFO 21062 --- [ocp-schedules-13,926c1eb8bac35150,09cfe383d89629fa] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:01:28.510 INFO 21062 --- [ocp-schedules-9,8c836732e02cf626,0774d813e71a4912] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:01:28.510 INFO 21062 --- [ocp-schedules-9,8c836732e02cf626,0774d813e71a4912] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:01:28.510 INFO 21062 --- [ocp-schedules-1,12313e758367b1c3,34f72cadd619c8ca] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:01:28.510 INFO 21062 --- [ocp-schedules-1,12313e758367b1c3,34f72cadd619c8ca] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:01:28.511 INFO 21062 --- [ocp-schedules-4,06e249eb481b6f05,dfad240904fa67ae] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:01:28.511 INFO 21062 --- [ocp-schedules-4,06e249eb481b6f05,dfad240904fa67ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:01:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:01:28.589583+08:00, lastProcessTime=2024-11-04T15:01:18.590767+08:00 2024-11-04 15:01:28.700 INFO 21062 --- [ocp-async-10,0e2b471e96184ba8,1c8a46827bf85834] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:01:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:01:28.589583+08:00 2024-11-04 15:01:28.928 INFO 21062 --- [ocp-schedules-3,debdce91a7c1f94b,d901fa7f4bea087a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:01:29.272 INFO 21062 --- [ocp-schedules-8,e0bcafa32402af63,c040c58d35146915] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:01:29.272 INFO 21062 --- [ocp-schedules-8,e0bcafa32402af63,c040c58d35146915] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:01:29.378 INFO 21062 --- [ocp-schedules-1,8cd49db05cc36833,2568cfbf64944928] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:01:29.405 WARN 21062 --- [ocp-async-1,ca7af5c4e2861fd6,db545c83fbd32602] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:29.409 INFO 21062 --- [ocp-schedules-30,b04a5b82e78cce07,487d602c34c97ce3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:01:29.414 INFO 21062 --- [ocp-schedules-30,b04a5b82e78cce07,487d602c34c97ce3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:01:29.414 INFO 21062 --- [ocp-schedules-30,b04a5b82e78cce07,487d602c34c97ce3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:01:29.439 INFO 21062 --- [ocp-async-10,eb5ce5dfb1ad24ca,6f9726ee1c017366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:29.562 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:29.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:29.643 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:01:29.643 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:01:29.652 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:01:29.652 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:01:29.652 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:01:29.663 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:01:29.663 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:01:29.663 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:01:29.667 INFO 21062 --- [ocp-schedules-23,c52d965f52b0af5d,5aa90b1631de1b4d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:01:29.831 INFO 21062 --- [ocp-schedules-16,d097c22295d71815,51701c5ba3799acc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:01:29.831 INFO 21062 --- [ocp-schedules-16,d097c22295d71815,51701c5ba3799acc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:01:29.839 INFO 21062 --- [ocp-schedules-16,d097c22295d71815,51701c5ba3799acc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:01:29.839 INFO 21062 --- [ocp-schedules-16,d097c22295d71815,51701c5ba3799acc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:01:29.844 INFO 21062 --- [ocp-schedules-16,d097c22295d71815,51701c5ba3799acc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:01:29.844 INFO 21062 --- [ocp-schedules-16,d097c22295d71815,51701c5ba3799acc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:01:29.983 INFO 21062 --- [prometheus-minute-schedule-1,f9a47bbbeab57175,cb37452964d7d3a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:01:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:01:31.197 INFO 21062 --- [ocp-schedules-15,94148a60b3d125bf,67851162675d01d5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:01:31.405 WARN 21062 --- [ocp-async-6,26d67871156f09f6,8221abfb723e6993] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:31.413 INFO 21062 --- [ocp-schedules-19,4bc4ae2aaed9cba8,e59a50b56e9cf7b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:01:31.439 INFO 21062 --- [ocp-async-3,0934192a25cdbb98,9a1c5a7a97c21cd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:32.440 INFO 21062 --- [ocp-schedules-22,ba7d607af5c6acc7,f44c10b77778673b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:01:32.445 INFO 21062 --- [ocp-schedules-22,ba7d607af5c6acc7,f44c10b77778673b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:01:32.445 INFO 21062 --- [ocp-schedules-22,ba7d607af5c6acc7,f44c10b77778673b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:01:32.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:32.564 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:32.609 INFO 21062 --- [ocp-schedules-9,882de552a9b01d7a,b12062365ee1a8aa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:01:32.609 INFO 21062 --- [ocp-schedules-9,882de552a9b01d7a,b12062365ee1a8aa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:01:32.635 INFO 21062 --- [ocp-schedules-9,882de552a9b01d7a,b12062365ee1a8aa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:01:32.635 INFO 21062 --- [ocp-schedules-9,882de552a9b01d7a,b12062365ee1a8aa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:01:32.635 INFO 21062 --- [ocp-schedules-24,0c9ea82974c9eb90,214ae6e8dda0b7fa] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:01:32.830 INFO 21062 --- [ocp-schedules-19,b7223f1d146eb97b,00bbf877ed3082d3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:01:32.830+08:00, clusters=[1] 2024-11-04 15:01:32.832 INFO 21062 --- [ocp-schedules-19,b7223f1d146eb97b,00bbf877ed3082d3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:01:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a8959f149f2fd141,515b9470108763e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:33.072 INFO 21062 --- [ocp-schedules-23,67353a0129e85405,4f5d668346f75944] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:01:33.072+08:00, clusters=[1] 2024-11-04 15:01:33.076 INFO 21062 --- [ocp-schedules-23,67353a0129e85405,4f5d668346f75944] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:01:33.405 WARN 21062 --- [ocp-async-2,41638bd5d20d1209,0fce1b324814f15c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:33.439 INFO 21062 --- [ocp-async-4,f7e604ab4fa1184d,a73f87f7f25a3c47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:34.573 INFO 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:01:34.574 INFO 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:01:34.574 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:01:34.574 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:01:34.583 INFO 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:01:34.583 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:01:34.584 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:01:34.585 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:01:34.585 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:01:34.586 WARN 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:01:34.586 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:01:34.586 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:01:34.586 INFO 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:01:34.587 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:01:34.587 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:01:34.587 WARN 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:01:34.594 INFO 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:01:34.594 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:01:34.594 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:01:34.605 ERROR 21062 --- [ocp-schedules-1,343f3a681f99c571,3c005d2719e37cb8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:01:35.405 WARN 21062 --- [ocp-async-5,ab9f37024b32032f,da851f81f1ffe5f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:35.439 INFO 21062 --- [ocp-async-7,64bddd1d1870251c,a78324b234a6a0cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:35.564 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:35.565 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5c228e0746b08fec,6f421de39566e3da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:37.405 WARN 21062 --- [ocp-async-10,31f616598b525da0,9a70e8c7c1304c87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:37.439 INFO 21062 --- [ocp-async-8,5b743983f0e21032,caf97713b0160eae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:38.505 INFO 21062 --- [ocp-schedules-18,fd5c51f9eb6ca4d4,79e44982ef2c1d27] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:01:38.505 INFO 21062 --- [ocp-schedules-6,ddb6ea0ad3600f9f,d84a6195eb6cf232] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:01:38.565 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:38.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:01:38.589687+08:00, lastProcessTime=2024-11-04T15:01:28.589583+08:00 2024-11-04 15:01:38.700 INFO 21062 --- [ocp-async-5,d906e54477dc3af5,54c0f68fc34e41ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:01:38.742 INFO 21062 --- [ocp-schedules-12,ee14de493798f433,faf4ae57451b7621] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:01:38.743 INFO 21062 --- [ocp-schedules-12,641490f40623352c,9a44e09032a05cd9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:01:38.745 INFO 21062 --- [ocp-schedules-12,641490f40623352c,9a44e09032a05cd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:01:38.589687+08:00 2024-11-04 15:01:38.774 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:01:38.774 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:01:38.983 INFO 21062 --- [prometheus-minute-schedule-1,8338e1399d727392,4bfb1f5bb4761365] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:39.213 INFO 21062 --- [ocp-schedules-14,f16d69703cfb1828,448f2e7484c66e93] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:01:39.216 INFO 21062 --- [ocp-schedules-14,f16d69703cfb1828,448f2e7484c66e93] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:39.217 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:01:39.405 WARN 21062 --- [ocp-async-2,33797bf2d5f030f6,cdfc6825cd7011c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:39.440 INFO 21062 --- [ocp-async-1,05c9df3d61e13990,aa9e8800206404ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:40.002 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:01:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:01:41.200 INFO 21062 --- [ocp-schedules-8,8c02e15b5fa5db6d,e121ebdcb0bf45c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:01:41.406 WARN 21062 --- [ocp-async-1,d0518ee35e4e2d90,5c6b553cad3d38fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:41.438 INFO 21062 --- [ocp-schedules-30,24f694095a0ef3a6,1e52d211a9046fff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:01:41.442 INFO 21062 --- [ocp-async-6,99c79c80e8c1bb45,6e556f1b00a74691] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:41.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:41.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:41.983 INFO 21062 --- [prometheus-minute-schedule-1,ad03d878031302d9,7eef80eefe0385c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:42.295 INFO 21062 --- [ocp-schedules-5,5d2f31be67741149,3ab5a3fc1cc34d63] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:01:42.299 INFO 21062 --- [ocp-schedules-5,5d2f31be67741149,3ab5a3fc1cc34d63] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:42.382 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:01:42.384 ERROR 21062 --- [ocp-schedules-5,5d2f31be67741149,3ab5a3fc1cc34d63] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:01:42.384 INFO 21062 --- [ocp-schedules-5,5d2f31be67741149,3ab5a3fc1cc34d63] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:01:43.406 WARN 21062 --- [ocp-async-6,470ee9cd96797017,53d35d0371016390] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:43.440 INFO 21062 --- [ocp-async-3,613bb14df6654d61,ca47e0b5496e2751] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:44.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:44.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d73de8048e18b34c,261258b557c4acbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:45.406 WARN 21062 --- [ocp-async-3,d72304351e818fb4,1c44342b708214e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:45.440 INFO 21062 --- [ocp-async-5,7f239ad153867941,8014cd467223148b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:47.406 WARN 21062 --- [ocp-async-1,d23f9726dffa4951,ff5d88014a1bc63d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:47.440 INFO 21062 --- [ocp-async-10,b261ffde881689b0,31c1223b7c0b180b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:47.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:47.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e635b066926934cb,392b83ec041671a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:01:48.590355+08:00, lastProcessTime=2024-11-04T15:01:38.589687+08:00 2024-11-04 15:01:48.701 INFO 21062 --- [ocp-async-8,f0fdeb63e99e5bf5,a6d2e597b58f4518] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:01:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:01:48.590355+08:00 2024-11-04 15:01:49.217 INFO 21062 --- [ocp-schedules-14,f16d69703cfb1828,448f2e7484c66e93] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:01:49.407 WARN 21062 --- [ocp-async-6,44859a1b451a3da6,9a8b77d07ec188c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:49.441 INFO 21062 --- [ocp-async-3,4b56b33e259dbe2f,a7de0e7409dc87b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:50.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:50.569 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d3dbae338fadde58,769c6f9e0deb735e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:51.203 INFO 21062 --- [ocp-schedules-4,39c7eb0bb69140e4,72096f65ea0f99a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:01:51.407 WARN 21062 --- [ocp-async-2,706ac80e7d4ec45d,89eca6c651cf3a49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:51.441 INFO 21062 --- [ocp-async-1,9a203454be85419e,ee0ceb57296eeaba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:51.515 INFO 21062 --- [ocp-schedules-13,81d279550bb990fb,e74cab7c1097bb95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:01:52.447 INFO 21062 --- [ocp-schedules-21,360249dd9f64bdb5,7c65606d8e365bc5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:01:52.453 INFO 21062 --- [ocp-schedules-21,360249dd9f64bdb5,7c65606d8e365bc5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:01:52.453 INFO 21062 --- [ocp-schedules-21,360249dd9f64bdb5,7c65606d8e365bc5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:01:53.407 WARN 21062 --- [ocp-async-5,cb7b467595961717,dbbcaf6103c57f89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:53.441 INFO 21062 --- [ocp-async-4,45ad7cf799b84b29,a86d91016a9934a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:53.569 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:53.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:53.983 INFO 21062 --- [prometheus-minute-schedule-1,905734a6c596b257,eb71d918fccfcd21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:55.407 WARN 21062 --- [ocp-async-4,d402b4d9fefa0746,09ece52cf4b94bce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:55.441 INFO 21062 --- [ocp-async-7,6437443b9de48a85,f010e53c748a331c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:56.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:56.571 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:56.983 INFO 21062 --- [prometheus-minute-schedule-1,2343ca0777fa6438,897b1d203a71b1e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:01:57.407 WARN 21062 --- [ocp-async-7,5404797afc3c5d15,3f5bdecf2274eb5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:57.441 INFO 21062 --- [ocp-async-3,62dd28d0145c65eb,3414ad4ba7333e5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:58.499 INFO 21062 --- [ocp-schedules-17,5b0d1bc7718afbc2,8843717e7021157e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:01:58.500 INFO 21062 --- [ocp-schedules-3,4b43b43a099b025c,c6ff9699fcc38e0d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:01:58.502 INFO 21062 --- [ocp-schedules-15,276bcd7068e25811,0fbcef03e434aa0a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:01:58.504 INFO 21062 --- [ocp-schedules-3,4b43b43a099b025c,c6ff9699fcc38e0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:58.504 INFO 21062 --- [ocp-schedules-17,5b0d1bc7718afbc2,8843717e7021157e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:01:58.505 INFO 21062 --- [ocp-schedules-25,34b6d878aedeaed2,82c0e09c6d60fbe5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:01:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:01:58.510 INFO 21062 --- [ocp-schedules-29,3c7972ddb9e4a82e,189579d344a2c99d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:01:58.510 INFO 21062 --- [ocp-schedules-14,e278da2d58375840,bba17f389d62208a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:01:58.510 INFO 21062 --- [ocp-schedules-29,3c7972ddb9e4a82e,189579d344a2c99d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:01:58.510 INFO 21062 --- [ocp-schedules-14,e278da2d58375840,bba17f389d62208a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:01:58.510 INFO 21062 --- [ocp-schedules-21,aa58630346b244f4,e6fbe1579355b804] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:01:58.510 INFO 21062 --- [ocp-schedules-21,aa58630346b244f4,e6fbe1579355b804] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:01:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:01:58.590489+08:00, lastProcessTime=2024-11-04T15:01:48.590355+08:00 2024-11-04 15:01:58.701 INFO 21062 --- [ocp-async-10,433e17ac370b3958,c738d9553c43739b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:01:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:01:58.590489+08:00 2024-11-04 15:01:59.381 INFO 21062 --- [ocp-schedules-22,5ec37f0c34a91560,daabd1b069706550] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:01:59.408 WARN 21062 --- [ocp-async-1,78abdab477ba3a8c,cc203ff985ba5eeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:01:59.442 INFO 21062 --- [ocp-async-10,382be701942872e8,e838eed966f44891] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:01:59.571 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:01:59.572 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:01:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7e55434c5e781e11,76c0ae9a42b14377] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703720, interval=60 2024-11-04 15:02:01.206 INFO 21062 --- [ocp-schedules-5,9882baadd56c85f6,452b29e23ce4281c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:02:01.408 WARN 21062 --- [ocp-async-6,9ebd2b10dce49355,5bb2ef1339ceca9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:01.442 INFO 21062 --- [ocp-async-8,15ebbc318d88f06a,469615fee9e504f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:01.541 INFO 21062 --- [ocp-schedules-4,158ec81f14c76a65,71370d3aed91a86d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:02:02.572 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:02.573 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:02.983 INFO 21062 --- [prometheus-minute-schedule-1,57e284f213b74a72,776a4067a8f760af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:03.408 WARN 21062 --- [ocp-async-2,a3fb8f24abbb09db,ef19242b0865320b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:03.442 INFO 21062 --- [ocp-async-9,0d9c3054a3324009,1dc016e167c86177] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:05.408 WARN 21062 --- [ocp-async-1,e4256145acb88504,b70a670419715ca0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:05.442 INFO 21062 --- [ocp-async-5,56da9acba70c10ec,bad634c2a296ecac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:05.573 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:05.574 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ecabe2fce6fd66ff,1f3be9eddfe9e6b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:07.408 WARN 21062 --- [ocp-async-4,c0d94e6701a73f1b,13c8a8c0d0b881c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:07.442 INFO 21062 --- [ocp-async-10,a15d196570acc3e6,cf6343105b9ac911] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:08.505 INFO 21062 --- [ocp-schedules-17,5b0d1bc7718afbc2,8843717e7021157e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:02:08.505 INFO 21062 --- [ocp-schedules-3,4b43b43a099b025c,c6ff9699fcc38e0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:02:08.574 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:08.575 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:02:08.590135+08:00, lastProcessTime=2024-11-04T15:01:58.590489+08:00 2024-11-04 15:02:08.701 INFO 21062 --- [ocp-async-9,86231443a0b734a4,4f814d10505f00a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:02:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:02:08.590135+08:00 2024-11-04 15:02:08.775 INFO 21062 --- [ocp-schedules-12,641490f40623352c,9a44e09032a05cd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:02:08.775 INFO 21062 --- [ocp-schedules-12,2f2e309aa9086b85,6a06fce0b483b303] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:02:08.778 INFO 21062 --- [ocp-schedules-12,2f2e309aa9086b85,6a06fce0b483b303] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:08.806 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:02:08.806 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:02:08.983 INFO 21062 --- [prometheus-minute-schedule-1,6a46239d00540ab5,ee2a5a4233090320] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:09.408 WARN 21062 --- [ocp-async-7,cb1b200682d6a465,3f47746c0eaeb8dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:09.442 INFO 21062 --- [ocp-async-9,46f3728a299437d8,190e1486ed003aef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:10.002 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:02:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:02:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:02:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:02:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:02:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:02:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:02:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:02:11.210 INFO 21062 --- [ocp-schedules-15,cc8db3a425a98983,9142b0a85165bfc1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:02:11.409 WARN 21062 --- [ocp-async-3,a8d42af7b452720e,b68fec10f8238cf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:11.443 INFO 21062 --- [ocp-async-5,7b27c643e9504a1b,d4e2a6acef602b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:11.575 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:11.576 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:11.582 INFO 21062 --- [ocp-schedules-25,58027a1709fd93d6,a6ecacfa6fdee5c5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:02:11.721 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=22, affectRows=22 2024-11-04 15:02:11.721 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=22, validCount=22 2024-11-04 15:02:11.721 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=39, affectRows=39 2024-11-04 15:02:11.721 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=32, affectRows=32 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=31, validCount=31 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=39, validCount=39 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=24, affectRows=24 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=69, affectRows=69 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=24, validCount=24 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=66, affectRows=66 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=67, validCount=67 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=41, affectRows=41 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=55, affectRows=55 2024-11-04 15:02:11.722 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703660, nodeCount=74, affectRows=74 2024-11-04 15:02:11.723 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=41, validCount=41 2024-11-04 15:02:11.723 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=66, validCount=66 2024-11-04 15:02:11.723 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=55, validCount=55 2024-11-04 15:02:11.723 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703660, nodeCount=74, validCount=74 2024-11-04 15:02:11.983 INFO 21062 --- [prometheus-minute-schedule-1,fabb95f879cbed43,f9cae7cec5dfe892] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:12.455 INFO 21062 --- [ocp-schedules-14,aaa1f79a2c90dffd,df48a672fb9fcaf3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:02:12.461 INFO 21062 --- [ocp-schedules-14,aaa1f79a2c90dffd,df48a672fb9fcaf3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:02:12.461 INFO 21062 --- [ocp-schedules-14,aaa1f79a2c90dffd,df48a672fb9fcaf3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:02:13.409 WARN 21062 --- [ocp-async-6,b0a056edbc4be74a,0d406e8c9c021a14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:13.443 INFO 21062 --- [ocp-async-10,652494c55c118bb8,554c17cb33c48fad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:13.984 INFO 21062 --- [prometheus-cache-monitor1,7bfd11429562cf0d,ce23ca341f7c3329] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":838} 2024-11-04 15:02:13.984 INFO 21062 --- [prometheus-cache-monitor1,7bfd11429562cf0d,ce23ca341f7c3329] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:02:14.576 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:14.577 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:14.983 INFO 21062 --- [prometheus-minute-schedule-1,6d1122284bcfbd75,ae31d02c51873201] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:15.409 WARN 21062 --- [ocp-async-7,6535b8f0f012aa72,e74533c2b30e35fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:15.443 INFO 21062 --- [ocp-async-2,7423b088595fb34d,607289a0635a4ce6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:17.409 WARN 21062 --- [ocp-async-9,05de4c9eac044787,81fef6ce8993756b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:17.443 INFO 21062 --- [ocp-async-1,2f1b9000f7cb4b89,bc27b1444817f6fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:17.577 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:17.577 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:17.983 INFO 21062 --- [prometheus-minute-schedule-1,a892429952a16f23,e862772d2e9c7f93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:02:18.590538+08:00, lastProcessTime=2024-11-04T15:02:08.590135+08:00 2024-11-04 15:02:18.701 INFO 21062 --- [ocp-async-10,1d20a2dd9a35df9e,1c02c488027a5dce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:02:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:02:18.590538+08:00 2024-11-04 15:02:19.409 WARN 21062 --- [ocp-async-6,fab4b89640ff37af,6f884cd752f7a85c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:19.443 INFO 21062 --- [ocp-async-10,be514caaabbda548,49ca3d77471a11a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:20.578 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:20.578 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:20.983 INFO 21062 --- [prometheus-minute-schedule-1,29573363cceb2433,c330bbce02327150] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:21.212 INFO 21062 --- [ocp-schedules-30,2f6dc057fa61f397,2e202464beab2fbe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:02:21.409 WARN 21062 --- [ocp-async-7,98977c20fdf897a4,589ac29aed9984b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:21.444 INFO 21062 --- [ocp-async-2,ee79f0eacf5510ed,d7d9eab81874ee3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:21.621 INFO 21062 --- [ocp-schedules-19,ff831718767cbe5b,7e4f494d86e0ce94] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:02:23.409 WARN 21062 --- [ocp-async-9,38754e05c93e7679,25d81a82dbce280e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:23.444 INFO 21062 --- [ocp-async-1,d6acdbf45c60e992,263714e6c2862333] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:23.578 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:23.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6c9cc5a66eedc46b,500bc6e657eed219] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:25.410 WARN 21062 --- [ocp-async-4,c62cc36c8fa699d5,f66b84b3a7b57672] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:25.444 INFO 21062 --- [ocp-async-6,70f47265a5461e3a,c7f0a5aee71e170f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:26.579 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:26.580 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:26.983 INFO 21062 --- [prometheus-minute-schedule-1,95f665e22d48fd06,58aa314cdb8cc409] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b8dd82230a537de8,ef551f9dda9c03cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:02:27.410 WARN 21062 --- [ocp-async-7,7e713d4de1ed60a6,50d564f4c5bb90e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:27.444 INFO 21062 --- [ocp-async-8,f887d5f66c085a48,74e61f78e869ab65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:28.498 INFO 21062 --- [ocp-schedules-21,c3afbe2a64377f32,a331bf4694f77ee5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:02:28.499 INFO 21062 --- [ocp-schedules-13,34911735709c40b6,667b7c2ce6cceb8a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:02:28.500 INFO 21062 --- [ocp-schedules-14,9c5cb6df82bfe6d6,f89987296d6aed29] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:02:28.500 INFO 21062 --- [ocp-schedules-28,6e499bccfbdf7aaa,5db341b7755099fa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:02:28.502 INFO 21062 --- [ocp-schedules-21,c3afbe2a64377f32,a331bf4694f77ee5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:02:28.502 INFO 21062 --- [ocp-schedules-25,c00872e6fd9b211c,406a3db3a1ba0232] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:02:28.503 INFO 21062 --- [ocp-schedules-14,9c5cb6df82bfe6d6,f89987296d6aed29] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:28.504 INFO 21062 --- [ocp-schedules-28,6e499bccfbdf7aaa,5db341b7755099fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:02:28.504 INFO 21062 --- [ocp-schedules-27,dbc6a88a108d1570,aa629b1c9cefbebf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:02:28.504 INFO 21062 --- [ocp-schedules-10,a32485146e3b5c86,0d73a009c77b8768] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:02:28.505 INFO 21062 --- [ocp-schedules-13,34911735709c40b6,667b7c2ce6cceb8a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:02:28.507 INFO 21062 --- [ocp-schedules-10,a32485146e3b5c86,0d73a009c77b8768] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:28.509 INFO 21062 --- [ocp-schedules-15,4f09deb2dc36a7b1,a58bfcb7fec125b8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:02:28.509 INFO 21062 --- [ocp-schedules-15,4f09deb2dc36a7b1,a58bfcb7fec125b8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:02:28.509 INFO 21062 --- [ocp-schedules-11,f514c6394fea68c6,c035387cb929a520] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:02:28.509 INFO 21062 --- [ocp-schedules-11,f514c6394fea68c6,c035387cb929a520] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:02:28.510 INFO 21062 --- [ocp-schedules-2,e4cbefdc5d2c8c97,2a40d5a933b3202f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:02:28.510 INFO 21062 --- [ocp-schedules-2,e4cbefdc5d2c8c97,2a40d5a933b3202f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:02:28.540 INFO 21062 --- [ocp-schedules-1,cbff93ebb8afa59a,4f9967afedb1be04] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:02:28.540 INFO 21062 --- [ocp-schedules-1,cbff93ebb8afa59a,4f9967afedb1be04] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:02:28.580 INFO 21062 --- [pool-12-thread-12,128768c178504bd7,dbac833922fe4d33] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:02:28.580 INFO 21062 --- [pool-12-thread-11,128768c178504bd7,a74da718c2b5d7e9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:02:28.581 INFO 21062 --- [pool-12-thread-9,128768c178504bd7,2abc2f95ed0879b0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:02:28.582 INFO 21062 --- [pool-12-thread-10,128768c178504bd7,ac2b37cd8343359d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:02:28.584 INFO 21062 --- [ocp-schedules-18,e05c80000d143e70,b1309ab9546f0af2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":389,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":838} 2024-11-04 15:02:28.584 INFO 21062 --- [ocp-schedules-18,e05c80000d143e70,b1309ab9546f0af2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:02:28.585 INFO 21062 --- [ForkJoinPool-11-worker-14,128768c178504bd7,2564d940668b643d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:02:28.585 INFO 21062 --- [ForkJoinPool-11-worker-14,128768c178504bd7,2564d940668b643d] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:02:28.585 INFO 21062 --- [ForkJoinPool-11-worker-14,128768c178504bd7,2564d940668b643d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:02:28.585 INFO 21062 --- [pool-12-thread-12,128768c178504bd7,dbac833922fe4d33] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:02:28.586 INFO 21062 --- [pool-12-thread-11,128768c178504bd7,a74da718c2b5d7e9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:02:28.586 INFO 21062 --- [pool-12-thread-11,128768c178504bd7,a74da718c2b5d7e9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:02:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:02:28.588876+08:00, lastProcessTime=2024-11-04T15:02:18.590538+08:00 2024-11-04 15:02:28.607 INFO 21062 --- [pool-12-thread-9,128768c178504bd7,2abc2f95ed0879b0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:02:28.615 INFO 21062 --- [pool-12-thread-9,128768c178504bd7,2abc2f95ed0879b0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:02:28.615 INFO 21062 --- [pool-12-thread-9,128768c178504bd7,2abc2f95ed0879b0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:02:28.618 INFO 21062 --- [ForkJoinPool-11-worker-14,128768c178504bd7,89e45e1bb27a3b26] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:02:28.618 INFO 21062 --- [pool-12-thread-10,128768c178504bd7,ac2b37cd8343359d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:02:28.701 INFO 21062 --- [ocp-async-4,d4899189ddeb4640,8e633dd27737fc9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:02:28.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:02:28.588876+08:00 2024-11-04 15:02:28.907 INFO 21062 --- [ocp-schedules-25,b52abc455c57ecfe,ffc59f23e6f88595] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:02:28.907 INFO 21062 --- [ocp-schedules-25,b52abc455c57ecfe,ffc59f23e6f88595] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:02:28.930 INFO 21062 --- [ocp-schedules-15,e861c2d586440fb3,b375c06a46cfd17d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:02:28.946 INFO 21062 --- [ocp-schedules-11,8d9bc937f27f6f5b,4dca2b4e2d34242a] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 15:02:28.946 INFO 21062 --- [ocp-schedules-11,8d9bc937f27f6f5b,4dca2b4e2d34242a] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 15:02:28.946 INFO 21062 --- [ocp-schedules-11,8d9bc937f27f6f5b,4dca2b4e2d34242a] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=422 2024-11-04 15:02:28.964 INFO 21062 --- [ocp-schedules-11,8d9bc937f27f6f5b,4dca2b4e2d34242a] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=422 2024-11-04 15:02:29.219 INFO 21062 --- [ocp-schedules-23,c21c89bd12753f29,75d10dbaeec5fbbe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 15:02:29.222 INFO 21062 --- [ocp-schedules-23,c21c89bd12753f29,75d10dbaeec5fbbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:29.260 ERROR 21062 --- [ocp-schedules-23,c21c89bd12753f29,75d10dbaeec5fbbe] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:02:29.260 INFO 21062 --- [ocp-schedules-23,c21c89bd12753f29,75d10dbaeec5fbbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 15:02:29.276 INFO 21062 --- [ocp-schedules-16,3089eabca165de9e,819e097a84f0bcef] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:02:29.276 INFO 21062 --- [ocp-schedules-16,3089eabca165de9e,819e097a84f0bcef] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:02:29.280 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:02:29.296 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:02:29.298 WARN 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:02:29.299 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:02:29.300 WARN 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:02:29.306 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:02:29.307 WARN 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:02:29.308 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:02:29.309 WARN 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:02:29.315 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:02:29.316 WARN 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:02:29.317 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:02:29.318 WARN 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:02:29.324 INFO 21062 --- [ocp-schedules-18,0a421e4c81752dc6,162c17c1024b2889] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:02:29.383 INFO 21062 --- [ocp-schedules-7,c697b44272ccb4e2,ea7098bf631b70cb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:02:29.410 WARN 21062 --- [ocp-async-5,27795146833f6771,00c6a14e3d240e5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:29.414 INFO 21062 --- [ocp-schedules-29,cdcdb3195d1e9171,a6d59d31b27aae72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:02:29.419 INFO 21062 --- [ocp-schedules-29,cdcdb3195d1e9171,a6d59d31b27aae72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:02:29.419 INFO 21062 --- [ocp-schedules-29,cdcdb3195d1e9171,a6d59d31b27aae72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:02:29.444 INFO 21062 --- [ocp-async-4,e48038c1b6c27cef,10ca7e6129ee7948] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:29.580 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:29.581 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:29.849 INFO 21062 --- [ocp-schedules-6,e14a247313f824b4,2fddc114eb11b722] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:02:29.849 INFO 21062 --- [ocp-schedules-6,e14a247313f824b4,2fddc114eb11b722] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:02:29.850 INFO 21062 --- [ocp-schedules-6,e14a247313f824b4,2fddc114eb11b722] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:02:29.850 INFO 21062 --- [ocp-schedules-6,e14a247313f824b4,2fddc114eb11b722] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:02:29.851 INFO 21062 --- [ocp-schedules-6,e14a247313f824b4,2fddc114eb11b722] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:02:29.851 INFO 21062 --- [ocp-schedules-6,e14a247313f824b4,2fddc114eb11b722] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:02:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7d0635ec310bac50,5e35015fd6226e76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:30.202 INFO 21062 --- [ocp-schedules-1,69ab2284b697eea1,747939acd1e1c4a5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:02:30.202 INFO 21062 --- [ocp-schedules-1,69ab2284b697eea1,747939acd1e1c4a5] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:02:30.230 INFO 21062 --- [pool-49-thread-1,69ab2284b697eea1,acde35d52e78a3cf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:02:30.256 WARN 21062 --- [ocp-schedules-1,69ab2284b697eea1,747939acd1e1c4a5] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:02:30.256 INFO 21062 --- [ocp-schedules-1,69ab2284b697eea1,747939acd1e1c4a5] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:02:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:02:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:02:31.215 INFO 21062 --- [ocp-schedules-25,810f7af7e9204b15,39ec1aa3f5d38770] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:02:31.410 WARN 21062 --- [ocp-async-6,7e900dbb90048a52,5a6e4b885c61d429] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:31.445 INFO 21062 --- [ocp-async-10,66627327c6c9522e,0c41210e8e7dd29a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:31.681 INFO 21062 --- [ocp-schedules-4,f389228fdeffda9a,e0f0ba7c069a59e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:02:32.464 INFO 21062 --- [ocp-schedules-24,075a3d128ed118e8,ae65578ec885915b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:02:32.470 INFO 21062 --- [ocp-schedules-24,075a3d128ed118e8,ae65578ec885915b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:02:32.470 INFO 21062 --- [ocp-schedules-24,075a3d128ed118e8,ae65578ec885915b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:02:32.581 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:32.582 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:32.635 INFO 21062 --- [ocp-schedules-25,dd3b719df7a67a42,2a4bf8ba4393006a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:02:32.635 INFO 21062 --- [ocp-schedules-25,dd3b719df7a67a42,2a4bf8ba4393006a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:02:32.662 INFO 21062 --- [ocp-schedules-25,dd3b719df7a67a42,2a4bf8ba4393006a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:02:32.662 INFO 21062 --- [ocp-schedules-25,dd3b719df7a67a42,2a4bf8ba4393006a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:02:32.663 INFO 21062 --- [ocp-schedules-13,35e3e2fe40ef6cb7,783b43495ab7e8af] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:02:32.858 INFO 21062 --- [ocp-schedules-21,3ff5ee50b9c28baf,ee6710dc6821d465] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:02:32.858+08:00, clusters=[1] 2024-11-04 15:02:32.859 INFO 21062 --- [ocp-schedules-21,3ff5ee50b9c28baf,ee6710dc6821d465] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:02:32.983 INFO 21062 --- [prometheus-minute-schedule-1,209eec06f8c8bfd4,976511974b5d1d3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:33.101 INFO 21062 --- [ocp-schedules-18,2922183198020a7a,19ab695b3badfb44] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:02:33.101+08:00, clusters=[1] 2024-11-04 15:02:33.105 INFO 21062 --- [ocp-schedules-18,2922183198020a7a,19ab695b3badfb44] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:02:33.120 INFO 21062 --- [ocp-schedules-26,d8a4b82d3f862348,0113188e65f5dadf] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:02:33.123 INFO 21062 --- [ocp-schedules-26,d8a4b82d3f862348,0113188e65f5dadf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:33.182 ERROR 21062 --- [ocp-schedules-26,d8a4b82d3f862348,0113188e65f5dadf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:02:33.182 INFO 21062 --- [ocp-schedules-26,d8a4b82d3f862348,0113188e65f5dadf] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:02:33.410 WARN 21062 --- [ocp-async-10,7c250738b488b5a4,d60a3ef3c84e39aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:33.445 INFO 21062 --- [ocp-async-3,1855070266e214f7,e80c112aa59464a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:34.614 INFO 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:02:34.614 INFO 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:02:34.614 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:02:34.614 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:02:34.624 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:02:34.624 INFO 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:02:34.624 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:02:34.625 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:02:34.625 WARN 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:02:34.626 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:02:34.626 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:02:34.626 INFO 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:02:34.626 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:02:34.627 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:02:34.627 WARN 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:02:34.627 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:02:34.633 INFO 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:02:34.633 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:02:34.633 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:02:34.643 ERROR 21062 --- [ocp-schedules-23,c96b37ce43292e2d,4ffe8c5b7941165f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:02:35.410 WARN 21062 --- [ocp-async-3,8ea5c89019b71291,42efff2bee8d1793] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:35.445 INFO 21062 --- [ocp-async-1,8caa3acdbb8b29df,3746594ff407460e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:35.582 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:35.583 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:35.983 INFO 21062 --- [prometheus-minute-schedule-1,77e890780b6775dc,ca9cef434e9907c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:37.411 WARN 21062 --- [ocp-async-4,ea6df7092d20ff12,f26b4d81c2855c7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:37.445 INFO 21062 --- [ocp-async-7,6c4a1145687969d6,e852dbc9590d73ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:38.504 INFO 21062 --- [ocp-schedules-28,6e499bccfbdf7aaa,5db341b7755099fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:02:38.504 INFO 21062 --- [ocp-schedules-14,9c5cb6df82bfe6d6,f89987296d6aed29] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:02:38.508 INFO 21062 --- [ocp-schedules-10,a32485146e3b5c86,0d73a009c77b8768] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:02:38.583 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:38.583 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:02:38.589708+08:00, lastProcessTime=2024-11-04T15:02:28.588876+08:00 2024-11-04 15:02:38.701 INFO 21062 --- [ocp-async-3,e4c9948577619698,c76308dcb9b6ce23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:02:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:02:38.589708+08:00 2024-11-04 15:02:38.806 INFO 21062 --- [ocp-schedules-12,2f2e309aa9086b85,6a06fce0b483b303] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:02:38.807 INFO 21062 --- [ocp-schedules-12,98ae946581760253,dc644d89f64e9a38] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:02:38.811 INFO 21062 --- [ocp-schedules-12,98ae946581760253,dc644d89f64e9a38] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:38.840 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:02:38.840 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:02:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0e5c4288bcfa4791,c3c89d8116910cc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:39.411 WARN 21062 --- [ocp-async-9,50dcc7749e70c54c,93e83a1ede7ff871] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:39.445 INFO 21062 --- [ocp-async-5,f0c982e36a558ef3,1cc65d6811e6a994] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:02:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:02:41.217 INFO 21062 --- [ocp-schedules-22,55573c4e1d8452e0,950b04599083b688] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:02:41.411 WARN 21062 --- [ocp-async-5,22fcb6a62af9c811,7b00bc44bf5f5c33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:41.446 INFO 21062 --- [ocp-async-1,980f8a866ad28959,ea26caff02b7dd59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:41.584 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:41.584 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:41.716 INFO 21062 --- [ocp-schedules-9,5f369d3db4744e3e,771d9e5b6f0385f5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:02:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:02:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:02:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:02:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:02:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:02:41.829 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:02:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 15:02:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a7364eeeed41d145,782eee5f73bf3d09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:42.384 INFO 21062 --- [ocp-schedules-1,70cff4fa81feebd2,0163bd02a1b4027b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:02:42.387 INFO 21062 --- [ocp-schedules-1,70cff4fa81feebd2,0163bd02a1b4027b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:42.472 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:02:42.474 ERROR 21062 --- [ocp-schedules-1,70cff4fa81feebd2,0163bd02a1b4027b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:02:42.474 INFO 21062 --- [ocp-schedules-1,70cff4fa81feebd2,0163bd02a1b4027b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:02:43.411 WARN 21062 --- [ocp-async-1,2eb4e893f3d21205,5786ef08ed54a4e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:43.446 INFO 21062 --- [ocp-async-10,437bb266532c63f7,8bbf72e2c5b6dfb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:44.585 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:44.585 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:44.983 INFO 21062 --- [prometheus-minute-schedule-1,91b8909034d53954,143701e4201345b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:45.411 WARN 21062 --- [ocp-async-10,915f35bbd9ea9186,49715466b7694734] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:45.446 INFO 21062 --- [ocp-async-3,0cb7a345de3ab32b,ff935dd713827552] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:47.411 WARN 21062 --- [ocp-async-3,2ac0b671f33e7b5f,a9ff4a4cd6e70206] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:47.446 INFO 21062 --- [ocp-async-4,76c91fe6a29d4653,947bfca4ecc3418e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:47.586 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:47.586 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:47.983 INFO 21062 --- [prometheus-minute-schedule-1,ae7902243ea1a142,965e5af2147557f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:02:48.590275+08:00, lastProcessTime=2024-11-04T15:02:38.589708+08:00 2024-11-04 15:02:48.702 INFO 21062 --- [ocp-async-7,9ba8389efa73eca4,f5ad80d88f06c23f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:02:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:02:48.590275+08:00 2024-11-04 15:02:49.218 INFO 21062 --- [ocp-schedules-8,4af727c3bee437df,1af7317027e8ae5b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:02:49.222 INFO 21062 --- [ocp-schedules-8,4af727c3bee437df,1af7317027e8ae5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:49.223 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:02:49.411 WARN 21062 --- [ocp-async-1,bdf791531927d57c,a5dc94e4a37a1817] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:49.446 INFO 21062 --- [ocp-async-10,7af6e69b3bbb18cc,0a6bb37c3b6bb1e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:50.587 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:50.588 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:50.983 INFO 21062 --- [prometheus-minute-schedule-1,50660630352486cb,5247f1171155eff3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:51.221 INFO 21062 --- [ocp-schedules-1,b4cc13f6beb85495,3634fc196cdf9963] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:02:51.412 WARN 21062 --- [ocp-async-9,88585c7014581db7,b71d930896808932] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:51.446 INFO 21062 --- [ocp-async-3,ecd2d8ad23665c42,beaf26ca050841e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:51.772 INFO 21062 --- [ocp-schedules-5,96600496f2c73a71,a1fac52bce76a6a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:02:52.472 INFO 21062 --- [ocp-schedules-18,5dd1e1cbe0920846,c1b4d66066315b2f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:02:52.478 INFO 21062 --- [ocp-schedules-18,5dd1e1cbe0920846,c1b4d66066315b2f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:02:52.478 INFO 21062 --- [ocp-schedules-18,5dd1e1cbe0920846,c1b4d66066315b2f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:02:53.412 WARN 21062 --- [ocp-async-5,b14386086594ad9d,d60f0481ab50c49e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:53.447 INFO 21062 --- [ocp-async-4,9d55c3477de1c101,8aa218cfe72aecce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:53.588 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:53.588 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c25ae1635498f102,12cde7eadee8f626] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:55.412 WARN 21062 --- [ocp-async-1,fa826069207cc47c,7b9b1bf010c08eeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:55.447 INFO 21062 --- [ocp-async-6,499d80497750e3c9,3089df68225f6eb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:56.589 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:56.590 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:56.983 INFO 21062 --- [prometheus-minute-schedule-1,174074ae067aac11,e2c6520b95fb359d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:02:57.412 WARN 21062 --- [ocp-async-9,4d68ac5d4f696d41,3c559c87ecb6c22b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:57.447 INFO 21062 --- [ocp-async-10,d7c1c70bcb255c67,15caddf9a222cb0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:58.500 INFO 21062 --- [ocp-schedules-20,1278dd27461f4708,9d256e9b20ed1f74] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:02:58.500 INFO 21062 --- [ocp-schedules-22,2be91f943188c1f1,cc2ca3272e0dfc43] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:02:58.502 INFO 21062 --- [ocp-schedules-13,283fa59a522e6843,5d2c1ba5da86953e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:02:58.503 INFO 21062 --- [ocp-schedules-20,1278dd27461f4708,9d256e9b20ed1f74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:02:58.504 INFO 21062 --- [ocp-schedules-22,2be91f943188c1f1,cc2ca3272e0dfc43] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:02:58.506 INFO 21062 --- [ocp-schedules-6,b6a40b3dba22eae5,ba07e8d310cdb534] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:02:58.512 INFO 21062 --- [ocp-schedules-21,527e103d3a72defd,7a6adb86419e4984] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:02:58.512 INFO 21062 --- [ocp-schedules-27,3bbf4b03583e0012,4a3137063b1d3cba] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:02:58.512 INFO 21062 --- [ocp-schedules-21,527e103d3a72defd,7a6adb86419e4984] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:02:58.512 INFO 21062 --- [ocp-schedules-27,3bbf4b03583e0012,4a3137063b1d3cba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:02:58.512 INFO 21062 --- [ocp-schedules-29,6cd4539c730be8b2,1cf01cdf663dc1ef] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:02:58.512 INFO 21062 --- [ocp-schedules-29,6cd4539c730be8b2,1cf01cdf663dc1ef] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:02:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:02:58.591006+08:00, lastProcessTime=2024-11-04T15:02:48.590275+08:00 2024-11-04 15:02:58.702 INFO 21062 --- [ocp-async-2,afe9f155ce90ff48,98d12bc56def8a7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:02:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:02:58.591006+08:00 2024-11-04 15:02:59.223 INFO 21062 --- [ocp-schedules-8,4af727c3bee437df,1af7317027e8ae5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:02:59.385 INFO 21062 --- [ocp-schedules-16,7f796705e88de3cb,839d3e26f3d61454] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:02:59.413 WARN 21062 --- [ocp-async-3,cb28eaec64f492fe,cfa5d297744c1257] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:02:59.447 INFO 21062 --- [ocp-async-2,e59a3e5636b09242,0748824125f4c88a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:02:59.590 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:02:59.591 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:02:59.983 INFO 21062 --- [prometheus-minute-schedule-1,c545914b2caf89a4,946513b49b8b92bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703780, interval=60 2024-11-04 15:03:01.223 INFO 21062 --- [ocp-schedules-14,2df8a9e1b9d808ef,b4785ba7d98337d8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:03:01.413 WARN 21062 --- [ocp-async-4,859e5d340d7853a9,dcffc5e43230e96e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:01.448 INFO 21062 --- [ocp-async-7,d80fadd2e7656150,6a0dcf41bff5c252] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:01.796 INFO 21062 --- [ocp-schedules-29,6ce3bae91e07b25b,758f57cb0bf39bfa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:03:02.591 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:02.592 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:02.983 INFO 21062 --- [prometheus-minute-schedule-1,82a9142cad895e55,a258198d7e1ec0b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:03.413 WARN 21062 --- [ocp-async-6,6abf3fdc030fe6a7,646e0cbd48d3a0ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:03.448 INFO 21062 --- [ocp-async-8,d1ef643fdd073e45,8aa45d95c57cf500] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:04.249 INFO 21062 --- [metric-archive-minute-52,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=21, affectRows=21 2024-11-04 15:03:04.249 INFO 21062 --- [metric-archive-minute-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=21, validCount=21 2024-11-04 15:03:04.261 INFO 21062 --- [metric-archive-minute-45,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=5, affectRows=5 2024-11-04 15:03:04.261 INFO 21062 --- [metric-archive-minute-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=5, validCount=5 2024-11-04 15:03:04.275 INFO 21062 --- [metric-archive-minute-55,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=18, affectRows=18 2024-11-04 15:03:04.275 INFO 21062 --- [metric-archive-minute-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=18, validCount=18 2024-11-04 15:03:04.276 INFO 21062 --- [metric-archive-minute-46,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=20, affectRows=20 2024-11-04 15:03:04.276 INFO 21062 --- [metric-archive-minute-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=20, validCount=20 2024-11-04 15:03:04.282 INFO 21062 --- [metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=13, affectRows=13 2024-11-04 15:03:04.282 INFO 21062 --- [metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=13, validCount=13 2024-11-04 15:03:04.293 INFO 21062 --- [metric-archive-minute-38,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=46, affectRows=46 2024-11-04 15:03:04.293 INFO 21062 --- [metric-archive-minute-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=46, validCount=46 2024-11-04 15:03:04.299 INFO 21062 --- [metric-archive-minute-54,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=79, affectRows=79 2024-11-04 15:03:04.299 INFO 21062 --- [metric-archive-minute-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=79, validCount=79 2024-11-04 15:03:04.320 INFO 21062 --- [metric-archive-minute-16,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=66, affectRows=66 2024-11-04 15:03:04.320 INFO 21062 --- [metric-archive-minute-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=66, validCount=66 2024-11-04 15:03:04.321 INFO 21062 --- [metric-archive-minute-13,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=62, affectRows=62 2024-11-04 15:03:04.321 INFO 21062 --- [metric-archive-minute-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=62, validCount=62 2024-11-04 15:03:04.326 INFO 21062 --- [metric-archive-minute-35,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=89, affectRows=89 2024-11-04 15:03:04.326 INFO 21062 --- [metric-archive-minute-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1730700000, nodeCount=89, validCount=89 2024-11-04 15:03:05.413 WARN 21062 --- [ocp-async-10,9ecc501d35d6bb98,132f59b804e9370f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:05.448 INFO 21062 --- [ocp-async-5,0b1784fc012f23f3,adbb357643e1b14e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:05.592 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:05.593 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:05.983 INFO 21062 --- [prometheus-minute-schedule-1,6f65ca9579cf738e,7ad77833c9d53a9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:07.413 WARN 21062 --- [ocp-async-4,9c387b666c6a6840,c519fea5a4e09c98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:07.448 INFO 21062 --- [ocp-async-2,bbad5a5b9bc51fa2,01d0fa2383daced8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:08.504 INFO 21062 --- [ocp-schedules-20,1278dd27461f4708,9d256e9b20ed1f74] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:03:08.504 INFO 21062 --- [ocp-schedules-22,2be91f943188c1f1,cc2ca3272e0dfc43] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:03:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:03:08.590353+08:00, lastProcessTime=2024-11-04T15:02:58.591006+08:00 2024-11-04 15:03:08.593 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:08.593 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:08.702 INFO 21062 --- [ocp-async-8,155847fe06a75865,4cf2c60c641e7e7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:03:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:03:08.590353+08:00 2024-11-04 15:03:08.840 INFO 21062 --- [ocp-schedules-12,98ae946581760253,dc644d89f64e9a38] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:03:08.841 INFO 21062 --- [ocp-schedules-12,1c0b2c12b3c91a58,61d0f36d5ff7d4bb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:03:08.844 INFO 21062 --- [ocp-schedules-12,1c0b2c12b3c91a58,61d0f36d5ff7d4bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:08.873 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:03:08.873 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:03:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d4ac67b05a2502fb,5e8a251dc9944986] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:09.413 WARN 21062 --- [ocp-async-9,db3c64b57fa0489e,e0c52eceae6c3009] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:09.448 INFO 21062 --- [ocp-async-8,e6a5f75d8c3e3cbe,9dabe1ab2ddb0976] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:03:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:03:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:03:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:03:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:03:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:03:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:03:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:03:11.227 INFO 21062 --- [ocp-schedules-29,9ad097afe0648603,f3f988eb422e0595] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:03:11.414 WARN 21062 --- [ocp-async-3,98a74d729617f658,38410801a6efd8e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:11.449 INFO 21062 --- [ocp-async-5,64adef9df564178a,ee06b6f380c87fc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:11.593 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:11.594 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:11.733 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=7, affectRows=7 2024-11-04 15:03:11.734 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=7, validCount=7 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=27, affectRows=27 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=24, affectRows=24 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=24, validCount=24 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=27, validCount=27 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=62, affectRows=62 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=68, affectRows=68 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=56, affectRows=56 2024-11-04 15:03:11.735 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=36, affectRows=36 2024-11-04 15:03:11.736 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=36, validCount=36 2024-11-04 15:03:11.736 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=43, affectRows=43 2024-11-04 15:03:11.736 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=56, validCount=56 2024-11-04 15:03:11.736 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=62, validCount=62 2024-11-04 15:03:11.736 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=68, validCount=68 2024-11-04 15:03:11.736 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=43, validCount=43 2024-11-04 15:03:11.737 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703720, nodeCount=99, affectRows=99 2024-11-04 15:03:11.737 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703720, nodeCount=96, validCount=96 2024-11-04 15:03:11.836 INFO 21062 --- [ocp-schedules-7,75a295fca3c75d09,eb49043874010f25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:03:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e9ba04fa3be45cf0,39fc2fd9eb8e20a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:12.480 INFO 21062 --- [ocp-schedules-20,73e6ef8517703f34,1edea80791390627] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:03:12.486 INFO 21062 --- [ocp-schedules-20,73e6ef8517703f34,1edea80791390627] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:03:12.486 INFO 21062 --- [ocp-schedules-20,73e6ef8517703f34,1edea80791390627] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:03:13.414 WARN 21062 --- [ocp-async-4,eea679a223577ec8,25ea47959dde00b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:13.449 INFO 21062 --- [ocp-async-2,81b60f84de9273bf,a89f9dea8336375d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:13.984 INFO 21062 --- [prometheus-cache-monitor1,1f6bfd6b917685d7,ea37a3fdde188c89] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:03:13.984 INFO 21062 --- [prometheus-cache-monitor1,1f6bfd6b917685d7,ea37a3fdde188c89] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:03:14.399 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T14:57:14.376, end=2024-11-04T15:02:14.397} 2024-11-04 15:03:14.400 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T14:57:14.376, end=2024-11-04T15:02:14.397} 2024-11-04 15:03:14.404 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:03:14.404 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:03:14.404 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:03:14.404 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:03:14.594 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:14.595 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:14.983 INFO 21062 --- [prometheus-minute-schedule-1,bde354bbd917a126,32405e595ec0ad0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:15.406 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T14:59:15.383, end=2024-11-04T15:01:15.403} 2024-11-04 15:03:15.410 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:03:15.410 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:03:15.414 WARN 21062 --- [ocp-async-7,b22c10e36b9cf010,8607c86cae5be3f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:15.449 INFO 21062 --- [ocp-async-6,bb360925ff59d08a,a9f0d3ee0dc32499] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:17.414 WARN 21062 --- [ocp-async-8,357ea33e116967a3,0ac3683bcdf224e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:17.449 INFO 21062 --- [ocp-async-10,b3ad27ac753567f1,3ba8339bf3c6cf07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:17.595 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:17.596 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:17.983 INFO 21062 --- [prometheus-minute-schedule-1,773c93f3ee3e793d,9a47c9e862de5011] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:03:18.591006+08:00, lastProcessTime=2024-11-04T15:03:08.590353+08:00 2024-11-04 15:03:18.702 INFO 21062 --- [ocp-async-1,9131830a6eb4aac5,77b563a80f806884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:03:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:03:18.591006+08:00 2024-11-04 15:03:19.414 WARN 21062 --- [ocp-async-4,59d5ec7894d97631,147dce8e6e5d31f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:19.449 INFO 21062 --- [ocp-async-2,ca0fd558ebbee5b0,3031f2c61600f777] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:20.596 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:20.598 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4e22f0d71650ffbf,8769e22ff7fa4d89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:21.229 INFO 21062 --- [ocp-schedules-15,f7e51c9d71927f28,c01b551c5dc67071] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:03:21.415 WARN 21062 --- [ocp-async-7,d7100d5e33d6005b,8b71866444e2f9fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:21.449 INFO 21062 --- [ocp-async-6,83d75ca09b581c80,9cf7f87deff6ee69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:21.862 INFO 21062 --- [ocp-schedules-8,3f3e51d3293b6ffa,31fa26ef10859474] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:03:23.415 WARN 21062 --- [ocp-async-6,148689070052607f,90bad4d2dfbf3ab4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:23.450 INFO 21062 --- [ocp-async-10,bba53d38e7500e05,a32cc25baea7bb57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:23.598 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:23.598 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:23.983 INFO 21062 --- [prometheus-minute-schedule-1,f98c6869423e4dac,09b62e6ca2002594] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:25.415 WARN 21062 --- [ocp-async-10,57f3e604a6c0dc6d,de58bc085b107494] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:25.450 INFO 21062 --- [ocp-async-5,4da13bff5edbe143,af5c47a9b7d3d0e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:26.599 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:26.599 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:26.983 INFO 21062 --- [prometheus-minute-schedule-1,cf1a8b8dfde2cef3,efe19fb77abcbfd1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c7f73832f4479ccf,048da3f4c747d3e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:03:27.415 WARN 21062 --- [ocp-async-7,93a3ea007db95267,f4c1bd3ab54475ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:27.450 INFO 21062 --- [ocp-async-2,bd26a2acdb3babcf,0296df2b7092ff4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:28.498 INFO 21062 --- [ocp-schedules-14,c928baa4d453c9a8,937317a8d0054527] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:03:28.499 INFO 21062 --- [ocp-schedules-5,3346d5ed075bed79,61105425e5c27a72] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:03:28.500 INFO 21062 --- [ocp-schedules-28,164734b1cbc0e1e0,57fdf967ed40112a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:03:28.500 INFO 21062 --- [ocp-schedules-11,721d309828418129,1a68393e28b12042] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:03:28.502 INFO 21062 --- [ocp-schedules-14,c928baa4d453c9a8,937317a8d0054527] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:03:28.502 INFO 21062 --- [ocp-schedules-25,82686177ff83356a,07baa8c26eb9704d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:03:28.504 INFO 21062 --- [ocp-schedules-28,164734b1cbc0e1e0,57fdf967ed40112a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:28.504 INFO 21062 --- [ocp-schedules-11,721d309828418129,1a68393e28b12042] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:03:28.505 INFO 21062 --- [ocp-schedules-27,31013b0ad143e5d1,88c70640ba1bba6b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:03:28.506 INFO 21062 --- [ocp-schedules-5,3346d5ed075bed79,61105425e5c27a72] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:03:28.510 INFO 21062 --- [ocp-schedules-1,1d6345deda7799c1,e4a7dbd37121c3b3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:03:28.510 INFO 21062 --- [ocp-schedules-24,19e4688812a4bfdf,194a7bb64b1d0850] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:03:28.510 INFO 21062 --- [ocp-schedules-8,b1cb359c465294ff,50e70f8af8ed05f6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:03:28.510 INFO 21062 --- [ocp-schedules-1,1d6345deda7799c1,e4a7dbd37121c3b3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:03:28.510 INFO 21062 --- [ocp-schedules-24,19e4688812a4bfdf,194a7bb64b1d0850] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:03:28.510 INFO 21062 --- [ocp-schedules-8,b1cb359c465294ff,50e70f8af8ed05f6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:03:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:03:28.588678+08:00, lastProcessTime=2024-11-04T15:03:18.591006+08:00 2024-11-04 15:03:28.597 INFO 21062 --- [ocp-schedules-21,8da9a551b1b96a87,a0e2848a7223f139] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 15:03:28.597 INFO 21062 --- [ocp-schedules-21,8da9a551b1b96a87,a0e2848a7223f139] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 15:03:28.652 INFO 21062 --- [ocp-schedules-30,135bf39c65633412,82bcf1402e670a39] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:03:28.653 INFO 21062 --- [ocp-schedules-30,135bf39c65633412,82bcf1402e670a39] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:03:28.702 INFO 21062 --- [ocp-async-4,d0b44f7b99ebe3b0,2785371317df6143] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:03:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:03:28.588678+08:00 2024-11-04 15:03:28.933 INFO 21062 --- [ocp-schedules-29,5b7c73a8ed8f2350,8f1bfbaab07d79f2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:03:29.279 INFO 21062 --- [ocp-schedules-5,e088a3b9309d62c4,c7786b942b1ab0f6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:03:29.279 INFO 21062 --- [ocp-schedules-5,e088a3b9309d62c4,c7786b942b1ab0f6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:03:29.388 INFO 21062 --- [ocp-schedules-3,8389b4f471cbd7e9,bfbfb7462064cf56] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:03:29.415 WARN 21062 --- [ocp-async-3,088d9c26737b8e98,7a11128a03afb337] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:29.419 INFO 21062 --- [ocp-schedules-21,f034ffa6bbb33fd9,c58447866b12e1b2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:03:29.424 INFO 21062 --- [ocp-schedules-21,f034ffa6bbb33fd9,c58447866b12e1b2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:03:29.424 INFO 21062 --- [ocp-schedules-21,f034ffa6bbb33fd9,c58447866b12e1b2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:03:29.450 INFO 21062 --- [ocp-async-1,a78aa3f7dc470948,766c1d4b4c085262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:29.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:29.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:29.852 INFO 21062 --- [ocp-schedules-18,f260154c0047bdb9,7e4b1fb491e502b1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:03:29.852 INFO 21062 --- [ocp-schedules-18,f260154c0047bdb9,7e4b1fb491e502b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:03:29.853 INFO 21062 --- [ocp-schedules-18,f260154c0047bdb9,7e4b1fb491e502b1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:03:29.853 INFO 21062 --- [ocp-schedules-18,f260154c0047bdb9,7e4b1fb491e502b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:03:29.854 INFO 21062 --- [ocp-schedules-18,f260154c0047bdb9,7e4b1fb491e502b1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:03:29.854 INFO 21062 --- [ocp-schedules-18,f260154c0047bdb9,7e4b1fb491e502b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:03:29.983 INFO 21062 --- [prometheus-minute-schedule-1,623e517fd98a0a30,e6faf3a3685be98b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:03:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:03:31.232 INFO 21062 --- [ocp-schedules-9,a273fcf4d0482809,6b514539272ded21] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:03:31.416 WARN 21062 --- [ocp-async-7,735a15d755f53127,99ba0bdf4bbea850] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:31.450 INFO 21062 --- [ocp-async-9,1a0964e4a3a93d40,8688578de79ac43f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:31.900 INFO 21062 --- [ocp-schedules-10,e32686684ce011b3,d659e39910038895] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:03:32.488 INFO 21062 --- [ocp-schedules-9,8b253804cb4b4105,1a5b67a9de90c8fe] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:03:32.493 INFO 21062 --- [ocp-schedules-9,8b253804cb4b4105,1a5b67a9de90c8fe] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:03:32.493 INFO 21062 --- [ocp-schedules-9,8b253804cb4b4105,1a5b67a9de90c8fe] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:03:32.600 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:32.601 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:32.662 INFO 21062 --- [ocp-schedules-18,66444ca8b4adf62f,f7780570b5560a59] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:03:32.662 INFO 21062 --- [ocp-schedules-18,66444ca8b4adf62f,f7780570b5560a59] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:03:32.689 INFO 21062 --- [ocp-schedules-18,66444ca8b4adf62f,f7780570b5560a59] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:03:32.689 INFO 21062 --- [ocp-schedules-18,66444ca8b4adf62f,f7780570b5560a59] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:03:32.690 INFO 21062 --- [ocp-schedules-15,77526942a4a8d967,401c9cc3e70ed1c4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:03:32.886 INFO 21062 --- [ocp-schedules-2,0aca333413a84586,9441388d688c1620] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:03:32.886+08:00, clusters=[1] 2024-11-04 15:03:32.888 INFO 21062 --- [ocp-schedules-2,0aca333413a84586,9441388d688c1620] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:03:32.983 INFO 21062 --- [prometheus-minute-schedule-1,fed5c64662d6bd63,11dd6db62fb621b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:33.131 INFO 21062 --- [ocp-schedules-16,22a95079d4ce6583,da53ab97a6196af9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:03:33.131+08:00, clusters=[1] 2024-11-04 15:03:33.134 INFO 21062 --- [ocp-schedules-16,22a95079d4ce6583,da53ab97a6196af9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:03:33.416 WARN 21062 --- [ocp-async-2,c7e0bfc75c3bc1bb,81d9183b9ca730e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:33.451 INFO 21062 --- [ocp-async-6,7623ccde98be1889,617e81c336dabbd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:34.652 INFO 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:03:34.652 INFO 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:03:34.652 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:03:34.652 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:03:34.663 INFO 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:03:34.664 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:03:34.664 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:03:34.665 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:03:34.665 WARN 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:03:34.665 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:03:34.666 INFO 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:03:34.666 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:03:34.666 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:03:34.667 WARN 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:03:34.667 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:03:34.667 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:03:34.674 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:03:34.674 INFO 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:03:34.674 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:03:34.691 ERROR 21062 --- [ocp-schedules-15,5a85336fc06b0021,08ba396622b744b7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:03:35.416 WARN 21062 --- [ocp-async-10,23fb1d08191ff238,8345b029720f7e47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:35.451 INFO 21062 --- [ocp-async-4,f053e31b091190d2,f0765b6759474b76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:35.601 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:35.602 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:35.983 INFO 21062 --- [prometheus-minute-schedule-1,1c65ab289fd6606b,bbcdc913889b2c94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:37.416 WARN 21062 --- [ocp-async-4,3cbb239885cbda13,d533aec3f1c57b08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:37.451 INFO 21062 --- [ocp-async-5,fe6bacd2c037908b,7edfdc715c7ac444] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:38.505 INFO 21062 --- [ocp-schedules-11,721d309828418129,1a68393e28b12042] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:03:38.505 INFO 21062 --- [ocp-schedules-28,164734b1cbc0e1e0,57fdf967ed40112a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:03:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:03:38.589716+08:00, lastProcessTime=2024-11-04T15:03:28.588678+08:00 2024-11-04 15:03:38.602 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:38.603 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:38.703 INFO 21062 --- [ocp-async-10,735651808f2aa872,b9aa310d134ef8c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:03:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:03:38.589716+08:00 2024-11-04 15:03:38.873 INFO 21062 --- [ocp-schedules-12,1c0b2c12b3c91a58,61d0f36d5ff7d4bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:03:38.873 INFO 21062 --- [ocp-schedules-12,1e1e001a2569d26d,17832a45a8cdea4f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:03:38.877 INFO 21062 --- [ocp-schedules-12,1e1e001a2569d26d,17832a45a8cdea4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:38.905 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:03:38.905 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:03:38.983 INFO 21062 --- [prometheus-minute-schedule-1,676a6c6e8bc94de7,dc00a8dbb481987c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:39.416 WARN 21062 --- [ocp-async-2,c634b88097fcfdc1,6379b4d63f949189] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:39.451 INFO 21062 --- [ocp-async-10,e3d6b668bdd80ed2,e2563e690b4ba783] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:03:40.004 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:03:41.235 INFO 21062 --- [ocp-schedules-28,029ad0ec54b4dc37,645752beb46889cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:03:41.416 WARN 21062 --- [ocp-async-3,7ea31a142d67b223,340bdac56e9a6435] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:41.451 INFO 21062 --- [ocp-async-4,33d6bd16cb3f6861,2dfb17fbd268cec5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:41.603 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:41.604 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:41.928 INFO 21062 --- [ocp-schedules-10,664bafd22676db92,8dbd97ba2ad6c2c8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:03:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2145f559c93ad85c,bf6f4e0eac072a22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:42.475 INFO 21062 --- [ocp-schedules-2,3783471d92d0c28f,216b73606130f69e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:03:42.478 INFO 21062 --- [ocp-schedules-2,3783471d92d0c28f,216b73606130f69e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:42.562 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:03:42.565 ERROR 21062 --- [ocp-schedules-2,3783471d92d0c28f,216b73606130f69e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:03:42.565 INFO 21062 --- [ocp-schedules-2,3783471d92d0c28f,216b73606130f69e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:03:43.417 WARN 21062 --- [ocp-async-4,7abfa110a37e4482,9a4a52ce9f81f74c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:43.452 INFO 21062 --- [ocp-async-5,58607ff1d4cfcec6,81372646b440a5d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:44.604 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:44.605 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:44.983 INFO 21062 --- [prometheus-minute-schedule-1,1cd308fdf7747486,3a8e0ddab30b63ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:45.417 WARN 21062 --- [ocp-async-5,29e597adcfb90b33,926c065617301ff9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:45.452 INFO 21062 --- [ocp-async-2,aba653fff4a62627,024cd1dd9fc02e7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:47.417 WARN 21062 --- [ocp-async-2,2d83726ef413eced,2b6fbfb484f52451] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:47.452 INFO 21062 --- [ocp-async-10,299e62fbe0094bc6,7d260604413aba4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:47.605 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:47.606 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b12f74a541bf0ac1,e993b31d22419cad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:03:48.591072+08:00, lastProcessTime=2024-11-04T15:03:38.589716+08:00 2024-11-04 15:03:48.703 INFO 21062 --- [ocp-async-9,abc054a306d7ab3d,c72d75aa802643af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:03:48.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:03:48.591072+08:00 2024-11-04 15:03:49.417 WARN 21062 --- [ocp-async-1,36cc5f7b91672c9d,817c30a4c0676a4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:49.452 INFO 21062 --- [ocp-async-9,21ed46eba9907f6d,e5f9b4457fb00d95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:50.606 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:50.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:50.983 INFO 21062 --- [prometheus-minute-schedule-1,788afab8f1503da6,fe65fb303205299c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:51.240 INFO 21062 --- [ocp-schedules-6,057bc21fd0c9ea56,a265168efaba3150] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:03:51.418 WARN 21062 --- [ocp-async-8,35b2ebb7da4aaa13,f7332b639f91bd08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:51.452 INFO 21062 --- [ocp-async-6,843c0826e940fbfa,685500ff941afb65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:51.971 INFO 21062 --- [ocp-schedules-30,f57253b9da508230,9538bab7139e5b51] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:03:52.496 INFO 21062 --- [ocp-schedules-17,bc159d534e1dd998,b78a84950f14d4fa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:03:52.501 INFO 21062 --- [ocp-schedules-17,bc159d534e1dd998,b78a84950f14d4fa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:03:52.501 INFO 21062 --- [ocp-schedules-17,bc159d534e1dd998,b78a84950f14d4fa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:03:53.418 WARN 21062 --- [ocp-async-3,8a352509acbead84,bf35e4d887b8d186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:53.453 INFO 21062 --- [ocp-async-10,45ccf15db5f06d77,d486bfae25361e30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:53.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:53.608 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6b366fb37fdec3da,ab575e67ed8a72d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:55.418 WARN 21062 --- [ocp-async-1,d998f3f19571a811,12c2acc2ffecf682] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:55.453 INFO 21062 --- [ocp-async-4,7282154a06e19529,fc76e1e58abe7376] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:56.609 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:56.610 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:56.983 INFO 21062 --- [prometheus-minute-schedule-1,77b490966a1e7f57,06eca587f63efc98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:03:57.418 WARN 21062 --- [ocp-async-8,17b1ead2ce78f702,b3dd42952c9754fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:57.453 INFO 21062 --- [ocp-async-5,7f137be45dca2b7a,7b56fe20ddca3378] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:58.500 INFO 21062 --- [ocp-schedules-29,5d3696e588666d4f,1f032f46b9c2b9c7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:03:58.500 INFO 21062 --- [ocp-schedules-27,dadea3a7171d1bfe,7b73ac02de448b51] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:03:58.502 INFO 21062 --- [ocp-schedules-2,a92f580ab44080fb,1f7aca61bd5e0176] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:03:58.504 INFO 21062 --- [ocp-schedules-29,5d3696e588666d4f,1f032f46b9c2b9c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:58.504 INFO 21062 --- [ocp-schedules-27,dadea3a7171d1bfe,7b73ac02de448b51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:03:58.505 INFO 21062 --- [ocp-schedules-6,b2d0144222b2504a,e6f32b3635a15a8a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:03:58.510 INFO 21062 --- [ocp-schedules-28,fb9ed998a882e2b6,dae66a8279ef0266] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:03:58.511 INFO 21062 --- [ocp-schedules-3,a603954760cb58c2,c952b26220bc5941] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:03:58.511 INFO 21062 --- [ocp-schedules-28,fb9ed998a882e2b6,dae66a8279ef0266] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:03:58.511 INFO 21062 --- [ocp-schedules-3,a603954760cb58c2,c952b26220bc5941] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:03:58.511 INFO 21062 --- [ocp-schedules-7,c3cb87670263441c,0dc478166fcc03a9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:03:58.511 INFO 21062 --- [ocp-schedules-7,c3cb87670263441c,0dc478166fcc03a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:03:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:03:58.590576+08:00, lastProcessTime=2024-11-04T15:03:48.591072+08:00 2024-11-04 15:03:58.703 INFO 21062 --- [ocp-async-10,2334376329808dc8,ab962f1b8fc0506d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:03:58.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:03:58.590576+08:00 2024-11-04 15:03:59.223 INFO 21062 --- [ocp-schedules-4,0a1fb162c56d01d6,73df79bd822a9b79] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:03:59.227 INFO 21062 --- [ocp-schedules-4,0a1fb162c56d01d6,73df79bd822a9b79] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:03:59.227 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:03:59.391 INFO 21062 --- [ocp-schedules-10,8b86d688ee5779f9,69df6fced61bbb68] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:03:59.418 WARN 21062 --- [ocp-async-2,57d3cea228cbf2f6,ccb01943a73246cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:03:59.453 INFO 21062 --- [ocp-async-10,e1833ce14f1a7389,30446f59062b0436] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:03:59.610 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:03:59.611 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:03:59.983 INFO 21062 --- [prometheus-minute-schedule-1,5b5dcf87ed14201c,8be6ee473e3b3d78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703840, interval=60 2024-11-04 15:04:01.244 INFO 21062 --- [ocp-schedules-15,bb59f3bd1ca7f7b8,a539f8dbe0591866] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:04:01.419 WARN 21062 --- [ocp-async-1,f8c8adcb9500678a,bdb224db8b106075] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:01.453 INFO 21062 --- [ocp-async-4,566bf010dba2224f,64b2e149ec78d76f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:01.995 INFO 21062 --- [ocp-schedules-5,5d3e3052c241ef5d,2f92ded652c2507a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:04:02.611 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:02.614 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:02.983 INFO 21062 --- [prometheus-minute-schedule-1,ba4dc4518cca165f,78c60d2068298822] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:03.419 WARN 21062 --- [ocp-async-8,92300ddac79e8eaa,2c06df6a2d3d843e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:03.453 INFO 21062 --- [ocp-async-5,c6233f64cb5bdb61,1985e64f7d9acbce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:05.419 WARN 21062 --- [ocp-async-6,79d9a778f18431d7,aca7d4b086f33023] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:05.454 INFO 21062 --- [ocp-async-3,0473d8c554fea7b5,6ebbc514076319ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:05.614 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:05.615 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0ef056df3fe58bf0,6b536eb90e68f586] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:07.419 WARN 21062 --- [ocp-async-10,f5d7776b2d2213e5,98bda1ebf6d3a2af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:07.454 INFO 21062 --- [ocp-async-7,3c8f80266bda01ed,bc6af80f66642a3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:08.505 INFO 21062 --- [ocp-schedules-27,dadea3a7171d1bfe,7b73ac02de448b51] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:04:08.505 INFO 21062 --- [ocp-schedules-29,5d3696e588666d4f,1f032f46b9c2b9c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:04:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:04:08.590692+08:00, lastProcessTime=2024-11-04T15:03:58.590576+08:00 2024-11-04 15:04:08.615 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:08.616 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:08.703 INFO 21062 --- [ocp-async-5,5b87ccba7fc81b19,a9efffba442243e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:04:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:04:08.590692+08:00 2024-11-04 15:04:08.906 INFO 21062 --- [ocp-schedules-12,1e1e001a2569d26d,17832a45a8cdea4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:04:08.906 INFO 21062 --- [ocp-schedules-12,01ed7456c094c59c,4717fbbf620a8150] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:04:08.909 INFO 21062 --- [ocp-schedules-12,01ed7456c094c59c,4717fbbf620a8150] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:08.938 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:04:08.938 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:04:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f4dbbd4fb6f703ac,2ccc20cbdd67d25d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:09.141 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:04:09.141 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:04:09.150 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:04:09.150 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:04:09.150 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:04:09.151 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:04:09.156 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:04:09.156 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:04:09.156 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:04:09.156 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:04:09.192 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:04:09.201 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:04:09.201 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:04:09.201 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:04:09.201 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:04:09.201 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:04:09.209 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:04:09.227 INFO 21062 --- [ocp-schedules-4,0a1fb162c56d01d6,73df79bd822a9b79] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:04:09.419 WARN 21062 --- [ocp-async-8,20b3a879f44fba46,68d5d6f9858dfe1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:09.454 INFO 21062 --- [ocp-async-5,cf57d93fdeaaf76a,f270c3a4f746e2dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:09.765 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:04:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:04:10.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:04:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:04:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:04:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:04:10.321 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.164869324S 2024-11-04 15:04:10.321 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:04:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:04:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:04:10.327 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:04:10.422 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:04:10.422 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:04:10.447 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:04:10.509 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:04:10.509 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:04:10.510 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:04:10.510 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:04:10.510 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:04:10.510 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:04:10.520 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:04:10.693 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] o.s.boot.SpringApplication : Started application in 1.543 seconds (JVM running for 9264.372) 2024-11-04 15:04:10.693 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.543561224S 2024-11-04 15:04:11.155 INFO 21062 --- [ocp-schedules-30,79f6195ae2a36a26,96d1ca1c509cb26e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:04:11.246 INFO 21062 --- [ocp-schedules-22,162dc2f72f8ae06c,c5b26c89a494dbd7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:04:11.419 WARN 21062 --- [ocp-async-6,829cd483b7f08dd2,3c44857356067883] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:11.454 INFO 21062 --- [ocp-async-3,838224f3707d2816,59542b5a3f2cf988] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:11.617 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:11.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:11.737 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=7, affectRows=7 2024-11-04 15:04:11.737 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=7, validCount=7 2024-11-04 15:04:11.738 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=41, affectRows=41 2024-11-04 15:04:11.738 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=25, affectRows=25 2024-11-04 15:04:11.738 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=58, affectRows=58 2024-11-04 15:04:11.738 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=17, affectRows=17 2024-11-04 15:04:11.738 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=17, validCount=17 2024-11-04 15:04:11.738 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=25, validCount=25 2024-11-04 15:04:11.738 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=41, validCount=41 2024-11-04 15:04:11.739 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=58, validCount=58 2024-11-04 15:04:11.739 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=62, affectRows=62 2024-11-04 15:04:11.739 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=45, affectRows=45 2024-11-04 15:04:11.739 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=62, validCount=62 2024-11-04 15:04:11.739 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=44, validCount=44 2024-11-04 15:04:11.739 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=78, affectRows=78 2024-11-04 15:04:11.739 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=78, validCount=78 2024-11-04 15:04:11.740 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703780, nodeCount=89, affectRows=89 2024-11-04 15:04:11.740 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703780, nodeCount=87, validCount=87 2024-11-04 15:04:11.983 INFO 21062 --- [prometheus-minute-schedule-1,00a11be31f2e26ec,95e388bc6cc11622] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:12.041 INFO 21062 --- [ocp-schedules-1,bc979a4eb28f2f8f,91d5d53c3abb5f82] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:04:12.503 INFO 21062 --- [ocp-schedules-5,e052a5003d142f4e,3098a808cb21718a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:04:12.511 INFO 21062 --- [ocp-schedules-5,e052a5003d142f4e,3098a808cb21718a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:04:12.511 INFO 21062 --- [ocp-schedules-5,e052a5003d142f4e,3098a808cb21718a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:04:13.420 WARN 21062 --- [ocp-async-10,1c7995e99eeb35bd,285126a4164f7997] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:13.454 INFO 21062 --- [ocp-async-7,0e93c9cf91ae6c9b,01cfe390bfe4d0b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:13.984 INFO 21062 --- [prometheus-cache-monitor1,f8bbb71e1c36443b,e32e2571bf8fa8ab] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:04:13.984 INFO 21062 --- [prometheus-cache-monitor1,f8bbb71e1c36443b,e32e2571bf8fa8ab] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:04:14.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:14.619 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:14.983 INFO 21062 --- [prometheus-minute-schedule-1,7aa387e98ba15c5a,29502e0726e64f98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:15.420 WARN 21062 --- [ocp-async-8,2b66cd880c66c5ab,e839e2c3fbffc9c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:15.455 INFO 21062 --- [ocp-async-9,e9f609f074b7ab26,441e127eabe40825] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:17.420 WARN 21062 --- [ocp-async-5,3849b43c9576d30d,b9a0193d1ed53c15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:17.455 INFO 21062 --- [ocp-async-2,ee398ac39f63382b,c147612db0e9886f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:17.619 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:17.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:17.983 INFO 21062 --- [prometheus-minute-schedule-1,6333190bc26065ea,9bbabfd45394484e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:04:18.590107+08:00, lastProcessTime=2024-11-04T15:04:08.590692+08:00 2024-11-04 15:04:18.738 INFO 21062 --- [ocp-async-1,19f442ea89f254c7,be453c07cfa54de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:04:18.798 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:04:18.590107+08:00 2024-11-04 15:04:19.420 WARN 21062 --- [ocp-async-3,3f3ac49a7e68e88f,4269cfcbcef45896] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:19.455 INFO 21062 --- [ocp-async-1,b8e4a9bf89442535,8a4c73dfd15ec07e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:20.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:20.621 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f7716a70e9a6b0e3,0d5a6ba5bbb8aa26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:21.249 INFO 21062 --- [ocp-schedules-29,29541fd16872eb19,dc416dcb5b11864b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:04:21.421 WARN 21062 --- [ocp-async-1,fbb9f51ffa06d48f,b5cd2d51392a4987] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:21.455 INFO 21062 --- [ocp-async-9,18b96637ac911ac0,fb93a0f9ff68dd63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:22.067 INFO 21062 --- [ocp-schedules-8,70535e47b6aede16,aafd234f5aca9c66] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:04:23.421 WARN 21062 --- [ocp-async-9,548686a1935d2268,806ef376d4f75bc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:23.455 INFO 21062 --- [ocp-async-2,9e071c6bc95a6e9d,20ed7c46072c860e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:23.621 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:23.622 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:23.983 INFO 21062 --- [prometheus-minute-schedule-1,f0bbebf6d12a79b7,ae44c51a11f3f1b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:25.421 WARN 21062 --- [ocp-async-2,94604ac93b11e880,5a0aac46c621afe2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:25.455 INFO 21062 --- [ocp-async-7,2de1dd04e16d9449,4188a55616d0ec2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:26.622 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:26.623 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:26.983 INFO 21062 --- [prometheus-minute-schedule-1,0ffffe345270c496,507146739e0261d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:26.984 INFO 21062 --- [prometheus-hour-schedule-1,9caf590c56e749ca,9e5aff9ecdfa684c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:04:27.421 WARN 21062 --- [ocp-async-7,af7fe59fada6db53,c20fe73548a12266] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:27.455 INFO 21062 --- [ocp-async-4,077fd15f56b7fff8,5bf4024b62a75f8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:28.498 INFO 21062 --- [ocp-schedules-3,a05b62e78ce6025e,685ecb0e15786e97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:04:28.499 INFO 21062 --- [ocp-schedules-13,13dbcf1739fc2777,7dec95d6461aefd1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:04:28.499 INFO 21062 --- [ocp-schedules-25,75a9a93b51d1316c,b0dc21ee51982ffd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:04:28.500 INFO 21062 --- [ocp-schedules-14,4d78664b48b8ac47,cc907db77eff0c57] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:04:28.501 INFO 21062 --- [ocp-schedules-3,a05b62e78ce6025e,685ecb0e15786e97] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:04:28.502 INFO 21062 --- [ocp-schedules-2,1d31d207611e1fb2,c01fe2fbaf402a2d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:04:28.503 INFO 21062 --- [ocp-schedules-25,75a9a93b51d1316c,b0dc21ee51982ffd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:28.504 INFO 21062 --- [ocp-schedules-14,4d78664b48b8ac47,cc907db77eff0c57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:04:28.504 INFO 21062 --- [ocp-schedules-17,b7cb1580d177e73e,88a88336551a71f9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:04:28.505 INFO 21062 --- [ocp-schedules-13,13dbcf1739fc2777,7dec95d6461aefd1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:04:28.509 INFO 21062 --- [ocp-schedules-29,dd418af3752be816,b012072db5e9d0ad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:04:28.509 INFO 21062 --- [ocp-schedules-29,dd418af3752be816,b012072db5e9d0ad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:04:28.509 INFO 21062 --- [ocp-schedules-19,af6bb4e25a588b03,a8651b779335e8e2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:04:28.509 INFO 21062 --- [ocp-schedules-19,af6bb4e25a588b03,a8651b779335e8e2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:04:28.509 INFO 21062 --- [ocp-schedules-8,63e2d53c3f4aa500,982d7ba3fcfe153d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:04:28.509 INFO 21062 --- [ocp-schedules-8,63e2d53c3f4aa500,982d7ba3fcfe153d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:04:28.584 INFO 21062 --- [ocp-schedules-26,a3ebae909c3b1479,a089b69adf6086d7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:04:28.584 INFO 21062 --- [ocp-schedules-26,a3ebae909c3b1479,a089b69adf6086d7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:04:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:04:28.589295+08:00, lastProcessTime=2024-11-04T15:04:18.590107+08:00 2024-11-04 15:04:28.610 INFO 21062 --- [ocp-schedules-9,0d84d9f89488d7cb,7d20a42368a094dc] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:04:28.610+08:00, tenantSize=0 2024-11-04 15:04:28.704 INFO 21062 --- [ocp-schedules-10,e1c5fbe943536f99,33b85179d8fc4001] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 15:04:28.705 INFO 21062 --- [ocp-schedules-10,e1c5fbe943536f99,33b85179d8fc4001] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 15:04:28.739 INFO 21062 --- [ocp-async-2,663ebf222e0535a5,0b14de7f6ce9c1a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:04:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:04:28.589295+08:00 2024-11-04 15:04:28.935 INFO 21062 --- [ocp-schedules-5,bd71398608106c44,22a8309984deda1e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:04:29.284 INFO 21062 --- [ocp-schedules-22,6c50d1af1f342412,2affc393991dfa0f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:04:29.284 INFO 21062 --- [ocp-schedules-22,6c50d1af1f342412,2affc393991dfa0f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:04:29.394 INFO 21062 --- [ocp-schedules-9,ef9d25f810c47874,0cc6dd87f7cb5bc7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:04:29.421 WARN 21062 --- [ocp-async-3,1d86061d1b4d447f,ca8f7968dfe7f4ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:29.424 INFO 21062 --- [ocp-schedules-20,ec0f192bd94c67d6,381fbc3d930a5b4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:04:29.429 INFO 21062 --- [ocp-schedules-20,ec0f192bd94c67d6,381fbc3d930a5b4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:04:29.429 INFO 21062 --- [ocp-schedules-20,ec0f192bd94c67d6,381fbc3d930a5b4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:04:29.456 INFO 21062 --- [ocp-async-2,94a5f7b78e708105,fb41679285350ac6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:29.623 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:29.624 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:29.668 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:04:29.668 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:04:29.677 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:04:29.677 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:04:29.677 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:04:29.686 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:04:29.686 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:04:29.686 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:04:29.690 INFO 21062 --- [ocp-schedules-6,240c90f4a8ccbcde,e8ff886d072bac98] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:04:29.855 INFO 21062 --- [ocp-schedules-1,6c18a250884ec35a,375e3f44d57c72e2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:04:29.855 INFO 21062 --- [ocp-schedules-1,6c18a250884ec35a,375e3f44d57c72e2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:04:29.860 INFO 21062 --- [ocp-schedules-1,6c18a250884ec35a,375e3f44d57c72e2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:04:29.860 INFO 21062 --- [ocp-schedules-1,6c18a250884ec35a,375e3f44d57c72e2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:04:29.866 INFO 21062 --- [ocp-schedules-1,6c18a250884ec35a,375e3f44d57c72e2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:04:29.866 INFO 21062 --- [ocp-schedules-1,6c18a250884ec35a,375e3f44d57c72e2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:04:29.983 INFO 21062 --- [prometheus-minute-schedule-1,975902caf4618477,341ef0f50b2720c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:04:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:04:31.252 INFO 21062 --- [ocp-schedules-7,a95eaf3742bebb53,72cdea9700f3c35c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:04:31.421 WARN 21062 --- [ocp-async-7,61ed8e482814c7fb,cff735969ea3974f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:31.456 INFO 21062 --- [ocp-async-1,c44431bbbca3cdb6,63edd4e3f9038982] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:32.107 INFO 21062 --- [ocp-schedules-30,4d7239cc890a5a13,95b05d7d52c7556a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:04:32.513 INFO 21062 --- [ocp-schedules-24,7a5d4ffa111d63ef,fd1faa10ae355015] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:04:32.518 INFO 21062 --- [ocp-schedules-24,7a5d4ffa111d63ef,fd1faa10ae355015] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:04:32.518 INFO 21062 --- [ocp-schedules-24,7a5d4ffa111d63ef,fd1faa10ae355015] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:04:32.624 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:32.625 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:32.690 INFO 21062 --- [ocp-schedules-5,b51b902358d8e74e,e18b720ed4a424fa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:04:32.690 INFO 21062 --- [ocp-schedules-5,b51b902358d8e74e,e18b720ed4a424fa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:04:32.717 INFO 21062 --- [ocp-schedules-5,b51b902358d8e74e,e18b720ed4a424fa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:04:32.717 INFO 21062 --- [ocp-schedules-5,b51b902358d8e74e,e18b720ed4a424fa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:04:32.718 INFO 21062 --- [ocp-schedules-29,cfbd211b02410095,564d6f3e27c1b3e4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:04:32.914 INFO 21062 --- [ocp-schedules-21,f2fa0a0a745342d6,f86a4fa983e77225] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:04:32.914+08:00, clusters=[1] 2024-11-04 15:04:32.915 INFO 21062 --- [ocp-schedules-21,f2fa0a0a745342d6,f86a4fa983e77225] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:04:32.983 INFO 21062 --- [prometheus-minute-schedule-1,614814319f46af61,5003f1771293f40b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:33.161 INFO 21062 --- [ocp-schedules-26,50f41e3204288da1,0f8df0d33b65ed28] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:04:33.161+08:00, clusters=[1] 2024-11-04 15:04:33.165 INFO 21062 --- [ocp-schedules-26,50f41e3204288da1,0f8df0d33b65ed28] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:04:33.182 INFO 21062 --- [ocp-schedules-4,b43b2ce0675f7ada,c6367467455b945d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:04:33.186 INFO 21062 --- [ocp-schedules-4,b43b2ce0675f7ada,c6367467455b945d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:33.246 ERROR 21062 --- [ocp-schedules-4,b43b2ce0675f7ada,c6367467455b945d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:04:33.246 INFO 21062 --- [ocp-schedules-4,b43b2ce0675f7ada,c6367467455b945d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:04:33.422 WARN 21062 --- [ocp-async-4,9b015979c582d741,1e2ddb7a04ec519a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:33.456 INFO 21062 --- [ocp-async-5,bc063eb1c0b51b8a,801d45d3897a7721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:34.700 INFO 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:04:34.700 INFO 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:04:34.700 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:04:34.700 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:04:34.711 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:04:34.711 INFO 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:04:34.711 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:04:34.713 WARN 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:04:34.713 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:04:34.713 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:04:34.714 INFO 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:04:34.714 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:04:34.714 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:04:34.715 WARN 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:04:34.715 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:04:34.716 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:04:34.723 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:04:34.723 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:04:34.723 INFO 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:04:34.735 ERROR 21062 --- [ocp-schedules-16,2e2119066475224a,149c8e464caf956d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:04:35.422 WARN 21062 --- [ocp-async-3,6aafcbb8092f10ac,95243c6ee760ed54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:35.456 INFO 21062 --- [ocp-async-6,1fff298a68dfb23d,5a257b8059bd3862] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:35.625 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:35.626 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0958f252c411f3cc,03ddcabb7c23aca4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:37.422 WARN 21062 --- [ocp-async-7,508c609522b8602b,9364435629f7ea5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:37.456 INFO 21062 --- [ocp-async-2,2c779482faf5f8dc,41ef5d01020413d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:38.504 INFO 21062 --- [ocp-schedules-25,75a9a93b51d1316c,b0dc21ee51982ffd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:04:38.504 INFO 21062 --- [ocp-schedules-14,4d78664b48b8ac47,cc907db77eff0c57] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:04:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:04:38.589467+08:00, lastProcessTime=2024-11-04T15:04:28.589295+08:00 2024-11-04 15:04:38.626 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:38.627 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:38.739 INFO 21062 --- [ocp-async-3,5aee51c415083a5f,d523502b233be69f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:04:38.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:04:38.589467+08:00 2024-11-04 15:04:38.938 INFO 21062 --- [ocp-schedules-12,01ed7456c094c59c,4717fbbf620a8150] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:04:38.938 INFO 21062 --- [ocp-schedules-12,c1917597db75247e,6ce4930985953ffd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:04:38.942 INFO 21062 --- [ocp-schedules-12,c1917597db75247e,6ce4930985953ffd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:38.969 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:04:38.969 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:04:38.983 INFO 21062 --- [prometheus-minute-schedule-1,7dcea2ce24b881ff,885d1092b43e8baa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:39.422 WARN 21062 --- [ocp-async-9,48c233d9ce0faeb3,992c732e27da16d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:39.457 INFO 21062 --- [ocp-async-3,74dc10018d872477,3aa862a6ddc4889d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:04:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:04:41.254 INFO 21062 --- [ocp-schedules-29,7a99058c6c1b4d15,a7d476de6dd984b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:04:41.422 WARN 21062 --- [ocp-async-10,73df6374581c826d,a7774a5c0d539457] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:41.457 INFO 21062 --- [ocp-async-7,298f42befe71a7dc,b1cdeb2cc5db6653] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:41.627 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:41.628 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:04:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:04:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:04:41.855 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:04:41.855 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:04:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c9bf57287501b763,2c0f470d1d65354e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:42.133 INFO 21062 --- [ocp-schedules-1,5eef9ae93930db4a,565ee079c50afbac] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:04:42.565 INFO 21062 --- [ocp-schedules-11,bec4fb120809ded9,d6254e71238c481a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:04:42.569 INFO 21062 --- [ocp-schedules-11,bec4fb120809ded9,d6254e71238c481a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:42.652 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:04:42.654 ERROR 21062 --- [ocp-schedules-11,bec4fb120809ded9,d6254e71238c481a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:04:42.654 INFO 21062 --- [ocp-schedules-11,bec4fb120809ded9,d6254e71238c481a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:04:43.422 WARN 21062 --- [ocp-async-6,98b391dba8bc746a,c7e76bd46c9d8be8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:43.457 INFO 21062 --- [ocp-async-8,477fa63e5c8b850a,49c811f8b272f0a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:44.102 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:04:44.103 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:04:44.104 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.104 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:04:44.105 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.106 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.106 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.107 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:04:44.122 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.123 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.123 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.124 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:04:44.136 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.137 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.137 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.138 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:04:44.152 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.153 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.153 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.154 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:04:44.166 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.167 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.167 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.168 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:04:44.173 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.173 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.174 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.174 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:04:44.178 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.179 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.179 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.179 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:04:44.184 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.184 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.185 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.185 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:04:44.191 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.191 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.192 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.192 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:04:44.197 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.198 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.198 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.199 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:04:44.209 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.210 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.210 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.211 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:04:44.219 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.220 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.220 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.221 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:04:44.234 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.234 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.236 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.236 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:04:44.241 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.241 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.241 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.242 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:04:44.248 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.248 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.249 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.249 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:04:44.254 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.255 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.255 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.256 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:04:44.260 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.260 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.261 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.261 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:04:44.266 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.266 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.267 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.267 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:04:44.272 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.272 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.273 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.274 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:04:44.278 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.279 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.279 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.280 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:04:44.284 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.285 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.286 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.286 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:04:44.291 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.291 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.292 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.292 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:04:44.297 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.297 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.298 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.298 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:04:44.302 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.302 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.303 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.303 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:04:44.308 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.308 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.309 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.309 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:04:44.313 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.313 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.314 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.314 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:04:44.318 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.319 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.319 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.320 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:04:44.324 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.325 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.325 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.326 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:04:44.330 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.330 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.331 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.331 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:04:44.335 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.335 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.336 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.336 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:04:44.341 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.341 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.342 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.342 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:04:44.347 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.347 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.348 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.348 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:04:44.352 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.353 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.353 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.354 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:04:44.358 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.359 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.359 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.359 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:04:44.366 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.367 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.367 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.368 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:04:44.374 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.374 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.374 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.375 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:04:44.380 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.380 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.381 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.381 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:04:44.386 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.387 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.387 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.388 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:04:44.391 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.391 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.392 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.392 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:04:44.395 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.396 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.396 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.396 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:04:44.400 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.400 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.401 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.401 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:04:44.405 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.405 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.406 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.406 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:04:44.415 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.416 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.416 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.417 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:04:44.439 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.440 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.441 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.441 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:04:44.447 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.448 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.449 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.449 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:04:44.455 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.456 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.457 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.457 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:04:44.463 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.463 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.463 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.464 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:04:44.468 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.468 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.469 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.469 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:04:44.473 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.474 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.475 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.475 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:04:44.479 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:04:44.479 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:04:44.480 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.481 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:04:44.482 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.482 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.483 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.483 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:04:44.488 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.488 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:04:44.489 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:04:44.489 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:04:44.494 INFO 21062 --- [ocp-schedules-5,833e6b6924d2009f,90b5635568ad2956] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:04:44.628 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:44.629 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d5ce8386241383a2,ef7094fcae726ff2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:45.423 WARN 21062 --- [ocp-async-9,6641062ef93f96be,e24fa72335c2c588] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:45.457 INFO 21062 --- [ocp-async-1,88628625d65b6a23,cd5fde9e38ed3b19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:47.423 WARN 21062 --- [ocp-async-10,145fd5aa7d8dc48e,428c8a56e59a3db5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:47.457 INFO 21062 --- [ocp-async-5,7c3f85cb3e9dd3ce,784fbc645e53570d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:47.629 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:47.630 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:47.983 INFO 21062 --- [prometheus-minute-schedule-1,56490c9d1c96a86b,cc95e48d378a12a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:04:48.591824+08:00, lastProcessTime=2024-11-04T15:04:38.589467+08:00 2024-11-04 15:04:48.739 INFO 21062 --- [ocp-async-8,b4947e0b462cc072,31b8a0bbbcf0a8af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:04:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:04:48.591824+08:00 2024-11-04 15:04:49.423 WARN 21062 --- [ocp-async-2,eb081026fac2cdeb,20076f732ad40e49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:49.457 INFO 21062 --- [ocp-async-8,bca6e12e790fa5ad,50557e6609ec5b82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:50.630 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:50.630 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8fbe41b62340b1ed,f7640e76c334cffa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:51.257 INFO 21062 --- [ocp-schedules-21,721c8661be0fab1c,fa04463bf0689d30] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:04:51.423 WARN 21062 --- [ocp-async-4,c7f5e8e26d640525,039c70de7cf96132] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:51.457 INFO 21062 --- [ocp-async-1,44ba9e61bf56e471,9508c49c5d5ccc73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:52.173 INFO 21062 --- [ocp-schedules-9,c617806b0f4d9334,8335840070cb8a74] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:04:52.521 INFO 21062 --- [ocp-schedules-13,d7745cee3569a843,2a5980c2baf1b7a6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:04:52.526 INFO 21062 --- [ocp-schedules-13,d7745cee3569a843,2a5980c2baf1b7a6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:04:52.526 INFO 21062 --- [ocp-schedules-13,d7745cee3569a843,2a5980c2baf1b7a6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:04:53.423 WARN 21062 --- [ocp-async-10,be8f65bdc115fc5c,6eee17a930316799] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:53.458 INFO 21062 --- [ocp-async-5,ccc33f25f64569ac,15dd10e85466cfba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:53.631 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:53.631 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9026f796eae9e666,43a84fd40930510b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:55.423 WARN 21062 --- [ocp-async-7,a69e82a31dc430f4,40ea3d66ab9fb915] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:55.458 INFO 21062 --- [ocp-async-6,ce6a229e1bb56971,0e9fbc1728f6199b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:56.632 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:56.632 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d4bf7a5e16931eff,a9e1d56664a10742] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:04:57.423 WARN 21062 --- [ocp-async-8,bbcd59df6d6a6890,c63ecc9e7c55847f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:57.458 INFO 21062 --- [ocp-async-9,5d6e5349b71acff0,e30eed91387457c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:58.500 INFO 21062 --- [ocp-schedules-13,b7edcdd4ad07168b,3bcf510c7b723271] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:04:58.500 INFO 21062 --- [ocp-schedules-17,acfd3477f490fe87,bc1e728f69d26c2f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:04:58.503 INFO 21062 --- [ocp-schedules-10,7cefb8a784727da6,8e8c00ca6a656332] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:04:58.504 INFO 21062 --- [ocp-schedules-25,18f4fbd3f0ff86f5,fcde8f96fc6f86fa] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:04:58.505 INFO 21062 --- [ocp-schedules-17,acfd3477f490fe87,bc1e728f69d26c2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:58.505 INFO 21062 --- [ocp-schedules-13,b7edcdd4ad07168b,3bcf510c7b723271] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:04:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:04:58.511 INFO 21062 --- [ocp-schedules-5,08722b8fb6fbd72b,57f80bc5da26e4dd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:04:58.511 INFO 21062 --- [ocp-schedules-29,f17d46b4425739b2,ba377d628d8a3717] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:04:58.511 INFO 21062 --- [ocp-schedules-16,4a604c3710919c7c,1469ef1d264a8115] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:04:58.511 INFO 21062 --- [ocp-schedules-5,08722b8fb6fbd72b,57f80bc5da26e4dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:04:58.511 INFO 21062 --- [ocp-schedules-16,4a604c3710919c7c,1469ef1d264a8115] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:04:58.511 INFO 21062 --- [ocp-schedules-29,f17d46b4425739b2,ba377d628d8a3717] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:04:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:04:58.589946+08:00, lastProcessTime=2024-11-04T15:04:48.591824+08:00 2024-11-04 15:04:58.739 INFO 21062 --- [ocp-async-5,d76f3a4c06a171e6,e39e0755476ff924] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:04:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:04:58.589946+08:00 2024-11-04 15:04:59.398 INFO 21062 --- [ocp-schedules-10,3c1a5c691edba465,1694688786fcd574] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:04:59.424 WARN 21062 --- [ocp-async-3,b9791edf17f49391,07d1e7abc2ca7f0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:04:59.458 INFO 21062 --- [ocp-async-5,4a476f815b121e46,c8e8c05be4a27037] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:04:59.632 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:04:59.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:04:59.983 INFO 21062 --- [prometheus-minute-schedule-1,11e623f97f7db8f9,4939f5a276fdc86c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703900, interval=60 2024-11-04 15:05:00.682 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,a582d5747635af8a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:00:01.0, nextRun=2024-11-04 15:05:00.0, currentDate=Mon Nov 04 15:05:00 CST 2024 2024-11-04 15:05:00.682 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,a582d5747635af8a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:00:01.0, nextRunTime=2024-11-04 15:05:00.0} 2024-11-04 15:05:00.712 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,a582d5747635af8a] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:05:00.715 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,a582d5747635af8a] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:05:00.715 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,a582d5747635af8a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:05:01.260 INFO 21062 --- [ocp-schedules-19,f48f2f124ad5b842,9ecb6f3a051f96e2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:05:01.424 WARN 21062 --- [ocp-async-7,43e284b30f41a201,53f36f0552320e71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:01.458 INFO 21062 --- [ocp-async-6,4933f590c3647cf0,0efec01ee6a4105a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:02.199 INFO 21062 --- [ocp-schedules-27,fe89e05e393d31b5,f9d1253d113631f5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:05:02.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:02.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:02.983 INFO 21062 --- [prometheus-minute-schedule-1,46e60e5d6eacaa70,3da463ae01c9c311] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:03.424 WARN 21062 --- [ocp-async-8,9917f47d3c3c2b4f,474ca899b935b718] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:03.459 INFO 21062 --- [ocp-async-9,c7330d5ec2a0ad1d,88332a3c8a4c6a5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:05.424 WARN 21062 --- [ocp-async-1,b9afdfd9ab735d24,d3887ba6fd093196] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:05.459 INFO 21062 --- [ocp-async-3,e366609c8cd7da21,766d1bb02d48902f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:05.634 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:05.635 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:05.983 INFO 21062 --- [prometheus-minute-schedule-1,efc9bee0d919048f,43434a51b9fb13ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:07.424 WARN 21062 --- [ocp-async-5,2b2ceedc62e796ba,857c96f533cc2cb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:07.459 INFO 21062 --- [ocp-async-2,2beed5195949a0f1,36c3428b77f611f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:08.506 INFO 21062 --- [ocp-schedules-17,acfd3477f490fe87,bc1e728f69d26c2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:05:08.506 INFO 21062 --- [ocp-schedules-13,b7edcdd4ad07168b,3bcf510c7b723271] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:05:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:05:08.590781+08:00, lastProcessTime=2024-11-04T15:04:58.589946+08:00 2024-11-04 15:05:08.635 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:08.636 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:08.739 INFO 21062 --- [ocp-async-9,ee43aaa1e55e3abc,073247750bb050e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:05:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:05:08.590781+08:00 2024-11-04 15:05:08.969 INFO 21062 --- [ocp-schedules-12,c1917597db75247e,6ce4930985953ffd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:05:08.969 INFO 21062 --- [ocp-schedules-12,de933ab22282ce8d,6fea47800c622c56] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:05:08.973 INFO 21062 --- [ocp-schedules-12,de933ab22282ce8d,6fea47800c622c56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:08.983 INFO 21062 --- [prometheus-minute-schedule-1,566e43b12d679daa,1133bf43e639e0da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:09.003 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:05:09.003 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:05:09.228 INFO 21062 --- [ocp-schedules-15,e4f69e7bae3761e8,a317f95e084d1640] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:05:09.231 INFO 21062 --- [ocp-schedules-15,e4f69e7bae3761e8,a317f95e084d1640] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:09.232 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:05:09.424 WARN 21062 --- [ocp-async-6,30a514fa22cb445f,ef71de3615a57263] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:09.459 INFO 21062 --- [ocp-async-9,fd3ef8cfd55ed09c,783921342933b96e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:10.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:05:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:05:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:05:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:05:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:05:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:05:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:05:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:05:11.262 INFO 21062 --- [ocp-schedules-20,d888871189a52628,6c4bdcdf73e7dce1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:05:11.424 WARN 21062 --- [ocp-async-9,37f8887cd77ae384,9be1b34384ae21f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:11.459 INFO 21062 --- [ocp-async-3,36881c5020b2b642,b7f99065ad02d504] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:11.636 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:11.637 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:11.743 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=1, affectRows=1 2024-11-04 15:05:11.743 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=1, validCount=1 2024-11-04 15:05:11.744 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=8, affectRows=8 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=8, validCount=8 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=11, affectRows=11 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=30, affectRows=30 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=39, affectRows=39 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=11, validCount=11 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=30, validCount=30 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=42, affectRows=42 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=39, validCount=39 2024-11-04 15:05:11.745 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=24, affectRows=24 2024-11-04 15:05:11.746 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=24, validCount=24 2024-11-04 15:05:11.746 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=42, validCount=42 2024-11-04 15:05:11.746 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=52, affectRows=52 2024-11-04 15:05:11.746 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=52, validCount=52 2024-11-04 15:05:11.746 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=61, affectRows=61 2024-11-04 15:05:11.747 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=61, validCount=61 2024-11-04 15:05:11.747 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=72, affectRows=72 2024-11-04 15:05:11.747 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703840, nodeCount=82, affectRows=82 2024-11-04 15:05:11.747 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=70, validCount=70 2024-11-04 15:05:11.747 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703840, nodeCount=81, validCount=81 2024-11-04 15:05:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b87ebf847eefdf4c,fb9408569c8b64e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:12.239 INFO 21062 --- [ocp-schedules-22,9e1b94846c7e3424,03e45857438f80e2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:05:12.529 INFO 21062 --- [ocp-schedules-23,9849107e145a29ea,d913746308417693] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:05:12.535 INFO 21062 --- [ocp-schedules-23,9849107e145a29ea,d913746308417693] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:05:12.535 INFO 21062 --- [ocp-schedules-23,9849107e145a29ea,d913746308417693] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:05:13.425 WARN 21062 --- [ocp-async-3,1aef722a58daa6b9,d41a19b0b30693b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:13.459 INFO 21062 --- [ocp-async-2,227388b86d052cfc,39d6879127001bd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:13.985 INFO 21062 --- [prometheus-cache-monitor1,3f8355a7f2e02ed7,133ad1e7ea9c5c5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:05:13.985 INFO 21062 --- [prometheus-cache-monitor1,3f8355a7f2e02ed7,133ad1e7ea9c5c5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:05:14.637 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:14.638 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:14.983 INFO 21062 --- [prometheus-minute-schedule-1,fe67cb1f64533dbf,553e525a0fde5e06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:15.421 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:01:15.403, end=2024-11-04T15:03:15.418} 2024-11-04 15:05:15.425 WARN 21062 --- [ocp-async-6,646bff49fece487c,1923dfeafdc1064e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:15.432 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:05:15.432 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:05:15.459 INFO 21062 --- [ocp-async-4,2a4104218d8bc782,7a9cd57972ea204a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:17.425 WARN 21062 --- [ocp-async-4,5eabc8d34dee6a9f,013e122a10c85817] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:17.460 INFO 21062 --- [ocp-async-10,c6f3b8913f84ea5f,82c4d120c03d61d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:17.638 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:17.639 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9776c4f608487895,6855326bfa044897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:05:18.589730+08:00, lastProcessTime=2024-11-04T15:05:08.590781+08:00 2024-11-04 15:05:18.739 INFO 21062 --- [ocp-async-7,51a8fc3da1abaf58,9fb27f48e181e28d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:05:18.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:05:18.589730+08:00 2024-11-04 15:05:19.232 INFO 21062 --- [ocp-schedules-15,e4f69e7bae3761e8,a317f95e084d1640] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:05:19.425 WARN 21062 --- [ocp-async-3,db77435c308b8163,4dafca7ee1b80255] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:19.460 INFO 21062 --- [ocp-async-7,7960b27c35de96da,8951d4323d61833f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:20.639 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:20.640 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4f81b39200e93286,c22550aab9b7a10e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:21.266 INFO 21062 --- [ocp-schedules-2,c2491687d5b4706a,2ec9d811ffa889a9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:05:21.425 WARN 21062 --- [ocp-async-7,2d3bfb37a2002981,f2682b23143c09dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:21.460 INFO 21062 --- [ocp-async-4,c8ebd01aba3d3be3,c855e1b62efb6907] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:22.267 INFO 21062 --- [ocp-schedules-14,1b3b1a851367d282,5806c8d7723fff91] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:05:23.426 WARN 21062 --- [ocp-async-4,0dfdefea0b625edf,62218bcc2a0ae578] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:23.460 INFO 21062 --- [ocp-async-10,4cdb44b9149b555d,6d9c3c04e5e3220f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:23.640 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:23.641 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:23.983 INFO 21062 --- [prometheus-minute-schedule-1,dd704fb4fae40705,d4ad07ef349e5ec7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:25.426 WARN 21062 --- [ocp-async-10,51368a697ca1e71a,3291676d66cfeae2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:25.460 INFO 21062 --- [ocp-async-6,619c183ed18cf48f,8b4fe282440c9380] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:26.641 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:26.642 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:26.983 INFO 21062 --- [prometheus-minute-schedule-1,59953d36b1768770,bc2f2645ec2ad9e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a4df9e6eb5a83211,59fc903a311982bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:05:27.426 WARN 21062 --- [ocp-async-6,4c3b273dd18d4f53,47a8b890ba46d5f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:27.461 INFO 21062 --- [ocp-async-8,cdbdea9ffeab1ff5,5cef914b99b96a40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:28.498 INFO 21062 --- [ocp-schedules-16,e8ba390a004ef839,8bf06da9ecbe4fe8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:05:28.499 INFO 21062 --- [ocp-schedules-15,721c851738239350,f54d6623f044e558] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:05:28.500 INFO 21062 --- [ocp-schedules-11,233880588c4034f6,e5f710bf42892882] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:05:28.500 INFO 21062 --- [ocp-schedules-8,125483427f0f272f,913e1bac3cfd23e6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:05:28.502 INFO 21062 --- [ocp-schedules-16,e8ba390a004ef839,8bf06da9ecbe4fe8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:05:28.503 INFO 21062 --- [ocp-schedules-5,1562211218d31966,dac63b713ecef987] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:05:28.503 INFO 21062 --- [ocp-schedules-11,233880588c4034f6,e5f710bf42892882] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:28.504 INFO 21062 --- [ocp-schedules-8,125483427f0f272f,913e1bac3cfd23e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:28.504 INFO 21062 --- [ocp-schedules-6,f7faecf11d6590e3,d86b186661a83fc5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:05:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:05:28.505 INFO 21062 --- [ocp-schedules-15,721c851738239350,f54d6623f044e558] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:05:28.509 INFO 21062 --- [ocp-schedules-29,866e99dc3e6fb22e,8e0487ad03b28021] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:05:28.509 INFO 21062 --- [ocp-schedules-14,b526b54ce621d739,1fcedcf93ab6d264] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:05:28.509 INFO 21062 --- [ocp-schedules-29,866e99dc3e6fb22e,8e0487ad03b28021] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:05:28.509 INFO 21062 --- [ocp-schedules-14,b526b54ce621d739,1fcedcf93ab6d264] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:05:28.509 INFO 21062 --- [ocp-schedules-4,37ab38dea2765722,c5c8d19ddb7a0292] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:05:28.509 INFO 21062 --- [ocp-schedules-4,37ab38dea2765722,c5c8d19ddb7a0292] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:05:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:05:28.588868+08:00, lastProcessTime=2024-11-04T15:05:18.589730+08:00 2024-11-04 15:05:28.740 INFO 21062 --- [ocp-async-3,baca1f6544aa9cfe,8975ff6060545388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:05:28.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:05:28.588868+08:00 2024-11-04 15:05:28.938 INFO 21062 --- [ocp-schedules-28,485a8bd97ff3c10c,4a0fe0a8047198f1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:05:29.288 INFO 21062 --- [ocp-schedules-29,953cbb678cce3243,2e5d171342e4e866] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:05:29.288 INFO 21062 --- [ocp-schedules-29,953cbb678cce3243,2e5d171342e4e866] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:05:29.400 INFO 21062 --- [ocp-schedules-4,ef4485e44e29cf03,8cb0a5f4d29253a1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:05:29.426 WARN 21062 --- [ocp-async-9,ef6848d7c86ed0d4,3139c4969de5798f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:29.429 INFO 21062 --- [ocp-schedules-16,24fb3e57aedeaaa5,77293d4d00124b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:05:29.436 INFO 21062 --- [ocp-schedules-16,24fb3e57aedeaaa5,77293d4d00124b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:05:29.436 INFO 21062 --- [ocp-schedules-16,24fb3e57aedeaaa5,77293d4d00124b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:05:29.461 INFO 21062 --- [ocp-async-3,e172fb51c21ddf7e,5287ab8d4c30243a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:29.642 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:29.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:29.873 INFO 21062 --- [ocp-schedules-10,59a97b3a23d3f951,0c627fd750eabce6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:05:29.873 INFO 21062 --- [ocp-schedules-10,59a97b3a23d3f951,0c627fd750eabce6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:05:29.876 INFO 21062 --- [ocp-schedules-10,59a97b3a23d3f951,0c627fd750eabce6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:05:29.876 INFO 21062 --- [ocp-schedules-10,59a97b3a23d3f951,0c627fd750eabce6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:05:29.876 INFO 21062 --- [ocp-schedules-10,59a97b3a23d3f951,0c627fd750eabce6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:05:29.876 INFO 21062 --- [ocp-schedules-10,59a97b3a23d3f951,0c627fd750eabce6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:05:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3f14b4cc7fd65c1b,28f736546668614c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:05:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:05:31.268 INFO 21062 --- [ocp-schedules-9,d1b5c1c068b87b42,ec91fdc54a278539] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:05:31.426 WARN 21062 --- [ocp-async-3,aa5ae092440025f9,5ac435d4ffb1748b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:31.461 INFO 21062 --- [ocp-async-7,1999649505dc324c,e82d7a3f23bbeef6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:32.307 INFO 21062 --- [ocp-schedules-21,95457fbc3fbdc531,6c85fc6f24e4b425] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:05:32.537 INFO 21062 --- [ocp-schedules-9,2fd151f9cbe40a5a,0af973654e199407] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:05:32.543 INFO 21062 --- [ocp-schedules-9,2fd151f9cbe40a5a,0af973654e199407] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:05:32.543 INFO 21062 --- [ocp-schedules-9,2fd151f9cbe40a5a,0af973654e199407] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:05:32.643 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:32.644 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:32.718 INFO 21062 --- [ocp-schedules-28,20d70dedb7654b82,9c0a455cb6f677c8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:05:32.718 INFO 21062 --- [ocp-schedules-28,20d70dedb7654b82,9c0a455cb6f677c8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:05:32.744 INFO 21062 --- [ocp-schedules-28,20d70dedb7654b82,9c0a455cb6f677c8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:05:32.744 INFO 21062 --- [ocp-schedules-28,20d70dedb7654b82,9c0a455cb6f677c8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:05:32.746 INFO 21062 --- [ocp-schedules-15,f4f6a72c3b8a123f,b381227c1dda772f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:05:32.941 INFO 21062 --- [ocp-schedules-30,66d38def62721e68,728bc1b06910073f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:05:32.941+08:00, clusters=[1] 2024-11-04 15:05:32.942 INFO 21062 --- [ocp-schedules-30,66d38def62721e68,728bc1b06910073f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:05:32.983 INFO 21062 --- [prometheus-minute-schedule-1,cfc7fed2dc86eda7,18f6945872258fe2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:33.189 INFO 21062 --- [ocp-schedules-10,127dd445cda027f0,fbf9dd1e62b0466e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:05:33.189+08:00, clusters=[1] 2024-11-04 15:05:33.193 INFO 21062 --- [ocp-schedules-10,127dd445cda027f0,fbf9dd1e62b0466e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:05:33.426 WARN 21062 --- [ocp-async-8,3692400e7490539f,0fea19f65be9f03f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:33.461 INFO 21062 --- [ocp-async-1,1af7e048b69768de,56f50d4246feab6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:34.746 INFO 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:05:34.746 INFO 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:05:34.746 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:05:34.746 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:05:34.757 INFO 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:05:34.757 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:05:34.758 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:05:34.759 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:05:34.759 WARN 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:05:34.759 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:05:34.759 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:05:34.760 INFO 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:05:34.760 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:05:34.761 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:05:34.761 WARN 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:05:34.761 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:05:34.768 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:05:34.768 INFO 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:05:34.768 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:05:34.782 ERROR 21062 --- [ocp-schedules-2,a0b7be6052860902,b8478c9b6c5719cb] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:05:35.427 WARN 21062 --- [ocp-async-9,800478ae4849219f,35cb58c417f6bcb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:35.461 INFO 21062 --- [ocp-async-10,3d66a66d01a71551,2cdc34776610a26d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:35.644 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:35.645 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:35.983 INFO 21062 --- [prometheus-minute-schedule-1,534e26e324c3aa29,a771ed86ed81b3c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:37.427 WARN 21062 --- [ocp-async-5,e33eb3d52f5f89cc,097ea58e717631c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:37.462 INFO 21062 --- [ocp-async-6,2d198071c1453b5a,16ea5a753e8b47f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:38.505 INFO 21062 --- [ocp-schedules-11,233880588c4034f6,e5f710bf42892882] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:05:38.505 INFO 21062 --- [ocp-schedules-8,125483427f0f272f,913e1bac3cfd23e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:05:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:05:38.590476+08:00, lastProcessTime=2024-11-04T15:05:28.588868+08:00 2024-11-04 15:05:38.682 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:38.683 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:38.740 INFO 21062 --- [ocp-async-9,1f9346a6fb17b7e5,f3ca4c0d207016bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:05:38.807 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:05:38.590476+08:00 2024-11-04 15:05:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ec992c0d6e402a3b,3dc446cfda297205] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:39.003 INFO 21062 --- [ocp-schedules-12,de933ab22282ce8d,6fea47800c622c56] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:05:39.004 INFO 21062 --- [ocp-schedules-12,7a947652e2466684,db8659ff5d33412d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:05:39.007 INFO 21062 --- [ocp-schedules-12,7a947652e2466684,db8659ff5d33412d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:39.034 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:05:39.034 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:05:39.427 WARN 21062 --- [ocp-async-7,7c8437eecc16dbaf,ee2a3b4d92895e41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:39.462 INFO 21062 --- [ocp-async-9,43583db79346675a,e2878adbd2784a67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:05:40.004 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:05:41.271 INFO 21062 --- [ocp-schedules-6,0c16265d9f7babd9,0d9cf43aee3c8367] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:05:41.427 WARN 21062 --- [ocp-async-1,57ae2663622e3336,f4eff0152fe8d328] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:41.462 INFO 21062 --- [ocp-async-5,55ec9c1c640901b4,0768cc89eed6ea4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:41.683 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:41.684 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:41.983 INFO 21062 --- [prometheus-minute-schedule-1,13a3e5f5d48963f3,67c54beb1d1f82e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:42.333 INFO 21062 --- [ocp-schedules-25,815dce5cbbdcb0df,2dd239252c6429c7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:05:42.654 INFO 21062 --- [ocp-schedules-18,31e41b63313908de,9e58fa25d29bedf9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:05:42.658 INFO 21062 --- [ocp-schedules-18,31e41b63313908de,9e58fa25d29bedf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:42.742 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:05:42.745 ERROR 21062 --- [ocp-schedules-18,31e41b63313908de,9e58fa25d29bedf9] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:05:42.745 INFO 21062 --- [ocp-schedules-18,31e41b63313908de,9e58fa25d29bedf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:05:43.427 WARN 21062 --- [ocp-async-3,0b98d6e9df44f69b,fb324c4518447a91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:43.462 INFO 21062 --- [ocp-async-2,7c10a8d6d9233a8f,e17a4a033dc4f858] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:44.684 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:44.685 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:44.983 INFO 21062 --- [prometheus-minute-schedule-1,70ca3061017b74c1,1d2cdc8984237818] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:45.427 WARN 21062 --- [ocp-async-8,3c154d154f633206,4a4f5bd158a1a4b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:45.462 INFO 21062 --- [ocp-async-4,b5412f78f4b93228,78b317822115ab0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:47.428 WARN 21062 --- [ocp-async-1,397e87bca8df3064,e765b3894be21584] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:47.463 INFO 21062 --- [ocp-async-10,3183473d10d499ef,560cfe3311be33f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:47.685 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:47.686 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:47.983 INFO 21062 --- [prometheus-minute-schedule-1,57cc13baa4c15954,b3480234d934a32f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:05:48.590243+08:00, lastProcessTime=2024-11-04T15:05:38.590476+08:00 2024-11-04 15:05:48.740 INFO 21062 --- [ocp-async-2,3e11f9931c61390d,01912c98eec98c8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:05:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:05:48.590243+08:00 2024-11-04 15:05:49.428 WARN 21062 --- [ocp-async-3,0fa0433ae1154bd2,6ee7272ffaaa5af8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:49.463 INFO 21062 --- [ocp-async-2,da40fa6f2c7c8c7e,95c671faeb3002b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:50.686 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:50.687 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4b4a0502d5bd8bc6,ab8c398e790be46f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:51.274 INFO 21062 --- [ocp-schedules-30,ee28f53a78510ea3,4da1cf5f5bc9a156] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:05:51.428 WARN 21062 --- [ocp-async-7,29abe210dad075c8,2b6c08dea5f94de0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:51.463 INFO 21062 --- [ocp-async-4,6c72a535be7ef516,e72c45f694221cc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:52.376 INFO 21062 --- [ocp-schedules-20,bb44b883d05ff5b7,122857bf5ae66384] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:05:52.545 INFO 21062 --- [ocp-schedules-2,912bd35c38ae28b9,e25f160729f0ee4f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:05:52.550 INFO 21062 --- [ocp-schedules-2,912bd35c38ae28b9,e25f160729f0ee4f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:05:52.550 INFO 21062 --- [ocp-schedules-2,912bd35c38ae28b9,e25f160729f0ee4f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:05:53.428 WARN 21062 --- [ocp-async-9,9e30fa32ae8a1547,ae6e0e57b971339f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:53.463 INFO 21062 --- [ocp-async-10,8e69b1327c1d5dc7,fbe7a33fc652fcd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:53.687 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:53.688 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a487ba3c7c26adbb,72e695a020b4ce34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:55.428 WARN 21062 --- [ocp-async-3,0cfdac601e399d45,af27cdb16dfb4108] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:55.464 INFO 21062 --- [ocp-async-6,3f2dc9097700b62b,c5026945334da7e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:56.688 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:56.689 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:56.983 INFO 21062 --- [prometheus-minute-schedule-1,483b5879bf275b75,820257a3bbb04794] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:05:57.428 WARN 21062 --- [ocp-async-2,037021e5bfd362b3,05a4ea26e85f5b10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:57.464 INFO 21062 --- [ocp-async-8,19a866f83202eafa,f685c50e8d255ee1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:58.500 INFO 21062 --- [ocp-schedules-27,4d491ceae090b006,92060dc6bbc5c11c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:05:58.500 INFO 21062 --- [ocp-schedules-19,b449e864892cd12c,90234de75e95daf6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:05:58.502 INFO 21062 --- [ocp-schedules-30,ccb8ef8863dbcf09,0d3347ca600d70bc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:05:58.504 INFO 21062 --- [ocp-schedules-27,4d491ceae090b006,92060dc6bbc5c11c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:58.504 INFO 21062 --- [ocp-schedules-19,b449e864892cd12c,90234de75e95daf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:05:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:05:58.504 INFO 21062 --- [ocp-schedules-9,e5e38d41092ca962,d918795149e5f262] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:05:58.509 INFO 21062 --- [ocp-schedules-24,8249c8ee421ebdfc,35cc41e7f841fb1d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:05:58.509 INFO 21062 --- [ocp-schedules-24,8249c8ee421ebdfc,35cc41e7f841fb1d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:05:58.509 INFO 21062 --- [ocp-schedules-3,e88ca643da5db8e6,6959378f22b61c42] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:05:58.509 INFO 21062 --- [ocp-schedules-3,e88ca643da5db8e6,6959378f22b61c42] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:05:58.510 INFO 21062 --- [ocp-schedules-20,aafba8aa0244ba53,8bfb93bb8e406340] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:05:58.510 INFO 21062 --- [ocp-schedules-20,aafba8aa0244ba53,8bfb93bb8e406340] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:05:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:05:58.589514+08:00, lastProcessTime=2024-11-04T15:05:48.590243+08:00 2024-11-04 15:05:58.740 INFO 21062 --- [ocp-async-10,3de56471cdbebcf4,1c7b2de077ccb726] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:05:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:05:58.589514+08:00 2024-11-04 15:05:59.403 INFO 21062 --- [ocp-schedules-25,ff73d9b7268c04c8,87774ce9e627cc22] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:05:59.429 WARN 21062 --- [ocp-async-9,874d9025f2e0569e,fdcf54e81416e26c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:05:59.464 INFO 21062 --- [ocp-async-10,95d2a64eeea3d199,bd3cf4507b9641de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:05:59.689 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:05:59.690 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:05:59.983 INFO 21062 --- [prometheus-minute-schedule-1,5f3895237a1c7661,340e63850a75f9e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730703960, interval=60 2024-11-04 15:06:01.276 INFO 21062 --- [ocp-schedules-13,24dc717d5885577e,034fe765a573f7c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:06:01.429 WARN 21062 --- [ocp-async-3,99171d7737c57350,a0722029029d29c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:01.464 INFO 21062 --- [ocp-async-6,e36d36d0e7c9c65a,3df132010797c534] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:02.404 INFO 21062 --- [ocp-schedules-15,8f91ed66054c1851,97a80d46d3a8528d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:06:02.690 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:02.690 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:02.983 INFO 21062 --- [prometheus-minute-schedule-1,8b7e6733790ac11c,fcd478bc15fe2d2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:03.429 WARN 21062 --- [ocp-async-2,b60541e01776c18e,eaea8b40d9b69a01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:03.464 INFO 21062 --- [ocp-async-8,20b3c58ea664ed83,9182fb287d095cfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:05.429 WARN 21062 --- [ocp-async-8,f02cbad3e6681c6f,681d2e7ddbebccc3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:05.465 INFO 21062 --- [ocp-async-1,36322d6d8b512e5f,8dc50fa1cc866a0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:05.691 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:05.692 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:05.983 INFO 21062 --- [prometheus-minute-schedule-1,63ac67805cccc761,8cd8905b380bcd6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:07.429 WARN 21062 --- [ocp-async-10,acec700bcc02ceea,693a72de610152d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:07.465 INFO 21062 --- [ocp-async-5,1cd4f1052d159483,aba209fb5226ff7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:08.505 INFO 21062 --- [ocp-schedules-27,4d491ceae090b006,92060dc6bbc5c11c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:06:08.505 INFO 21062 --- [ocp-schedules-19,b449e864892cd12c,90234de75e95daf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:06:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:06:08.589731+08:00, lastProcessTime=2024-11-04T15:05:58.589514+08:00 2024-11-04 15:06:08.692 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:08.693 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:08.740 INFO 21062 --- [ocp-async-9,e7127dee6e1a73d4,4bd5f41210161db5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:06:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:06:08.589731+08:00 2024-11-04 15:06:08.983 INFO 21062 --- [prometheus-minute-schedule-1,270c3059b7bf3a05,dc9f589f8362330d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:09.035 INFO 21062 --- [ocp-schedules-12,7a947652e2466684,db8659ff5d33412d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:06:09.035 INFO 21062 --- [ocp-schedules-12,8ea7498237705028,044d41ab79841c2d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:06:09.039 INFO 21062 --- [ocp-schedules-12,8ea7498237705028,044d41ab79841c2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:09.070 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:06:09.070 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:06:09.429 WARN 21062 --- [ocp-async-2,70dc0a725eea5214,dee28897623e0dfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:09.465 INFO 21062 --- [ocp-async-9,8a10fa8dc8a396d3,a1147a05d1593205] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:10.002 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:06:10.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:06:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:06:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:06:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:06:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:06:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:06:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:06:11.279 INFO 21062 --- [ocp-schedules-30,c8c372045d6aea0d,1761bd5522a95c2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:06:11.430 WARN 21062 --- [ocp-async-8,04aeea1e2898d2f2,a5889093529f248a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:11.465 INFO 21062 --- [ocp-async-10,61f93ca75f328066,efb0ee83781f47f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:11.693 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:11.694 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:11.744 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=5, affectRows=5 2024-11-04 15:06:11.744 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=5, validCount=5 2024-11-04 15:06:11.744 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=3, affectRows=3 2024-11-04 15:06:11.744 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=3, validCount=3 2024-11-04 15:06:11.745 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=5, affectRows=5 2024-11-04 15:06:11.745 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=5, validCount=5 2024-11-04 15:06:11.745 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=19, affectRows=19 2024-11-04 15:06:11.745 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=19, validCount=19 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=7, affectRows=7 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=27, affectRows=27 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=43, affectRows=43 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=7, validCount=7 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=28, affectRows=28 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=33, affectRows=33 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=26, validCount=26 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=28, validCount=28 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=43, validCount=43 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=33, validCount=33 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=64, affectRows=64 2024-11-04 15:06:11.746 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=69, affectRows=69 2024-11-04 15:06:11.747 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=64, validCount=64 2024-11-04 15:06:11.747 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=51, affectRows=51 2024-11-04 15:06:11.747 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=68, validCount=68 2024-11-04 15:06:11.747 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=51, validCount=51 2024-11-04 15:06:11.747 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703900, nodeCount=68, affectRows=68 2024-11-04 15:06:11.747 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703900, nodeCount=67, validCount=67 2024-11-04 15:06:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c0dd4334ac6e5701,8224cd166aea8a15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:12.450 INFO 21062 --- [ocp-schedules-5,4301d3ad03273d96,5699af90619abe7d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:06:12.552 INFO 21062 --- [ocp-schedules-17,f3535ea79b0572bf,de59f06e5cf3fe6f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:06:12.558 INFO 21062 --- [ocp-schedules-17,f3535ea79b0572bf,de59f06e5cf3fe6f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:06:12.558 INFO 21062 --- [ocp-schedules-17,f3535ea79b0572bf,de59f06e5cf3fe6f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:06:13.430 WARN 21062 --- [ocp-async-3,76d624c33a2c77ad,ec334cd9fa77e9fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:13.465 INFO 21062 --- [ocp-async-5,96e9e4cd2d738bf8,3d553f51419040e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:13.984 INFO 21062 --- [prometheus-cache-monitor1,8c92e181741c285f,40cebbd1cde166aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:06:13.984 INFO 21062 --- [prometheus-cache-monitor1,8c92e181741c285f,40cebbd1cde166aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:06:14.694 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:14.695 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:14.983 INFO 21062 --- [prometheus-minute-schedule-1,04a6e120f0390ec7,39209d796b2a4419] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:15.430 WARN 21062 --- [ocp-async-2,fd2f4a8bb017af34,1be1176b9f510d28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:15.465 INFO 21062 --- [ocp-async-7,d378b11fe9fcbce8,58f8d8c441e8de76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:17.430 WARN 21062 --- [ocp-async-8,c6d71b8b7cd954f6,5bfe8b7697493d0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:17.466 INFO 21062 --- [ocp-async-4,2c14023094746dc1,a163b349e8b780cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:17.695 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:17.696 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:17.983 INFO 21062 --- [prometheus-minute-schedule-1,b0b5fba2f232d69e,1d9246f519cd1793] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:06:18.590270+08:00, lastProcessTime=2024-11-04T15:06:08.589731+08:00 2024-11-04 15:06:18.741 INFO 21062 --- [ocp-async-1,99d969f973276d19,dbcc4ee70b0e0658] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:06:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:06:18.590270+08:00 2024-11-04 15:06:19.233 INFO 21062 --- [ocp-schedules-19,3ec900731828ca6d,6505a332124d4de0] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:06:19.236 INFO 21062 --- [ocp-schedules-19,3ec900731828ca6d,6505a332124d4de0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:19.237 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:06:19.430 WARN 21062 --- [ocp-async-3,506ca0495e0a4865,62c578d5d3291ace] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:19.466 INFO 21062 --- [ocp-async-2,a37ad03a3e769c5a,7e704386e4345670] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:20.696 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:20.698 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:20.983 INFO 21062 --- [prometheus-minute-schedule-1,566d325c78dffb08,98fee3c5cbc348c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:21.282 INFO 21062 --- [ocp-schedules-27,07ebfa0ea1306ca3,1be03f25e536be24] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:06:21.431 WARN 21062 --- [ocp-async-5,cd8766d2f637e40b,e2efa2f5dd020fd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:21.466 INFO 21062 --- [ocp-async-8,ed533c8963034fb0,357ffc35d7b9b56b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:22.477 INFO 21062 --- [ocp-schedules-23,e6db4b8daaf32969,c0c167fa0be7b8f9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:06:23.431 WARN 21062 --- [ocp-async-8,26da54d3336115ca,bc223c7ad9b257c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:23.466 INFO 21062 --- [ocp-async-4,b3ccf3adfec58c08,0973a6316b9cce19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:23.698 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:23.699 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8d1e28b58536e6cb,d4273c6752ad4d42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:25.431 WARN 21062 --- [ocp-async-4,59812235d6dfb5a3,2e8201d3bd3321a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:25.466 INFO 21062 --- [ocp-async-1,17b133140e6b076a,6553ea932526463f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:26.699 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:26.701 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:26.983 INFO 21062 --- [prometheus-minute-schedule-1,c72e9e87fcfe10bd,2904255da1cc25cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:26.984 INFO 21062 --- [prometheus-hour-schedule-1,143037fb7f9c7a1c,29abdb9ce92800d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:06:27.431 WARN 21062 --- [ocp-async-1,c5f2f6458fb35133,e1c6477a559f01ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:27.467 INFO 21062 --- [ocp-async-2,52a397cad7a855aa,72a5edcb52690d06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:28.498 INFO 21062 --- [ocp-schedules-17,1290c7b49ecf8fac,398d0e12a8468c93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:06:28.499 INFO 21062 --- [ocp-schedules-2,f65e0be0ec5bf22d,6a9c9e8949e3295f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:06:28.500 INFO 21062 --- [ocp-schedules-18,709041f2285a7265,89e53c5d4c640e37] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:06:28.500 INFO 21062 --- [ocp-schedules-23,4139cff1140d516e,3414aec7de24a923] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:06:28.503 INFO 21062 --- [ocp-schedules-3,20600403d1157ae4,0f476af00eb3c7ca] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:06:28.503 INFO 21062 --- [ocp-schedules-17,1290c7b49ecf8fac,398d0e12a8468c93] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:06:28.503 INFO 21062 --- [ocp-schedules-18,709041f2285a7265,89e53c5d4c640e37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:28.504 INFO 21062 --- [ocp-schedules-23,4139cff1140d516e,3414aec7de24a923] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:06:28.505 INFO 21062 --- [ocp-schedules-26,9b1926a96c91f06e,a82fcb0b106db032] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:06:28.506 INFO 21062 --- [ocp-schedules-2,f65e0be0ec5bf22d,6a9c9e8949e3295f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:06:28.511 INFO 21062 --- [ocp-schedules-22,c66252bcca5f2375,edc37ab23c624693] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:06:28.511 INFO 21062 --- [ocp-schedules-16,996768743c978411,1cd152fbc1b762ac] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:06:28.511 INFO 21062 --- [ocp-schedules-22,c66252bcca5f2375,edc37ab23c624693] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:06:28.511 INFO 21062 --- [ocp-schedules-16,996768743c978411,1cd152fbc1b762ac] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:06:28.511 INFO 21062 --- [ocp-schedules-28,1830c7849f673882,85f0e11d239feb81] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:06:28.511 INFO 21062 --- [ocp-schedules-28,1830c7849f673882,85f0e11d239feb81] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:06:28.584 INFO 21062 --- [ocp-schedules-21,d0d137cbbc6d2490,deca2408a8016226] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:06:28.584 INFO 21062 --- [ocp-schedules-21,d0d137cbbc6d2490,deca2408a8016226] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:06:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:06:28.589516+08:00, lastProcessTime=2024-11-04T15:06:18.590270+08:00 2024-11-04 15:06:28.653 INFO 21062 --- [ocp-schedules-24,a31360894d6b01c2,1b71576c1b45f2e4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:06:28.654 INFO 21062 --- [ocp-schedules-24,a31360894d6b01c2,1b71576c1b45f2e4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:06:28.741 INFO 21062 --- [ocp-async-3,cbdd9fbac624ea66,993460cca843a163] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:06:28.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:06:28.589516+08:00 2024-11-04 15:06:28.917 INFO 21062 --- [ocp-schedules-30,4e373256f0ed4852,a3ba7e886b4f4f18] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:06:28.918 INFO 21062 --- [ocp-schedules-30,4e373256f0ed4852,a3ba7e886b4f4f18] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:06:28.940 INFO 21062 --- [ocp-schedules-3,4ee22bb2b8a3e28b,99993c3672f40060] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:06:29.237 INFO 21062 --- [ocp-schedules-19,3ec900731828ca6d,6505a332124d4de0] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:06:29.292 INFO 21062 --- [ocp-schedules-28,459ec20d86d5760c,04fc098b71ec9d2c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:06:29.292 INFO 21062 --- [ocp-schedules-28,459ec20d86d5760c,04fc098b71ec9d2c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:06:29.407 INFO 21062 --- [ocp-schedules-6,f80baa68e16aa062,3c10f799d9ef2097] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:06:29.431 WARN 21062 --- [ocp-async-4,da1d37a262b9724b,f2caeb8a2c75c4cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:29.436 INFO 21062 --- [ocp-schedules-1,c8e02530967a23b5,413c04390e3d6fcb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:06:29.441 INFO 21062 --- [ocp-schedules-1,c8e02530967a23b5,413c04390e3d6fcb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:06:29.441 INFO 21062 --- [ocp-schedules-1,c8e02530967a23b5,413c04390e3d6fcb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:06:29.467 INFO 21062 --- [ocp-async-3,e6d2df1415192165,07bd518ce89d2258] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:29.701 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:29.702 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:29.877 INFO 21062 --- [ocp-schedules-2,4cb3bfc6f10113c5,6312b633e455816b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:06:29.877 INFO 21062 --- [ocp-schedules-2,4cb3bfc6f10113c5,6312b633e455816b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:06:29.880 INFO 21062 --- [ocp-schedules-2,4cb3bfc6f10113c5,6312b633e455816b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:06:29.880 INFO 21062 --- [ocp-schedules-2,4cb3bfc6f10113c5,6312b633e455816b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:06:29.881 INFO 21062 --- [ocp-schedules-2,4cb3bfc6f10113c5,6312b633e455816b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:06:29.881 INFO 21062 --- [ocp-schedules-2,4cb3bfc6f10113c5,6312b633e455816b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:06:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7beae948ddccb896,7282b7685acbbc2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:06:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:06:31.285 INFO 21062 --- [ocp-schedules-29,0c0b58720b270949,f49a143ed75a1ec0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:06:31.432 WARN 21062 --- [ocp-async-3,d784653ac0786034,b4c623882907fd38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:31.467 INFO 21062 --- [ocp-async-5,1894efdca4e6aba4,94cde358f3c850c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:32.518 INFO 21062 --- [ocp-schedules-17,dac6eacc322d3f70,0c8873de847a146b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:06:32.561 INFO 21062 --- [ocp-schedules-29,8126e87ed3b48eeb,4be36ce7c2b36dfb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:06:32.566 INFO 21062 --- [ocp-schedules-29,8126e87ed3b48eeb,4be36ce7c2b36dfb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:06:32.566 INFO 21062 --- [ocp-schedules-29,8126e87ed3b48eeb,4be36ce7c2b36dfb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:06:32.702 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:32.702 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:32.744 INFO 21062 --- [ocp-schedules-27,1b5e76aa194bdc7b,d1797dd2a46a51fa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:06:32.744 INFO 21062 --- [ocp-schedules-27,1b5e76aa194bdc7b,d1797dd2a46a51fa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:06:32.771 INFO 21062 --- [ocp-schedules-27,1b5e76aa194bdc7b,d1797dd2a46a51fa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:06:32.771 INFO 21062 --- [ocp-schedules-27,1b5e76aa194bdc7b,d1797dd2a46a51fa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:06:32.772 INFO 21062 --- [ocp-schedules-22,f5c7719140c208e6,67047fdb1945b0ff] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:06:32.968 INFO 21062 --- [ocp-schedules-4,41e0cfefd80f31bf,9b43e6ec615838cd] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:06:32.968+08:00, clusters=[1] 2024-11-04 15:06:32.970 INFO 21062 --- [ocp-schedules-4,41e0cfefd80f31bf,9b43e6ec615838cd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:06:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3b97b9eb72b1ae45,fa372facad6ea500] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:33.218 INFO 21062 --- [ocp-schedules-1,e0a4ce0a0ee2941c,0b7e2ee215df5cf1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:06:33.218+08:00, clusters=[1] 2024-11-04 15:06:33.222 INFO 21062 --- [ocp-schedules-1,e0a4ce0a0ee2941c,0b7e2ee215df5cf1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:06:33.246 INFO 21062 --- [ocp-schedules-5,1b6728628f7b8799,29138fa66295e685] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:06:33.250 INFO 21062 --- [ocp-schedules-5,1b6728628f7b8799,29138fa66295e685] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:33.313 ERROR 21062 --- [ocp-schedules-5,1b6728628f7b8799,29138fa66295e685] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:06:33.313 INFO 21062 --- [ocp-schedules-5,1b6728628f7b8799,29138fa66295e685] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:06:33.432 WARN 21062 --- [ocp-async-5,80f53e5447ad7676,bb16497b5212a090] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:33.467 INFO 21062 --- [ocp-async-7,927fb45515e1e864,448c3ef13450fc31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:34.791 INFO 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:06:34.792 INFO 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:06:34.792 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:06:34.792 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:06:34.802 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:06:34.802 INFO 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:06:34.803 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:06:34.804 WARN 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:06:34.804 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:06:34.805 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:06:34.805 INFO 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:06:34.805 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:06:34.806 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:06:34.806 WARN 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:06:34.806 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:06:34.807 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:06:34.814 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:06:34.814 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:06:34.814 INFO 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:06:34.824 ERROR 21062 --- [ocp-schedules-28,11570c8237b6762c,7b5b1dd9421749eb] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:06:35.434 WARN 21062 --- [ocp-async-8,0aabe9d811b86a2e,8008d9e8640f6526] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:35.467 INFO 21062 --- [ocp-async-9,f5e05de498864d0b,d768f86ae1792bca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:35.703 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:35.704 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:35.983 INFO 21062 --- [prometheus-minute-schedule-1,ffb2cd0c40bf2ebe,d5cbac809dcdc699] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:37.432 WARN 21062 --- [ocp-async-3,84c2cf522236eb28,e98bca8049dcbe48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:37.467 INFO 21062 --- [ocp-async-1,e2553cbbe3fe92b7,e11a583c717bd8a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:38.504 INFO 21062 --- [ocp-schedules-23,4139cff1140d516e,3414aec7de24a923] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:06:38.505 INFO 21062 --- [ocp-schedules-18,709041f2285a7265,89e53c5d4c640e37] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:06:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:06:38.589294+08:00, lastProcessTime=2024-11-04T15:06:28.589516+08:00 2024-11-04 15:06:38.704 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:38.705 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:38.741 INFO 21062 --- [ocp-async-8,e5693f2efadd7b1f,f2436d3fcf003245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:06:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:06:38.589294+08:00 2024-11-04 15:06:38.983 INFO 21062 --- [prometheus-minute-schedule-1,9f9ed4193ab0f05d,655cd01ae7822480] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:39.071 INFO 21062 --- [ocp-schedules-12,8ea7498237705028,044d41ab79841c2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:06:39.071 INFO 21062 --- [ocp-schedules-12,5bcc761666774c60,fd43872c13370f4d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:06:39.074 INFO 21062 --- [ocp-schedules-12,5bcc761666774c60,fd43872c13370f4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:39.101 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:06:39.101 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:06:39.432 WARN 21062 --- [ocp-async-7,5efd88f22049d39a,507235d9734ad074] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:39.468 INFO 21062 --- [ocp-async-8,a62db7d4155f6f49,d00da4fde2db4f8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:06:40.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:06:41.289 INFO 21062 --- [ocp-schedules-7,404ccd58e1936fb2,ba20d230d0a01b85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:06:41.432 WARN 21062 --- [ocp-async-4,6d2c7472182eee8c,febe682bfd8cd4fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:41.468 INFO 21062 --- [ocp-async-3,dd2f3f24626af019,a45dee82f6c31cfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:41.705 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:41.706 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:06:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:06:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:06:41.983 INFO 21062 --- [prometheus-minute-schedule-1,45b0a24a94c74626,0e7cfbac8933b28e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:42.543 INFO 21062 --- [ocp-schedules-6,f131791abbbaa071,3eff3b4e0953dfea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:06:42.745 INFO 21062 --- [ocp-schedules-20,a407d297111a30ea,a531b2c5dc7bb40b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:06:42.749 INFO 21062 --- [ocp-schedules-20,a407d297111a30ea,a531b2c5dc7bb40b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:42.833 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:06:42.837 ERROR 21062 --- [ocp-schedules-20,a407d297111a30ea,a531b2c5dc7bb40b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:06:42.837 INFO 21062 --- [ocp-schedules-20,a407d297111a30ea,a531b2c5dc7bb40b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:06:43.433 WARN 21062 --- [ocp-async-10,c78933387f5eb130,816f9184e45ccb16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:43.468 INFO 21062 --- [ocp-async-6,25fd8c95fa792292,c487cd2f20001468] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:44.706 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:44.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:44.983 INFO 21062 --- [prometheus-minute-schedule-1,2fe28d8bdda6491e,b8cd08691f71309b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:45.433 WARN 21062 --- [ocp-async-5,23c204734a1fda01,52cf753dddb0e97c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:45.468 INFO 21062 --- [ocp-async-2,aa5eada12e81c459,82ac3ca1c16568f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:47.433 WARN 21062 --- [ocp-async-4,8e972b3f23e4ec24,c8333ec151052b53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:47.468 INFO 21062 --- [ocp-async-9,4c196563bdd5ff26,4eaae6e2c80540bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:47.707 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:47.708 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:47.983 INFO 21062 --- [prometheus-minute-schedule-1,290aa5e3c6c6a695,cf0d7787102a6e90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:06:48.591089+08:00, lastProcessTime=2024-11-04T15:06:38.589294+08:00 2024-11-04 15:06:48.741 INFO 21062 --- [ocp-async-6,fefebd0477b216b3,77664dc15440f94e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:06:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:06:48.591089+08:00 2024-11-04 15:06:49.433 WARN 21062 --- [ocp-async-1,8936982138bd0af1,a3d931149f7f8267] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:49.469 INFO 21062 --- [ocp-async-6,d237c5a618037464,0eba93a70f63cab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:50.708 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:50.709 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:50.983 INFO 21062 --- [prometheus-minute-schedule-1,728d42bd7e8bf2d8,63f8d03ba9c2aad5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:51.292 INFO 21062 --- [ocp-schedules-8,f1cf9eb62f9c7be2,bbb6ba00eddc6707] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:06:51.433 WARN 21062 --- [ocp-async-5,933a9ea69ce4f17a,26296e0641d3a79c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:51.469 INFO 21062 --- [ocp-async-2,5f9ae730d580913e,33104619dffdf68e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:52.568 INFO 21062 --- [ocp-schedules-16,76868259926ae35e,f87e0cdad46e6e26] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:06:52.575 INFO 21062 --- [ocp-schedules-16,76868259926ae35e,f87e0cdad46e6e26] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:06:52.575 INFO 21062 --- [ocp-schedules-16,76868259926ae35e,f87e0cdad46e6e26] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:06:52.586 INFO 21062 --- [ocp-schedules-20,e5f1744f127b82fe,fb310f3829c98227] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:06:53.434 WARN 21062 --- [ocp-async-4,270890762d33c4cb,0fc8c3029ee72707] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:53.469 INFO 21062 --- [ocp-async-9,e663b0220436ef88,50ece3cce84309f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:53.709 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:53.710 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:53.983 INFO 21062 --- [prometheus-minute-schedule-1,e8583c618242a98f,b6c29c2d07927983] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:55.434 WARN 21062 --- [ocp-async-3,80dbaebf0c63afef,3edaa5bd675ba2b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:55.469 INFO 21062 --- [ocp-async-10,77efd627b175af16,aa0b670d6cf6d92f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:56.710 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:56.710 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ee84fef5b7706c4e,50e9fe46de82a930] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:06:57.434 WARN 21062 --- [ocp-async-6,7f6c21b0396e0a7d,89129386334a6cd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:57.469 INFO 21062 --- [ocp-async-5,a8a8588b7d3c07be,82d750ce4f43f26b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:58.500 INFO 21062 --- [ocp-schedules-13,3beb80c8d4df820d,0ffd1319cd0e76b7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:06:58.500 INFO 21062 --- [ocp-schedules-10,35e4e95349c9f47f,3053e30a6c272d8b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:06:58.502 INFO 21062 --- [ocp-schedules-5,26a42059e4d0edaa,2007d13017282443] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:06:58.504 INFO 21062 --- [ocp-schedules-13,3beb80c8d4df820d,0ffd1319cd0e76b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:58.504 INFO 21062 --- [ocp-schedules-25,ad70fc1b0d693c1a,83fefe3d027c3132] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:06:58.504 INFO 21062 --- [ocp-schedules-10,35e4e95349c9f47f,3053e30a6c272d8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:06:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:06:58.509 INFO 21062 --- [ocp-schedules-29,3437504890836544,fcecc387693db91c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:06:58.509 INFO 21062 --- [ocp-schedules-28,d31d0da837998b48,683442c0233b1978] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:06:58.509 INFO 21062 --- [ocp-schedules-4,1f9af2ccba2366d4,1e1d224435bcf723] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:06:58.510 INFO 21062 --- [ocp-schedules-29,3437504890836544,fcecc387693db91c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:06:58.510 INFO 21062 --- [ocp-schedules-28,d31d0da837998b48,683442c0233b1978] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:06:58.510 INFO 21062 --- [ocp-schedules-4,1f9af2ccba2366d4,1e1d224435bcf723] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:06:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:06:58.589566+08:00, lastProcessTime=2024-11-04T15:06:48.591089+08:00 2024-11-04 15:06:58.741 INFO 21062 --- [ocp-async-9,d920a4aa46f3f945,615f49d5de97f9ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:06:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:06:58.589566+08:00 2024-11-04 15:06:59.410 INFO 21062 --- [ocp-schedules-7,81dddf640fe5104f,fd71b7cf1c192bc8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:06:59.434 WARN 21062 --- [ocp-async-4,f6ad3b35bea2fee8,cb8437f1bc1cab78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:06:59.470 INFO 21062 --- [ocp-async-9,49d43aff8c518547,a7fed0570235b66b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:06:59.711 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:06:59.712 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:06:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f704fe5fed4a0d71,131225a622d9b4f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704020, interval=60 2024-11-04 15:07:01.294 INFO 21062 --- [ocp-schedules-14,67df12d55f2a965b,2d145c9f49a3f87d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:07:01.434 WARN 21062 --- [ocp-async-1,5ffd36f2bb29c335,730d5f3e60714289] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:01.470 INFO 21062 --- [ocp-async-10,1e16edc506501620,ee50d59cd1332aaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:02.612 INFO 21062 --- [ocp-schedules-15,903498cbc86d5354,3adde87d44d6967a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:07:02.712 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:02.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:02.983 INFO 21062 --- [prometheus-minute-schedule-1,41d9ef28ceb92078,4a4f006ee243603a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:03.435 WARN 21062 --- [ocp-async-7,fa5f5f1487340b40,77fb144d404e5a53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:03.470 INFO 21062 --- [ocp-async-5,1b7193f632782378,0ec8bb9f29b8e5e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:05.435 WARN 21062 --- [ocp-async-4,da5551ed67adb924,581db016969ead38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:05.470 INFO 21062 --- [ocp-async-8,d6baccb1ab418704,cf99ac92378bd887] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:05.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:05.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8fb729730af14f56,016b025de6e4178b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:07.435 WARN 21062 --- [ocp-async-9,e1fba4216ba1adde,d1536ce7d63dd4c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:07.470 INFO 21062 --- [ocp-async-3,e429ebc982cd06c1,cd528f645cc79b73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:08.507 INFO 21062 --- [ocp-schedules-10,35e4e95349c9f47f,3053e30a6c272d8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:07:08.507 INFO 21062 --- [ocp-schedules-13,3beb80c8d4df820d,0ffd1319cd0e76b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:07:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:07:08.590349+08:00, lastProcessTime=2024-11-04T15:06:58.589566+08:00 2024-11-04 15:07:08.713 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:08.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:08.742 INFO 21062 --- [ocp-async-5,8c76398593fbd5c6,068f6d1e8223de38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:07:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:07:08.590349+08:00 2024-11-04 15:07:08.983 INFO 21062 --- [prometheus-minute-schedule-1,0b79b0217db58b0b,be5c73570b8e320c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:09.102 INFO 21062 --- [ocp-schedules-12,5bcc761666774c60,fd43872c13370f4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:07:09.102 INFO 21062 --- [ocp-schedules-12,4999cbf829798d6c,b7eee133eee997b6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:07:09.106 INFO 21062 --- [ocp-schedules-12,4999cbf829798d6c,b7eee133eee997b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:09.136 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:07:09.136 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:07:09.435 WARN 21062 --- [ocp-async-7,0eb943d36ca66029,c8b7c56f9d8bbe29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:09.471 INFO 21062 --- [ocp-async-5,927b64ec03fec961,5da51bc19f5a9935] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:10.002 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:07:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:07:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:07:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:07:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:07:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:07:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:07:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:07:11.155 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:07:11.155 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:07:11.163 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:07:11.163 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:07:11.163 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:07:11.163 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:07:11.169 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:07:11.169 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:07:11.169 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:07:11.169 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:07:11.206 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:07:11.215 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:07:11.215 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:07:11.215 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:07:11.215 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:07:11.215 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:07:11.224 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:07:11.298 INFO 21062 --- [ocp-schedules-19,17326d23f25759eb,45778c13943ef786] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:07:11.435 WARN 21062 --- [ocp-async-4,e5c8e46795cb8263,629591af58642fb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:11.471 INFO 21062 --- [ocp-async-9,494c99094f164837,a4b42512c69417b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:11.714 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:11.715 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:11.747 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:07:11.748 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=5, affectRows=5 2024-11-04 15:07:11.748 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=5, validCount=5 2024-11-04 15:07:11.748 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=19, affectRows=19 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=27, affectRows=27 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=31, affectRows=31 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=53, affectRows=53 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=19, validCount=19 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=27, validCount=27 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=48, affectRows=48 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=31, validCount=31 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=53, validCount=53 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=52, affectRows=52 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=71, affectRows=71 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=48, validCount=48 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=52, validCount=52 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=70, validCount=70 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=54, affectRows=54 2024-11-04 15:07:11.749 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730703960, nodeCount=62, affectRows=62 2024-11-04 15:07:11.750 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=54, validCount=54 2024-11-04 15:07:11.750 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730703960, nodeCount=60, validCount=60 2024-11-04 15:07:11.983 INFO 21062 --- [prometheus-minute-schedule-1,264e062ef4e67257,5de79bf35dd1e01a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:12.309 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.140236412S 2024-11-04 15:07:12.310 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:07:12.410 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:07:12.410 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:07:12.434 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:07:12.494 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:07:12.494 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:07:12.494 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:07:12.494 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:07:12.494 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:07:12.494 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:07:12.505 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:07:12.577 INFO 21062 --- [ocp-schedules-1,4803afe082a96f52,df451f5b9a7e40db] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:07:12.583 INFO 21062 --- [ocp-schedules-1,4803afe082a96f52,df451f5b9a7e40db] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:07:12.583 INFO 21062 --- [ocp-schedules-1,4803afe082a96f52,df451f5b9a7e40db] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:07:12.682 INFO 21062 --- [ocp-schedules-19,a12a555c2bd5365a,8fa73c6d5a239166] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:07:12.729 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] o.s.boot.SpringApplication : Started application in 1.565 seconds (JVM running for 9446.408) 2024-11-04 15:07:12.729 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.566025804S 2024-11-04 15:07:13.137 INFO 21062 --- [ocp-schedules-23,3c6a3ed689c94917,5c52ca86c61360da] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:07:13.435 WARN 21062 --- [ocp-async-1,d425707861d33f70,fefb76fa24612acb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:13.471 INFO 21062 --- [ocp-async-3,387c25f87f3f4275,7083db82ac3f4855] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:13.984 INFO 21062 --- [prometheus-cache-monitor1,85852cd35e117f60,9560383d15b0eeca] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:07:13.985 INFO 21062 --- [prometheus-cache-monitor1,85852cd35e117f60,9560383d15b0eeca] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:07:14.715 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:14.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:14.983 INFO 21062 --- [prometheus-minute-schedule-1,7ad6c9b384a13c09,c0f693348762d4f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:15.435 WARN 21062 --- [ocp-async-3,c95ddb5e5908cd66,545f6fc8cb9140c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:15.443 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:03:15.418, end=2024-11-04T15:05:15.441} 2024-11-04 15:07:15.450 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:07:15.450 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:07:15.471 INFO 21062 --- [ocp-async-10,24020df8cfb228e4,a8992109552c65f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:17.436 WARN 21062 --- [ocp-async-10,c2917a13db630e72,70303c5ec277aef4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:17.471 INFO 21062 --- [ocp-async-2,be4bf6ea002a4b8d,1c993a7094b3e3b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:17.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:17.717 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:17.983 INFO 21062 --- [prometheus-minute-schedule-1,c68352c5c1e9f7ca,dd34fe3c82f6a8fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:07:18.590351+08:00, lastProcessTime=2024-11-04T15:07:08.590349+08:00 2024-11-04 15:07:18.742 INFO 21062 --- [ocp-async-8,6d60b66688874c94,f02fdee35b11d6bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:07:18.748 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:07:18.590351+08:00 2024-11-04 15:07:19.436 WARN 21062 --- [ocp-async-1,5bd7def389a425d4,4161a82087fc2eb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:19.471 INFO 21062 --- [ocp-async-7,4aeab812a223610b,146c0c6b5a83da0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:20.717 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:20.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ba2080a773b7a697,a68d2a0c8ab4ad7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:21.301 INFO 21062 --- [ocp-schedules-11,62297731913c3326,2dd45b9b0c9258b3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:07:21.436 WARN 21062 --- [ocp-async-7,74f5ab3fe9056b8a,e61b5aec8f88d515] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:21.472 INFO 21062 --- [ocp-async-10,417fb04119ec801d,3cee42f8cea24cb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:22.706 INFO 21062 --- [ocp-schedules-16,3192e27bbcafdc5a,a0edf06e69aaa469] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:07:23.436 WARN 21062 --- [ocp-async-10,92ed18f840f7cbd4,ceb3a894f8359b7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:23.472 INFO 21062 --- [ocp-async-2,c4580fd5ed0a5600,ce6dc96a5937da22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:23.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:23.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:23.983 INFO 21062 --- [prometheus-minute-schedule-1,7b8f85a4ec494591,2432ee131bc12908] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:25.436 WARN 21062 --- [ocp-async-2,f4e151181890617d,95f9d20f25890973] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:25.472 INFO 21062 --- [ocp-async-8,15571c3561857acd,9b3e2f7655490b27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:26.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:26.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:26.983 INFO 21062 --- [prometheus-minute-schedule-1,91aeccf572f9005e,8af4eb8b3acec98c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c121554620fbaf31,6694a2b59d3b3592] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:07:27.437 WARN 21062 --- [ocp-async-8,472f849ae1831392,75a1c018ee11cf82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:27.472 INFO 21062 --- [ocp-async-6,ead948ffff369007,9a9f2a87df1b7df5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:28.498 INFO 21062 --- [ocp-schedules-9,f7ac1968d68c5ce4,271bf14748c27bda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:07:28.499 INFO 21062 --- [ocp-schedules-2,a6e41faee9387a07,fa37653da689180a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:07:28.500 INFO 21062 --- [ocp-schedules-16,21347ec684be7929,8dc23beb46ef6dac] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:07:28.500 INFO 21062 --- [ocp-schedules-26,d10c9e0f5eee20be,952927a94e73804f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:07:28.502 INFO 21062 --- [ocp-schedules-9,f7ac1968d68c5ce4,271bf14748c27bda] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:07:28.502 INFO 21062 --- [ocp-schedules-5,a3a3e51051b2a988,f263ea0b0a5c9f65] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:07:28.503 INFO 21062 --- [ocp-schedules-16,21347ec684be7929,8dc23beb46ef6dac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:28.504 INFO 21062 --- [ocp-schedules-26,d10c9e0f5eee20be,952927a94e73804f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:28.507 INFO 21062 --- [ocp-schedules-1,201c89d45c17270b,ddfc5a58e62e27d9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:07:28.507 INFO 21062 --- [ocp-schedules-2,a6e41faee9387a07,fa37653da689180a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:07:28.507 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:07:28.509 INFO 21062 --- [ocp-schedules-3,e1dec161c8a8513d,7fcb21b2cc48f695] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:07:28.509 INFO 21062 --- [ocp-schedules-19,ae3a32f37ee208b1,0036b83f6fc6bc86] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:07:28.509 INFO 21062 --- [ocp-schedules-22,929c1fc7132a3d88,7c4bb46579b78b75] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:07:28.509 INFO 21062 --- [ocp-schedules-22,929c1fc7132a3d88,7c4bb46579b78b75] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:07:28.509 INFO 21062 --- [ocp-schedules-19,ae3a32f37ee208b1,0036b83f6fc6bc86] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:07:28.509 INFO 21062 --- [ocp-schedules-3,e1dec161c8a8513d,7fcb21b2cc48f695] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:07:28.509 INFO 21062 --- [ocp-schedules-15,5c12e79142184b86,89f6b66aab0d7cd3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:07:28.512 INFO 21062 --- [ocp-schedules-15,5c12e79142184b86,89f6b66aab0d7cd3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:28.540 INFO 21062 --- [ocp-schedules-29,c3a3a786fa32fe92,40b6a4549801e64a] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:07:28.540 INFO 21062 --- [ocp-schedules-29,c3a3a786fa32fe92,40b6a4549801e64a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:07:28.578 INFO 21062 --- [pool-12-thread-16,87ea9e971e9c5563,503fbf6958afda64] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:07:28.578 INFO 21062 --- [pool-12-thread-15,87ea9e971e9c5563,66fc7a4cd45eee18] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:07:28.578 INFO 21062 --- [pool-12-thread-13,87ea9e971e9c5563,743c5fd4b7731b94] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:07:28.579 INFO 21062 --- [pool-12-thread-14,87ea9e971e9c5563,ae974b215b4fe3cc] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:07:28.584 INFO 21062 --- [ForkJoinPool-11-worker-7,87ea9e971e9c5563,d8380e6c1b0f695d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:07:28.584 INFO 21062 --- [ForkJoinPool-11-worker-7,87ea9e971e9c5563,d8380e6c1b0f695d] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:07:28.584 INFO 21062 --- [ForkJoinPool-11-worker-7,87ea9e971e9c5563,d8380e6c1b0f695d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:07:28.584 INFO 21062 --- [pool-12-thread-13,87ea9e971e9c5563,743c5fd4b7731b94] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:07:28.584 INFO 21062 --- [pool-12-thread-14,87ea9e971e9c5563,ae974b215b4fe3cc] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:07:28.584 INFO 21062 --- [pool-12-thread-14,87ea9e971e9c5563,ae974b215b4fe3cc] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:07:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:07:28.588848+08:00, lastProcessTime=2024-11-04T15:07:18.590351+08:00 2024-11-04 15:07:28.601 INFO 21062 --- [pool-12-thread-15,87ea9e971e9c5563,66fc7a4cd45eee18] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:07:28.602 INFO 21062 --- [pool-12-thread-15,87ea9e971e9c5563,66fc7a4cd45eee18] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:07:28.603 INFO 21062 --- [pool-12-thread-15,87ea9e971e9c5563,66fc7a4cd45eee18] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:07:28.606 INFO 21062 --- [ForkJoinPool-11-worker-7,87ea9e971e9c5563,2a01ecb72614dd5f] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:07:28.606 INFO 21062 --- [pool-12-thread-16,87ea9e971e9c5563,503fbf6958afda64] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:07:28.740 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:07:28.588848+08:00 2024-11-04 15:07:28.742 INFO 21062 --- [ocp-async-1,a9b93a94a506d0f9,36c3c9dcfe4b472d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:07:28.943 INFO 21062 --- [ocp-schedules-19,cfee4ae81666d58f,452d72afe57a479f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:07:29.237 INFO 21062 --- [ocp-schedules-9,61d03c5af3040104,72241455d2a1260d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:07:29.241 INFO 21062 --- [ocp-schedules-9,61d03c5af3040104,72241455d2a1260d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:29.242 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:07:29.297 INFO 21062 --- [ocp-schedules-24,4a8a4534134fd2bc,cfcb9b647d3a217a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:07:29.297 INFO 21062 --- [ocp-schedules-24,4a8a4534134fd2bc,cfcb9b647d3a217a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:07:29.325 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:07:29.342 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:07:29.344 WARN 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:07:29.344 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:07:29.345 WARN 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:07:29.351 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:07:29.352 WARN 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:07:29.353 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:07:29.353 WARN 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:07:29.360 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:07:29.361 WARN 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:07:29.361 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:07:29.363 WARN 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:07:29.368 INFO 21062 --- [ocp-schedules-18,cfd0db7ea1b1ed20,f2b1450a918a90c9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:07:29.413 INFO 21062 --- [ocp-schedules-25,907822534e7f76e5,a05bda28bc2d0a7b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:07:29.437 WARN 21062 --- [ocp-async-5,d38244025844df8a,7b1f47eae33eb06b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:29.441 INFO 21062 --- [ocp-schedules-4,8cdf6e7d61fa6c47,0d6dad31a92d5386] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:07:29.446 INFO 21062 --- [ocp-schedules-4,8cdf6e7d61fa6c47,0d6dad31a92d5386] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:07:29.446 INFO 21062 --- [ocp-schedules-4,8cdf6e7d61fa6c47,0d6dad31a92d5386] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:07:29.472 INFO 21062 --- [ocp-async-1,9b766152147e30a8,9e8551424fe27439] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:29.690 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:07:29.690 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:07:29.699 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:07:29.699 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:07:29.699 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:07:29.709 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:07:29.709 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:07:29.710 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:07:29.713 INFO 21062 --- [ocp-schedules-27,658287b039775186,749e48449585bfc7] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:07:29.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:29.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:29.882 INFO 21062 --- [ocp-schedules-10,2f6bc2cbe69f02fe,20ac2480384b7a03] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:07:29.882 INFO 21062 --- [ocp-schedules-10,2f6bc2cbe69f02fe,20ac2480384b7a03] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:07:29.887 INFO 21062 --- [ocp-schedules-10,2f6bc2cbe69f02fe,20ac2480384b7a03] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:07:29.887 INFO 21062 --- [ocp-schedules-10,2f6bc2cbe69f02fe,20ac2480384b7a03] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:07:29.891 INFO 21062 --- [ocp-schedules-10,2f6bc2cbe69f02fe,20ac2480384b7a03] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:07:29.891 INFO 21062 --- [ocp-schedules-10,2f6bc2cbe69f02fe,20ac2480384b7a03] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:07:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6f2af973f793899a,a8dd0ac339fa301c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:30.257 INFO 21062 --- [ocp-schedules-3,216954721f6e854d,425fe2bda144a3ca] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:07:30.257 INFO 21062 --- [ocp-schedules-3,216954721f6e854d,425fe2bda144a3ca] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:07:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:07:30.284 INFO 21062 --- [pool-50-thread-1,216954721f6e854d,9e60d192d3340a31] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:07:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:07:30.312 WARN 21062 --- [ocp-schedules-3,216954721f6e854d,425fe2bda144a3ca] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:07:30.312 INFO 21062 --- [ocp-schedules-3,216954721f6e854d,425fe2bda144a3ca] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:07:31.304 INFO 21062 --- [ocp-schedules-11,463d78ab18945f14,9492afd63893c230] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:07:31.437 WARN 21062 --- [ocp-async-8,87c7eeafc6978ebb,b3e1702b77d22c53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:31.473 INFO 21062 --- [ocp-async-3,9414207dd39a19c9,b0e4c694c263c6ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:32.586 INFO 21062 --- [ocp-schedules-11,4096b83e587294a4,e2ea2b090fe954cd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:07:32.592 INFO 21062 --- [ocp-schedules-11,4096b83e587294a4,e2ea2b090fe954cd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:07:32.592 INFO 21062 --- [ocp-schedules-11,4096b83e587294a4,e2ea2b090fe954cd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:07:32.721 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:32.721 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:32.732 INFO 21062 --- [ocp-schedules-3,a45ee6270c61a2e2,9f3f5bc026626a2f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:07:32.771 INFO 21062 --- [ocp-schedules-23,1ec4379f19a955bd,98bd12215a0bc01b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:07:32.771 INFO 21062 --- [ocp-schedules-23,1ec4379f19a955bd,98bd12215a0bc01b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:07:32.798 INFO 21062 --- [ocp-schedules-23,1ec4379f19a955bd,98bd12215a0bc01b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:07:32.798 INFO 21062 --- [ocp-schedules-23,1ec4379f19a955bd,98bd12215a0bc01b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:07:32.799 INFO 21062 --- [ocp-schedules-6,05ecde5ef7925ec6,1c399f8b8be1af23] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:07:32.983 INFO 21062 --- [prometheus-minute-schedule-1,078228a0769d360e,5d15207fe9540dd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:32.996 INFO 21062 --- [ocp-schedules-10,8cdbf607bd108304,f56ae16942f42079] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:07:32.996+08:00, clusters=[1] 2024-11-04 15:07:32.997 INFO 21062 --- [ocp-schedules-10,8cdbf607bd108304,f56ae16942f42079] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:07:33.248 INFO 21062 --- [ocp-schedules-18,1bdd6891c71d0287,84f184282ace2a3f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:07:33.248+08:00, clusters=[1] 2024-11-04 15:07:33.252 INFO 21062 --- [ocp-schedules-18,1bdd6891c71d0287,84f184282ace2a3f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:07:33.437 WARN 21062 --- [ocp-async-6,6abc9e51e8f6064e,653d955fe7debd11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:33.473 INFO 21062 --- [ocp-async-10,1ab5da23c124b8b6,c927792a9582b7f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:34.833 INFO 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:07:34.833 INFO 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:07:34.833 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:07:34.833 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:07:34.843 INFO 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:07:34.843 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:07:34.843 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:07:34.844 WARN 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:07:34.844 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:07:34.844 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:07:34.845 INFO 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:07:34.845 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:07:34.845 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:07:34.846 WARN 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:07:34.846 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:07:34.846 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:07:34.852 INFO 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:07:34.852 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:07:34.852 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:07:34.861 ERROR 21062 --- [ocp-schedules-28,06910dd8b9ab187e,819ac9ae78ad51f0] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:07:35.437 WARN 21062 --- [ocp-async-10,f5cc4a9cca24339c,81d8cc06d9974b41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:35.473 INFO 21062 --- [ocp-async-5,277891fe2b65c1d9,88ce05156e4a909e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:35.721 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:35.722 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:35.983 INFO 21062 --- [prometheus-minute-schedule-1,15dd5c550d9744fc,e73c07a906f83041] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:37.437 WARN 21062 --- [ocp-async-8,ad14911a4b103e9f,4e730432d7f5a004] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:37.473 INFO 21062 --- [ocp-async-1,b3bde24de9263a2e,d8abdf56ed4d9c00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:38.507 INFO 21062 --- [ocp-schedules-26,d10c9e0f5eee20be,952927a94e73804f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:07:38.508 INFO 21062 --- [ocp-schedules-16,21347ec684be7929,8dc23beb46ef6dac] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:07:38.512 INFO 21062 --- [ocp-schedules-15,5c12e79142184b86,89f6b66aab0d7cd3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:07:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:07:38.590568+08:00, lastProcessTime=2024-11-04T15:07:28.588848+08:00 2024-11-04 15:07:38.722 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:38.723 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:38.742 INFO 21062 --- [ocp-async-10,85182f806b7c2276,4f04c9033c8f2982] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:07:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:07:38.590568+08:00 2024-11-04 15:07:38.983 INFO 21062 --- [prometheus-minute-schedule-1,2a440c358bdf85b4,6a398bc3d852eff8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:39.137 INFO 21062 --- [ocp-schedules-12,4999cbf829798d6c,b7eee133eee997b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:07:39.137 INFO 21062 --- [ocp-schedules-12,133e41be8b26216a,e6ef05a1a58a99eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:07:39.141 INFO 21062 --- [ocp-schedules-12,133e41be8b26216a,e6ef05a1a58a99eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:39.169 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:07:39.169 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:07:39.242 INFO 21062 --- [ocp-schedules-9,61d03c5af3040104,72241455d2a1260d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:07:39.438 WARN 21062 --- [ocp-async-3,7a04a9c1e3486f09,b393d8f10ca8d64b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:39.473 INFO 21062 --- [ocp-async-10,540ee3baf6d382de,cfe6f0ee5ca194fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:07:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:07:41.307 INFO 21062 --- [ocp-schedules-8,cc8cb83d1666c099,480284a780d0bff7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:07:41.438 WARN 21062 --- [ocp-async-5,d87a634c2672646e,456f96f85f869e17] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:41.473 INFO 21062 --- [ocp-async-8,555fc87553c1422d,6b72f7fe58043925] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:41.723 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:41.724 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:41.983 INFO 21062 --- [prometheus-minute-schedule-1,98a1f81afb7c1c66,ca3231e54943db4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:42.756 INFO 21062 --- [ocp-schedules-4,2a5d1474ccea962b,1d0a6c825b05fa1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:07:42.837 INFO 21062 --- [ocp-schedules-22,4deae6c1af29882d,6488542049909b94] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:07:42.841 INFO 21062 --- [ocp-schedules-22,4deae6c1af29882d,6488542049909b94] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:42.924 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:07:42.928 ERROR 21062 --- [ocp-schedules-22,4deae6c1af29882d,6488542049909b94] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:07:42.928 INFO 21062 --- [ocp-schedules-22,4deae6c1af29882d,6488542049909b94] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:07:43.438 WARN 21062 --- [ocp-async-9,45625e3e39a701de,5e31d7f7c8614d83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:43.474 INFO 21062 --- [ocp-async-7,19e4c01ffd6ae595,cdef4047f4049b8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:44.724 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:44.725 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:44.983 INFO 21062 --- [prometheus-minute-schedule-1,6d17d46689039893,f33103ee91a47bc3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:45.438 WARN 21062 --- [ocp-async-6,2a0730c3d4e43b39,e75db48c284014eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:45.474 INFO 21062 --- [ocp-async-3,3f4e841e3e7aac2f,a6a7bb510c7cc851] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:47.438 WARN 21062 --- [ocp-async-10,6d5889c65e1c33e6,56b3b922659fb8b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:47.474 INFO 21062 --- [ocp-async-2,86cff9dc2eb5a2ac,97e984e29f6ce744] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:47.725 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:47.726 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:47.983 INFO 21062 --- [prometheus-minute-schedule-1,5121f759e8b97f6c,adf4fae61973f270] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:07:48.589963+08:00, lastProcessTime=2024-11-04T15:07:38.590568+08:00 2024-11-04 15:07:48.742 INFO 21062 --- [ocp-async-7,f5370e987a43bc9b,479dbefdd82b9cc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:07:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:07:48.589963+08:00 2024-11-04 15:07:49.439 WARN 21062 --- [ocp-async-9,4ac6e1005fd9775c,d745bb5c03eaf50f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:49.474 INFO 21062 --- [ocp-async-7,f35f4f37b04a28d7,f41835db052d6838] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:50.726 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:50.727 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:50.983 INFO 21062 --- [prometheus-minute-schedule-1,98104b0d86274e62,5d96f416e88ab94c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:51.311 INFO 21062 --- [ocp-schedules-30,a5e192293caa1551,8684763fcb24df39] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:07:51.439 WARN 21062 --- [ocp-async-6,f801ed741f3c7cb0,a2d1caf14af47a74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:51.474 INFO 21062 --- [ocp-async-3,79635f497fac5f8a,fceb94e655bda372] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:52.594 INFO 21062 --- [ocp-schedules-5,9fc1628297f23551,6519114ef83481b0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:07:52.599 INFO 21062 --- [ocp-schedules-5,9fc1628297f23551,6519114ef83481b0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:07:52.599 INFO 21062 --- [ocp-schedules-5,9fc1628297f23551,6519114ef83481b0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:07:52.797 INFO 21062 --- [ocp-schedules-30,263479b7b12256e0,96e146b18af85eef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:07:53.439 WARN 21062 --- [ocp-async-5,ae1138831731c805,e1afdec46f4a78b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:53.474 INFO 21062 --- [ocp-async-2,7ec75368a7e7e2e3,4f595e5c4bd57f8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:53.727 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:53.728 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ed38a7a7ed473ffc,4fd0939e46c3f9fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:55.439 WARN 21062 --- [ocp-async-9,e362b70c405fadd6,76c2562a1c6e4bfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:55.475 INFO 21062 --- [ocp-async-1,541fc1a293bab6e3,9d623d858e90ea31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:56.728 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:56.729 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:56.983 INFO 21062 --- [prometheus-minute-schedule-1,9694a4a1eabdf56e,994ae80ede1afb64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:07:57.439 WARN 21062 --- [ocp-async-7,488f4d77c20f72d7,26f15c31dbf6ce26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:57.475 INFO 21062 --- [ocp-async-4,82346d80bdc5cda4,005d1232895435d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:58.500 INFO 21062 --- [ocp-schedules-14,29fc7dc1bd4f5e33,86189dd40d95bb2e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:07:58.500 INFO 21062 --- [ocp-schedules-21,423ab947fffb1d5a,1b7ceb7f9909811e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:07:58.502 INFO 21062 --- [ocp-schedules-26,bcbfe66c8de55409,eca4eecb7f82f51c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:07:58.503 INFO 21062 --- [ocp-schedules-14,29fc7dc1bd4f5e33,86189dd40d95bb2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:58.504 INFO 21062 --- [ocp-schedules-21,423ab947fffb1d5a,1b7ceb7f9909811e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:07:58.504 INFO 21062 --- [ocp-schedules-4,b149ab01c8262081,8520d7fda9337df8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:07:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:07:58.509 INFO 21062 --- [ocp-schedules-11,85491cb152482b8d,fb59f6d41ad05c6a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:07:58.509 INFO 21062 --- [ocp-schedules-20,2e237fc5cf6fa725,2779a13ac9d59f69] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:07:58.509 INFO 21062 --- [ocp-schedules-20,2e237fc5cf6fa725,2779a13ac9d59f69] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:07:58.509 INFO 21062 --- [ocp-schedules-11,85491cb152482b8d,fb59f6d41ad05c6a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:07:58.509 INFO 21062 --- [ocp-schedules-19,c29d15dfb874dd2f,8bbb2f4dd0b1b041] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:07:58.509 INFO 21062 --- [ocp-schedules-19,c29d15dfb874dd2f,8bbb2f4dd0b1b041] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:07:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:07:58.589730+08:00, lastProcessTime=2024-11-04T15:07:48.589963+08:00 2024-11-04 15:07:58.742 INFO 21062 --- [ocp-async-2,3377124f43c3dd92,0d3a727d7e400c61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:07:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:07:58.589730+08:00 2024-11-04 15:07:59.416 INFO 21062 --- [ocp-schedules-10,4cf45d6f773dabcd,2dffad36af035167] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:07:59.440 WARN 21062 --- [ocp-async-5,8e10d33c874bb466,fed10d7b0cf34663] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:07:59.475 INFO 21062 --- [ocp-async-2,3607f7ab7068a676,4200e679aa52a231] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:07:59.729 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:07:59.730 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:07:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f72af2c3da89b8c5,b17f7701a3adf6c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704080, interval=60 2024-11-04 15:08:01.314 INFO 21062 --- [ocp-schedules-23,3e15291fd0431ee4,f6d70b49ab24e0d6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:08:01.440 WARN 21062 --- [ocp-async-9,1368863c3408ce1b,f5d1793f1e19f922] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:01.475 INFO 21062 --- [ocp-async-1,08dfacfb939b6c5e,7311332a111aff00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:02.730 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:02.730 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:02.821 INFO 21062 --- [ocp-schedules-27,1443b1edad40493f,8603bd812f01d759] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:08:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4d7bbf4944595d10,1611d45ef8952fb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:03.440 WARN 21062 --- [ocp-async-7,dda3e5c15273eea4,d18462b40165a242] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:03.475 INFO 21062 --- [ocp-async-4,04e3f8a8ddf054b5,bbd1b3c01c287a19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:05.440 WARN 21062 --- [ocp-async-3,fb5a44ebbfb12a00,7682daf7a5da00dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:05.476 INFO 21062 --- [ocp-async-10,4d9ba67e0c59a325,7f4bee003b594da9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:05.731 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:05.731 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:05.983 INFO 21062 --- [prometheus-minute-schedule-1,988bc3e1220e3e9f,8246ff271a771570] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:07.440 WARN 21062 --- [ocp-async-9,69c0c9734d09f9ba,6b7def14757c27c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:07.476 INFO 21062 --- [ocp-async-8,007c79212c2aee10,4764c639f42b5e67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:08.504 INFO 21062 --- [ocp-schedules-14,29fc7dc1bd4f5e33,86189dd40d95bb2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:08:08.504 INFO 21062 --- [ocp-schedules-21,423ab947fffb1d5a,1b7ceb7f9909811e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:08:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:08:08.590538+08:00, lastProcessTime=2024-11-04T15:07:58.589730+08:00 2024-11-04 15:08:08.732 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:08.732 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:08.743 INFO 21062 --- [ocp-async-4,92d6b9c7c87eb460,8834aa23d97bea57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:08:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:08:08.590538+08:00 2024-11-04 15:08:08.983 INFO 21062 --- [prometheus-minute-schedule-1,0518997751857b5f,16dfb0c2597a2d61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:09.169 INFO 21062 --- [ocp-schedules-12,133e41be8b26216a,e6ef05a1a58a99eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:08:09.170 INFO 21062 --- [ocp-schedules-12,6b25b4ba4572d685,c544b0bc837f6919] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:08:09.173 INFO 21062 --- [ocp-schedules-12,6b25b4ba4572d685,c544b0bc837f6919] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:09.202 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:08:09.202 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:08:09.440 WARN 21062 --- [ocp-async-7,342faafe0997f2f0,ecf12f05afca3689] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:09.476 INFO 21062 --- [ocp-async-4,c09d3f1c84b197df,9da2fefb5f97e98f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:08:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:08:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:08:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:08:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:08:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:08:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:08:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:08:11.317 INFO 21062 --- [ocp-schedules-29,7215fece79a76c42,bf535b6f9b650c81] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:08:11.441 WARN 21062 --- [ocp-async-5,1a30a264eab9a612,9d7f0d6a99c9f782] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:11.476 INFO 21062 --- [ocp-async-10,0b2b8bae7a3e200e,0d479b9502a42e06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:11.732 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:11.733 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:11.754 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704020, nodeCount=94, affectRows=94 2024-11-04 15:08:11.754 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704020, nodeCount=88, affectRows=88 2024-11-04 15:08:11.754 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704020, nodeCount=108, affectRows=108 2024-11-04 15:08:11.755 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704020, nodeCount=94, validCount=94 2024-11-04 15:08:11.755 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704020, nodeCount=88, validCount=88 2024-11-04 15:08:11.755 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704020, nodeCount=107, validCount=107 2024-11-04 15:08:11.755 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704020, nodeCount=132, affectRows=132 2024-11-04 15:08:11.756 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704020, nodeCount=130, validCount=130 2024-11-04 15:08:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4a9ebed70c9e5dc6,c1e9d69e5c9eff1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:12.602 INFO 21062 --- [ocp-schedules-26,83cebe77cee66a2f,b3f5071b3b131034] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:08:12.607 INFO 21062 --- [ocp-schedules-26,83cebe77cee66a2f,b3f5071b3b131034] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:08:12.607 INFO 21062 --- [ocp-schedules-26,83cebe77cee66a2f,b3f5071b3b131034] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:08:12.865 INFO 21062 --- [ocp-schedules-11,00a86f98bd38a40a,03998a6d86afc9c0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:08:13.441 WARN 21062 --- [ocp-async-10,93edd5427312a8fe,1c4c0ece73e1436f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:13.476 INFO 21062 --- [ocp-async-8,89c09277615af355,af11983f793e4570] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:13.984 INFO 21062 --- [prometheus-cache-monitor1,63f4a77da814081e,9532612bb498bce2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:08:13.984 INFO 21062 --- [prometheus-cache-monitor1,63f4a77da814081e,9532612bb498bce2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:08:14.415 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:02:14.397, end=2024-11-04T15:07:14.413} 2024-11-04 15:08:14.416 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:02:14.397, end=2024-11-04T15:07:14.413} 2024-11-04 15:08:14.420 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:08:14.420 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:08:14.420 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:08:14.420 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:08:14.733 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:14.734 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:14.983 INFO 21062 --- [prometheus-minute-schedule-1,fcb6e67fd08bfa5e,0fcea9bc3c944664] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:15.441 WARN 21062 --- [ocp-async-7,ca911a594eea6b5b,c53b7af1fa974588] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:15.476 INFO 21062 --- [ocp-async-1,6f6f1ccd25bd8598,60fe75d760ac8f94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:17.441 WARN 21062 --- [ocp-async-5,5f36219c446b6d61,69fa3762cb344c43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:17.477 INFO 21062 --- [ocp-async-4,bd6a1ae5bde5b5c2,1f72c8af6faf722a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:17.734 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:17.735 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:17.983 INFO 21062 --- [prometheus-minute-schedule-1,90bc3bae4247cff7,611233d8cfc36c8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:08:18.590299+08:00, lastProcessTime=2024-11-04T15:08:08.590538+08:00 2024-11-04 15:08:18.743 INFO 21062 --- [ocp-async-2,d685ee94a146161f,b34d802175d6628e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:08:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:08:18.590299+08:00 2024-11-04 15:08:19.441 WARN 21062 --- [ocp-async-9,4718171c7bd57c42,3ed5b8f4a2fee169] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:19.477 INFO 21062 --- [ocp-async-7,d361e39866b16552,e39d578a7f73e852] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:20.735 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:20.736 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:20.983 INFO 21062 --- [prometheus-minute-schedule-1,58f8afe6407a0fdc,b37c279c84ef822e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:21.320 INFO 21062 --- [ocp-schedules-16,00e16fb1b0b46d91,9795b98a0c9e7114] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:08:21.441 WARN 21062 --- [ocp-async-8,910e67836bb4855e,fdd619836912bcd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:21.477 INFO 21062 --- [ocp-async-5,3034f47c45631e92,e1ac4a97c06e473e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:22.890 INFO 21062 --- [ocp-schedules-30,347194e553ef7297,72502ae452a999d5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:08:23.442 WARN 21062 --- [ocp-async-1,9e7418d7650d60ce,a9f8d9ecbac7d822] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:23.477 INFO 21062 --- [ocp-async-4,bda6c1068bd2ce17,8382e059a625fc92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:23.736 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:23.737 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:23.983 INFO 21062 --- [prometheus-minute-schedule-1,39b5db08cae22848,cb91819ebf5b87da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:25.442 WARN 21062 --- [ocp-async-9,fcfe45c98fb7e70a,74d94468f454618b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:25.477 INFO 21062 --- [ocp-async-10,e2ff35745f2de0fc,e533de18603b6b22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:26.737 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:26.738 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:26.983 INFO 21062 --- [prometheus-minute-schedule-1,a0de7c903cc07cc2,c6f09529870d1f2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:26.984 INFO 21062 --- [prometheus-hour-schedule-1,bd9fc8ee9501c95f,ecc0d25293db023a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:08:27.442 WARN 21062 --- [ocp-async-8,2a6b19a41e9a3e8d,99ef2c5c783d4eae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:27.478 INFO 21062 --- [ocp-async-7,324eed8c8612d653,2683e6e512c24503] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:28.498 INFO 21062 --- [ocp-schedules-9,f8627146446409e3,b1378876f0eacc89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:08:28.500 INFO 21062 --- [ocp-schedules-1,2c9514a7c3e3b0da,6bb3db6b464e7606] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:08:28.500 INFO 21062 --- [ocp-schedules-19,50328a3da12c5b8d,41a09f57aa6e2351] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:08:28.500 INFO 21062 --- [ocp-schedules-6,d7b81335d0ce8d5a,14c779a6bb358966] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:08:28.502 INFO 21062 --- [ocp-schedules-23,cda97c7d6e2179d0,4331067e1a35ed22] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:08:28.503 INFO 21062 --- [ocp-schedules-9,f8627146446409e3,b1378876f0eacc89] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:08:28.504 INFO 21062 --- [ocp-schedules-1,2c9514a7c3e3b0da,6bb3db6b464e7606] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:28.504 INFO 21062 --- [ocp-schedules-6,d7b81335d0ce8d5a,14c779a6bb358966] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:08:28.505 INFO 21062 --- [ocp-schedules-30,7f8b0ac82f579d48,7620652c8046cdfa] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:08:28.506 INFO 21062 --- [ocp-schedules-19,50328a3da12c5b8d,41a09f57aa6e2351] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:08:28.510 INFO 21062 --- [ocp-schedules-26,017e8479561c467c,2390fef184a9ebd4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:08:28.510 INFO 21062 --- [ocp-schedules-7,c2bdc8e868d7456f,f0fb0ffdf865030d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:08:28.510 INFO 21062 --- [ocp-schedules-26,017e8479561c467c,2390fef184a9ebd4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:08:28.510 INFO 21062 --- [ocp-schedules-7,c2bdc8e868d7456f,f0fb0ffdf865030d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:08:28.510 INFO 21062 --- [ocp-schedules-25,60aa582f3a1a33bf,70032bfdbb092052] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:08:28.510 INFO 21062 --- [ocp-schedules-25,60aa582f3a1a33bf,70032bfdbb092052] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:08:28.585 INFO 21062 --- [ocp-schedules-16,b0e92f47208ddf49,45a094d471bfb24e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:08:28.585 INFO 21062 --- [ocp-schedules-16,b0e92f47208ddf49,45a094d471bfb24e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:08:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:08:28.588858+08:00, lastProcessTime=2024-11-04T15:08:18.590299+08:00 2024-11-04 15:08:28.743 INFO 21062 --- [ocp-async-4,9847df1e7fc26a8f,e55a7386d684ddc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:08:28.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:08:28.588858+08:00 2024-11-04 15:08:28.946 INFO 21062 --- [ocp-schedules-4,ed2c31a1109908cf,21d2824574efc9f5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:08:29.301 INFO 21062 --- [ocp-schedules-26,192d2d3a4f12743f,cba0bd981c40ec38] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:08:29.301 INFO 21062 --- [ocp-schedules-26,192d2d3a4f12743f,cba0bd981c40ec38] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:08:29.419 INFO 21062 --- [ocp-schedules-9,cbf175031892664e,c9efc36bdd7a08ac] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:08:29.442 WARN 21062 --- [ocp-async-9,3ab9ba3c1ce5fcdf,bd82a90cb1e58283] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:29.446 INFO 21062 --- [ocp-schedules-8,4de7c40b2af6a95a,6e792f51c08cba78] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:08:29.452 INFO 21062 --- [ocp-schedules-8,4de7c40b2af6a95a,6e792f51c08cba78] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:08:29.452 INFO 21062 --- [ocp-schedules-8,4de7c40b2af6a95a,6e792f51c08cba78] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:08:29.478 INFO 21062 --- [ocp-async-4,0806851005d14fe8,bae098c62fa2d3af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:29.738 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:29.739 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:29.895 INFO 21062 --- [ocp-schedules-13,a2bf9aaa5ded87a7,9f68457618a3434d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:08:29.895 INFO 21062 --- [ocp-schedules-13,a2bf9aaa5ded87a7,9f68457618a3434d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:08:29.899 INFO 21062 --- [ocp-schedules-13,a2bf9aaa5ded87a7,9f68457618a3434d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:08:29.899 INFO 21062 --- [ocp-schedules-13,a2bf9aaa5ded87a7,9f68457618a3434d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:08:29.900 INFO 21062 --- [ocp-schedules-13,a2bf9aaa5ded87a7,9f68457618a3434d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:08:29.900 INFO 21062 --- [ocp-schedules-13,a2bf9aaa5ded87a7,9f68457618a3434d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:08:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b90419b0e9428682,63aafa1ab522551d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:08:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:08:31.323 INFO 21062 --- [ocp-schedules-22,28c4e41ffe4a232e,dc5f2175eee72fea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:08:31.442 WARN 21062 --- [ocp-async-8,f3f23f0526404346,0da60d2b14530d69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:31.478 INFO 21062 --- [ocp-async-10,f21ecaf9ee717473,5ba4249cb76142dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:32.609 INFO 21062 --- [ocp-schedules-5,0809b9a2ac93b86c,897ee8a0368584d5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:08:32.615 INFO 21062 --- [ocp-schedules-5,0809b9a2ac93b86c,897ee8a0368584d5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:08:32.615 INFO 21062 --- [ocp-schedules-5,0809b9a2ac93b86c,897ee8a0368584d5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:08:32.739 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:32.740 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:32.798 INFO 21062 --- [ocp-schedules-26,e97860df0eacf2da,0f74068fa4199515] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:08:32.798 INFO 21062 --- [ocp-schedules-26,e97860df0eacf2da,0f74068fa4199515] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:08:32.826 INFO 21062 --- [ocp-schedules-9,689fcc538dc96fcc,440f54762f7a4249] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:08:32.826 INFO 21062 --- [ocp-schedules-26,e97860df0eacf2da,0f74068fa4199515] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:08:32.826 INFO 21062 --- [ocp-schedules-26,e97860df0eacf2da,0f74068fa4199515] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:08:32.933 INFO 21062 --- [ocp-schedules-10,e1e7c8b9b616b583,a9f9b7c0035a4563] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:08:32.983 INFO 21062 --- [prometheus-minute-schedule-1,46e61b76b2147e46,be31dfe0d26c05d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:33.023 INFO 21062 --- [ocp-schedules-13,5b2ecc7525dca7a3,fd4d98c12508a025] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:08:33.023+08:00, clusters=[1] 2024-11-04 15:08:33.024 INFO 21062 --- [ocp-schedules-13,5b2ecc7525dca7a3,fd4d98c12508a025] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:08:33.279 INFO 21062 --- [ocp-schedules-18,cac02be6f94cb8e3,a12c126b6c635cc2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:08:33.279+08:00, clusters=[1] 2024-11-04 15:08:33.282 INFO 21062 --- [ocp-schedules-18,cac02be6f94cb8e3,a12c126b6c635cc2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:08:33.313 INFO 21062 --- [ocp-schedules-2,dcba28e8cf688cfb,149d56c6077f8bf9] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:08:33.317 INFO 21062 --- [ocp-schedules-2,dcba28e8cf688cfb,149d56c6077f8bf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:33.373 ERROR 21062 --- [ocp-schedules-2,dcba28e8cf688cfb,149d56c6077f8bf9] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:08:33.373 INFO 21062 --- [ocp-schedules-2,dcba28e8cf688cfb,149d56c6077f8bf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:08:33.443 WARN 21062 --- [ocp-async-10,0d206f9dffbd2cd0,df42a40a4fd93e3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:33.478 INFO 21062 --- [ocp-async-5,24f2b06a9807ec81,43db1b2dfff062bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:34.871 INFO 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:08:34.871 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:08:34.871 INFO 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:08:34.871 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:08:34.881 INFO 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:08:34.881 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:08:34.881 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:08:34.882 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:08:34.882 WARN 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:08:34.882 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:08:34.883 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:08:34.883 INFO 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:08:34.884 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:08:34.885 WARN 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:08:34.885 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:08:34.885 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:08:34.892 INFO 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:08:34.892 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:08:34.892 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:08:34.903 ERROR 21062 --- [ocp-schedules-10,191d90204d3dbdc7,0f0df301fa0dc979] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:08:35.443 WARN 21062 --- [ocp-async-9,f2ae1d6e17c040ef,0baa5ac8496a1063] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:35.478 INFO 21062 --- [ocp-async-3,2f0d96333064abfe,70802bfe4792166d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:35.740 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:35.741 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:35.983 INFO 21062 --- [prometheus-minute-schedule-1,851045df900d07f9,6d7bcfd489450b29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:37.443 WARN 21062 --- [ocp-async-8,7e1b04b40dbe1781,546d2db2a34ff5dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:37.479 INFO 21062 --- [ocp-async-4,9221d408c59d1f43,6e9215f9574dbba6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:38.505 INFO 21062 --- [ocp-schedules-6,d7b81335d0ce8d5a,14c779a6bb358966] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:08:38.505 INFO 21062 --- [ocp-schedules-1,2c9514a7c3e3b0da,6bb3db6b464e7606] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:08:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:08:38.590332+08:00, lastProcessTime=2024-11-04T15:08:28.588858+08:00 2024-11-04 15:08:38.741 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:38.742 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:38.743 INFO 21062 --- [ocp-async-9,e780e6d16e25b11e,66035c3760803dda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:08:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:08:38.590332+08:00 2024-11-04 15:08:38.983 INFO 21062 --- [prometheus-minute-schedule-1,77ec57db257e7a63,59b06074294719e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:39.203 INFO 21062 --- [ocp-schedules-12,6b25b4ba4572d685,c544b0bc837f6919] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:08:39.203 INFO 21062 --- [ocp-schedules-12,ca4a46c4d7aaf3ed,cf3ed1f05b80d65c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:08:39.207 INFO 21062 --- [ocp-schedules-12,ca4a46c4d7aaf3ed,cf3ed1f05b80d65c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:39.236 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:08:39.236 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:08:39.242 INFO 21062 --- [ocp-schedules-6,0d9cf925f9f38583,5a1d09ba0c3fee47] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:08:39.246 INFO 21062 --- [ocp-schedules-6,0d9cf925f9f38583,5a1d09ba0c3fee47] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:39.246 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:08:39.443 WARN 21062 --- [ocp-async-5,b789811898d4b68d,21df61b6cff3c231] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:39.479 INFO 21062 --- [ocp-async-9,59d9025939e9589f,a3dc3e2d22bc1370] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:40.002 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:08:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:08:41.326 INFO 21062 --- [ocp-schedules-11,c4692ffda0b50346,52559d5a94725c83] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:08:41.443 WARN 21062 --- [ocp-async-3,78f4b4fb8748467b,e491fc1858dbb3fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:41.479 INFO 21062 --- [ocp-async-8,00a65259d9aba503,bbbc67ba825ad6c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:41.742 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:41.743 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:08:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:08:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:08:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:08:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:08:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:08:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:08:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bf48a04c8539ab9c,13922b289f0804f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:42.928 INFO 21062 --- [ocp-schedules-23,35c77a96f7c3869c,13806c15e31357a4] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:08:42.932 INFO 21062 --- [ocp-schedules-23,35c77a96f7c3869c,13806c15e31357a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:42.959 INFO 21062 --- [ocp-schedules-24,a5a1c09b4313b3d7,fdbe6ec4f3942492] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:08:43.022 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:08:43.025 ERROR 21062 --- [ocp-schedules-23,35c77a96f7c3869c,13806c15e31357a4] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:08:43.025 INFO 21062 --- [ocp-schedules-23,35c77a96f7c3869c,13806c15e31357a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:08:43.444 WARN 21062 --- [ocp-async-4,7f9dc4330819848f,277b2ea596c0f151] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:43.479 INFO 21062 --- [ocp-async-6,e658c4af7e10891f,287b0e0c553a5279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:44.743 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:44.744 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:44.983 INFO 21062 --- [prometheus-minute-schedule-1,033be23092e9328f,1c4164cafd1a4df4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:45.444 WARN 21062 --- [ocp-async-5,86f5dc513b7fb77f,3757664b616cab3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:45.479 INFO 21062 --- [ocp-async-7,01843d289b6f542b,50702a97c19e5bc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:47.444 WARN 21062 --- [ocp-async-3,fbab9730a744caef,9aaa61186544127a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:47.480 INFO 21062 --- [ocp-async-1,d6970483afcad923,5dd6ec79134afac0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:47.744 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:47.745 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:47.983 INFO 21062 --- [prometheus-minute-schedule-1,525254491e047110,70758ede0117590d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:08:48.590692+08:00, lastProcessTime=2024-11-04T15:08:38.590332+08:00 2024-11-04 15:08:48.743 INFO 21062 --- [ocp-async-6,454d9ff8f7b887b0,7b0fd1d45d9055b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:08:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:08:48.590692+08:00 2024-11-04 15:08:49.247 INFO 21062 --- [ocp-schedules-6,0d9cf925f9f38583,5a1d09ba0c3fee47] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:08:49.444 WARN 21062 --- [ocp-async-2,c95ab4a9f0400b1c,8842919a40b43dbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:49.480 INFO 21062 --- [ocp-async-6,8a0f27be379b2ab8,78d34485c92f702f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:50.745 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:50.747 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:50.983 INFO 21062 --- [prometheus-minute-schedule-1,226bf9ec8a84f867,d892a6d6e78d8f58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:51.330 INFO 21062 --- [ocp-schedules-13,f53e68b66959f845,fa5a78def650a1e7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:08:51.444 WARN 21062 --- [ocp-async-5,da58801d0942fad4,73cf303860910968] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:51.480 INFO 21062 --- [ocp-async-7,f46d95086d913a7d,20bfc4f1811eb4a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:52.617 INFO 21062 --- [ocp-schedules-9,6921aecf7f7e83d5,f5bd499348cd89bc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:08:52.623 INFO 21062 --- [ocp-schedules-9,6921aecf7f7e83d5,f5bd499348cd89bc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:08:52.623 INFO 21062 --- [ocp-schedules-9,6921aecf7f7e83d5,f5bd499348cd89bc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:08:53.002 INFO 21062 --- [ocp-schedules-15,c1e57e33f11ad9fd,faaac285928e6d55] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:08:53.444 WARN 21062 --- [ocp-async-9,e91d88bb13cb73a6,86645300f75ae221] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:53.480 INFO 21062 --- [ocp-async-1,123137e6ed86da91,d14f21626da97987] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:53.747 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:53.748 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d0d712dfed52e243,51fd8d88bb7a0dca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:55.445 WARN 21062 --- [ocp-async-2,e14174fb03f68d73,c606fd0afeaec7f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:55.480 INFO 21062 --- [ocp-async-4,860cf58c925a6ef0,27ee00e6c5c969f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:56.748 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:56.749 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:56.983 INFO 21062 --- [prometheus-minute-schedule-1,aaa1bfa834daa092,01c37c04657fdb69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:08:57.445 WARN 21062 --- [ocp-async-6,6ca3148f1eec2ac1,4ddabbba24e3ec02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:57.480 INFO 21062 --- [ocp-async-10,87817650ccee2932,77cd5a2cf1f16544] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:58.500 INFO 21062 --- [ocp-schedules-18,452c35425a7b20ba,7607ad12f5330664] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:08:58.500 INFO 21062 --- [ocp-schedules-9,f884ecec9d54bc48,4fa6174e55954a4b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:08:58.502 INFO 21062 --- [ocp-schedules-26,b3d652248e9a0a07,56432aa1e62311ba] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:08:58.504 INFO 21062 --- [ocp-schedules-18,452c35425a7b20ba,7607ad12f5330664] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:58.504 INFO 21062 --- [ocp-schedules-9,f884ecec9d54bc48,4fa6174e55954a4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:08:58.504 INFO 21062 --- [ocp-schedules-20,b658e41c4161cf59,42092fee97eaefda] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:08:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:08:58.510 INFO 21062 --- [ocp-schedules-4,11d2a5f9e8b7aa5b,53402f2c22085a48] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:08:58.510 INFO 21062 --- [ocp-schedules-1,fcaafe390929df75,7c46e344c3f35eed] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:08:58.510 INFO 21062 --- [ocp-schedules-29,5ca173f50ea3a013,75467938630289ea] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:08:58.510 INFO 21062 --- [ocp-schedules-29,5ca173f50ea3a013,75467938630289ea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:08:58.510 INFO 21062 --- [ocp-schedules-4,11d2a5f9e8b7aa5b,53402f2c22085a48] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:08:58.510 INFO 21062 --- [ocp-schedules-1,fcaafe390929df75,7c46e344c3f35eed] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:08:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:08:58.589468+08:00, lastProcessTime=2024-11-04T15:08:48.590692+08:00 2024-11-04 15:08:58.744 INFO 21062 --- [ocp-async-1,d6602e540fcd7969,665643f4374fe27c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:08:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:08:58.589468+08:00 2024-11-04 15:08:59.423 INFO 21062 --- [ocp-schedules-25,a2e73550eaa810e4,41591199bb643bcf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:08:59.445 WARN 21062 --- [ocp-async-9,301464ed696f6d52,9a446728516aed83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:08:59.481 INFO 21062 --- [ocp-async-1,1b84e38ab1f02c5b,b9326baab1c6d1d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:08:59.749 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:08:59.750 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:08:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f75c0e1c5411f48b,56e2834dfe1ee60f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704140, interval=60 2024-11-04 15:09:01.333 INFO 21062 --- [ocp-schedules-21,b13fe47200bde8a5,bc58e47c6e3c7c8c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:09:01.445 WARN 21062 --- [ocp-async-2,e356b5f990b293bd,24562aea1d3c2029] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:01.481 INFO 21062 --- [ocp-async-4,868e9c03a1bafb22,bd7d753fb186af14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:02.750 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:02.751 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1763bc599d8a20a4,a3e88eb7d28481bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:03.029 INFO 21062 --- [ocp-schedules-7,5ded13d7c81fbf21,b76511bd7999f66e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:09:03.445 WARN 21062 --- [ocp-async-5,441e83dfee103d41,6121966f9d2b770f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:03.481 INFO 21062 --- [ocp-async-10,fb22536a5d6138aa,207331a3660f2a37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:05.445 WARN 21062 --- [ocp-async-10,22fe83dbd941c6ce,11b415e14488e8dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:05.481 INFO 21062 --- [ocp-async-3,46d1f37a681a52d6,1e3565d0b41e8bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:05.751 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:05.752 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2e9fbe62c741a006,4db25ed14e073bc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:07.446 WARN 21062 --- [ocp-async-3,413a8016b83f8aec,ad0204502368dc15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:07.481 INFO 21062 --- [ocp-async-8,ff022420f5007f75,ed71f07de9ab68e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:08.505 INFO 21062 --- [ocp-schedules-9,f884ecec9d54bc48,4fa6174e55954a4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:09:08.505 INFO 21062 --- [ocp-schedules-18,452c35425a7b20ba,7607ad12f5330664] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:09:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:09:08.590131+08:00, lastProcessTime=2024-11-04T15:08:58.589468+08:00 2024-11-04 15:09:08.744 INFO 21062 --- [ocp-async-6,02bbdd84e0a610db,3b7a4d3eb9f6f5ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:09:08.752 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:08.753 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:09:08.590131+08:00 2024-11-04 15:09:08.983 INFO 21062 --- [prometheus-minute-schedule-1,8711d84d919cf1d3,919ed5a2326ca650] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:09.237 INFO 21062 --- [ocp-schedules-12,ca4a46c4d7aaf3ed,cf3ed1f05b80d65c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:09:09.237 INFO 21062 --- [ocp-schedules-12,340034245224e4ef,620eaf9e7c2f6b5e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:09:09.241 INFO 21062 --- [ocp-schedules-12,340034245224e4ef,620eaf9e7c2f6b5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:09.273 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:09:09.273 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:09:09.446 WARN 21062 --- [ocp-async-5,98bea902596826e3,93ee8d6a978501c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:09.481 INFO 21062 --- [ocp-async-6,a373e6777a830dee,cb50135cb25ccaf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:09:10.004 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:09:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:09:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:09:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:09:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:09:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:09:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:09:11.336 INFO 21062 --- [ocp-schedules-2,bf256d1dfb71f83a,6a0a8b9976443a30] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:09:11.446 WARN 21062 --- [ocp-async-9,5f6e38ec8b2e8682,33325fa6bd5dc38d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:11.482 INFO 21062 --- [ocp-async-1,2c0bd5c33edf1a1c,d35bdbf0aac6c843] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:11.754 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:11.757 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=8, affectRows=8 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=16, affectRows=16 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=22, affectRows=22 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=15, affectRows=15 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=8, validCount=8 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=15, validCount=15 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=15, validCount=15 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=22, validCount=22 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=29, affectRows=29 2024-11-04 15:09:11.760 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=54, affectRows=54 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=41, affectRows=41 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=33, affectRows=33 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=29, validCount=29 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=41, validCount=41 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=33, validCount=33 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=54, validCount=54 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=68, affectRows=68 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=51, affectRows=51 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=51, validCount=51 2024-11-04 15:09:11.761 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=67, validCount=67 2024-11-04 15:09:11.762 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704080, nodeCount=85, affectRows=85 2024-11-04 15:09:11.762 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704080, nodeCount=84, validCount=84 2024-11-04 15:09:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a1d5cbf66a4a52fb,701486fa220b14f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:12.626 INFO 21062 --- [ocp-schedules-29,06bfd31abffc7391,9ceb7551e8324a07] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:09:12.632 INFO 21062 --- [ocp-schedules-29,06bfd31abffc7391,9ceb7551e8324a07] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:09:12.632 INFO 21062 --- [ocp-schedules-29,06bfd31abffc7391,9ceb7551e8324a07] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:09:13.071 INFO 21062 --- [ocp-schedules-2,ab02efc063aa618d,09f4dcac04d4e50e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:09:13.446 WARN 21062 --- [ocp-async-2,26fe738d99b821ab,0f63324e6a783bb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:13.482 INFO 21062 --- [ocp-async-8,282b93f0b082ee98,73d1f02e35370b08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:13.984 INFO 21062 --- [prometheus-cache-monitor1,b419bcc8ade06338,c56a1dc603f3c2da] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:09:13.984 INFO 21062 --- [prometheus-cache-monitor1,b419bcc8ade06338,c56a1dc603f3c2da] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:09:14.757 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:14.759 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:14.983 INFO 21062 --- [prometheus-minute-schedule-1,10e8fb3e6df96a9a,7551ef884d054f2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:15.446 WARN 21062 --- [ocp-async-5,6304dbe218bbdc5c,1f2829e99437f147] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:15.461 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:05:15.441, end=2024-11-04T15:07:15.458} 2024-11-04 15:09:15.466 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:09:15.466 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:09:15.482 INFO 21062 --- [ocp-async-10,df95b534ac660ce0,573ac8e5dcf0ac63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:17.447 WARN 21062 --- [ocp-async-9,63e9c760caacf5c4,d2a30ad66625a573] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:17.482 INFO 21062 --- [ocp-async-7,deb688c278ec4c84,4ec3338e4b11f1c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:17.759 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:17.760 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4b62a8bbbd981c14,eecc09a16268c9c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:09:18.590473+08:00, lastProcessTime=2024-11-04T15:09:08.590131+08:00 2024-11-04 15:09:18.744 INFO 21062 --- [ocp-async-3,71655fa92ede6c8e,44fc2f4660287707] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:09:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:09:18.590473+08:00 2024-11-04 15:09:19.447 WARN 21062 --- [ocp-async-2,7dfd43c0fef00bed,a4a852391fb638b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:19.482 INFO 21062 --- [ocp-async-5,76920e3b665f056e,af929f3e4cc89e03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:20.760 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:20.761 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8b155647d6b71c13,d5678499295d29b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:21.339 INFO 21062 --- [ocp-schedules-26,53468f3f4114b113,8024e9f2f0bd94bf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:09:21.447 WARN 21062 --- [ocp-async-5,403206d3b954a406,8662586b526ec71b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:21.482 INFO 21062 --- [ocp-async-9,2282ffb41505a9c5,cddac0b5e2f6cad7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:23.095 INFO 21062 --- [ocp-schedules-27,8be30b811e58c97d,b374157e24dac696] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:09:23.447 WARN 21062 --- [ocp-async-9,102efe035c60021b,63ba72e8a7eb862a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:23.483 INFO 21062 --- [ocp-async-7,1bf715cecb344169,b2b356339c725554] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:23.761 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:23.762 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:23.983 INFO 21062 --- [prometheus-minute-schedule-1,2f3b2bdfb03fa247,3438d615a90f0dd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:25.447 WARN 21062 --- [ocp-async-7,774eefdaee67501c,e18ba32e25b45481] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:25.483 INFO 21062 --- [ocp-async-1,32da485ebd86c666,7ab8008ffae00863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:26.762 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:26.763 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f4dfe600c09b67d3,49aa2b79a6896c73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:26.984 INFO 21062 --- [prometheus-hour-schedule-1,cb35de1aad1001ee,3140586bdfb8aa44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:09:27.448 WARN 21062 --- [ocp-async-1,348d333cdf2bf403,3ecc5f346c9e45d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:27.483 INFO 21062 --- [ocp-async-8,4aa86bc65c168f7c,a6a82e2c7b30772b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:28.498 INFO 21062 --- [ocp-schedules-21,56ef3a5c5dc5fa80,0b94db9b0614ff46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:09:28.500 INFO 21062 --- [ocp-schedules-22,298ddabf8f35ff1a,5f77c110df24f3e1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:09:28.500 INFO 21062 --- [ocp-schedules-14,690f5cfad1f457da,5f8c2a9e74049bec] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:09:28.500 INFO 21062 --- [ocp-schedules-3,3557ba4117a61bc0,987271f3d4360c4c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:09:28.502 INFO 21062 --- [ocp-schedules-21,56ef3a5c5dc5fa80,0b94db9b0614ff46] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:09:28.502 INFO 21062 --- [ocp-schedules-6,7b050fce571a556b,5b2953c6df0c99ee] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:09:28.503 INFO 21062 --- [ocp-schedules-22,298ddabf8f35ff1a,5f77c110df24f3e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:28.504 INFO 21062 --- [ocp-schedules-3,3557ba4117a61bc0,987271f3d4360c4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:09:28.504 INFO 21062 --- [ocp-schedules-30,739220ef04cbbd0e,de0b8a3c7177ba20] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:09:28.505 INFO 21062 --- [ocp-schedules-14,690f5cfad1f457da,5f8c2a9e74049bec] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:09:28.509 INFO 21062 --- [ocp-schedules-8,13a92870d5cca29f,f6d81871f33ac0ea] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:09:28.509 INFO 21062 --- [ocp-schedules-8,13a92870d5cca29f,f6d81871f33ac0ea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:09:28.510 INFO 21062 --- [ocp-schedules-4,e082364e0ed3d981,315574ed0fef6f87] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:09:28.510 INFO 21062 --- [ocp-schedules-4,e082364e0ed3d981,315574ed0fef6f87] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:09:28.510 INFO 21062 --- [ocp-schedules-10,e5fdb03aa01274d7,b0f9e60e6f11a9d1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:09:28.510 INFO 21062 --- [ocp-schedules-10,e5fdb03aa01274d7,b0f9e60e6f11a9d1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:09:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:09:28.590349+08:00, lastProcessTime=2024-11-04T15:09:18.590473+08:00 2024-11-04 15:09:28.612 INFO 21062 --- [ocp-schedules-27,7a8321c273b092e6,e4821ef635f7f347] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:09:28.612+08:00, tenantSize=0 2024-11-04 15:09:28.655 INFO 21062 --- [ocp-schedules-15,4d97d1b109d45e0c,1355898c7711ed64] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:09:28.656 INFO 21062 --- [ocp-schedules-15,4d97d1b109d45e0c,1355898c7711ed64] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:09:28.744 INFO 21062 --- [ocp-async-2,259421f749d7df69,877ecf8e024ef434] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:09:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:09:28.590349+08:00 2024-11-04 15:09:28.948 INFO 21062 --- [ocp-schedules-24,ed0f10217accca20,a2e1abe8ce29047e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:09:29.306 INFO 21062 --- [ocp-schedules-4,85253beea20e9ae8,8df55c6889e2709a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:09:29.306 INFO 21062 --- [ocp-schedules-4,85253beea20e9ae8,8df55c6889e2709a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:09:29.425 INFO 21062 --- [ocp-schedules-5,49be472df066b4bc,4d05ab59a6a05ed8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:09:29.448 WARN 21062 --- [ocp-async-7,7e0585f741c67c95,3096801bc7bdd34f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:29.452 INFO 21062 --- [ocp-schedules-25,e7ddb2b5d59ab258,431ec79cb069fb8f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:09:29.458 INFO 21062 --- [ocp-schedules-25,e7ddb2b5d59ab258,431ec79cb069fb8f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:09:29.458 INFO 21062 --- [ocp-schedules-25,e7ddb2b5d59ab258,431ec79cb069fb8f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:09:29.483 INFO 21062 --- [ocp-async-2,f400a2d4ae934459,3a4f559d47072f99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:29.764 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:29.764 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:29.901 INFO 21062 --- [ocp-schedules-2,197956583f8bc30e,022e7cefa2698498] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:09:29.901 INFO 21062 --- [ocp-schedules-2,197956583f8bc30e,022e7cefa2698498] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:09:29.903 INFO 21062 --- [ocp-schedules-2,197956583f8bc30e,022e7cefa2698498] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:09:29.903 INFO 21062 --- [ocp-schedules-2,197956583f8bc30e,022e7cefa2698498] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:09:29.904 INFO 21062 --- [ocp-schedules-2,197956583f8bc30e,022e7cefa2698498] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:09:29.904 INFO 21062 --- [ocp-schedules-2,197956583f8bc30e,022e7cefa2698498] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:09:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e0a3a969dcf37206,61a8554f89bce3b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:09:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:09:31.342 INFO 21062 --- [ocp-schedules-23,cb9ca046b2917fb7,91c2a36307acac26] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:09:31.448 WARN 21062 --- [ocp-async-1,1ba0559824970c14,f9cd6531504be5c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:31.483 INFO 21062 --- [ocp-async-5,b03cce9ff125e13a,c7be497910fecee3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:32.634 INFO 21062 --- [ocp-schedules-4,048b5509d9dc36d3,1e535113e0c2fd36] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:09:32.640 INFO 21062 --- [ocp-schedules-4,048b5509d9dc36d3,1e535113e0c2fd36] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:09:32.640 INFO 21062 --- [ocp-schedules-4,048b5509d9dc36d3,1e535113e0c2fd36] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:09:32.764 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:32.766 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:32.826 INFO 21062 --- [ocp-schedules-10,9c0722eaff051ea9,04b3556fc2ec9358] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:09:32.826 INFO 21062 --- [ocp-schedules-10,9c0722eaff051ea9,04b3556fc2ec9358] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:09:32.854 INFO 21062 --- [ocp-schedules-10,9c0722eaff051ea9,04b3556fc2ec9358] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:09:32.854 INFO 21062 --- [ocp-schedules-10,9c0722eaff051ea9,04b3556fc2ec9358] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:09:32.855 INFO 21062 --- [ocp-schedules-6,083b0b038b770f3a,f0d56e426503ec52] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:09:32.983 INFO 21062 --- [prometheus-minute-schedule-1,5c83c30f785b8c76,39424884f2bad894] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:33.049 INFO 21062 --- [ocp-schedules-5,87eac0a8542306ab,15d5b62f5db46252] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:09:33.049+08:00, clusters=[1] 2024-11-04 15:09:33.050 INFO 21062 --- [ocp-schedules-5,87eac0a8542306ab,15d5b62f5db46252] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:09:33.138 INFO 21062 --- [ocp-schedules-19,06fc6bbf0e0831ad,5cc8626609211d5e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:09:33.309 INFO 21062 --- [ocp-schedules-11,8ff3920fd43a5d9b,3147c4526ec94eac] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:09:33.309+08:00, clusters=[1] 2024-11-04 15:09:33.312 INFO 21062 --- [ocp-schedules-11,8ff3920fd43a5d9b,3147c4526ec94eac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:09:33.448 WARN 21062 --- [ocp-async-5,1238420cbde18b9d,5bc1647a7a6fe861] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:33.484 INFO 21062 --- [ocp-async-9,098ffe7573679ea8,435a2f69b873a7f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:34.912 INFO 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:09:34.913 INFO 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:09:34.913 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:09:34.913 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:09:34.923 INFO 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:09:34.924 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:09:34.924 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:09:34.925 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:09:34.925 WARN 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:09:34.926 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:09:34.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:09:34.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:09:34.926 INFO 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:09:34.927 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:09:34.928 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:09:34.928 WARN 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:09:34.935 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:09:34.935 INFO 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:09:34.935 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:09:34.948 ERROR 21062 --- [ocp-schedules-6,f1ed350d19782280,4d539799921ddccd] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:09:35.448 WARN 21062 --- [ocp-async-10,51f6a2436a4a9252,e80f5ec4e1748463] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:35.484 INFO 21062 --- [ocp-async-6,765b26e8b6891b30,55061f8c6328660f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:35.766 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:35.767 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:35.983 INFO 21062 --- [prometheus-minute-schedule-1,f99ed97d4437cd05,d791314083af9f3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:37.448 WARN 21062 --- [ocp-async-1,a401ffc49ec42676,d4bc8e18fe381400] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:37.484 INFO 21062 --- [ocp-async-2,d848eeae5dad1f67,bc09e95ee0903d77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:38.504 INFO 21062 --- [ocp-schedules-3,3557ba4117a61bc0,987271f3d4360c4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:09:38.504 INFO 21062 --- [ocp-schedules-22,298ddabf8f35ff1a,5f77c110df24f3e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:09:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:09:38.589644+08:00, lastProcessTime=2024-11-04T15:09:28.590349+08:00 2024-11-04 15:09:38.744 INFO 21062 --- [ocp-async-10,1e97a6adfc2aef46,62bafab2467c38f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:09:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:09:38.589644+08:00 2024-11-04 15:09:38.767 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:38.768 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:38.983 INFO 21062 --- [prometheus-minute-schedule-1,01de538163110a8d,4ac22100aaa1bed4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:39.274 INFO 21062 --- [ocp-schedules-12,340034245224e4ef,620eaf9e7c2f6b5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:09:39.274 INFO 21062 --- [ocp-schedules-12,1c6fc09aee5e8e0e,0bccf465b0a384ef] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:09:39.278 INFO 21062 --- [ocp-schedules-12,1c6fc09aee5e8e0e,0bccf465b0a384ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:39.307 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:09:39.307 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:09:39.448 WARN 21062 --- [ocp-async-9,1339edebb0bf1c67,edb79706aa2c1016] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:39.484 INFO 21062 --- [ocp-async-10,0724a126f01771af,3a8f1c8c0c66d31a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:09:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:09:41.345 INFO 21062 --- [ocp-schedules-2,5b8ebb19c088fc7c,07fc3f4fc99b8a5d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:09:41.449 WARN 21062 --- [ocp-async-6,5b60abf2a07aa901,da4cfacdf6983ebf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:41.484 INFO 21062 --- [ocp-async-1,5c2d9f1f9bbfa607,809688f9e2a9598c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:41.768 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:41.769 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2c93c4e25043c72c,b14653153b6a0004] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:43.026 INFO 21062 --- [ocp-schedules-25,18ad26ebf82f7b20,211a758bbc9bf0c4] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:09:43.030 INFO 21062 --- [ocp-schedules-25,18ad26ebf82f7b20,211a758bbc9bf0c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:43.115 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:09:43.117 ERROR 21062 --- [ocp-schedules-25,18ad26ebf82f7b20,211a758bbc9bf0c4] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:09:43.117 INFO 21062 --- [ocp-schedules-25,18ad26ebf82f7b20,211a758bbc9bf0c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:09:43.165 INFO 21062 --- [ocp-schedules-1,cfcd610fd4c770aa,1ddc675ec192022f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:09:43.449 WARN 21062 --- [ocp-async-2,5d5e893b30a452cb,154bd6f19057e7bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:43.485 INFO 21062 --- [ocp-async-4,239d25367983982a,95f6f269d286ad4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:44.769 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:44.770 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:44.983 INFO 21062 --- [prometheus-minute-schedule-1,4ca3c0c56e43ed08,b002a808f81c90e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:45.449 WARN 21062 --- [ocp-async-8,3791d9516942cb12,81946528d410125c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:45.485 INFO 21062 --- [ocp-async-5,878c093f449ee351,82c63b31c039fe59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:47.449 WARN 21062 --- [ocp-async-6,8e972ca07fed8ec4,85e30b57f2f4a351] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:47.485 INFO 21062 --- [ocp-async-7,0ab13c8417991182,adc8075ec2787458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:47.770 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:47.771 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:47.983 INFO 21062 --- [prometheus-minute-schedule-1,faf480601779f67c,1e84ea9e0f726074] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:09:48.591807+08:00, lastProcessTime=2024-11-04T15:09:38.589644+08:00 2024-11-04 15:09:48.744 INFO 21062 --- [ocp-async-3,6df0bb52c259e017,529e8d9061d1086b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:09:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:09:48.591807+08:00 2024-11-04 15:09:49.247 INFO 21062 --- [ocp-schedules-9,eed10f0d9ef997f9,a2a95a16c19ba883] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:09:49.250 INFO 21062 --- [ocp-schedules-9,eed10f0d9ef997f9,a2a95a16c19ba883] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:49.251 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:09:49.449 WARN 21062 --- [ocp-async-2,b3b1f20f18e26d9d,058dec539b618cdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:49.485 INFO 21062 --- [ocp-async-3,c09056d07c176053,10dc531ccfc3b65b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:50.771 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:50.773 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:50.983 INFO 21062 --- [prometheus-minute-schedule-1,44f8dbe5e051b682,d849a1324c09a269] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:51.347 INFO 21062 --- [ocp-schedules-6,a7afbbf43d91a1ec,091b917b5017a614] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:09:51.449 WARN 21062 --- [ocp-async-9,6d0f97874e86e58e,e62023010c83ec40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:51.485 INFO 21062 --- [ocp-async-5,f6c880ec0d0eb318,e29b392d39625c0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:52.641 INFO 21062 --- [ocp-schedules-6,a44b7f263b7df202,72f201a57464b08d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:09:52.647 INFO 21062 --- [ocp-schedules-6,a44b7f263b7df202,72f201a57464b08d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:09:52.647 INFO 21062 --- [ocp-schedules-6,a44b7f263b7df202,72f201a57464b08d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:09:53.206 INFO 21062 --- [ocp-schedules-10,e3b10f6cd2221919,0ef18be579596c80] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:09:53.450 WARN 21062 --- [ocp-async-6,ac6f35a948ab7380,cc49b0c2f2ebf14d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:53.485 INFO 21062 --- [ocp-async-7,a84588716dc717a7,b985fc985fbb1f84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:53.773 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:53.773 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:53.983 INFO 21062 --- [prometheus-minute-schedule-1,80054066ca71c2c9,6b19b52ea58e323f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:55.450 WARN 21062 --- [ocp-async-1,fd0f4e6d6ae3b59a,0795fc60c2cca52f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:55.486 INFO 21062 --- [ocp-async-4,da9bcc2f2201dd75,9b2b5b655d6ab159] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:56.774 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:56.775 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fde03816a15d87e2,437bca545c06d1a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:09:57.450 WARN 21062 --- [ocp-async-3,46374397fb0dc090,95c0fc710e1da530] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:57.486 INFO 21062 --- [ocp-async-8,bd92b0788040920b,5aea68490e789202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:58.499 INFO 21062 --- [ocp-schedules-8,2a20cac2e31ae97c,f3888666a489f4d4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:09:58.500 INFO 21062 --- [ocp-schedules-6,b5160f745ad73f08,52beab0ac5ffb855] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:09:58.502 INFO 21062 --- [ocp-schedules-4,66698c5df2d41c38,a4b604418c766353] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:09:58.504 INFO 21062 --- [ocp-schedules-8,2a20cac2e31ae97c,f3888666a489f4d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:58.504 INFO 21062 --- [ocp-schedules-6,b5160f745ad73f08,52beab0ac5ffb855] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:09:58.507 INFO 21062 --- [ocp-schedules-5,30200238d0bd0838,01e5a3f1e95d49b9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:09:58.507 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:09:58.509 INFO 21062 --- [ocp-schedules-14,d5f350a7022abdf4,a37d502cbbaab900] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:09:58.509 INFO 21062 --- [ocp-schedules-14,d5f350a7022abdf4,a37d502cbbaab900] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:09:58.509 INFO 21062 --- [ocp-schedules-21,c8bb669c57b0386e,fa48a3a9ff122fd2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:09:58.509 INFO 21062 --- [ocp-schedules-21,c8bb669c57b0386e,fa48a3a9ff122fd2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:09:58.509 INFO 21062 --- [ocp-schedules-17,9290e6e183f060ac,6afdcfa4c089b4fb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:09:58.509 INFO 21062 --- [ocp-schedules-17,9290e6e183f060ac,6afdcfa4c089b4fb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:09:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:09:58.589297+08:00, lastProcessTime=2024-11-04T15:09:48.591807+08:00 2024-11-04 15:09:58.745 INFO 21062 --- [ocp-async-5,10a0101a74f40b14,425485ada473150d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:09:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:09:58.589297+08:00 2024-11-04 15:09:59.251 INFO 21062 --- [ocp-schedules-9,eed10f0d9ef997f9,a2a95a16c19ba883] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:09:59.428 INFO 21062 --- [ocp-schedules-20,cd02061cbbc9e09b,cb0e8268141681f5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:09:59.450 WARN 21062 --- [ocp-async-10,e1f9f947844dfa15,a40f5b42db6d0f5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:09:59.486 INFO 21062 --- [ocp-async-1,65775d7896dafe15,628152dcbc6c41ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:09:59.775 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:09:59.776 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:09:59.983 INFO 21062 --- [prometheus-minute-schedule-1,68503d06ef1f1d0f,fe2ba8039eb34400] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704200, interval=60 2024-11-04 15:10:00.417 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,08b7090573a96036] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:05:01.0, nextRun=2024-11-04 15:10:00.0, currentDate=Mon Nov 04 15:10:00 CST 2024 2024-11-04 15:10:00.417 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,08b7090573a96036] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:05:01.0, nextRunTime=2024-11-04 15:10:00.0} 2024-11-04 15:10:00.445 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,08b7090573a96036] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:10:00.449 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,08b7090573a96036] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:10:00.449 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,08b7090573a96036] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:10:01.350 INFO 21062 --- [ocp-schedules-3,24f35f98ef71bf9a,4c116c58d923cd7f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:10:01.450 WARN 21062 --- [ocp-async-2,48f84f68d9a2acb3,8be1bae0c8dbb50b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:01.486 INFO 21062 --- [ocp-async-3,31383ef0873060d0,35e5d0316cb1adb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:02.776 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:02.777 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:02.983 INFO 21062 --- [prometheus-minute-schedule-1,3ba0591698c5f244,f06797c90687fcef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:03.231 INFO 21062 --- [ocp-schedules-16,3a360007028819fb,08d132619d365f9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:10:03.451 WARN 21062 --- [ocp-async-3,ad116a5994badea9,943784460695afa8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:03.486 INFO 21062 --- [ocp-async-8,fd487d29e7159e80,f0042c48ae7d54cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:04.518 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecb9632799e7c251] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:00:05.0, nextRun=2024-11-04 15:10:04.0, currentDate=Mon Nov 04 15:10:04 CST 2024 2024-11-04 15:10:04.518 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecb9632799e7c251] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 15:00:05.0, nextRunTime=2024-11-04 15:10:04.0} 2024-11-04 15:10:04.525 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecb9632799e7c251] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 15:10:04.525 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecb9632799e7c251] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T15:10:04.525+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 15:10:05.383 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,bd8addc65d45eb63] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 86, operation:EXECUTE, state: RUNNING 2024-11-04 15:10:05.403 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=86, context=Context{parallelIdx=-1, stringMap={task_instance_id=23, task_operation=execute, latest_execution_start_time=2024-11-04T15:10:05.375+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=86}, listMap={}}, executor=10.46.128.24 2024-11-04 15:10:05.408 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:10:05.409 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:10:05.411 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.411 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:10:05.413 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.413 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.414 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.414 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:10:05.429 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.429 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.430 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.430 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:10:05.443 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.443 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.444 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.444 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:10:05.451 WARN 21062 --- [ocp-async-10,506534cd5e04e43d,77bd0a52bf19a117] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:05.459 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.459 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.460 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.460 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:10:05.473 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.474 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.474 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.475 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:10:05.481 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.481 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.482 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.482 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:10:05.487 INFO 21062 --- [ocp-async-5,b136ae76ab45342e,130a5f5800ddeb41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:05.487 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.487 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.488 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.488 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:10:05.492 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.493 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.493 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.494 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:10:05.500 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.501 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.501 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.502 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:10:05.506 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.507 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.507 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.508 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:10:05.514 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.514 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.515 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.515 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:10:05.523 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.524 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.524 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.525 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:10:05.539 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.539 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.540 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.540 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:10:05.545 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.545 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.546 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.546 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:10:05.553 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.553 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.554 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.554 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:10:05.560 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.561 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.561 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.561 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:10:05.566 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.566 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.567 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.567 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:10:05.572 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.572 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.572 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.573 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:10:05.577 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.578 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.578 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.579 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:10:05.583 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.584 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.584 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.585 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:10:05.588 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.589 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.590 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.590 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:10:05.595 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.595 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.596 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.596 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:10:05.600 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.601 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.601 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.601 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:10:05.606 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.606 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.607 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.607 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:10:05.611 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.611 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.612 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.612 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:10:05.616 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.617 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.617 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.617 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:10:05.622 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.622 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.622 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.623 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:10:05.627 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.628 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.628 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.628 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:10:05.632 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.633 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.633 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.633 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:10:05.638 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.638 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.639 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.639 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:10:05.644 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.645 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.645 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.645 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:10:05.650 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.650 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.651 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.651 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:10:05.655 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.656 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.656 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.657 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:10:05.661 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.661 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.662 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.662 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:10:05.669 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.670 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.670 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.671 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:10:05.676 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.676 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.677 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.677 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:10:05.683 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.684 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.684 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.685 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:10:05.691 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.691 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.692 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.692 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:10:05.696 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.696 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.697 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.697 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:10:05.700 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.701 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.701 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.702 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:10:05.705 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.706 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.706 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.707 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:10:05.710 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.711 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.711 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.712 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:10:05.720 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.720 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.721 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.721 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:10:05.741 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.741 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.742 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.743 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:10:05.748 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.749 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.749 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.749 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:10:05.755 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.755 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.756 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.756 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:10:05.762 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.762 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.763 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.763 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:10:05.767 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.767 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.768 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.768 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:10:05.772 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.772 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.773 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.774 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:10:05.777 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:05.778 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:05.779 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:10:05.779 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:10:05.780 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.781 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:10:05.782 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.782 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.782 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.783 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:10:05.788 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.789 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:10:05.789 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:05.790 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:10:05.794 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:10:05.802 INFO 21062 --- [subtask-executor19,dd193f8f67a5a423,da2f5370b8def0dd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 86, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 15:10:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2652b98f8befc2c1,6f06483b1071ba94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:06.675 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:23, status:SUCCESS 2024-11-04 15:10:06.676 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:23, status:SUCCESS 2024-11-04 15:10:07.451 WARN 21062 --- [ocp-async-5,eb985b33c62a0fd1,1194be8970d1203f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:07.487 INFO 21062 --- [ocp-async-7,1d0c74327d35ca5e,befe0b5a63a0fd58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:08.507 INFO 21062 --- [ocp-schedules-8,2a20cac2e31ae97c,f3888666a489f4d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:10:08.507 INFO 21062 --- [ocp-schedules-6,b5160f745ad73f08,52beab0ac5ffb855] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:10:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:10:08.591189+08:00, lastProcessTime=2024-11-04T15:09:58.589297+08:00 2024-11-04 15:10:08.745 INFO 21062 --- [ocp-async-9,360a3f8777d40097,c18939b397589a55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:10:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:10:08.591189+08:00 2024-11-04 15:10:08.778 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:08.779 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:08.983 INFO 21062 --- [prometheus-minute-schedule-1,461ced4d06d9f6cf,de3f37d2a238cc4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:09.308 INFO 21062 --- [ocp-schedules-12,1c6fc09aee5e8e0e,0bccf465b0a384ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:10:09.308 INFO 21062 --- [ocp-schedules-12,32c52e8aadcb5d6b,a8ae98c0740ed8a0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:10:09.312 INFO 21062 --- [ocp-schedules-12,32c52e8aadcb5d6b,a8ae98c0740ed8a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:09.344 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:10:09.344 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:10:09.451 WARN 21062 --- [ocp-async-3,2bf2c1b7722de6d6,59a5649d93eed9e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:09.487 INFO 21062 --- [ocp-async-10,e2c17110effec77e,145801cf3a65cf30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:10:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:10:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:10:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:10:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:10:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:10:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:10:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:10:11.352 INFO 21062 --- [ocp-schedules-22,fd0aedaf11513e8c,c064db423e296cf1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:10:11.451 WARN 21062 --- [ocp-async-10,2afa3e8902606fa9,be92192a946740f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:11.487 INFO 21062 --- [ocp-async-5,b40618d794129fb7,01e4efa9145c65dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:11.764 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=3, affectRows=3 2024-11-04 15:10:11.764 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=3, validCount=3 2024-11-04 15:10:11.765 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=13, affectRows=13 2024-11-04 15:10:11.765 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=12, validCount=12 2024-11-04 15:10:11.765 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=35, affectRows=35 2024-11-04 15:10:11.765 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=33, affectRows=33 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=35, validCount=35 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=33, validCount=33 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=46, affectRows=46 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=27, affectRows=27 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=27, validCount=27 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=46, validCount=46 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=56, affectRows=56 2024-11-04 15:10:11.766 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=56, validCount=56 2024-11-04 15:10:11.767 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=104, affectRows=104 2024-11-04 15:10:11.767 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=103, validCount=103 2024-11-04 15:10:11.767 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704140, nodeCount=105, affectRows=105 2024-11-04 15:10:11.768 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704140, nodeCount=104, validCount=104 2024-11-04 15:10:11.779 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:11.780 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:11.983 INFO 21062 --- [prometheus-minute-schedule-1,037a69ee8caf20b2,0002473c84e5b8c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:12.650 INFO 21062 --- [ocp-schedules-20,d5388ff702e2341e,aa3bbb212c569495] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:10:12.654 INFO 21062 --- [ocp-schedules-20,d5388ff702e2341e,aa3bbb212c569495] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:10:12.654 INFO 21062 --- [ocp-schedules-20,d5388ff702e2341e,aa3bbb212c569495] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:10:13.137 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:10:13.137 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:10:13.146 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:10:13.146 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:10:13.146 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:10:13.146 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:10:13.152 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:10:13.152 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:10:13.152 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:10:13.152 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:10:13.191 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:10:13.199 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:10:13.199 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:10:13.200 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:10:13.200 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:10:13.200 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:10:13.209 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:10:13.273 INFO 21062 --- [ocp-schedules-7,2c4d6c3647f67ae8,5b74cd034432404a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:10:13.452 WARN 21062 --- [ocp-async-2,67fb974e45c42ced,b9178a1d4bfbbbb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:13.487 INFO 21062 --- [ocp-async-7,ca65f9face6065dc,7b75b4087a10a810] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:13.756 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:10:13.984 INFO 21062 --- [prometheus-cache-monitor1,671962a7512d7822,050be96c4d1e0e1f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:10:13.984 INFO 21062 --- [prometheus-cache-monitor1,671962a7512d7822,050be96c4d1e0e1f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:10:14.351 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.199350075S 2024-11-04 15:10:14.351 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:10:14.448 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:10:14.448 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:10:14.473 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:10:14.531 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:10:14.531 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:10:14.531 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:10:14.531 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:10:14.531 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:10:14.531 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:10:14.542 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:10:14.717 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] o.s.boot.SpringApplication : Started application in 1.571 seconds (JVM running for 9628.397) 2024-11-04 15:10:14.717 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.571946162S 2024-11-04 15:10:14.780 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:14.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2f1a70bf2a528172,4634610fced30c36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:15.165 INFO 21062 --- [ocp-schedules-10,58f155b7b2113095,c0fa5874c083d65f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:10:15.452 WARN 21062 --- [ocp-async-7,49c6bd4111d5d2f6,39177edeea47bcfc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:15.487 INFO 21062 --- [ocp-async-4,2c34aa62a5e66227,0f7259347e2e66f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:17.452 WARN 21062 --- [ocp-async-10,004548517bc1c85b,84440ae144b64d3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:17.488 INFO 21062 --- [ocp-async-8,2d8a35a041b03982,9846ed9c60c5d582] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:17.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:17.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:17.983 INFO 21062 --- [prometheus-minute-schedule-1,8805ffd45e6b51cc,fff46da32a7cee40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:10:18.591011+08:00, lastProcessTime=2024-11-04T15:10:08.591189+08:00 2024-11-04 15:10:18.745 INFO 21062 --- [ocp-async-1,95662073479a9ff6,bc12e49bc3366258] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:10:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:10:18.591011+08:00 2024-11-04 15:10:19.452 WARN 21062 --- [ocp-async-5,0c0c9679bf6f50ed,1f99a9792396ad4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:19.488 INFO 21062 --- [ocp-async-1,daf0d4274c15048c,6a9b027f0d4c1551] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:20.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:20.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e5522653f7029cf7,95b5e54c82ff8222] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:21.355 INFO 21062 --- [ocp-schedules-22,e980661754044da0,8280ce678786d8ee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:10:21.452 WARN 21062 --- [ocp-async-3,a38409083d26c169,1cd9eee68d91366c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:21.488 INFO 21062 --- [ocp-async-9,5a5ec3085287a3ba,8d1153350d5950a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:23.303 INFO 21062 --- [ocp-schedules-18,6fc0f0a1db1672d1,3fff3a35e79f953d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:10:23.453 WARN 21062 --- [ocp-async-4,c5bb8d87469b3578,672b44426c7988df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:23.488 INFO 21062 --- [ocp-async-6,0b49f504830e751a,92c4bc1d2a7212cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:23.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:23.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b394406ab8f1fe84,ce3b746697fa74d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:25.453 WARN 21062 --- [ocp-async-5,1370d6264a7d939a,6bd0337dd963c313] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:25.488 INFO 21062 --- [ocp-async-2,eebf7c62c642fd13,81fabe01f972ddc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:26.784 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:26.784 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:26.983 INFO 21062 --- [prometheus-minute-schedule-1,39c82298720608ac,242bc61864d381e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:26.984 INFO 21062 --- [prometheus-hour-schedule-1,dd42f3c6cb463f38,c00d4f71d6cb51b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:10:27.453 WARN 21062 --- [ocp-async-1,f775f93e13143d7e,0d147e16daa0aeea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:27.488 INFO 21062 --- [ocp-async-7,6cec252fd2fcea1a,44c69f1430eb869b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:28.498 INFO 21062 --- [ocp-schedules-26,770727d585295e45,65447748c495b8c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:10:28.499 INFO 21062 --- [ocp-schedules-3,d5144b8c54799403,b06e26f107e557de] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:10:28.500 INFO 21062 --- [ocp-schedules-8,1906009c271b9b61,4474791e80ef5bad] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:10:28.500 INFO 21062 --- [ocp-schedules-18,3f4c0e2441ea8d91,5db6ddebe76e40f0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:10:28.502 INFO 21062 --- [ocp-schedules-10,0ce5a82933851898,32a82537a21b6eba] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:10:28.503 INFO 21062 --- [ocp-schedules-8,1906009c271b9b61,4474791e80ef5bad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:28.503 INFO 21062 --- [ocp-schedules-26,770727d585295e45,65447748c495b8c8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:10:28.504 INFO 21062 --- [ocp-schedules-18,3f4c0e2441ea8d91,5db6ddebe76e40f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:10:28.504 INFO 21062 --- [ocp-schedules-16,73abf582500a93a5,d0b7055dd81478f6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:10:28.506 INFO 21062 --- [ocp-schedules-3,d5144b8c54799403,b06e26f107e557de] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:10:28.510 INFO 21062 --- [ocp-schedules-21,2e5ffce0edf99669,cd38fc68779b3df1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:10:28.510 INFO 21062 --- [ocp-schedules-27,dc4b6047d2162e79,9d52679ce770d62b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:10:28.510 INFO 21062 --- [ocp-schedules-7,4ffc90cd71cc2946,0d96acfaa1c498d6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:10:28.510 INFO 21062 --- [ocp-schedules-21,2e5ffce0edf99669,cd38fc68779b3df1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:10:28.510 INFO 21062 --- [ocp-schedules-7,4ffc90cd71cc2946,0d96acfaa1c498d6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:10:28.510 INFO 21062 --- [ocp-schedules-27,dc4b6047d2162e79,9d52679ce770d62b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:10:28.585 INFO 21062 --- [ocp-schedules-13,867499c506a758a0,be1dfc678b305981] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:10:28.585 INFO 21062 --- [ocp-schedules-13,867499c506a758a0,be1dfc678b305981] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:10:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:10:28.590179+08:00, lastProcessTime=2024-11-04T15:10:18.591011+08:00 2024-11-04 15:10:28.745 INFO 21062 --- [ocp-async-8,5c6471d3a15b45fb,7035353e93e795ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:10:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:10:28.590179+08:00 2024-11-04 15:10:28.927 INFO 21062 --- [ocp-schedules-22,4f50306573e08fc7,d3544a2a676e0177] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:10:28.927 INFO 21062 --- [ocp-schedules-22,4f50306573e08fc7,d3544a2a676e0177] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:10:28.950 INFO 21062 --- [ocp-schedules-3,df85d1449d9c0d5b,05bf714a9b2d3512] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:10:29.310 INFO 21062 --- [ocp-schedules-26,09cada81952f58fc,a268766c815a2d32] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:10:29.310 INFO 21062 --- [ocp-schedules-26,09cada81952f58fc,a268766c815a2d32] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:10:29.431 INFO 21062 --- [ocp-schedules-1,84e2ea1cd9fd3b0f,7eca4f910034558e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:10:29.453 WARN 21062 --- [ocp-async-6,9b10dca9c048fa8f,0ceb81911fd2118b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:29.458 INFO 21062 --- [ocp-schedules-23,d94a5023f952ce20,81634096db11678e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:10:29.463 INFO 21062 --- [ocp-schedules-23,d94a5023f952ce20,81634096db11678e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:10:29.463 INFO 21062 --- [ocp-schedules-23,d94a5023f952ce20,81634096db11678e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:10:29.488 INFO 21062 --- [ocp-async-8,e68998d37e1872bb,8138656277f1c120] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:29.713 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:10:29.713 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:10:29.722 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:10:29.722 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:10:29.722 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:10:29.732 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:10:29.732 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:10:29.732 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:10:29.736 INFO 21062 --- [ocp-schedules-28,f2ef4544ae2a12f5,8a7ac556be50bbcb] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:10:29.785 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:29.785 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:29.905 INFO 21062 --- [ocp-schedules-2,5b9cf21ba6dd5a1d,1c3fc3a259306d9c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:10:29.905 INFO 21062 --- [ocp-schedules-2,5b9cf21ba6dd5a1d,1c3fc3a259306d9c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:10:29.911 INFO 21062 --- [ocp-schedules-2,5b9cf21ba6dd5a1d,1c3fc3a259306d9c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:10:29.911 INFO 21062 --- [ocp-schedules-2,5b9cf21ba6dd5a1d,1c3fc3a259306d9c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:10:29.917 INFO 21062 --- [ocp-schedules-2,5b9cf21ba6dd5a1d,1c3fc3a259306d9c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:10:29.917 INFO 21062 --- [ocp-schedules-2,5b9cf21ba6dd5a1d,1c3fc3a259306d9c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:10:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e2d572cbe4473da8,3160feadce40c061] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:10:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:10:31.358 INFO 21062 --- [ocp-schedules-22,a27acae6a3da57b0,7cec9c97faf64f1b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:10:31.453 WARN 21062 --- [ocp-async-2,37fa8845594afd55,5f58464d483fe492] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:31.489 INFO 21062 --- [ocp-async-3,cad35672b587257f,94494f36ea72ce87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:32.657 INFO 21062 --- [ocp-schedules-13,4f4b76a926c21eb0,3efff6d28cbe532b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:10:32.663 INFO 21062 --- [ocp-schedules-13,4f4b76a926c21eb0,3efff6d28cbe532b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:10:32.663 INFO 21062 --- [ocp-schedules-13,4f4b76a926c21eb0,3efff6d28cbe532b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:10:32.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:32.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:32.854 INFO 21062 --- [ocp-schedules-5,922d885d30213623,eae723ad16906954] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:10:32.854 INFO 21062 --- [ocp-schedules-5,922d885d30213623,eae723ad16906954] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:10:32.883 INFO 21062 --- [ocp-schedules-5,922d885d30213623,eae723ad16906954] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:10:32.883 INFO 21062 --- [ocp-schedules-5,922d885d30213623,eae723ad16906954] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:10:32.885 INFO 21062 --- [ocp-schedules-16,20d47b987147c20a,f502423c598f396e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:10:32.983 INFO 21062 --- [prometheus-minute-schedule-1,736ac580ba089c32,eefdfc0a252d3611] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:33.077 INFO 21062 --- [ocp-schedules-19,33bbe2b6a8185bf4,8ac536d4dc850f89] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:10:33.077+08:00, clusters=[1] 2024-11-04 15:10:33.080 INFO 21062 --- [ocp-schedules-19,33bbe2b6a8185bf4,8ac536d4dc850f89] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:10:33.339 INFO 21062 --- [ocp-schedules-28,67b5742661b874b6,8b69cd959486972c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:10:33.339+08:00, clusters=[1] 2024-11-04 15:10:33.340 INFO 21062 --- [ocp-schedules-20,a4f6b195b4ac6ed9,18b79c5ee40098c3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:10:33.342 INFO 21062 --- [ocp-schedules-28,67b5742661b874b6,8b69cd959486972c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:10:33.373 INFO 21062 --- [ocp-schedules-9,af1e0983877c0fd8,f209ecad665324a4] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:10:33.376 INFO 21062 --- [ocp-schedules-9,af1e0983877c0fd8,f209ecad665324a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:33.439 ERROR 21062 --- [ocp-schedules-9,af1e0983877c0fd8,f209ecad665324a4] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:10:33.439 INFO 21062 --- [ocp-schedules-9,af1e0983877c0fd8,f209ecad665324a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:10:33.453 WARN 21062 --- [ocp-async-7,63e7643fe7b4c5d4,c38f720fb6ae8514] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:33.489 INFO 21062 --- [ocp-async-9,947e837efb407449,0a91459f0ee76986] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:34.957 INFO 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:10:34.957 INFO 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:10:34.957 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:10:34.957 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:10:34.968 INFO 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:10:34.968 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:10:34.969 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:10:34.970 WARN 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:10:34.970 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:10:34.970 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:10:34.971 INFO 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:10:34.971 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:10:34.971 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:10:34.973 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:10:34.973 WARN 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:10:34.973 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:10:34.981 INFO 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:10:34.981 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:10:34.981 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:10:35.007 ERROR 21062 --- [ocp-schedules-19,1d78231bd9fdb240,105139b11ff39490] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:10:35.454 WARN 21062 --- [ocp-async-6,93d8ec369e616e1d,460215871aab361c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:35.489 INFO 21062 --- [ocp-async-5,f01dea1457034564,5f1c9e5a7aee7aee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:35.787 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:35.787 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e6e3fa2cef13e701,336d1ecdde1542ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:37.454 WARN 21062 --- [ocp-async-2,d28602e97d32cf32,0cbf8332d9743e95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:37.489 INFO 21062 --- [ocp-async-1,ed4eff291629d76e,ac690cb4cbb4aff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:38.504 INFO 21062 --- [ocp-schedules-8,1906009c271b9b61,4474791e80ef5bad] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:10:38.504 INFO 21062 --- [ocp-schedules-18,3f4c0e2441ea8d91,5db6ddebe76e40f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:10:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:10:38.589307+08:00, lastProcessTime=2024-11-04T15:10:28.590179+08:00 2024-11-04 15:10:38.745 INFO 21062 --- [ocp-async-6,2a502b6e922125d3,a63890812453ce39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:10:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:10:38.589307+08:00 2024-11-04 15:10:38.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:38.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:38.983 INFO 21062 --- [prometheus-minute-schedule-1,e9aa150f64e9589e,24415b2a47744055] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:39.345 INFO 21062 --- [ocp-schedules-12,32c52e8aadcb5d6b,a8ae98c0740ed8a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:10:39.345 INFO 21062 --- [ocp-schedules-12,8d670b74fa0b630e,e8b5336123d410e3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:10:39.348 INFO 21062 --- [ocp-schedules-12,8d670b74fa0b630e,e8b5336123d410e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:39.376 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:10:39.376 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:10:39.454 WARN 21062 --- [ocp-async-9,42b1627b464fe010,d547f1ad2fcc27b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:39.489 INFO 21062 --- [ocp-async-6,bb4c510eeeba6a23,b39c83379133ec48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:40.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:10:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:10:41.361 INFO 21062 --- [ocp-schedules-4,2b4477eb43f459b2,c25beef44b174c0c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:10:41.454 WARN 21062 --- [ocp-async-5,f30678bbfe487109,8296a857cc4aac18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:41.489 INFO 21062 --- [ocp-async-2,1acca0a3edc538a3,3858df4df2e86f98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:41.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:41.790 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:10:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:10:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:10:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:10:41.855 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:10:41.855 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:10:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c3f6b9490b550d38,aed4af50f4e345ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:43.117 INFO 21062 --- [ocp-schedules-26,d088df5a400f6d31,e87f703bc891fcce] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:10:43.122 INFO 21062 --- [ocp-schedules-26,d088df5a400f6d31,e87f703bc891fcce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:43.205 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:10:43.207 ERROR 21062 --- [ocp-schedules-26,d088df5a400f6d31,e87f703bc891fcce] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:10:43.207 INFO 21062 --- [ocp-schedules-26,d088df5a400f6d31,e87f703bc891fcce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:10:43.365 INFO 21062 --- [ocp-schedules-20,bace2cf21f75f719,f37fce41554373be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:10:43.454 WARN 21062 --- [ocp-async-1,45f8040d506a2296,68996cb7467b0063] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:43.490 INFO 21062 --- [ocp-async-3,cf7c12c750d8934a,8101d6711e10ebf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:44.790 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:44.790 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:44.983 INFO 21062 --- [prometheus-minute-schedule-1,ce1fff82c4a83a11,be89b66f140b21a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:45.454 WARN 21062 --- [ocp-async-9,9562a6eda3bb071f,6c0916da8e259229] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:45.490 INFO 21062 --- [ocp-async-4,b176adfd560f9bfa,e0adfcd46c6fb62c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:47.455 WARN 21062 --- [ocp-async-5,ee5927adac8c0021,f4ea929478de7586] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:47.490 INFO 21062 --- [ocp-async-10,5a9686acdd87eefb,6301a83b7b07bb9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:47.791 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:47.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fda4cf7e082dc135,b2e45ea08f4095f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:10:48.592009+08:00, lastProcessTime=2024-11-04T15:10:38.589307+08:00 2024-11-04 15:10:48.745 INFO 21062 --- [ocp-async-8,45936912ace8b57b,390cd73d66c1cb40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:10:48.778 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:10:48.592009+08:00 2024-11-04 15:10:49.455 WARN 21062 --- [ocp-async-1,e926773f034f9f4a,4a636dc28fa2d16d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:49.490 INFO 21062 --- [ocp-async-8,71b9d6b9e79cb37a,a40ebcf6b5682440] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:50.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:50.793 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:50.983 INFO 21062 --- [prometheus-minute-schedule-1,0183d948c27564ac,1a78a64a97d6e448] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:51.365 INFO 21062 --- [ocp-schedules-17,5b9b2016416bb432,6b09c5d3134aee13] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:10:51.455 WARN 21062 --- [ocp-async-8,35d770354879a5da,5460fbba7c3000be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:51.491 INFO 21062 --- [ocp-async-4,87af6cd6f13e6993,accaf6edf1e8b9a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:52.665 INFO 21062 --- [ocp-schedules-4,99509f9aad863c14,82f9d1ef291b79bb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:10:52.672 INFO 21062 --- [ocp-schedules-4,99509f9aad863c14,82f9d1ef291b79bb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:10:52.672 INFO 21062 --- [ocp-schedules-4,99509f9aad863c14,82f9d1ef291b79bb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:10:53.409 INFO 21062 --- [ocp-schedules-28,2776d00663eee7d6,0bdf70c7c0c3a6d3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:10:53.455 WARN 21062 --- [ocp-async-5,7b5769fee532f4b9,ebc5deb10b9981b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:53.491 INFO 21062 --- [ocp-async-10,80f21e7fb30b7316,5c1ca4abae3f9e60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:53.793 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:53.794 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:53.983 INFO 21062 --- [prometheus-minute-schedule-1,6c2b5724a177458f,af4bbe5e24412092] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:55.455 WARN 21062 --- [ocp-async-2,de704dd97ffcee1a,f6355dcca96d61bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:55.491 INFO 21062 --- [ocp-async-1,1c96a1d8cea514e5,1653a1fa0e5bce58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:56.794 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:56.795 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:56.983 INFO 21062 --- [prometheus-minute-schedule-1,544442f206db9cff,e4d89fb91a1467f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:10:57.456 WARN 21062 --- [ocp-async-1,f9bf624dc357a652,6fcad537b5cdae7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:57.491 INFO 21062 --- [ocp-async-9,6f7933a87b5678ef,a7bedd6453b22e6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:58.500 INFO 21062 --- [ocp-schedules-2,297214b6cafe9ed3,10d5e56114191d0b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:10:58.500 INFO 21062 --- [ocp-schedules-24,7ae2075411cbf031,0f4cdcf4258d4aa0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:10:58.502 INFO 21062 --- [ocp-schedules-9,08f5aec6f9820a05,9023874cce445faf] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:10:58.503 INFO 21062 --- [ocp-schedules-2,297214b6cafe9ed3,10d5e56114191d0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:10:58.504 INFO 21062 --- [ocp-schedules-24,7ae2075411cbf031,0f4cdcf4258d4aa0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:58.504 INFO 21062 --- [ocp-schedules-8,f19f0a63d969b44f,5528ff91f584c4e0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:10:58.509 INFO 21062 --- [ocp-schedules-19,c11c3f378f770916,1a59ebcaa9efe2ad] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:10:58.509 INFO 21062 --- [ocp-schedules-19,c11c3f378f770916,1a59ebcaa9efe2ad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:10:58.509 INFO 21062 --- [ocp-schedules-28,f12560b299c916e2,444db9846c12a64a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:10:58.510 INFO 21062 --- [ocp-schedules-28,f12560b299c916e2,444db9846c12a64a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:10:58.510 INFO 21062 --- [ocp-schedules-26,d23158c1dc0a7330,62a2cd5b3b7f69e4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:10:58.510 INFO 21062 --- [ocp-schedules-26,d23158c1dc0a7330,62a2cd5b3b7f69e4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:10:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:10:58.588876+08:00, lastProcessTime=2024-11-04T15:10:48.592009+08:00 2024-11-04 15:10:58.746 INFO 21062 --- [ocp-async-6,6e5ee5fd80a2e333,00df85d6d7c26bcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:10:58.800 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:10:58.588876+08:00 2024-11-04 15:10:59.251 INFO 21062 --- [ocp-schedules-3,0c7c3dba3bb4b6e4,178f2394480b8eb4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:10:59.255 INFO 21062 --- [ocp-schedules-3,0c7c3dba3bb4b6e4,178f2394480b8eb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:10:59.256 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:10:59.434 INFO 21062 --- [ocp-schedules-13,1970bc19c7b8a67a,2f713cb0f2325149] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:10:59.456 WARN 21062 --- [ocp-async-5,05c7620ecbb1f413,b913285a7619e62a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:10:59.491 INFO 21062 --- [ocp-async-2,d48e290475c5122a,e1baa984f2b9d60b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:10:59.795 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:10:59.796 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:10:59.983 INFO 21062 --- [prometheus-minute-schedule-1,fbf445534a314a0a,fb24c7bb8eb831c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704260, interval=60 2024-11-04 15:11:01.367 INFO 21062 --- [ocp-schedules-14,8324466c7d1aa466,fb040a31903fa753] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:11:01.456 WARN 21062 --- [ocp-async-10,e7f83e61a09cfda5,24f409200957d2e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:01.491 INFO 21062 --- [ocp-async-1,abcbacef7189b36b,c665199f574b1607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:02.796 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:02.797 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:02.983 INFO 21062 --- [prometheus-minute-schedule-1,514bd82c39624650,a027573442d714c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:03.435 INFO 21062 --- [ocp-schedules-5,90440acefb34f792,4a88e6fd27eb0c72] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:11:03.456 WARN 21062 --- [ocp-async-1,4eb128e2377de7a9,ec7bbf9710b4fab3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:03.492 INFO 21062 --- [ocp-async-9,e9919a2cd7029648,0f0a4a80ca2e1b1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:05.456 WARN 21062 --- [ocp-async-9,c8416c3b4c907d82,e571e78da96a26fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:05.492 INFO 21062 --- [ocp-async-6,28cb87ce5ea7d822,acdddfbba1ac3d20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:05.797 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:05.798 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e13b407dedb2cc11,eb6a9870aed78130] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:07.457 WARN 21062 --- [ocp-async-6,ef79c36b0aa4374c,af152e0aeb453e2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:07.492 INFO 21062 --- [ocp-async-3,980f989a303c6c07,16a1c94148054238] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:08.504 INFO 21062 --- [ocp-schedules-2,297214b6cafe9ed3,10d5e56114191d0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:11:08.504 INFO 21062 --- [ocp-schedules-24,7ae2075411cbf031,0f4cdcf4258d4aa0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:11:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:11:08.590349+08:00, lastProcessTime=2024-11-04T15:10:58.588876+08:00 2024-11-04 15:11:08.746 INFO 21062 --- [ocp-async-7,70c533c9d54f6c40,832b0cb6ce068034] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:11:08.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:11:08.590349+08:00 2024-11-04 15:11:08.798 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:08.799 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:08.983 INFO 21062 --- [prometheus-minute-schedule-1,30d3e4f56cdda535,b7e3b2db32629ba3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:09.256 INFO 21062 --- [ocp-schedules-3,0c7c3dba3bb4b6e4,178f2394480b8eb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:11:09.377 INFO 21062 --- [ocp-schedules-12,8d670b74fa0b630e,e8b5336123d410e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:11:09.377 INFO 21062 --- [ocp-schedules-12,18898dcdcb16d9ae,838fc18b5d96a4ed] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:11:09.381 INFO 21062 --- [ocp-schedules-12,18898dcdcb16d9ae,838fc18b5d96a4ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:11:09.410 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:11:09.410 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:11:09.459 WARN 21062 --- [ocp-async-8,56ccd11bf9911088,f89a7503c660d2fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:09.492 INFO 21062 --- [ocp-async-9,92520784fabd3e81,1951e0ba58e8fa04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:10.004 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:11:10.004 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:11:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:11:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:11:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:11:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:11:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:11:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:11:11.370 INFO 21062 --- [ocp-schedules-7,4bfabc9ae20a3cd9,c6af970c3a29b527] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:11:11.457 WARN 21062 --- [ocp-async-9,1ed4f9bb1b32ed39,e9006ba9205f8c44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:11.492 INFO 21062 --- [ocp-async-4,67a8b8644fb9def0,6c0abdd9215bdac0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:11.764 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=47, affectRows=47 2024-11-04 15:11:11.764 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=60, affectRows=60 2024-11-04 15:11:11.764 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=46, affectRows=46 2024-11-04 15:11:11.764 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=49, affectRows=49 2024-11-04 15:11:11.764 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=38, affectRows=38 2024-11-04 15:11:11.764 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=49, affectRows=49 2024-11-04 15:11:11.764 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=54, affectRows=54 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=38, validCount=38 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=45, validCount=45 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=47, validCount=47 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=49, validCount=49 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=49, validCount=49 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=60, validCount=60 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704200, nodeCount=79, affectRows=79 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=54, validCount=54 2024-11-04 15:11:11.765 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704200, nodeCount=77, validCount=77 2024-11-04 15:11:11.799 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:11.800 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b72feb4b2113fb69,4456f89a2cfbefc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:12.674 INFO 21062 --- [ocp-schedules-29,c3e09b1febd96f2b,b7a87021f1ff3790] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:11:12.680 INFO 21062 --- [ocp-schedules-29,c3e09b1febd96f2b,b7a87021f1ff3790] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:11:12.680 INFO 21062 --- [ocp-schedules-29,c3e09b1febd96f2b,b7a87021f1ff3790] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:11:13.457 WARN 21062 --- [ocp-async-10,9112dc64aeab01fd,a64bf9c75a1178fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:13.480 INFO 21062 --- [ocp-schedules-13,dbe8b17a315b6ec7,448f71a54cdd1afe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:11:13.493 INFO 21062 --- [ocp-async-2,09f8a996bd72f286,6deb608534ac9a0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:13.984 INFO 21062 --- [prometheus-cache-monitor1,def3205810f334d6,148c80c92edfe839] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:11:13.984 INFO 21062 --- [prometheus-cache-monitor1,def3205810f334d6,148c80c92edfe839] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:11:14.800 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:14.801 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:14.983 INFO 21062 --- [prometheus-minute-schedule-1,c60ee4de588af467,2aa546444654e006] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:15.458 WARN 21062 --- [ocp-async-8,82c227afe8cf46dc,1faf79904ac05a85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:15.479 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:07:15.458, end=2024-11-04T15:09:15.477} 2024-11-04 15:11:15.489 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:11:15.489 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:11:15.493 INFO 21062 --- [ocp-async-1,bc922fdd00417572,fd0e3562ca1369c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:17.458 WARN 21062 --- [ocp-async-1,337202343c0c4b71,456b45ae909ad07b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:17.493 INFO 21062 --- [ocp-async-5,2e3a3a00d50b8d06,429284ae51d2ae51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:17.801 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:17.802 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ca403db85c79662a,2f4bf8adabe022c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:11:18.590056+08:00, lastProcessTime=2024-11-04T15:11:08.590349+08:00 2024-11-04 15:11:18.746 INFO 21062 --- [ocp-async-3,ec81dcc54db8a7ef,93a4ea7dc78949d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:11:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:11:18.590056+08:00 2024-11-04 15:11:19.458 WARN 21062 --- [ocp-async-10,fb389345cf36c511,3e4567dcf3ab0079] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:19.493 INFO 21062 --- [ocp-async-3,658c5a1b1aeeab83,a9eadd4b46cdcc87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:20.802 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:20.802 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9d5af1c4680bfef5,70c1140ab2800483] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:21.373 INFO 21062 --- [ocp-schedules-5,154447d494ef9f3c,26578715a0121163] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:11:21.458 WARN 21062 --- [ocp-async-8,197fb7831795fe4b,7910e032da4322df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:21.493 INFO 21062 --- [ocp-async-7,7849a6d87385ecae,fb02e704b11fa345] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:23.458 WARN 21062 --- [ocp-async-1,c719781ead5db32e,fa6e481277f9bef3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:23.494 INFO 21062 --- [ocp-async-5,5a731298167dd3c3,bf7b7559af1beb39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:23.510 INFO 21062 --- [ocp-schedules-20,e77dbe777c7136b9,ba7122ca6d67506b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:11:23.803 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:23.803 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:23.983 INFO 21062 --- [prometheus-minute-schedule-1,9ee29d01185fcfe5,29a4710db2aa2bc3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:25.459 WARN 21062 --- [ocp-async-10,5f29d98ff2a5b45c,289b6a36435d8736] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:25.494 INFO 21062 --- [ocp-async-4,ce1f91ffdb1971b1,057a8a0a5ea8da36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:26.804 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:26.805 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:26.983 INFO 21062 --- [prometheus-minute-schedule-1,1488a4fdebe00409,debc7d465d4674fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4d6501ba6ac644be,ee36fa08aeec0f92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:11:27.459 WARN 21062 --- [ocp-async-3,05fb4a5a65be1f28,30b3bbb2ec93ee31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:27.494 INFO 21062 --- [ocp-async-2,90df9b82875d8232,cb73b975c7b614ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:28.498 INFO 21062 --- [ocp-schedules-1,c7018f0cd53156cd,642a653b328f0b45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:11:28.499 INFO 21062 --- [ocp-schedules-22,e44366f02db21d0d,5b7c3a7993737582] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:11:28.500 INFO 21062 --- [ocp-schedules-16,f9253d600947b9e5,a98a9dfc5be83cd0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:11:28.500 INFO 21062 --- [ocp-schedules-29,b0b074ca97e430ad,9051ac2329e4b9a9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:11:28.503 INFO 21062 --- [ocp-schedules-19,c41b78b7e4f97caa,7947f6f8e54dc072] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:11:28.503 INFO 21062 --- [ocp-schedules-22,e44366f02db21d0d,5b7c3a7993737582] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:11:28.504 INFO 21062 --- [ocp-schedules-29,b0b074ca97e430ad,9051ac2329e4b9a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:11:28.504 INFO 21062 --- [ocp-schedules-1,c7018f0cd53156cd,642a653b328f0b45] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:11:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:11:28.504 INFO 21062 --- [ocp-schedules-13,7fd4f117988b9f3a,0f0ec37492151bc0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:11:28.506 INFO 21062 --- [ocp-schedules-16,f9253d600947b9e5,a98a9dfc5be83cd0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:11:28.511 INFO 21062 --- [ocp-schedules-24,1db26ee62ae991c6,6f072e3c703bee57] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:11:28.511 INFO 21062 --- [ocp-schedules-24,1db26ee62ae991c6,6f072e3c703bee57] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:11:28.511 INFO 21062 --- [ocp-schedules-9,1ee049f26275dc12,37a45bda4b6aa9be] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:11:28.511 INFO 21062 --- [ocp-schedules-9,1ee049f26275dc12,37a45bda4b6aa9be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:11:28.511 INFO 21062 --- [ocp-schedules-10,836dd5c3417b8960,b8754e5225cfa32d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:11:28.511 INFO 21062 --- [ocp-schedules-10,836dd5c3417b8960,b8754e5225cfa32d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:11:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:11:28.589812+08:00, lastProcessTime=2024-11-04T15:11:18.590056+08:00 2024-11-04 15:11:28.746 INFO 21062 --- [ocp-async-6,7715082c5b243973,dde31ae24e49b26f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:11:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:11:28.589812+08:00 2024-11-04 15:11:28.953 INFO 21062 --- [ocp-schedules-28,72c76a2ac8b23b0b,c54d609e2a00d6d9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:11:29.315 INFO 21062 --- [ocp-schedules-13,4f69e05de472a676,7cbbff5b83a3a0fe] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:11:29.315 INFO 21062 --- [ocp-schedules-13,4f69e05de472a676,7cbbff5b83a3a0fe] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:11:29.437 INFO 21062 --- [ocp-schedules-10,7dfcf2c146a8b60a,71c9f97f39440533] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:11:29.459 WARN 21062 --- [ocp-async-10,359e6aa4fddbbb1d,9bccf60aaacaa4ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:29.463 INFO 21062 --- [ocp-schedules-27,44af5f643534ace9,f74dcfdb7c5fdd85] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:11:29.468 INFO 21062 --- [ocp-schedules-27,44af5f643534ace9,f74dcfdb7c5fdd85] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:11:29.468 INFO 21062 --- [ocp-schedules-27,44af5f643534ace9,f74dcfdb7c5fdd85] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:11:29.494 INFO 21062 --- [ocp-async-6,606776f45df412b6,f4003cd692a13967] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:29.805 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:29.806 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:29.923 INFO 21062 --- [ocp-schedules-26,ffd3c925b98de30f,b7e782934702cb8f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:11:29.923 INFO 21062 --- [ocp-schedules-26,ffd3c925b98de30f,b7e782934702cb8f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:11:29.924 INFO 21062 --- [ocp-schedules-26,ffd3c925b98de30f,b7e782934702cb8f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:11:29.924 INFO 21062 --- [ocp-schedules-26,ffd3c925b98de30f,b7e782934702cb8f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:11:29.924 INFO 21062 --- [ocp-schedules-26,ffd3c925b98de30f,b7e782934702cb8f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:11:29.925 INFO 21062 --- [ocp-schedules-26,ffd3c925b98de30f,b7e782934702cb8f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:11:29.983 INFO 21062 --- [prometheus-minute-schedule-1,9b9c60961b805de7,1eb88a90a6c7c84c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:11:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:11:31.377 INFO 21062 --- [ocp-schedules-19,2905108112f51160,b2285bfc9de33a99] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:11:31.459 WARN 21062 --- [ocp-async-4,06c72e9a32f52925,c6584a6337ec632d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:31.494 INFO 21062 --- [ocp-async-8,9dcd6e40bcc2ed5a,ba02f8ab07e5b557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:32.683 INFO 21062 --- [ocp-schedules-26,e826be65d2af9186,4a3e63c1cb20a893] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:11:32.688 INFO 21062 --- [ocp-schedules-26,e826be65d2af9186,4a3e63c1cb20a893] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:11:32.688 INFO 21062 --- [ocp-schedules-26,e826be65d2af9186,4a3e63c1cb20a893] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:11:32.806 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:32.806 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:32.883 INFO 21062 --- [ocp-schedules-21,2b894613e6273f73,82a36b67cc6c22c6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:11:32.883 INFO 21062 --- [ocp-schedules-21,2b894613e6273f73,82a36b67cc6c22c6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:11:32.909 INFO 21062 --- [ocp-schedules-21,2b894613e6273f73,82a36b67cc6c22c6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:11:32.909 INFO 21062 --- [ocp-schedules-21,2b894613e6273f73,82a36b67cc6c22c6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:11:32.911 INFO 21062 --- [ocp-schedules-13,b8a05a622690be0b,2e866f488e30c8fa] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:11:32.983 INFO 21062 --- [prometheus-minute-schedule-1,5a388822bd98b95c,4e5233882f917c41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:33.106 INFO 21062 --- [ocp-schedules-17,90b97787df6787a8,9b6785abf34de097] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:11:33.106+08:00, clusters=[1] 2024-11-04 15:11:33.108 INFO 21062 --- [ocp-schedules-17,90b97787df6787a8,9b6785abf34de097] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:11:33.370 INFO 21062 --- [ocp-schedules-3,0717f90335ab026a,7e8ad7b9e24b340e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:11:33.370+08:00, clusters=[1] 2024-11-04 15:11:33.373 INFO 21062 --- [ocp-schedules-3,0717f90335ab026a,7e8ad7b9e24b340e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:11:33.459 WARN 21062 --- [ocp-async-1,8547740382a3057b,9a5deab151b73a5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:33.495 INFO 21062 --- [ocp-async-7,5532cc78e6233326,3d1cd47cc74dc798] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:33.551 INFO 21062 --- [ocp-schedules-5,ff52bc39ebd68ff7,1586ceff1de3cd4d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:11:35.016 INFO 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:11:35.016 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:11:35.016 INFO 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:11:35.016 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:11:35.026 INFO 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:11:35.027 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:11:35.027 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:11:35.028 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:11:35.028 WARN 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:11:35.028 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:11:35.029 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:11:35.029 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:11:35.029 INFO 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:11:35.030 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:11:35.030 WARN 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:11:35.030 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:11:35.037 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:11:35.038 INFO 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:11:35.038 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:11:35.054 ERROR 21062 --- [ocp-schedules-13,42dbf96fe46d1b92,991e6d7d88483a04] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:11:35.459 WARN 21062 --- [ocp-async-10,407a0d22befb6abb,87114a706723a49a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:35.495 INFO 21062 --- [ocp-async-5,4330b6fc2165527b,136441be58b3ecba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:35.807 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:35.807 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:35.983 INFO 21062 --- [prometheus-minute-schedule-1,572c2996f3a93368,6a58af267d7bc348] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:37.460 WARN 21062 --- [ocp-async-4,aeac865addf82e6b,724d4e6a587daa96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:37.495 INFO 21062 --- [ocp-async-3,f05e7617e961955d,a8ed9080375f5d09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:38.504 INFO 21062 --- [ocp-schedules-29,b0b074ca97e430ad,9051ac2329e4b9a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:11:38.504 INFO 21062 --- [ocp-schedules-22,e44366f02db21d0d,5b7c3a7993737582] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:11:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:11:38.589285+08:00, lastProcessTime=2024-11-04T15:11:28.589812+08:00 2024-11-04 15:11:38.747 INFO 21062 --- [ocp-async-10,1d87aca336345f7c,ce7ab00bbcf315a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:11:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:11:38.589285+08:00 2024-11-04 15:11:38.807 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:38.808 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:38.983 INFO 21062 --- [prometheus-minute-schedule-1,05478604bab41192,6dbee17be10792ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:39.410 INFO 21062 --- [ocp-schedules-12,18898dcdcb16d9ae,838fc18b5d96a4ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:11:39.410 INFO 21062 --- [ocp-schedules-12,f5c259e0b75e16da,cf480fb4efad16cb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:11:39.414 INFO 21062 --- [ocp-schedules-12,f5c259e0b75e16da,cf480fb4efad16cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:11:39.444 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:11:39.444 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:11:39.460 WARN 21062 --- [ocp-async-2,bb1513f96f11f9cf,1d05b70322a02e1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:39.495 INFO 21062 --- [ocp-async-10,cff2bda2cb95b873,aef9f6a47ab85245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:11:40.004 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:11:41.380 INFO 21062 --- [ocp-schedules-2,955732b0bfa2f51d,fdc035f15f014312] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:11:41.460 WARN 21062 --- [ocp-async-9,fc3923be38fe9748,fa4bf3e823e11796] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:41.495 INFO 21062 --- [ocp-async-4,0102442fe9b2b475,0a992dda5af31409] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:41.808 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:41.809 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:41.983 INFO 21062 --- [prometheus-minute-schedule-1,a314560bba4fca8e,3076f6becdc5a024] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:43.207 INFO 21062 --- [ocp-schedules-17,cd8d12d5651e7ebb,ef6c1c245e657fbe] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:11:43.210 INFO 21062 --- [ocp-schedules-17,cd8d12d5651e7ebb,ef6c1c245e657fbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:11:43.296 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:11:43.298 ERROR 21062 --- [ocp-schedules-17,cd8d12d5651e7ebb,ef6c1c245e657fbe] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:11:43.298 INFO 21062 --- [ocp-schedules-17,cd8d12d5651e7ebb,ef6c1c245e657fbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:11:43.460 WARN 21062 --- [ocp-async-3,3454b4162a06c737,92a3c23906b5412e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:43.495 INFO 21062 --- [ocp-async-8,dc2d7bd7f88d22b7,4340001281f232cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:43.576 INFO 21062 --- [ocp-schedules-14,83db5bb4b7c8bf4a,da1d66f203231479] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:11:44.809 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:44.810 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3d54aa98059dd0d1,c1b77dd5b05a66ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:45.461 WARN 21062 --- [ocp-async-2,0a804e3b259e611d,6bdfac6313ea7291] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:45.496 INFO 21062 --- [ocp-async-7,b8789164def23806,88442ef56123747e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:47.461 WARN 21062 --- [ocp-async-9,b8343e6bb88a94b1,31c054fc18c43323] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:47.496 INFO 21062 --- [ocp-async-5,7d1e2de5454cf3a9,22d1cdbcee29d614] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:47.810 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:47.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:47.983 INFO 21062 --- [prometheus-minute-schedule-1,13c19c92dbee8496,df6556bc92d4990d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:11:48.592700+08:00, lastProcessTime=2024-11-04T15:11:38.589285+08:00 2024-11-04 15:11:48.747 INFO 21062 --- [ocp-async-6,f97051d14d9ef50a,5f9ca757e266fee2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:11:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:11:48.592700+08:00 2024-11-04 15:11:49.461 WARN 21062 --- [ocp-async-6,cbf5ca54555a7c93,03d11cb63a00e736] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:49.496 INFO 21062 --- [ocp-async-8,6dfeb0376b1a9895,61bed1e6a97c12d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:50.811 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:50.812 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:50.983 INFO 21062 --- [prometheus-minute-schedule-1,69419c41715a0f84,3b77ba3c1ba533bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:51.383 INFO 21062 --- [ocp-schedules-27,1950fb607ca593c7,97c2af04a2680bc5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:11:51.461 WARN 21062 --- [ocp-async-2,f52559e6d3157dc9,f96a102d17ed815e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:51.496 INFO 21062 --- [ocp-async-7,3fa75d35dbf91f54,650242bc42fe6ccb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:52.691 INFO 21062 --- [ocp-schedules-27,50328e5871c23870,9538ab540db66d4b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:11:52.697 INFO 21062 --- [ocp-schedules-27,50328e5871c23870,9538ab540db66d4b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:11:52.697 INFO 21062 --- [ocp-schedules-27,50328e5871c23870,9538ab540db66d4b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:11:53.461 WARN 21062 --- [ocp-async-9,fb4b714ba487ba58,9593f4d7bb89470e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:53.496 INFO 21062 --- [ocp-async-5,b880895914b471dd,2fa8b89f9f5c1763] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:53.619 INFO 21062 --- [ocp-schedules-17,d728dc39882d2a02,c52207bf95786fda] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:11:53.812 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:53.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:53.983 INFO 21062 --- [prometheus-minute-schedule-1,0d0e856f2c664ac5,806dcba1d90969fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:55.461 WARN 21062 --- [ocp-async-4,c2be794d4763e05d,1225412cde75ae6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:55.497 INFO 21062 --- [ocp-async-3,1725aafe7e35bbf9,48f460cbc84e55ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:56.813 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:56.814 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:56.983 INFO 21062 --- [prometheus-minute-schedule-1,4e2d10895656e299,82828da53fb27a1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:11:57.461 WARN 21062 --- [ocp-async-2,18c6c0775c43bab0,b4a87887662baff1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:57.497 INFO 21062 --- [ocp-async-8,77479a6e69a171b7,5a468394b1f909a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:58.500 INFO 21062 --- [ocp-schedules-13,6feaf9cec08385fe,357b5e25ca1b0fc9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:11:58.500 INFO 21062 --- [ocp-schedules-15,1efb43c0e9343c91,24c90734c8c2e9f8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:11:58.503 INFO 21062 --- [ocp-schedules-30,a1766b93b4d3fc14,76956350f7ca05e9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:11:58.504 INFO 21062 --- [ocp-schedules-13,6feaf9cec08385fe,357b5e25ca1b0fc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:11:58.504 INFO 21062 --- [ocp-schedules-15,1efb43c0e9343c91,24c90734c8c2e9f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:11:58.504 INFO 21062 --- [ocp-schedules-5,92d186d758a3452c,0ae67ee257135696] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:11:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:11:58.511 INFO 21062 --- [ocp-schedules-1,f40089809bf69bae,39fc0a392124dc76] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:11:58.511 INFO 21062 --- [ocp-schedules-16,f09b0aec1d57e038,64d16afd5759322f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:11:58.511 INFO 21062 --- [ocp-schedules-16,f09b0aec1d57e038,64d16afd5759322f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:11:58.511 INFO 21062 --- [ocp-schedules-1,f40089809bf69bae,39fc0a392124dc76] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:11:58.511 INFO 21062 --- [ocp-schedules-26,6f4b8c66e6127d9b,2357aaf64f76e99b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:11:58.512 INFO 21062 --- [ocp-schedules-26,6f4b8c66e6127d9b,2357aaf64f76e99b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:11:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:11:58.590759+08:00, lastProcessTime=2024-11-04T15:11:48.592700+08:00 2024-11-04 15:11:58.747 INFO 21062 --- [ocp-async-10,a57aa760cf5d83de,5e74f839a6e35c28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:11:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:11:58.590759+08:00 2024-11-04 15:11:59.440 INFO 21062 --- [ocp-schedules-30,7877b5395aefb201,fa531148034e4188] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:11:59.462 WARN 21062 --- [ocp-async-7,b7c0f6dfeea90451,8411c448a8e5cfc7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:11:59.497 INFO 21062 --- [ocp-async-4,b84a3eba795ebab4,e969c3a69b2fa56b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:11:59.814 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:11:59.815 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:11:59.983 INFO 21062 --- [prometheus-minute-schedule-1,4f69d47c6025d374,1ffb27af5b498920] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704320, interval=60 2024-11-04 15:12:01.387 INFO 21062 --- [ocp-schedules-8,3c9f9b950db99155,888d68f5a774450c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:12:01.462 WARN 21062 --- [ocp-async-5,94fbc63ceb66fa85,8cb6be612835bb5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:01.497 INFO 21062 --- [ocp-async-2,e12690a7bbc5d2ad,995215edc89c1b65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:02.815 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:02.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:02.983 INFO 21062 --- [prometheus-minute-schedule-1,fe0deefdf1b8c895,04773c158514050b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:03.462 WARN 21062 --- [ocp-async-3,f13a40ce5904c114,f6e2e305e10367db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:03.497 INFO 21062 --- [ocp-async-8,10c91da5382e3092,87bffef5c809760a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:03.645 INFO 21062 --- [ocp-schedules-19,f197766126ad8505,8410400af2eaa049] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:12:05.462 WARN 21062 --- [ocp-async-7,3161ba7e299a26d6,cf0aca06aafcc73a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:05.498 INFO 21062 --- [ocp-async-10,833d23cc61c04538,aebec2992dc3f592] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:05.816 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:05.817 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:05.983 INFO 21062 --- [prometheus-minute-schedule-1,28171efaf12a971f,4fa04c0119afa111] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:07.462 WARN 21062 --- [ocp-async-10,0fc795df9ff85f19,d14d918d071ea0ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:07.498 INFO 21062 --- [ocp-async-6,764614edfefdd61d,ad31e718ad39be03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:08.505 INFO 21062 --- [ocp-schedules-15,1efb43c0e9343c91,24c90734c8c2e9f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:12:08.505 INFO 21062 --- [ocp-schedules-13,6feaf9cec08385fe,357b5e25ca1b0fc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:12:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:12:08.589720+08:00, lastProcessTime=2024-11-04T15:11:58.590759+08:00 2024-11-04 15:12:08.747 INFO 21062 --- [ocp-async-1,395b0f01a37e7b10,f4f246389b39408f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:12:08.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:12:08.589720+08:00 2024-11-04 15:12:08.817 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:08.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:08.983 INFO 21062 --- [prometheus-minute-schedule-1,c9d2b68caebe1f11,295148b0498fe9ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:09.256 INFO 21062 --- [ocp-schedules-16,6c703f3da7dedb5a,7451e8990cece390] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:12:09.260 INFO 21062 --- [ocp-schedules-16,6c703f3da7dedb5a,7451e8990cece390] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:09.261 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:12:09.444 INFO 21062 --- [ocp-schedules-12,f5c259e0b75e16da,cf480fb4efad16cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:12:09.444 INFO 21062 --- [ocp-schedules-12,d64f01328292a6f4,89d3dedd760d317b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:12:09.448 INFO 21062 --- [ocp-schedules-12,d64f01328292a6f4,89d3dedd760d317b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:09.463 WARN 21062 --- [ocp-async-3,8cc12d5b3bb02156,c97d2b9342ca398c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:09.476 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:12:09.476 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:12:09.498 INFO 21062 --- [ocp-async-7,c0da689c92511c60,28a684224fdcec1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:10.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:12:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:12:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:12:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:12:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:12:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:12:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:12:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:12:11.390 INFO 21062 --- [ocp-schedules-26,726a52b287bdc488,f8cc40c42a79f586] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:12:11.463 WARN 21062 --- [ocp-async-7,1aae5d55fd42dca2,2825a1824792b93a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:11.498 INFO 21062 --- [ocp-async-9,807b87c369e27c33,ee4fc678243118ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:11.771 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704260, nodeCount=78, affectRows=78 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704260, nodeCount=36, affectRows=36 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704260, nodeCount=75, affectRows=75 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704260, nodeCount=36, validCount=36 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704260, nodeCount=58, affectRows=58 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704260, nodeCount=77, affectRows=77 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704260, nodeCount=78, validCount=78 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704260, nodeCount=74, validCount=74 2024-11-04 15:12:11.772 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704260, nodeCount=58, validCount=58 2024-11-04 15:12:11.773 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704260, nodeCount=76, validCount=76 2024-11-04 15:12:11.774 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704260, nodeCount=98, affectRows=98 2024-11-04 15:12:11.775 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704260, nodeCount=97, validCount=97 2024-11-04 15:12:11.818 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:11.819 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:11.983 INFO 21062 --- [prometheus-minute-schedule-1,1b2dde0389bb1b74,750b679cf7e0e14a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:12.699 INFO 21062 --- [ocp-schedules-10,9a69ab978b2bab1c,ae4796611030ee92] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:12:12.705 INFO 21062 --- [ocp-schedules-10,9a69ab978b2bab1c,ae4796611030ee92] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:12:12.705 INFO 21062 --- [ocp-schedules-10,9a69ab978b2bab1c,ae4796611030ee92] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:12:13.463 WARN 21062 --- [ocp-async-5,994baaa8101f372f,165626b6bec08158] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:13.498 INFO 21062 --- [ocp-async-4,1e0aa67d60a9d4f5,111d5269e1c57201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:13.688 INFO 21062 --- [ocp-schedules-19,6dc484ee9fc47bb2,16eb8738933b9ffd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:12:13.984 INFO 21062 --- [prometheus-cache-monitor1,d3b2ae324941632c,d347069494fbd0b4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:12:13.984 INFO 21062 --- [prometheus-cache-monitor1,d3b2ae324941632c,d347069494fbd0b4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:12:14.819 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:14.820 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:14.983 INFO 21062 --- [prometheus-minute-schedule-1,892f4d9d0f1bd9d5,bb7102eb3279cce3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:15.463 WARN 21062 --- [ocp-async-4,051fd21754b19a9e,32a8a0ccd0219ff9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:15.499 INFO 21062 --- [ocp-async-2,55fd2b9f105d623e,d61c4e4ed3b1d9d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:17.463 WARN 21062 --- [ocp-async-1,4789784954028335,3d55502472059d4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:17.499 INFO 21062 --- [ocp-async-8,7bf93907d77950a7,3063c5a7e5f5c818] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:17.820 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:17.821 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:17.983 INFO 21062 --- [prometheus-minute-schedule-1,bb06865b2bf535a5,ed25ff8b77764e18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:12:18.589913+08:00, lastProcessTime=2024-11-04T15:12:08.589720+08:00 2024-11-04 15:12:18.747 INFO 21062 --- [ocp-async-6,bafa66ba478600b6,0f4327d83ab4aca3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:12:18.779 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:12:18.589913+08:00 2024-11-04 15:12:19.261 INFO 21062 --- [ocp-schedules-16,6c703f3da7dedb5a,7451e8990cece390] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:12:19.464 WARN 21062 --- [ocp-async-5,62516e6564565c02,98a7f3f500480fb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:19.499 INFO 21062 --- [ocp-async-6,d84834c27b009b40,d3db0b18d2d0df91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:20.821 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:20.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4e5b3b0b9e5c3200,c658ba742f2a6826] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:21.393 INFO 21062 --- [ocp-schedules-19,777901da7f0014a3,7ffd61bc71981bbb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:12:21.464 WARN 21062 --- [ocp-async-3,330654412a8a55c5,38cdc50fa0aa34f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:21.499 INFO 21062 --- [ocp-async-2,7d06fed40a1d3811,8bd0ac9d5aadd5f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:23.464 WARN 21062 --- [ocp-async-7,785d2b3ad7b6be90,fd9f619d1b6b9c42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:23.499 INFO 21062 --- [ocp-async-8,832a65a2e89b3428,e6eba1fce70ffa87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:23.751 INFO 21062 --- [ocp-schedules-26,bc61e40e23d62ab4,905b242e369c62db] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:12:23.822 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:23.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6d6d514aeb16aac3,ed4cc68469e93578] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:25.464 WARN 21062 --- [ocp-async-5,b0a1c3bfa7da7b24,88d95e9f5d7e879d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:25.499 INFO 21062 --- [ocp-async-9,a5563df36abda677,78f224012acc2c11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:26.823 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:26.824 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:26.983 INFO 21062 --- [prometheus-minute-schedule-1,bc3454d13cc114bf,c7dc6a5bcec8e96a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f45127e9d8cd9929,683b1e3d01009a63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:12:27.464 WARN 21062 --- [ocp-async-6,870b38f578b174f9,a66bb109205f6152] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:27.500 INFO 21062 --- [ocp-async-4,4fbd7e5bfd8655c5,657ed4ff62706748] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:28.498 INFO 21062 --- [ocp-schedules-13,e6870870ef51bba9,ed4aff1713ac67cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:12:28.499 INFO 21062 --- [ocp-schedules-30,793f78094aa371f0,6a1f8572a6e21fa8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:12:28.500 INFO 21062 --- [ocp-schedules-20,f6298a4d3cc3380a,c95097df11a7dc92] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:12:28.500 INFO 21062 --- [ocp-schedules-19,60b5577d8955b90d,29cd20abcca63f0f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:12:28.502 INFO 21062 --- [ocp-schedules-13,e6870870ef51bba9,ed4aff1713ac67cb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:12:28.502 INFO 21062 --- [ocp-schedules-7,ef3d77baa1c8a20c,3735a9b8cc7bb58c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:12:28.503 INFO 21062 --- [ocp-schedules-20,f6298a4d3cc3380a,c95097df11a7dc92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:12:28.504 INFO 21062 --- [ocp-schedules-19,60b5577d8955b90d,29cd20abcca63f0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:28.504 INFO 21062 --- [ocp-schedules-24,d62a4bc64b56318e,77b0d6e9ba030371] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:12:28.504 INFO 21062 --- [ocp-schedules-11,120bfa578c466939,a2363e1a4c162013] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:12:28.505 INFO 21062 --- [ocp-schedules-30,793f78094aa371f0,6a1f8572a6e21fa8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:12:28.507 INFO 21062 --- [ocp-schedules-11,120bfa578c466939,a2363e1a4c162013] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:28.509 INFO 21062 --- [ocp-schedules-9,6ccefd82bfc4fb49,35a1cddaf2ed42f7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:12:28.509 INFO 21062 --- [ocp-schedules-9,6ccefd82bfc4fb49,35a1cddaf2ed42f7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:12:28.510 INFO 21062 --- [ocp-schedules-15,85fdb427dc789617,a7d4e5673789b169] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:12:28.510 INFO 21062 --- [ocp-schedules-18,00fe231dc2032480,ec5064662a9a22ec] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:12:28.510 INFO 21062 --- [ocp-schedules-18,00fe231dc2032480,ec5064662a9a22ec] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:12:28.510 INFO 21062 --- [ocp-schedules-15,85fdb427dc789617,a7d4e5673789b169] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:12:28.541 INFO 21062 --- [ocp-schedules-3,a8416c8d6e814469,2e068a9fd7c69a4a] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:12:28.541 INFO 21062 --- [ocp-schedules-3,a8416c8d6e814469,2e068a9fd7c69a4a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:12:28.584 INFO 21062 --- [pool-12-thread-1,a4741a59b92a7fc3,d268944045e93eaa] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:12:28.584 INFO 21062 --- [pool-12-thread-3,a4741a59b92a7fc3,115fa04320586b5b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:12:28.584 INFO 21062 --- [pool-12-thread-3,a4741a59b92a7fc3,115fa04320586b5b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:12:28.584 INFO 21062 --- [pool-12-thread-2,a4741a59b92a7fc3,b5559fb669047b93] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:12:28.584 INFO 21062 --- [pool-12-thread-4,a4741a59b92a7fc3,0b0c567933b27e36] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:12:28.585 INFO 21062 --- [ocp-schedules-22,481aeb724edce551,1d68f05545439ed0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:12:28.585 INFO 21062 --- [ocp-schedules-22,481aeb724edce551,1d68f05545439ed0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:12:28.585 INFO 21062 --- [pool-12-thread-3,a4741a59b92a7fc3,115fa04320586b5b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:12:28.586 INFO 21062 --- [pool-12-thread-3,a4741a59b92a7fc3,115fa04320586b5b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:12:28.589 INFO 21062 --- [ForkJoinPool-11-worker-9,a4741a59b92a7fc3,75af9b9d02195a9b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:12:28.589 INFO 21062 --- [ForkJoinPool-11-worker-0,a4741a59b92a7fc3,8b6e62d680261f33] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:12:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:12:28.590131+08:00, lastProcessTime=2024-11-04T15:12:18.589913+08:00 2024-11-04 15:12:28.589 INFO 21062 --- [ForkJoinPool-11-worker-9,a4741a59b92a7fc3,75af9b9d02195a9b] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:12:28.590 INFO 21062 --- [ForkJoinPool-11-worker-9,a4741a59b92a7fc3,75af9b9d02195a9b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:12:28.590 INFO 21062 --- [pool-12-thread-4,a4741a59b92a7fc3,0b0c567933b27e36] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:12:28.590 INFO 21062 --- [pool-12-thread-1,a4741a59b92a7fc3,d268944045e93eaa] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:12:28.592 INFO 21062 --- [pool-12-thread-2,a4741a59b92a7fc3,b5559fb669047b93] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:12:28.592 INFO 21062 --- [pool-12-thread-2,a4741a59b92a7fc3,b5559fb669047b93] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:12:28.657 INFO 21062 --- [ocp-schedules-28,1a9ce92f80e9125b,dd53dd81aa271c3b] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:12:28.658 INFO 21062 --- [ocp-schedules-28,1a9ce92f80e9125b,dd53dd81aa271c3b] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:12:28.748 INFO 21062 --- [ocp-async-5,7bc3b6432e756ffd,b7fac8baee5a3940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:12:28.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:12:28.590131+08:00 2024-11-04 15:12:28.955 INFO 21062 --- [ocp-schedules-16,b17cdfb11d3a152a,204a7bff74162e3a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:12:28.968 INFO 21062 --- [ocp-schedules-15,ac3297e2f0d2ed50,31347557d73f46e5] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 15:12:28.968 INFO 21062 --- [ocp-schedules-15,ac3297e2f0d2ed50,31347557d73f46e5] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 15:12:28.969 INFO 21062 --- [ocp-schedules-15,ac3297e2f0d2ed50,31347557d73f46e5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=422 2024-11-04 15:12:28.987 INFO 21062 --- [ocp-schedules-15,ac3297e2f0d2ed50,31347557d73f46e5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=422 2024-11-04 15:12:29.260 INFO 21062 --- [ocp-schedules-14,cb49c90cb9db3633,99901b4873adf720] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 15:12:29.264 INFO 21062 --- [ocp-schedules-14,cb49c90cb9db3633,99901b4873adf720] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:29.307 ERROR 21062 --- [ocp-schedules-14,cb49c90cb9db3633,99901b4873adf720] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.GeneratedMethodAccessor1314.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:12:29.307 INFO 21062 --- [ocp-schedules-14,cb49c90cb9db3633,99901b4873adf720] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 15:12:29.320 INFO 21062 --- [ocp-schedules-29,4e68a9159a3b51db,0b33c2164b2cdf29] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:12:29.320 INFO 21062 --- [ocp-schedules-29,4e68a9159a3b51db,0b33c2164b2cdf29] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:12:29.368 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:12:29.386 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:12:29.387 WARN 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:12:29.388 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:12:29.389 WARN 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:12:29.396 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:12:29.397 WARN 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:12:29.398 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:12:29.399 WARN 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:12:29.405 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:12:29.406 WARN 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:12:29.407 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:12:29.408 WARN 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:12:29.414 INFO 21062 --- [ocp-schedules-26,c5e457fac76d70ba,5f9b8f713be0de45] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:12:29.443 INFO 21062 --- [ocp-schedules-28,881233d0e0b88442,76e531b43fe621e4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:12:29.464 WARN 21062 --- [ocp-async-8,f8aa0566a78b9523,2538cb44b282741b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:29.468 INFO 21062 --- [ocp-schedules-2,f3a3526a132131df,385f8ad4b8d0c752] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:12:29.473 INFO 21062 --- [ocp-schedules-2,f3a3526a132131df,385f8ad4b8d0c752] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:12:29.473 INFO 21062 --- [ocp-schedules-2,f3a3526a132131df,385f8ad4b8d0c752] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:12:29.500 INFO 21062 --- [ocp-async-10,2ada6cdf03df8d11,e8601bf478cfeb8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:29.824 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:29.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:29.925 INFO 21062 --- [ocp-schedules-7,37f2c2422b6b38fb,cf6b43b17d201f3c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:12:29.925 INFO 21062 --- [ocp-schedules-7,37f2c2422b6b38fb,cf6b43b17d201f3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:12:29.926 INFO 21062 --- [ocp-schedules-7,37f2c2422b6b38fb,cf6b43b17d201f3c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:12:29.926 INFO 21062 --- [ocp-schedules-7,37f2c2422b6b38fb,cf6b43b17d201f3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:12:29.927 INFO 21062 --- [ocp-schedules-7,37f2c2422b6b38fb,cf6b43b17d201f3c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:12:29.927 INFO 21062 --- [ocp-schedules-7,37f2c2422b6b38fb,cf6b43b17d201f3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:12:29.983 INFO 21062 --- [prometheus-minute-schedule-1,9dc1bd6916e50349,f6723f2d0843b607] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:12:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:12:30.313 INFO 21062 --- [ocp-schedules-13,14ea88853ac286af,521ddad657b9bfdd] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:12:30.313 INFO 21062 --- [ocp-schedules-13,14ea88853ac286af,521ddad657b9bfdd] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:12:30.342 INFO 21062 --- [pool-51-thread-1,14ea88853ac286af,7ebc54bfc2bd0711] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:12:30.371 WARN 21062 --- [ocp-schedules-13,14ea88853ac286af,521ddad657b9bfdd] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:12:30.371 INFO 21062 --- [ocp-schedules-13,14ea88853ac286af,521ddad657b9bfdd] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:12:31.397 INFO 21062 --- [ocp-schedules-29,9da3e1a33e46075d,0130c6399692c973] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:12:31.465 WARN 21062 --- [ocp-async-9,5c27e2a653e4c0c9,9ce3ed45650f2795] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:31.500 INFO 21062 --- [ocp-async-3,fe68c196049fc3c3,69fd0f70af25df26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:32.708 INFO 21062 --- [ocp-schedules-21,dd3cc91769e250c0,b8011bb3c13f71a0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:12:32.713 INFO 21062 --- [ocp-schedules-21,dd3cc91769e250c0,b8011bb3c13f71a0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:12:32.713 INFO 21062 --- [ocp-schedules-21,dd3cc91769e250c0,b8011bb3c13f71a0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:12:32.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:32.825 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:32.909 INFO 21062 --- [ocp-schedules-3,ac5461f649807b07,83403869906556d9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:12:32.909 INFO 21062 --- [ocp-schedules-3,ac5461f649807b07,83403869906556d9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:12:32.936 INFO 21062 --- [ocp-schedules-3,ac5461f649807b07,83403869906556d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:12:32.936 INFO 21062 --- [ocp-schedules-3,ac5461f649807b07,83403869906556d9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:12:32.937 INFO 21062 --- [ocp-schedules-28,0adb8f6ea60238a7,996100fff87920b1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:12:32.983 INFO 21062 --- [prometheus-minute-schedule-1,aa8701777858bbc5,08a813b7c5b8a579] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:33.135 INFO 21062 --- [ocp-schedules-8,e1e01420e9cfdce7,001e9e260c075907] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:12:33.135+08:00, clusters=[1] 2024-11-04 15:12:33.137 INFO 21062 --- [ocp-schedules-8,e1e01420e9cfdce7,001e9e260c075907] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:12:33.399 INFO 21062 --- [ocp-schedules-23,dfa3e2f1d407f92a,c8886e6cb73ffdc7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:12:33.399+08:00, clusters=[1] 2024-11-04 15:12:33.403 INFO 21062 --- [ocp-schedules-23,dfa3e2f1d407f92a,c8886e6cb73ffdc7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:12:33.439 INFO 21062 --- [ocp-schedules-10,0c3750ef571443d1,b7e9199dd2f7aea2] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:12:33.442 INFO 21062 --- [ocp-schedules-10,0c3750ef571443d1,b7e9199dd2f7aea2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:33.465 WARN 21062 --- [ocp-async-3,11ee44646e362faa,806a136399ade4cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:33.502 ERROR 21062 --- [ocp-schedules-10,0c3750ef571443d1,b7e9199dd2f7aea2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:12:33.502 INFO 21062 --- [ocp-async-2,1bea2d581c3212b3,0d1e68c857fb24aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:33.502 INFO 21062 --- [ocp-schedules-10,0c3750ef571443d1,b7e9199dd2f7aea2] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:12:33.792 INFO 21062 --- [ocp-schedules-25,cd0daf8318342a65,624129f400c379d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:12:35.064 INFO 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:12:35.064 INFO 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:12:35.064 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:12:35.064 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:12:35.075 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:12:35.075 INFO 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:12:35.076 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:12:35.077 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:12:35.077 WARN 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:12:35.077 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:12:35.078 INFO 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:12:35.078 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:12:35.078 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:12:35.079 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:12:35.079 WARN 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:12:35.079 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:12:35.086 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:12:35.086 INFO 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:12:35.086 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:12:35.101 ERROR 21062 --- [ocp-schedules-15,42f9b286e30dde39,062fba5a1af91726] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:12:35.465 WARN 21062 --- [ocp-async-1,6736cda3636d8e69,5770b8c8c2fa8fa1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:35.502 INFO 21062 --- [ocp-async-5,49176904618ffc1a,3e0ccb59a479c6fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:35.826 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:35.826 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:35.983 INFO 21062 --- [prometheus-minute-schedule-1,faef211b33fe623c,fb012872913805b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:37.465 WARN 21062 --- [ocp-async-5,a72abf59ebd76488,e9599691ad5df767] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:37.502 INFO 21062 --- [ocp-async-6,f61e546bdbc0a761,5291b6877f79ef26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:38.504 INFO 21062 --- [ocp-schedules-20,f6298a4d3cc3380a,c95097df11a7dc92] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:12:38.504 INFO 21062 --- [ocp-schedules-19,60b5577d8955b90d,29cd20abcca63f0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:12:38.508 INFO 21062 --- [ocp-schedules-11,120bfa578c466939,a2363e1a4c162013] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:12:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:12:38.589518+08:00, lastProcessTime=2024-11-04T15:12:28.590131+08:00 2024-11-04 15:12:38.748 INFO 21062 --- [ocp-async-1,631970412adb4bd3,ba0dcc78d0e8e60a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:12:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:12:38.589518+08:00 2024-11-04 15:12:38.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:38.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:38.983 INFO 21062 --- [prometheus-minute-schedule-1,d39e21198eb38e6a,44e8a73c0270d3a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:39.465 WARN 21062 --- [ocp-async-7,ea81dcb677e4e4cb,53f087da3eb6a994] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:39.476 INFO 21062 --- [ocp-schedules-12,d64f01328292a6f4,89d3dedd760d317b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:12:39.477 INFO 21062 --- [ocp-schedules-12,fd2958b490098f19,e5e78839addaacd6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:12:39.480 INFO 21062 --- [ocp-schedules-12,fd2958b490098f19,e5e78839addaacd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:39.502 INFO 21062 --- [ocp-async-2,5bacd3f189c2a938,b6206655fb9c0495] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:39.508 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:12:39.509 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:12:40.002 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:12:40.002 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:12:41.400 INFO 21062 --- [ocp-schedules-21,5dc882f696decd77,68b325e4cf3e519b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:12:41.466 WARN 21062 --- [ocp-async-2,c2475e5e0dc87dd1,c77c9aa3c3abbf6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:41.502 INFO 21062 --- [ocp-async-10,cc1f1be1ce6477db,d949c8f61ca86c3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:12:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:12:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:12:41.827 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:41.828 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:41.983 INFO 21062 --- [prometheus-minute-schedule-1,0e852b2dc5133d57,296cfece0a696997] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:43.298 INFO 21062 --- [ocp-schedules-5,a7582ef7a2b5e909,6fd289c0ce825882] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:12:43.302 INFO 21062 --- [ocp-schedules-5,a7582ef7a2b5e909,6fd289c0ce825882] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:43.388 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:12:43.390 ERROR 21062 --- [ocp-schedules-5,a7582ef7a2b5e909,6fd289c0ce825882] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:12:43.390 INFO 21062 --- [ocp-schedules-5,a7582ef7a2b5e909,6fd289c0ce825882] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:12:43.466 WARN 21062 --- [ocp-async-10,0fc59dbf18668537,27055bf942cb1453] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:43.503 INFO 21062 --- [ocp-async-3,05c678fa548e74fc,cb3ebe02884f38b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:43.816 INFO 21062 --- [ocp-schedules-17,e0033be109a7b669,19dbdbe9cb11ace4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:12:44.828 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:44.830 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d05be37d39c2cfa1,1e340c262f4b7f34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:45.466 WARN 21062 --- [ocp-async-3,12f58837efb815c1,928f6ec87acee691] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:45.503 INFO 21062 --- [ocp-async-1,c6f8c2fb16f89305,0abda527ccd169f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:47.466 WARN 21062 --- [ocp-async-2,94981272200a907b,87b3eaab7f406f4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:47.503 INFO 21062 --- [ocp-async-5,86eb1eb40b0c2940,e879de5160d6ec56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:47.830 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:47.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:47.983 INFO 21062 --- [prometheus-minute-schedule-1,cc73b59003c77839,680664fae1d6c23b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:12:48.590998+08:00, lastProcessTime=2024-11-04T15:12:38.589518+08:00 2024-11-04 15:12:48.748 INFO 21062 --- [ocp-async-6,585b5818b57b1469,5a82cc0316db1f05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:12:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:12:48.590998+08:00 2024-11-04 15:12:49.466 WARN 21062 --- [ocp-async-9,0764f33c7cde43aa,547108b3e37b9b57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:49.503 INFO 21062 --- [ocp-async-3,5fddc73e7706d953,93e23a2a239eaafa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:50.831 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:50.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1c1c0222b91f0992,5c2fe3a3310441b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:51.403 INFO 21062 --- [ocp-schedules-6,be6d2a54e1ff6bbe,6aa9ee5df347ba90] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:12:51.467 WARN 21062 --- [ocp-async-3,0900a85b164bc0db,1a58d4ac7cd84b15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:51.503 INFO 21062 --- [ocp-async-2,e42f0b92030d39ac,4feecd8423989247] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:52.716 INFO 21062 --- [ocp-schedules-14,67989d38a1dc51c9,6953a93e06ca7739] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:12:52.720 INFO 21062 --- [ocp-schedules-14,67989d38a1dc51c9,6953a93e06ca7739] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:12:52.720 INFO 21062 --- [ocp-schedules-14,67989d38a1dc51c9,6953a93e06ca7739] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:12:53.467 WARN 21062 --- [ocp-async-2,6bb3e6d3cf9a163f,4fdf3c2c7f3e6874] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:53.503 INFO 21062 --- [ocp-async-5,624454bdd079ce4c,174811b3bfe9108f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:53.832 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:53.833 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:53.859 INFO 21062 --- [ocp-schedules-16,32b0e700e7432b1a,efa7fedf478762d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:12:53.983 INFO 21062 --- [prometheus-minute-schedule-1,96d867771eddd533,0946c6f9fd6034f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:55.467 WARN 21062 --- [ocp-async-5,2e67f97f7b8643f7,696c9e0cec5ad5d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:55.503 INFO 21062 --- [ocp-async-10,adfbbeacfc5b6264,62c98f247b4e992a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:56.833 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:56.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:56.983 INFO 21062 --- [prometheus-minute-schedule-1,93035c40f6935889,4c4b788c4d886125] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:12:57.467 WARN 21062 --- [ocp-async-10,df6e4b87153c76a6,dfa83aa9c7de2233] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:57.504 INFO 21062 --- [ocp-async-7,314c341be4ac42d3,a94682105359e684] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:58.499 INFO 21062 --- [ocp-schedules-9,9c50094dac17ba1a,89e3b6137d3a4208] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:12:58.500 INFO 21062 --- [ocp-schedules-18,c4fa32b109c4068a,c401a59e097bc7fc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:12:58.502 INFO 21062 --- [ocp-schedules-2,0eed4b9f641cdb9d,b27d130a5f807c3b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:12:58.504 INFO 21062 --- [ocp-schedules-7,77d0e1a811bca45b,ff980f885ec5f274] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:12:58.505 INFO 21062 --- [ocp-schedules-9,9c50094dac17ba1a,89e3b6137d3a4208] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:58.505 INFO 21062 --- [ocp-schedules-18,c4fa32b109c4068a,c401a59e097bc7fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:12:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:12:58.509 INFO 21062 --- [ocp-schedules-26,c3abfc5b25838666,c5ef840beaa9ab7d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:12:58.509 INFO 21062 --- [ocp-schedules-26,c3abfc5b25838666,c5ef840beaa9ab7d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:12:58.510 INFO 21062 --- [ocp-schedules-13,298a187d46f2c744,98d2d28b3b49a065] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:12:58.510 INFO 21062 --- [ocp-schedules-8,e44fbe96e821b467,6747bb78664d0829] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:12:58.510 INFO 21062 --- [ocp-schedules-13,298a187d46f2c744,98d2d28b3b49a065] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:12:58.510 INFO 21062 --- [ocp-schedules-8,e44fbe96e821b467,6747bb78664d0829] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:12:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:12:58.589420+08:00, lastProcessTime=2024-11-04T15:12:48.590998+08:00 2024-11-04 15:12:58.748 INFO 21062 --- [ocp-async-8,706c7c6df6b59473,ff5245f55fbb4c81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:12:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:12:58.589420+08:00 2024-11-04 15:12:59.446 INFO 21062 --- [ocp-schedules-2,c3e03d2e1244196d,cf3ed9e690abc328] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:12:59.467 WARN 21062 --- [ocp-async-2,311e421ceec09843,d9777294ee92ce6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:12:59.504 INFO 21062 --- [ocp-async-8,7ce8eda6f1a72f75,dd1ebf62a414addd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:12:59.665 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 15:12:59.665 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/task/2, method=10.130.47.17, client=GET, traceId=] 2024-11-04 15:12:59.665 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 15:12:59.835 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:12:59.836 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:12:59.983 INFO 21062 --- [prometheus-minute-schedule-1,c2863961acdf3987,0e08195293f1d613] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704380, interval=60 2024-11-04 15:13:00.536 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,d0c7dbdf6ebe48da,dea51784c10dfdcc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.130.47.17, traceId=d0c7dbdf6ebe48da, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2024-11-04 15:13:00.539 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,d0c7dbdf6ebe48da,dea51784c10dfdcc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.130.47.17, traceId=d0c7dbdf6ebe48da, duration=9 ms] 2024-11-04 15:13:00.654 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,faffcdce305e631e,660ef7eaa12112cd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=faffcdce305e631e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 15:13:00.654 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,faffcdce305e631e,660ef7eaa12112cd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=faffcdce305e631e, duration=5 ms] 2024-11-04 15:13:00.656 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,79d50fef73daee9c,16370e00cc633f2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=79d50fef73daee9c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 15:13:00.662 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,79d50fef73daee9c,16370e00cc633f2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=79d50fef73daee9c, duration=10 ms] 2024-11-04 15:13:00.759 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,961b3e2fbf222133,852a5acf151add6c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=961b3e2fbf222133, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:13:00.775 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,1d799fa8adbe56a2,2d0b0bb4f4f78db6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=1d799fa8adbe56a2, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 15:13:00.779 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,1d799fa8adbe56a2,2d0b0bb4f4f78db6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=1d799fa8adbe56a2, duration=9 ms] 2024-11-04 15:13:00.800 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,961b3e2fbf222133,852a5acf151add6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=961b3e2fbf222133, duration=47 ms] 2024-11-04 15:13:00.905 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8f5a1f36fd53466c,8a5b7352dbc965a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=8f5a1f36fd53466c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 15:13:01.079 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8f5a1f36fd53466c,8a5b7352dbc965a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=8f5a1f36fd53466c, duration=181 ms] 2024-11-04 15:13:01.406 INFO 21062 --- [ocp-schedules-30,a49ecc1477a6e6cb,1d1a5fb49784ed05] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:13:01.468 WARN 21062 --- [ocp-async-8,f96ac465a9a3ae80,7bdc24936d09f96f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:01.504 INFO 21062 --- [ocp-async-6,8f6c3702fccd2b83,c1ab0deae66242b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:02.432 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,bfb1a71198e669b6,01eb5f3533e60dd6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=bfb1a71198e669b6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 15:13:02.443 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,bfb1a71198e669b6,01eb5f3533e60dd6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=bfb1a71198e669b6, duration=16 ms] 2024-11-04 15:13:02.836 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:02.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:02.983 INFO 21062 --- [prometheus-minute-schedule-1,8b9271c4578f11d5,5906f06f971d2727] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:03.468 WARN 21062 --- [ocp-async-10,aa5e3d1307b86419,bf9d50d45c9b7740] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:03.504 INFO 21062 --- [ocp-async-4,cbfdf2775078a3af,2c11f8299bf13ccc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:03.808 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8f5c65c3e541fa6f,634510d707aec3a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=8f5c65c3e541fa6f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 15:13:03.808 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8f5c65c3e541fa6f,634510d707aec3a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=8f5c65c3e541fa6f, duration=4 ms] 2024-11-04 15:13:03.810 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f4f3bf9cfb062d01,6a89c516096e9916] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=f4f3bf9cfb062d01, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 15:13:03.810 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,f4f3bf9cfb062d01,6a89c516096e9916] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=f4f3bf9cfb062d01, duration=6 ms] 2024-11-04 15:13:03.810 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,1b7f0d33bcdcc226,3a6cd2091b3dfaa5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=1b7f0d33bcdcc226, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 15:13:03.810 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,1b7f0d33bcdcc226,3a6cd2091b3dfaa5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=1b7f0d33bcdcc226, duration=3 ms] 2024-11-04 15:13:03.815 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 15:13:03.815 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 15:13:03.815 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 15:13:03.820 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 15:13:03.820 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 15:13:03.820 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 15:13:03.882 INFO 21062 --- [ocp-schedules-3,042b8e10c3c73405,5de9475495f11434] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:13:05.356 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,ae40030b688f7d81,25982e18380ce1c7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=ae40030b688f7d81, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 15:13:05.356 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,ae40030b688f7d81,25982e18380ce1c7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=ae40030b688f7d81, duration=6 ms] 2024-11-04 15:13:05.468 WARN 21062 --- [ocp-async-7,fe4b12cde1eedd6b,a97ddd9b985b2d4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:05.504 INFO 21062 --- [ocp-async-1,0e3777c605445366,3eab45a46149e7f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:05.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:05.837 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:05.983 INFO 21062 --- [prometheus-minute-schedule-1,a5b425f6ff357c2b,880054ead0936c0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:07.468 WARN 21062 --- [ocp-async-5,16c7d43f8f45303a,e0af0a8cfdc3b518] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:07.505 INFO 21062 --- [ocp-async-8,b5985cef818fb0a2,609f67e8859f85a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:08.506 INFO 21062 --- [ocp-schedules-9,9c50094dac17ba1a,89e3b6137d3a4208] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:13:08.506 INFO 21062 --- [ocp-schedules-18,c4fa32b109c4068a,c401a59e097bc7fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:13:08.538 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1a082f0892adcae0,13f8834eee7420f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=1a082f0892adcae0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 15:13:08.546 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1a082f0892adcae0,13f8834eee7420f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=1a082f0892adcae0, duration=15 ms] 2024-11-04 15:13:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:13:08.590990+08:00, lastProcessTime=2024-11-04T15:12:58.589420+08:00 2024-11-04 15:13:08.748 INFO 21062 --- [ocp-async-4,2123c485c9e5f929,7360da79ee69a9df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:13:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:13:08.590990+08:00 2024-11-04 15:13:08.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:08.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:08.983 INFO 21062 --- [prometheus-minute-schedule-1,62e7a682ec493217,46a7a50ce1c42112] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:09.468 WARN 21062 --- [ocp-async-3,d09e6860fdcabf67,94a27e86d4938c75] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:09.505 INFO 21062 --- [ocp-async-4,44d9cb9f39fdae3e,d01bf5b7cb347003] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:09.509 INFO 21062 --- [ocp-schedules-12,fd2958b490098f19,e5e78839addaacd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:13:09.509 INFO 21062 --- [ocp-schedules-12,0a0f85c5b280784f,fc318742fce5d3f9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:13:09.512 INFO 21062 --- [ocp-schedules-12,0a0f85c5b280784f,fc318742fce5d3f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:09.541 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:13:09.541 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:13:09.671 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,14e0b3ffdf222755,a501fe515b86e0e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/51/logSegments?taskInstanceId=2&subtaskInstanceId=51, client=10.130.47.17, traceId=14e0b3ffdf222755, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,51,] 2024-11-04 15:13:09.677 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,14e0b3ffdf222755,a501fe515b86e0e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/51/logSegments client=10.130.47.17, traceId=14e0b3ffdf222755, duration=14 ms] 2024-11-04 15:13:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:13:10.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:13:10.295 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:13:10.295 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:13:10.296 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:13:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:13:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:13:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:13:11.409 INFO 21062 --- [ocp-schedules-6,46dd4c10cdf89a72,7586c91ed75c1006] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:13:11.468 WARN 21062 --- [ocp-async-2,49024f2301727b38,826853838cf1e9a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:11.505 INFO 21062 --- [ocp-async-1,e4b77f597cea734d,b4f50091cb8473f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=20, affectRows=20 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=17, affectRows=17 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=5, affectRows=5 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=2, affectRows=2 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=34, affectRows=34 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=5, validCount=5 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=2, validCount=2 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=17, validCount=17 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=20, validCount=20 2024-11-04 15:13:11.788 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=33, validCount=33 2024-11-04 15:13:11.791 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=67, affectRows=67 2024-11-04 15:13:11.792 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=39, affectRows=39 2024-11-04 15:13:11.792 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=39, validCount=39 2024-11-04 15:13:11.792 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=67, validCount=67 2024-11-04 15:13:11.792 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=78, affectRows=78 2024-11-04 15:13:11.792 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=76, affectRows=76 2024-11-04 15:13:11.792 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704320, nodeCount=84, affectRows=84 2024-11-04 15:13:11.792 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=78, validCount=78 2024-11-04 15:13:11.793 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=74, validCount=74 2024-11-04 15:13:11.793 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704320, nodeCount=84, validCount=84 2024-11-04 15:13:11.838 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:11.839 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8b409ec58fecb9f2,982368a08fe0d28b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:12.723 INFO 21062 --- [ocp-schedules-15,c3e34464340bac2b,bb4ae0f5a55f3296] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:13:12.729 INFO 21062 --- [ocp-schedules-15,c3e34464340bac2b,bb4ae0f5a55f3296] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:13:12.729 INFO 21062 --- [ocp-schedules-15,c3e34464340bac2b,bb4ae0f5a55f3296] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:13:13.469 WARN 21062 --- [ocp-async-5,c117c130bdf94489,31abbe25709d7029] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:13.505 INFO 21062 --- [ocp-async-8,1ea69679d5e991c5,7a031ee357a52b4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:13.922 INFO 21062 --- [ocp-schedules-11,f17fa37292beb229,29d50e4f209cb084] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:13:13.984 INFO 21062 --- [prometheus-cache-monitor1,d54d2bd17529b631,5df31b90667c68f4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:13:13.984 INFO 21062 --- [prometheus-cache-monitor1,d54d2bd17529b631,5df31b90667c68f4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:13:14.432 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:07:14.413, end=2024-11-04T15:12:14.429} 2024-11-04 15:13:14.432 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:07:14.413, end=2024-11-04T15:12:14.43} 2024-11-04 15:13:14.436 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:13:14.437 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:13:14.437 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:13:14.437 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:13:14.839 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:14.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:14.983 INFO 21062 --- [prometheus-minute-schedule-1,4155ffa2eda6f93f,b310e645e3f89e38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:15.165 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:13:15.165 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:13:15.174 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:13:15.174 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:13:15.174 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:13:15.174 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:13:15.180 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:13:15.180 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:13:15.180 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:13:15.180 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:13:15.217 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:13:15.225 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:13:15.225 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:13:15.226 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:13:15.226 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:13:15.226 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:13:15.236 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:13:15.501 WARN 21062 --- [ocp-async-3,e10458617bae9b78,c8aebcdde5428e93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:15.505 INFO 21062 --- [ocp-async-10,878be0731c0277af,a2b2ab63ae9e646e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:15.509 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:09:15.477, end=2024-11-04T15:11:15.507} 2024-11-04 15:13:15.513 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:13:15.513 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:13:15.788 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:13:16.334 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.153904683S 2024-11-04 15:13:16.334 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:13:16.430 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:13:16.430 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:13:16.457 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:13:16.512 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:13:16.512 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:13:16.512 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:13:16.512 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:13:16.512 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:13:16.512 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:13:16.520 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:13:16.645 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] o.s.boot.SpringApplication : Started application in 1.471 seconds (JVM running for 9810.324) 2024-11-04 15:13:16.645 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.471324366S 2024-11-04 15:13:17.013 INFO 21062 --- [ocp-schedules-29,4657f854aaa40a2f,ea55fbba8ced2631] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:13:17.501 WARN 21062 --- [ocp-async-4,947b7786fde3c824,bc0e2f0e2425aea8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:17.505 INFO 21062 --- [ocp-async-7,b6294b14bec797a2,afbb9916354bb22d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:17.840 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:17.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:17.983 INFO 21062 --- [prometheus-minute-schedule-1,c91eb8a3b1e86914,9c329435ddc9374b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:13:18.590728+08:00, lastProcessTime=2024-11-04T15:13:08.590990+08:00 2024-11-04 15:13:18.748 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:13:18.590728+08:00 2024-11-04 15:13:18.748 INFO 21062 --- [ocp-async-8,431e243dee0181d5,e2cbaf96a3e0206c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:13:19.261 INFO 21062 --- [ocp-schedules-20,27e7cffeaaba7df2,9fcfd37fca8f5737] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:13:19.265 INFO 21062 --- [ocp-schedules-20,27e7cffeaaba7df2,9fcfd37fca8f5737] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:19.265 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:13:19.501 WARN 21062 --- [ocp-async-5,d42146464a6c7c09,305d6f9427c64b21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:19.505 INFO 21062 --- [ocp-async-8,2d085debd0aef491,da7d9a8519deb468] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:20.841 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:20.842 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e8863b582884b09f,fba9f84c2084999e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:21.412 INFO 21062 --- [ocp-schedules-25,f415d9c96d08fa6f,b24117c8f71d2f6c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:13:21.501 WARN 21062 --- [ocp-async-3,e05604760ea08066,acd306ec42b1a9ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:21.506 INFO 21062 --- [ocp-async-10,dc55db4a76947324,cc3b5cfb4252d28c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:23.502 WARN 21062 --- [ocp-async-2,417df6270ad53161,fd26cc55092e8cc1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:23.506 INFO 21062 --- [ocp-async-7,26088076b5a3b190,4ea157bcfc4b5710] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:23.842 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:23.843 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:23.947 INFO 21062 --- [ocp-schedules-2,b3eee9f1f84dd6de,9658bd693da058df] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:13:23.983 INFO 21062 --- [prometheus-minute-schedule-1,3858226941547d52,457bc9ae7fb14561] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:25.502 WARN 21062 --- [ocp-async-1,1665acdc131330e2,c37e10f3fde45730] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:25.506 INFO 21062 --- [ocp-async-9,582bdc34c47813cc,2d1d7a190eb5bfaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:26.843 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:26.844 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:26.983 INFO 21062 --- [prometheus-minute-schedule-1,aa739441ed38824d,94f816e24bd2c833] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:26.984 INFO 21062 --- [prometheus-hour-schedule-1,91e12e779c237667,72d27e899848f962] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:13:27.502 WARN 21062 --- [ocp-async-3,cd08a7c1033226ad,5021ce55cac7d5a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:27.506 INFO 21062 --- [ocp-async-6,ed36b21e7a2260c1,7810601fdca89ba2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:28.498 INFO 21062 --- [ocp-schedules-2,a856fab1f4db8a43,cb79b27e76ba2b14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:13:28.499 INFO 21062 --- [ocp-schedules-9,349299a5b17ba2f3,a2d1441a617dee5c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:13:28.500 INFO 21062 --- [ocp-schedules-7,a3a37992c086cc0c,ca44c687b232be14] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:13:28.500 INFO 21062 --- [ocp-schedules-30,8289fc91014a476a,7737d7314618d11d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:13:28.502 INFO 21062 --- [ocp-schedules-2,a856fab1f4db8a43,cb79b27e76ba2b14] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:13:28.502 INFO 21062 --- [ocp-schedules-29,f0526e3979633014,519b24b378d929d6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:13:28.503 INFO 21062 --- [ocp-schedules-7,a3a37992c086cc0c,ca44c687b232be14] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:13:28.504 INFO 21062 --- [ocp-schedules-30,8289fc91014a476a,7737d7314618d11d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:28.504 INFO 21062 --- [ocp-schedules-8,0eca398526d3dbca,353818acb7109318] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:13:28.505 INFO 21062 --- [ocp-schedules-9,349299a5b17ba2f3,a2d1441a617dee5c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:13:28.511 INFO 21062 --- [ocp-schedules-6,58ff0e7f2a283e35,81abe4bb802fe93c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:13:28.511 INFO 21062 --- [ocp-schedules-26,81457670a5dfedc1,c149186ad25b4305] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:13:28.511 INFO 21062 --- [ocp-schedules-6,58ff0e7f2a283e35,81abe4bb802fe93c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:13:28.511 INFO 21062 --- [ocp-schedules-26,81457670a5dfedc1,c149186ad25b4305] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:13:28.511 INFO 21062 --- [ocp-schedules-21,d8331b92e19c08a4,85f03e2851885d29] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:13:28.511 INFO 21062 --- [ocp-schedules-21,d8331b92e19c08a4,85f03e2851885d29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:13:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:13:28.589956+08:00, lastProcessTime=2024-11-04T15:13:18.590728+08:00 2024-11-04 15:13:28.600 INFO 21062 --- [ocp-schedules-18,e99a487fc8bec0cb,6cc885cc6982eb2e] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 15:13:28.600 INFO 21062 --- [ocp-schedules-18,e99a487fc8bec0cb,6cc885cc6982eb2e] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 15:13:28.742 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:13:28.589956+08:00 2024-11-04 15:13:28.749 INFO 21062 --- [ocp-async-1,5d481b3d869e5842,4a803162fec7e291] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:13:28.958 INFO 21062 --- [ocp-schedules-17,617e53d32c78127f,24618cc1324fd5ce] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:13:29.266 INFO 21062 --- [ocp-schedules-20,27e7cffeaaba7df2,9fcfd37fca8f5737] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:13:29.324 INFO 21062 --- [ocp-schedules-6,3248b8d344e03f8c,950df9592d8e0db7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:13:29.324 INFO 21062 --- [ocp-schedules-6,3248b8d344e03f8c,950df9592d8e0db7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:13:29.448 INFO 21062 --- [ocp-schedules-18,a03f36719f861765,abe1705d834572a6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:13:29.473 INFO 21062 --- [ocp-schedules-16,1479d601ea13e670,48fd5ee8c75b17c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:13:29.478 INFO 21062 --- [ocp-schedules-16,1479d601ea13e670,48fd5ee8c75b17c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:13:29.478 INFO 21062 --- [ocp-schedules-16,1479d601ea13e670,48fd5ee8c75b17c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:13:29.502 WARN 21062 --- [ocp-async-7,4a6c5eb2465149e5,51a8192c2b32832c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:29.506 INFO 21062 --- [ocp-async-5,aadfcae283892b86,46e12f47df5b6c92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:29.736 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:13:29.736 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:13:29.745 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:13:29.745 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:13:29.745 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:13:29.756 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:13:29.756 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:13:29.756 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:13:29.760 INFO 21062 --- [ocp-schedules-10,31522e372831903e,9e444522d8662f4c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:13:29.845 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:29.845 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:29.928 INFO 21062 --- [ocp-schedules-4,9196c9c3d984b088,701977a6787a3c9e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:13:29.928 INFO 21062 --- [ocp-schedules-4,9196c9c3d984b088,701977a6787a3c9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:13:29.932 INFO 21062 --- [ocp-schedules-4,9196c9c3d984b088,701977a6787a3c9e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:13:29.932 INFO 21062 --- [ocp-schedules-4,9196c9c3d984b088,701977a6787a3c9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:13:29.936 INFO 21062 --- [ocp-schedules-4,9196c9c3d984b088,701977a6787a3c9e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:13:29.936 INFO 21062 --- [ocp-schedules-4,9196c9c3d984b088,701977a6787a3c9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:13:29.983 INFO 21062 --- [prometheus-minute-schedule-1,984ed16ccbf9a6f4,8105a593a16bc172] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:13:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:13:31.415 INFO 21062 --- [ocp-schedules-8,1372663458bad679,29115216cf08d316] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:13:31.502 WARN 21062 --- [ocp-async-9,bcb0e7fd68f6cd97,daf9b978fee2c717] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:31.507 INFO 21062 --- [ocp-async-8,e826b78747ca7d5e,55d9304dc16258ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:32.732 INFO 21062 --- [ocp-schedules-4,23d2081750e0b17c,572b74237e905146] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:13:32.737 INFO 21062 --- [ocp-schedules-4,23d2081750e0b17c,572b74237e905146] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:13:32.737 INFO 21062 --- [ocp-schedules-4,23d2081750e0b17c,572b74237e905146] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:13:32.846 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:32.846 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:32.937 INFO 21062 --- [ocp-schedules-2,f1e9b6ea29b44646,d656dd352c40b73e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:13:32.937 INFO 21062 --- [ocp-schedules-2,f1e9b6ea29b44646,d656dd352c40b73e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:13:32.962 INFO 21062 --- [ocp-schedules-2,f1e9b6ea29b44646,d656dd352c40b73e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:13:32.962 INFO 21062 --- [ocp-schedules-2,f1e9b6ea29b44646,d656dd352c40b73e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:13:32.963 INFO 21062 --- [ocp-schedules-18,fbf2c031bd3d8f72,5e7348b19f3c87c9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:13:32.983 INFO 21062 --- [prometheus-minute-schedule-1,88cc67d457090c2e,f9028d95c133c02e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:33.163 INFO 21062 --- [ocp-schedules-25,c5fdb5a729fb1029,aba5904385346633] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:13:33.163+08:00, clusters=[1] 2024-11-04 15:13:33.164 INFO 21062 --- [ocp-schedules-25,c5fdb5a729fb1029,aba5904385346633] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:13:33.431 INFO 21062 --- [ocp-schedules-1,39fcf490cb27e020,ab2fcf1f3b9f3959] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:13:33.431+08:00, clusters=[1] 2024-11-04 15:13:33.435 INFO 21062 --- [ocp-schedules-1,39fcf490cb27e020,ab2fcf1f3b9f3959] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:13:33.502 WARN 21062 --- [ocp-async-6,1a074ccebda737bf,58cf80b679f14034] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:33.507 INFO 21062 --- [ocp-async-10,03a87ac83b0d8c8a,7e6ee927d3674d49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:33.971 INFO 21062 --- [ocp-schedules-3,aa0e128326e34900,2920584c75a3ce09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:13:35.111 INFO 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:13:35.111 INFO 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:13:35.111 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:13:35.111 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:13:35.120 INFO 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:13:35.121 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:13:35.121 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:13:35.122 WARN 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:13:35.122 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:13:35.122 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:13:35.123 INFO 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:13:35.123 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:13:35.123 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:13:35.124 WARN 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:13:35.124 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:13:35.124 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:13:35.131 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:13:35.131 INFO 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:13:35.131 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:13:35.142 ERROR 21062 --- [ocp-schedules-19,f60f6bb2d802e981,eb7a849ee92adef3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:13:35.503 WARN 21062 --- [ocp-async-7,bca9bef88d7b4360,4eee82209d0aca08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:35.507 INFO 21062 --- [ocp-async-1,16c8166a6bc96973,7b64c9c8268c2634] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:35.846 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:35.848 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0932e2adeab533a8,3c03d03ffd46f3c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:37.503 WARN 21062 --- [ocp-async-9,b69c71de17b39bcd,7b4c6dc30cd28f99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:37.507 INFO 21062 --- [ocp-async-3,309daa92bc3c5d5f,7bebc0f7f4f78b4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:38.504 INFO 21062 --- [ocp-schedules-7,a3a37992c086cc0c,ca44c687b232be14] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:13:38.504 INFO 21062 --- [ocp-schedules-30,8289fc91014a476a,7737d7314618d11d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:13:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:13:38.589512+08:00, lastProcessTime=2024-11-04T15:13:28.589956+08:00 2024-11-04 15:13:38.749 INFO 21062 --- [ocp-async-10,98f7a81293f3f2a1,0efb6fdc1d29501e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:13:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:13:38.589512+08:00 2024-11-04 15:13:38.848 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:38.849 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:38.983 INFO 21062 --- [prometheus-minute-schedule-1,bdc3de424b7962bd,c47819de7bcd8dd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:39.503 WARN 21062 --- [ocp-async-10,394d052af5c7d5cd,2d08bc3ea3f62fa9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:39.507 INFO 21062 --- [ocp-async-7,b173c224b687181b,6727d4e7fc1a7c36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:39.542 INFO 21062 --- [ocp-schedules-12,0a0f85c5b280784f,fc318742fce5d3f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:13:39.542 INFO 21062 --- [ocp-schedules-12,d50f4915bde39c9d,c9b5043215518c4f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:13:39.545 INFO 21062 --- [ocp-schedules-12,d50f4915bde39c9d,c9b5043215518c4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:39.574 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:13:39.574 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:13:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:13:40.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:13:41.418 INFO 21062 --- [ocp-schedules-13,6b48f0c002e8dba0,deb5e4891f892004] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:13:41.503 WARN 21062 --- [ocp-async-4,4189178e694d52c8,281b59c7899e1fde] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:41.507 INFO 21062 --- [ocp-async-9,79533382c19f42f6,12492796f941d471] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:41.849 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:41.850 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:41.983 INFO 21062 --- [prometheus-minute-schedule-1,504d9945b9c7db67,9e8f5dbce7dc5fb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:43.390 INFO 21062 --- [ocp-schedules-16,650140d679848da2,1d75a0252befc70d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:13:43.396 INFO 21062 --- [ocp-schedules-16,650140d679848da2,1d75a0252befc70d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:43.483 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:13:43.486 ERROR 21062 --- [ocp-schedules-16,650140d679848da2,1d75a0252befc70d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:13:43.486 INFO 21062 --- [ocp-schedules-16,650140d679848da2,1d75a0252befc70d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:13:43.503 WARN 21062 --- [ocp-async-5,12185d8d2230299f,10a0a1208db6684e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:43.508 INFO 21062 --- [ocp-async-8,ccf5964f3dc7ec70,ecb1b3280ed089bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:44.010 INFO 21062 --- [ocp-schedules-3,fa9dbdd20cff4abd,810e2f12e1abacea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:13:44.850 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:44.851 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:44.983 INFO 21062 --- [prometheus-minute-schedule-1,88cf7228b7c01097,7bf849e57fc42356] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:45.504 WARN 21062 --- [ocp-async-2,2552a82af4d6a75f,79ac3277f4095bcd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:45.508 INFO 21062 --- [ocp-async-6,47c95f52e7904eb8,2ce02557a03ad7a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:46.859 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,963de903449e98eb,691d6d46bbdccc3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=963de903449e98eb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 15:13:46.870 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,963de903449e98eb,691d6d46bbdccc3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=963de903449e98eb, duration=25 ms] 2024-11-04 15:13:47.504 WARN 21062 --- [ocp-async-7,eed5df899370d5ec,a8fe981c87558945] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:47.508 INFO 21062 --- [ocp-async-4,50386662d3a79ec9,9ac4b7071f3808d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:47.852 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:47.853 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:47.983 INFO 21062 --- [prometheus-minute-schedule-1,1846bb6b066ac008,ffef6173783c3755] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:13:48.591400+08:00, lastProcessTime=2024-11-04T15:13:38.589512+08:00 2024-11-04 15:13:48.749 INFO 21062 --- [ocp-async-3,877a0443169329d8,a1579ab9d963d665] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:13:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:13:48.591400+08:00 2024-11-04 15:13:49.504 WARN 21062 --- [ocp-async-5,3deffee271911890,fc479bb2a4ab286a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:49.508 INFO 21062 --- [ocp-async-8,2b9d01f4c6851b51,6f3f24ad45a54605] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:50.853 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:50.854 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b359f45569bdce69,b95ee7c22b00ac7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:51.422 INFO 21062 --- [ocp-schedules-18,91fc86cdf1de623b,ca388a874339e6f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:13:51.504 WARN 21062 --- [ocp-async-2,d6a485c5cbdcc16e,2b5ac9eea4b80556] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:51.508 INFO 21062 --- [ocp-async-6,132b39960513eb92,03ea7b2660176d84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:52.739 INFO 21062 --- [ocp-schedules-3,2afcdbda3c9038ba,7427a322df424e52] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:13:52.743 INFO 21062 --- [ocp-schedules-3,2afcdbda3c9038ba,7427a322df424e52] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:13:52.743 INFO 21062 --- [ocp-schedules-3,2afcdbda3c9038ba,7427a322df424e52] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:13:53.504 WARN 21062 --- [ocp-async-6,4758683503344175,0fdc17b474814dde] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:53.508 INFO 21062 --- [ocp-async-4,4f249982e64b936c,44890171f89e40bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:53.854 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:53.855 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9e905b41a0b2dfcc,0eff685bbc49b0d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:54.065 INFO 21062 --- [ocp-schedules-15,da4733655d47aed9,41680fbf4137ee1f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:13:55.504 WARN 21062 --- [ocp-async-9,d909b1dbed465fb5,ffe605dbdc99a721] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:55.509 INFO 21062 --- [ocp-async-5,047c69ee324614ae,74de0b5cebeb916b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:56.855 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:56.856 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:56.983 INFO 21062 --- [prometheus-minute-schedule-1,353b5273a2fe3f21,2a3160758f526c2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:13:57.505 WARN 21062 --- [ocp-async-5,a44c654c1f6aa729,23a575711d1fe88b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:57.509 INFO 21062 --- [ocp-async-2,71d5d9b2fe034f81,2a67ce9eb483bb49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:58.500 INFO 21062 --- [ocp-schedules-30,8aacab5e970a7e07,335e8960d10431e5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:13:58.500 INFO 21062 --- [ocp-schedules-8,6ce84fa5558b27e0,a706848ea1fe8f9b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:13:58.502 INFO 21062 --- [ocp-schedules-19,224064ac3fee4758,028144324d5ba91c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:13:58.504 INFO 21062 --- [ocp-schedules-8,6ce84fa5558b27e0,a706848ea1fe8f9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:58.504 INFO 21062 --- [ocp-schedules-6,61e652d66767940d,a33133d44e5fffc2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:13:58.505 INFO 21062 --- [ocp-schedules-30,8aacab5e970a7e07,335e8960d10431e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:13:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:13:58.511 INFO 21062 --- [ocp-schedules-3,8d6d6b76303db1c5,8fa16706ca1156d1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:13:58.511 INFO 21062 --- [ocp-schedules-17,509481d7b5aeba5f,ef610efb84d432c6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:13:58.511 INFO 21062 --- [ocp-schedules-17,509481d7b5aeba5f,ef610efb84d432c6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:13:58.511 INFO 21062 --- [ocp-schedules-3,8d6d6b76303db1c5,8fa16706ca1156d1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:13:58.511 INFO 21062 --- [ocp-schedules-25,9685dd3bc71696f2,ea8c19c4f7fa9b8a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:13:58.511 INFO 21062 --- [ocp-schedules-25,9685dd3bc71696f2,ea8c19c4f7fa9b8a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:13:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:13:58.588403+08:00, lastProcessTime=2024-11-04T15:13:48.591400+08:00 2024-11-04 15:13:58.749 INFO 21062 --- [ocp-async-7,0969fd9f16cfee30,4e5f4d7287de3a2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:13:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:13:58.588403+08:00 2024-11-04 15:13:59.452 INFO 21062 --- [ocp-schedules-5,1886ed40e01c0ca0,eb6a0eba1bf0ecc9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:13:59.505 WARN 21062 --- [ocp-async-6,29aa650952cc606f,10cddddd2a2edf09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:13:59.509 INFO 21062 --- [ocp-async-7,1afeb149469ae122,8f9c3ffd3e301c6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:13:59.856 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:13:59.857 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:13:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ef14fc83fa35caa5,a43a32b339650ddd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704440, interval=60 2024-11-04 15:14:01.426 INFO 21062 --- [ocp-schedules-21,b875de5f14ad0fc5,687e3666a4e892f5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:14:01.505 WARN 21062 --- [ocp-async-7,2a5fd0a8e6e99a11,43c9ce91eff95e43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:01.509 INFO 21062 --- [ocp-async-4,2db6ee6e3c3b2ffc,6c8cdad4819c9815] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:02.857 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:02.858 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:02.983 INFO 21062 --- [prometheus-minute-schedule-1,615a51c05f34f59b,2062725a61abfb4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:03.186 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,485d0d471ed44922,c8e42900606f1053] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=485d0d471ed44922, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 15:14:03.364 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,485d0d471ed44922,c8e42900606f1053] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=485d0d471ed44922, duration=187 ms] 2024-11-04 15:14:03.505 WARN 21062 --- [ocp-async-8,c2e4b3adc15632ab,0486d22762e7da99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:03.509 INFO 21062 --- [ocp-async-10,30112fb3e7801e84,a488e213831fa99d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:04.098 INFO 21062 --- [ocp-schedules-15,c805f95cdb2974a5,42153e1a09c758c5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:14:05.506 WARN 21062 --- [ocp-async-1,a629865bb0e1a777,1cc68c05ba7cd93b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:05.510 INFO 21062 --- [ocp-async-6,a21ad8452fdacc3f,d9ef02f65c0d41e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:05.858 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:05.859 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:05.983 INFO 21062 --- [prometheus-minute-schedule-1,aa0620c0274e66d7,ca4d4ad517bae775] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:07.233 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d1a3198d1724a710,6ff191540bbfe94f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=d1a3198d1724a710, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 15:14:07.234 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,d1a3198d1724a710,6ff191540bbfe94f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=d1a3198d1724a710, duration=9 ms] 2024-11-04 15:14:07.506 WARN 21062 --- [ocp-async-7,f6e6cc2f59edcaa0,6a8526e9563148f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:07.510 INFO 21062 --- [ocp-async-3,ff3cbcc395dfb3cd,c9c6d57af309c78a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:08.505 INFO 21062 --- [ocp-schedules-8,6ce84fa5558b27e0,a706848ea1fe8f9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:14:08.505 INFO 21062 --- [ocp-schedules-30,8aacab5e970a7e07,335e8960d10431e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:14:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:14:08.589295+08:00, lastProcessTime=2024-11-04T15:13:58.588403+08:00 2024-11-04 15:14:08.749 INFO 21062 --- [ocp-async-10,a99c27433b5e1f61,3ac652ed21c76bca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:14:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:14:08.589295+08:00 2024-11-04 15:14:08.859 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:08.860 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2504c6691b0bcb0b,8f8b3623ef0cb051] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:09.506 WARN 21062 --- [ocp-async-8,361adcefd1292b8c,8fa2698b4cc7005f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:09.510 INFO 21062 --- [ocp-async-10,36352b2b660b0c71,fb04714e3bf787cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:09.574 INFO 21062 --- [ocp-schedules-12,d50f4915bde39c9d,c9b5043215518c4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:14:09.574 INFO 21062 --- [ocp-schedules-12,39ed95b6e9802b7e,0c53affee1d4141a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:14:09.578 INFO 21062 --- [ocp-schedules-12,39ed95b6e9802b7e,0c53affee1d4141a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:09.608 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:14:09.608 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:14:10.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:14:10.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:14:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:14:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:14:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:14:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:14:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:14:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:14:11.429 INFO 21062 --- [ocp-schedules-6,347ea9e9e16432b8,1ab60ca063652426] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:14:11.506 WARN 21062 --- [ocp-async-2,05f159d193d394c5,84b35dacba8fdd1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:11.510 INFO 21062 --- [ocp-async-6,0509f1f32a4cba0e,fccfeb8e6fcbd93b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:11.778 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=1, affectRows=1 2024-11-04 15:14:11.779 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=1, validCount=1 2024-11-04 15:14:11.780 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=7, affectRows=7 2024-11-04 15:14:11.780 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=7, validCount=7 2024-11-04 15:14:11.780 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=14, affectRows=14 2024-11-04 15:14:11.780 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=14, validCount=14 2024-11-04 15:14:11.781 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=6, affectRows=6 2024-11-04 15:14:11.781 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=6, validCount=6 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=36, affectRows=36 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=23, affectRows=23 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=36, validCount=36 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=23, validCount=23 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=50, affectRows=50 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=45, affectRows=45 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=32, affectRows=32 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=71, affectRows=71 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=32, validCount=32 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=45, validCount=45 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=50, affectRows=50 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=50, validCount=50 2024-11-04 15:14:11.782 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=70, validCount=70 2024-11-04 15:14:11.783 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=50, validCount=50 2024-11-04 15:14:11.783 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704380, nodeCount=87, affectRows=87 2024-11-04 15:14:11.783 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704380, nodeCount=85, validCount=85 2024-11-04 15:14:11.860 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:11.861 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:11.983 INFO 21062 --- [prometheus-minute-schedule-1,61f8ec8577b92598,f63966def94de1ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:12.746 INFO 21062 --- [ocp-schedules-24,745ced332a0be0c1,13bdb161d0a64f43] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:14:12.752 INFO 21062 --- [ocp-schedules-24,745ced332a0be0c1,13bdb161d0a64f43] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:14:12.752 INFO 21062 --- [ocp-schedules-24,745ced332a0be0c1,13bdb161d0a64f43] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:14:13.506 WARN 21062 --- [ocp-async-7,e857a23f366d1eda,76ea5a376f7b23c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:13.510 INFO 21062 --- [ocp-async-3,ed438d5ef1d0b6e0,dd60dbe190fd2e50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:13.984 INFO 21062 --- [prometheus-cache-monitor1,26a7c4171d0e65e8,4295ed2efa59283c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:14:13.984 INFO 21062 --- [prometheus-cache-monitor1,26a7c4171d0e65e8,4295ed2efa59283c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:14:14.145 INFO 21062 --- [ocp-schedules-7,286393c463aa6c4d,b3b34d4f6cc75d13] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:14:14.861 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:14.862 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d22fc008f763993a,698a0bdc894c1dd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:15.506 WARN 21062 --- [ocp-async-5,450e67d79488cd85,371c2ac637ba7ade] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:15.510 INFO 21062 --- [ocp-async-4,e8b2af05b683ddcb,904b0caec3b5a530] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:17.506 WARN 21062 --- [ocp-async-10,c4490f4f526e1a72,095e2249325965bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:17.511 INFO 21062 --- [ocp-async-1,2c1ebbeea28bc2ac,1820e008caed61d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:17.862 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:17.863 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:17.983 INFO 21062 --- [prometheus-minute-schedule-1,46c398b5f44ebde4,cb538a84e117eb8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:14:18.589512+08:00, lastProcessTime=2024-11-04T15:14:08.589295+08:00 2024-11-04 15:14:18.750 INFO 21062 --- [ocp-async-9,1329d54c8e9f6bec,bb0e837716ddc8e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:14:18.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:14:18.589512+08:00 2024-11-04 15:14:19.507 WARN 21062 --- [ocp-async-9,3bf105652404ae48,68bea58196c13c04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:19.511 INFO 21062 --- [ocp-async-3,a07f55c50dd42013,c2db89520942986e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:20.864 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:20.865 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8d8e5cbf5c144261,4136e15e34414e7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:21.432 INFO 21062 --- [ocp-schedules-16,c67ead77c1c006b3,2449ae2dc3cbadd9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:14:21.507 WARN 21062 --- [ocp-async-8,d1e205bc62757b5c,f52691007871fae8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:21.511 INFO 21062 --- [ocp-async-4,9d1c41d5ff75d773,7ba8ebeb1335c819] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:23.507 WARN 21062 --- [ocp-async-2,92f8671b075135ad,fb68532e0674b5f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:23.511 INFO 21062 --- [ocp-async-1,38c2129894ad6cd6,bae76c53e3840116] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:23.865 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:23.865 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:23.983 INFO 21062 --- [prometheus-minute-schedule-1,74a58ded59a8023e,b27a8cd04386b05c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:24.172 INFO 21062 --- [ocp-schedules-1,7849f5ca552153ef,999ad1a25db16ca3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:14:25.507 WARN 21062 --- [ocp-async-9,72b8d557fcabe5ea,26130bcecdf12b9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:25.511 INFO 21062 --- [ocp-async-7,d6fc0d2266389df1,54f6c2e3b5993177] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:26.866 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:26.866 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:26.983 INFO 21062 --- [prometheus-minute-schedule-1,463e1a1cbbd72195,ffffb134a2a7386e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e03aaedf542bf38f,adfce23fcef34d06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:14:27.508 WARN 21062 --- [ocp-async-3,51cabecf3c4ab4b1,a87c2551ac2aeb63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:27.512 INFO 21062 --- [ocp-async-5,5247e1f9cbea28ba,65d564fac9500c90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:28.498 INFO 21062 --- [ocp-schedules-17,84c2c0579c5f6214,cc85ef179a213f4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:14:28.500 INFO 21062 --- [ocp-schedules-6,08233bc4554d435d,9db599c59253170a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:14:28.500 INFO 21062 --- [ocp-schedules-24,a24493b2b234b670,cb1905428b792e5e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:14:28.500 INFO 21062 --- [ocp-schedules-26,d0a48b3be159dc70,5a3f73d8f574e13b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:14:28.503 INFO 21062 --- [ocp-schedules-15,24ee362eb754ecd4,d3e15a5c67e42c39] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:14:28.504 INFO 21062 --- [ocp-schedules-17,84c2c0579c5f6214,cc85ef179a213f4b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:14:28.504 INFO 21062 --- [ocp-schedules-24,a24493b2b234b670,cb1905428b792e5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:28.504 INFO 21062 --- [ocp-schedules-26,d0a48b3be159dc70,5a3f73d8f574e13b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:28.504 INFO 21062 --- [ocp-schedules-1,ebd7a9c48e8a0d4f,db1cece7a2a3b761] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:14:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:14:28.506 INFO 21062 --- [ocp-schedules-6,08233bc4554d435d,9db599c59253170a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:14:28.510 INFO 21062 --- [ocp-schedules-8,1daf59f849a8f10f,fc6d0047eb0c5820] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:14:28.510 INFO 21062 --- [ocp-schedules-20,d37856c3608258a6,68eaa3d92a377da7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:14:28.510 INFO 21062 --- [ocp-schedules-19,51bfc7c56864905e,68e0cab64b7a47e2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:14:28.510 INFO 21062 --- [ocp-schedules-20,d37856c3608258a6,68eaa3d92a377da7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:14:28.510 INFO 21062 --- [ocp-schedules-19,51bfc7c56864905e,68e0cab64b7a47e2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:14:28.510 INFO 21062 --- [ocp-schedules-8,1daf59f849a8f10f,fc6d0047eb0c5820] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:14:28.586 INFO 21062 --- [ocp-schedules-25,34b1bb8ddf5ea46b,89daa5fe0c14d36d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:14:28.586 INFO 21062 --- [ocp-schedules-25,34b1bb8ddf5ea46b,89daa5fe0c14d36d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:14:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:14:28.589904+08:00, lastProcessTime=2024-11-04T15:14:18.589512+08:00 2024-11-04 15:14:28.615 INFO 21062 --- [ocp-schedules-18,f34bd4c36b2f7f1f,8e535e470be98140] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:14:28.615+08:00, tenantSize=0 2024-11-04 15:14:28.715 INFO 21062 --- [ocp-schedules-10,96d648eb949c8340,86b50945aa7f2d95] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 15:14:28.715 INFO 21062 --- [ocp-schedules-10,96d648eb949c8340,86b50945aa7f2d95] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 15:14:28.750 INFO 21062 --- [ocp-async-9,3a39c1c4865a5f9b,e3a53c262c7e7ed7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:14:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:14:28.589904+08:00 2024-11-04 15:14:28.941 INFO 21062 --- [ocp-schedules-6,c639684f02c7d122,98388ea30ef4ef0a] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:14:28.941 INFO 21062 --- [ocp-schedules-6,c639684f02c7d122,98388ea30ef4ef0a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:14:28.960 INFO 21062 --- [ocp-schedules-15,6c83dc48b55edcd5,64a4561f37587254] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:14:29.266 INFO 21062 --- [ocp-schedules-19,38f0d1f889dbe1ed,5f0db52a8cc14190] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:14:29.270 INFO 21062 --- [ocp-schedules-19,38f0d1f889dbe1ed,5f0db52a8cc14190] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:29.271 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:14:29.329 INFO 21062 --- [ocp-schedules-30,dfcafeeb318e9a0e,4a1ef81b39e2d55c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:14:29.329 INFO 21062 --- [ocp-schedules-30,dfcafeeb318e9a0e,4a1ef81b39e2d55c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:14:29.454 INFO 21062 --- [ocp-schedules-11,5003eea5ea18c39a,b131d9e87a6e2095] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:14:29.478 INFO 21062 --- [ocp-schedules-29,76007ad97361bc8b,98d06db04177e744] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:14:29.484 INFO 21062 --- [ocp-schedules-29,76007ad97361bc8b,98d06db04177e744] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:14:29.484 INFO 21062 --- [ocp-schedules-29,76007ad97361bc8b,98d06db04177e744] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:14:29.508 WARN 21062 --- [ocp-async-9,fcef7f3314a3293f,765f5594c42c0cc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:29.512 INFO 21062 --- [ocp-async-6,de80506f314f377b,243d304fa088e3a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:29.867 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:29.868 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:29.940 INFO 21062 --- [ocp-schedules-28,711b580c4c78fa5c,ac355850258b74be] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:14:29.940 INFO 21062 --- [ocp-schedules-28,711b580c4c78fa5c,ac355850258b74be] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:14:29.943 INFO 21062 --- [ocp-schedules-28,711b580c4c78fa5c,ac355850258b74be] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:14:29.943 INFO 21062 --- [ocp-schedules-28,711b580c4c78fa5c,ac355850258b74be] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:14:29.944 INFO 21062 --- [ocp-schedules-28,711b580c4c78fa5c,ac355850258b74be] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:14:29.944 INFO 21062 --- [ocp-schedules-28,711b580c4c78fa5c,ac355850258b74be] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:14:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b5b12ce9f3020170,14f678d5dede29aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:14:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:14:31.434 INFO 21062 --- [ocp-schedules-20,3b43eaeb734338b1,6fe25bf903185d2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:14:31.508 WARN 21062 --- [ocp-async-3,7868241b1372d786,f796f2a7c68fce92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:31.512 INFO 21062 --- [ocp-async-8,e253953afaac2200,dba8704d4e10e3e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:32.754 INFO 21062 --- [ocp-schedules-5,ad5ab8aabf9a61b7,d8bf81b4602e945e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:14:32.759 INFO 21062 --- [ocp-schedules-5,ad5ab8aabf9a61b7,d8bf81b4602e945e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:14:32.759 INFO 21062 --- [ocp-schedules-5,ad5ab8aabf9a61b7,d8bf81b4602e945e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:14:32.868 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:32.869 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:32.962 INFO 21062 --- [ocp-schedules-22,807feec995fd6c02,d556e933af09ece5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:14:32.962 INFO 21062 --- [ocp-schedules-22,807feec995fd6c02,d556e933af09ece5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:14:32.983 INFO 21062 --- [prometheus-minute-schedule-1,02f3688706966349,ef2cbcc151d83fa5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:32.988 INFO 21062 --- [ocp-schedules-22,807feec995fd6c02,d556e933af09ece5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:14:32.988 INFO 21062 --- [ocp-schedules-22,807feec995fd6c02,d556e933af09ece5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:14:32.990 INFO 21062 --- [ocp-schedules-29,2481d5234c19b44f,bc1df5a218bea865] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:14:33.191 INFO 21062 --- [ocp-schedules-4,a587286d3d19803f,9ac6d44b3dc588af] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:14:33.191+08:00, clusters=[1] 2024-11-04 15:14:33.192 INFO 21062 --- [ocp-schedules-4,a587286d3d19803f,9ac6d44b3dc588af] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:14:33.463 INFO 21062 --- [ocp-schedules-9,97cce1cab600aada,84ff4c3736779b0f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:14:33.463+08:00, clusters=[1] 2024-11-04 15:14:33.467 INFO 21062 --- [ocp-schedules-9,97cce1cab600aada,84ff4c3736779b0f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:14:33.502 INFO 21062 --- [ocp-schedules-10,bdae3740eb0b5ff4,ac1e8edb415358a1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:14:33.505 INFO 21062 --- [ocp-schedules-10,bdae3740eb0b5ff4,ac1e8edb415358a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:33.508 WARN 21062 --- [ocp-async-2,a847f16020a3390c,7b421e1281cd036c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:33.512 INFO 21062 --- [ocp-async-4,3cc32e514965d4d1,7c0332644efb6894] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:33.562 ERROR 21062 --- [ocp-schedules-10,bdae3740eb0b5ff4,ac1e8edb415358a1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:14:33.562 INFO 21062 --- [ocp-schedules-10,bdae3740eb0b5ff4,ac1e8edb415358a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:14:34.214 INFO 21062 --- [ocp-schedules-27,71c3e9f46fcd9cdf,b6bc211a14765469] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:14:35.152 INFO 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:14:35.152 INFO 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:14:35.152 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:14:35.152 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:14:35.164 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:14:35.164 INFO 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:14:35.164 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:14:35.166 WARN 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:14:35.166 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:14:35.166 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:14:35.167 INFO 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:14:35.167 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:14:35.167 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:14:35.168 WARN 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:14:35.168 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:14:35.169 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:14:35.177 INFO 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:14:35.177 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:14:35.177 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:14:35.189 ERROR 21062 --- [ocp-schedules-22,7b0768214b762a4b,06f3cbe20aa86848] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:14:35.508 WARN 21062 --- [ocp-async-9,4cbd264858e9949b,b555f94c85a8bd9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:35.512 INFO 21062 --- [ocp-async-1,2a06504a024040b0,a2b85bdd34c3c196] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:35.869 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:35.870 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:35.983 INFO 21062 --- [prometheus-minute-schedule-1,5f1a4e55a51d0a44,3a6c02136f01dca0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:37.508 WARN 21062 --- [ocp-async-3,29d9b61588bab4ec,f40496ddadc3c4ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:37.513 INFO 21062 --- [ocp-async-7,7bc80ba3df181784,5dfed7fb1de9d3b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:38.504 INFO 21062 --- [ocp-schedules-26,d0a48b3be159dc70,5a3f73d8f574e13b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:14:38.504 INFO 21062 --- [ocp-schedules-24,a24493b2b234b670,cb1905428b792e5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:14:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:14:38.589468+08:00, lastProcessTime=2024-11-04T15:14:28.589904+08:00 2024-11-04 15:14:38.750 INFO 21062 --- [ocp-async-4,8867af3b93ce3de5,bb4e3efcee6e02d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:14:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:14:38.589468+08:00 2024-11-04 15:14:38.870 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:38.870 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:38.983 INFO 21062 --- [prometheus-minute-schedule-1,20b673809e5ff442,5c4ef8f30f66ce7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:39.271 INFO 21062 --- [ocp-schedules-19,38f0d1f889dbe1ed,5f0db52a8cc14190] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:14:39.509 WARN 21062 --- [ocp-async-4,b27d182858c8fb0b,783e84033fd770b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:39.513 INFO 21062 --- [ocp-async-9,f42e5bf9e217cdcd,c94179fa8db06006] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:39.609 INFO 21062 --- [ocp-schedules-12,39ed95b6e9802b7e,0c53affee1d4141a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:14:39.609 INFO 21062 --- [ocp-schedules-12,7eef2211bb68bbb7,d019055785aa8182] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:14:39.613 INFO 21062 --- [ocp-schedules-12,7eef2211bb68bbb7,d019055785aa8182] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:39.644 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:14:39.644 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:14:40.002 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:14:40.002 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:14:41.437 INFO 21062 --- [ocp-schedules-13,6ee2ca77038c9bea,957eb76e7212031b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:14:41.509 WARN 21062 --- [ocp-async-10,395d6be12a562088,195c7e2d36d8cc13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:41.513 INFO 21062 --- [ocp-async-3,d3a89736cbb30a34,6c9fbcb2b10fb8d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:14:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:14:41.829 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:14:41.855 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:14:41.855 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:14:41.871 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:41.871 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:41.983 INFO 21062 --- [prometheus-minute-schedule-1,63f9057593bc7e5e,a5ed479514263449] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:43.487 INFO 21062 --- [ocp-schedules-21,f16ad61d1b28d1f5,0a74d502ebaaa862] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:14:43.492 INFO 21062 --- [ocp-schedules-21,f16ad61d1b28d1f5,0a74d502ebaaa862] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:43.509 WARN 21062 --- [ocp-async-6,2d603b5273b7fe59,2a76bf99dc44a62f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:43.513 INFO 21062 --- [ocp-async-7,b570e98c87f75b2d,9a1aea1d1e339c18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:43.578 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:14:43.580 ERROR 21062 --- [ocp-schedules-21,f16ad61d1b28d1f5,0a74d502ebaaa862] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:14:43.581 INFO 21062 --- [ocp-schedules-21,f16ad61d1b28d1f5,0a74d502ebaaa862] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:14:44.237 INFO 21062 --- [ocp-schedules-16,cd29d0e13c01e826,4b9d14b9fbe48ec6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:14:44.502 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:14:44.502 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:14:44.503 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.503 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:14:44.505 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.506 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.506 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.507 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:14:44.522 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.523 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.524 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.524 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:14:44.536 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.537 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.537 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.538 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:14:44.552 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.552 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.553 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.554 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:14:44.567 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.567 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.568 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.568 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:14:44.574 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.574 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.575 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.575 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:14:44.579 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.579 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.580 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.580 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:14:44.585 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.585 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.586 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.586 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:14:44.592 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.592 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.593 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.593 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:14:44.598 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.599 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.600 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.601 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:14:44.606 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.607 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.607 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.608 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:14:44.616 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.617 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.617 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.618 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:14:44.631 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.631 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.632 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.632 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:14:44.637 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.637 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.638 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.638 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:14:44.644 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.644 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.645 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.645 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:14:44.650 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.651 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.651 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.652 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:14:44.655 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.656 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.656 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.657 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:14:44.661 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.661 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.661 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.662 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:14:44.667 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.667 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.668 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.668 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:14:44.673 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.673 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.673 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.674 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:14:44.678 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.678 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.679 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.679 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:14:44.684 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.684 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.685 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.685 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:14:44.690 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.690 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.691 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.691 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:14:44.695 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.696 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.696 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.696 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:14:44.701 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.701 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.702 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.702 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:14:44.706 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.706 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.707 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.707 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:14:44.713 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.713 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.714 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.714 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:14:44.719 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.719 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.720 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.720 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:14:44.724 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.725 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.725 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.726 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:14:44.730 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.731 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.731 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.732 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:14:44.735 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.735 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.736 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.736 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:14:44.740 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.741 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.741 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.742 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:14:44.746 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.746 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.747 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.747 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:14:44.751 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.752 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.752 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.753 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:14:44.761 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.761 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.761 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.762 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:14:44.769 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.769 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.769 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.770 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:14:44.775 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.775 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.776 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.776 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:14:44.781 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.782 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.782 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.783 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:14:44.786 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.787 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.787 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.788 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:14:44.792 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.792 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.792 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.793 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:14:44.796 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.797 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.797 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.797 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:14:44.801 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.801 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.802 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.802 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:14:44.811 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.812 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.812 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.813 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:14:44.833 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.833 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.834 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.834 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:14:44.840 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.840 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.840 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.841 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:14:44.847 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.847 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.847 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.848 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:14:44.854 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.855 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.855 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.856 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:14:44.860 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.860 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.861 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.861 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:14:44.865 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.866 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.866 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.867 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:14:44.871 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:14:44.871 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:14:44.871 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:44.872 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.872 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:14:44.872 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:44.874 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.874 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.874 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.875 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:14:44.880 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.880 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:14:44.881 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:14:44.881 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:14:44.885 INFO 21062 --- [ocp-schedules-9,c44520d8715edce2,68a3585f34ea298c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:14:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d3430effb8e59193,0dd0b218b6398e81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:45.509 WARN 21062 --- [ocp-async-7,099ff4c8cda61be7,e5d97f0e41d2d307] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:45.513 INFO 21062 --- [ocp-async-4,c2ecae3f2e8988d3,f8072a300aa2b14a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:47.509 WARN 21062 --- [ocp-async-4,816189a63ff2b514,04fcd9704d4645c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:47.514 INFO 21062 --- [ocp-async-1,d9f5f30f9d3efc16,3497f340fa069eb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:47.873 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:47.873 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:47.983 INFO 21062 --- [prometheus-minute-schedule-1,2e7d8564004139b3,503cef49b59ad107] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:14:48.590958+08:00, lastProcessTime=2024-11-04T15:14:38.589468+08:00 2024-11-04 15:14:48.750 INFO 21062 --- [ocp-async-3,60bbed9696590f07,8329216f3af4d239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:14:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:14:48.590958+08:00 2024-11-04 15:14:49.510 WARN 21062 --- [ocp-async-6,659d47093db9e90d,6558a715336af447] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:49.514 INFO 21062 --- [ocp-async-3,b954d36ae4072ba5,386caa6f3d04f6b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:50.873 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:50.875 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:50.983 INFO 21062 --- [prometheus-minute-schedule-1,e80e84105ab4a2dc,c134e8b760fc5690] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:51.440 INFO 21062 --- [ocp-schedules-21,a82e8db46c0da81a,79847b656d6ebf99] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:14:51.510 WARN 21062 --- [ocp-async-2,66caf24880b69d13,0e7410aa69c3d23c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:51.514 INFO 21062 --- [ocp-async-5,0ffeed75befefee6,e4f7eb97c351d217] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:52.763 INFO 21062 --- [ocp-schedules-23,613cc5f269deb5fe,93ec245fb465ba8c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:14:52.769 INFO 21062 --- [ocp-schedules-23,613cc5f269deb5fe,93ec245fb465ba8c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:14:52.769 INFO 21062 --- [ocp-schedules-23,613cc5f269deb5fe,93ec245fb465ba8c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:14:53.510 WARN 21062 --- [ocp-async-10,324c20ac6bc4ecf8,5d319714dc7adbf8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:53.514 INFO 21062 --- [ocp-async-9,d5c61523511fc0c4,8c6849bb09ec79d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:53.875 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:53.876 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:53.983 INFO 21062 --- [prometheus-minute-schedule-1,0e74a67b29375f7d,fb233e99a02457d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:54.280 INFO 21062 --- [ocp-schedules-23,a7fb29f0ed7b4772,b646727d121d6252] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:14:55.510 WARN 21062 --- [ocp-async-6,9e56212c0ae96db0,020d2fbf298ac9b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:55.514 INFO 21062 --- [ocp-async-8,03104c6d80eaa61e,fc530d786d427482] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:56.876 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:56.877 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:56.983 INFO 21062 --- [prometheus-minute-schedule-1,6606e5a71aa69c25,d52153496f199e0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:14:57.510 WARN 21062 --- [ocp-async-7,298bb0c92418255a,2932d259df8c04c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:57.514 INFO 21062 --- [ocp-async-3,07b6675abd206a6e,68b71761059e6b2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:58.500 INFO 21062 --- [ocp-schedules-26,e96d5e897da4b30e,c6dfeb365c71da4a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:14:58.500 INFO 21062 --- [ocp-schedules-28,bff9b16c2f9594b3,ef843c9e6ca8401f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:14:58.502 INFO 21062 --- [ocp-schedules-10,e6cfe967caac3ed2,ef145cd37f729827] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:14:58.504 INFO 21062 --- [ocp-schedules-26,e96d5e897da4b30e,c6dfeb365c71da4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:58.504 INFO 21062 --- [ocp-schedules-28,bff9b16c2f9594b3,ef843c9e6ca8401f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:14:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:14:58.504 INFO 21062 --- [ocp-schedules-24,6974302ff7f51e05,3b8190ec94ca16f2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:14:58.509 INFO 21062 --- [ocp-schedules-9,64050ca816c40b5e,e49e2c51d46bd1f2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:14:58.510 INFO 21062 --- [ocp-schedules-20,99b4190c795b3a8b,af6d7ed96a8f7cc1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:14:58.510 INFO 21062 --- [ocp-schedules-9,64050ca816c40b5e,e49e2c51d46bd1f2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:14:58.510 INFO 21062 --- [ocp-schedules-20,99b4190c795b3a8b,af6d7ed96a8f7cc1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:14:58.510 INFO 21062 --- [ocp-schedules-7,bf5fcda06e169289,455342ec61dac797] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:14:58.510 INFO 21062 --- [ocp-schedules-7,bf5fcda06e169289,455342ec61dac797] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:14:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:14:58.589647+08:00, lastProcessTime=2024-11-04T15:14:48.590958+08:00 2024-11-04 15:14:58.750 INFO 21062 --- [ocp-async-5,a8cdcc749908edcc,7e89031cd759ef00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:14:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:14:58.589647+08:00 2024-11-04 15:14:59.457 INFO 21062 --- [ocp-schedules-29,0001a53ffa04bd19,582594be879bd437] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:14:59.510 WARN 21062 --- [ocp-async-10,b6dedc268e62db45,c944dfa7a2f7bcf1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:14:59.514 INFO 21062 --- [ocp-async-9,0bdeadf6e4a9b41b,05371f31f38730b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:14:59.877 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:14:59.878 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:14:59.983 INFO 21062 --- [prometheus-minute-schedule-1,b85ffeb58d59b2e9,50d770cecec19b23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704500, interval=60 2024-11-04 15:15:00.205 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b57d01198ebab2cd] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:10:00.0, nextRun=2024-11-04 15:15:00.0, currentDate=Mon Nov 04 15:15:00 CST 2024 2024-11-04 15:15:00.206 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b57d01198ebab2cd] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:10:00.0, nextRunTime=2024-11-04 15:15:00.0} 2024-11-04 15:15:00.232 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b57d01198ebab2cd] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:15:00.237 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b57d01198ebab2cd] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:15:00.237 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,b57d01198ebab2cd] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:15:01.444 INFO 21062 --- [ocp-schedules-2,a3c624b27a641e42,e7a5be4bda875d6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:15:01.511 WARN 21062 --- [ocp-async-1,271e0dddd0a92fb3,a4efeac6434627a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:01.515 INFO 21062 --- [ocp-async-8,be45f344291917d8,eb84cd107e3204a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:02.878 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:02.880 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:02.983 INFO 21062 --- [prometheus-minute-schedule-1,7ccd075c5771e168,77749eb0a1255ad3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:03.511 WARN 21062 --- [ocp-async-2,96765125aa26edbe,215e647258b1e4a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:03.515 INFO 21062 --- [ocp-async-3,19869a7a850653df,bbf1ed6a2f197413] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:04.307 INFO 21062 --- [ocp-schedules-8,227c9855d20aaa39,d1d872376bc59410] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:15:05.511 WARN 21062 --- [ocp-async-10,e822bca08700e03d,a9bfbd6ba130dbd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:05.515 INFO 21062 --- [ocp-async-5,6608c0dc64069df8,4302c10117a0c103] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:05.880 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:05.882 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:05.983 INFO 21062 --- [prometheus-minute-schedule-1,4fb1f0ff788f3727,d92fe7fa23cfce5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:07.511 WARN 21062 --- [ocp-async-1,f4234cafafcc2d97,4bbf309a6aba40e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:07.515 INFO 21062 --- [ocp-async-6,2e7e716bd83d68f7,4eb8136c72d5288b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:08.001 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:32:07+08:00, updateTime=2024-11-04T15:14:37+08:00, expireTime=null, isLeader=false} 2024-11-04 15:15:08.002 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=10.46.128.24:8090, expireInSeconds=30, createTime=2024-11-04T12:32:07+08:00, updateTime=2024-11-04T15:14:37+08:00, expireTime=null, isLeader=false} 2024-11-04 15:15:08.003 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2024-11-04 15:15:08.005 WARN 21062 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2024-11-04 15:15:08.505 INFO 21062 --- [ocp-schedules-26,e96d5e897da4b30e,c6dfeb365c71da4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:15:08.505 INFO 21062 --- [ocp-schedules-28,bff9b16c2f9594b3,ef843c9e6ca8401f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:15:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:15:08.590132+08:00, lastProcessTime=2024-11-04T15:14:58.589647+08:00 2024-11-04 15:15:08.751 INFO 21062 --- [ocp-async-7,d332bad0391fc45f,f3c821503f9e63cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:15:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:15:08.590132+08:00 2024-11-04 15:15:08.882 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:08.882 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:08.983 INFO 21062 --- [prometheus-minute-schedule-1,5f23c02a3171334e,93b1783bee74c706] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:09.511 WARN 21062 --- [ocp-async-7,24909baeab508b32,3251be09bf921a2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:09.516 INFO 21062 --- [ocp-async-3,f2786149cdced4e0,63abea33f1d038df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:09.645 INFO 21062 --- [ocp-schedules-12,7eef2211bb68bbb7,d019055785aa8182] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:15:09.645 INFO 21062 --- [ocp-schedules-12,68bfe08b67ca730b,ce852feba92ff240] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:15:09.648 INFO 21062 --- [ocp-schedules-12,68bfe08b67ca730b,ce852feba92ff240] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:09.678 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:15:09.678 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:15:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:15:10.003 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:15:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:15:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:15:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:15:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:15:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:15:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:15:11.447 INFO 21062 --- [ocp-schedules-24,fa85d5e1939796a9,793cd776f9d27b41] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:15:11.511 WARN 21062 --- [ocp-async-4,bb49efac21f4ce5b,bae99a67b859e687] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:11.516 INFO 21062 --- [ocp-async-5,6e5946e3d95eb303,be4bd524d2891a90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:11.784 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704440, nodeCount=55, affectRows=55 2024-11-04 15:15:11.784 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704440, nodeCount=64, affectRows=64 2024-11-04 15:15:11.784 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704440, nodeCount=54, validCount=54 2024-11-04 15:15:11.784 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704440, nodeCount=64, validCount=64 2024-11-04 15:15:11.784 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704440, nodeCount=99, affectRows=99 2024-11-04 15:15:11.785 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704440, nodeCount=85, affectRows=85 2024-11-04 15:15:11.785 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704440, nodeCount=98, validCount=98 2024-11-04 15:15:11.785 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704440, nodeCount=119, affectRows=119 2024-11-04 15:15:11.785 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704440, nodeCount=85, validCount=85 2024-11-04 15:15:11.785 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704440, nodeCount=118, validCount=118 2024-11-04 15:15:11.883 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:11.883 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:11.983 INFO 21062 --- [prometheus-minute-schedule-1,78957bf98ec1b64f,4d7f5a01883db302] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:12.771 INFO 21062 --- [ocp-schedules-29,adc6a480f63c4c67,4d551a70b57b11aa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:15:12.776 INFO 21062 --- [ocp-schedules-29,adc6a480f63c4c67,4d551a70b57b11aa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:15:12.776 INFO 21062 --- [ocp-schedules-29,adc6a480f63c4c67,4d551a70b57b11aa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:15:13.512 WARN 21062 --- [ocp-async-9,0b25e7fb93cd0785,a83c4f2b0bdfcf44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:13.516 INFO 21062 --- [ocp-async-6,37c24a7555fffe7f,e03775ae9a384ac7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:13.984 INFO 21062 --- [prometheus-cache-monitor1,dff8ea0cd5a34aff,33b0afd094d573ac] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:15:13.984 INFO 21062 --- [prometheus-cache-monitor1,dff8ea0cd5a34aff,33b0afd094d573ac] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:15:14.347 INFO 21062 --- [ocp-schedules-7,4bc55d73246e7d44,ef2eaf569e11ad22] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:15:14.883 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:14.884 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5589ea367bf5b5e3,cd8162410d5f7cf3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:15.512 WARN 21062 --- [ocp-async-7,adfee8516f6f2678,856cac4946c3b723] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:15.516 INFO 21062 --- [ocp-async-2,664c9cd7c2a48e40,87560ce81ff8d929] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:15.526 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:11:15.507, end=2024-11-04T15:13:15.524} 2024-11-04 15:15:15.535 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:15:15.535 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:15:17.512 WARN 21062 --- [ocp-async-3,2da16177f48465ec,900525f9e2b77a18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:17.517 INFO 21062 --- [ocp-async-10,6121ea78fabc6372,05e709b29ffc6af8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:17.884 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:17.885 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5321f52db1d40134,97d5b1abb8cdc4e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:15:18.591539+08:00, lastProcessTime=2024-11-04T15:15:08.590132+08:00 2024-11-04 15:15:18.751 INFO 21062 --- [ocp-async-1,b96d3a6f4c3c32fd,9b376923e17f8a89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:15:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:15:18.591539+08:00 2024-11-04 15:15:19.512 WARN 21062 --- [ocp-async-9,8e11aef6a5ca888f,8b8ee9e9c13fa24b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:19.517 INFO 21062 --- [ocp-async-1,1a35c810056c55f5,c5d615019bf94572] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:20.885 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:20.886 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:20.983 INFO 21062 --- [prometheus-minute-schedule-1,63a487f5dd792562,19689cbfcaefd387] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:21.449 INFO 21062 --- [ocp-schedules-9,de4735ea058b3cdf,7cbc843c2307ea4b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:15:21.512 WARN 21062 --- [ocp-async-7,a681c3a86e953e36,1aaf9dbd936c91a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:21.517 INFO 21062 --- [ocp-async-2,5f0248e2fb94d49e,a94018cb320f7621] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:22.846 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ea85892f3809e0a7,9dc7dcb3c09d3230] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=ea85892f3809e0a7, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 15:15:22.855 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,ea85892f3809e0a7,9dc7dcb3c09d3230] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=ea85892f3809e0a7, duration=20 ms] 2024-11-04 15:15:23.512 WARN 21062 --- [ocp-async-2,cba323885bda3050,b67a746262632be4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:23.517 INFO 21062 --- [ocp-async-10,93780352565f6f31,1d4acc11ce3545c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:23.886 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:23.887 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4ba4335b05292e23,ebcb961e59e6ddb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:24.372 INFO 21062 --- [ocp-schedules-6,6fc267a3034a3943,0a6925ff30b3e724] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:15:24.614 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6e45b9c94f9d2734,7e6329224116d746] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments?taskInstanceId=2&subtaskInstanceId=42, client=10.130.47.17, traceId=6e45b9c94f9d2734, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,42,] 2024-11-04 15:15:24.623 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,6e45b9c94f9d2734,7e6329224116d746] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/42/logSegments client=10.130.47.17, traceId=6e45b9c94f9d2734, duration=16 ms] 2024-11-04 15:15:25.513 WARN 21062 --- [ocp-async-10,a4c8b674ddebe864,cf1637e7394d14af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:25.517 INFO 21062 --- [ocp-async-9,e29e992424c6fa1c,4445a0103b3a9b33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:26.887 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:26.888 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:26.983 INFO 21062 --- [prometheus-minute-schedule-1,67041f7da7a2e722,0663f094db178c44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:26.984 INFO 21062 --- [prometheus-hour-schedule-1,63e6771bd9df0a0d,053ee8b00014ff90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:15:27.513 WARN 21062 --- [ocp-async-1,613b04b44f1865ae,563c3ebf16328fd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:27.518 INFO 21062 --- [ocp-async-7,69fea2c302285d61,7f62bc3fbc974edc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:28.498 INFO 21062 --- [ocp-schedules-26,2f0d20106b4e3994,bc67ea032d353325] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:15:28.499 INFO 21062 --- [ocp-schedules-11,3ec44478c3825de3,c5c7a01e0e32259a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:15:28.500 INFO 21062 --- [ocp-schedules-24,c665663702cf2214,123d28db4e65c31f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:15:28.500 INFO 21062 --- [ocp-schedules-7,6550423a3fcb0b48,4216b5762ba254f6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:15:28.502 INFO 21062 --- [ocp-schedules-30,762779f6083a4eac,24a6fb00c6934384] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:15:28.503 INFO 21062 --- [ocp-schedules-26,2f0d20106b4e3994,bc67ea032d353325] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:15:28.503 INFO 21062 --- [ocp-schedules-24,c665663702cf2214,123d28db4e65c31f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:15:28.504 INFO 21062 --- [ocp-schedules-7,6550423a3fcb0b48,4216b5762ba254f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:28.505 INFO 21062 --- [ocp-schedules-20,96375d5424298cad,30a3a0269fb60237] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:15:28.505 INFO 21062 --- [ocp-schedules-11,3ec44478c3825de3,c5c7a01e0e32259a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:15:28.509 INFO 21062 --- [ocp-schedules-29,4608cd2b3a17b848,c907784268d78899] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:15:28.509 INFO 21062 --- [ocp-schedules-29,4608cd2b3a17b848,c907784268d78899] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:15:28.510 INFO 21062 --- [ocp-schedules-14,961f8517c3545e82,6dd4974f7aa6a52a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:15:28.510 INFO 21062 --- [ocp-schedules-14,961f8517c3545e82,6dd4974f7aa6a52a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:15:28.510 INFO 21062 --- [ocp-schedules-16,27fbfe542e43d9a5,d5abf1fda7f4270a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:15:28.510 INFO 21062 --- [ocp-schedules-16,27fbfe542e43d9a5,d5abf1fda7f4270a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:15:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:15:28.589697+08:00, lastProcessTime=2024-11-04T15:15:18.591539+08:00 2024-11-04 15:15:28.659 INFO 21062 --- [ocp-schedules-8,1bd1cd4fb8ef16c4,02bb7c2dab766ae4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:15:28.660 INFO 21062 --- [ocp-schedules-8,1bd1cd4fb8ef16c4,02bb7c2dab766ae4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:15:28.751 INFO 21062 --- [ocp-async-5,46fff6fcf494175d,5985e362bfd806bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:15:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:15:28.589697+08:00 2024-11-04 15:15:28.962 INFO 21062 --- [ocp-schedules-27,bfe21036ef0e7e4b,e40844d23281a417] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:15:29.333 INFO 21062 --- [ocp-schedules-16,07650214bccd34d8,294d2df228bf0e7a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:15:29.333 INFO 21062 --- [ocp-schedules-16,07650214bccd34d8,294d2df228bf0e7a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:15:29.459 INFO 21062 --- [ocp-schedules-19,e9c55510e289c253,e4f55a022d8d4b51] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:15:29.484 INFO 21062 --- [ocp-schedules-23,53a1bb384baee32f,9c4eaba5183b51d5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:15:29.489 INFO 21062 --- [ocp-schedules-23,53a1bb384baee32f,9c4eaba5183b51d5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:15:29.489 INFO 21062 --- [ocp-schedules-23,53a1bb384baee32f,9c4eaba5183b51d5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:15:29.513 WARN 21062 --- [ocp-async-10,1a0d7082c0c8af1c,db81da2073423fd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:29.518 INFO 21062 --- [ocp-async-5,9d03f44c33152feb,cdf769cdba6f38ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:29.888 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:29.889 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:29.944 INFO 21062 --- [ocp-schedules-25,ec13097d979106f7,25c5f036dc5e38ed] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:15:29.945 INFO 21062 --- [ocp-schedules-25,ec13097d979106f7,25c5f036dc5e38ed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:15:29.946 INFO 21062 --- [ocp-schedules-25,ec13097d979106f7,25c5f036dc5e38ed] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:15:29.946 INFO 21062 --- [ocp-schedules-25,ec13097d979106f7,25c5f036dc5e38ed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:15:29.947 INFO 21062 --- [ocp-schedules-25,ec13097d979106f7,25c5f036dc5e38ed] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:15:29.947 INFO 21062 --- [ocp-schedules-25,ec13097d979106f7,25c5f036dc5e38ed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:15:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4ca05fc5d2a757e8,85570a5bb96f48ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:15:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:15:31.452 INFO 21062 --- [ocp-schedules-1,e95dd6493f4925a4,2e15302dcbef7f9d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:15:31.513 WARN 21062 --- [ocp-async-1,ce0f9029a68feb1b,cad1d3dbcf9503cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:31.518 INFO 21062 --- [ocp-async-9,0c29abd420c45b25,d86b3d1901fcc31c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:32.779 INFO 21062 --- [ocp-schedules-27,7f9a24640b820691,08cc0f05cc3d40f9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:15:32.784 INFO 21062 --- [ocp-schedules-27,7f9a24640b820691,08cc0f05cc3d40f9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:15:32.784 INFO 21062 --- [ocp-schedules-27,7f9a24640b820691,08cc0f05cc3d40f9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:15:32.889 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:32.890 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d337c41df2ea5ced,a34942a62821d12f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:32.988 INFO 21062 --- [ocp-schedules-11,073e015c3b6588e2,90dd8b91f474e54d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:15:32.988 INFO 21062 --- [ocp-schedules-11,073e015c3b6588e2,90dd8b91f474e54d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:15:33.017 INFO 21062 --- [ocp-schedules-11,073e015c3b6588e2,90dd8b91f474e54d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:15:33.017 INFO 21062 --- [ocp-schedules-11,073e015c3b6588e2,90dd8b91f474e54d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:15:33.017 INFO 21062 --- [ocp-schedules-6,2dfeb83e7b8732f3,1092b2a9a4ef85a2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:15:33.220 INFO 21062 --- [ocp-schedules-30,7ad5b981e789235d,d897fddbf206b70a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:15:33.220+08:00, clusters=[1] 2024-11-04 15:15:33.221 INFO 21062 --- [ocp-schedules-30,7ad5b981e789235d,d897fddbf206b70a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:15:33.494 INFO 21062 --- [ocp-schedules-8,94232f041628608e,67e82cd6a269f616] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:15:33.494+08:00, clusters=[1] 2024-11-04 15:15:33.498 INFO 21062 --- [ocp-schedules-8,94232f041628608e,67e82cd6a269f616] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:15:33.513 WARN 21062 --- [ocp-async-2,7ed1a50222ee52b7,852262112deeec08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:33.518 INFO 21062 --- [ocp-async-9,6b503eb9785602f5,56be44707ad874c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:34.415 INFO 21062 --- [ocp-schedules-25,f433481aa3ea4a46,566bcda10549c9df] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:15:35.199 INFO 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:15:35.199 INFO 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:15:35.199 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:15:35.199 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:15:35.210 INFO 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:15:35.210 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:15:35.211 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:15:35.212 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:15:35.212 WARN 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:15:35.212 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:15:35.213 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:15:35.213 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:15:35.213 INFO 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:15:35.214 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:15:35.214 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:15:35.214 WARN 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:15:35.222 INFO 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:15:35.222 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:15:35.222 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:15:35.233 ERROR 21062 --- [ocp-schedules-27,8b17d07f1d010b99,95f2e404820a587d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:15:35.513 WARN 21062 --- [ocp-async-4,fcb36e3e1e0b91d6,44ca20037ef51db9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:35.518 INFO 21062 --- [ocp-async-3,b9573dba68e1a839,1dc41709260218bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:35.890 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:35.891 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:35.983 INFO 21062 --- [prometheus-minute-schedule-1,7780ffd66ce4ca45,185343e450d67675] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:37.514 WARN 21062 --- [ocp-async-6,65440f7ab49ce83f,e7166dcafd3576ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:37.518 INFO 21062 --- [ocp-async-5,fc4f940d678e7198,1a53ca3897e35b08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:38.504 INFO 21062 --- [ocp-schedules-24,c665663702cf2214,123d28db4e65c31f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:15:38.504 INFO 21062 --- [ocp-schedules-7,6550423a3fcb0b48,4216b5762ba254f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:15:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:15:38.588969+08:00, lastProcessTime=2024-11-04T15:15:28.589697+08:00 2024-11-04 15:15:38.751 INFO 21062 --- [ocp-async-4,cd5dca00bd27bef9,5fde4be7f1ba3363] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:15:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:15:38.588969+08:00 2024-11-04 15:15:38.891 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:38.892 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a38eaf71994487c7,9f6f25edd9fe897f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:39.271 INFO 21062 --- [ocp-schedules-24,6a684acde6454a1d,b58ff5345a6910aa] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:15:39.275 INFO 21062 --- [ocp-schedules-24,6a684acde6454a1d,b58ff5345a6910aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:39.276 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:15:39.514 WARN 21062 --- [ocp-async-9,5aa0c3448abd9c44,b94b1020c467e28a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:39.519 INFO 21062 --- [ocp-async-4,71ee264156da686d,49c51f6cda067124] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:39.678 INFO 21062 --- [ocp-schedules-12,68bfe08b67ca730b,ce852feba92ff240] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:15:39.678 INFO 21062 --- [ocp-schedules-12,fc7bc5cf4c6710cf,ea7276f99166fe1c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:15:39.682 INFO 21062 --- [ocp-schedules-12,fc7bc5cf4c6710cf,ea7276f99166fe1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:39.711 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:15:39.711 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:15:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:15:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:15:41.455 INFO 21062 --- [ocp-schedules-15,077e5fed077e8644,1428334921f82e6c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:15:41.514 WARN 21062 --- [ocp-async-3,c2fb016450d24a2e,e273cf7d145a2ff4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:41.519 INFO 21062 --- [ocp-async-6,3f9d1647bc0abaa7,0cdba9707413c4d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:41.892 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:41.892 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:41.983 INFO 21062 --- [prometheus-minute-schedule-1,7071c2763bd2fe0a,d32b04db316971ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:43.514 WARN 21062 --- [ocp-async-5,0e6de34cfe2164b8,8ce9779a95dcad70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:43.519 INFO 21062 --- [ocp-async-8,e632c563cc790f71,c1e9c0b172ee9519] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:43.581 INFO 21062 --- [ocp-schedules-28,ace9b89fccc516ed,f6cc955fba8e506b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:15:43.585 INFO 21062 --- [ocp-schedules-28,ace9b89fccc516ed,f6cc955fba8e506b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:43.672 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:15:43.674 ERROR 21062 --- [ocp-schedules-28,ace9b89fccc516ed,f6cc955fba8e506b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:15:43.674 INFO 21062 --- [ocp-schedules-28,ace9b89fccc516ed,f6cc955fba8e506b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:15:44.441 INFO 21062 --- [ocp-schedules-26,b71a0fa3d09258a5,7d9fd35fc7de4581] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:15:44.893 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:44.894 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:44.983 INFO 21062 --- [prometheus-minute-schedule-1,45b890bd4ac57200,fdbee5ffebffe9c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:45.514 WARN 21062 --- [ocp-async-9,8a9a4b289e5593c1,4e3a3297649798c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:45.519 INFO 21062 --- [ocp-async-7,880901cce4835a2b,280a3f7350f9aadb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:47.515 WARN 21062 --- [ocp-async-3,e1226683ad26c214,636576299092ba6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:47.519 INFO 21062 --- [ocp-async-10,22aea0e6352f6699,37030b5f9cdd1a54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:47.894 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:47.895 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e5eea9a0eda9f72a,9b88153ef0b4acb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:15:48.589254+08:00, lastProcessTime=2024-11-04T15:15:38.588969+08:00 2024-11-04 15:15:48.751 INFO 21062 --- [ocp-async-1,bd2eb10d71ee60dd,0ec9330090d525ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:15:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:15:48.589254+08:00 2024-11-04 15:15:49.276 INFO 21062 --- [ocp-schedules-24,6a684acde6454a1d,b58ff5345a6910aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:15:49.515 WARN 21062 --- [ocp-async-8,8c515bea148a4914,d89459c8cb24b9dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:49.519 INFO 21062 --- [ocp-async-1,fd43346f7d7ce422,55154e22238438c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:50.895 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:50.896 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d75d13c899de4ec6,594a28880344fe29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:51.459 INFO 21062 --- [ocp-schedules-28,b897be7336435c6a,208d310d2dc82f6a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:15:51.515 WARN 21062 --- [ocp-async-2,5d623641825a1336,f64dc92d6953d962] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:51.520 INFO 21062 --- [ocp-async-7,3f9913d43512340c,66dbed96ae24601f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:52.787 INFO 21062 --- [ocp-schedules-30,b3fd4802b5d57650,d592c325c71b019c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:15:52.792 INFO 21062 --- [ocp-schedules-30,b3fd4802b5d57650,d592c325c71b019c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:15:52.792 INFO 21062 --- [ocp-schedules-30,b3fd4802b5d57650,d592c325c71b019c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:15:53.515 WARN 21062 --- [ocp-async-4,c5b369cbf40cba96,fab20d61c3a928cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:53.520 INFO 21062 --- [ocp-async-10,54e79fcc82c35a87,dd58f24eaee59dce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:53.896 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:53.897 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d54ad73550483688,7ec9f5787044069b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:54.482 INFO 21062 --- [ocp-schedules-24,227be94a077c2675,0c792b8dd1b46120] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:15:55.515 WARN 21062 --- [ocp-async-8,808dfb1b60b6a621,222ceffd60c340b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:55.520 INFO 21062 --- [ocp-async-5,53a14b9286088f21,c14c25c90c5ed60d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:56.897 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:56.898 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:56.983 INFO 21062 --- [prometheus-minute-schedule-1,920cdfc8b56458ff,6b84fe25b8a51cd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:15:57.515 WARN 21062 --- [ocp-async-2,53a5a72e52d6fcbf,3f06c78634181745] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:57.520 INFO 21062 --- [ocp-async-9,feb1741863404b56,8d16450c323edb87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:58.500 INFO 21062 --- [ocp-schedules-13,d431c551b8a576d5,8de7c2e57227f2c0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:15:58.500 INFO 21062 --- [ocp-schedules-9,acc9c2fe5ccfa7a6,dc239a47b3ff5a0c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:15:58.502 INFO 21062 --- [ocp-schedules-28,d689e0cbeaf65532,ca2938cf4dd063ad] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:15:58.504 INFO 21062 --- [ocp-schedules-9,acc9c2fe5ccfa7a6,dc239a47b3ff5a0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:58.504 INFO 21062 --- [ocp-schedules-13,d431c551b8a576d5,8de7c2e57227f2c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:15:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:15:58.505 INFO 21062 --- [ocp-schedules-30,5821cfd7f1169df0,4ad899113148bd17] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:15:58.509 INFO 21062 --- [ocp-schedules-29,6b2fd28594ac02d7,eba08d0afc4a8d2d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:15:58.509 INFO 21062 --- [ocp-schedules-16,aa307b6046f9c7d8,9dd003cdc3d9947c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:15:58.509 INFO 21062 --- [ocp-schedules-17,e8812c2fe4e7e4b5,41c847ce99b80bff] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:15:58.509 INFO 21062 --- [ocp-schedules-17,e8812c2fe4e7e4b5,41c847ce99b80bff] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:15:58.509 INFO 21062 --- [ocp-schedules-29,6b2fd28594ac02d7,eba08d0afc4a8d2d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:15:58.509 INFO 21062 --- [ocp-schedules-16,aa307b6046f9c7d8,9dd003cdc3d9947c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:15:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:15:58.590348+08:00, lastProcessTime=2024-11-04T15:15:48.589254+08:00 2024-11-04 15:15:58.752 INFO 21062 --- [ocp-async-3,00ff603e3b562858,e74327a084eca7ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:15:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:15:58.590348+08:00 2024-11-04 15:15:59.463 INFO 21062 --- [ocp-schedules-19,7777284b75c4ae98,b0e422ee466dde9a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:15:59.516 WARN 21062 --- [ocp-async-3,6c82838e54b8c3aa,f6fd21326f309a87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:15:59.520 INFO 21062 --- [ocp-async-10,2c27b007189e815e,7a4b0d840e0bbb07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:15:59.898 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:15:59.899 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:15:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a377b6b502fdf2ef,fd49e147f6f10ca3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704560, interval=60 2024-11-04 15:16:01.462 INFO 21062 --- [ocp-schedules-1,0627e93c20811939,cb3dafbbfcac2561] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:16:01.516 WARN 21062 --- [ocp-async-6,d6d656039c55276a,c587f82fbf61df52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:01.520 INFO 21062 --- [ocp-async-5,2e18aa1769a4cc71,bed851a6d3b046c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:02.899 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:02.900 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c8f2b0bc8c0f1f1b,0f9531329dd504dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:03.516 WARN 21062 --- [ocp-async-2,1ded8d0cc87a6331,91e5f215b62d0543] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:03.521 INFO 21062 --- [ocp-async-9,66392036b0004a5d,9dc311e70d55aff9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:04.508 INFO 21062 --- [ocp-schedules-20,06839366f195c6de,db1f18af770b76a5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:16:05.516 WARN 21062 --- [ocp-async-3,860c08c1b00a0ec4,6939b95bfce2a6b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:05.521 INFO 21062 --- [ocp-async-4,5eda140b805d4a51,d96d0537bbe6904d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:05.900 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:05.901 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:05.983 INFO 21062 --- [prometheus-minute-schedule-1,6d6c1ebef6c428a7,c30b29f15c8326ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:07.516 WARN 21062 --- [ocp-async-6,cd6f294c76fa80bf,fab0f25bc59103e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:07.521 INFO 21062 --- [ocp-async-8,77ec115a82642396,3a39f19944293cc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:08.505 INFO 21062 --- [ocp-schedules-9,acc9c2fe5ccfa7a6,dc239a47b3ff5a0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:16:08.505 INFO 21062 --- [ocp-schedules-13,d431c551b8a576d5,8de7c2e57227f2c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:16:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:16:08.591620+08:00, lastProcessTime=2024-11-04T15:15:58.590348+08:00 2024-11-04 15:16:08.752 INFO 21062 --- [ocp-async-1,344a2aa91a4c3fdc,2549c86befb9563c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:16:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:16:08.591620+08:00 2024-11-04 15:16:08.901 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:08.902 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:08.983 INFO 21062 --- [prometheus-minute-schedule-1,93e75b377ba0e6f9,f36dcaa89844d30a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:09.517 WARN 21062 --- [ocp-async-1,ccabae520dcfac86,bb517acfaf525fdf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:09.521 INFO 21062 --- [ocp-async-9,2cb966dfed80e9fd,3065b2c81b9c1bab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:09.711 INFO 21062 --- [ocp-schedules-12,fc7bc5cf4c6710cf,ea7276f99166fe1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:16:09.712 INFO 21062 --- [ocp-schedules-12,8fe2fd1d4254db88,b56e3dc66f908002] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:16:09.715 INFO 21062 --- [ocp-schedules-12,8fe2fd1d4254db88,b56e3dc66f908002] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:09.743 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:16:09.743 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:16:10.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:16:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:16:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:16:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:16:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:16:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:16:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:16:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:16:11.465 INFO 21062 --- [ocp-schedules-9,da87e7450b935d9c,d17b6d21102c619b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:16:11.517 WARN 21062 --- [ocp-async-3,00d99b3f06eb364f,0b8d4afffa9b1ea1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:11.521 INFO 21062 --- [ocp-async-4,a0b418653412e1f9,7f8803968388da70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:11.789 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=1, affectRows=1 2024-11-04 15:16:11.790 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=1, validCount=1 2024-11-04 15:16:11.790 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=9, affectRows=9 2024-11-04 15:16:11.790 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=9, validCount=9 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=6, affectRows=6 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=20, affectRows=20 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=6, validCount=6 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=20, validCount=20 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=7, affectRows=7 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=7, validCount=7 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=27, affectRows=27 2024-11-04 15:16:11.791 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=27, validCount=27 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=45, affectRows=45 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=55, affectRows=55 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=62, affectRows=62 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=45, validCount=45 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=55, validCount=55 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=62, validCount=62 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=32, affectRows=32 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=66, affectRows=66 2024-11-04 15:16:11.792 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=31, validCount=31 2024-11-04 15:16:11.793 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=66, validCount=66 2024-11-04 15:16:11.793 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704500, nodeCount=92, affectRows=92 2024-11-04 15:16:11.794 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704500, nodeCount=90, validCount=90 2024-11-04 15:16:11.902 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:11.903 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4594f45a75806ae2,2f19fc45ad24c501] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:12.795 INFO 21062 --- [ocp-schedules-16,31059723c8de89b8,714325f41b4731e6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:16:12.802 INFO 21062 --- [ocp-schedules-16,31059723c8de89b8,714325f41b4731e6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:16:12.802 INFO 21062 --- [ocp-schedules-16,31059723c8de89b8,714325f41b4731e6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:16:13.517 WARN 21062 --- [ocp-async-10,5018594b7684a200,c61f7304ebd28909] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:13.522 INFO 21062 --- [ocp-async-8,ff1f6122a77fe267,b82a6ee3fe689355] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:13.984 INFO 21062 --- [prometheus-cache-monitor1,97fb72a71c2a81cc,452fdf55e27f4738] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:16:13.985 INFO 21062 --- [prometheus-cache-monitor1,97fb72a71c2a81cc,452fdf55e27f4738] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:16:14.589 INFO 21062 --- [ocp-schedules-11,2b0fce01103516df,481582a55280bdcc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:16:14.903 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:14.904 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:14.983 INFO 21062 --- [prometheus-minute-schedule-1,788484dff164e2e3,e21cb170e955425d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:15.517 WARN 21062 --- [ocp-async-5,a4a69274697ff697,1ee6d285b7db933b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:15.522 INFO 21062 --- [ocp-async-2,73cd089f85749d02,caf5564c7058e6e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:17.013 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:16:17.013 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:16:17.022 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:16:17.022 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:16:17.022 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:16:17.022 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:16:17.028 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:16:17.028 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:16:17.028 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:16:17.028 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:16:17.062 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:16:17.072 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:16:17.072 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:16:17.072 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:16:17.072 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:16:17.072 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:16:17.081 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:16:17.517 WARN 21062 --- [ocp-async-9,3b22867a5a88eb46,5224e7700747b150] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:17.522 INFO 21062 --- [ocp-async-7,d58cebe214bad801,7b4027d7b4af3aaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:17.613 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:16:17.904 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:17.905 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:17.983 INFO 21062 --- [prometheus-minute-schedule-1,713a961f00e74673,79083aa86084ff68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:18.168 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.140198738S 2024-11-04 15:16:18.168 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:16:18.264 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:16:18.264 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:16:18.289 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:16:18.344 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:16:18.344 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:16:18.344 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:16:18.344 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:16:18.344 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:16:18.344 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:16:18.355 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:16:18.483 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] o.s.boot.SpringApplication : Started application in 1.461 seconds (JVM running for 9992.163) 2024-11-04 15:16:18.484 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.461995164S 2024-11-04 15:16:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:16:18.590567+08:00, lastProcessTime=2024-11-04T15:16:08.591620+08:00 2024-11-04 15:16:18.752 INFO 21062 --- [ocp-async-6,787767502e55b0f8,39db91ac0f448849] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:16:18.777 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:16:18.590567+08:00 2024-11-04 15:16:18.933 INFO 21062 --- [ocp-schedules-11,c49edf544b613442,5f08c2be613b082a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:16:19.518 WARN 21062 --- [ocp-async-10,d42cec738689cbb3,0359886d2c58347a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:19.522 INFO 21062 --- [ocp-async-8,b9de5b15f7129f33,c050c6accf043d84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:20.905 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:20.906 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:20.983 INFO 21062 --- [prometheus-minute-schedule-1,12d4b953d751e401,e003abc657d3cd75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:21.468 INFO 21062 --- [ocp-schedules-14,251ce5aedcbe5f35,6c8be61e1eb019b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:16:21.518 WARN 21062 --- [ocp-async-5,39fd0071f9b4107d,2c31ca690e8b93b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:21.522 INFO 21062 --- [ocp-async-2,cba3456064e403ae,53addb91f05f520d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:23.518 WARN 21062 --- [ocp-async-2,ed5435df124f15f9,16c0895410063e2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:23.523 INFO 21062 --- [ocp-async-3,a2d2f34c7ea96a42,487a85e77bbf4345] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:23.907 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:23.907 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:23.983 INFO 21062 --- [prometheus-minute-schedule-1,51852421552c5c9e,6624bca89c06b32d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:24.613 INFO 21062 --- [ocp-schedules-10,2e7e731cb3e8302c,4eeeff3ce3479fb7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:16:25.518 WARN 21062 --- [ocp-async-3,a18d96900c76faa1,ed2250e36ad7b947] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:25.523 INFO 21062 --- [ocp-async-10,2fbc57ac28bed7e3,3a6f0d40194d7349] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:26.908 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:26.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ca88bb98d5d24d4c,f703d4db98a62ef3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:26.984 INFO 21062 --- [prometheus-hour-schedule-1,802dc2cb807072cf,b7ff8c7db6dcd35a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:16:27.518 WARN 21062 --- [ocp-async-10,b396aa116cf48f74,aac0f4c73a14eae6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:27.523 INFO 21062 --- [ocp-async-1,d7d52e41c64beabc,15ad961931d40cee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:28.498 INFO 21062 --- [ocp-schedules-25,c67859d78fd49438,8a8618192382d25a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:16:28.500 INFO 21062 --- [ocp-schedules-3,068d8c562899f7bf,3e055efa45eb8be1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:16:28.500 INFO 21062 --- [ocp-schedules-5,36ac1ddec8a121ad,0d2ff94175d0f9e2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:16:28.500 INFO 21062 --- [ocp-schedules-13,4e4166ae8168f75b,9865f6a8413f7728] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:16:28.502 INFO 21062 --- [ocp-schedules-30,931e6130840b9843,5f0ffbdfc6e68c7b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:16:28.504 INFO 21062 --- [ocp-schedules-1,896260a08ba0596d,001227653f8add8c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:16:28.505 INFO 21062 --- [ocp-schedules-13,4e4166ae8168f75b,9865f6a8413f7728] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:28.505 INFO 21062 --- [ocp-schedules-5,36ac1ddec8a121ad,0d2ff94175d0f9e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:28.505 INFO 21062 --- [ocp-schedules-25,c67859d78fd49438,8a8618192382d25a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:16:28.505 INFO 21062 --- [ocp-schedules-3,068d8c562899f7bf,3e055efa45eb8be1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:16:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:16:28.510 INFO 21062 --- [ocp-schedules-22,27531c108300b017,c572d1a819b4c777] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:16:28.510 INFO 21062 --- [ocp-schedules-18,999e42a5694324b8,c84d608e03fdf891] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:16:28.510 INFO 21062 --- [ocp-schedules-6,5ef7795a2c53e740,ce047f9471dc5419] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:16:28.510 INFO 21062 --- [ocp-schedules-22,27531c108300b017,c572d1a819b4c777] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:16:28.510 INFO 21062 --- [ocp-schedules-18,999e42a5694324b8,c84d608e03fdf891] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:16:28.510 INFO 21062 --- [ocp-schedules-6,5ef7795a2c53e740,ce047f9471dc5419] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:16:28.586 INFO 21062 --- [ocp-schedules-10,e5e1afdf845f22a9,b95b13398a777cb9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:16:28.586 INFO 21062 --- [ocp-schedules-10,e5e1afdf845f22a9,b95b13398a777cb9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:16:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:16:28.589692+08:00, lastProcessTime=2024-11-04T15:16:18.590567+08:00 2024-11-04 15:16:28.743 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:16:28.589692+08:00 2024-11-04 15:16:28.752 INFO 21062 --- [ocp-async-4,d6a42154ec915afa,c10424d709f19373] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:16:28.965 INFO 21062 --- [ocp-schedules-17,678d92d8290c176b,a60de9b892622232] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:16:29.338 INFO 21062 --- [ocp-schedules-24,8c84c6c6b2a0de0f,0bca2b7dba0a4eab] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:16:29.338 INFO 21062 --- [ocp-schedules-24,8c84c6c6b2a0de0f,0bca2b7dba0a4eab] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:16:29.466 INFO 21062 --- [ocp-schedules-11,9f3de67565269c96,34060e70706ba517] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:16:29.489 INFO 21062 --- [ocp-schedules-19,206e50c7ed925d1f,562ff5dff876c325] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:16:29.496 INFO 21062 --- [ocp-schedules-19,206e50c7ed925d1f,562ff5dff876c325] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:16:29.496 INFO 21062 --- [ocp-schedules-19,206e50c7ed925d1f,562ff5dff876c325] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:16:29.518 WARN 21062 --- [ocp-async-7,da6011b03bdf4afb,5c61c08b2d4f2ec6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:29.523 INFO 21062 --- [ocp-async-4,c5b067077b1a2f7d,e8f825f908c2c36a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:29.760 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:16:29.760 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:16:29.770 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:16:29.770 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:16:29.770 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:16:29.781 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:16:29.781 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:16:29.781 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:16:29.784 INFO 21062 --- [ocp-schedules-7,01f01c25b0cd82ae,e1ea4dd5e0f6df0d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:16:29.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:29.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:29.949 INFO 21062 --- [ocp-schedules-16,ee7feb7cd34c64a5,da96911c77c4ef5b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:16:29.949 INFO 21062 --- [ocp-schedules-16,ee7feb7cd34c64a5,da96911c77c4ef5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:16:29.954 INFO 21062 --- [ocp-schedules-16,ee7feb7cd34c64a5,da96911c77c4ef5b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:16:29.954 INFO 21062 --- [ocp-schedules-16,ee7feb7cd34c64a5,da96911c77c4ef5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:16:29.958 INFO 21062 --- [ocp-schedules-16,ee7feb7cd34c64a5,da96911c77c4ef5b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:16:29.958 INFO 21062 --- [ocp-schedules-16,ee7feb7cd34c64a5,da96911c77c4ef5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:16:29.983 INFO 21062 --- [prometheus-minute-schedule-1,0119db7b9afbd97f,8cc9ef230ccddc02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:16:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:16:31.471 INFO 21062 --- [ocp-schedules-11,042cad6c5cd4837f,06243f892d3f2ab9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:16:31.519 WARN 21062 --- [ocp-async-4,17fd997a201259c2,cceca7c7421a2ec2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:31.523 INFO 21062 --- [ocp-async-5,3a0c1ddbd619e0bd,0a1a972e3e84dfc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:32.804 INFO 21062 --- [ocp-schedules-21,300003d25dc785e9,008ba807f27c294c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:16:32.809 INFO 21062 --- [ocp-schedules-21,300003d25dc785e9,008ba807f27c294c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:16:32.809 INFO 21062 --- [ocp-schedules-21,300003d25dc785e9,008ba807f27c294c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:16:32.909 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:32.910 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d092e2f153be9ea9,8e6e41de659bd35e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:33.017 INFO 21062 --- [ocp-schedules-2,8825078e00fe7e59,f4e6b8cdb3214070] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:16:33.017 INFO 21062 --- [ocp-schedules-2,8825078e00fe7e59,f4e6b8cdb3214070] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:16:33.041 INFO 21062 --- [ocp-schedules-2,8825078e00fe7e59,f4e6b8cdb3214070] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:16:33.041 INFO 21062 --- [ocp-schedules-2,8825078e00fe7e59,f4e6b8cdb3214070] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:16:33.043 INFO 21062 --- [ocp-schedules-15,3e355c4f2325459d,c43689f0c64f939b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:16:33.246 INFO 21062 --- [ocp-schedules-23,846d91ab1a0f0144,1349cc38afe062b7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:16:33.246+08:00, clusters=[1] 2024-11-04 15:16:33.248 INFO 21062 --- [ocp-schedules-23,846d91ab1a0f0144,1349cc38afe062b7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:16:33.519 WARN 21062 --- [ocp-async-8,ab8e07c811c75318,44fa2a00a4a6d085] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:33.523 INFO 21062 --- [ocp-async-1,92a56bb27a01df04,63a85fc7cadc1846] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:33.525 INFO 21062 --- [ocp-schedules-17,d9d6d5aec4a9e5d5,c0219bc8a6a14034] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:16:33.525+08:00, clusters=[1] 2024-11-04 15:16:33.529 INFO 21062 --- [ocp-schedules-17,d9d6d5aec4a9e5d5,c0219bc8a6a14034] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:16:33.563 INFO 21062 --- [ocp-schedules-10,a02bf17aedf61ba8,53ee07ec17363534] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:16:33.566 INFO 21062 --- [ocp-schedules-10,a02bf17aedf61ba8,53ee07ec17363534] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:33.626 ERROR 21062 --- [ocp-schedules-10,a02bf17aedf61ba8,53ee07ec17363534] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:16:33.626 INFO 21062 --- [ocp-schedules-10,a02bf17aedf61ba8,53ee07ec17363534] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:16:34.643 INFO 21062 --- [ocp-schedules-14,0d7fbcd214d9a0c2,6ae4ce773080e87f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:16:35.243 INFO 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:16:35.243 INFO 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:16:35.243 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:16:35.243 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:16:35.253 INFO 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:16:35.253 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:16:35.253 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:16:35.255 WARN 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:16:35.255 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:16:35.255 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:16:35.255 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:16:35.255 INFO 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:16:35.255 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:16:35.256 WARN 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:16:35.256 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:16:35.256 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:16:35.263 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:16:35.263 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:16:35.263 INFO 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:16:35.266 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.24, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null) 2024-11-04 15:16:35.267 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - BY_APPLICATION 2024-11-04 15:16:35.269 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=http, hostAddress=10.46.128.24, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null) 2024-11-04 15:16:35.282 ERROR 21062 --- [ocp-schedules-22,91c88114e39153ec,21b7ef05926fb9b0] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:16:35.519 WARN 21062 --- [ocp-async-9,533afa606fa05db1,d4751294b71e6181] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:35.523 INFO 21062 --- [ocp-async-3,33864f9063d00224,65a79476f8b90a99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:35.910 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:35.911 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b610316e37c61d50,84870e59a14dd059] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:37.519 WARN 21062 --- [ocp-async-6,39254bfb6d4d4648,0898b06f6d8a1369] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:37.523 INFO 21062 --- [ocp-async-10,d1e25a3f228e41ac,fc26001e5621dc8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:38.506 INFO 21062 --- [ocp-schedules-5,36ac1ddec8a121ad,0d2ff94175d0f9e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:16:38.506 INFO 21062 --- [ocp-schedules-13,4e4166ae8168f75b,9865f6a8413f7728] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:16:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:16:38.589252+08:00, lastProcessTime=2024-11-04T15:16:28.589692+08:00 2024-11-04 15:16:38.747 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:16:38.589252+08:00 2024-11-04 15:16:38.752 INFO 21062 --- [ocp-async-9,fe5f90816ef89e19,80d46b5da3953e1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:16:38.911 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:38.912 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:38.983 INFO 21062 --- [prometheus-minute-schedule-1,faccf7b793c9200c,ed2db89c505b10a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:39.520 WARN 21062 --- [ocp-async-2,5f9200410bf50730,7c9867c59fab84d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:39.524 INFO 21062 --- [ocp-async-9,d9dd72e967d98647,8b805e13808741dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:39.744 INFO 21062 --- [ocp-schedules-12,8fe2fd1d4254db88,b56e3dc66f908002] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:16:39.744 INFO 21062 --- [ocp-schedules-12,175c75e7cb1533aa,53986864e32fa45f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:16:39.748 INFO 21062 --- [ocp-schedules-12,175c75e7cb1533aa,53986864e32fa45f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:39.776 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:16:39.776 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:16:40.002 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:16:40.002 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:16:41.475 INFO 21062 --- [ocp-schedules-25,eedd6af221b330fd,e4039d55d8ec33eb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:16:41.520 WARN 21062 --- [ocp-async-7,dbdb8591934411ba,efa9d1b92ae15372] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:41.524 INFO 21062 --- [ocp-async-6,27534f155b9e1cb9,d5e26eb697331b7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:41.822 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:16:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:16:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:16:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:16:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:16:41.851 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:16:41.851 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:16:41.912 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:41.913 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:41.983 INFO 21062 --- [prometheus-minute-schedule-1,92bbf0c263b775a0,99d2f2f22f102796] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:43.520 WARN 21062 --- [ocp-async-10,fb237e9035f017e0,36e95d5a1eb826a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:43.524 INFO 21062 --- [ocp-async-8,17fe39efb2254d01,fb7010aa227babdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:43.674 INFO 21062 --- [ocp-schedules-10,61f7e07380505449,5a725561b2af3b50] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:16:43.679 INFO 21062 --- [ocp-schedules-10,61f7e07380505449,5a725561b2af3b50] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:43.762 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:16:43.768 ERROR 21062 --- [ocp-schedules-10,61f7e07380505449,5a725561b2af3b50] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:16:43.768 INFO 21062 --- [ocp-schedules-10,61f7e07380505449,5a725561b2af3b50] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:16:44.668 INFO 21062 --- [ocp-schedules-25,0737b37f05d1af7f,3535b54484ad01df] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:16:44.913 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:44.913 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:44.983 INFO 21062 --- [prometheus-minute-schedule-1,fb0db10e50602c64,2fc38f16a7c298a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:45.520 WARN 21062 --- [ocp-async-2,b2ee00bf36a1db20,62cf52cab361d555] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:45.524 INFO 21062 --- [ocp-async-1,eedab547501d643b,b751dc9f808bfbdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:47.520 WARN 21062 --- [ocp-async-9,fd3ebcb5311b14c6,56d7781f022dc9f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:47.524 INFO 21062 --- [ocp-async-3,41ef4c77190a7637,cdb1845fc855158f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:47.914 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:47.915 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:47.983 INFO 21062 --- [prometheus-minute-schedule-1,0ec77ace12c54d69,63624959312f4f9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:16:48.590482+08:00, lastProcessTime=2024-11-04T15:16:38.589252+08:00 2024-11-04 15:16:48.752 INFO 21062 --- [ocp-async-4,b0452812f67564bf,c95d857720e399bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:16:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:16:48.590482+08:00 2024-11-04 15:16:49.276 INFO 21062 --- [ocp-schedules-30,7b90421901a5e3ae,b44f1700cc7733b8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:16:49.280 INFO 21062 --- [ocp-schedules-30,7b90421901a5e3ae,b44f1700cc7733b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:49.281 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:16:49.520 WARN 21062 --- [ocp-async-4,ce9d8deec3414cdd,0e62f4d9ffee1dc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:49.525 INFO 21062 --- [ocp-async-8,6b45a8e4a09892ea,3d86df85561a8c91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:50.915 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:50.917 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d7ebc14aeedf17fd,bf526bd63ea9ed48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:51.477 INFO 21062 --- [ocp-schedules-8,f39fcb5aed6535ca,fa1563acbcf3d8f3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:16:51.521 WARN 21062 --- [ocp-async-2,3dda845d379d4d43,9708407ca84cdfb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:51.525 INFO 21062 --- [ocp-async-1,e11a1901f2467e71,c6d137f5c622652c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:52.812 INFO 21062 --- [ocp-schedules-10,94ce355b5fec3d2f,52709de9fd87cff1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:16:52.818 INFO 21062 --- [ocp-schedules-10,94ce355b5fec3d2f,52709de9fd87cff1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:16:52.818 INFO 21062 --- [ocp-schedules-10,94ce355b5fec3d2f,52709de9fd87cff1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:16:53.521 WARN 21062 --- [ocp-async-7,b844fcb79cb20d53,f15600c400254066] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:53.525 INFO 21062 --- [ocp-async-3,2474fc47048e339a,b48585c9ffbad5b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:53.917 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:53.918 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:53.983 INFO 21062 --- [prometheus-minute-schedule-1,2bea8c6405b22f10,f0be262c035ea215] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:54.710 INFO 21062 --- [ocp-schedules-23,69ce20fa8cf7eb46,19a4f83e25fc1a33] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:16:55.521 WARN 21062 --- [ocp-async-4,f58a0b9b97c45954,cf80a1f9a62442d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:55.525 INFO 21062 --- [ocp-async-10,0b25267bec12c1af,a37609066bfd4e5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:56.918 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:56.919 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:56.983 INFO 21062 --- [prometheus-minute-schedule-1,a0905a1914f33449,7bbd1665e13960f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:16:57.521 WARN 21062 --- [ocp-async-8,6127d7b416bbbefc,b223487913928d28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:57.525 INFO 21062 --- [ocp-async-2,093b5a6bbc711c56,6d4f5c71a9a79fb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:58.500 INFO 21062 --- [ocp-schedules-18,9913880f4c5d9055,1d25a46ccbdb822c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:16:58.500 INFO 21062 --- [ocp-schedules-24,a936b7bf809db8fd,00c562b1ab0c3250] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:16:58.503 INFO 21062 --- [ocp-schedules-8,2add83e1fb25625a,e758c9054d68aa0f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:16:58.504 INFO 21062 --- [ocp-schedules-18,9913880f4c5d9055,1d25a46ccbdb822c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:58.504 INFO 21062 --- [ocp-schedules-24,a936b7bf809db8fd,00c562b1ab0c3250] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:16:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:16:58.506 INFO 21062 --- [ocp-schedules-13,ef0d6950008b7ffe,7e35807175b8ee47] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:16:58.510 INFO 21062 --- [ocp-schedules-1,2ee3e90f98ca9174,0ce0ffba387f207d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:16:58.510 INFO 21062 --- [ocp-schedules-3,18e30641970b786a,f707d5ab1920c27a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:16:58.510 INFO 21062 --- [ocp-schedules-1,2ee3e90f98ca9174,0ce0ffba387f207d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:16:58.510 INFO 21062 --- [ocp-schedules-3,18e30641970b786a,f707d5ab1920c27a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:16:58.511 INFO 21062 --- [ocp-schedules-27,9bcff5e2cad65342,ae42d61471f2166b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:16:58.511 INFO 21062 --- [ocp-schedules-27,9bcff5e2cad65342,ae42d61471f2166b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:16:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:16:58.590521+08:00, lastProcessTime=2024-11-04T15:16:48.590482+08:00 2024-11-04 15:16:58.753 INFO 21062 --- [ocp-async-9,996dd3f752b4015a,160a6d8e060fa472] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:16:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:16:58.590521+08:00 2024-11-04 15:16:59.281 INFO 21062 --- [ocp-schedules-30,7b90421901a5e3ae,b44f1700cc7733b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:16:59.469 INFO 21062 --- [ocp-schedules-15,6ede890cb17c654c,212a3ba78c8cfb23] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:16:59.521 WARN 21062 --- [ocp-async-7,1cac51ad5a35ead1,53d1df369762d019] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:16:59.525 INFO 21062 --- [ocp-async-3,f813859b1b13fb08,cf19a9c87b15e0fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:16:59.920 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:16:59.920 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:16:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2132100cb75b0bb9,b06c4db78990c253] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704620, interval=60 2024-11-04 15:17:01.480 INFO 21062 --- [ocp-schedules-22,bc3973765ff6b789,f3c7833698708334] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:17:01.522 WARN 21062 --- [ocp-async-6,1486ba7cacfdc9b1,97b872dfd002b4f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:01.526 INFO 21062 --- [ocp-async-10,dd16ef1c2758635f,96f47a8538eaef7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:02.921 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:02.922 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:02.983 INFO 21062 --- [prometheus-minute-schedule-1,26d243414098335d,7837369a2668f13d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:03.522 WARN 21062 --- [ocp-async-5,f5219627f338d131,8b8361335b01c354] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:03.526 INFO 21062 --- [ocp-async-2,f17985f811018d8c,0844233c0bb3de9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:04.735 INFO 21062 --- [ocp-schedules-10,936c2fed3c550fb2,b2a02a6e9a6672b4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:17:05.522 WARN 21062 --- [ocp-async-7,9d277a0ec7f13e35,356f9374c5fd86f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:05.526 INFO 21062 --- [ocp-async-9,9dce207553166777,0703816471aa036f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:05.922 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:05.923 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:05.983 INFO 21062 --- [prometheus-minute-schedule-1,dbc83799e3a487ef,bf378e3d454ec60c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:07.522 WARN 21062 --- [ocp-async-6,9d11310cbf649311,f81b63556412a682] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:07.526 INFO 21062 --- [ocp-async-4,03870ff62482fcf5,cb79482b3c7b2c57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:08.505 INFO 21062 --- [ocp-schedules-24,a936b7bf809db8fd,00c562b1ab0c3250] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:17:08.505 INFO 21062 --- [ocp-schedules-18,9913880f4c5d9055,1d25a46ccbdb822c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:17:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:17:08.591188+08:00, lastProcessTime=2024-11-04T15:16:58.590521+08:00 2024-11-04 15:17:08.753 INFO 21062 --- [ocp-async-8,20dbea4837587566,d15536651a76806c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:17:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:17:08.591188+08:00 2024-11-04 15:17:08.923 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:08.924 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:08.983 INFO 21062 --- [prometheus-minute-schedule-1,10672119b602bf85,52c5f26c0ab2084b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:09.522 WARN 21062 --- [ocp-async-8,eddc292799205c63,0862eb85bb4352aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:09.526 INFO 21062 --- [ocp-async-2,d3c716e91f618740,9268571922876101] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:09.777 INFO 21062 --- [ocp-schedules-12,175c75e7cb1533aa,53986864e32fa45f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:17:09.777 INFO 21062 --- [ocp-schedules-12,a1f2ef80ee564ed9,47dbf955b1ed202e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:17:09.780 INFO 21062 --- [ocp-schedules-12,a1f2ef80ee564ed9,47dbf955b1ed202e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:09.810 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:17:09.810 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:17:10.002 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:17:10.004 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:17:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:17:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:17:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:17:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:17:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:17:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:17:11.484 INFO 21062 --- [ocp-schedules-29,a6cb0f6ba7b99e9e,5316393870568ea2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:17:11.522 WARN 21062 --- [ocp-async-1,2c78691260c80812,a767fa4b01a70ea1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:11.526 INFO 21062 --- [ocp-async-9,e76e10956a2110ba,43583c0509a7f7f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:11.796 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=4, affectRows=4 2024-11-04 15:17:11.796 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=4, validCount=4 2024-11-04 15:17:11.796 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=26, affectRows=26 2024-11-04 15:17:11.796 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=17, affectRows=17 2024-11-04 15:17:11.797 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=17, validCount=17 2024-11-04 15:17:11.797 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=25, validCount=25 2024-11-04 15:17:11.797 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=13, affectRows=13 2024-11-04 15:17:11.797 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=13, validCount=13 2024-11-04 15:17:11.797 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=21, affectRows=21 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=21, validCount=21 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=89, affectRows=89 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=42, affectRows=42 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=25, affectRows=25 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=67, affectRows=67 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=58, affectRows=58 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704560, nodeCount=60, affectRows=60 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=25, validCount=25 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=42, validCount=42 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=87, validCount=87 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=67, validCount=67 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=58, validCount=58 2024-11-04 15:17:11.798 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704560, nodeCount=60, validCount=60 2024-11-04 15:17:11.924 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:11.925 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4bcc8cde83b292c1,57e3399a4e8267f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:12.822 INFO 21062 --- [ocp-schedules-30,31d4719cced23d98,d9c0e6d319eba79c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:17:12.828 INFO 21062 --- [ocp-schedules-30,31d4719cced23d98,d9c0e6d319eba79c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:17:12.828 INFO 21062 --- [ocp-schedules-30,31d4719cced23d98,d9c0e6d319eba79c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:17:13.523 WARN 21062 --- [ocp-async-3,a55be8a19aec1458,f8055080ed748016] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:13.527 INFO 21062 --- [ocp-async-4,53713352ecc15b04,fba0898a1c11e9e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:13.984 INFO 21062 --- [prometheus-cache-monitor1,bd879a3101352fa7,56816b5c5cc2226f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:17:13.984 INFO 21062 --- [prometheus-cache-monitor1,bd879a3101352fa7,56816b5c5cc2226f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:17:14.774 INFO 21062 --- [ocp-schedules-14,9e4fbdfda12869ef,8dd9c84690e6cdb9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:17:14.925 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:14.926 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:14.983 INFO 21062 --- [prometheus-minute-schedule-1,35f0eb364683be21,4b8c6d8b6925864c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:15.523 WARN 21062 --- [ocp-async-8,884dfeba7b61d80a,858c395d60601ea8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:15.527 INFO 21062 --- [ocp-async-5,4d60f82422417334,d6f7d76c38b85880] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:15.546 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:13:15.524, end=2024-11-04T15:15:15.543} 2024-11-04 15:17:15.555 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:17:15.555 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:17:17.523 WARN 21062 --- [ocp-async-2,1a73cb1431620455,c915b005f4338429] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:17.527 INFO 21062 --- [ocp-async-7,9f185bc0ddf54bf0,23ba581e1befb629] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:17.926 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:17.927 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:17.983 INFO 21062 --- [prometheus-minute-schedule-1,c0bd5dc681aa6989,a669a4396c52596f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:17:18.589649+08:00, lastProcessTime=2024-11-04T15:17:08.591188+08:00 2024-11-04 15:17:18.753 INFO 21062 --- [ocp-async-6,42f87ba973ae8c97,67463dc54ca70776] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:17:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:17:18.589649+08:00 2024-11-04 15:17:19.523 WARN 21062 --- [ocp-async-3,7aa9065475ec72d1,afd82e77e2e8117b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:19.527 INFO 21062 --- [ocp-async-4,5c45a9927f903e94,9efa61629fc0144e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:20.927 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:20.928 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:20.983 INFO 21062 --- [prometheus-minute-schedule-1,332da814c99034cf,d8e0795e300f349d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:21.487 INFO 21062 --- [ocp-schedules-17,6349751a74f55765,cece844c17c77f40] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:17:21.523 WARN 21062 --- [ocp-async-4,7e8fbb9391e309dc,088eefb3dbd434d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:21.527 INFO 21062 --- [ocp-async-5,a3363851eb996fad,4a231e1088d3ca5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:23.523 WARN 21062 --- [ocp-async-5,16dc7517eb924763,d3faa9d213767a7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:23.528 INFO 21062 --- [ocp-async-1,a98d3f799feee16e,e34a6a436ba8f176] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:23.928 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:23.929 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6110bfeb7b45ee03,952f5285f78cd3a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:24.800 INFO 21062 --- [ocp-schedules-24,023c533fc50351ab,52a1c11dcdce621c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:17:25.524 WARN 21062 --- [ocp-async-3,cae9c4461a3c8676,3d17b8b42f0b65c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:25.528 INFO 21062 --- [ocp-async-6,049d5bec81714cf4,6c99cc360345ea83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:26.929 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:26.930 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:26.983 INFO 21062 --- [prometheus-minute-schedule-1,93c7ff2c81021a73,15d705d175f97670] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:26.984 INFO 21062 --- [prometheus-hour-schedule-1,6ac89e26fc92f05d,97f4cbbd9399bbc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:17:27.524 WARN 21062 --- [ocp-async-4,0b3ce50eb86af91e,dd560720173333cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:27.528 INFO 21062 --- [ocp-async-10,2974f247aa91e5c0,ab4920e062d8af37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:28.498 INFO 21062 --- [ocp-schedules-23,91637364e73a3ae8,a4100292ff1d83b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:17:28.499 INFO 21062 --- [ocp-schedules-15,df44994403b308ce,6c3722893028c063] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:17:28.500 INFO 21062 --- [ocp-schedules-4,a6a6eadec4c585f8,8d0907c31133df35] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:17:28.500 INFO 21062 --- [ocp-schedules-29,366f9e128733613c,a6c744e5ad31aad3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:17:28.502 INFO 21062 --- [ocp-schedules-23,91637364e73a3ae8,a4100292ff1d83b6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:17:28.502 INFO 21062 --- [ocp-schedules-20,12e889dcc5f79b2a,65b59c45be9d0e52] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:17:28.504 INFO 21062 --- [ocp-schedules-4,a6a6eadec4c585f8,8d0907c31133df35] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:28.504 INFO 21062 --- [ocp-schedules-29,366f9e128733613c,a6c744e5ad31aad3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:17:28.504 INFO 21062 --- [ocp-schedules-14,f642c07dd0022815,7fe8fa59e0bd0021] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:17:28.504 INFO 21062 --- [ocp-schedules-1,de7fb5e059b85aa9,05dd295c9cd1d223] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:17:28.505 INFO 21062 --- [ocp-schedules-15,df44994403b308ce,6c3722893028c063] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:17:28.507 INFO 21062 --- [ocp-schedules-1,de7fb5e059b85aa9,05dd295c9cd1d223] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:28.509 INFO 21062 --- [ocp-schedules-5,e4d1c0c3f120fc93,e4df4736bbf9bcad] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:17:28.509 INFO 21062 --- [ocp-schedules-5,e4d1c0c3f120fc93,e4df4736bbf9bcad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:17:28.509 INFO 21062 --- [ocp-schedules-28,ed874dcf14997b78,dabeacbdd64f38a3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:17:28.509 INFO 21062 --- [ocp-schedules-28,ed874dcf14997b78,dabeacbdd64f38a3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:17:28.509 INFO 21062 --- [ocp-schedules-2,415e91e5d2a1f72e,8fdab1d32e0af857] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:17:28.509 INFO 21062 --- [ocp-schedules-2,415e91e5d2a1f72e,8fdab1d32e0af857] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:17:28.541 INFO 21062 --- [ocp-schedules-25,1bb5202f834e4e17,713a30115d664e24] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:17:28.541 INFO 21062 --- [ocp-schedules-25,1bb5202f834e4e17,713a30115d664e24] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:17:28.580 INFO 21062 --- [pool-12-thread-8,887f7e5d6f026a4f,f1193b28579a8eb5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:17:28.580 INFO 21062 --- [pool-12-thread-6,887f7e5d6f026a4f,20817359d89a58af] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:17:28.581 INFO 21062 --- [pool-12-thread-7,887f7e5d6f026a4f,36dca2cb18b1ee88] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:17:28.581 INFO 21062 --- [pool-12-thread-5,887f7e5d6f026a4f,60ec9a9acb0c4253] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:17:28.581 INFO 21062 --- [pool-12-thread-7,887f7e5d6f026a4f,36dca2cb18b1ee88] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:17:28.582 INFO 21062 --- [pool-12-thread-7,887f7e5d6f026a4f,36dca2cb18b1ee88] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:17:28.584 INFO 21062 --- [pool-12-thread-7,887f7e5d6f026a4f,36dca2cb18b1ee88] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:17:28.585 INFO 21062 --- [pool-12-thread-6,887f7e5d6f026a4f,20817359d89a58af] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:17:28.585 INFO 21062 --- [pool-12-thread-6,887f7e5d6f026a4f,20817359d89a58af] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:17:28.586 INFO 21062 --- [ForkJoinPool-11-worker-2,887f7e5d6f026a4f,ee3d8f7828957211] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:17:28.586 INFO 21062 --- [pool-12-thread-8,887f7e5d6f026a4f,f1193b28579a8eb5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:17:28.586 INFO 21062 --- [ForkJoinPool-11-worker-11,887f7e5d6f026a4f,fe8e30b6d8db3ee8] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:17:28.586 INFO 21062 --- [ForkJoinPool-11-worker-11,887f7e5d6f026a4f,fe8e30b6d8db3ee8] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:17:28.586 INFO 21062 --- [ForkJoinPool-11-worker-11,887f7e5d6f026a4f,fe8e30b6d8db3ee8] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:17:28.586 INFO 21062 --- [pool-12-thread-5,887f7e5d6f026a4f,60ec9a9acb0c4253] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:17:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:17:28.588636+08:00, lastProcessTime=2024-11-04T15:17:18.589649+08:00 2024-11-04 15:17:28.753 INFO 21062 --- [ocp-async-1,28d95a3882cf2292,d918b39612ec76ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:17:28.799 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:17:28.588636+08:00 2024-11-04 15:17:28.967 INFO 21062 --- [ocp-schedules-5,47dfcdebe9f089a9,252941b5c368ee69] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:17:29.343 INFO 21062 --- [ocp-schedules-10,b16895639504bf4c,6a37b064d0ae3b14] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:17:29.343 INFO 21062 --- [ocp-schedules-10,b16895639504bf4c,6a37b064d0ae3b14] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:17:29.414 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:17:29.431 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:17:29.433 WARN 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:17:29.434 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:17:29.435 WARN 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:17:29.442 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:17:29.443 WARN 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:17:29.443 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:17:29.444 WARN 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:17:29.450 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:17:29.452 WARN 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:17:29.452 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:17:29.453 WARN 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:17:29.459 INFO 21062 --- [ocp-schedules-16,82ba62bf8b2d88f2,44a5ce6febb6aecb] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:17:29.471 INFO 21062 --- [ocp-schedules-11,4d3102cca4197b42,114cab78abf69132] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:17:29.496 INFO 21062 --- [ocp-schedules-21,f5b90010f6facd03,c57d1c9121b7edf9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:17:29.501 INFO 21062 --- [ocp-schedules-21,f5b90010f6facd03,c57d1c9121b7edf9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:17:29.501 INFO 21062 --- [ocp-schedules-21,f5b90010f6facd03,c57d1c9121b7edf9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:17:29.524 WARN 21062 --- [ocp-async-7,a3c2ab6943348f9b,1341270b5018e852] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:29.528 INFO 21062 --- [ocp-async-1,1a3a2c40b40f25a3,c659a43caeb6aff5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:29.930 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:29.931 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:29.962 INFO 21062 --- [ocp-schedules-15,b7347bbb4c045005,ad4fcde889395f36] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:17:29.962 INFO 21062 --- [ocp-schedules-15,b7347bbb4c045005,ad4fcde889395f36] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:17:29.963 INFO 21062 --- [ocp-schedules-15,b7347bbb4c045005,ad4fcde889395f36] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:17:29.963 INFO 21062 --- [ocp-schedules-15,b7347bbb4c045005,ad4fcde889395f36] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:17:29.963 INFO 21062 --- [ocp-schedules-15,b7347bbb4c045005,ad4fcde889395f36] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:17:29.963 INFO 21062 --- [ocp-schedules-15,b7347bbb4c045005,ad4fcde889395f36] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:17:29.983 INFO 21062 --- [prometheus-minute-schedule-1,9e900a18466a3652,1be299a7225a1044] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:17:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:17:30.371 INFO 21062 --- [ocp-schedules-19,b3847011db59a634,652ac00aab877eb1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:17:30.371 INFO 21062 --- [ocp-schedules-19,b3847011db59a634,652ac00aab877eb1] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:17:30.400 INFO 21062 --- [pool-52-thread-1,b3847011db59a634,0fd77a0188c42fef] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:17:30.430 WARN 21062 --- [ocp-schedules-19,b3847011db59a634,652ac00aab877eb1] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:17:30.430 INFO 21062 --- [ocp-schedules-19,b3847011db59a634,652ac00aab877eb1] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:17:31.490 INFO 21062 --- [ocp-schedules-3,410ae75420d76f64,1fb9d81c6cc7fff5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:17:31.524 WARN 21062 --- [ocp-async-4,3c3057e052e92d77,1e2af60da8e4e52d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:31.528 INFO 21062 --- [ocp-async-6,4d3611db03b6f36e,3397657052958019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:32.831 INFO 21062 --- [ocp-schedules-8,f0dca1183700ce59,038ba1053ede6e25] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:17:32.836 INFO 21062 --- [ocp-schedules-8,f0dca1183700ce59,038ba1053ede6e25] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:17:32.836 INFO 21062 --- [ocp-schedules-8,f0dca1183700ce59,038ba1053ede6e25] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:17:32.931 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:32.933 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:32.983 INFO 21062 --- [prometheus-minute-schedule-1,102ad1898a9a22d0,e939a080f42a710b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:33.041 INFO 21062 --- [ocp-schedules-6,f16ca6bde6bcb1e9,9b1565dc2f6e6567] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:17:33.041 INFO 21062 --- [ocp-schedules-6,f16ca6bde6bcb1e9,9b1565dc2f6e6567] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:17:33.069 INFO 21062 --- [ocp-schedules-6,f16ca6bde6bcb1e9,9b1565dc2f6e6567] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:17:33.069 INFO 21062 --- [ocp-schedules-6,f16ca6bde6bcb1e9,9b1565dc2f6e6567] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:17:33.070 INFO 21062 --- [ocp-schedules-27,2a2dcccc9e94f6ca,db6dd4e8f38d7061] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:17:33.272 INFO 21062 --- [ocp-schedules-9,08b0f6b767e9149b,295a595969456389] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:17:33.272+08:00, clusters=[1] 2024-11-04 15:17:33.273 INFO 21062 --- [ocp-schedules-9,08b0f6b767e9149b,295a595969456389] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:17:33.524 WARN 21062 --- [ocp-async-8,bf86504b3e58e0e2,89e9a11ca27d72e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:33.528 INFO 21062 --- [ocp-async-10,3f3097be1c2c2edc,6e559574db261f02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:33.555 INFO 21062 --- [ocp-schedules-3,1b1b654587c6c4e8,cf976f0a430e97f1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:17:33.555+08:00, clusters=[1] 2024-11-04 15:17:33.559 INFO 21062 --- [ocp-schedules-3,1b1b654587c6c4e8,cf976f0a430e97f1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:17:34.842 INFO 21062 --- [ocp-schedules-16,4426b118f5560d06,a675ef07306bb8ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:17:35.292 INFO 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:17:35.292 INFO 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:17:35.292 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:17:35.292 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:17:35.302 INFO 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:17:35.302 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:17:35.302 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:17:35.304 WARN 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:17:35.304 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:17:35.304 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:17:35.304 INFO 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:17:35.304 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:17:35.304 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:17:35.305 WARN 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:17:35.306 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:17:35.306 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:17:35.312 INFO 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:17:35.312 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:17:35.312 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:17:35.325 ERROR 21062 --- [ocp-schedules-25,ef9886d074422112,c6568bd7d7ce1377] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:17:35.524 WARN 21062 --- [ocp-async-7,47e8b463c8c336f4,25d14802a1a4975d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:35.529 INFO 21062 --- [ocp-async-3,4ca5744eeec64c17,ebe053f18917e398] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:35.933 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:35.934 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4071017cd4529ba9,eeca18e4c8ae7777] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:37.524 WARN 21062 --- [ocp-async-1,1775bf5d5726a64e,8b43bee12e9d5a7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:37.529 INFO 21062 --- [ocp-async-9,bf8302e19249f190,667c328b53d4298c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:38.505 INFO 21062 --- [ocp-schedules-4,a6a6eadec4c585f8,8d0907c31133df35] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:17:38.505 INFO 21062 --- [ocp-schedules-29,366f9e128733613c,a6c744e5ad31aad3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:17:38.508 INFO 21062 --- [ocp-schedules-1,de7fb5e059b85aa9,05dd295c9cd1d223] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:17:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:17:38.589905+08:00, lastProcessTime=2024-11-04T15:17:28.588636+08:00 2024-11-04 15:17:38.753 INFO 21062 --- [ocp-async-10,9e8ffd136a8d0329,7365301e443ed8d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:17:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:17:38.589905+08:00 2024-11-04 15:17:38.934 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:38.935 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:38.983 INFO 21062 --- [prometheus-minute-schedule-1,cd475e18c6d0b479,e5c0ac060dc20c52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:39.525 WARN 21062 --- [ocp-async-10,0b9b1713cc9167c3,1eb44f2e72112dcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:39.529 INFO 21062 --- [ocp-async-7,d64a49d8c719db5c,c3a5d2122e3e27bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:39.810 INFO 21062 --- [ocp-schedules-12,a1f2ef80ee564ed9,47dbf955b1ed202e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:17:39.810 INFO 21062 --- [ocp-schedules-12,5b8543805214cda2,aa3fe1b31dc68f92] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:17:39.814 INFO 21062 --- [ocp-schedules-12,5b8543805214cda2,aa3fe1b31dc68f92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:39.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:17:39.845 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:17:40.004 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:17:40.004 INFO 21062 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:17:41.492 INFO 21062 --- [ocp-schedules-14,b3440caf35a7311a,b80a26239629be64] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:17:41.525 WARN 21062 --- [ocp-async-2,1bb56858b882cb40,d499e96b7ad6ab3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:41.529 INFO 21062 --- [ocp-async-1,c72462a4253fd2c2,ecd5e4e8ac101447] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:41.935 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:41.935 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:41.983 INFO 21062 --- [prometheus-minute-schedule-1,98b533647bab98e7,042cbdb79d43e8d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:43.525 WARN 21062 --- [ocp-async-4,a6932f4b0544cac8,7c8b246a1b283882] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:43.529 INFO 21062 --- [ocp-async-6,d3272080ab5ff2d8,ec10a659f5bc3c24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:43.768 INFO 21062 --- [ocp-schedules-10,e46b89aff76d061f,2e2acce68656037b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:17:43.772 INFO 21062 --- [ocp-schedules-10,e46b89aff76d061f,2e2acce68656037b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:43.857 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:17:43.860 ERROR 21062 --- [ocp-schedules-10,e46b89aff76d061f,2e2acce68656037b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:17:43.860 INFO 21062 --- [ocp-schedules-10,e46b89aff76d061f,2e2acce68656037b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:17:44.870 INFO 21062 --- [ocp-schedules-30,92ac324b7bee416c,fdde2b6810612c30] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:17:44.936 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:44.936 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e6d3fe546d58db11,1e8ed6dd98d5bbe0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:45.525 WARN 21062 --- [ocp-async-8,7d51d9da74593559,cb87979fb6e3f109] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:45.530 INFO 21062 --- [ocp-async-5,ab7e732ddd3f6dec,3297c0fcc0f67dc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:47.525 WARN 21062 --- [ocp-async-2,7a5bd7720bf3810b,4189fc4a3c337950] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:47.530 INFO 21062 --- [ocp-async-3,bbd4ec12b6f0a025,8ad12c618d874dc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:47.936 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:47.937 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:47.983 INFO 21062 --- [prometheus-minute-schedule-1,a1a8e521b0cf02a0,892e28ad99c35e4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:17:48.593026+08:00, lastProcessTime=2024-11-04T15:17:38.589905+08:00 2024-11-04 15:17:48.753 INFO 21062 --- [ocp-async-9,85c266d55bd3d09e,c5f33110bed979c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:17:48.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:17:48.593026+08:00 2024-11-04 15:17:49.526 WARN 21062 --- [ocp-async-4,b2fa7b0d134b4659,cdcd8a99347e1df9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:49.530 INFO 21062 --- [ocp-async-6,0b2899ecfabe210a,1f5674443665aa99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:50.938 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:50.939 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:50.983 INFO 21062 --- [prometheus-minute-schedule-1,1307ce13548c016f,9f24e1b00b61184b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:51.495 INFO 21062 --- [ocp-schedules-2,bdf305db17da0bc7,8a83155484286e2b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:17:51.526 WARN 21062 --- [ocp-async-8,b0627df0e5369ec5,1e3895616f3271b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:51.530 INFO 21062 --- [ocp-async-5,b573892758572bc4,9e1de9490ef9c7f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:52.839 INFO 21062 --- [ocp-schedules-26,730f311a1b1e06bd,f5a8500480950bbc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:17:52.844 INFO 21062 --- [ocp-schedules-26,730f311a1b1e06bd,f5a8500480950bbc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:17:52.844 INFO 21062 --- [ocp-schedules-26,730f311a1b1e06bd,f5a8500480950bbc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:17:53.526 WARN 21062 --- [ocp-async-7,d67c0e3be713f12e,50f8198b95574c51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:53.530 INFO 21062 --- [ocp-async-3,b4edfadfbe257baa,86533c575624d719] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:53.939 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:53.940 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:53.983 INFO 21062 --- [prometheus-minute-schedule-1,df08138453c2e2d3,63a6693996545730] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:54.914 INFO 21062 --- [ocp-schedules-29,c0e0de52388619dc,074690d217e652a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:17:55.526 WARN 21062 --- [ocp-async-1,ed779a3a0bd7c4b4,b50f7b1deb0b1f30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:55.531 INFO 21062 --- [ocp-async-9,b53e2ab77b624ca7,ded980a05c587319] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:56.940 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:56.941 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:56.983 INFO 21062 --- [prometheus-minute-schedule-1,18a5b4da8432b160,7d12d5ed776b69a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:17:57.526 WARN 21062 --- [ocp-async-8,99b9fc6285254882,696c4039d132f7c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:57.531 INFO 21062 --- [ocp-async-10,beac616068888c46,65d4ddae8d10cff7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:58.500 INFO 21062 --- [ocp-schedules-9,d23365c2bb5721ce,e44815d5a4c97597] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:17:58.500 INFO 21062 --- [ocp-schedules-4,1671e259f17dd730,a671dc906dd6b678] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:17:58.502 INFO 21062 --- [ocp-schedules-8,8bcb2cf3b00ccd92,2a35c1b271aba751] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:17:58.503 INFO 21062 --- [ocp-schedules-9,d23365c2bb5721ce,e44815d5a4c97597] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:17:58.504 INFO 21062 --- [ocp-schedules-4,1671e259f17dd730,a671dc906dd6b678] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:58.505 INFO 21062 --- [ocp-schedules-18,a3bba8514c7265e8,c7c1bdfa80bfa210] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:17:58.509 INFO 21062 --- [ocp-schedules-15,ab6fafb8ccb9d291,e51b1565cde5e34c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:17:58.509 INFO 21062 --- [ocp-schedules-15,ab6fafb8ccb9d291,e51b1565cde5e34c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:17:58.509 INFO 21062 --- [ocp-schedules-30,61204d662f646006,adadde940d63a772] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:17:58.509 INFO 21062 --- [ocp-schedules-30,61204d662f646006,adadde940d63a772] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:17:58.509 INFO 21062 --- [ocp-schedules-26,c0952cf238835880,7a656eefacbeb26c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:17:58.509 INFO 21062 --- [ocp-schedules-26,c0952cf238835880,7a656eefacbeb26c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:17:58.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:17:58.591518+08:00, lastProcessTime=2024-11-04T15:17:48.593026+08:00 2024-11-04 15:17:58.754 INFO 21062 --- [ocp-async-2,3cd90526bd2a103b,9ae5a89b87955ca8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:17:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:17:58.591518+08:00 2024-11-04 15:17:59.281 INFO 21062 --- [ocp-schedules-29,2e1bc82dc0a76ccd,8160040a00dd747e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:17:59.285 INFO 21062 --- [ocp-schedules-29,2e1bc82dc0a76ccd,8160040a00dd747e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:17:59.286 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:17:59.475 INFO 21062 --- [ocp-schedules-28,aba9c26d558b172c,facbac5863033cb3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:17:59.527 WARN 21062 --- [ocp-async-7,f0763a216f0fd3a5,a21d6bd00fd28691] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:17:59.531 INFO 21062 --- [ocp-async-3,3aa9f33ca4d7738b,a56e93a304aa8b41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:17:59.941 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:17:59.942 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:17:59.983 INFO 21062 --- [prometheus-minute-schedule-1,9ad65e7606325edd,d89a4dd54de67be6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704680, interval=60 2024-11-04 15:18:01.498 INFO 21062 --- [ocp-schedules-28,b1a67c85f91955c0,42138c873668af55] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:18:01.527 WARN 21062 --- [ocp-async-1,b7c42086254f8642,6856d0c7baa95bbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:01.531 INFO 21062 --- [ocp-async-9,83374b27e2e888e4,7a052d3ac0e4bebf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:02.942 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:02.943 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2b4e31fd33c4e926,e38d2feaa0359e1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:03.527 WARN 21062 --- [ocp-async-6,0e04548d95190c95,d26a88e7b99614b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:03.531 INFO 21062 --- [ocp-async-10,921a76293e87b7c7,da7778ae0407c8ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:04.938 INFO 21062 --- [ocp-schedules-19,d0c92424dbad00f8,9ca464cc3468f039] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:18:05.527 WARN 21062 --- [ocp-async-2,f26dfcbbb7539813,a5c7d083c03526e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:05.531 INFO 21062 --- [ocp-async-7,5e266c11e0460c2f,7e9872a32ec810cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:05.943 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:05.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c4414501c931ffb6,58f2b232920fe392] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:07.527 WARN 21062 --- [ocp-async-4,ff49a0bb0e8d50e6,d97839fe97fae68c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:07.532 INFO 21062 --- [ocp-async-1,88127253edace435,3d02a92549ef5ac5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:08.504 INFO 21062 --- [ocp-schedules-9,d23365c2bb5721ce,e44815d5a4c97597] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:18:08.504 INFO 21062 --- [ocp-schedules-4,1671e259f17dd730,a671dc906dd6b678] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:18:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:18:08.589411+08:00, lastProcessTime=2024-11-04T15:17:58.591518+08:00 2024-11-04 15:18:08.754 INFO 21062 --- [ocp-async-8,852765a3e3d4948a,fe8ad7928b5249f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:18:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:18:08.589411+08:00 2024-11-04 15:18:08.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:08.944 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:08.983 INFO 21062 --- [prometheus-minute-schedule-1,57157235d4152a4a,530350e8118b6e6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:09.286 INFO 21062 --- [ocp-schedules-29,2e1bc82dc0a76ccd,8160040a00dd747e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:18:09.527 WARN 21062 --- [ocp-async-8,654a20441a5c5788,e228edfbc3f14fd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:09.532 INFO 21062 --- [ocp-async-10,a66822e1df426294,e8649c318c23d174] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:09.845 INFO 21062 --- [ocp-schedules-12,5b8543805214cda2,aa3fe1b31dc68f92] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:18:09.845 INFO 21062 --- [ocp-schedules-12,5dd0b346a372ed24,9a64bfb121dda9e2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:18:09.849 INFO 21062 --- [ocp-schedules-12,5dd0b346a372ed24,9a64bfb121dda9e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:09.877 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:18:09.877 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:18:10.004 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:18:10.004 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:18:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:18:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:18:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:18:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:18:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:18:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:18:11.501 INFO 21062 --- [ocp-schedules-17,cd95cbf90d8b7646,29a08701d3d22ec2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:18:11.528 WARN 21062 --- [ocp-async-5,45168441d0625d31,8c42a0aea1a1e9f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:11.532 INFO 21062 --- [ocp-async-7,53abf97345a9624e,da83a43bf8c5e224] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:11.797 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=33, affectRows=33 2024-11-04 15:18:11.797 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=33, validCount=33 2024-11-04 15:18:11.797 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=49, affectRows=49 2024-11-04 15:18:11.797 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=26, affectRows=26 2024-11-04 15:18:11.797 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=44, affectRows=44 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=26, validCount=26 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=49, validCount=49 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=51, affectRows=51 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=43, validCount=43 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=54, affectRows=54 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=55, affectRows=55 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=36, affectRows=36 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=50, validCount=50 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=54, validCount=54 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=54, validCount=54 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=36, validCount=36 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=32, affectRows=32 2024-11-04 15:18:11.798 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=32, validCount=32 2024-11-04 15:18:11.799 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704620, nodeCount=42, affectRows=42 2024-11-04 15:18:11.799 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704620, nodeCount=42, validCount=42 2024-11-04 15:18:11.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:11.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:11.983 INFO 21062 --- [prometheus-minute-schedule-1,7e282ad85529ae88,735e68d35015e02c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:12.847 INFO 21062 --- [ocp-schedules-14,46bcfdc43cda47a6,f3090411c6251e45] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:18:12.852 INFO 21062 --- [ocp-schedules-14,46bcfdc43cda47a6,f3090411c6251e45] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:18:12.852 INFO 21062 --- [ocp-schedules-14,46bcfdc43cda47a6,f3090411c6251e45] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:18:13.528 WARN 21062 --- [ocp-async-4,125e2c01fe8965f8,b3cc97699431575b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:13.532 INFO 21062 --- [ocp-async-1,a774074961652fe5,7ee8572b0e4dbccd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:13.984 INFO 21062 --- [prometheus-cache-monitor1,9cea395a777ec97e,1bbd64a1ea41a97d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:18:13.984 INFO 21062 --- [prometheus-cache-monitor1,9cea395a777ec97e,1bbd64a1ea41a97d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:18:14.449 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:12:14.43, end=2024-11-04T15:17:14.446} 2024-11-04 15:18:14.449 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:12:14.429, end=2024-11-04T15:17:14.446} 2024-11-04 15:18:14.453 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:18:14.453 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:18:14.453 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:18:14.453 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:18:14.945 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:14.946 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:14.980 INFO 21062 --- [ocp-schedules-16,a30195bfb9881815,851fdca9fedfe40f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:18:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a315cce6d0c21260,eac27cd6025f2678] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:15.528 WARN 21062 --- [ocp-async-1,243b74a36d66f265,27e2ccfcd418d928] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:15.532 INFO 21062 --- [ocp-async-8,59dd521431810d8a,2650255c5a24c2f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:17.528 WARN 21062 --- [ocp-async-8,5333c625792ec552,8cf3025f3bdb5846] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:17.532 INFO 21062 --- [ocp-async-5,dd931f293b58b1aa,e86e83e5f0170d25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:17.946 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:17.947 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:17.983 INFO 21062 --- [prometheus-minute-schedule-1,e8ad047bb1982fc6,ebe207da33ed1002] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:18:18.589515+08:00, lastProcessTime=2024-11-04T15:18:08.589411+08:00 2024-11-04 15:18:18.754 INFO 21062 --- [ocp-async-7,4b69a419f10f41e3,ea9e2470afb92524] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:18:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:18:18.589515+08:00 2024-11-04 15:18:19.528 WARN 21062 --- [ocp-async-4,1f69fff8efec1cf1,f139e698cc1e4113] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:19.532 INFO 21062 --- [ocp-async-7,b3e9c803b9a18ec2,2f1dcaf5e319cffc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:20.947 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:20.948 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:20.983 INFO 21062 --- [prometheus-minute-schedule-1,cfadddba74d8be60,fd6e8dbfa23517a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:21.504 INFO 21062 --- [ocp-schedules-14,740fca0277c71782,3714fc86cfa22205] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:18:21.528 WARN 21062 --- [ocp-async-7,dfa67f62e52badb6,824237027077fe5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:21.533 INFO 21062 --- [ocp-async-9,297b9c321ce4263e,215f486a420123d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:23.529 WARN 21062 --- [ocp-async-8,5875a10b7f658c0b,d6cce42423d353be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:23.533 INFO 21062 --- [ocp-async-2,75676e04759947d9,61366ab977cdc20e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:23.948 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:23.949 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ddf6123d6ba94771,c899ab86bde2dc43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:25.005 INFO 21062 --- [ocp-schedules-19,976e40a09427f839,881e9d76abc53f64] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:18:25.529 WARN 21062 --- [ocp-async-4,d711ff80b5246210,7a90f8d58675cdc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:25.533 INFO 21062 --- [ocp-async-3,74e78da438f24cd9,e615c4bfe8a7a523] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:26.949 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:26.950 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:26.983 INFO 21062 --- [prometheus-minute-schedule-1,8c792a3439e06a8f,ebbb0ecbf6f0bc1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:26.984 INFO 21062 --- [prometheus-hour-schedule-1,fd20abcd898da440,c41abc6668a9765a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:18:27.529 WARN 21062 --- [ocp-async-1,32970b8c5e1e4b11,198db22413f74277] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:27.533 INFO 21062 --- [ocp-async-6,20279d4fddbf373c,7d9a0bd6ab602dba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:28.498 INFO 21062 --- [ocp-schedules-18,c7b58be97d2f59b3,5be38b752d1fd20b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:18:28.499 INFO 21062 --- [ocp-schedules-20,24f64b7ee62f696e,0518b0a76ae4a8ba] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:18:28.500 INFO 21062 --- [ocp-schedules-26,836753944d8303ef,0fd506b428ed3c91] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:18:28.500 INFO 21062 --- [ocp-schedules-23,6eb6739e4c0da847,00904e7d17599e5d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:18:28.502 INFO 21062 --- [ocp-schedules-18,c7b58be97d2f59b3,5be38b752d1fd20b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:18:28.502 INFO 21062 --- [ocp-schedules-9,42305081981565a8,b9a68a67b1671723] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:18:28.504 INFO 21062 --- [ocp-schedules-23,6eb6739e4c0da847,00904e7d17599e5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:28.504 INFO 21062 --- [ocp-schedules-26,836753944d8303ef,0fd506b428ed3c91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:18:28.504 INFO 21062 --- [ocp-schedules-11,e7f02e94c6454e03,69c4ac03a0a7777f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:18:28.505 INFO 21062 --- [ocp-schedules-20,24f64b7ee62f696e,0518b0a76ae4a8ba] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:18:28.509 INFO 21062 --- [ocp-schedules-6,f2b4de159ebee106,75b1bd2607eff1cf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:18:28.509 INFO 21062 --- [ocp-schedules-6,f2b4de159ebee106,75b1bd2607eff1cf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:18:28.509 INFO 21062 --- [ocp-schedules-17,56766dd651f6f11b,9e0848e6c16376f7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:18:28.509 INFO 21062 --- [ocp-schedules-19,88bedfc67be04aee,4d781fcd8c06aa1c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:18:28.509 INFO 21062 --- [ocp-schedules-17,56766dd651f6f11b,9e0848e6c16376f7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:18:28.509 INFO 21062 --- [ocp-schedules-19,88bedfc67be04aee,4d781fcd8c06aa1c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:18:28.587 INFO 21062 --- [ocp-schedules-25,67677ba23953c13d,e804709c795da42e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:18:28.587 INFO 21062 --- [ocp-schedules-25,67677ba23953c13d,e804709c795da42e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:18:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:18:28.590133+08:00, lastProcessTime=2024-11-04T15:18:18.589515+08:00 2024-11-04 15:18:28.661 INFO 21062 --- [ocp-schedules-15,eb013f8432915934,ec6a1c6debb249dc] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:18:28.662 INFO 21062 --- [ocp-schedules-15,eb013f8432915934,ec6a1c6debb249dc] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:18:28.754 INFO 21062 --- [ocp-async-4,2c5953230ccdecd2,18093eff50e5e2e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:18:28.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:18:28.590133+08:00 2024-11-04 15:18:28.953 INFO 21062 --- [ocp-schedules-10,0c71282b938cd30a,adfa2f8dc34a4836] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:18:28.953 INFO 21062 --- [ocp-schedules-10,0c71282b938cd30a,adfa2f8dc34a4836] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:18:28.970 INFO 21062 --- [ocp-schedules-20,1028eb1734ec1c0c,463f39918553da92] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:18:29.347 INFO 21062 --- [ocp-schedules-9,9d0c5ef2eefb0434,84f99135d8aaf406] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:18:29.347 INFO 21062 --- [ocp-schedules-9,9d0c5ef2eefb0434,84f99135d8aaf406] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:18:29.478 INFO 21062 --- [ocp-schedules-29,9e1d1492d3e92731,df04ec3451b0c0fb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:18:29.501 INFO 21062 --- [ocp-schedules-22,ad6ba480fe87990b,24e7445464d4667b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:18:29.506 INFO 21062 --- [ocp-schedules-22,ad6ba480fe87990b,24e7445464d4667b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:18:29.506 INFO 21062 --- [ocp-schedules-22,ad6ba480fe87990b,24e7445464d4667b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:18:29.529 WARN 21062 --- [ocp-async-2,ae60d1b89983f3e9,9634a645f8b14dfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:29.533 INFO 21062 --- [ocp-async-4,3dff08cc5de3a5f2,9970a82f6ed29886] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:29.950 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:29.951 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:29.964 INFO 21062 --- [ocp-schedules-7,c8932210a175586d,64b4395e239d941e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:18:29.964 INFO 21062 --- [ocp-schedules-7,c8932210a175586d,64b4395e239d941e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:18:29.966 INFO 21062 --- [ocp-schedules-7,c8932210a175586d,64b4395e239d941e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:18:29.966 INFO 21062 --- [ocp-schedules-7,c8932210a175586d,64b4395e239d941e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:18:29.966 INFO 21062 --- [ocp-schedules-7,c8932210a175586d,64b4395e239d941e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:18:29.966 INFO 21062 --- [ocp-schedules-7,c8932210a175586d,64b4395e239d941e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:18:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6e81161dfcc37e80,a1780a46df8c0ae6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:18:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:18:31.507 INFO 21062 --- [ocp-schedules-19,c75a14ec6daf9bc1,5fa85bebf27bc43e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:18:31.529 WARN 21062 --- [ocp-async-3,f8cff501b0655d24,c9413a1fc40926f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:31.534 INFO 21062 --- [ocp-async-7,edfe66df8b45986e,95ee1c70a6076c4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:32.854 INFO 21062 --- [ocp-schedules-8,ac8e63eaaa564113,5724a0d2718827de] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:18:32.860 INFO 21062 --- [ocp-schedules-8,ac8e63eaaa564113,5724a0d2718827de] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:18:32.860 INFO 21062 --- [ocp-schedules-8,ac8e63eaaa564113,5724a0d2718827de] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:18:32.951 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:32.952 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:32.983 INFO 21062 --- [prometheus-minute-schedule-1,58445dedcd84a791,8fa7584ced0b91cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:33.069 INFO 21062 --- [ocp-schedules-22,e01020f6acde5086,99387b7a53d47120] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:18:33.069 INFO 21062 --- [ocp-schedules-22,e01020f6acde5086,99387b7a53d47120] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:18:33.096 INFO 21062 --- [ocp-schedules-5,f00e73aa0ed56c31,561642798b4c6ba0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:18:33.097 INFO 21062 --- [ocp-schedules-22,e01020f6acde5086,99387b7a53d47120] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:18:33.097 INFO 21062 --- [ocp-schedules-22,e01020f6acde5086,99387b7a53d47120] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:18:33.300 INFO 21062 --- [ocp-schedules-4,52c6fd18fbc09374,00c6eb50b3eb85a6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:18:33.300+08:00, clusters=[1] 2024-11-04 15:18:33.301 INFO 21062 --- [ocp-schedules-4,52c6fd18fbc09374,00c6eb50b3eb85a6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:18:33.529 WARN 21062 --- [ocp-async-6,25608c67d0531c9e,931bf977ab8262fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:33.534 INFO 21062 --- [ocp-async-10,8bbda7ef3190b5fa,ad7a4b4f8e076991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:33.585 INFO 21062 --- [ocp-schedules-10,afd8f7f4e954fa6b,1b9fd2b5b96192ec] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:18:33.585+08:00, clusters=[1] 2024-11-04 15:18:33.589 INFO 21062 --- [ocp-schedules-10,afd8f7f4e954fa6b,1b9fd2b5b96192ec] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:18:33.626 INFO 21062 --- [ocp-schedules-6,e11f618da0937472,0c5c0e804b5ccce8] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:18:33.630 INFO 21062 --- [ocp-schedules-6,e11f618da0937472,0c5c0e804b5ccce8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:33.688 ERROR 21062 --- [ocp-schedules-6,e11f618da0937472,0c5c0e804b5ccce8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:18:33.688 INFO 21062 --- [ocp-schedules-6,e11f618da0937472,0c5c0e804b5ccce8] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:18:35.047 INFO 21062 --- [ocp-schedules-11,da5781ed81f7ff10,622a0e1b7c3ce647] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:18:35.334 INFO 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:18:35.334 INFO 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:18:35.334 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:18:35.334 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:18:35.344 INFO 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:18:35.345 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:18:35.345 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:18:35.346 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:18:35.346 WARN 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:18:35.346 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:18:35.347 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:18:35.347 INFO 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:18:35.347 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:18:35.348 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:18:35.348 WARN 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:18:35.348 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:18:35.355 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:18:35.355 INFO 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:18:35.355 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:18:35.369 ERROR 21062 --- [ocp-schedules-18,640be9f99a91a5e3,c845db365906aef9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:18:35.530 WARN 21062 --- [ocp-async-9,41769cca86d169a1,b0bca5bb11dd40f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:35.534 INFO 21062 --- [ocp-async-5,bb71310cf8867bad,88ac916549f2a96d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:35.952 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:35.953 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:35.983 INFO 21062 --- [prometheus-minute-schedule-1,743d3bd5f5937b62,1c57fb41a9591d8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:37.530 WARN 21062 --- [ocp-async-3,c3cb43cec299df8d,2ddbefa68ca33ba9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:37.534 INFO 21062 --- [ocp-async-1,2a59ea3b8a2e185e,141d78ec0718ddab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:38.504 INFO 21062 --- [ocp-schedules-23,6eb6739e4c0da847,00904e7d17599e5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:18:38.504 INFO 21062 --- [ocp-schedules-26,836753944d8303ef,0fd506b428ed3c91] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:18:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:18:38.589687+08:00, lastProcessTime=2024-11-04T15:18:28.590133+08:00 2024-11-04 15:18:38.754 INFO 21062 --- [ocp-async-10,f6d0d18b8e252bb1,29ad47d615e6a13d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:18:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:18:38.589687+08:00 2024-11-04 15:18:38.954 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:38.954 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:38.983 INFO 21062 --- [prometheus-minute-schedule-1,6dfd6f3bf7b8c764,2bc5e47f5cdaea6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:39.530 WARN 21062 --- [ocp-async-8,43f52ee5cc6c40ff,46948946f3a22309] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:39.534 INFO 21062 --- [ocp-async-9,f05cb2e9aefbcf37,397a755454881312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:39.877 INFO 21062 --- [ocp-schedules-12,5dd0b346a372ed24,9a64bfb121dda9e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:18:39.878 INFO 21062 --- [ocp-schedules-12,88d106f65a502a3a,8f46ce150bc39b5b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:18:39.883 INFO 21062 --- [ocp-schedules-12,88d106f65a502a3a,8f46ce150bc39b5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:39.911 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:18:39.911 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:18:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:18:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:18:41.510 INFO 21062 --- [ocp-schedules-29,19d95bc8bfb4e2a0,4213d4a5c108fc8c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:18:41.531 WARN 21062 --- [ocp-async-5,9f7324e556359c0c,c9ce97a2fd3755ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:41.535 INFO 21062 --- [ocp-async-3,eeb492cdec1fb74a,8790850dd389c052] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:18:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:18:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:18:41.954 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:41.955 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:41.983 INFO 21062 --- [prometheus-minute-schedule-1,313782ca04622d7b,7d628231279b82f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:43.531 WARN 21062 --- [ocp-async-1,f302c95c4df1f7b0,0c8d9bbd41227eaf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:43.535 INFO 21062 --- [ocp-async-7,be755b76c47c99cd,63510dfa0e75559b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:43.860 INFO 21062 --- [ocp-schedules-9,fbdee5cad1c92de2,d25d24d3e73b00ce] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:18:43.864 INFO 21062 --- [ocp-schedules-9,fbdee5cad1c92de2,d25d24d3e73b00ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:43.950 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:18:43.953 ERROR 21062 --- [ocp-schedules-9,fbdee5cad1c92de2,d25d24d3e73b00ce] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:18:43.953 INFO 21062 --- [ocp-schedules-9,fbdee5cad1c92de2,d25d24d3e73b00ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:18:44.955 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:44.956 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:44.983 INFO 21062 --- [prometheus-minute-schedule-1,945a72bd89efa24d,81108829aafdf85e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:45.071 INFO 21062 --- [ocp-schedules-6,c718279ff2e38df1,a9343a5ea33a950f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:18:45.531 WARN 21062 --- [ocp-async-8,94b1f19c253db468,07b77dd8d0c8c0eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:45.535 INFO 21062 --- [ocp-async-10,79c0bc7059656c26,8b6e5fb1cef44402] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:47.531 WARN 21062 --- [ocp-async-5,e626b5e7ce8043d7,813f56d734ddf001] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:47.535 INFO 21062 --- [ocp-async-2,814c82f4a2ad8fe1,886463f9c86b31b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:47.956 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:47.957 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:47.983 INFO 21062 --- [prometheus-minute-schedule-1,7177fbbdb7f6bad4,5f981a5704946664] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:18:48.589420+08:00, lastProcessTime=2024-11-04T15:18:38.589687+08:00 2024-11-04 15:18:48.754 INFO 21062 --- [ocp-async-4,96468bc783d312f8,3b0fd77a1c95bcd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:18:48.802 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:18:48.589420+08:00 2024-11-04 15:18:49.531 WARN 21062 --- [ocp-async-4,4e1725136497b23f,2ded24e602116d99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:49.535 INFO 21062 --- [ocp-async-7,90b6796cc9f2265c,7d58d7aac5116171] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:50.957 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:50.958 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:50.983 INFO 21062 --- [prometheus-minute-schedule-1,c7901bd249e6d924,25a0b7acb764df25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:51.513 INFO 21062 --- [ocp-schedules-16,968efd5dc2db2463,b6f9d836d1491510] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:18:51.532 WARN 21062 --- [ocp-async-6,0c25e8004f72536f,16c375f3e6042861] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:51.535 INFO 21062 --- [ocp-async-10,4f36a485e016f50f,2d193ba83c38474e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:52.863 INFO 21062 --- [ocp-schedules-20,f0ceaf3784227b8d,65ae76225c66d632] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:18:52.868 INFO 21062 --- [ocp-schedules-20,f0ceaf3784227b8d,65ae76225c66d632] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:18:52.868 INFO 21062 --- [ocp-schedules-20,f0ceaf3784227b8d,65ae76225c66d632] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:18:53.532 WARN 21062 --- [ocp-async-9,ec34f752eb1b0e4d,f84452b38d2e9108] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:53.536 INFO 21062 --- [ocp-async-2,914a7a6606386cac,ab10e2ee2212eaf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:53.958 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:53.959 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:53.983 INFO 21062 --- [prometheus-minute-schedule-1,37a3a21b5c1cdb19,96da272d662c2321] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:55.113 INFO 21062 --- [ocp-schedules-17,83057796f8c40006,aee99cdeb5b01cf1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:18:55.532 WARN 21062 --- [ocp-async-3,23f0f40938dc0512,4132ae7cd6c6b8e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:55.536 INFO 21062 --- [ocp-async-1,210c06b4404affae,dfccb8da08ac2b51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:56.959 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:56.960 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:56.983 INFO 21062 --- [prometheus-minute-schedule-1,f1fdc2a08318ce1c,13d5ac83f4ccb9da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:18:57.532 WARN 21062 --- [ocp-async-6,2a4a1bb3eb6b1350,6b99608cbcfd2266] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:57.536 INFO 21062 --- [ocp-async-8,178a80edce73b365,9d608a1e60b43f9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:58.500 INFO 21062 --- [ocp-schedules-17,fc063b68b1f0414f,42f6c32a9552284d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:18:58.500 INFO 21062 --- [ocp-schedules-5,c0ef4a8536fbed50,e91297a493f8a298] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:18:58.502 INFO 21062 --- [ocp-schedules-18,a694cee4c3b2699f,fc71f40fbeaa1b44] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:18:58.504 INFO 21062 --- [ocp-schedules-5,c0ef4a8536fbed50,e91297a493f8a298] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:58.504 INFO 21062 --- [ocp-schedules-17,fc063b68b1f0414f,42f6c32a9552284d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:18:58.504 INFO 21062 --- [ocp-schedules-21,d09b182b9a980fcf,54b692a3c8d07957] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:18:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:18:58.512 INFO 21062 --- [ocp-schedules-3,9fc867db7743c3df,78d4481619d63191] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:18:58.512 INFO 21062 --- [ocp-schedules-20,8803bcf2290a7779,a4b7f6215e23a585] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:18:58.512 INFO 21062 --- [ocp-schedules-3,9fc867db7743c3df,78d4481619d63191] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:18:58.512 INFO 21062 --- [ocp-schedules-20,8803bcf2290a7779,a4b7f6215e23a585] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:18:58.512 INFO 21062 --- [ocp-schedules-9,98a50f986b253592,0b1c5e1711399025] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:18:58.513 INFO 21062 --- [ocp-schedules-9,98a50f986b253592,0b1c5e1711399025] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:18:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:18:58.588349+08:00, lastProcessTime=2024-11-04T15:18:48.589420+08:00 2024-11-04 15:18:58.755 INFO 21062 --- [ocp-async-5,4e83c05e2936beb1,e20ecc4206e66ae1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:18:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:18:58.588349+08:00 2024-11-04 15:18:59.481 INFO 21062 --- [ocp-schedules-19,b9846270b6f54b42,4826d2660fb4a17b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:18:59.532 WARN 21062 --- [ocp-async-5,81334e8306350f42,99a5bb8eacb8f1e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:18:59.536 INFO 21062 --- [ocp-async-2,faf42df9c940358b,cd1ae0300ce11716] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:18:59.960 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:18:59.961 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:18:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7b684ee2037c65f1,5b54f291a287d9d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704740, interval=60 2024-11-04 15:19:01.517 INFO 21062 --- [ocp-schedules-29,7e0c3a8d2ce2567f,0f0f56dfcc24f6ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:19:01.533 WARN 21062 --- [ocp-async-3,259605d129f1dd98,7458501fd7acaf14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:01.536 INFO 21062 --- [ocp-async-1,069113e1603e8802,08482e6ac69ec746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:02.961 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:02.962 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:02.983 INFO 21062 --- [prometheus-minute-schedule-1,b4082d7aa25e92e5,1f6be8b38ed92c43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:03.533 WARN 21062 --- [ocp-async-6,4fa99b8471da281f,884642fcf9882d6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:03.537 INFO 21062 --- [ocp-async-8,bf1c4d374ad21845,44d8d0e999094c61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:05.153 INFO 21062 --- [ocp-schedules-10,516698edea6aa855,95ab3967a9ebc917] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:19:05.533 WARN 21062 --- [ocp-async-5,4170c25e672d2a2e,8de24135361a76ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:05.537 INFO 21062 --- [ocp-async-9,22fdcd3446442a65,9ee35606810a591e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:05.962 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:05.963 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d9a67c2400486ede,a634b3e115f203ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:07.533 WARN 21062 --- [ocp-async-3,4b4760d6a4a5d2f6,92c175f545db78db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:07.537 INFO 21062 --- [ocp-async-4,5ea4fed99f7468ac,99e6bd40e2531841] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:08.505 INFO 21062 --- [ocp-schedules-5,c0ef4a8536fbed50,e91297a493f8a298] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:19:08.505 INFO 21062 --- [ocp-schedules-17,fc063b68b1f0414f,42f6c32a9552284d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:19:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:19:08.589295+08:00, lastProcessTime=2024-11-04T15:18:58.588349+08:00 2024-11-04 15:19:08.755 INFO 21062 --- [ocp-async-7,988397e2638a05e1,30ecf0b99b76741e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:19:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:19:08.589295+08:00 2024-11-04 15:19:08.963 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:08.963 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a497877776eeb3cc,36b245e9ac63cf85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:09.286 INFO 21062 --- [ocp-schedules-13,9887de88d26d02d9,2497da68262205e6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:19:09.290 INFO 21062 --- [ocp-schedules-13,9887de88d26d02d9,2497da68262205e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:09.291 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:19:09.533 WARN 21062 --- [ocp-async-7,df30683a3da9863f,fa591b6fd09c8905] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:09.537 INFO 21062 --- [ocp-async-8,6c51f725f8fa99af,5e82f692695a0ee9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:09.912 INFO 21062 --- [ocp-schedules-12,88d106f65a502a3a,8f46ce150bc39b5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:19:09.912 INFO 21062 --- [ocp-schedules-12,9a13470430a179c7,96d982734fdd5e92] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:19:09.916 INFO 21062 --- [ocp-schedules-12,9a13470430a179c7,96d982734fdd5e92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:09.946 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:19:09.946 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:19:10.002 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:19:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:19:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:19:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:19:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:19:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:19:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:19:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:19:11.519 INFO 21062 --- [ocp-schedules-19,4c21f99efc534760,27f066fcc897b0da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:19:11.534 WARN 21062 --- [ocp-async-10,98c6eb642fcf16b1,973bc7b97026c29a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:11.537 INFO 21062 --- [ocp-async-9,c11b32231cb9892e,c5af7ef60faf6456] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=8, affectRows=8 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=8, validCount=8 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=40, affectRows=40 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=16, affectRows=16 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=11, affectRows=11 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=11, validCount=11 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=16, validCount=16 2024-11-04 15:19:11.807 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=39, validCount=39 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=29, affectRows=29 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=18, affectRows=18 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=30, affectRows=30 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=58, affectRows=58 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=50, affectRows=50 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=27, affectRows=27 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=18, validCount=18 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=29, validCount=29 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=30, validCount=30 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=65, affectRows=65 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=27, validCount=27 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=50, validCount=50 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=58, validCount=58 2024-11-04 15:19:11.808 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=63, validCount=63 2024-11-04 15:19:11.809 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704680, nodeCount=70, affectRows=70 2024-11-04 15:19:11.809 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704680, nodeCount=70, validCount=70 2024-11-04 15:19:11.964 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:11.965 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:11.983 INFO 21062 --- [prometheus-minute-schedule-1,dd6c3706ffab8919,b837a86136ac5f8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:12.871 INFO 21062 --- [ocp-schedules-18,017fab932b669af9,c7be274738edbb55] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:19:12.877 INFO 21062 --- [ocp-schedules-18,017fab932b669af9,c7be274738edbb55] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:19:12.877 INFO 21062 --- [ocp-schedules-18,017fab932b669af9,c7be274738edbb55] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:19:13.534 WARN 21062 --- [ocp-async-9,2fe6b85a097875f7,583699bd87d295e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:13.537 INFO 21062 --- [ocp-async-4,9d1941dd24120cfb,b8b44f62e9148655] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:13.984 INFO 21062 --- [prometheus-cache-monitor1,299823a76afd337a,7f1735bb75ea1839] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:19:13.984 INFO 21062 --- [prometheus-cache-monitor1,299823a76afd337a,7f1735bb75ea1839] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:19:14.965 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:14.966 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:14.983 INFO 21062 --- [prometheus-minute-schedule-1,bec6a3e5863c875f,d990e5e717f66cc3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:15.196 INFO 21062 --- [ocp-schedules-1,f4dd399ed8c67d81,47a7c404b0da3d13] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:19:15.534 WARN 21062 --- [ocp-async-4,a5ba187c0a08f8c1,5941d0b4d9d92296] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:15.538 INFO 21062 --- [ocp-async-7,f75688e9bb7df0a9,d6df59927d76ae8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:15.565 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:15:15.543, end=2024-11-04T15:17:15.563} 2024-11-04 15:19:15.570 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:19:15.570 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:19:17.534 WARN 21062 --- [ocp-async-7,40932bb450de0ba3,8a1ec693034d3c34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:17.538 INFO 21062 --- [ocp-async-10,63ba90e9c419ada5,c0d3a30d23426620] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:17.966 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:17.967 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:17.983 INFO 21062 --- [prometheus-minute-schedule-1,0dbba3df68ffdff8,2a34b09750945ff6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:19:18.590677+08:00, lastProcessTime=2024-11-04T15:19:08.589295+08:00 2024-11-04 15:19:18.755 INFO 21062 --- [ocp-async-3,0a46252da9915fce,849137cb15707424] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:19:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:19:18.590677+08:00 2024-11-04 15:19:18.933 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:19:18.933 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:19:18.942 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:19:18.942 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:19:18.942 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:19:18.943 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:19:18.948 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:19:18.948 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:19:18.948 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:19:18.948 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:19:18.985 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:19:18.994 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:19:18.994 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:19:18.995 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:19:18.995 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:19:18.995 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:19:19.002 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:19:19.291 INFO 21062 --- [ocp-schedules-13,9887de88d26d02d9,2497da68262205e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:19:19.522 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:19:19.534 WARN 21062 --- [ocp-async-2,54506cdc9532e934,8cefa0d7ff0d2017] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:19.538 INFO 21062 --- [ocp-async-3,d4717aeef2c15303,32700cd35fb6260f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:20.118 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.169702528S 2024-11-04 15:19:20.118 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:19:20.216 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:19:20.216 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:19:20.246 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:19:20.308 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:19:20.308 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:19:20.308 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:19:20.308 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:19:20.308 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:19:20.309 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:19:20.321 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:19:20.490 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] o.s.boot.SpringApplication : Started application in 1.547 seconds (JVM running for 10174.169) 2024-11-04 15:19:20.490 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.547552694S 2024-11-04 15:19:20.935 INFO 21062 --- [ocp-schedules-2,c6d6c2a7dcb50809,8996e8109fed92f5] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:19:20.967 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:20.968 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ff6fb34f2da584e2,97fbc3f922f43ae1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:21.522 INFO 21062 --- [ocp-schedules-5,91d4e155982383e0,a316732798d0df61] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:19:21.534 WARN 21062 --- [ocp-async-3,8802c15c8325dc94,8b48c6552356e3bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:21.538 INFO 21062 --- [ocp-async-1,56c0f4b547588fbc,9b4f5678e6b38ce0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:23.535 WARN 21062 --- [ocp-async-8,b757ff597713bd4d,3ea859c9400d649b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:23.538 INFO 21062 --- [ocp-async-5,6342b4fb8bcfd844,d4e0f34705c603dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:23.968 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:23.969 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:23.983 INFO 21062 --- [prometheus-minute-schedule-1,fb8ec09da9c09fa9,bddf6a42d3331dc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:25.221 INFO 21062 --- [ocp-schedules-23,b6bf9bf9ca20d847,487708c46e347046] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:19:25.535 WARN 21062 --- [ocp-async-2,4149c0f23263148d,91f9e6dfa4c0bd91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:25.538 INFO 21062 --- [ocp-async-9,76a8f43080c0b6d8,6710d47961928b8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:26.969 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:26.971 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:26.983 INFO 21062 --- [prometheus-minute-schedule-1,eae2c564d824095c,7162eb659991a1c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:26.984 INFO 21062 --- [prometheus-hour-schedule-1,647e4500c261ef16,faa53ba8e456c155] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:19:27.535 WARN 21062 --- [ocp-async-3,ba8f380b2e3fcd9c,af73ba8e97dede9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:27.539 INFO 21062 --- [ocp-async-6,6e65f6d801848bd0,cb1c060524651a03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:28.498 INFO 21062 --- [ocp-schedules-4,2db66ce121d72fa6,0635652c61ffb977] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:19:28.499 INFO 21062 --- [ocp-schedules-26,b44e1149fec75662,fc430da20d1ff74c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:19:28.500 INFO 21062 --- [ocp-schedules-17,9f07a0af9d7dd945,3a52831a1edbdfb5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:19:28.500 INFO 21062 --- [ocp-schedules-29,13c5db2faf0fc806,aa9771213ff69274] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:19:28.502 INFO 21062 --- [ocp-schedules-4,2db66ce121d72fa6,0635652c61ffb977] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:19:28.502 INFO 21062 --- [ocp-schedules-28,0e8b90e841f9a6b5,2ce129737a1c6adb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:19:28.504 INFO 21062 --- [ocp-schedules-17,9f07a0af9d7dd945,3a52831a1edbdfb5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:28.504 INFO 21062 --- [ocp-schedules-29,13c5db2faf0fc806,aa9771213ff69274] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:19:28.505 INFO 21062 --- [ocp-schedules-26,b44e1149fec75662,fc430da20d1ff74c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:19:28.505 INFO 21062 --- [ocp-schedules-2,49b6b93d45b473c6,ed7358195b314e34] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:19:28.509 INFO 21062 --- [ocp-schedules-21,b5af6fbbe6fa0a03,048160d72900b82a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:19:28.509 INFO 21062 --- [ocp-schedules-21,b5af6fbbe6fa0a03,048160d72900b82a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:19:28.509 INFO 21062 --- [ocp-schedules-20,fe7a8dc1db394f8b,ab40c2383a171436] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:19:28.510 INFO 21062 --- [ocp-schedules-20,fe7a8dc1db394f8b,ab40c2383a171436] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:19:28.510 INFO 21062 --- [ocp-schedules-11,8348e761b4f94014,72556456418d669b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:19:28.510 INFO 21062 --- [ocp-schedules-11,8348e761b4f94014,72556456418d669b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:19:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:19:28.588891+08:00, lastProcessTime=2024-11-04T15:19:18.590677+08:00 2024-11-04 15:19:28.617 INFO 21062 --- [ocp-schedules-13,8e135c8e44399b2b,db53136141dc2f81] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:19:28.617+08:00, tenantSize=0 2024-11-04 15:19:28.745 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:19:28.588891+08:00 2024-11-04 15:19:28.755 INFO 21062 --- [ocp-async-2,989286c43bd047c3,d4be327ce009a67e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:19:28.972 INFO 21062 --- [ocp-schedules-7,0ae41adf12e121d4,6ccfefacce9dc673] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:19:29.352 INFO 21062 --- [ocp-schedules-11,53c535fee3a8c3ce,9f412bf2cc130657] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:19:29.352 INFO 21062 --- [ocp-schedules-11,53c535fee3a8c3ce,9f412bf2cc130657] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:19:29.483 INFO 21062 --- [ocp-schedules-8,b31ed1b3f5ae703d,3e591b895067285c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:19:29.506 INFO 21062 --- [ocp-schedules-23,1e14b91602cec8fa,d2044b4bbc1badd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:19:29.511 INFO 21062 --- [ocp-schedules-23,1e14b91602cec8fa,d2044b4bbc1badd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:19:29.512 INFO 21062 --- [ocp-schedules-23,1e14b91602cec8fa,d2044b4bbc1badd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:19:29.535 WARN 21062 --- [ocp-async-2,5191d91ca2ef968f,6cd3479ab7446afc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:29.539 INFO 21062 --- [ocp-async-10,38bef5047e0c6c69,c49b51cad6208ce0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:29.785 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:19:29.785 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:19:29.793 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:19:29.793 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:19:29.793 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:19:29.803 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:19:29.803 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:19:29.803 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:19:29.808 INFO 21062 --- [ocp-schedules-24,0a9106ab5502ae91,c35da2d0224c598b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:19:29.967 INFO 21062 --- [ocp-schedules-19,607428c939656334,f59dd704912d0b74] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:19:29.967 INFO 21062 --- [ocp-schedules-19,607428c939656334,f59dd704912d0b74] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:19:29.971 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:29.972 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:29.975 INFO 21062 --- [ocp-schedules-19,607428c939656334,f59dd704912d0b74] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:19:29.975 INFO 21062 --- [ocp-schedules-19,607428c939656334,f59dd704912d0b74] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:19:29.979 INFO 21062 --- [ocp-schedules-19,607428c939656334,f59dd704912d0b74] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:19:29.979 INFO 21062 --- [ocp-schedules-19,607428c939656334,f59dd704912d0b74] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:19:29.983 INFO 21062 --- [prometheus-minute-schedule-1,203a6b47c63d15df,1bad9882bd07ce1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:19:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:19:31.525 INFO 21062 --- [ocp-schedules-16,0cc68be4876e2f5e,a3c0638edc672276] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:19:31.535 WARN 21062 --- [ocp-async-9,b57c8b49e3b6e61c,71a5b6c4951135b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:31.539 INFO 21062 --- [ocp-async-4,3ad7110a837050c7,3918f9d419348860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:32.880 INFO 21062 --- [ocp-schedules-20,d317878324f0ea20,81058a36c5a6b3c8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:19:32.886 INFO 21062 --- [ocp-schedules-20,d317878324f0ea20,81058a36c5a6b3c8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:19:32.886 INFO 21062 --- [ocp-schedules-20,d317878324f0ea20,81058a36c5a6b3c8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:19:32.972 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:32.973 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:32.983 INFO 21062 --- [prometheus-minute-schedule-1,e462fe6c2459c2c0,da32608b944d3cc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:33.097 INFO 21062 --- [ocp-schedules-5,263a258b87b4830c,5c2b9d32273aa944] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:19:33.097 INFO 21062 --- [ocp-schedules-5,263a258b87b4830c,5c2b9d32273aa944] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:19:33.122 INFO 21062 --- [ocp-schedules-5,263a258b87b4830c,5c2b9d32273aa944] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:19:33.122 INFO 21062 --- [ocp-schedules-5,263a258b87b4830c,5c2b9d32273aa944] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:19:33.124 INFO 21062 --- [ocp-schedules-21,7203325407903242,28d30c1cd3278bf3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:19:33.327 INFO 21062 --- [ocp-schedules-27,e2f3e9e0193d9196,4170afc54e031855] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:19:33.327+08:00, clusters=[1] 2024-11-04 15:19:33.329 INFO 21062 --- [ocp-schedules-27,e2f3e9e0193d9196,4170afc54e031855] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:19:33.536 WARN 21062 --- [ocp-async-6,573da66c86831ca3,c170e522ea8b5f40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:33.539 INFO 21062 --- [ocp-async-1,a9b682a30bd328a6,45abb446300e29ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:33.616 INFO 21062 --- [ocp-schedules-11,239162c185af62a5,387774227f42fb2c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:19:33.616+08:00, clusters=[1] 2024-11-04 15:19:33.619 INFO 21062 --- [ocp-schedules-11,239162c185af62a5,387774227f42fb2c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:19:35.256 INFO 21062 --- [ocp-schedules-23,2507fd54c54020bd,9d86c8afea9c16d2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:19:35.377 INFO 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:19:35.377 INFO 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:19:35.377 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:19:35.377 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:19:35.386 INFO 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:19:35.387 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:19:35.387 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:19:35.388 WARN 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:19:35.388 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:19:35.389 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:19:35.389 INFO 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:19:35.389 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:19:35.390 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:19:35.390 WARN 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:19:35.390 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:19:35.391 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:19:35.397 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:19:35.397 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:19:35.397 INFO 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:19:35.406 ERROR 21062 --- [ocp-schedules-3,ca50cc02609eb07f,5ae0a69cc2827647] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:19:35.536 WARN 21062 --- [ocp-async-7,6b7802dadb4413a9,726c834be100d3cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:35.539 INFO 21062 --- [ocp-async-5,6134213de2f6acc5,b4c5bce9bb9fcff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:35.973 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:35.978 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:35.983 INFO 21062 --- [prometheus-minute-schedule-1,53100bdc5550820a,8b6566da61c9b671] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:37.536 WARN 21062 --- [ocp-async-3,b3f28869250eae70,dc85a0a06d79cf49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:37.540 INFO 21062 --- [ocp-async-9,b307a53fc82af385,52448da5636eab7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:38.504 INFO 21062 --- [ocp-schedules-17,9f07a0af9d7dd945,3a52831a1edbdfb5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:19:38.504 INFO 21062 --- [ocp-schedules-29,13c5db2faf0fc806,aa9771213ff69274] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:19:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:19:38.588893+08:00, lastProcessTime=2024-11-04T15:19:28.588891+08:00 2024-11-04 15:19:38.745 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:19:38.588893+08:00 2024-11-04 15:19:38.755 INFO 21062 --- [ocp-async-1,738279193834548f,ab600a40295d5459] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:19:38.978 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:38.979 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:38.983 INFO 21062 --- [prometheus-minute-schedule-1,71c188549023529e,4d1f6dcf98c9c4cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:39.536 WARN 21062 --- [ocp-async-8,21f8539675c4b41a,ffbf08278232d4e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:39.540 INFO 21062 --- [ocp-async-2,fa20d14e9a80a82c,6b0adc09c6cfcfcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:39.947 INFO 21062 --- [ocp-schedules-12,9a13470430a179c7,96d982734fdd5e92] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:19:39.947 INFO 21062 --- [ocp-schedules-12,dabb4c8243272e86,639e4077f6de4d4d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:19:39.951 INFO 21062 --- [ocp-schedules-12,dabb4c8243272e86,639e4077f6de4d4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:39.981 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:19:39.981 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:19:40.002 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:19:40.002 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:19:41.528 INFO 21062 --- [ocp-schedules-19,33acc67308c601cb,c56909b673c5d8bb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:19:41.536 WARN 21062 --- [ocp-async-5,4215bd4c59a7a3b1,808a04c17c4a59d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:41.540 INFO 21062 --- [ocp-async-3,4755bd21492cdede,ff841ec1279ef43d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:41.979 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:41.980 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:41.983 INFO 21062 --- [prometheus-minute-schedule-1,70657ede055331b8,5d3767ae6c43a9a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:43.537 WARN 21062 --- [ocp-async-9,f533c6d30fbca33e,2208b213963e03c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:43.540 INFO 21062 --- [ocp-async-4,0666ec3c74103340,02df9f8130692f24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:43.954 INFO 21062 --- [ocp-schedules-19,0e5bb31dd6fdf889,33d538cab781f5fd] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:19:43.957 INFO 21062 --- [ocp-schedules-19,0e5bb31dd6fdf889,33d538cab781f5fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:44.044 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:19:44.052 ERROR 21062 --- [ocp-schedules-19,0e5bb31dd6fdf889,33d538cab781f5fd] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:19:44.052 INFO 21062 --- [ocp-schedules-19,0e5bb31dd6fdf889,33d538cab781f5fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:19:44.980 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:44.980 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:44.983 INFO 21062 --- [prometheus-minute-schedule-1,674d016f48c0ac19,ca6c42db2897662a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:45.279 INFO 21062 --- [ocp-schedules-5,ed300654302dfbc8,7599d6ca84d4978e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:19:45.537 WARN 21062 --- [ocp-async-8,13b7ed942df88acf,a2404cc0260ff9a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:45.541 INFO 21062 --- [ocp-async-1,ba94815da72d1c88,843bce0da2099022] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:47.537 WARN 21062 --- [ocp-async-5,e9abc95c2de6cf4d,f01bc0a90fd83d46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:47.541 INFO 21062 --- [ocp-async-7,6dc3d995671388bc,9b499e8c0dc7297e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:47.981 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:47.981 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:47.983 INFO 21062 --- [prometheus-minute-schedule-1,bda4a69543adbd7f,f09318e7614811aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:19:48.590738+08:00, lastProcessTime=2024-11-04T15:19:38.588893+08:00 2024-11-04 15:19:48.755 INFO 21062 --- [ocp-async-10,c56aba169ea66103,a31300bd7fbaab8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:19:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:19:48.590738+08:00 2024-11-04 15:19:49.537 WARN 21062 --- [ocp-async-9,f365930d1a2d241b,9b3c0b5b1b547455] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:49.541 INFO 21062 --- [ocp-async-4,e4f9beb02660d2a6,4a485fbf99103d3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:50.981 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:50.982 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:50.983 INFO 21062 --- [prometheus-minute-schedule-1,35115092b8fe68de,091781614af3352b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:51.531 INFO 21062 --- [ocp-schedules-17,9c13bb9dc3b5b171,16d8c144ca320b5e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:19:51.537 WARN 21062 --- [ocp-async-6,90b0a17b59b545e4,1b10c7727b7cb34a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:51.541 INFO 21062 --- [ocp-async-1,b1178447848cadf2,76de5a09000b489d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:52.889 INFO 21062 --- [ocp-schedules-11,b71ef214109ac6f1,fbb99d0c75adff79] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:19:52.895 INFO 21062 --- [ocp-schedules-11,b71ef214109ac6f1,fbb99d0c75adff79] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:19:52.895 INFO 21062 --- [ocp-schedules-11,b71ef214109ac6f1,fbb99d0c75adff79] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:19:53.537 WARN 21062 --- [ocp-async-5,c6aca4ecf792a57b,6022c185cfc8022e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:53.541 INFO 21062 --- [ocp-async-7,8066ab0050265baa,c5fe02af443c4bf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:53.983 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:53.983 INFO 21062 --- [prometheus-minute-schedule-1,31b8156191170210,e800cbc65c4a5cb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:53.983 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:55.318 INFO 21062 --- [ocp-schedules-18,7692a0ecacbe38ef,4710715342e67cb2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:19:55.538 WARN 21062 --- [ocp-async-9,9426324e2e258897,00102fe047ecf6c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:55.542 INFO 21062 --- [ocp-async-10,ff51ffef99ee0e55,8422e7ab4b1bb631] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:56.983 INFO 21062 --- [prometheus-minute-schedule-1,fa4d8fc721ede86c,bc65cfe5f7f949a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:56.983 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:56.984 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:19:57.538 WARN 21062 --- [ocp-async-6,8ef1d7ef28596652,879a0a5d59cac891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:57.542 INFO 21062 --- [ocp-async-8,1de8546069e7d0bd,07722def1a89eb12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:58.500 INFO 21062 --- [ocp-schedules-24,68628fb8b9f88848,4657c1e29c4b3e67] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:19:58.500 INFO 21062 --- [ocp-schedules-23,edaefa2be4d33a34,3ca8259c3dd3bf0f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:19:58.502 INFO 21062 --- [ocp-schedules-27,d019300c0717f097,a723e1297a10dea9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:19:58.503 INFO 21062 --- [ocp-schedules-24,68628fb8b9f88848,4657c1e29c4b3e67] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:58.504 INFO 21062 --- [ocp-schedules-23,edaefa2be4d33a34,3ca8259c3dd3bf0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:19:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:19:58.505 INFO 21062 --- [ocp-schedules-17,a3f4caa01188c575,f29b7cce9cdcdb00] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:19:58.511 INFO 21062 --- [ocp-schedules-25,817b4b725db873c1,1ec206de447ccd2c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:19:58.511 INFO 21062 --- [ocp-schedules-25,817b4b725db873c1,1ec206de447ccd2c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:19:58.511 INFO 21062 --- [ocp-schedules-1,eb0b86c325f5be6c,fa3c23479ecde56d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:19:58.511 INFO 21062 --- [ocp-schedules-1,eb0b86c325f5be6c,fa3c23479ecde56d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:19:58.511 INFO 21062 --- [ocp-schedules-29,e772e1c9e01d8597,671630564fdebb7e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:19:58.511 INFO 21062 --- [ocp-schedules-29,e772e1c9e01d8597,671630564fdebb7e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:19:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:19:58.589511+08:00, lastProcessTime=2024-11-04T15:19:48.590738+08:00 2024-11-04 15:19:58.756 INFO 21062 --- [ocp-async-2,587659c55652b2a4,93dffaabb627fca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:19:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:19:58.589511+08:00 2024-11-04 15:19:59.486 INFO 21062 --- [ocp-schedules-5,eaaf6a8cd2824a66,b9c95e88dfb508d0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:19:59.538 WARN 21062 --- [ocp-async-2,bc2ffa5e96a8a38f,f21042d54f522956] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:19:59.542 INFO 21062 --- [ocp-async-7,d64b60f803cdbeeb,1d2905d9b54c8b89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:19:59.983 INFO 21062 --- [prometheus-minute-schedule-1,834755e9a593e294,39b0563f42d93a84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704800, interval=60 2024-11-04 15:19:59.985 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:19:59.985 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:00.951 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2a2c2545f13bfffe] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:15:00.0, nextRun=2024-11-04 15:20:00.0, currentDate=Mon Nov 04 15:20:00 CST 2024 2024-11-04 15:20:00.951 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2a2c2545f13bfffe] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:15:00.0, nextRunTime=2024-11-04 15:20:00.0} 2024-11-04 15:20:00.978 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2a2c2545f13bfffe] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:20:01.011 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2a2c2545f13bfffe] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:20:01.011 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,2a2c2545f13bfffe] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:20:01.534 INFO 21062 --- [ocp-schedules-9,c7b1b6fd577c0966,223b822f84f8be8b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:20:01.538 WARN 21062 --- [ocp-async-9,42db8765b24badfe,0d8b10632dcd3612] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:01.542 INFO 21062 --- [ocp-async-10,2489bc2d78176464,73f4d0334e98f9fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:02.983 INFO 21062 --- [prometheus-minute-schedule-1,551cbdfcbd0b24c0,71b90999c70007f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:02.985 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:02.986 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:03.538 WARN 21062 --- [ocp-async-6,a095de85e16a4e37,f443fa459ae0a8d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:03.542 INFO 21062 --- [ocp-async-4,52ff4729b0d0c29d,13bd5627472f5f6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:04.062 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,08f64048ae61d9d6] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:10:05.0, nextRun=2024-11-04 15:20:04.0, currentDate=Mon Nov 04 15:20:04 CST 2024 2024-11-04 15:20:04.063 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,08f64048ae61d9d6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 15:10:05.0, nextRunTime=2024-11-04 15:20:04.0} 2024-11-04 15:20:04.068 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,08f64048ae61d9d6] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 15:20:04.068 INFO 21062 --- [task-coordinator3,dd193f8f67a5a423,08f64048ae61d9d6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T15:20:04.068+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 15:20:04.797 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,a30b0b33fa158192] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 87, operation:EXECUTE, state: RUNNING 2024-11-04 15:20:04.816 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=87, context=Context{parallelIdx=-1, stringMap={task_instance_id=24, task_operation=execute, latest_execution_start_time=2024-11-04T15:20:04.788+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=87}, listMap={}}, executor=10.46.128.24 2024-11-04 15:20:04.821 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:20:04.821 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:20:04.823 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.824 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:20:04.825 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.826 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.826 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.827 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:20:04.843 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.843 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.844 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.844 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:20:04.857 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.857 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.858 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.858 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:20:04.872 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.872 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.873 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.874 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:20:04.886 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.887 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.887 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.887 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:20:04.893 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.893 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.893 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.894 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:20:04.898 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.898 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.899 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.899 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:20:04.903 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.903 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.904 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.904 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:20:04.910 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.910 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.911 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.911 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:20:04.916 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.917 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.917 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.918 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:20:04.922 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.923 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.923 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.924 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:20:04.931 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.932 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.932 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.933 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:20:04.946 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.947 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.947 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.948 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:20:04.952 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.953 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.953 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.954 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:20:04.959 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.959 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.960 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.960 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:20:04.965 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.966 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.966 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.967 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:20:04.971 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.971 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.972 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.972 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:20:04.976 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.977 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.977 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.978 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:20:04.982 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.983 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.983 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.984 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:20:04.989 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.989 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.990 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.991 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:20:04.994 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.995 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:04.996 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:04.996 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:20:05.001 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.001 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.002 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.002 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:20:05.006 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.007 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.007 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.008 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:20:05.012 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.012 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.013 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.013 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:20:05.017 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.018 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.018 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.018 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:20:05.022 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.023 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.023 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.023 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:20:05.028 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.028 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.029 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.029 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:20:05.033 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.033 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.034 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.035 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:20:05.039 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.039 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.040 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.040 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:20:05.045 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.045 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.046 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.046 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:20:05.050 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.050 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.051 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.051 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:20:05.058 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.059 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.060 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.060 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:20:05.064 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.065 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.065 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.066 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:20:05.070 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.070 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.071 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.072 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:20:05.079 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.080 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.080 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.080 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:20:05.087 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.088 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.088 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.089 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:20:05.094 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.095 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.095 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.096 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:20:05.101 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.101 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.102 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.102 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:20:05.106 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.106 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.107 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.107 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:20:05.110 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.111 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.111 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.111 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:20:05.115 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.115 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.116 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.116 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:20:05.120 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.120 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.120 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.121 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:20:05.129 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.130 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.131 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.131 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:20:05.152 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.152 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.153 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.153 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:20:05.159 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.159 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.160 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.161 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:20:05.166 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.166 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.167 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.167 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:20:05.173 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.173 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.174 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.174 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:20:05.178 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.178 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.179 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.179 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:20:05.183 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.184 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.184 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.185 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:20:05.189 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:20:05.189 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:20:05.190 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.191 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:20:05.192 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.192 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.193 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.194 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:20:05.199 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.199 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:20:05.200 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:05.200 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:20:05.204 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:20:05.212 INFO 21062 --- [subtask-executor20,dd193f8f67a5a423,533991af5a06efa8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 87, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 15:20:05.343 INFO 21062 --- [ocp-schedules-15,3b3e9cd5d0c42aed,6e4f3a6e968494c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:20:05.446 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:24, status:SUCCESS 2024-11-04 15:20:05.447 INFO 21062 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:24, status:SUCCESS 2024-11-04 15:20:05.538 WARN 21062 --- [ocp-async-1,05cce05fe7cbe455,80a6b637ed509822] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:05.542 INFO 21062 --- [ocp-async-2,9084629fceb354a8,394cc33bd83823d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:05.983 INFO 21062 --- [prometheus-minute-schedule-1,7516bf219933f0df,b0a4f38d4e7a7b01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:05.986 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:05.987 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:07.539 WARN 21062 --- [ocp-async-9,474ef1f27a593b84,021b22049bc945ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:07.543 INFO 21062 --- [ocp-async-3,362749691d313f65,161026457ac351d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:08.504 INFO 21062 --- [ocp-schedules-24,68628fb8b9f88848,4657c1e29c4b3e67] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:20:08.504 INFO 21062 --- [ocp-schedules-23,edaefa2be4d33a34,3ca8259c3dd3bf0f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:20:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:20:08.589295+08:00, lastProcessTime=2024-11-04T15:19:58.589511+08:00 2024-11-04 15:20:08.756 INFO 21062 --- [ocp-async-10,5ff2356ad25e02f7,12afe9ba970f98e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:20:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:20:08.589295+08:00 2024-11-04 15:20:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2114e8605a6bcdbe,5a33bcb040259d0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:08.987 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:08.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:09.539 WARN 21062 --- [ocp-async-8,3805f07577a1a9b7,e3e44f002896fe3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:09.543 INFO 21062 --- [ocp-async-10,88c8eccfbae03c84,84211b1c1e0a688a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:09.981 INFO 21062 --- [ocp-schedules-12,dabb4c8243272e86,639e4077f6de4d4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:20:09.981 INFO 21062 --- [ocp-schedules-12,141dddd04a5b536e,5080bc6c4e989f71] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:20:09.985 INFO 21062 --- [ocp-schedules-12,141dddd04a5b536e,5080bc6c4e989f71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:10.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:20:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:20:10.014 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:20:10.014 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:20:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:20:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:20:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:20:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:20:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:20:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:20:11.537 INFO 21062 --- [ocp-schedules-5,781b65ea0b483c1d,94a2d5c70354b12e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:20:11.539 WARN 21062 --- [ocp-async-1,a7cf2f13fdedea4b,1907233f46fc5242] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:11.543 INFO 21062 --- [ocp-async-5,0a219f71eb7de7e7,c224419f6ca169c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:11.806 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=7, affectRows=7 2024-11-04 15:20:11.806 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=7, validCount=7 2024-11-04 15:20:11.806 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=18, affectRows=18 2024-11-04 15:20:11.806 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=18, validCount=18 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=43, affectRows=43 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=43, validCount=43 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=56, affectRows=56 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=71, affectRows=71 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=27, affectRows=27 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=36, affectRows=36 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=56, validCount=56 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=79, affectRows=79 2024-11-04 15:20:11.807 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=26, validCount=26 2024-11-04 15:20:11.808 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=36, validCount=36 2024-11-04 15:20:11.808 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=71, validCount=71 2024-11-04 15:20:11.808 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=77, validCount=77 2024-11-04 15:20:11.808 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704740, nodeCount=85, affectRows=85 2024-11-04 15:20:11.808 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704740, nodeCount=85, validCount=85 2024-11-04 15:20:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0ddd1e178a56da82,e3587df1488a74e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:11.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:11.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:12.898 INFO 21062 --- [ocp-schedules-23,69f43bf66463aa74,ff23121e62e8c4aa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:20:12.903 INFO 21062 --- [ocp-schedules-23,69f43bf66463aa74,ff23121e62e8c4aa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:20:12.903 INFO 21062 --- [ocp-schedules-23,69f43bf66463aa74,ff23121e62e8c4aa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:20:13.539 WARN 21062 --- [ocp-async-2,f307295af5664ba7,c49dbe8f44234285] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:13.543 INFO 21062 --- [ocp-async-7,b754b445ec814f32,946c164cef196afe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:13.984 INFO 21062 --- [prometheus-cache-monitor1,ade932f48d4902f7,4bde9c0de081a1c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:20:13.984 INFO 21062 --- [prometheus-cache-monitor1,ade932f48d4902f7,4bde9c0de081a1c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:20:14.983 INFO 21062 --- [prometheus-minute-schedule-1,5ca1afa8459df856,d27b45d4d04ea961] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:14.988 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:14.989 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:15.384 INFO 21062 --- [ocp-schedules-2,0f06ea9f8a3d0bd9,db1abb0380149639] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:20:15.539 WARN 21062 --- [ocp-async-3,157fa1739d209136,e73ae0892d450cdb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:15.543 INFO 21062 --- [ocp-async-6,fc35034380fe612d,603e9130913f52a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:17.540 WARN 21062 --- [ocp-async-1,55127d3e2c793b3f,d518dfbbcf867649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:17.544 INFO 21062 --- [ocp-async-4,92bdb7816ab8f1b4,f24eb645dde42d10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:17.983 INFO 21062 --- [prometheus-minute-schedule-1,d6c86598892ed4d1,64bcb4ef8cfbd4b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:17.989 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:17.990 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:20:18.589744+08:00, lastProcessTime=2024-11-04T15:20:08.589295+08:00 2024-11-04 15:20:18.756 INFO 21062 --- [ocp-async-9,b3105772c8668f48,26a2a2339cd320fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:20:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:20:18.589744+08:00 2024-11-04 15:20:19.292 INFO 21062 --- [ocp-schedules-7,a3b5dbf34d90591b,8ec76b63b318c5f0] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:20:19.295 INFO 21062 --- [ocp-schedules-7,a3b5dbf34d90591b,8ec76b63b318c5f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:19.296 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:20:19.540 WARN 21062 --- [ocp-async-9,8c6ea9afac5479ff,a9c2e8fe6bff7b4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:19.544 INFO 21062 --- [ocp-async-7,f99fb9e21cb0805f,b1da631a6b960455] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:20.983 INFO 21062 --- [prometheus-minute-schedule-1,2ffa68b5c022a4b1,19345759cd494b10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:20.990 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:20.991 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:21.540 WARN 21062 --- [ocp-async-8,fe6bf425c1bb20fa,40cf1d02a563a57a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:21.541 INFO 21062 --- [ocp-schedules-8,955ccd25d6da8693,4f79cb57800bee8d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:20:21.544 INFO 21062 --- [ocp-async-6,e75f830027d43692,a86e209f7ac15ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:23.540 WARN 21062 --- [ocp-async-10,60522e9b085ff4c4,f7e81fe2e45f5e71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:23.544 INFO 21062 --- [ocp-async-4,5dec6c12ef399969,1d3e97809e11ca4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:23.983 INFO 21062 --- [prometheus-minute-schedule-1,e8f2b6b2da852464,ab42d7030f6bf98b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:23.991 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:23.992 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:25.410 INFO 21062 --- [ocp-schedules-9,f0c910f4e8679d04,55262beebc7b9b25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:20:25.540 WARN 21062 --- [ocp-async-5,e1e50ed1d1b8f04b,f8b7ab9fa7a72530] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:25.544 INFO 21062 --- [ocp-async-2,3c22b207d253bed1,08a881282a4222ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:26.983 INFO 21062 --- [prometheus-minute-schedule-1,bc098daf09ef9866,d9a3928a0b529c88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:26.984 INFO 21062 --- [prometheus-hour-schedule-1,761e144b76a4abe7,e53eb09ba05f6370] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:20:26.992 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:26.992 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:27.540 WARN 21062 --- [ocp-async-7,2ba3899caea87b02,0b41c8976c4c5e92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:27.544 INFO 21062 --- [ocp-async-3,6a7d32571a537c41,f4d4ed6e68315dd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:28.498 INFO 21062 --- [ocp-schedules-26,8f2d5609882d0edf,43bcd88f5b785792] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:20:28.499 INFO 21062 --- [ocp-schedules-11,ddc4d57bae075976,cd160e3feb242819] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:20:28.500 INFO 21062 --- [ocp-schedules-22,ccc98e7aaab92a04,76959e1325dfd0a6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:20:28.500 INFO 21062 --- [ocp-schedules-29,de2bbcc7241455aa,8cbb5896ccb0b17c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:20:28.501 INFO 21062 --- [ocp-schedules-26,8f2d5609882d0edf,43bcd88f5b785792] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:20:28.502 INFO 21062 --- [ocp-schedules-17,e9cd30adfa0ded3e,317c6937d3ee6045] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:20:28.503 INFO 21062 --- [ocp-schedules-22,ccc98e7aaab92a04,76959e1325dfd0a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:20:28.503 INFO 21062 --- [ocp-schedules-29,de2bbcc7241455aa,8cbb5896ccb0b17c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:28.504 INFO 21062 --- [ocp-schedules-3,51657b1b7945b67c,785edc708262e3bb] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:20:28.505 INFO 21062 --- [ocp-schedules-11,ddc4d57bae075976,cd160e3feb242819] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:20:28.509 INFO 21062 --- [ocp-schedules-15,0a01d3afe1436402,9cddf5dfda8182c5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:20:28.510 INFO 21062 --- [ocp-schedules-15,0a01d3afe1436402,9cddf5dfda8182c5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:20:28.510 INFO 21062 --- [ocp-schedules-30,db6d375d8565addd,a3743d431228ff16] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:20:28.510 INFO 21062 --- [ocp-schedules-10,0840e37858ed139a,11f37824a2691782] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:20:28.510 INFO 21062 --- [ocp-schedules-10,0840e37858ed139a,11f37824a2691782] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:20:28.510 INFO 21062 --- [ocp-schedules-30,db6d375d8565addd,a3743d431228ff16] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:20:28.587 INFO 21062 --- [ocp-schedules-27,b3aed22974239382,3202dd985d3e6b04] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:20:28.587 INFO 21062 --- [ocp-schedules-27,b3aed22974239382,3202dd985d3e6b04] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:20:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:20:28.590121+08:00, lastProcessTime=2024-11-04T15:20:18.589744+08:00 2024-11-04 15:20:28.756 INFO 21062 --- [ocp-async-5,42bcf327952ee1bc,5e878e8d64ac5c49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:20:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:20:28.590121+08:00 2024-11-04 15:20:28.975 INFO 21062 --- [ocp-schedules-21,85d5909d65d9887c,60ec9b10351ca464] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:20:29.296 INFO 21062 --- [ocp-schedules-7,a3b5dbf34d90591b,8ec76b63b318c5f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:20:29.357 INFO 21062 --- [ocp-schedules-26,249bb53d6fcbedb1,9fa083ca8f544866] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:20:29.357 INFO 21062 --- [ocp-schedules-26,249bb53d6fcbedb1,9fa083ca8f544866] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:20:29.490 INFO 21062 --- [ocp-schedules-1,2db00fae97983591,88033b8311fbc798] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:20:29.512 INFO 21062 --- [ocp-schedules-4,d1fca3ad366e85f2,500f9571d0921733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:20:29.516 INFO 21062 --- [ocp-schedules-4,d1fca3ad366e85f2,500f9571d0921733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:20:29.517 INFO 21062 --- [ocp-schedules-4,d1fca3ad366e85f2,500f9571d0921733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:20:29.541 WARN 21062 --- [ocp-async-4,5b9c891b0e6df150,b786fabf51b893c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:29.545 INFO 21062 --- [ocp-async-9,ef66c0c2f2b9b844,9e9a38c95d05c549] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:29.983 INFO 21062 --- [prometheus-minute-schedule-1,e96cd9f582978c06,6a1b8a07aff8fd70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:29.985 INFO 21062 --- [ocp-schedules-9,9206fa85c30bc763,78f97051a2fb5b19] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:20:29.985 INFO 21062 --- [ocp-schedules-9,9206fa85c30bc763,78f97051a2fb5b19] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:20:29.986 INFO 21062 --- [ocp-schedules-9,9206fa85c30bc763,78f97051a2fb5b19] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:20:29.986 INFO 21062 --- [ocp-schedules-9,9206fa85c30bc763,78f97051a2fb5b19] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:20:29.986 INFO 21062 --- [ocp-schedules-9,9206fa85c30bc763,78f97051a2fb5b19] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:20:29.986 INFO 21062 --- [ocp-schedules-9,9206fa85c30bc763,78f97051a2fb5b19] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:20:29.993 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:29.993 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:20:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:20:31.541 WARN 21062 --- [ocp-async-7,8dfb2b8b6134391c,f90083840038bddb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:31.544 INFO 21062 --- [ocp-schedules-30,e92c6e3c29a5699d,40cfa2f4b1f3565d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:20:31.545 INFO 21062 --- [ocp-async-8,f37a7aad9014128f,77e0d4ba48ccc184] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:32.905 INFO 21062 --- [ocp-schedules-23,5ba301cb01bee59c,a200c0fe770e951d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:20:32.911 INFO 21062 --- [ocp-schedules-23,5ba301cb01bee59c,a200c0fe770e951d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:20:32.911 INFO 21062 --- [ocp-schedules-23,5ba301cb01bee59c,a200c0fe770e951d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:20:32.983 INFO 21062 --- [prometheus-minute-schedule-1,406429a1e82833ed,09d4a9f909c5bee2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:32.993 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:32.994 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:33.123 INFO 21062 --- [ocp-schedules-18,f7333c8e798e89e7,7f94e2f401094626] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:20:33.123 INFO 21062 --- [ocp-schedules-18,f7333c8e798e89e7,7f94e2f401094626] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:20:33.151 INFO 21062 --- [ocp-schedules-5,03ae320d0b49cdf9,a51d436ac0fd5dd9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:20:33.152 INFO 21062 --- [ocp-schedules-18,f7333c8e798e89e7,7f94e2f401094626] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:20:33.152 INFO 21062 --- [ocp-schedules-18,f7333c8e798e89e7,7f94e2f401094626] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:20:33.355 INFO 21062 --- [ocp-schedules-2,2eaf2aff999de62e,c1554517b4631520] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:20:33.355+08:00, clusters=[1] 2024-11-04 15:20:33.357 INFO 21062 --- [ocp-schedules-2,2eaf2aff999de62e,c1554517b4631520] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:20:33.541 WARN 21062 --- [ocp-async-3,62369138ee149ffa,6132e6c95e914252] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:33.545 INFO 21062 --- [ocp-async-6,5246435ba1f4425c,a2b9502e681fab9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:33.647 INFO 21062 --- [ocp-schedules-26,292ea629f7824a5c,7456562b0a2f5f00] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:20:33.647+08:00, clusters=[1] 2024-11-04 15:20:33.651 INFO 21062 --- [ocp-schedules-26,292ea629f7824a5c,7456562b0a2f5f00] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:20:33.688 INFO 21062 --- [ocp-schedules-25,c2b87e26fe766164,5078578ca2a28b7b] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:20:33.691 INFO 21062 --- [ocp-schedules-25,c2b87e26fe766164,5078578ca2a28b7b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:33.751 ERROR 21062 --- [ocp-schedules-25,c2b87e26fe766164,5078578ca2a28b7b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:20:33.751 INFO 21062 --- [ocp-schedules-25,c2b87e26fe766164,5078578ca2a28b7b] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:20:35.417 INFO 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:20:35.417 INFO 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:20:35.417 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:20:35.417 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:20:35.427 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:20:35.427 INFO 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:20:35.428 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:20:35.429 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:20:35.429 WARN 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:20:35.429 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:20:35.430 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:20:35.430 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:20:35.430 INFO 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:20:35.431 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:20:35.431 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:20:35.431 WARN 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:20:35.438 INFO 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:20:35.438 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:20:35.439 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:20:35.451 ERROR 21062 --- [ocp-schedules-13,528e8d6f33c8c485,e8ec700dda008c0b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:20:35.451 INFO 21062 --- [ocp-schedules-23,e79fa7620db03f58,170cfd77f95350bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:20:35.541 WARN 21062 --- [ocp-async-1,6a6ff42722fdc995,7577c8480bb6f03f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:35.545 INFO 21062 --- [ocp-async-5,f6c6393e3501fac1,288530f5e2cb4a0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:35.983 INFO 21062 --- [prometheus-minute-schedule-1,56522418891fc4fb,1386aa7c83f2921c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:35.994 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:35.995 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:37.541 WARN 21062 --- [ocp-async-7,c7c96b849a5cad59,17b91fe3a0944702] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:37.545 INFO 21062 --- [ocp-async-2,2e554531cecff845,3381b7f5e8b6ac87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:38.504 INFO 21062 --- [ocp-schedules-22,ccc98e7aaab92a04,76959e1325dfd0a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:20:38.504 INFO 21062 --- [ocp-schedules-29,de2bbcc7241455aa,8cbb5896ccb0b17c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:20:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:20:38.589564+08:00, lastProcessTime=2024-11-04T15:20:28.590121+08:00 2024-11-04 15:20:38.757 INFO 21062 --- [ocp-async-6,3e727cf04b923331,f5906a5cde51126d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:20:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:20:38.589564+08:00 2024-11-04 15:20:38.983 INFO 21062 --- [prometheus-minute-schedule-1,02e697a3dbc540e8,f4bf44d220dd91b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:38.995 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:38.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:39.542 WARN 21062 --- [ocp-async-6,8bf073e212203dfb,65908c102acb897b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:39.546 INFO 21062 --- [ocp-async-1,5d4d2d03580719f8,976b4abf778b5e32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:20:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:20:40.014 INFO 21062 --- [ocp-schedules-12,141dddd04a5b536e,5080bc6c4e989f71] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:20:40.014 INFO 21062 --- [ocp-schedules-12,b098f7d9cea590e7,a982ef2280768e7d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:20:40.019 INFO 21062 --- [ocp-schedules-12,b098f7d9cea590e7,a982ef2280768e7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:40.047 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:20:40.047 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:20:41.542 WARN 21062 --- [ocp-async-4,568daa5923383d6f,e0098559130fc94b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:41.546 INFO 21062 --- [ocp-async-7,92d9018f0f92cacb,81dc73f7688a449d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:41.548 INFO 21062 --- [ocp-schedules-27,46bf34335142493a,80c396ceeb9e5911] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:20:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:20:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:20:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:20:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:20:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:20:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:20:41.983 INFO 21062 --- [prometheus-minute-schedule-1,82e9c5bc3f1d654f,242708d4f6386960] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:41.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:41.996 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:43.542 WARN 21062 --- [ocp-async-9,3a7e3dceb01ba7ab,048f2259356384b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:43.546 INFO 21062 --- [ocp-async-8,7918f35daa025710,39b326f233acb395] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:44.053 INFO 21062 --- [ocp-schedules-20,6658632e3aaf741e,bf6329f39754f04c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:20:44.057 INFO 21062 --- [ocp-schedules-20,6658632e3aaf741e,bf6329f39754f04c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:44.144 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:20:44.147 ERROR 21062 --- [ocp-schedules-20,6658632e3aaf741e,bf6329f39754f04c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:20:44.147 INFO 21062 --- [ocp-schedules-20,6658632e3aaf741e,bf6329f39754f04c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:20:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9907129a21e24ed7,80cc5cbf2b0db247] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:44.997 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:44.997 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:45.477 INFO 21062 --- [ocp-schedules-4,fa7ea671e09a466d,055041752f04ad5b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:20:45.542 WARN 21062 --- [ocp-async-3,8130969d59486e3b,7f486abc5ccf6c7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:45.546 INFO 21062 --- [ocp-async-10,43ca586b8429f963,683e2c254b01a006] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:47.542 WARN 21062 --- [ocp-async-1,5c46ebef95bc56a5,3c6ccbf73a9a7af3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:47.546 INFO 21062 --- [ocp-async-5,266c4a61e58dbca7,38bf9149f508c8a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:47.983 INFO 21062 --- [prometheus-minute-schedule-1,95b7b7e5f6a2a3d3,f6cf93fdef26517e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:47.998 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:47.998 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:48.593 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:20:48.593017+08:00, lastProcessTime=2024-11-04T15:20:38.589564+08:00 2024-11-04 15:20:48.757 INFO 21062 --- [ocp-async-2,6e1e7b66f5440a90,2b84b1bad90d62cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:20:48.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:20:48.593017+08:00 2024-11-04 15:20:49.542 WARN 21062 --- [ocp-async-9,5e042ede95a44956,88676610752b6b28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:49.546 INFO 21062 --- [ocp-async-8,adb7249ca6dcda36,f312913601ee5280] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d484fea522677bdb,6416930de157a999] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:50.999 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:50.999 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:51.543 WARN 21062 --- [ocp-async-3,4ff7a1e0e6c146cc,42b50bf296966cfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:51.547 INFO 21062 --- [ocp-async-10,7062170f0c61f0de,85307a43fc4d9dd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:51.550 INFO 21062 --- [ocp-schedules-27,e6649b700498dadb,e5e222e7fa6348f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:20:52.914 INFO 21062 --- [ocp-schedules-19,732615ecacd30565,f7195a8a73be234d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:20:52.919 INFO 21062 --- [ocp-schedules-19,732615ecacd30565,f7195a8a73be234d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:20:52.919 INFO 21062 --- [ocp-schedules-19,732615ecacd30565,f7195a8a73be234d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:20:53.543 WARN 21062 --- [ocp-async-1,dcb757f649dfad03,88d73a757461130f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:53.547 INFO 21062 --- [ocp-async-5,81796eb00bcf6705,0e6b6c56067a0c2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:53.983 INFO 21062 --- [prometheus-minute-schedule-1,0b440886b3411aa4,a2bdab98c42c4ea8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:53.999 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:54.000 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:55.517 INFO 21062 --- [ocp-schedules-18,1bbfd814ebe6e95a,3b1c4426d4e38749] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:20:55.543 WARN 21062 --- [ocp-async-7,efde509a4e1b6456,eee6ed3427e1735f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:55.547 INFO 21062 --- [ocp-async-2,4925c3655071361f,a0724f60c7a36358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:56.983 INFO 21062 --- [prometheus-minute-schedule-1,69fc6f35048042cc,a21872038164fc1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:20:57.000 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:20:57.001 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:20:57.543 WARN 21062 --- [ocp-async-8,ea1b7d3cef75a7c1,462df077d4e465ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:57.547 INFO 21062 --- [ocp-async-6,c2556cd29bc6510a,0af67ecccd1abec6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:58.500 INFO 21062 --- [ocp-schedules-1,1d4add57c011fff6,4a65337cd0e678b8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:20:58.500 INFO 21062 --- [ocp-schedules-9,9b666e0e7305b052,7c96f7c850da41fd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:20:58.502 INFO 21062 --- [ocp-schedules-23,3fcddfb51529cdce,b985f1a84bdc65ab] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:20:58.503 INFO 21062 --- [ocp-schedules-1,1d4add57c011fff6,4a65337cd0e678b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:58.503 INFO 21062 --- [ocp-schedules-9,9b666e0e7305b052,7c96f7c850da41fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:20:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:20:58.505 INFO 21062 --- [ocp-schedules-15,c468f8bf13214a2c,ab0f6bf38e02d65a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:20:58.511 INFO 21062 --- [ocp-schedules-2,88e8001c497b7ddf,09dba69b0742ea15] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:20:58.511 INFO 21062 --- [ocp-schedules-5,3b1d081d676eb577,96097f54d03a6c06] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:20:58.511 INFO 21062 --- [ocp-schedules-5,3b1d081d676eb577,96097f54d03a6c06] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:20:58.511 INFO 21062 --- [ocp-schedules-2,88e8001c497b7ddf,09dba69b0742ea15] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:20:58.511 INFO 21062 --- [ocp-schedules-21,fb3f69fdb187d6a4,166cc9c197ca72aa] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:20:58.511 INFO 21062 --- [ocp-schedules-21,fb3f69fdb187d6a4,166cc9c197ca72aa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:20:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:20:58.588854+08:00, lastProcessTime=2024-11-04T15:20:48.593017+08:00 2024-11-04 15:20:58.757 INFO 21062 --- [ocp-async-4,582d373864cf2877,390f881d608dfa62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:20:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:20:58.588854+08:00 2024-11-04 15:20:59.493 INFO 21062 --- [ocp-schedules-24,4f941bb2787407ef,5493ae7f9c401048] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:20:59.543 WARN 21062 --- [ocp-async-10,c31925ae58747861,b3ab0af629bd243f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:20:59.547 INFO 21062 --- [ocp-async-5,965fbeb98fdc983d,74440c89cdd32266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:20:59.983 INFO 21062 --- [prometheus-minute-schedule-1,85a3618961725981,60d5a45c93c5ec28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704860, interval=60 2024-11-04 15:21:00.001 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:00.002 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:01.543 WARN 21062 --- [ocp-async-5,0458ec069fe0e463,d7a00aaddf664e04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:01.547 INFO 21062 --- [ocp-async-9,33a8a9393f6a20c7,3f9d40a308b7a5fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:01.553 INFO 21062 --- [ocp-schedules-23,92b9926ceaf48c6a,84cc6bff7b6ddfad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:21:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d08e23bb2ed0bc3d,d5f6dcb80c36290e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:03.002 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:03.003 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:03.544 WARN 21062 --- [ocp-async-9,ddf95387292065ce,c6d7f6f569602d7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:03.548 INFO 21062 --- [ocp-async-3,f69011fadc4860c3,eac31acf0991c613] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:05.544 INFO 21062 --- [ocp-schedules-13,cdf87c21447f7c62,71d923d53e75467f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:21:05.544 WARN 21062 --- [ocp-async-3,a8076647726d6265,ab51076ac85d0671] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:05.548 INFO 21062 --- [ocp-async-4,6d264996bdd201e0,3b0fab1ad57b44a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:05.983 INFO 21062 --- [prometheus-minute-schedule-1,974c5bdb096f15ad,d0718a3102ebb731] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:06.003 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:06.004 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:07.544 WARN 21062 --- [ocp-async-4,b6350a7d3b95278e,79a5b7cb7d4a1305] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:07.548 INFO 21062 --- [ocp-async-2,bef7417bfcbf27eb,71050fb093c89047] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:08.504 INFO 21062 --- [ocp-schedules-9,9b666e0e7305b052,7c96f7c850da41fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:21:08.505 INFO 21062 --- [ocp-schedules-1,1d4add57c011fff6,4a65337cd0e678b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:21:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:21:08.589013+08:00, lastProcessTime=2024-11-04T15:20:58.588854+08:00 2024-11-04 15:21:08.757 INFO 21062 --- [ocp-async-8,e3c48b80b79c5e72,1e10b3f02aad1f54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:21:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:21:08.589013+08:00 2024-11-04 15:21:08.983 INFO 21062 --- [prometheus-minute-schedule-1,655b83820f0bf4d7,6cc26f1cc39ae036] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:09.004 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:09.004 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:09.544 WARN 21062 --- [ocp-async-6,2dfcbc7064e63c8c,1e099c9b8db63599] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:09.548 INFO 21062 --- [ocp-async-8,6ffaedeb27a4ede9,d8bbeebf4ddc3c13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:21:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:21:10.047 INFO 21062 --- [ocp-schedules-12,b098f7d9cea590e7,a982ef2280768e7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:21:10.047 INFO 21062 --- [ocp-schedules-12,9fe345ac42c42ffe,fcaf021ddc366580] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:21:10.051 INFO 21062 --- [ocp-schedules-12,9fe345ac42c42ffe,fcaf021ddc366580] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:10.078 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:21:10.078 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:21:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:21:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:21:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:21:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:21:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:21:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:21:11.544 WARN 21062 --- [ocp-async-1,0dbfd2cb0b77a79b,f01a3f31f48e6d33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:11.548 INFO 21062 --- [ocp-async-10,7498e4bac0a2681f,236f8088b444dcc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:11.555 INFO 21062 --- [ocp-schedules-4,908049f6267facac,458be463da2c6616] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:21:11.813 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704800, nodeCount=120, affectRows=120 2024-11-04 15:21:11.814 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704800, nodeCount=163, affectRows=163 2024-11-04 15:21:11.814 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704800, nodeCount=139, affectRows=139 2024-11-04 15:21:11.814 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704800, nodeCount=120, validCount=120 2024-11-04 15:21:11.815 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704800, nodeCount=162, validCount=162 2024-11-04 15:21:11.815 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704800, nodeCount=137, validCount=137 2024-11-04 15:21:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4283346f6bdda84e,b71ee3ecf7049333] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:12.005 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:12.005 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:12.922 INFO 21062 --- [ocp-schedules-18,0812c70d7127f034,89e430ea406a14e0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:21:12.928 INFO 21062 --- [ocp-schedules-18,0812c70d7127f034,89e430ea406a14e0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:21:12.928 INFO 21062 --- [ocp-schedules-18,0812c70d7127f034,89e430ea406a14e0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:21:13.545 WARN 21062 --- [ocp-async-5,d355a957745ba9f6,aadc7f80054e5d2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:13.549 INFO 21062 --- [ocp-async-2,1e8a1fb7d8a2821f,cef1b0c602346b3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:13.984 INFO 21062 --- [prometheus-cache-monitor1,33d595640c7438be,c6d2d82db551c73f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:21:13.984 INFO 21062 --- [prometheus-cache-monitor1,33d595640c7438be,c6d2d82db551c73f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:21:14.983 INFO 21062 --- [prometheus-minute-schedule-1,064b704fa1f98556,ee0da58e82cf7ce1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:15.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:15.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:15.545 WARN 21062 --- [ocp-async-6,5c361b5bbe90c068,23270e45e2de59c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:15.549 INFO 21062 --- [ocp-async-9,c788b4d222d4cdda,f5f4ac07b2f213c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:15.583 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:17:15.563, end=2024-11-04T15:19:15.581} 2024-11-04 15:21:15.584 INFO 21062 --- [ocp-schedules-18,57caa97df2029fe4,10e0bbc875472ba4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:21:15.597 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:21:15.597 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:21:17.545 WARN 21062 --- [ocp-async-8,cf44def01acdaf3e,42677a60d5fff043] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:17.549 INFO 21062 --- [ocp-async-3,cf710922eace458c,8099616708c68cf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7c528d10d8ed1a99,dad699a493d60ce0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:18.006 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:18.007 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:21:18.589252+08:00, lastProcessTime=2024-11-04T15:21:08.589013+08:00 2024-11-04 15:21:18.757 INFO 21062 --- [ocp-async-4,930e9dbbfbcde2d7,e52ace3fdb6c6a96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:21:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:21:18.589252+08:00 2024-11-04 15:21:19.545 WARN 21062 --- [ocp-async-5,bdd6367610f3693b,6a082b7ce68213a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:19.549 INFO 21062 --- [ocp-async-2,3fe49046a1fac45c,a19800434dbde4cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:20.983 INFO 21062 --- [prometheus-minute-schedule-1,d137638485d3993d,7efda9fc43764bb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:21.007 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:21.008 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:21.545 WARN 21062 --- [ocp-async-6,17c5682bb5dde5f8,bd3a541eec66ce42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:21.549 INFO 21062 --- [ocp-async-9,31121dd9c1cd47c4,5d426baf17523252] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:21.558 INFO 21062 --- [ocp-schedules-20,5e9873e534ac9806,7788509b688c9a2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:21:23.545 WARN 21062 --- [ocp-async-8,f097e9866cbe88ab,1291f81b5c5f02f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:23.549 INFO 21062 --- [ocp-async-3,e6c955e4e2f0cca4,c5f87bc80b0c2538] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:23.983 INFO 21062 --- [prometheus-minute-schedule-1,de0682b07e37ed47,accf346e7586ae47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:24.008 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:24.009 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:25.546 WARN 21062 --- [ocp-async-10,ba4897805718b1b9,01178b8acff22dec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:25.550 INFO 21062 --- [ocp-async-4,348672a299912474,498cc418e54ac583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:25.612 INFO 21062 --- [ocp-schedules-26,381efcdc61d5f6ac,882b4c80fdb52f73] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:21:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3d5258e97df84800,fecb50356b37a53e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4773d4eac3f8ba3f,354cc89e15f001f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:21:27.009 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:27.010 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:27.546 WARN 21062 --- [ocp-async-6,cb3dc6b64c8c6a56,4c7d206d01953834] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:27.550 INFO 21062 --- [ocp-async-7,f63821de1daeda6d,a63f662aa424b94e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:28.498 INFO 21062 --- [ocp-schedules-2,85cf18614fd1dd98,577c2230e2573a2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:21:28.499 INFO 21062 --- [ocp-schedules-24,4a406f36bf7a6cc0,eba79707683c44a5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:21:28.500 INFO 21062 --- [ocp-schedules-11,c1975c9609892dbc,67a1c9b809f1143f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:21:28.500 INFO 21062 --- [ocp-schedules-19,97cdea915a4fa115,920edfb52a4326c5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:21:28.502 INFO 21062 --- [ocp-schedules-2,85cf18614fd1dd98,577c2230e2573a2e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:21:28.502 INFO 21062 --- [ocp-schedules-6,3879a1dc54213d87,f51592a16237ebee] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:21:28.504 INFO 21062 --- [ocp-schedules-11,c1975c9609892dbc,67a1c9b809f1143f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:28.504 INFO 21062 --- [ocp-schedules-19,97cdea915a4fa115,920edfb52a4326c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:21:28.505 INFO 21062 --- [ocp-schedules-24,4a406f36bf7a6cc0,eba79707683c44a5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:21:28.505 INFO 21062 --- [ocp-schedules-22,1e4690ea4139e8a7,e9f7343ecbfac95e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:21:28.510 INFO 21062 --- [ocp-schedules-9,1f1ef70f15db12f3,1625c93827091c10] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:21:28.510 INFO 21062 --- [ocp-schedules-9,1f1ef70f15db12f3,1625c93827091c10] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:21:28.511 INFO 21062 --- [ocp-schedules-21,8b06ac7aefddbebe,53e417f6b90b2c7b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:21:28.511 INFO 21062 --- [ocp-schedules-21,8b06ac7aefddbebe,53e417f6b90b2c7b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:21:28.511 INFO 21062 --- [ocp-schedules-23,47f05919c6b0b51b,993b6c1daeb749be] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:21:28.511 INFO 21062 --- [ocp-schedules-23,47f05919c6b0b51b,993b6c1daeb749be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:21:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:21:28.590134+08:00, lastProcessTime=2024-11-04T15:21:18.589252+08:00 2024-11-04 15:21:28.663 INFO 21062 --- [ocp-schedules-18,c1fb8ed977dd1a03,3f06b32cec6df69e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:21:28.663 INFO 21062 --- [ocp-schedules-18,c1fb8ed977dd1a03,3f06b32cec6df69e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:21:28.757 INFO 21062 --- [ocp-async-10,5f77a53cadecc373,116b94f830904219] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:21:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:21:28.590134+08:00 2024-11-04 15:21:28.977 INFO 21062 --- [ocp-schedules-27,c87619350e1331b2,40ef8855fc596fcd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:21:29.297 INFO 21062 --- [ocp-schedules-6,2455b9a45d9ef9e4,f99737aa286575b6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:21:29.302 INFO 21062 --- [ocp-schedules-6,2455b9a45d9ef9e4,f99737aa286575b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:29.302 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:21:29.362 INFO 21062 --- [ocp-schedules-21,a78bedc979e29122,9d79c51d80d6a654] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:21:29.362 INFO 21062 --- [ocp-schedules-21,a78bedc979e29122,9d79c51d80d6a654] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:21:29.495 INFO 21062 --- [ocp-schedules-2,f00dc67e1ff4f9c5,1867852f5aadc25b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:21:29.517 INFO 21062 --- [ocp-schedules-30,314c596d58db63fa,0ac55d27965bb327] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:21:29.522 INFO 21062 --- [ocp-schedules-30,314c596d58db63fa,0ac55d27965bb327] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:21:29.522 INFO 21062 --- [ocp-schedules-30,314c596d58db63fa,0ac55d27965bb327] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:21:29.546 WARN 21062 --- [ocp-async-10,4b9ce66dd16a4e75,c12482e155b7e48d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:29.550 INFO 21062 --- [ocp-async-5,6bc12d16944afffb,cf8e06675c61f95a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4bf610e1bccf7c84,f6e5eb05ea013915] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:29.987 INFO 21062 --- [ocp-schedules-25,a56c2b07c22c2914,fd5395c12a1c1f63] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:21:29.987 INFO 21062 --- [ocp-schedules-25,a56c2b07c22c2914,fd5395c12a1c1f63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:21:29.990 INFO 21062 --- [ocp-schedules-25,a56c2b07c22c2914,fd5395c12a1c1f63] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:21:29.990 INFO 21062 --- [ocp-schedules-25,a56c2b07c22c2914,fd5395c12a1c1f63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:21:29.991 INFO 21062 --- [ocp-schedules-25,a56c2b07c22c2914,fd5395c12a1c1f63] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:21:29.991 INFO 21062 --- [ocp-schedules-25,a56c2b07c22c2914,fd5395c12a1c1f63] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:21:30.010 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:30.011 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:21:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:21:31.546 WARN 21062 --- [ocp-async-4,b2a25e5bafda274a,c63a7626b5046b10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:31.550 INFO 21062 --- [ocp-async-2,e2d8ee4115a70a5b,14dfd27dfea410ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:31.561 INFO 21062 --- [ocp-schedules-8,b406dd22b2c93be3,db6d387e8539982b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:21:32.931 INFO 21062 --- [ocp-schedules-9,2ad4bf6638844a2c,1331a175d99d0b6d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:21:32.937 INFO 21062 --- [ocp-schedules-9,2ad4bf6638844a2c,1331a175d99d0b6d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:21:32.937 INFO 21062 --- [ocp-schedules-9,2ad4bf6638844a2c,1331a175d99d0b6d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:21:32.983 INFO 21062 --- [prometheus-minute-schedule-1,432e42bc4bd899d2,0c85a70c992a803f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:33.011 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:33.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:33.152 INFO 21062 --- [ocp-schedules-13,b8aaa1b04b8a608d,31d79d9d1242964a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:21:33.152 INFO 21062 --- [ocp-schedules-13,b8aaa1b04b8a608d,31d79d9d1242964a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:21:33.180 INFO 21062 --- [ocp-schedules-13,b8aaa1b04b8a608d,31d79d9d1242964a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:21:33.180 INFO 21062 --- [ocp-schedules-13,b8aaa1b04b8a608d,31d79d9d1242964a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:21:33.180 INFO 21062 --- [ocp-schedules-26,30e77080058d054e,be89e20851134554] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:21:33.385 INFO 21062 --- [ocp-schedules-27,7b61855bfbaf5beb,d3f93a613e722ee3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:21:33.385+08:00, clusters=[1] 2024-11-04 15:21:33.386 INFO 21062 --- [ocp-schedules-27,7b61855bfbaf5beb,d3f93a613e722ee3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:21:33.546 WARN 21062 --- [ocp-async-8,8bc2763271a966cb,5fe8201bca802bd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:33.550 INFO 21062 --- [ocp-async-9,155efb904fd5e1bc,6bc9432cade6ac5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:33.676 INFO 21062 --- [ocp-schedules-25,610298ed123570e3,f937ad2f5c924abb] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:21:33.676+08:00, clusters=[1] 2024-11-04 15:21:33.680 INFO 21062 --- [ocp-schedules-25,610298ed123570e3,f937ad2f5c924abb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:21:35.460 INFO 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:21:35.460 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:21:35.460 INFO 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:21:35.460 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:21:35.469 INFO 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:21:35.470 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:21:35.471 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:21:35.471 WARN 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:21:35.471 INFO 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:21:35.472 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:21:35.472 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:21:35.472 WARN 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:21:35.472 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:21:35.472 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:21:35.473 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:21:35.474 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:21:35.480 INFO 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:21:35.480 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:21:35.480 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:21:35.490 ERROR 21062 --- [ocp-schedules-21,85f45e908548020a,904c9b7c40459484] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:21:35.547 WARN 21062 --- [ocp-async-10,65efa665ced2e8c9,204602ffac784628] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:35.551 INFO 21062 --- [ocp-async-3,b9217105ab14fbf2,9cf40f2a5a269170] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:35.656 INFO 21062 --- [ocp-schedules-20,ba5dbb77b52940e5,a4efc810fb4cf9e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:21:35.983 INFO 21062 --- [prometheus-minute-schedule-1,168a4e17cc6a1e37,01cd24296b5835cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:36.012 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:36.013 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:37.547 WARN 21062 --- [ocp-async-4,a6bd4e3080f3af02,6baef21b3ea2b0b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:37.551 INFO 21062 --- [ocp-async-6,e2c48b823f0c9eb4,a05f725ac134a679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:38.505 INFO 21062 --- [ocp-schedules-19,97cdea915a4fa115,920edfb52a4326c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:21:38.505 INFO 21062 --- [ocp-schedules-11,c1975c9609892dbc,67a1c9b809f1143f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:21:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:21:38.588675+08:00, lastProcessTime=2024-11-04T15:21:28.590134+08:00 2024-11-04 15:21:38.758 INFO 21062 --- [ocp-async-9,f18aa96c986f504a,48115d93b8202b0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:21:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:21:38.588675+08:00 2024-11-04 15:21:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c7751e87bcae6d39,134fb3bf8006c0db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:39.013 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:39.014 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:39.303 INFO 21062 --- [ocp-schedules-6,2455b9a45d9ef9e4,f99737aa286575b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:21:39.547 WARN 21062 --- [ocp-async-9,90f098ff866be5be,bf91c5dd5db5b259] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:39.551 INFO 21062 --- [ocp-async-10,8dfe474c23a1e302,9d7bb18788b18f10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:40.004 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:21:40.005 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:21:40.079 INFO 21062 --- [ocp-schedules-12,9fe345ac42c42ffe,fcaf021ddc366580] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:21:40.079 INFO 21062 --- [ocp-schedules-12,9fd24f50174e5cca,10ca14ef128912eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:21:40.083 INFO 21062 --- [ocp-schedules-12,9fd24f50174e5cca,10ca14ef128912eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:40.112 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:21:40.113 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:21:41.547 WARN 21062 --- [ocp-async-3,e3608f37af7fca04,2a4a84a383419e7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:41.551 INFO 21062 --- [ocp-async-4,0192b942610fd478,a1eba5e676525bc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:41.563 INFO 21062 --- [ocp-schedules-9,d3f5f8a88592e3c4,a3adbf0778a2cf5f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:21:41.983 INFO 21062 --- [prometheus-minute-schedule-1,f4b31ce5ef3c0b4e,387c33583cb1d07c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:42.014 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:42.015 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:43.547 WARN 21062 --- [ocp-async-6,4a80d1519670d1b0,c42041232742b3a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:43.551 INFO 21062 --- [ocp-async-2,de4551507aeb1a2c,85c4dd42c605717e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:44.147 INFO 21062 --- [ocp-schedules-9,041aaf6fadff831e,8b82cdfabb839f42] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:21:44.152 INFO 21062 --- [ocp-schedules-9,041aaf6fadff831e,8b82cdfabb839f42] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:44.238 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:21:44.241 ERROR 21062 --- [ocp-schedules-9,041aaf6fadff831e,8b82cdfabb839f42] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:21:44.241 INFO 21062 --- [ocp-schedules-9,041aaf6fadff831e,8b82cdfabb839f42] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:21:44.983 INFO 21062 --- [prometheus-minute-schedule-1,ce402010b412b306,fcde07f1722f0599] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:45.015 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:45.015 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:45.547 WARN 21062 --- [ocp-async-9,6bd11e68a145984f,1ee59b324efedf00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:45.551 INFO 21062 --- [ocp-async-7,96d58396e79ff6ec,796255cd5b2f58ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:45.681 INFO 21062 --- [ocp-schedules-8,947f9ecf1e23f8ae,d54d4a194c801060] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:21:47.548 WARN 21062 --- [ocp-async-10,e8853ec1529e4149,106a7beb04d04d2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:47.552 INFO 21062 --- [ocp-async-1,5b7f437d710fc9be,a6dfde5d524dbb72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:47.983 INFO 21062 --- [prometheus-minute-schedule-1,f98709ad8ac720ec,da4141314060126f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:48.016 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:48.016 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:21:48.590262+08:00, lastProcessTime=2024-11-04T15:21:38.588675+08:00 2024-11-04 15:21:48.758 INFO 21062 --- [ocp-async-5,21405e45d241184c,66e3de606745a9da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:21:48.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:21:48.590262+08:00 2024-11-04 15:21:49.548 WARN 21062 --- [ocp-async-5,cd344c33453e1319,a23bbe7c7cdd0e3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:49.552 INFO 21062 --- [ocp-async-2,2ab3c4bb68924427,f1c69bdaf8252123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8a9ddc3086a53cb9,a84f91d7cee38f83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:51.017 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:51.017 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:51.548 WARN 21062 --- [ocp-async-9,f5962b02332b844d,2c39f32db6cda1e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:51.552 INFO 21062 --- [ocp-async-7,0fb52dec01840615,8c67dd0c3a4a05cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:51.566 INFO 21062 --- [ocp-schedules-10,040a7091e698c91a,535767e00ddd80bf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:21:52.940 INFO 21062 --- [ocp-schedules-22,d3a086f630968188,dcf31ef2958e3fab] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:21:52.946 INFO 21062 --- [ocp-schedules-22,d3a086f630968188,dcf31ef2958e3fab] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:21:52.947 INFO 21062 --- [ocp-schedules-22,d3a086f630968188,dcf31ef2958e3fab] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:21:53.548 WARN 21062 --- [ocp-async-10,1fd118b4baeac3e5,da9186a50386a12e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:53.552 INFO 21062 --- [ocp-async-1,3f7adf56cb18c0ac,7aa7027eb17c18c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1129a58a49667543,86910f775e7d97f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:54.018 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:54.018 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:55.548 WARN 21062 --- [ocp-async-4,c39d3d4dda705bb2,952390ea76240748] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:55.552 INFO 21062 --- [ocp-async-6,1275e57392246de7,ac06ae024619a2d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:55.720 INFO 21062 --- [ocp-schedules-17,d1a283dace36d9cd,c3d0531b0f140837] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:21:56.983 INFO 21062 --- [prometheus-minute-schedule-1,b72c9438098a8d05,0d03bfa12e1f2ba6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:21:57.018 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:21:57.019 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:21:57.549 WARN 21062 --- [ocp-async-9,9a682b15aed9edf0,ce9d241a8b47d3a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:57.553 INFO 21062 --- [ocp-async-8,c647dcf979d407a1,2817eca34f505d9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:58.500 INFO 21062 --- [ocp-schedules-28,1e6544b4d7498c97,decf004ae99e2f35] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:21:58.500 INFO 21062 --- [ocp-schedules-20,c2f9b78932dc7c11,f1b2d541669a75ce] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:21:58.502 INFO 21062 --- [ocp-schedules-10,477fa76a8db096bf,f6db1d7e285cb9ae] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:21:58.503 INFO 21062 --- [ocp-schedules-28,1e6544b4d7498c97,decf004ae99e2f35] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:58.504 INFO 21062 --- [ocp-schedules-20,c2f9b78932dc7c11,f1b2d541669a75ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:21:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:21:58.504 INFO 21062 --- [ocp-schedules-27,db9f5a638a5ddde7,0522184d5b7c8d42] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:21:58.511 INFO 21062 --- [ocp-schedules-16,04a38a13f99ba107,24392536d7b0bb5d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:21:58.511 INFO 21062 --- [ocp-schedules-6,bc8e752fb3a1e8f9,dab86e39e7c6b396] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:21:58.511 INFO 21062 --- [ocp-schedules-16,04a38a13f99ba107,24392536d7b0bb5d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:21:58.511 INFO 21062 --- [ocp-schedules-6,bc8e752fb3a1e8f9,dab86e39e7c6b396] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:21:58.511 INFO 21062 --- [ocp-schedules-15,2a787650c37883c7,79667c4b8e194361] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:21:58.511 INFO 21062 --- [ocp-schedules-15,2a787650c37883c7,79667c4b8e194361] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:21:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:21:58.588415+08:00, lastProcessTime=2024-11-04T15:21:48.590262+08:00 2024-11-04 15:21:58.758 INFO 21062 --- [ocp-async-3,c1f2ce2f3186173f,23a5743dec74a465] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:21:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:21:58.588415+08:00 2024-11-04 15:21:59.498 INFO 21062 --- [ocp-schedules-14,1ca17b811d2b6f66,8aad5a3b3e2f17e4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:21:59.549 WARN 21062 --- [ocp-async-10,98ace31e8d0d4084,80eb4d018d7333b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:21:59.553 INFO 21062 --- [ocp-async-1,e2c4701916a78134,312c159933073760] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:21:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f092f6325a2f7aeb,0ea9d57b185a354b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704920, interval=60 2024-11-04 15:22:00.019 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:00.020 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:01.549 WARN 21062 --- [ocp-async-1,e8098db383dc9ce7,fabdf2719f9bb512] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:01.553 INFO 21062 --- [ocp-async-6,e95e74c3ba542eac,66cec4f850f0212f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:01.569 INFO 21062 --- [ocp-schedules-4,81518eb332fb619e,c97b32aab6073d19] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:22:02.983 INFO 21062 --- [prometheus-minute-schedule-1,fb084aa193712bea,32e3d712fae9b260] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:03.020 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:03.021 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:03.549 WARN 21062 --- [ocp-async-9,3a7cc9853266d666,27a1e91ee14665a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:03.553 INFO 21062 --- [ocp-async-2,2b9535ab35c14f5f,a03750c7b75d6439] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:05.549 WARN 21062 --- [ocp-async-2,539af653defcef2a,1f4dcd80dc897b37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:05.553 INFO 21062 --- [ocp-async-10,eaf8903d423090a3,0bacd0876b080864] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:05.744 INFO 21062 --- [ocp-schedules-27,3178c9209c2e9b17,865c1760f9fba9a6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:22:05.983 INFO 21062 --- [prometheus-minute-schedule-1,e0d549f070058c28,631737840babe852] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:06.021 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:06.022 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:07.550 WARN 21062 --- [ocp-async-1,4632970cf538eba7,741cd6943ca6c99e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:07.553 INFO 21062 --- [ocp-async-4,f66861f824f9234a,072cc30cc01bea74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:08.505 INFO 21062 --- [ocp-schedules-28,1e6544b4d7498c97,decf004ae99e2f35] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:22:08.505 INFO 21062 --- [ocp-schedules-20,c2f9b78932dc7c11,f1b2d541669a75ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:22:08.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:22:08.591593+08:00, lastProcessTime=2024-11-04T15:21:58.588415+08:00 2024-11-04 15:22:08.758 INFO 21062 --- [ocp-async-6,e064ddf77dd98dc2,262b2930197abcd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:22:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:22:08.591593+08:00 2024-11-04 15:22:08.983 INFO 21062 --- [prometheus-minute-schedule-1,eda20b72a41c300c,9e263ec333af2034] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:09.022 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:09.023 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:09.550 WARN 21062 --- [ocp-async-8,3375daaafe444eef,9cc6bc043dd56593] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:09.553 INFO 21062 --- [ocp-async-6,bd4f4aecfa7b10a1,86e0e813530e0472] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:10.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:22:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:22:10.113 INFO 21062 --- [ocp-schedules-12,9fd24f50174e5cca,10ca14ef128912eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:22:10.113 INFO 21062 --- [ocp-schedules-12,1a5f3944f71ccd7f,cce3daaab5da5141] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:22:10.117 INFO 21062 --- [ocp-schedules-12,1a5f3944f71ccd7f,cce3daaab5da5141] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:10.148 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:22:10.148 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:22:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:22:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:22:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:22:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:22:10.325 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:22:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:22:11.550 WARN 21062 --- [ocp-async-7,a171fa23c66f0ee1,b82a15b7f3c73b6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:11.554 INFO 21062 --- [ocp-async-3,22dae1230eccb78f,b3dc0cd2ea9ca95d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:11.571 INFO 21062 --- [ocp-schedules-18,470155b2089789bf,bc94c8bcc313cf9f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:22:11.817 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=1, affectRows=1 2024-11-04 15:22:11.817 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=1, validCount=1 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=9, affectRows=9 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=9, validCount=9 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=11, affectRows=11 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=39, affectRows=39 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=11, validCount=11 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=34, affectRows=34 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=46, affectRows=46 2024-11-04 15:22:11.818 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=39, validCount=39 2024-11-04 15:22:11.819 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=34, validCount=34 2024-11-04 15:22:11.819 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=46, validCount=46 2024-11-04 15:22:11.819 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=49, affectRows=49 2024-11-04 15:22:11.819 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=34, affectRows=34 2024-11-04 15:22:11.819 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=34, validCount=34 2024-11-04 15:22:11.819 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=49, validCount=49 2024-11-04 15:22:11.821 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=25, affectRows=25 2024-11-04 15:22:11.821 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=25, validCount=25 2024-11-04 15:22:11.821 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704860, nodeCount=174, affectRows=174 2024-11-04 15:22:11.822 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704860, nodeCount=171, validCount=171 2024-11-04 15:22:11.983 INFO 21062 --- [prometheus-minute-schedule-1,10e976629aadbb98,72f2b8d77ba96286] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:12.023 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:12.024 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:12.949 INFO 21062 --- [ocp-schedules-27,5f23c65260170dcb,ffb855e4df4966b3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:22:12.954 INFO 21062 --- [ocp-schedules-27,5f23c65260170dcb,ffb855e4df4966b3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:22:12.954 INFO 21062 --- [ocp-schedules-27,5f23c65260170dcb,ffb855e4df4966b3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:22:13.550 WARN 21062 --- [ocp-async-10,735ad53dc36170b7,e60f66cd4381d0a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:13.554 INFO 21062 --- [ocp-async-5,6c86f8efaf562d7e,dd32ac92cf2c30d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:13.984 INFO 21062 --- [prometheus-cache-monitor1,78282646dc0076c0,7dc268aece7788fa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:22:13.985 INFO 21062 --- [prometheus-cache-monitor1,78282646dc0076c0,7dc268aece7788fa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:22:14.983 INFO 21062 --- [prometheus-minute-schedule-1,73789d093be38dca,f358a372fa5d9b60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:15.024 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:15.025 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:15.550 WARN 21062 --- [ocp-async-8,338f591c95eb4923,4774e02faaeaff10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:15.554 INFO 21062 --- [ocp-async-9,77dc9d9ff0276579,8aad9fc3a622beab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:15.787 INFO 21062 --- [ocp-schedules-27,9204a6c8fe19d63b,127fbda873ff48c1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:22:17.550 WARN 21062 --- [ocp-async-6,9c17707965185de3,0d4c2408528314cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:17.554 INFO 21062 --- [ocp-async-2,34433e4fd4f3c220,fcefd1d3e9a9eec4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:17.983 INFO 21062 --- [prometheus-minute-schedule-1,c4f83d23c0d779b6,f14476f3f43f9d8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:18.025 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:18.026 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:22:18.590351+08:00, lastProcessTime=2024-11-04T15:22:08.591593+08:00 2024-11-04 15:22:18.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:22:18.590351+08:00 2024-11-04 15:22:18.758 INFO 21062 --- [ocp-async-1,8f978402f9e6be6a,f134a438fab7d3fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:22:19.551 WARN 21062 --- [ocp-async-10,432dc402d02a9fe6,5e29cc7829202946] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:19.554 INFO 21062 --- [ocp-async-5,847aa8d2a547a252,e75f2b42b1e6e9e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:20.935 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:22:20.935 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:22:20.944 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:22:20.944 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:22:20.944 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:22:20.944 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:22:20.949 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:22:20.949 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:22:20.949 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:22:20.949 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:22:20.983 INFO 21062 --- [prometheus-minute-schedule-1,24a851d2b1ee9ef8,b75b8b38d3414682] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:20.985 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:22:20.993 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:22:20.993 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:22:20.994 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:22:20.994 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:22:20.994 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:22:21.002 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:22:21.026 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:21.027 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:21.533 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:22:21.551 WARN 21062 --- [ocp-async-4,0661062af0db0dbe,3c05616eea2540ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:21.555 INFO 21062 --- [ocp-async-9,cbdd0433f5782504,a117ae2314e28d43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:21.574 INFO 21062 --- [ocp-schedules-9,00f1c2df5fad351f,6318dc9028693c80] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:22:22.132 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.183267982S 2024-11-04 15:22:22.133 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:22:22.231 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:22:22.231 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:22:22.256 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:22:22.319 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:22:22.319 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:22:22.319 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:22:22.319 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:22:22.319 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:22:22.319 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:22:22.330 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:22:22.508 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] o.s.boot.SpringApplication : Started application in 1.564 seconds (JVM running for 10356.188) 2024-11-04 15:22:22.509 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.565174011S 2024-11-04 15:22:22.982 INFO 21062 --- [ocp-schedules-22,2fdc1d70c84d54bb,a5076e3ea8bfceb3] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:22:23.551 WARN 21062 --- [ocp-async-7,cf9a8f6d1eccc0d6,95eef96fcc00a0ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:23.555 INFO 21062 --- [ocp-async-2,e7c7e7e667b79e42,77198fc04f95f8a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:23.983 INFO 21062 --- [prometheus-minute-schedule-1,07c4c2074a5c5ee5,6d33fddff8f65feb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:24.027 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:24.028 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:25.551 WARN 21062 --- [ocp-async-3,133eb4b7d69aeb2c,c99321fa1e1030d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:25.555 INFO 21062 --- [ocp-async-1,46097698219a2a5b,87db55dac5b510f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:25.813 INFO 21062 --- [ocp-schedules-11,ca1802a1031bed09,682254613cc95500] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:22:26.983 INFO 21062 --- [prometheus-minute-schedule-1,483436bf074e26ae,b6301f2e95d2b340] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:26.984 INFO 21062 --- [prometheus-hour-schedule-1,98b29762c60a691f,b76af9789363d1c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:22:27.029 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:27.029 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:27.551 WARN 21062 --- [ocp-async-5,72a80195b590388c,40d23e712cf86866] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:27.555 INFO 21062 --- [ocp-async-8,217912b8a8cdef85,4ba9e05705f8a42e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:28.498 INFO 21062 --- [ocp-schedules-16,9160c085f2d08b6b,244b03f27f2a71b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:22:28.500 INFO 21062 --- [ocp-schedules-17,02dc6080f84a9de7,e8dc40e044828c47] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:22:28.500 INFO 21062 --- [ocp-schedules-6,9edbbcf27477af1b,33db52108c727fb1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:22:28.500 INFO 21062 --- [ocp-schedules-13,c2fd8a6739abe694,22f904a9f6855060] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:22:28.502 INFO 21062 --- [ocp-schedules-16,9160c085f2d08b6b,244b03f27f2a71b8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:22:28.502 INFO 21062 --- [ocp-schedules-26,41d4eb028a46079d,ec8e43591412d04e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:22:28.503 INFO 21062 --- [ocp-schedules-17,02dc6080f84a9de7,e8dc40e044828c47] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:22:28.504 INFO 21062 --- [ocp-schedules-7,701acaaee1a2b6b3,74604eb5bf65c97b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:22:28.505 INFO 21062 --- [ocp-schedules-21,84596dfaac9c9b7f,3725a58b8cbe444f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:22:28.505 INFO 21062 --- [ocp-schedules-6,9edbbcf27477af1b,33db52108c727fb1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:22:28.506 INFO 21062 --- [ocp-schedules-13,c2fd8a6739abe694,22f904a9f6855060] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:28.508 INFO 21062 --- [ocp-schedules-21,84596dfaac9c9b7f,3725a58b8cbe444f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:28.510 INFO 21062 --- [ocp-schedules-2,fea70b849c66f1c8,039b36ab2062017c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:22:28.510 INFO 21062 --- [ocp-schedules-28,f812f8c39f50238f,e5c604e15cad9240] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:22:28.510 INFO 21062 --- [ocp-schedules-1,75220b1508e0e836,8ff63f5965240a4f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:22:28.510 INFO 21062 --- [ocp-schedules-2,fea70b849c66f1c8,039b36ab2062017c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:22:28.510 INFO 21062 --- [ocp-schedules-28,f812f8c39f50238f,e5c604e15cad9240] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:22:28.510 INFO 21062 --- [ocp-schedules-1,75220b1508e0e836,8ff63f5965240a4f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:22:28.541 INFO 21062 --- [ocp-schedules-3,59a320387d6ad59c,96132ade65e8fc7c] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:22:28.541 INFO 21062 --- [ocp-schedules-3,59a320387d6ad59c,96132ade65e8fc7c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:22:28.581 INFO 21062 --- [pool-12-thread-11,78bede315961958f,8168ec8b493f39c4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:22:28.581 INFO 21062 --- [pool-12-thread-10,78bede315961958f,ade0805ff922e5e3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:22:28.581 INFO 21062 --- [pool-12-thread-12,78bede315961958f,b8fe4356610287e6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:22:28.582 INFO 21062 --- [pool-12-thread-9,78bede315961958f,c2f719223700df90] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:22:28.587 INFO 21062 --- [ForkJoinPool-11-worker-4,78bede315961958f,8e154824e5fce196] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:22:28.587 INFO 21062 --- [pool-12-thread-11,78bede315961958f,8168ec8b493f39c4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:22:28.587 INFO 21062 --- [ForkJoinPool-11-worker-4,78bede315961958f,8e154824e5fce196] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:22:28.587 INFO 21062 --- [ForkJoinPool-11-worker-4,78bede315961958f,8e154824e5fce196] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:22:28.587 INFO 21062 --- [pool-12-thread-11,78bede315961958f,8168ec8b493f39c4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:22:28.587 INFO 21062 --- [pool-12-thread-9,78bede315961958f,c2f719223700df90] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:22:28.587 INFO 21062 --- [ocp-schedules-11,1777312863c5c175,15f4a2d66e4f04ea] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:22:28.587 INFO 21062 --- [ocp-schedules-11,1777312863c5c175,15f4a2d66e4f04ea] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:22:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:22:28.589965+08:00, lastProcessTime=2024-11-04T15:22:18.590351+08:00 2024-11-04 15:22:28.607 INFO 21062 --- [pool-12-thread-12,78bede315961958f,b8fe4356610287e6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:22:28.608 INFO 21062 --- [pool-12-thread-12,78bede315961958f,b8fe4356610287e6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:22:28.609 INFO 21062 --- [pool-12-thread-12,78bede315961958f,b8fe4356610287e6] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:22:28.612 INFO 21062 --- [ForkJoinPool-11-worker-4,78bede315961958f,d988a16b0499f702] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:22:28.612 INFO 21062 --- [pool-12-thread-10,78bede315961958f,ade0805ff922e5e3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:22:28.759 INFO 21062 --- [ocp-async-3,a12c873bcbe5b9c0,80900e7f8ec30dfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:22:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:22:28.589965+08:00 2024-11-04 15:22:28.963 INFO 21062 --- [ocp-schedules-1,0b8caabece36ea12,5bc0572a2a8bf424] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:22:28.963 INFO 21062 --- [ocp-schedules-1,0b8caabece36ea12,5bc0572a2a8bf424] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:22:28.979 INFO 21062 --- [ocp-schedules-7,b6a259c408a664e4,47b4e5d8244174eb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:22:28.991 INFO 21062 --- [ocp-schedules-16,c5db2776cb43bb8b,115d1b889165976c] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 15:22:28.991 INFO 21062 --- [ocp-schedules-16,c5db2776cb43bb8b,115d1b889165976c] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 15:22:28.991 INFO 21062 --- [ocp-schedules-16,c5db2776cb43bb8b,115d1b889165976c] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=422 2024-11-04 15:22:29.014 INFO 21062 --- [ocp-schedules-16,c5db2776cb43bb8b,115d1b889165976c] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=422 2024-11-04 15:22:29.308 INFO 21062 --- [ocp-schedules-3,a45cee9b4016f4d2,0c4ed4e3ff132197] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 15:22:29.311 INFO 21062 --- [ocp-schedules-3,a45cee9b4016f4d2,0c4ed4e3ff132197] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:29.350 ERROR 21062 --- [ocp-schedules-3,a45cee9b4016f4d2,0c4ed4e3ff132197] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.GeneratedMethodAccessor1314.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:22:29.350 INFO 21062 --- [ocp-schedules-3,a45cee9b4016f4d2,0c4ed4e3ff132197] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 15:22:29.367 INFO 21062 --- [ocp-schedules-11,83efc9b3dcb9ef56,efcdc13098ad2789] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:22:29.367 INFO 21062 --- [ocp-schedules-11,83efc9b3dcb9ef56,efcdc13098ad2789] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:22:29.459 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:22:29.477 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:22:29.478 WARN 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:22:29.479 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:22:29.481 WARN 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:22:29.488 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:22:29.489 WARN 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:22:29.490 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:22:29.491 WARN 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:22:29.497 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:22:29.498 WARN 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:22:29.499 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:22:29.500 WARN 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:22:29.501 INFO 21062 --- [ocp-schedules-18,a3dffbde49731fd4,b2fb735eea90e57f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:22:29.505 INFO 21062 --- [ocp-schedules-15,279a5e39ae66247c,cb2b0e029ab032bd] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:22:29.522 INFO 21062 --- [ocp-schedules-29,e75125f4f9a9e559,8fa4cebab44b5bc2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:22:29.527 INFO 21062 --- [ocp-schedules-29,e75125f4f9a9e559,8fa4cebab44b5bc2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:22:29.527 INFO 21062 --- [ocp-schedules-29,e75125f4f9a9e559,8fa4cebab44b5bc2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:22:29.551 WARN 21062 --- [ocp-async-2,a377ca99d49c1cc1,39c175184ecf41ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:29.555 INFO 21062 --- [ocp-async-10,eb47e57dfe1cfc4a,0126847142bf6ceb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:29.808 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:22:29.808 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:22:29.817 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:22:29.817 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:22:29.817 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:22:29.827 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:22:29.827 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:22:29.827 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:22:29.830 INFO 21062 --- [ocp-schedules-6,61683c3140c4cb3f,86c923a80e3b8ede] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:22:29.983 INFO 21062 --- [prometheus-minute-schedule-1,f54390b70d2b7e89,ab2b251e74a17b09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:29.992 INFO 21062 --- [ocp-schedules-14,1811af03f4dee625,ef61e3b259f088dc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:22:29.992 INFO 21062 --- [ocp-schedules-14,1811af03f4dee625,ef61e3b259f088dc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:22:29.998 INFO 21062 --- [ocp-schedules-14,1811af03f4dee625,ef61e3b259f088dc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:22:29.998 INFO 21062 --- [ocp-schedules-14,1811af03f4dee625,ef61e3b259f088dc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:22:30.003 INFO 21062 --- [ocp-schedules-14,1811af03f4dee625,ef61e3b259f088dc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:22:30.003 INFO 21062 --- [ocp-schedules-14,1811af03f4dee625,ef61e3b259f088dc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:22:30.029 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:30.030 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:22:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:22:30.430 INFO 21062 --- [ocp-schedules-1,981ae627a039e253,78bb39c65d5f4f6a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:22:30.430 INFO 21062 --- [ocp-schedules-1,981ae627a039e253,78bb39c65d5f4f6a] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:22:30.457 INFO 21062 --- [pool-53-thread-1,981ae627a039e253,ff534fcf28990349] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:22:30.486 WARN 21062 --- [ocp-schedules-1,981ae627a039e253,78bb39c65d5f4f6a] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:22:30.486 INFO 21062 --- [ocp-schedules-1,981ae627a039e253,78bb39c65d5f4f6a] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:22:31.552 WARN 21062 --- [ocp-async-1,375e73372c55977a,f34e1288906562a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:31.556 INFO 21062 --- [ocp-async-4,0e48a0b3f49a6545,68b547a8341deac2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:31.615 INFO 21062 --- [ocp-schedules-5,c83d0ccabc4f3e7b,35aab0cf21126d0c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:22:32.956 INFO 21062 --- [ocp-schedules-8,777fae988d52251c,239bd2fa25ed5e45] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:22:32.963 INFO 21062 --- [ocp-schedules-8,777fae988d52251c,239bd2fa25ed5e45] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:22:32.963 INFO 21062 --- [ocp-schedules-8,777fae988d52251c,239bd2fa25ed5e45] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:22:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f6791c13398c7341,163dac530f3f6b85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:33.030 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:33.031 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:33.180 INFO 21062 --- [ocp-schedules-23,bcf3ab6322248cfc,20a135d74e368199] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:22:33.180 INFO 21062 --- [ocp-schedules-23,bcf3ab6322248cfc,20a135d74e368199] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:22:33.208 INFO 21062 --- [ocp-schedules-23,bcf3ab6322248cfc,20a135d74e368199] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:22:33.208 INFO 21062 --- [ocp-schedules-23,bcf3ab6322248cfc,20a135d74e368199] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:22:33.209 INFO 21062 --- [ocp-schedules-19,306a8c1cdd57de6e,ef70256db41a64e0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:22:33.414 INFO 21062 --- [ocp-schedules-25,325c1112a2cfc5fa,ade9adc19f09fd15] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:22:33.414+08:00, clusters=[1] 2024-11-04 15:22:33.416 INFO 21062 --- [ocp-schedules-25,325c1112a2cfc5fa,ade9adc19f09fd15] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:22:33.552 WARN 21062 --- [ocp-async-8,e1f41ae65eda7f83,5913394d01da5569] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:33.556 INFO 21062 --- [ocp-async-7,0dc76fd24926e330,e64864d62bf31dd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:33.707 INFO 21062 --- [ocp-schedules-5,9347fd07e896b479,5649e0f36c5fd2da] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:22:33.707+08:00, clusters=[1] 2024-11-04 15:22:33.710 INFO 21062 --- [ocp-schedules-5,9347fd07e896b479,5649e0f36c5fd2da] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:22:33.752 INFO 21062 --- [ocp-schedules-15,452f7da5f27a260b,af68b50f6d835f12] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:22:33.755 INFO 21062 --- [ocp-schedules-15,452f7da5f27a260b,af68b50f6d835f12] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:33.813 ERROR 21062 --- [ocp-schedules-15,452f7da5f27a260b,af68b50f6d835f12] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:22:33.813 INFO 21062 --- [ocp-schedules-15,452f7da5f27a260b,af68b50f6d835f12] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:22:35.499 INFO 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:22:35.499 INFO 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:22:35.499 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:22:35.499 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:22:35.508 INFO 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:22:35.509 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:22:35.509 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:22:35.510 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:22:35.510 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:22:35.510 WARN 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:22:35.511 INFO 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:22:35.511 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:22:35.511 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:22:35.512 WARN 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:22:35.513 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:22:35.513 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:22:35.520 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:22:35.520 INFO 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:22:35.520 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:22:35.532 ERROR 21062 --- [ocp-schedules-20,cdd234ab99632423,55fe56be0f596768] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:22:35.552 WARN 21062 --- [ocp-async-6,aa8d79a9871caf02,95dc603de84fb493] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:35.556 INFO 21062 --- [ocp-async-3,3c74cec9181e30c6,7dfcfbf1269942ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:35.852 INFO 21062 --- [ocp-schedules-27,35e7122875519b07,910bdfc11ca483b0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:22:35.983 INFO 21062 --- [prometheus-minute-schedule-1,cb7da371921f45eb,afdff7e74659d5be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:36.031 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:36.032 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:37.552 WARN 21062 --- [ocp-async-1,a3e668ae9ed2fe00,cce0789bdfb95d5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:37.556 INFO 21062 --- [ocp-async-5,846aeb1e514b6532,cdd0a20de3acd617] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:38.504 INFO 21062 --- [ocp-schedules-17,02dc6080f84a9de7,e8dc40e044828c47] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:22:38.506 INFO 21062 --- [ocp-schedules-13,c2fd8a6739abe694,22f904a9f6855060] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:22:38.508 INFO 21062 --- [ocp-schedules-21,84596dfaac9c9b7f,3725a58b8cbe444f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:22:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:22:38.589728+08:00, lastProcessTime=2024-11-04T15:22:28.589965+08:00 2024-11-04 15:22:38.759 INFO 21062 --- [ocp-async-7,7f5be21dedf8e027,8c9c43503b8a1b59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:22:38.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:22:38.589728+08:00 2024-11-04 15:22:38.983 INFO 21062 --- [prometheus-minute-schedule-1,d6ff6a87d425660b,b5f3eff7a5f10d50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:39.032 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:39.033 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:39.303 INFO 21062 --- [ocp-schedules-8,7efc8dc6ddb0410b,641d65cc6f4edb93] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:22:39.307 INFO 21062 --- [ocp-schedules-8,7efc8dc6ddb0410b,641d65cc6f4edb93] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:39.307 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:22:39.552 WARN 21062 --- [ocp-async-7,4c431cfd6bfee2bc,d9c8ddd106b5fabb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:39.556 INFO 21062 --- [ocp-async-6,a93de370f6781084,ca557f29b50f259b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:40.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:22:40.003 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:22:40.148 INFO 21062 --- [ocp-schedules-12,1a5f3944f71ccd7f,cce3daaab5da5141] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:22:40.148 INFO 21062 --- [ocp-schedules-12,ab54c7dfab4b143b,eb84f5203bbebe8b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:22:40.152 INFO 21062 --- [ocp-schedules-12,ab54c7dfab4b143b,eb84f5203bbebe8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:40.180 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:22:40.180 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:22:41.552 WARN 21062 --- [ocp-async-2,da39a736646b96db,7f87dfa23495a2da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:41.556 INFO 21062 --- [ocp-async-1,19aabf29ac8ecda2,b9767128547351ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:41.618 INFO 21062 --- [ocp-schedules-18,7c723761e1db4f0f,4d4c2044a368fd09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:22:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:22:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:22:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:22:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:22:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:22:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:22:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:22:41.983 INFO 21062 --- [prometheus-minute-schedule-1,835ec2823c3a6d20,bded16b276c0841e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:42.033 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:42.034 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:43.553 WARN 21062 --- [ocp-async-1,12b68f4eb537bdd0,0ed820e88a44e6c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:43.557 INFO 21062 --- [ocp-async-4,bfc063866003ecd5,b8e30b2cb564b10b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:44.242 INFO 21062 --- [ocp-schedules-25,0e461e3564e264bb,91e08993d257f076] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:22:44.245 INFO 21062 --- [ocp-schedules-25,0e461e3564e264bb,91e08993d257f076] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:44.332 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:22:44.335 ERROR 21062 --- [ocp-schedules-25,0e461e3564e264bb,91e08993d257f076] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:22:44.335 INFO 21062 --- [ocp-schedules-25,0e461e3564e264bb,91e08993d257f076] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:22:44.983 INFO 21062 --- [prometheus-minute-schedule-1,2353a5bba5ae9f6d,121ae4103523abc2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:45.034 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:45.035 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:45.553 WARN 21062 --- [ocp-async-8,8043b28ae3fdf0df,52bc669fddb08f3a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:45.557 INFO 21062 --- [ocp-async-7,14350928dd3ad41b,069fc74fe0fcf96b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:45.880 INFO 21062 --- [ocp-schedules-10,792ecbe762c54d34,ef7868ba1975e870] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:22:47.553 WARN 21062 --- [ocp-async-6,338eea988bb667dd,a5bb862b013b6188] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:47.557 INFO 21062 --- [ocp-async-2,fe667230fa5971f2,658f7885baf91370] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:47.983 INFO 21062 --- [prometheus-minute-schedule-1,c2c563f41b152413,bfdc1b41b7d8b146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:48.035 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:48.036 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:48.597 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:22:48.597194+08:00, lastProcessTime=2024-11-04T15:22:38.589728+08:00 2024-11-04 15:22:48.759 INFO 21062 --- [ocp-async-5,5484e5c82b2e357c,12e8b08b2c0481a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:22:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:22:48.597194+08:00 2024-11-04 15:22:49.307 INFO 21062 --- [ocp-schedules-8,7efc8dc6ddb0410b,641d65cc6f4edb93] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:22:49.553 WARN 21062 --- [ocp-async-10,0420c8293a1d9766,9c7fcca2d2003533] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:49.557 INFO 21062 --- [ocp-async-5,e3f84f77f4e3cb20,c3d1c9d43b66b041] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:50.983 INFO 21062 --- [prometheus-minute-schedule-1,cb2cbdb061128277,3681593ffb1f512f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:51.036 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:51.037 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:51.553 WARN 21062 --- [ocp-async-9,a4772196f8e92873,99a0e7780c91d836] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:51.557 INFO 21062 --- [ocp-async-4,54b5aee3420d7cbe,27ef69172f5756b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:51.621 INFO 21062 --- [ocp-schedules-27,47a0384d0181e9b2,51b4dd0dc302d670] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:22:52.965 INFO 21062 --- [ocp-schedules-23,1344bf9ddef3f6cd,6ad3260e01ab8a82] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:22:52.971 INFO 21062 --- [ocp-schedules-23,1344bf9ddef3f6cd,6ad3260e01ab8a82] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:22:52.971 INFO 21062 --- [ocp-schedules-23,1344bf9ddef3f6cd,6ad3260e01ab8a82] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:22:53.554 WARN 21062 --- [ocp-async-7,d8e95b6e1b7506f5,e823b57c4119262f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:53.557 INFO 21062 --- [ocp-async-3,d6cd00159ede0d4f,496d97b66d8a2e66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:53.983 INFO 21062 --- [prometheus-minute-schedule-1,bea17866a1e76d6b,912e552c0d467dca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:54.037 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:54.038 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:55.554 WARN 21062 --- [ocp-async-10,154571cbafd40f78,760e05239c5fb0c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:55.558 INFO 21062 --- [ocp-async-1,c4f34b07e0fea7ca,5f4efa784dc56991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:55.919 INFO 21062 --- [ocp-schedules-21,2f11e820d39cb7a0,f863ce1ccf442ce6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:22:56.983 INFO 21062 --- [prometheus-minute-schedule-1,1097c736227aabed,efd63c43071b7b50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:22:57.038 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:22:57.039 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:22:57.554 WARN 21062 --- [ocp-async-8,b2c8fea48446a706,33026dddb8484b8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:57.558 INFO 21062 --- [ocp-async-5,ef3d545e16730f2c,6ed327ea0fc8a9fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:58.499 INFO 21062 --- [ocp-schedules-23,5c36c4f1fd56b0df,9ac8ba6085c638dd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:22:58.500 INFO 21062 --- [ocp-schedules-4,f1f480e6564bb667,8f0f2ec851aac986] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:22:58.502 INFO 21062 --- [ocp-schedules-13,1badf09b94de0c0b,a49598917b8f65f2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:22:58.504 INFO 21062 --- [ocp-schedules-23,5c36c4f1fd56b0df,9ac8ba6085c638dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:22:58.505 INFO 21062 --- [ocp-schedules-28,fd9add47be9b38af,7fed9096ebef9f82] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:22:58.505 INFO 21062 --- [ocp-schedules-4,f1f480e6564bb667,8f0f2ec851aac986] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:22:58.510 INFO 21062 --- [ocp-schedules-7,d63dc5050a616097,5a1d938a9f91f5c7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:22:58.510 INFO 21062 --- [ocp-schedules-7,d63dc5050a616097,5a1d938a9f91f5c7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:22:58.510 INFO 21062 --- [ocp-schedules-3,862ff20eaf765123,d64241758b88d85a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:22:58.510 INFO 21062 --- [ocp-schedules-25,68b54dc65105984a,403f02dc937c69fd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:22:58.510 INFO 21062 --- [ocp-schedules-3,862ff20eaf765123,d64241758b88d85a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:22:58.510 INFO 21062 --- [ocp-schedules-25,68b54dc65105984a,403f02dc937c69fd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:22:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:22:58.588862+08:00, lastProcessTime=2024-11-04T15:22:48.597194+08:00 2024-11-04 15:22:58.759 INFO 21062 --- [ocp-async-4,4460fee24831e503,eb628c297bdb479c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:22:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:22:58.588862+08:00 2024-11-04 15:22:59.504 INFO 21062 --- [ocp-schedules-8,958760be4fbff22f,e3102b5190a71975] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:22:59.554 WARN 21062 --- [ocp-async-7,d1a15273699cf5a9,00073a7cafe6cf6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:22:59.558 INFO 21062 --- [ocp-async-3,40185ae19800cfb2,ada8471c8d1324ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:22:59.983 INFO 21062 --- [prometheus-minute-schedule-1,81fb03ef256b454d,be419fecba8fb1cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730704980, interval=60 2024-11-04 15:23:00.039 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:00.040 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:01.554 WARN 21062 --- [ocp-async-10,349c2eab51418bdf,d653a11ef9ca83e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:01.558 INFO 21062 --- [ocp-async-1,85148d7120f94212,e07961a9d77386db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:01.624 INFO 21062 --- [ocp-schedules-2,5cb9dad93fd4e126,e8689661230496ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:23:02.983 INFO 21062 --- [prometheus-minute-schedule-1,fc3aaf4109747900,5199be674ad380f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:03.040 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:03.042 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:03.555 WARN 21062 --- [ocp-async-8,95a6d01b89d7552b,3c2de27e3bf12bfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:03.558 INFO 21062 --- [ocp-async-5,e08a8e8e6e575664,4347ca70d4515e6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:05.555 WARN 21062 --- [ocp-async-6,c8d51d565f70ca0d,d6124160822ed21c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:05.558 INFO 21062 --- [ocp-async-4,c94e82c5847ef787,6154ebc3e59a87fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:05.943 INFO 21062 --- [ocp-schedules-30,765d2195a6330830,3fce0035e5563ef5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:23:05.983 INFO 21062 --- [prometheus-minute-schedule-1,bdfb5c64fe72f537,714e867d623d17c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:06.042 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:06.042 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:07.555 WARN 21062 --- [ocp-async-3,e32158a0247c8a8b,808ea73f9d1cae71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:07.559 INFO 21062 --- [ocp-async-2,d0414bf4d71a14b8,08774411ccd4d8bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:08.505 INFO 21062 --- [ocp-schedules-23,5c36c4f1fd56b0df,9ac8ba6085c638dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:23:08.505 INFO 21062 --- [ocp-schedules-4,f1f480e6564bb667,8f0f2ec851aac986] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:23:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:23:08.590784+08:00, lastProcessTime=2024-11-04T15:22:58.588862+08:00 2024-11-04 15:23:08.759 INFO 21062 --- [ocp-async-9,09d77443edb07691,968918d788f5ba44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:23:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:23:08.590784+08:00 2024-11-04 15:23:08.983 INFO 21062 --- [prometheus-minute-schedule-1,034ea493bf0c0c30,e9b8189150389dca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:09.043 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:09.043 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:09.555 WARN 21062 --- [ocp-async-8,119ddf0c93cbcb04,0b2cf052f7a1d1fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:09.559 INFO 21062 --- [ocp-async-5,7fc1eeab360717bf,f3d2b72b8134b11e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:10.002 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:23:10.006 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:23:10.181 INFO 21062 --- [ocp-schedules-12,ab54c7dfab4b143b,eb84f5203bbebe8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:23:10.181 INFO 21062 --- [ocp-schedules-12,2401de40e0ba4960,56b98d83b54e6188] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:23:10.184 INFO 21062 --- [ocp-schedules-12,2401de40e0ba4960,56b98d83b54e6188] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:10.214 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:23:10.214 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:23:10.307 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:23:10.307 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:23:10.307 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:23:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:23:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:23:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:23:11.555 WARN 21062 --- [ocp-async-6,05d9eff4ebfaa910,c59d0d134cf4aa30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:11.559 INFO 21062 --- [ocp-async-4,c598c74c5f077b5b,2e69395c9fea5ec2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:11.627 INFO 21062 --- [ocp-schedules-29,da8557b237852e65,2e3a67fdba78d5be] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:23:11.818 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=5, affectRows=5 2024-11-04 15:23:11.818 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=5, validCount=5 2024-11-04 15:23:11.818 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=15, affectRows=15 2024-11-04 15:23:11.818 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=21, affectRows=21 2024-11-04 15:23:11.818 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=30, affectRows=30 2024-11-04 15:23:11.818 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=15, validCount=15 2024-11-04 15:23:11.819 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=21, validCount=21 2024-11-04 15:23:11.819 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=30, validCount=30 2024-11-04 15:23:11.819 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=65, affectRows=65 2024-11-04 15:23:11.819 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=68, affectRows=68 2024-11-04 15:23:11.819 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=65, validCount=65 2024-11-04 15:23:11.819 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=68, validCount=68 2024-11-04 15:23:11.820 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=73, affectRows=73 2024-11-04 15:23:11.820 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=71, validCount=71 2024-11-04 15:23:11.820 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=24, affectRows=24 2024-11-04 15:23:11.820 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=24, validCount=24 2024-11-04 15:23:11.821 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704920, nodeCount=121, affectRows=121 2024-11-04 15:23:11.821 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704920, nodeCount=120, validCount=120 2024-11-04 15:23:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e6c2d8c88c5e5f22,0bea0136cf325bbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:12.044 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:12.045 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:12.973 INFO 21062 --- [ocp-schedules-2,c26e7e5943e66129,e587f8d7204baf9d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:23:12.979 INFO 21062 --- [ocp-schedules-2,c26e7e5943e66129,e587f8d7204baf9d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:23:12.979 INFO 21062 --- [ocp-schedules-2,c26e7e5943e66129,e587f8d7204baf9d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:23:13.555 WARN 21062 --- [ocp-async-3,3f7b88c5cae10f05,3e2dcda6ef816afb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:13.559 INFO 21062 --- [ocp-async-2,d2c30984311553cd,c9e1527dd1758ccd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:13.984 INFO 21062 --- [prometheus-cache-monitor1,a42de2fad1bfeb8b,8435f17f8abe2681] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:23:13.984 INFO 21062 --- [prometheus-cache-monitor1,a42de2fad1bfeb8b,8435f17f8abe2681] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:23:14.465 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:17:14.446, end=2024-11-04T15:22:14.462} 2024-11-04 15:23:14.465 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:17:14.446, end=2024-11-04T15:22:14.463} 2024-11-04 15:23:14.474 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:23:14.474 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:23:14.474 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:23:14.474 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:23:14.983 INFO 21062 --- [prometheus-minute-schedule-1,39e165ff8c0a5627,9cdf08c3483a094d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:15.045 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:15.045 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:15.555 WARN 21062 --- [ocp-async-1,8bf06ae7937a7bb0,00eab8971bcf3c38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:15.559 INFO 21062 --- [ocp-async-9,1e03655a4cccd4cc,eb5295a4a31b11a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:15.607 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:19:15.581, end=2024-11-04T15:21:15.605} 2024-11-04 15:23:15.611 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:23:15.611 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:23:15.985 INFO 21062 --- [ocp-schedules-16,535ce2322ea73e85,582b5eaab2cd3bbf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:23:17.556 WARN 21062 --- [ocp-async-5,71940cdf68622242,18a7630c67716ad3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:17.560 INFO 21062 --- [ocp-async-7,cdc4671d3d7c8e87,caf207fc26d53097] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:17.983 INFO 21062 --- [prometheus-minute-schedule-1,04c55a46f3664b11,f3143e4c48404768] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:18.045 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:18.046 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:23:18.590351+08:00, lastProcessTime=2024-11-04T15:23:08.590784+08:00 2024-11-04 15:23:18.760 INFO 21062 --- [ocp-async-10,ef599115cb6015be,07dc51033cb78dd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:23:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:23:18.590351+08:00 2024-11-04 15:23:19.556 WARN 21062 --- [ocp-async-3,0a0b77afafb29e8a,5cd26e1d131865fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:19.560 INFO 21062 --- [ocp-async-2,159b42b58ea21794,fe5854afae2fceb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:20.983 INFO 21062 --- [prometheus-minute-schedule-1,6e0734f5ffcd31ed,f5163c70185a8355] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:21.046 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:21.047 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:21.556 WARN 21062 --- [ocp-async-1,c51c41a6ee825158,6763c2410c291557] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:21.560 INFO 21062 --- [ocp-async-9,f4e49033499a049a,6c3875ab6b3d02dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:21.630 INFO 21062 --- [ocp-schedules-28,11ed60897f39f904,eba88c95b5bdeea8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:23:23.556 WARN 21062 --- [ocp-async-5,5f2ab6adeb0efd0b,7788dacd68bc0c7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:23.560 INFO 21062 --- [ocp-async-7,1ef6ae4d7dec8daa,c8a9114814df4a13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:23.983 INFO 21062 --- [prometheus-minute-schedule-1,e98fc56b967ea1a5,dd4a3042c2c63f61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:24.047 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:24.048 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:25.556 WARN 21062 --- [ocp-async-4,e22f732bd6d348b2,dce9eb0b2e0d6d89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:25.560 INFO 21062 --- [ocp-async-10,755e62c7b9e4f372,bc3493c23fc9b5b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:26.008 INFO 21062 --- [ocp-schedules-22,07057760c5561f0a,8aec1f020f434a62] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:23:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ccf8e03498a3869c,9a47afe8a9374007] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:26.984 INFO 21062 --- [prometheus-hour-schedule-1,18237401bb3061dd,28303f2a91cd7e43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:23:27.048 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:27.049 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:27.556 WARN 21062 --- [ocp-async-2,d616c2f244963cfa,7421b38214ba3776] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:27.560 INFO 21062 --- [ocp-async-8,9d211444d535c0f5,8eb8298a0804f5ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:28.498 INFO 21062 --- [ocp-schedules-26,d9b4e39e7b566077,72fade72f0605209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:23:28.500 INFO 21062 --- [ocp-schedules-6,d2a9672178aa7f78,ffb0ba45d76fe073] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:23:28.500 INFO 21062 --- [ocp-schedules-30,5981ad0438d251d5,98fe5f85cb03b054] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:23:28.500 INFO 21062 --- [ocp-schedules-20,1be8b273bfee74f1,07f6e14df1121de6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:23:28.502 INFO 21062 --- [ocp-schedules-26,d9b4e39e7b566077,72fade72f0605209] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:23:28.502 INFO 21062 --- [ocp-schedules-16,6ba386e7703b81d9,7ed81d2fc3295d07] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:23:28.503 INFO 21062 --- [ocp-schedules-6,d2a9672178aa7f78,ffb0ba45d76fe073] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:28.504 INFO 21062 --- [ocp-schedules-20,1be8b273bfee74f1,07f6e14df1121de6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:23:28.505 INFO 21062 --- [ocp-schedules-11,5b6d110b78c9502c,5622e2f16b1ad07a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:23:28.506 INFO 21062 --- [ocp-schedules-30,5981ad0438d251d5,98fe5f85cb03b054] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:23:28.510 INFO 21062 --- [ocp-schedules-1,3efe73c4f6c42c0c,be5057d46bc41f55] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:23:28.510 INFO 21062 --- [ocp-schedules-18,b014c514e0804513,4c43361c8e2914de] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:23:28.510 INFO 21062 --- [ocp-schedules-1,3efe73c4f6c42c0c,be5057d46bc41f55] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:23:28.510 INFO 21062 --- [ocp-schedules-18,b014c514e0804513,4c43361c8e2914de] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:23:28.511 INFO 21062 --- [ocp-schedules-10,f660120a56384b91,bacfbfc70719587e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:23:28.511 INFO 21062 --- [ocp-schedules-10,f660120a56384b91,bacfbfc70719587e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:23:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:23:28.589721+08:00, lastProcessTime=2024-11-04T15:23:18.590351+08:00 2024-11-04 15:23:28.603 INFO 21062 --- [ocp-schedules-7,bc6d33a980eb8824,d3b9772d55a31ad8] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 15:23:28.603 INFO 21062 --- [ocp-schedules-7,bc6d33a980eb8824,d3b9772d55a31ad8] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 15:23:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:23:28.589721+08:00 2024-11-04 15:23:28.760 INFO 21062 --- [ocp-async-4,a8fcffa1308254fb,31844a0d60c40334] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:23:28.982 INFO 21062 --- [ocp-schedules-30,3573df5dd556fa65,99be154fb140a8f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:23:29.371 INFO 21062 --- [ocp-schedules-7,4a97792d9fad2630,3adb96fa066da266] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:23:29.372 INFO 21062 --- [ocp-schedules-7,4a97792d9fad2630,3adb96fa066da266] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:23:29.506 INFO 21062 --- [ocp-schedules-5,8fcd8b1cb5336c59,d15706c47f17e8a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:23:29.527 INFO 21062 --- [ocp-schedules-2,5ce2d52a1530d562,f7b9cd3c2e7b6bfd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:23:29.532 INFO 21062 --- [ocp-schedules-2,5ce2d52a1530d562,f7b9cd3c2e7b6bfd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:23:29.532 INFO 21062 --- [ocp-schedules-2,5ce2d52a1530d562,f7b9cd3c2e7b6bfd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:23:29.557 WARN 21062 --- [ocp-async-7,03ab474a8a1a8f19,6b8ecc8c064a579f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:29.561 INFO 21062 --- [ocp-async-3,ddf9fb288e5a3d3d,91bdb0f91841ffae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:29.983 INFO 21062 --- [prometheus-minute-schedule-1,97747241289af602,1798dc94ae1b6ea6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:30.004 INFO 21062 --- [ocp-schedules-14,cf629b397adb9172,c26b3630130b56e6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:23:30.004 INFO 21062 --- [ocp-schedules-14,cf629b397adb9172,c26b3630130b56e6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:23:30.006 INFO 21062 --- [ocp-schedules-14,cf629b397adb9172,c26b3630130b56e6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:23:30.006 INFO 21062 --- [ocp-schedules-14,cf629b397adb9172,c26b3630130b56e6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:23:30.006 INFO 21062 --- [ocp-schedules-14,cf629b397adb9172,c26b3630130b56e6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:23:30.006 INFO 21062 --- [ocp-schedules-14,cf629b397adb9172,c26b3630130b56e6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:23:30.049 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:30.050 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:23:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:23:31.557 WARN 21062 --- [ocp-async-10,74135572dcbf826c,bf88c509811eed61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:31.561 INFO 21062 --- [ocp-async-1,8c8ce953839957ed,1ca10344510aa3e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:31.633 INFO 21062 --- [ocp-schedules-8,eb523d558e6ce8b0,15afae5da4eed806] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:23:32.981 INFO 21062 --- [ocp-schedules-9,84408fdaef4648dc,2c755250c4f0e126] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:23:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7b6ed2fb9819c561,2cb0017a8e0e1471] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:32.987 INFO 21062 --- [ocp-schedules-9,84408fdaef4648dc,2c755250c4f0e126] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:23:32.987 INFO 21062 --- [ocp-schedules-9,84408fdaef4648dc,2c755250c4f0e126] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:23:33.053 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:33.054 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:33.208 INFO 21062 --- [ocp-schedules-24,4a9588804d28ad36,b4c33ce6c94a5867] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:23:33.208 INFO 21062 --- [ocp-schedules-24,4a9588804d28ad36,b4c33ce6c94a5867] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:23:33.236 INFO 21062 --- [ocp-schedules-24,4a9588804d28ad36,b4c33ce6c94a5867] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:23:33.236 INFO 21062 --- [ocp-schedules-24,4a9588804d28ad36,b4c33ce6c94a5867] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:23:33.237 INFO 21062 --- [ocp-schedules-8,cc2d0aed65d70a72,a053f5fbe2bb4c7a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:23:33.443 INFO 21062 --- [ocp-schedules-3,efaeb3d3a68ef1cb,047c444553c682c3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:23:33.443+08:00, clusters=[1] 2024-11-04 15:23:33.444 INFO 21062 --- [ocp-schedules-3,efaeb3d3a68ef1cb,047c444553c682c3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:23:33.557 WARN 21062 --- [ocp-async-9,b1b3236662545ce2,61a6ce4322ac2619] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:33.561 INFO 21062 --- [ocp-async-5,6f191e4896f18a4d,90c8732ce2e0ec34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:33.739 INFO 21062 --- [ocp-schedules-15,6b6e7d01db435f38,c4c78568869a75e8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:23:33.739+08:00, clusters=[1] 2024-11-04 15:23:33.743 INFO 21062 --- [ocp-schedules-15,6b6e7d01db435f38,c4c78568869a75e8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:23:35.541 INFO 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:23:35.541 INFO 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:23:35.541 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:23:35.541 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:23:35.552 INFO 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:23:35.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:23:35.552 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:23:35.554 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:23:35.554 WARN 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:23:35.554 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:23:35.555 INFO 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:23:35.555 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:23:35.555 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:23:35.556 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:23:35.556 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:23:35.556 WARN 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:23:35.557 WARN 21062 --- [ocp-async-6,0b33b269154a3b73,14370eb711e850a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:35.561 INFO 21062 --- [ocp-async-4,e363fd69a6d528d0,3cfd6c98f19cc381] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:35.563 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:23:35.563 INFO 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:23:35.563 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:23:35.575 ERROR 21062 --- [ocp-schedules-22,102f1755cb3eac1a,07831518263c7679] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:23:35.983 INFO 21062 --- [prometheus-minute-schedule-1,4117d2f578d0d211,45d18bb0a461478f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:36.049 INFO 21062 --- [ocp-schedules-28,b0b078d39d060bf9,877ed0868ff01cc7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:23:36.054 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:36.055 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:37.557 WARN 21062 --- [ocp-async-3,2cef0dbfec7cb63f,fb6502c3ca54b5ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:37.561 INFO 21062 --- [ocp-async-2,e210601c3a668d6a,3d8909b32a2032f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:38.504 INFO 21062 --- [ocp-schedules-20,1be8b273bfee74f1,07f6e14df1121de6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:23:38.504 INFO 21062 --- [ocp-schedules-6,d2a9672178aa7f78,ffb0ba45d76fe073] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:23:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:23:38.590131+08:00, lastProcessTime=2024-11-04T15:23:28.589721+08:00 2024-11-04 15:23:38.760 INFO 21062 --- [ocp-async-5,07afd30de35e88cb,5d46f60fd153f1b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:23:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:23:38.590131+08:00 2024-11-04 15:23:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a9133ee7640d93e7,c63acf5cb032cc96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:39.055 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:39.056 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:39.558 WARN 21062 --- [ocp-async-8,eccdb46df9f8d1b2,0a6d829e22e74719] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:39.561 INFO 21062 --- [ocp-async-6,939ed50b213b1806,a9a528ce266c6280] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:40.002 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:23:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:23:40.214 INFO 21062 --- [ocp-schedules-12,2401de40e0ba4960,56b98d83b54e6188] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:23:40.214 INFO 21062 --- [ocp-schedules-12,27a3e703862a73d3,5e269aa786592991] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:23:40.218 INFO 21062 --- [ocp-schedules-12,27a3e703862a73d3,5e269aa786592991] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:40.247 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:23:40.247 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:23:41.558 WARN 21062 --- [ocp-async-7,75144608642873ef,33e5a20a64a2907c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:41.562 INFO 21062 --- [ocp-async-3,ab1f79d843ffadd2,b8cc1a2ae0854699] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:41.635 INFO 21062 --- [ocp-schedules-15,64971f7bd3b06757,acb8456dd8c46da2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:23:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2ad43d9835e275fd,b5d57c46b73b918d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:42.056 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:42.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:43.558 WARN 21062 --- [ocp-async-10,474d53e896186e59,f5ab88db48cbcd64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:43.562 INFO 21062 --- [ocp-async-2,8f39657b85e04982,57cc1cc96ac83d00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:44.335 INFO 21062 --- [ocp-schedules-5,cbf430a0dd78294a,1fe9736da8a18a0a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:23:44.340 INFO 21062 --- [ocp-schedules-5,cbf430a0dd78294a,1fe9736da8a18a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:44.425 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:23:44.428 ERROR 21062 --- [ocp-schedules-5,cbf430a0dd78294a,1fe9736da8a18a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:23:44.428 INFO 21062 --- [ocp-schedules-5,cbf430a0dd78294a,1fe9736da8a18a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:23:44.983 INFO 21062 --- [prometheus-minute-schedule-1,9fd1cf22c92dbc67,fa570229cb8e230a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:45.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:45.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:45.558 WARN 21062 --- [ocp-async-2,ed85bb8af6998716,f1b3b8d0dccaaba3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:45.562 INFO 21062 --- [ocp-async-5,7f97aa63dee243dd,b9d44d1587bfaff1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:46.074 INFO 21062 --- [ocp-schedules-5,b0221e9b8b973cb4,1cbd70935c4a2aab] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:23:47.558 WARN 21062 --- [ocp-async-5,e63168090fa4d561,fc115e80058656a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:47.562 INFO 21062 --- [ocp-async-4,2ffbfc4490259fea,51c6ff8bc0bf627b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:47.983 INFO 21062 --- [prometheus-minute-schedule-1,acf28b3dc4c6c20b,8448f5a7947a0e9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:48.057 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:48.058 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:23:48.592803+08:00, lastProcessTime=2024-11-04T15:23:38.590131+08:00 2024-11-04 15:23:48.760 INFO 21062 --- [ocp-async-10,0c4b7e9101e71000,2211f97aabaf4f75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:23:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:23:48.592803+08:00 2024-11-04 15:23:49.308 INFO 21062 --- [ocp-schedules-30,ef45684e62e87c6f,421caf93defada24] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:23:49.312 INFO 21062 --- [ocp-schedules-30,ef45684e62e87c6f,421caf93defada24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:49.313 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:23:49.558 WARN 21062 --- [ocp-async-3,f34c87a6a5c4e1f8,3d4d418613aec515] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:49.562 INFO 21062 --- [ocp-async-10,cac1ca96e718c892,476aefd67d02eb89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:50.983 INFO 21062 --- [prometheus-minute-schedule-1,529bf173529b2aca,70520209cd068bda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:51.058 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:51.059 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:51.559 WARN 21062 --- [ocp-async-10,c7f01e26a55ac0f0,2b86666f6540ba70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:51.562 INFO 21062 --- [ocp-async-8,a6a6c60744e21ffa,3ef4dfc029a22569] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:51.639 INFO 21062 --- [ocp-schedules-17,b45bef5cb8878d47,ec2d60f149e78fb4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:23:52.989 INFO 21062 --- [ocp-schedules-1,0768a283ca218da1,4a08b98b9f1dacd5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:23:52.995 INFO 21062 --- [ocp-schedules-1,0768a283ca218da1,4a08b98b9f1dacd5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:23:52.995 INFO 21062 --- [ocp-schedules-1,0768a283ca218da1,4a08b98b9f1dacd5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:23:53.559 WARN 21062 --- [ocp-async-8,d8dd948b601fa228,f7481d9dc13f717d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:53.563 INFO 21062 --- [ocp-async-6,21a54969221140e5,12408404e211289c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:53.983 INFO 21062 --- [prometheus-minute-schedule-1,2417e1c092989c29,da27c89d097aa423] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:54.059 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:54.060 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:55.559 WARN 21062 --- [ocp-async-6,588cc57605d91f61,f24813fe8ed9d29f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:55.563 INFO 21062 --- [ocp-async-1,92eb181bfac37d4a,1946cce0b3b6551a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:56.116 INFO 21062 --- [ocp-schedules-5,345330e6767b8cb5,bc5b8d69ec30cb75] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:23:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ccb98b46eaf6771a,81dec66a5e421476] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:23:57.060 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:23:57.061 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:23:57.559 WARN 21062 --- [ocp-async-9,199fd14038ad6f82,16e43a6122b53763] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:57.563 INFO 21062 --- [ocp-async-2,de8a21d26310963c,79d39dcf357856d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:58.500 INFO 21062 --- [ocp-schedules-1,3f5a3119b1cbb4fa,3b62f9be10c28737] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:23:58.500 INFO 21062 --- [ocp-schedules-20,eee117bd82b44fc8,688d75cb199a418a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:23:58.502 INFO 21062 --- [ocp-schedules-6,7a645c94bcc68aa7,ed4dd5e8af545b0d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:23:58.504 INFO 21062 --- [ocp-schedules-1,3f5a3119b1cbb4fa,3b62f9be10c28737] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:58.504 INFO 21062 --- [ocp-schedules-20,eee117bd82b44fc8,688d75cb199a418a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:23:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:23:58.505 INFO 21062 --- [ocp-schedules-28,a0a8e0caf5817dc7,c5751747e0e9e8f7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:23:58.509 INFO 21062 --- [ocp-schedules-4,29313f500a9feca6,df4dd7d4f1ff5f61] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:23:58.510 INFO 21062 --- [ocp-schedules-3,ac5a0d472e544475,5aa3ec269a2cb093] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:23:58.510 INFO 21062 --- [ocp-schedules-29,58a5688af599b477,ef7c811217b2b3a0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:23:58.510 INFO 21062 --- [ocp-schedules-4,29313f500a9feca6,df4dd7d4f1ff5f61] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:23:58.510 INFO 21062 --- [ocp-schedules-3,ac5a0d472e544475,5aa3ec269a2cb093] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:23:58.510 INFO 21062 --- [ocp-schedules-29,58a5688af599b477,ef7c811217b2b3a0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:23:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:23:58.590102+08:00, lastProcessTime=2024-11-04T15:23:48.592803+08:00 2024-11-04 15:23:58.760 INFO 21062 --- [ocp-async-7,253d13f492bc29e7,d990352b28f38f34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:23:58.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:23:58.590102+08:00 2024-11-04 15:23:59.313 INFO 21062 --- [ocp-schedules-30,ef45684e62e87c6f,421caf93defada24] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:23:59.510 INFO 21062 --- [ocp-schedules-13,6401d82343e00592,1905768f1eee160d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:23:59.559 WARN 21062 --- [ocp-async-8,3ba7928662c0e680,6767d44b52054665] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:23:59.563 INFO 21062 --- [ocp-async-4,f5b8aaa4da4b5d09,4261d58565e482dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:23:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f224e4b21dbda9a7,77364e6349a8e68a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705040, interval=60 2024-11-04 15:24:00.061 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:00.062 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:01.560 WARN 21062 --- [ocp-async-6,3428e69a80dd5ab3,d3c6f76dce20ec06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:01.563 INFO 21062 --- [ocp-async-1,99c81d66780a9c2e,1ded2a54aa78f054] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:01.642 INFO 21062 --- [ocp-schedules-23,a7fd096853ecbd16,14f70c033a78c845] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:24:02.983 INFO 21062 --- [prometheus-minute-schedule-1,32fa15e794e3edbd,02b12cb229707519] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:03.062 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:03.063 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:03.560 WARN 21062 --- [ocp-async-9,e884b85fab87da6b,f054e633fd547b12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:03.563 INFO 21062 --- [ocp-async-2,312f18d9fd334c87,421c42d990434ecb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:05.560 WARN 21062 --- [ocp-async-5,b1a21dbb00b77d50,721b0c6e92e7d23c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:05.564 INFO 21062 --- [ocp-async-7,f4065165ac5d41f8,ba4cc7524661d720] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:05.983 INFO 21062 --- [prometheus-minute-schedule-1,7e5b9270eaa3820f,a5aa5f9c7d917bf2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:06.063 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:06.064 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:06.140 INFO 21062 --- [ocp-schedules-7,8e09ea2bb4f21e29,7b5ca312fde9c5b0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:24:07.560 WARN 21062 --- [ocp-async-4,72bcaec582e08fc6,2d83cf7b33a5e790] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:07.564 INFO 21062 --- [ocp-async-3,aa374e8ba92b5f8b,e4c307a1ecca5dc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:08.505 INFO 21062 --- [ocp-schedules-20,eee117bd82b44fc8,688d75cb199a418a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:24:08.505 INFO 21062 --- [ocp-schedules-1,3f5a3119b1cbb4fa,3b62f9be10c28737] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:24:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:24:08.590304+08:00, lastProcessTime=2024-11-04T15:23:58.590102+08:00 2024-11-04 15:24:08.760 INFO 21062 --- [ocp-async-10,2b61df17a0edbba3,5404c850b7269a85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:24:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:24:08.590304+08:00 2024-11-04 15:24:08.983 INFO 21062 --- [prometheus-minute-schedule-1,29da824cb2d4ccde,870b50964b590248] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:09.064 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:09.065 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:09.560 WARN 21062 --- [ocp-async-9,6ef342502f240a18,481fab48d396b8dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:09.564 INFO 21062 --- [ocp-async-2,9f20271bb826f2d0,bda4b5d57a494256] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:24:10.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:24:10.247 INFO 21062 --- [ocp-schedules-12,27a3e703862a73d3,5e269aa786592991] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:24:10.247 INFO 21062 --- [ocp-schedules-12,759c02ccc315fe11,f222944990e32645] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:24:10.251 INFO 21062 --- [ocp-schedules-12,759c02ccc315fe11,f222944990e32645] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:10.280 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:24:10.280 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:24:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:24:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:24:10.296 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:24:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:24:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:24:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:24:11.561 WARN 21062 --- [ocp-async-5,e76b91db1ab9e534,68a632c0127df158] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:11.564 INFO 21062 --- [ocp-async-7,2b5e43771fccba11,b114a9bfcf823f70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:11.645 INFO 21062 --- [ocp-schedules-5,14d49f6387dcfd44,2f51dbb78d8eb142] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=5, affectRows=5 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=5, validCount=5 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=15, affectRows=15 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=26, affectRows=26 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=15, validCount=15 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=26, validCount=26 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=25, affectRows=25 2024-11-04 15:24:11.822 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=25, validCount=25 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=62, affectRows=62 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=62, validCount=62 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=44, affectRows=44 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=49, affectRows=49 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=44, validCount=44 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=49, validCount=49 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=68, affectRows=68 2024-11-04 15:24:11.823 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=68, affectRows=68 2024-11-04 15:24:11.824 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=67, validCount=67 2024-11-04 15:24:11.824 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730704980, nodeCount=60, affectRows=60 2024-11-04 15:24:11.824 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=68, validCount=68 2024-11-04 15:24:11.824 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730704980, nodeCount=58, validCount=58 2024-11-04 15:24:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0334eadada2d19d8,d767995cdb7aa0e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:12.065 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:12.066 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:12.997 INFO 21062 --- [ocp-schedules-26,d49b3bf8b53d2d5b,7fb72e84e29a2252] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:24:13.002 INFO 21062 --- [ocp-schedules-26,d49b3bf8b53d2d5b,7fb72e84e29a2252] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:24:13.002 INFO 21062 --- [ocp-schedules-26,d49b3bf8b53d2d5b,7fb72e84e29a2252] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:24:13.561 WARN 21062 --- [ocp-async-4,954536a405618a76,d6d88803950ecd38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:13.565 INFO 21062 --- [ocp-async-3,c5686d6a7dd95351,5dc8373958cdd809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:13.984 INFO 21062 --- [prometheus-cache-monitor1,51f2841a1d844ff5,871080ce43ead2ea] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:24:13.984 INFO 21062 --- [prometheus-cache-monitor1,51f2841a1d844ff5,871080ce43ead2ea] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:24:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d47d7021ce00fd63,1a19ff8213376a7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:15.066 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:15.067 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:15.561 WARN 21062 --- [ocp-async-1,83c2a629266313b0,12fc982f55071734] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:15.565 INFO 21062 --- [ocp-async-10,a3304d4aaeabaa46,4d1974b97e020146] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:16.228 INFO 21062 --- [ocp-schedules-5,f3ce7ed597db279e,e9fc1b9307e5042a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:24:17.561 WARN 21062 --- [ocp-async-2,fc0c4eddc2b7d162,448663f015c5730d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:17.565 INFO 21062 --- [ocp-async-8,3285a70e826a56b6,1fe62b0852089233] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:17.983 INFO 21062 --- [prometheus-minute-schedule-1,0dbecca4f7c253bd,2f496ecb545e6302] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:18.067 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:18.068 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:24:18.590139+08:00, lastProcessTime=2024-11-04T15:24:08.590304+08:00 2024-11-04 15:24:18.761 INFO 21062 --- [ocp-async-6,3f67b4a230057dbf,943dba803b08e227] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:24:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:24:18.590139+08:00 2024-11-04 15:24:19.561 WARN 21062 --- [ocp-async-4,a1e4844b920925a8,8872693277738e59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:19.565 INFO 21062 --- [ocp-async-3,cf207709e46c9058,41d1cde565dd4679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:20.983 INFO 21062 --- [prometheus-minute-schedule-1,107650f61e3f2b5a,211101ad973b404a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:21.068 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:21.069 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:21.561 WARN 21062 --- [ocp-async-1,b0910249745c9be2,9f8b641a184c3e7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:21.565 INFO 21062 --- [ocp-async-10,08fcba82cc6f7445,326eb2e2fcda4bb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:21.649 INFO 21062 --- [ocp-schedules-29,bcfd1148ea95fae6,91d0bff0fd97180b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:24:23.562 WARN 21062 --- [ocp-async-2,cee38808c97edb93,b4215d78ab93a5ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:23.565 INFO 21062 --- [ocp-async-8,5c2c13f0d45147e7,84765a23cd57c5f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:23.983 INFO 21062 --- [prometheus-minute-schedule-1,faa97845592d67b7,ed5d24e5be0429f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:24.069 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:24.070 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:25.562 WARN 21062 --- [ocp-async-7,488d270b7e5dd5e7,a5559146ffde8b43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:25.566 INFO 21062 --- [ocp-async-6,15670517aeb9ee14,83c889b0bfc5575b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:26.254 INFO 21062 --- [ocp-schedules-22,7f224a1b42c15ff6,87d960f31d3d0474] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:24:26.983 INFO 21062 --- [prometheus-minute-schedule-1,6419e7244ca91a8f,41715d9ff3a659e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:26.984 INFO 21062 --- [prometheus-hour-schedule-1,6b703afad2ac6d6b,844c9e20d3084224] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:24:27.070 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:27.071 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:27.562 WARN 21062 --- [ocp-async-3,b19d417b4394cdad,5e12d0ac144581da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:27.566 INFO 21062 --- [ocp-async-9,2e7e1f61087eec72,b550554900d0e32a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:28.498 INFO 21062 --- [ocp-schedules-3,f9650731cf3fb3b1,b807c333ea893150] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:24:28.499 INFO 21062 --- [ocp-schedules-5,40b5a7f99c622109,fd0a26eca6c6dd70] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:24:28.500 INFO 21062 --- [ocp-schedules-19,d5966da20458b68c,1ed63da180e71385] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:24:28.500 INFO 21062 --- [ocp-schedules-14,6be3de1165c00603,3bfef0c9b2d404b8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:24:28.502 INFO 21062 --- [ocp-schedules-3,f9650731cf3fb3b1,b807c333ea893150] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:24:28.502 INFO 21062 --- [ocp-schedules-2,f53a6897ff80c25a,a23d3d697d779959] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:24:28.504 INFO 21062 --- [ocp-schedules-14,6be3de1165c00603,3bfef0c9b2d404b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:28.504 INFO 21062 --- [ocp-schedules-19,d5966da20458b68c,1ed63da180e71385] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:24:28.505 INFO 21062 --- [ocp-schedules-24,a86a2e144f71db1b,30c36b4ef0e85561] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:24:28.505 INFO 21062 --- [ocp-schedules-5,40b5a7f99c622109,fd0a26eca6c6dd70] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:24:28.509 INFO 21062 --- [ocp-schedules-16,f09878530190c630,d0736be48822204e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:24:28.509 INFO 21062 --- [ocp-schedules-16,f09878530190c630,d0736be48822204e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:24:28.509 INFO 21062 --- [ocp-schedules-26,79a371047cda4eae,02ffcb3f8e98a1f1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:24:28.509 INFO 21062 --- [ocp-schedules-26,79a371047cda4eae,02ffcb3f8e98a1f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:24:28.509 INFO 21062 --- [ocp-schedules-9,b5787b9e40c13577,a3f4c755cd2713e2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:24:28.510 INFO 21062 --- [ocp-schedules-9,b5787b9e40c13577,a3f4c755cd2713e2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:24:28.588 INFO 21062 --- [ocp-schedules-21,40cf1fd03a3e452a,3636a29ebca9adb8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:24:28.588 INFO 21062 --- [ocp-schedules-21,40cf1fd03a3e452a,3636a29ebca9adb8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:24:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:24:28.589301+08:00, lastProcessTime=2024-11-04T15:24:18.590139+08:00 2024-11-04 15:24:28.620 INFO 21062 --- [ocp-schedules-11,c93d8e24cf25835b,9fb56ae296b2b576] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:24:28.620+08:00, tenantSize=0 2024-11-04 15:24:28.664 INFO 21062 --- [ocp-schedules-15,77fbfe88db347d92,2b5bd8f75da31485] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:24:28.665 INFO 21062 --- [ocp-schedules-15,77fbfe88db347d92,2b5bd8f75da31485] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:24:28.725 INFO 21062 --- [ocp-schedules-30,ecd3a739355d1380,45505b2965340df5] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 15:24:28.725 INFO 21062 --- [ocp-schedules-30,ecd3a739355d1380,45505b2965340df5] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 15:24:28.761 INFO 21062 --- [ocp-async-7,11eb471ed1a7d155,592c459a1b7f05b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:24:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:24:28.589301+08:00 2024-11-04 15:24:28.984 INFO 21062 --- [ocp-schedules-5,3257e3da597c60ac,805c0ed96ae3b4c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:24:29.376 INFO 21062 --- [ocp-schedules-28,40b10ed3cac58c19,ab7d257f46987983] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:24:29.376 INFO 21062 --- [ocp-schedules-28,40b10ed3cac58c19,ab7d257f46987983] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:24:29.512 INFO 21062 --- [ocp-schedules-11,02c2befb90975fe8,8064e07711228d7f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:24:29.532 INFO 21062 --- [ocp-schedules-17,dfe8bfd4dda91233,2bcd83b044e82031] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:24:29.538 INFO 21062 --- [ocp-schedules-17,dfe8bfd4dda91233,2bcd83b044e82031] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:24:29.538 INFO 21062 --- [ocp-schedules-17,dfe8bfd4dda91233,2bcd83b044e82031] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:24:29.562 WARN 21062 --- [ocp-async-8,9b06674d0951fa16,78e94d644cc9ca86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:29.566 INFO 21062 --- [ocp-async-4,b13af1f12b347870,734544df5de7da51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:29.983 INFO 21062 --- [prometheus-minute-schedule-1,ce70e355be1d939c,a9b66defddced9e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:30.007 INFO 21062 --- [ocp-schedules-10,e6a26627fcd50926,76825dfc7bae7d0f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:24:30.007 INFO 21062 --- [ocp-schedules-10,e6a26627fcd50926,76825dfc7bae7d0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:24:30.008 INFO 21062 --- [ocp-schedules-10,e6a26627fcd50926,76825dfc7bae7d0f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:24:30.008 INFO 21062 --- [ocp-schedules-10,e6a26627fcd50926,76825dfc7bae7d0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:24:30.009 INFO 21062 --- [ocp-schedules-10,e6a26627fcd50926,76825dfc7bae7d0f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:24:30.009 INFO 21062 --- [ocp-schedules-10,e6a26627fcd50926,76825dfc7bae7d0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:24:30.071 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:30.071 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:24:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:24:31.562 WARN 21062 --- [ocp-async-6,ce13bb3af24174b7,0fe397f3ae762247] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:31.566 INFO 21062 --- [ocp-async-1,de3f48d0bb0763ac,433b6ce69c8693fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:31.651 INFO 21062 --- [ocp-schedules-7,9b8af9a3353f9090,ee6755d31effec01] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:24:32.983 INFO 21062 --- [prometheus-minute-schedule-1,df010f249cc228d8,2bd670e14ba249c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:33.005 INFO 21062 --- [ocp-schedules-26,dfe1174c10d1312f,105eb66684256c7a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:24:33.010 INFO 21062 --- [ocp-schedules-26,dfe1174c10d1312f,105eb66684256c7a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:24:33.011 INFO 21062 --- [ocp-schedules-26,dfe1174c10d1312f,105eb66684256c7a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:24:33.072 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:33.073 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:33.236 INFO 21062 --- [ocp-schedules-17,40a7f68a3addf0b6,2f0b3c2d5503ee62] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:24:33.236 INFO 21062 --- [ocp-schedules-17,40a7f68a3addf0b6,2f0b3c2d5503ee62] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:24:33.262 INFO 21062 --- [ocp-schedules-27,61b67cf4e03be8cd,8b05f873a0ff5152] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:24:33.263 INFO 21062 --- [ocp-schedules-17,40a7f68a3addf0b6,2f0b3c2d5503ee62] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:24:33.263 INFO 21062 --- [ocp-schedules-17,40a7f68a3addf0b6,2f0b3c2d5503ee62] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:24:33.471 INFO 21062 --- [ocp-schedules-24,0d6be2abcff07d32,14eb5fb6714aac34] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:24:33.471+08:00, clusters=[1] 2024-11-04 15:24:33.473 INFO 21062 --- [ocp-schedules-24,0d6be2abcff07d32,14eb5fb6714aac34] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:24:33.563 WARN 21062 --- [ocp-async-10,be40cc978ac5d224,80f1847686818ddd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:33.566 INFO 21062 --- [ocp-async-2,48d3201227f73364,e4a18be8c6ae1575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:33.771 INFO 21062 --- [ocp-schedules-10,c60b503055585e91,cce1dc364d1a4c69] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:24:33.771+08:00, clusters=[1] 2024-11-04 15:24:33.774 INFO 21062 --- [ocp-schedules-10,c60b503055585e91,cce1dc364d1a4c69] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:24:33.813 INFO 21062 --- [ocp-schedules-1,f389a7b6dc2fe35b,4933fe8a089fd736] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:24:33.816 INFO 21062 --- [ocp-schedules-1,f389a7b6dc2fe35b,4933fe8a089fd736] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:33.873 ERROR 21062 --- [ocp-schedules-1,f389a7b6dc2fe35b,4933fe8a089fd736] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:24:33.873 INFO 21062 --- [ocp-schedules-1,f389a7b6dc2fe35b,4933fe8a089fd736] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:24:35.563 WARN 21062 --- [ocp-async-2,c617a16f3c8368dd,f41a5a83b0390c3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:35.566 INFO 21062 --- [ocp-async-7,ca7fa71328b6b2df,616a8ddc874f0631] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:35.584 INFO 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:24:35.584 INFO 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:24:35.584 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:24:35.584 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:24:35.594 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:24:35.595 INFO 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:24:35.595 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:24:35.596 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:24:35.597 WARN 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:24:35.597 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:24:35.597 INFO 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:24:35.597 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:24:35.597 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:24:35.598 WARN 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:24:35.599 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:24:35.599 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:24:35.606 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:24:35.606 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:24:35.606 INFO 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:24:35.618 ERROR 21062 --- [ocp-schedules-27,ee6220e107e4a1f4,23b75e6c232307f7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:24:35.983 INFO 21062 --- [prometheus-minute-schedule-1,3434bed418ec8033,812b3fbdbf56ec73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:36.073 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:36.074 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:36.295 INFO 21062 --- [ocp-schedules-15,61b75889d6ef7653,1a2cc114cc51ebfe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:24:37.563 WARN 21062 --- [ocp-async-7,a69796c0d68b4b37,54c9dcb749a2e5dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:37.567 INFO 21062 --- [ocp-async-6,268ccf15c3907c02,0ffde18ceaffc8e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:38.505 INFO 21062 --- [ocp-schedules-14,6be3de1165c00603,3bfef0c9b2d404b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:24:38.505 INFO 21062 --- [ocp-schedules-19,d5966da20458b68c,1ed63da180e71385] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:24:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:24:38.590122+08:00, lastProcessTime=2024-11-04T15:24:28.589301+08:00 2024-11-04 15:24:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:24:38.590122+08:00 2024-11-04 15:24:38.761 INFO 21062 --- [ocp-async-9,f1d1d703101ee5d7,e37ddfe22cf75581] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:24:38.983 INFO 21062 --- [prometheus-minute-schedule-1,6bc4e601ff341a87,7fa2fc337d48d8e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:39.074 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:39.075 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:39.563 WARN 21062 --- [ocp-async-10,b7bc635134bec230,fde45d57ded8e7fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:39.567 INFO 21062 --- [ocp-async-9,1086f1714bc85fc2,3839793af74b9fa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:40.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:24:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:24:40.280 INFO 21062 --- [ocp-schedules-12,759c02ccc315fe11,f222944990e32645] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:24:40.281 INFO 21062 --- [ocp-schedules-12,ed1396c0dad71ea2,3453bc34f5592ae5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:24:40.284 INFO 21062 --- [ocp-schedules-12,ed1396c0dad71ea2,3453bc34f5592ae5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:40.314 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:24:40.314 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:24:41.563 WARN 21062 --- [ocp-async-9,f1e56bc9f44ad950,8c691688f3fec175] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:41.567 INFO 21062 --- [ocp-async-7,512ec220f4e20f23,c491850979bd00c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:41.654 INFO 21062 --- [ocp-schedules-6,68a98eae31b2ea8b,61b958e3e4b96679] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:24:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:24:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:24:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:24:41.983 INFO 21062 --- [prometheus-minute-schedule-1,48d35edef8895b5d,e6f4a5e31155f62b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:42.075 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:42.075 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:43.563 WARN 21062 --- [ocp-async-7,84c51ffe35b5f4ca,a9bdc3dcc0ec9d2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:43.567 INFO 21062 --- [ocp-async-6,5fe822e2f9f7d403,1f88238121b508ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:44.429 INFO 21062 --- [ocp-schedules-9,da8a76d1de55009a,12860976cf291b5c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:24:44.432 INFO 21062 --- [ocp-schedules-9,da8a76d1de55009a,12860976cf291b5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:44.516 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:24:44.519 ERROR 21062 --- [ocp-schedules-9,da8a76d1de55009a,12860976cf291b5c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:24:44.519 INFO 21062 --- [ocp-schedules-9,da8a76d1de55009a,12860976cf291b5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:24:44.892 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:24:44.892 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:24:44.894 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.894 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:24:44.895 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.896 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.896 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.897 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:24:44.911 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.912 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.912 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.913 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:24:44.926 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.926 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.927 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.927 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:24:44.942 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.942 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.943 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.943 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:24:44.956 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.956 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.957 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.957 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:24:44.963 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.963 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.964 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.964 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:24:44.968 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.968 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.969 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.969 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:24:44.973 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.974 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.974 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.975 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:24:44.981 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.981 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.981 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.982 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:24:44.983 INFO 21062 --- [prometheus-minute-schedule-1,672c7db170b3ee62,e22b6b130625b3df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:44.987 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.987 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.988 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.988 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:24:44.993 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.994 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:44.995 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:44.995 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:24:45.003 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.003 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.004 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.004 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:24:45.018 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.018 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.019 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.019 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:24:45.024 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.024 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.025 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.025 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:24:45.033 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.034 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.034 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.034 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:24:45.040 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.041 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.041 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.042 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:24:45.046 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.046 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.047 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.047 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:24:45.051 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.051 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.052 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.052 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:24:45.056 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.056 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.057 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.057 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:24:45.062 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.062 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.063 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.063 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:24:45.067 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.068 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.068 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.069 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:24:45.073 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.074 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.074 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.074 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:24:45.076 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:45.076 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:45.078 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.079 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.079 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.080 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:24:45.084 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.084 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.085 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.085 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:24:45.089 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.089 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.090 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.090 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:24:45.096 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.096 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.097 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.097 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:24:45.102 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.102 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.103 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.103 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:24:45.107 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.108 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.108 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.109 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:24:45.113 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.114 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.114 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.114 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:24:45.118 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.119 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.119 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.120 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:24:45.124 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.124 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.125 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.125 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:24:45.129 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.130 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.130 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.131 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:24:45.135 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.135 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.136 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.136 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:24:45.140 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.140 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.141 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.141 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:24:45.148 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.149 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.149 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.150 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:24:45.155 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.155 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.156 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.156 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:24:45.162 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.162 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.163 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.163 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:24:45.168 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.169 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.169 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.170 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:24:45.173 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.174 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.175 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.175 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:24:45.178 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.179 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.179 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.180 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:24:45.183 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.184 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.184 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.185 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:24:45.188 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.189 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.189 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.190 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:24:45.200 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.200 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.201 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.201 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:24:45.220 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.221 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.221 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.222 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:24:45.227 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.228 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.228 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.228 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:24:45.234 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.234 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.235 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.235 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:24:45.241 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.241 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.242 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.242 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:24:45.246 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.247 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.247 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.248 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:24:45.251 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.251 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.252 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.252 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:24:45.256 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:24:45.256 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:24:45.258 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.258 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:24:45.259 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.260 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.260 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.260 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:24:45.266 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.266 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:24:45.267 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:24:45.267 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:24:45.271 INFO 21062 --- [ocp-schedules-22,263e9330963aaaed,c0ce9fe05800d736] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:24:45.563 WARN 21062 --- [ocp-async-1,6f15b79e2051ef6c,07f8d10be77a21fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:45.567 INFO 21062 --- [ocp-async-2,9d43048bfa232f6a,f2508608758b4397] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:46.322 INFO 21062 --- [ocp-schedules-4,2a1bcbeed198c7a3,7d0774904a7af4ee] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:24:47.564 WARN 21062 --- [ocp-async-8,d6fd57fe18981401,23ba2988722aaa45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:47.567 INFO 21062 --- [ocp-async-5,18b0e419cc8dcd18,b1c1f1ed5966a9ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:47.983 INFO 21062 --- [prometheus-minute-schedule-1,32b94040eaed25df,73bcc44306d9ad07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:48.076 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:48.077 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:24:48.590526+08:00, lastProcessTime=2024-11-04T15:24:38.590122+08:00 2024-11-04 15:24:48.761 INFO 21062 --- [ocp-async-4,208fd13e90bee8a1,e5ddf715f9f9c1f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:24:48.775 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:24:48.590526+08:00 2024-11-04 15:24:49.564 WARN 21062 --- [ocp-async-7,4ff619a599c38e9c,fbf04e25bb1a68cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:49.567 INFO 21062 --- [ocp-async-6,29a1fff7a66bffa7,4aff97f9edd14fa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:50.983 INFO 21062 --- [prometheus-minute-schedule-1,56d0aa205cc3d197,305f456d2f05f8f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:51.077 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:51.078 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:51.564 WARN 21062 --- [ocp-async-1,2d6da58c1afa0a5d,c509c28a6cb81e4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:51.568 INFO 21062 --- [ocp-async-2,cd4f3f1eeab55897,fab90a00b8a355dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:51.657 INFO 21062 --- [ocp-schedules-24,3fffc1755a1e143e,00548a83b5edb42d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:24:53.014 INFO 21062 --- [ocp-schedules-28,d251b468685bfaca,a646ec1da2414ed3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:24:53.021 INFO 21062 --- [ocp-schedules-28,d251b468685bfaca,a646ec1da2414ed3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:24:53.021 INFO 21062 --- [ocp-schedules-28,d251b468685bfaca,a646ec1da2414ed3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:24:53.564 WARN 21062 --- [ocp-async-8,daeb40fda239bf93,72a88e2389a50ebd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:53.568 INFO 21062 --- [ocp-async-5,49ed11c193175a4e,e86bc865df93b269] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:53.983 INFO 21062 --- [prometheus-minute-schedule-1,f7e68018268bed8b,5e2c484a3713c863] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:54.078 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:54.079 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:55.564 WARN 21062 --- [ocp-async-3,b8f15a89e6eb1fb2,ec579bba10f5b5e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:55.568 INFO 21062 --- [ocp-async-4,bf3cf0dac22e29eb,a550927318d6bef8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:56.365 INFO 21062 --- [ocp-schedules-9,5bab37c3f9c025b8,f293005af7bf681b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:24:56.983 INFO 21062 --- [prometheus-minute-schedule-1,6940df9f2f33f2e5,f77d9dffcd81bc31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:24:57.079 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:24:57.080 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:24:57.565 WARN 21062 --- [ocp-async-6,8c7906d2796e486a,20cd0a73f3f68f20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:57.568 INFO 21062 --- [ocp-async-10,39af30268e8ee606,517673519f21d669] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:58.500 INFO 21062 --- [ocp-schedules-20,895c5a5b49b263ff,a714822ea1a3164d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:24:58.500 INFO 21062 --- [ocp-schedules-3,b09f82980804c211,61717185cfb443af] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:24:58.502 INFO 21062 --- [ocp-schedules-2,3636c8608d154dd8,0ea23e4bacb1a416] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:24:58.504 INFO 21062 --- [ocp-schedules-20,895c5a5b49b263ff,a714822ea1a3164d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:24:58.505 INFO 21062 --- [ocp-schedules-3,b09f82980804c211,61717185cfb443af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:58.505 INFO 21062 --- [ocp-schedules-11,68b3770cfbd2a9ba,95f95dbedbca79b4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:24:58.510 INFO 21062 --- [ocp-schedules-17,8f7f6abe3e06985b,8cb11fc2d3e6132c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:24:58.510 INFO 21062 --- [ocp-schedules-28,5e7d590a0a3b54de,1a8722a36a3fd27f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:24:58.510 INFO 21062 --- [ocp-schedules-23,9c17f4a5b7c484e4,917206e00bcb0372] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:24:58.510 INFO 21062 --- [ocp-schedules-17,8f7f6abe3e06985b,8cb11fc2d3e6132c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:24:58.510 INFO 21062 --- [ocp-schedules-23,9c17f4a5b7c484e4,917206e00bcb0372] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:24:58.510 INFO 21062 --- [ocp-schedules-28,5e7d590a0a3b54de,1a8722a36a3fd27f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:24:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:24:58.589729+08:00, lastProcessTime=2024-11-04T15:24:48.590526+08:00 2024-11-04 15:24:58.761 INFO 21062 --- [ocp-async-9,b7d7f71a885bd1ae,291efbbeada62021] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:24:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:24:58.589729+08:00 2024-11-04 15:24:59.314 INFO 21062 --- [ocp-schedules-14,28bfde5338137cd8,d889c0e3e4b31b9d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:24:59.318 INFO 21062 --- [ocp-schedules-14,28bfde5338137cd8,d889c0e3e4b31b9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:24:59.318 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:24:59.515 INFO 21062 --- [ocp-schedules-17,f381b72aa813a227,a8eac4a723e2be02] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:24:59.565 WARN 21062 --- [ocp-async-8,a1f62ca8e591d3f5,14e4e2bd5413c5dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:24:59.568 INFO 21062 --- [ocp-async-5,f6d42105781610bd,9604bbaf4a1cfa11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:24:59.983 INFO 21062 --- [prometheus-minute-schedule-1,867122171f8144a4,091f5bf35bcc937c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705100, interval=60 2024-11-04 15:25:00.080 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:00.080 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:00.720 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8b6533bcd3c99a51] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:20:01.0, nextRun=2024-11-04 15:25:00.0, currentDate=Mon Nov 04 15:25:00 CST 2024 2024-11-04 15:25:00.721 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8b6533bcd3c99a51] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:20:01.0, nextRunTime=2024-11-04 15:25:00.0} 2024-11-04 15:25:00.748 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8b6533bcd3c99a51] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:25:00.752 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8b6533bcd3c99a51] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:25:00.752 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,8b6533bcd3c99a51] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:25:01.565 WARN 21062 --- [ocp-async-3,0540db4b4fd5fc7c,b5bf5fc9cad756cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:01.568 INFO 21062 --- [ocp-async-4,82d686d731921f94,b00f6bab305eaa6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:01.660 INFO 21062 --- [ocp-schedules-5,54fdece357abd8b5,f9bb28ed2282dc74] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:25:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c68c4a572a849ca0,1801d0613ead2a6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:03.081 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:03.081 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:03.565 WARN 21062 --- [ocp-async-6,cd3c5f2b7970a038,03c0f4041a9eff3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:03.569 INFO 21062 --- [ocp-async-10,23acd42b4a612daa,8f0ef04f54e812ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:05.565 WARN 21062 --- [ocp-async-2,9175cc050cab0c37,20ff4676f6c2229c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:05.569 INFO 21062 --- [ocp-async-9,539afd8928050423,b88e5400ae893a14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:05.983 INFO 21062 --- [prometheus-minute-schedule-1,055cc8b49c94ef75,8b227e9b01420cd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:06.081 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:06.082 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:06.392 INFO 21062 --- [ocp-schedules-27,7502b0934da91330,7a6277eecc50581c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:25:07.565 WARN 21062 --- [ocp-async-5,fc954b0a4788a346,b6bb641df9dee694] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:07.569 INFO 21062 --- [ocp-async-7,4fbc15cc37206753,3f80b58d2824a9e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:08.505 INFO 21062 --- [ocp-schedules-20,895c5a5b49b263ff,a714822ea1a3164d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:25:08.505 INFO 21062 --- [ocp-schedules-3,b09f82980804c211,61717185cfb443af] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:25:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:25:08.591583+08:00, lastProcessTime=2024-11-04T15:24:58.589729+08:00 2024-11-04 15:25:08.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:25:08.591583+08:00 2024-11-04 15:25:08.762 INFO 21062 --- [ocp-async-1,edbbb2825352a902,4b438ad9e45c3ff6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:25:08.983 INFO 21062 --- [prometheus-minute-schedule-1,3d6a1ffc504b01ad,e54935a4325ed0d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:09.082 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:09.083 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:09.319 INFO 21062 --- [ocp-schedules-14,28bfde5338137cd8,d889c0e3e4b31b9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:25:09.566 WARN 21062 --- [ocp-async-6,5fb06faa8fd3d114,4b024354866e10cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:09.569 INFO 21062 --- [ocp-async-10,3c20483c877ba13f,6d008316bfa34a2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:10.004 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:25:10.004 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:25:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:25:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:25:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:25:10.314 INFO 21062 --- [ocp-schedules-12,ed1396c0dad71ea2,3453bc34f5592ae5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:25:10.314 INFO 21062 --- [ocp-schedules-12,7660d2e33978ee95,9d6584b6179bd90c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:25:10.318 INFO 21062 --- [ocp-schedules-12,7660d2e33978ee95,9d6584b6179bd90c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:25:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:25:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:25:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:25:10.346 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:25:10.346 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:25:11.566 WARN 21062 --- [ocp-async-2,365e423370d5c352,5c656fb7006fceda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:11.569 INFO 21062 --- [ocp-async-9,82e524c2a94f57dd,b5e742938a936a43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:11.664 INFO 21062 --- [ocp-schedules-15,5c08136744857434,62bcfa6dcffe6ec5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:25:11.831 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=10, affectRows=10 2024-11-04 15:25:11.831 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=1, affectRows=1 2024-11-04 15:25:11.831 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=1, validCount=1 2024-11-04 15:25:11.831 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=10, validCount=10 2024-11-04 15:25:11.831 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=19, affectRows=19 2024-11-04 15:25:11.831 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=21, affectRows=21 2024-11-04 15:25:11.831 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=19, validCount=19 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=31, affectRows=31 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=21, validCount=21 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=25, affectRows=25 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=31, validCount=31 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=25, validCount=25 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=27, affectRows=27 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=38, affectRows=38 2024-11-04 15:25:11.832 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=80, affectRows=80 2024-11-04 15:25:11.833 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=27, validCount=27 2024-11-04 15:25:11.833 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=38, validCount=38 2024-11-04 15:25:11.833 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=79, affectRows=79 2024-11-04 15:25:11.833 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=79, validCount=79 2024-11-04 15:25:11.833 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=79, validCount=79 2024-11-04 15:25:11.833 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705040, nodeCount=91, affectRows=91 2024-11-04 15:25:11.834 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705040, nodeCount=89, validCount=89 2024-11-04 15:25:11.983 INFO 21062 --- [prometheus-minute-schedule-1,57ab37761dcc1682,e133cf30ee2db2d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:12.083 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:12.084 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:13.023 INFO 21062 --- [ocp-schedules-6,53b5472498cc7e5a,2b2be0d3920add4d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:25:13.028 INFO 21062 --- [ocp-schedules-6,53b5472498cc7e5a,2b2be0d3920add4d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:25:13.028 INFO 21062 --- [ocp-schedules-6,53b5472498cc7e5a,2b2be0d3920add4d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:25:13.566 WARN 21062 --- [ocp-async-5,4d861f35b2afd3d0,f801943853da6fba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:13.570 INFO 21062 --- [ocp-async-7,c811151f68a4cb41,248806f97a0c5dd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:13.985 INFO 21062 --- [prometheus-cache-monitor1,9af85fa4ca79e6c0,38992f0b6cfb6f49] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:25:13.985 INFO 21062 --- [prometheus-cache-monitor1,9af85fa4ca79e6c0,38992f0b6cfb6f49] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:25:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a6eb0ca3caf18ba8,17187f9ceddcba38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:15.084 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:15.085 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:15.566 WARN 21062 --- [ocp-async-4,9df77fe1b1fd0b5f,692e1c06415c3cb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:15.570 INFO 21062 --- [ocp-async-1,293a69c377c08c90,c7f65d45c4dbaff3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:15.622 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:21:15.605, end=2024-11-04T15:23:15.619} 2024-11-04 15:25:15.626 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:25:15.626 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:25:16.433 INFO 21062 --- [ocp-schedules-22,4737a7d6d5082728,303c0e47d35e0722] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:25:17.566 WARN 21062 --- [ocp-async-10,63102edae370131e,5f06eea90741c670] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:17.570 INFO 21062 --- [ocp-async-8,4ed697388ae578ad,ee9626fcaa349160] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:17.983 INFO 21062 --- [prometheus-minute-schedule-1,a72669eaad0a13cd,2d86e5eac478074d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:18.085 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:18.086 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:25:18.590325+08:00, lastProcessTime=2024-11-04T15:25:08.591583+08:00 2024-11-04 15:25:18.762 INFO 21062 --- [ocp-async-3,a6470ff1154e2c79,294516862527226a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:25:18.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:25:18.590325+08:00 2024-11-04 15:25:19.566 WARN 21062 --- [ocp-async-5,e04942f4a8bc5604,fc94e28ba20a1f6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:19.570 INFO 21062 --- [ocp-async-7,7b69a57663ad1e43,350b9b506b4d69b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:20.983 INFO 21062 --- [prometheus-minute-schedule-1,9ddb893effce6aa9,550cbcc1957cc46f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:21.086 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:21.086 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:21.567 WARN 21062 --- [ocp-async-4,5217421b73baf5a5,a77494e7aecdcf84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:21.571 INFO 21062 --- [ocp-async-1,a6b50db29ae220ce,3ad06ba89a0dabb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:21.667 INFO 21062 --- [ocp-schedules-25,59ebe83e64d4ee12,134c74b8552b9ef3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:25:22.982 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:25:22.982 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:25:22.990 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:25:22.990 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:25:22.990 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:25:22.990 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:25:22.996 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:25:22.996 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:25:22.996 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:25:22.996 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:25:23.033 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:25:23.042 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:25:23.042 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:25:23.042 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:25:23.042 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:25:23.042 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:25:23.050 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:25:23.567 WARN 21062 --- [ocp-async-10,7cbed6fb2ae3b648,6109e00d48af7780] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:23.571 INFO 21062 --- [ocp-async-8,95861bc99619a67d,cbad9b77e559afc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:23.596 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:25:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8312b239f644a1ff,bc9beef4950156f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:24.087 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:24.087 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:24.153 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.156915386S 2024-11-04 15:25:24.153 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:25:24.250 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:25:24.250 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:25:24.277 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:25:24.338 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:25:24.338 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:25:24.338 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:25:24.338 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:25:24.338 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:25:24.338 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:25:24.349 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:25:24.521 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] o.s.boot.SpringApplication : Started application in 1.53 seconds (JVM running for 10538.2) 2024-11-04 15:25:24.521 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.531206829S 2024-11-04 15:25:24.974 INFO 21062 --- [ocp-schedules-22,f889a307c08988b0,0831d1ff351f29a6] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:25:25.567 WARN 21062 --- [ocp-async-9,8486a05a28423593,4676bc551d1fdefd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:25.571 INFO 21062 --- [ocp-async-3,b4c3e4ff3c6b0d4d,e23611428fe6b0e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:26.461 INFO 21062 --- [ocp-schedules-15,f8270725117817aa,e45c0347eec13489] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:25:26.983 INFO 21062 --- [prometheus-minute-schedule-1,82f7b1ca725dd715,ebd166fed7323ee9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:26.984 INFO 21062 --- [prometheus-hour-schedule-1,fe26fc8bb2023d80,5fb9772bc288b757] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:25:27.087 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:27.088 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:27.567 WARN 21062 --- [ocp-async-7,ad0fb9021d9cb452,3bb4699b4178ab4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:27.571 INFO 21062 --- [ocp-async-6,e76638e220fa3f46,009059902b27c459] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:28.498 INFO 21062 --- [ocp-schedules-25,a917fa2b7021ecd0,033c06bfd2faa34b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:25:28.500 INFO 21062 --- [ocp-schedules-16,9c62f8087949d6e9,649303a5fc641f92] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:25:28.500 INFO 21062 --- [ocp-schedules-3,2439191e7f075037,9785028f0c5a0772] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:25:28.500 INFO 21062 --- [ocp-schedules-9,9e60bdf121882d67,a0a8770ca7836c9b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:25:28.502 INFO 21062 --- [ocp-schedules-25,a917fa2b7021ecd0,033c06bfd2faa34b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:25:28.502 INFO 21062 --- [ocp-schedules-21,f9f4e35c5a439b92,e62663fe407ff07d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:25:28.503 INFO 21062 --- [ocp-schedules-16,9c62f8087949d6e9,649303a5fc641f92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:25:28.504 INFO 21062 --- [ocp-schedules-9,9e60bdf121882d67,a0a8770ca7836c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:25:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:25:28.504 INFO 21062 --- [ocp-schedules-18,7dc410c1e61f0de8,150da599820b291a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:25:28.505 INFO 21062 --- [ocp-schedules-3,2439191e7f075037,9785028f0c5a0772] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:25:28.510 INFO 21062 --- [ocp-schedules-10,f56c43f0c1e92cf5,206b82859ace9711] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:25:28.510 INFO 21062 --- [ocp-schedules-17,b0bc508b4e6bc29e,d11bb02c621f121e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:25:28.510 INFO 21062 --- [ocp-schedules-24,9b5c9c00ac6a04cd,d4364718cc5d1635] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:25:28.511 INFO 21062 --- [ocp-schedules-17,b0bc508b4e6bc29e,d11bb02c621f121e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:25:28.511 INFO 21062 --- [ocp-schedules-24,9b5c9c00ac6a04cd,d4364718cc5d1635] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:25:28.511 INFO 21062 --- [ocp-schedules-10,f56c43f0c1e92cf5,206b82859ace9711] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:25:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:25:28.589075+08:00, lastProcessTime=2024-11-04T15:25:18.590325+08:00 2024-11-04 15:25:28.762 INFO 21062 --- [ocp-async-8,47e064c85b3be975,eff6520ba5367338] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:25:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:25:28.589075+08:00 2024-11-04 15:25:28.987 INFO 21062 --- [ocp-schedules-11,4c7c125c09f3d82d,29a19d144cab70ea] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:25:29.380 INFO 21062 --- [ocp-schedules-25,6d1c35046ae6100e,523986216aa9dc23] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:25:29.380 INFO 21062 --- [ocp-schedules-25,6d1c35046ae6100e,523986216aa9dc23] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:25:29.518 INFO 21062 --- [ocp-schedules-2,3d85aad113d79328,c23be176e217c4c6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:25:29.538 INFO 21062 --- [ocp-schedules-22,54e0e629d60ce2b2,bf34d339f87394f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:25:29.544 INFO 21062 --- [ocp-schedules-22,54e0e629d60ce2b2,bf34d339f87394f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:25:29.544 INFO 21062 --- [ocp-schedules-22,54e0e629d60ce2b2,bf34d339f87394f7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:25:29.567 WARN 21062 --- [ocp-async-9,6bab4917535cca0d,645c74acedae8a0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:29.571 INFO 21062 --- [ocp-async-5,402d1169f966274b,9811b2693076253b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:29.830 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:25:29.831 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:25:29.840 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:25:29.840 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:25:29.840 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:25:29.852 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:25:29.852 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:25:29.852 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:25:29.856 INFO 21062 --- [ocp-schedules-7,7e83e7f6fe6f34d3,3b71767b8ae1de68] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:25:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b5dc35393ca380c3,cc3c7920a66188bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:30.010 INFO 21062 --- [ocp-schedules-15,3f05277e0c2504ab,636214fe6247012d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:25:30.010 INFO 21062 --- [ocp-schedules-15,3f05277e0c2504ab,636214fe6247012d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:25:30.016 INFO 21062 --- [ocp-schedules-15,3f05277e0c2504ab,636214fe6247012d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:25:30.016 INFO 21062 --- [ocp-schedules-15,3f05277e0c2504ab,636214fe6247012d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:25:30.021 INFO 21062 --- [ocp-schedules-15,3f05277e0c2504ab,636214fe6247012d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:25:30.021 INFO 21062 --- [ocp-schedules-15,3f05277e0c2504ab,636214fe6247012d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:25:30.089 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:30.089 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:25:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:25:31.568 WARN 21062 --- [ocp-async-5,2de3762ca2c952e7,5efba7f2716aa1f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:31.571 INFO 21062 --- [ocp-async-4,eb341d65e1450265,7a63c05416ce46df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:31.669 INFO 21062 --- [ocp-schedules-25,6d7299ca2f9dc6a6,8861d6745d9c11a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:25:32.983 INFO 21062 --- [prometheus-minute-schedule-1,c9cd1e75c45fe956,2455f99ce2258c1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:33.031 INFO 21062 --- [ocp-schedules-15,46937ca7d04c1f94,58a284d77c71f5d9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:25:33.037 INFO 21062 --- [ocp-schedules-15,46937ca7d04c1f94,58a284d77c71f5d9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:25:33.037 INFO 21062 --- [ocp-schedules-15,46937ca7d04c1f94,58a284d77c71f5d9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:25:33.089 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:33.090 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:33.263 INFO 21062 --- [ocp-schedules-23,bcf3c4e53d95253a,9113ba63b64108ca] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:25:33.263 INFO 21062 --- [ocp-schedules-23,bcf3c4e53d95253a,9113ba63b64108ca] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:25:33.292 INFO 21062 --- [ocp-schedules-23,bcf3c4e53d95253a,9113ba63b64108ca] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:25:33.292 INFO 21062 --- [ocp-schedules-23,bcf3c4e53d95253a,9113ba63b64108ca] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:25:33.292 INFO 21062 --- [ocp-schedules-28,9d9ca0c65c011c6f,728d75796815447d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:25:33.500 INFO 21062 --- [ocp-schedules-13,d04a10c1a6f35ea7,dea3e84749360f2d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:25:33.500+08:00, clusters=[1] 2024-11-04 15:25:33.502 INFO 21062 --- [ocp-schedules-13,d04a10c1a6f35ea7,dea3e84749360f2d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:25:33.568 WARN 21062 --- [ocp-async-4,f1d97ce211dedd7f,f48a516ab8f15bac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:33.571 INFO 21062 --- [ocp-async-6,ea6df29fb9524432,945091ce6ff7ee03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:33.804 INFO 21062 --- [ocp-schedules-10,a9897ab2fac48ef1,f310f99f4988afc4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:25:33.804+08:00, clusters=[1] 2024-11-04 15:25:33.807 INFO 21062 --- [ocp-schedules-10,a9897ab2fac48ef1,f310f99f4988afc4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:25:35.568 WARN 21062 --- [ocp-async-6,3f0c98e3d98c58d4,be746de87f93eec3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:35.572 INFO 21062 --- [ocp-async-9,e22e89df3eeea15f,8b9ff242172395c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:35.627 INFO 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:25:35.627 INFO 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:25:35.627 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:25:35.627 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:25:35.637 INFO 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:25:35.637 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:25:35.638 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:25:35.638 WARN 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:25:35.639 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:25:35.639 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:25:35.639 INFO 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:25:35.639 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:25:35.640 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:25:35.640 WARN 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:25:35.640 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:25:35.641 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:25:35.648 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:25:35.648 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:25:35.648 INFO 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:25:35.659 ERROR 21062 --- [ocp-schedules-28,b04650453752ae2c,bb53b27ff9180fc4] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:25:35.983 INFO 21062 --- [prometheus-minute-schedule-1,56f395d877f15b02,c6486e910d185763] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:36.090 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:36.091 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:36.505 INFO 21062 --- [ocp-schedules-7,2110ce0b8f75937c,b210a9ded33e086a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:25:37.568 WARN 21062 --- [ocp-async-9,62bd8d92f36999b1,bcf065b78939ae5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:37.572 INFO 21062 --- [ocp-async-3,ae0d0f5b2b844968,27251e89667dac13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:38.504 INFO 21062 --- [ocp-schedules-9,9e60bdf121882d67,a0a8770ca7836c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:25:38.504 INFO 21062 --- [ocp-schedules-16,9c62f8087949d6e9,649303a5fc641f92] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:25:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:25:38.589731+08:00, lastProcessTime=2024-11-04T15:25:28.589075+08:00 2024-11-04 15:25:38.762 INFO 21062 --- [ocp-async-1,8efee8cbcb650a8d,bbe8700e273ac07d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:25:38.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:25:38.589731+08:00 2024-11-04 15:25:38.983 INFO 21062 --- [prometheus-minute-schedule-1,25a33e4c7548e261,090318acb95096af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:39.091 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:39.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:39.568 WARN 21062 --- [ocp-async-10,3f728fb39ddd8e2e,053bf50c974a9e2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:39.572 INFO 21062 --- [ocp-async-6,fa4f55d621aa16ee,0d036f47a062a450] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:40.002 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:25:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:25:40.346 INFO 21062 --- [ocp-schedules-12,7660d2e33978ee95,9d6584b6179bd90c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:25:40.347 INFO 21062 --- [ocp-schedules-12,8538e728a4db8379,0edddaf9317b026a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:25:40.350 INFO 21062 --- [ocp-schedules-12,8538e728a4db8379,0edddaf9317b026a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:25:40.378 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:25:40.378 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:25:41.569 WARN 21062 --- [ocp-async-6,bf85e72ac01ee25a,55736eef3912873f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:41.572 INFO 21062 --- [ocp-async-9,7648ce90b6c1d5cf,eadfcb7551cb5188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:41.672 INFO 21062 --- [ocp-schedules-24,e320f03aa4887dfd,316b3daff0ffbec7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:25:41.983 INFO 21062 --- [prometheus-minute-schedule-1,4ac9926e5a5b4b73,03834b0b54ba0944] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:42.092 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:42.093 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:43.569 WARN 21062 --- [ocp-async-5,f8fa84e973f16c42,4fbc4eb0e7e6a16f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:43.572 INFO 21062 --- [ocp-async-3,e346eda9a3e7e38b,7421aaaac776735a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:44.519 INFO 21062 --- [ocp-schedules-28,0aee6149ea584dd9,3321ae3d9b950e4a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:25:44.523 INFO 21062 --- [ocp-schedules-28,0aee6149ea584dd9,3321ae3d9b950e4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:25:44.607 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:25:44.611 ERROR 21062 --- [ocp-schedules-28,0aee6149ea584dd9,3321ae3d9b950e4a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:25:44.611 INFO 21062 --- [ocp-schedules-28,0aee6149ea584dd9,3321ae3d9b950e4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:25:44.983 INFO 21062 --- [prometheus-minute-schedule-1,eb2796ce2c2342c5,e9ba53a56a487c82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:45.093 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:45.094 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:45.569 WARN 21062 --- [ocp-async-4,89b05cbb654fe6e0,945a8f898dd717af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:45.573 INFO 21062 --- [ocp-async-1,56136b23948108e3,65ca12d8c6368041] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:46.529 INFO 21062 --- [ocp-schedules-11,452499a9da8196c9,f735b55003e8a90b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:25:47.569 WARN 21062 --- [ocp-async-2,d510b4e80483ea91,d0f61052fc2aaf45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:47.573 INFO 21062 --- [ocp-async-8,d4167fe6bec3a1ff,df84c99843fa2c38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b4c0f673b027f828,5fdc4ced039b1f16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:48.094 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:48.094 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:25:48.590470+08:00, lastProcessTime=2024-11-04T15:25:38.589731+08:00 2024-11-04 15:25:48.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:25:48.590470+08:00 2024-11-04 15:25:48.762 INFO 21062 --- [ocp-async-7,25a1a561dca68cf7,36c7f3c114c9a4e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:25:49.570 WARN 21062 --- [ocp-async-5,a7ae65951e06d8a0,84876c07a163f10e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:49.573 INFO 21062 --- [ocp-async-3,0446ea3e79447cf7,7481cd6af08ccbdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:50.983 INFO 21062 --- [prometheus-minute-schedule-1,5fe0ff2130ec43bf,df7f48b07d70b875] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:51.095 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:51.095 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:51.570 WARN 21062 --- [ocp-async-4,034733966c4552d3,f344864a3c44bbf1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:51.573 INFO 21062 --- [ocp-async-1,dd14829decbea766,c87a6ef8e7e14c99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:51.674 INFO 21062 --- [ocp-schedules-9,4935c41ff32b6a89,82461528737b7515] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:25:53.040 INFO 21062 --- [ocp-schedules-8,fd2518faa43db10f,aeb9277712c5e6e4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:25:53.046 INFO 21062 --- [ocp-schedules-8,fd2518faa43db10f,aeb9277712c5e6e4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:25:53.046 INFO 21062 --- [ocp-schedules-8,fd2518faa43db10f,aeb9277712c5e6e4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:25:53.570 WARN 21062 --- [ocp-async-2,bea20e920595893d,fb16d2675ab3c13e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:53.573 INFO 21062 --- [ocp-async-8,ab950e16c6f4644c,c066c85c03669161] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:53.983 INFO 21062 --- [prometheus-minute-schedule-1,b65c50c7016a9a21,011f97dc974fd2df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:54.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:54.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:55.570 WARN 21062 --- [ocp-async-9,0394b522b0245c96,6c92e45f490f403e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:55.574 INFO 21062 --- [ocp-async-7,9338763ccf8d5348,b5f4712892f1d4c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:56.569 INFO 21062 --- [ocp-schedules-16,4f642586faacf16d,feda05b1df2529d6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:25:56.983 INFO 21062 --- [prometheus-minute-schedule-1,940946c1cf5479b5,699330423747b710] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:25:57.096 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:25:57.097 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:25:57.570 WARN 21062 --- [ocp-async-3,ceaae1e69eaec663,b961bdee5d638e27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:57.574 INFO 21062 --- [ocp-async-10,f2bf8b3f855ffbf0,2f8fb3ad887ddf3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:58.500 INFO 21062 --- [ocp-schedules-17,711fd9ff0b2314eb,1f24ab3503b2c02c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:25:58.500 INFO 21062 --- [ocp-schedules-3,9bae46e3cb025461,0bd883f915dc063a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:25:58.502 INFO 21062 --- [ocp-schedules-24,d25c53d19a8b256f,6824e006662c6ac6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:25:58.504 INFO 21062 --- [ocp-schedules-3,9bae46e3cb025461,0bd883f915dc063a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:25:58.504 INFO 21062 --- [ocp-schedules-17,711fd9ff0b2314eb,1f24ab3503b2c02c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:25:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:25:58.505 INFO 21062 --- [ocp-schedules-8,2c6cd3f0788201d7,61970e85b24a34c0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:25:58.509 INFO 21062 --- [ocp-schedules-14,83f13a1929de2a51,bd64f1c49ecfb79f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:25:58.509 INFO 21062 --- [ocp-schedules-14,83f13a1929de2a51,bd64f1c49ecfb79f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:25:58.509 INFO 21062 --- [ocp-schedules-10,2940f76ffe061994,434896cb74fc84da] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:25:58.509 INFO 21062 --- [ocp-schedules-10,2940f76ffe061994,434896cb74fc84da] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:25:58.509 INFO 21062 --- [ocp-schedules-28,5a0c8085d99f150c,faabe4ead0de0b9c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:25:58.509 INFO 21062 --- [ocp-schedules-28,5a0c8085d99f150c,faabe4ead0de0b9c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:25:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:25:58.589292+08:00, lastProcessTime=2024-11-04T15:25:48.590470+08:00 2024-11-04 15:25:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:25:58.589292+08:00 2024-11-04 15:25:58.762 INFO 21062 --- [ocp-async-6,edb3b9e812c2e01a,f355b68141949469] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:25:59.520 INFO 21062 --- [ocp-schedules-24,a6237d78f1bb27d6,4f70aa444608c8dc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:25:59.571 WARN 21062 --- [ocp-async-2,c6ea566eeaa74006,7c2bbb92fbf49a75] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:25:59.574 INFO 21062 --- [ocp-async-8,be875f08b4a67d4a,fc222f77d1f8896f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:25:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e8102c02f6748ea3,3351bac3bef3d876] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705160, interval=60 2024-11-04 15:26:00.097 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:00.098 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:01.571 WARN 21062 --- [ocp-async-9,afc11b0ce5386170,c822063c065a79de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:01.574 INFO 21062 --- [ocp-async-7,90f3d7dfc04205be,5a3cf71f6b2a92c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:01.679 INFO 21062 --- [ocp-schedules-26,7f0f61385dab0cf7,3009c244c3391329] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:26:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4e39d423fb3cc022,446abeb3f95b8637] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:03.098 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:03.099 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:03.571 WARN 21062 --- [ocp-async-3,1ed544f04ef57e35,14aae97f97e69d02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:03.574 INFO 21062 --- [ocp-async-10,3228d7a85d775388,13b4c2469bceb61e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:05.571 WARN 21062 --- [ocp-async-1,9209095dc5d1256e,190a359f7563ded2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:05.574 INFO 21062 --- [ocp-async-6,3b8c4b8ced468249,244be47f48127ed3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:05.983 INFO 21062 --- [prometheus-minute-schedule-1,98e7d53cfe927d96,a67634f8c8ae66b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:06.099 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:06.100 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:06.595 INFO 21062 --- [ocp-schedules-21,07ec8b78b599480a,4508b417b86946ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:26:07.571 WARN 21062 --- [ocp-async-8,9b153373b7182c9b,0b5f341dd5c19c63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:07.575 INFO 21062 --- [ocp-async-5,1c028505ef0b9488,dff03a0e9c774b7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:08.505 INFO 21062 --- [ocp-schedules-3,9bae46e3cb025461,0bd883f915dc063a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:26:08.505 INFO 21062 --- [ocp-schedules-17,711fd9ff0b2314eb,1f24ab3503b2c02c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:26:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:26:08.590079+08:00, lastProcessTime=2024-11-04T15:25:58.589292+08:00 2024-11-04 15:26:08.763 INFO 21062 --- [ocp-async-4,2d35a1730a990ba9,a8bb24ed58f013c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:26:08.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:26:08.590079+08:00 2024-11-04 15:26:08.983 INFO 21062 --- [prometheus-minute-schedule-1,0b95b8863f637abb,e91bc86ccdc749cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:09.100 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:09.101 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:09.319 INFO 21062 --- [ocp-schedules-28,4c50b2b5376907cf,9ba0fe8081de2517] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:26:09.323 INFO 21062 --- [ocp-schedules-28,4c50b2b5376907cf,9ba0fe8081de2517] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:09.323 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:26:09.572 WARN 21062 --- [ocp-async-3,71e770731c19edb9,01c367269107c03d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:09.575 INFO 21062 --- [ocp-async-10,76f51d9c02b4af0e,f9408fb436f18e90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:26:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:26:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:26:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:26:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:26:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:26:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:26:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:26:10.379 INFO 21062 --- [ocp-schedules-12,8538e728a4db8379,0edddaf9317b026a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:26:10.379 INFO 21062 --- [ocp-schedules-12,5c12d94a696306c1,2aac18f58198704e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:26:10.383 INFO 21062 --- [ocp-schedules-12,5c12d94a696306c1,2aac18f58198704e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:10.413 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:26:10.413 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:26:11.572 WARN 21062 --- [ocp-async-1,abbbaa2e99cdc797,a7790f12287a13cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:11.575 INFO 21062 --- [ocp-async-6,576f85420f25de4c,afddb0da08324db7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:11.682 INFO 21062 --- [ocp-schedules-20,d942a62ca7cd677e,17940eb02b23a04f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:26:11.830 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=3, affectRows=3 2024-11-04 15:26:11.830 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=3, validCount=3 2024-11-04 15:26:11.830 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=2, affectRows=2 2024-11-04 15:26:11.830 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=2, validCount=2 2024-11-04 15:26:11.831 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=23, affectRows=23 2024-11-04 15:26:11.831 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=22, validCount=22 2024-11-04 15:26:11.831 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=32, affectRows=32 2024-11-04 15:26:11.831 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=49, affectRows=49 2024-11-04 15:26:11.831 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=32, validCount=32 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=74, affectRows=74 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=41, affectRows=41 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=49, validCount=49 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=66, affectRows=66 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=61, affectRows=61 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705100, nodeCount=71, affectRows=71 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=41, validCount=41 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=73, validCount=73 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=61, validCount=61 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=65, validCount=65 2024-11-04 15:26:11.832 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705100, nodeCount=71, validCount=71 2024-11-04 15:26:11.983 INFO 21062 --- [prometheus-minute-schedule-1,1e6fdbed08c49e01,ea9f84f03cd7b688] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:12.101 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:12.102 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:13.049 INFO 21062 --- [ocp-schedules-15,ee618ff239d7e492,cbc0a75db349a5d7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:26:13.055 INFO 21062 --- [ocp-schedules-15,ee618ff239d7e492,cbc0a75db349a5d7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:26:13.055 INFO 21062 --- [ocp-schedules-15,ee618ff239d7e492,cbc0a75db349a5d7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:26:13.572 WARN 21062 --- [ocp-async-8,1cb5b36daef816c1,1aa0f9df3cc5e812] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:13.575 INFO 21062 --- [ocp-async-5,8a1a6f43d9a0a4c6,ae80501a39ad7428] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:13.984 INFO 21062 --- [prometheus-cache-monitor1,0ea1769c89db0073,71e93fb9758f4260] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:26:13.984 INFO 21062 --- [prometheus-cache-monitor1,0ea1769c89db0073,71e93fb9758f4260] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:26:14.983 INFO 21062 --- [prometheus-minute-schedule-1,2029739ffe728764,0b51cf81cc7587c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:15.102 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:15.102 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:15.572 WARN 21062 --- [ocp-async-7,49a7b1ecafde14b4,9a123a3d5a47d00b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:15.575 INFO 21062 --- [ocp-async-4,c5eed4b458e2bb41,1a3c9c245d48fdfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:16.639 INFO 21062 --- [ocp-schedules-14,ec12e768be7acd69,ed7f023da65590f3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:26:17.572 WARN 21062 --- [ocp-async-10,986da69a57229d42,7258249313be1ad5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:17.576 INFO 21062 --- [ocp-async-2,cfb210d5145cd47e,16a4a0ffdfdf4aad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4211f3ed7e613c13,618bcb5c630a150f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:18.102 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:18.103 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:26:18.589510+08:00, lastProcessTime=2024-11-04T15:26:08.590079+08:00 2024-11-04 15:26:18.763 INFO 21062 --- [ocp-async-9,e028243c4e4f0292,d5ceaff60fbe4064] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:26:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:26:18.589510+08:00 2024-11-04 15:26:19.324 INFO 21062 --- [ocp-schedules-28,4c50b2b5376907cf,9ba0fe8081de2517] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:26:19.573 WARN 21062 --- [ocp-async-8,375abb3ffd651fa5,9a45c3e554d68075] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:19.576 INFO 21062 --- [ocp-async-5,316740cd077b982b,eb0f4e511ea105d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8fb37eeca23a34f0,0eda1e39aa1524fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:21.103 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:21.104 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:21.573 WARN 21062 --- [ocp-async-7,d64825f56ca8ed05,390967cf78e46bc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:21.576 INFO 21062 --- [ocp-async-4,e1bd98eac0ac09d7,b8b08042b764820c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:21.684 INFO 21062 --- [ocp-schedules-24,a4b0c4e6eb764a84,947310a57f41773d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:26:23.573 WARN 21062 --- [ocp-async-10,81fef2f3ae8c4eac,c196629896e6d431] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:23.576 INFO 21062 --- [ocp-async-2,96a2400e45b09c82,4ee765f94711f0c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:23.983 INFO 21062 --- [prometheus-minute-schedule-1,ff0b3b642d2f0fae,3edd60a541033c32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:24.104 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:24.105 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:25.573 WARN 21062 --- [ocp-async-6,c2e9f2e683e6886e,845d3c43da838e73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:25.577 INFO 21062 --- [ocp-async-9,7453e0bf8aca376f,ed53de0683d5a492] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:26.665 INFO 21062 --- [ocp-schedules-5,eee3bb6617b4875f,6677276e32cac2aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:26:26.983 INFO 21062 --- [prometheus-minute-schedule-1,4d4de2fa3793b75c,37ab26ded8ead4c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8344016d308a1ec5,686312b62848b9b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:26:27.105 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:27.105 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:27.573 WARN 21062 --- [ocp-async-5,1b372c601a224e8f,bfb7819bc1d2c07e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:27.577 INFO 21062 --- [ocp-async-3,346a98044613771c,0013da1ea4515d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:28.498 INFO 21062 --- [ocp-schedules-6,27ba16b9b2ae7e17,e98ba055362fa0cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:26:28.500 INFO 21062 --- [ocp-schedules-1,ee6893f0be55b7cc,e675f51630a6f437] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:26:28.500 INFO 21062 --- [ocp-schedules-15,f439b991604e940c,0d74b15207399fd0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:26:28.500 INFO 21062 --- [ocp-schedules-18,274aa753ea9f6671,dca7552b462fe7ae] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:26:28.502 INFO 21062 --- [ocp-schedules-4,f5428a56c591a2fd,32a2a4c015db4d44] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:26:28.503 INFO 21062 --- [ocp-schedules-6,27ba16b9b2ae7e17,e98ba055362fa0cd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:26:28.503 INFO 21062 --- [ocp-schedules-1,ee6893f0be55b7cc,e675f51630a6f437] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:28.504 INFO 21062 --- [ocp-schedules-18,274aa753ea9f6671,dca7552b462fe7ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:26:28.504 INFO 21062 --- [ocp-schedules-19,08e7f601f8de3700,69b1e6e783286c03] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:26:28.505 INFO 21062 --- [ocp-schedules-15,f439b991604e940c,0d74b15207399fd0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:26:28.510 INFO 21062 --- [ocp-schedules-3,f0c53ad0b76ccd62,6371b50683d01f45] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:26:28.510 INFO 21062 --- [ocp-schedules-3,f0c53ad0b76ccd62,6371b50683d01f45] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:26:28.510 INFO 21062 --- [ocp-schedules-30,8ceaf8d25b2e5ecc,959a785077514e07] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:26:28.510 INFO 21062 --- [ocp-schedules-30,8ceaf8d25b2e5ecc,959a785077514e07] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:26:28.510 INFO 21062 --- [ocp-schedules-23,b9c8e7b751c9042a,e29ccc14d7608010] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:26:28.510 INFO 21062 --- [ocp-schedules-23,b9c8e7b751c9042a,e29ccc14d7608010] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:26:28.588 INFO 21062 --- [ocp-schedules-25,9417dacfcf805411,30f0ee6d8c21f616] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:26:28.588 INFO 21062 --- [ocp-schedules-25,9417dacfcf805411,30f0ee6d8c21f616] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:26:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:26:28.589653+08:00, lastProcessTime=2024-11-04T15:26:18.589510+08:00 2024-11-04 15:26:28.763 INFO 21062 --- [ocp-async-2,d4c939254e17dfdb,3403cca731e89465] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:26:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:26:28.589653+08:00 2024-11-04 15:26:28.975 INFO 21062 --- [ocp-schedules-27,fac662cba95b0ac6,b94515f7f269e104] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:26:28.975 INFO 21062 --- [ocp-schedules-27,fac662cba95b0ac6,b94515f7f269e104] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:26:28.990 INFO 21062 --- [ocp-schedules-14,720e973020bbec96,4315203dc58b7b9b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:26:29.385 INFO 21062 --- [ocp-schedules-7,4e2f83cc402c3eac,4e60feea5ca4045c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:26:29.385 INFO 21062 --- [ocp-schedules-7,4e2f83cc402c3eac,4e60feea5ca4045c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:26:29.523 INFO 21062 --- [ocp-schedules-24,2b8f28fb1798956f,b01b0251c6206dfa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:26:29.544 INFO 21062 --- [ocp-schedules-10,2dd52daec82830fe,996d927f587c1a54] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:26:29.549 INFO 21062 --- [ocp-schedules-10,2dd52daec82830fe,996d927f587c1a54] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:26:29.549 INFO 21062 --- [ocp-schedules-10,2dd52daec82830fe,996d927f587c1a54] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:26:29.573 WARN 21062 --- [ocp-async-6,9e41529a409eb5cf,9829208fe9c6f05a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:29.577 INFO 21062 --- [ocp-async-8,e63fc10322c71b4c,05ee766a62025ea2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:29.983 INFO 21062 --- [prometheus-minute-schedule-1,92cb3266c1426c25,efd26b9669e1ab2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:30.026 INFO 21062 --- [ocp-schedules-26,0428c0cfe01e58eb,bfe6a56e2870ac15] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:26:30.026 INFO 21062 --- [ocp-schedules-26,0428c0cfe01e58eb,bfe6a56e2870ac15] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:26:30.027 INFO 21062 --- [ocp-schedules-26,0428c0cfe01e58eb,bfe6a56e2870ac15] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:26:30.027 INFO 21062 --- [ocp-schedules-26,0428c0cfe01e58eb,bfe6a56e2870ac15] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:26:30.028 INFO 21062 --- [ocp-schedules-26,0428c0cfe01e58eb,bfe6a56e2870ac15] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:26:30.028 INFO 21062 --- [ocp-schedules-26,0428c0cfe01e58eb,bfe6a56e2870ac15] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:26:30.106 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:30.106 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:26:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:26:31.574 WARN 21062 --- [ocp-async-8,3e4117a02a91c1dc,f80f764ec869f580] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:31.577 INFO 21062 --- [ocp-async-7,0485cfc41e7f1243,2641adaba8097fcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:31.686 INFO 21062 --- [ocp-schedules-3,bea9ae728775c25e,876f98b792cda974] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:26:32.983 INFO 21062 --- [prometheus-minute-schedule-1,77b2da3c80370838,d3c4e2eeb1c5c3de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:33.058 INFO 21062 --- [ocp-schedules-3,856cfdfa306eb94c,8a5c25db9dc5bcc5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:26:33.064 INFO 21062 --- [ocp-schedules-3,856cfdfa306eb94c,8a5c25db9dc5bcc5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:26:33.064 INFO 21062 --- [ocp-schedules-3,856cfdfa306eb94c,8a5c25db9dc5bcc5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:26:33.106 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:33.107 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:33.292 INFO 21062 --- [ocp-schedules-26,4bd3185b84896099,2e508a8a6b19c0f3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:26:33.292 INFO 21062 --- [ocp-schedules-26,4bd3185b84896099,2e508a8a6b19c0f3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:26:33.319 INFO 21062 --- [ocp-schedules-26,4bd3185b84896099,2e508a8a6b19c0f3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:26:33.319 INFO 21062 --- [ocp-schedules-26,4bd3185b84896099,2e508a8a6b19c0f3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:26:33.319 INFO 21062 --- [ocp-schedules-8,db1cd3026d972039,b2389fefdbb9780a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:26:33.527 INFO 21062 --- [ocp-schedules-4,33d3921f1e131e73,a90dd3527f932f76] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:26:33.527+08:00, clusters=[1] 2024-11-04 15:26:33.529 INFO 21062 --- [ocp-schedules-4,33d3921f1e131e73,a90dd3527f932f76] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:26:33.574 WARN 21062 --- [ocp-async-7,06bc25c2c41fcd31,6e63411fcbf5362a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:33.577 INFO 21062 --- [ocp-async-3,0063c29aff5bdd6b,d790d9bc209b97df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:33.834 INFO 21062 --- [ocp-schedules-9,fbc18acd083dec80,9d682f48f7011210] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:26:33.834+08:00, clusters=[1] 2024-11-04 15:26:33.839 INFO 21062 --- [ocp-schedules-9,fbc18acd083dec80,9d682f48f7011210] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:26:33.874 INFO 21062 --- [ocp-schedules-20,abf6f7f1f3749117,c3c98c13fa4faad3] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:26:33.877 INFO 21062 --- [ocp-schedules-20,abf6f7f1f3749117,c3c98c13fa4faad3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:33.936 ERROR 21062 --- [ocp-schedules-20,abf6f7f1f3749117,c3c98c13fa4faad3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:26:33.936 INFO 21062 --- [ocp-schedules-20,abf6f7f1f3749117,c3c98c13fa4faad3] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:26:35.574 WARN 21062 --- [ocp-async-3,832b4ec42f9a5c79,8c3481586d57a828] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:35.578 INFO 21062 --- [ocp-async-6,177722584532eb37,7a07746274db9f20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:35.669 INFO 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:26:35.669 INFO 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:26:35.669 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:26:35.669 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:26:35.679 INFO 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:26:35.679 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:26:35.679 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:26:35.680 WARN 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:26:35.681 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:26:35.681 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:26:35.681 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:26:35.681 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:26:35.681 INFO 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:26:35.683 WARN 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:26:35.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:26:35.683 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:26:35.690 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:26:35.690 INFO 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:26:35.690 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:26:35.701 ERROR 21062 --- [ocp-schedules-26,75e0ca98411ced90,5135198794c78524] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:26:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b3824bee0adbb6a9,165e6c7efa5c7960] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:36.107 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:36.108 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:36.708 INFO 21062 --- [ocp-schedules-16,607cffe06de3483b,209f5a42da632a39] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:26:37.574 WARN 21062 --- [ocp-async-6,988610f30f9fb998,8b55d7ce8e42b9ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:37.578 INFO 21062 --- [ocp-async-9,7622d5f4e40c4849,25b4f8846bdc0e2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:38.504 INFO 21062 --- [ocp-schedules-1,ee6893f0be55b7cc,e675f51630a6f437] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:26:38.505 INFO 21062 --- [ocp-schedules-18,274aa753ea9f6671,dca7552b462fe7ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:26:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:26:38.590521+08:00, lastProcessTime=2024-11-04T15:26:28.589653+08:00 2024-11-04 15:26:38.763 INFO 21062 --- [ocp-async-4,f0ec3aefec529cfa,dd9b2b9208e278f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:26:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:26:38.590521+08:00 2024-11-04 15:26:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a197a54c01cc30de,fbef2cc39657c81a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:39.108 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:39.109 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:39.574 WARN 21062 --- [ocp-async-10,afd290666e88cc7e,bd560b3a17e0414c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:39.578 INFO 21062 --- [ocp-async-3,de4306898e544299,92e2ba89c3bbb54f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:26:40.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:26:40.414 INFO 21062 --- [ocp-schedules-12,5c12d94a696306c1,2aac18f58198704e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:26:40.414 INFO 21062 --- [ocp-schedules-12,60d25adcd2ee2fb2,c05f9fbc18e1092c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:26:40.417 INFO 21062 --- [ocp-schedules-12,60d25adcd2ee2fb2,c05f9fbc18e1092c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:40.447 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:26:40.447 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:26:41.575 WARN 21062 --- [ocp-async-3,6b1c478e9357a604,9ab4ea4b05cc15a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:41.578 INFO 21062 --- [ocp-async-6,1b379700d8d699e1,4591a1f1baeb28a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:41.689 INFO 21062 --- [ocp-schedules-18,e00913d15d8a964f,bb55c034af3cd22e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:26:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:26:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:26:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:26:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:26:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:26:41.852 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:26:41.852 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:26:41.983 INFO 21062 --- [prometheus-minute-schedule-1,eb7cff2a1c41076e,e37e7b79ce1231b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:42.109 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:42.110 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:43.575 WARN 21062 --- [ocp-async-8,c5eca776aa4270ca,ff6a9436d05e6c46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:43.578 INFO 21062 --- [ocp-async-9,a25f2cd32b9372f8,33ab4f1c700d8482] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:44.611 INFO 21062 --- [ocp-schedules-18,f0f8a29a2065905d,d583fe166d260954] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:26:44.615 INFO 21062 --- [ocp-schedules-18,f0f8a29a2065905d,d583fe166d260954] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:44.702 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:26:44.704 ERROR 21062 --- [ocp-schedules-18,f0f8a29a2065905d,d583fe166d260954] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:26:44.704 INFO 21062 --- [ocp-schedules-18,f0f8a29a2065905d,d583fe166d260954] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:26:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f3c11b873430934e,d34a8670934d8591] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:45.110 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:45.110 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:45.575 WARN 21062 --- [ocp-async-7,b3eec394f7291fc0,2c7f0c2578292865] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:45.579 INFO 21062 --- [ocp-async-4,93edb62df209b3e6,0a9eba0f114bd96f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:46.733 INFO 21062 --- [ocp-schedules-11,5a17df8bfdb26243,b6b6fc5652b656d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:26:47.575 WARN 21062 --- [ocp-async-1,b629c03f7d08b2da,03325f79743c6a01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:47.579 INFO 21062 --- [ocp-async-2,52d1e6c104db0fe4,d36845fdbb5fc3a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e229cea9848bb7bb,ec4de4d539d2ad46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:48.111 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:48.111 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:26:48.591201+08:00, lastProcessTime=2024-11-04T15:26:38.590521+08:00 2024-11-04 15:26:48.763 INFO 21062 --- [ocp-async-5,2c08110f33ad3566,3eaf65bc60bd6954] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:26:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:26:48.591201+08:00 2024-11-04 15:26:49.575 WARN 21062 --- [ocp-async-8,26cc08d5535dd28d,e47de15f9f3cc982] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:49.579 INFO 21062 --- [ocp-async-9,99dc3ec8aca17db9,55f2a64d36381bc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:50.983 INFO 21062 --- [prometheus-minute-schedule-1,4a5294915c93dc47,dcacbd3ca1e04e1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:51.111 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:51.112 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:51.575 WARN 21062 --- [ocp-async-7,5cfec4d00d8ae9c0,c7e5d6fc0004ecfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:51.579 INFO 21062 --- [ocp-async-4,1bc765e7f192ba32,db6a329ed3f07a69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:51.692 INFO 21062 --- [ocp-schedules-13,0e1fbbb35334d06f,5280508da659c0b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:26:53.067 INFO 21062 --- [ocp-schedules-11,ea3ae6f256e7fbaf,d6dbc9735edeb41b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:26:53.073 INFO 21062 --- [ocp-schedules-11,ea3ae6f256e7fbaf,d6dbc9735edeb41b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:26:53.073 INFO 21062 --- [ocp-schedules-11,ea3ae6f256e7fbaf,d6dbc9735edeb41b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:26:53.575 WARN 21062 --- [ocp-async-1,53354b61d004cabe,34abb30b39a56241] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:53.579 INFO 21062 --- [ocp-async-2,28a6955798d2aa8a,9b5807d00b815832] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:53.984 INFO 21062 --- [prometheus-minute-schedule-1,cd82acc000759384,e046ee972a530176] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:54.112 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:54.113 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:55.576 WARN 21062 --- [ocp-async-6,4796dd28e95a24de,8627f0f9ce76ac11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:55.579 INFO 21062 --- [ocp-async-5,825de933a602dfb5,906f5d734571c099] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:56.774 INFO 21062 --- [ocp-schedules-22,8a1487115c82c1e5,163924eef2870a91] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:26:56.983 INFO 21062 --- [prometheus-minute-schedule-1,8bb418ad5c3d25ec,486e4cd9b1e40bef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:26:57.113 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:26:57.114 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:26:57.576 WARN 21062 --- [ocp-async-9,46f21fcf75665aee,814947ed144f57c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:57.580 INFO 21062 --- [ocp-async-10,c0007276a8eaa10f,779841553050be0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:58.500 INFO 21062 --- [ocp-schedules-2,f53e0a45d0c33ef8,d6fc5944321bc6e6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:26:58.500 INFO 21062 --- [ocp-schedules-28,4030f843b4c20e09,bed47b49804a8bf9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:26:58.502 INFO 21062 --- [ocp-schedules-26,0133b870ac785bfe,be0fc121033c1513] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:26:58.505 INFO 21062 --- [ocp-schedules-2,f53e0a45d0c33ef8,d6fc5944321bc6e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:58.505 INFO 21062 --- [ocp-schedules-28,4030f843b4c20e09,bed47b49804a8bf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:26:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:26:58.505 INFO 21062 --- [ocp-schedules-4,5b3ee4c6b3886a28,81e9e5b287052a2b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:26:58.512 INFO 21062 --- [ocp-schedules-16,3dfe3aabf8390457,e7d430c3b111d2f8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:26:58.513 INFO 21062 --- [ocp-schedules-11,d971e8526205b6ce,e9ad7cbfc3dbbb35] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:26:58.513 INFO 21062 --- [ocp-schedules-16,3dfe3aabf8390457,e7d430c3b111d2f8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:26:58.513 INFO 21062 --- [ocp-schedules-11,d971e8526205b6ce,e9ad7cbfc3dbbb35] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:26:58.513 INFO 21062 --- [ocp-schedules-8,bd00b5094e1b8a80,1a41572cf365de8c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:26:58.513 INFO 21062 --- [ocp-schedules-8,bd00b5094e1b8a80,1a41572cf365de8c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:26:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:26:58.589079+08:00, lastProcessTime=2024-11-04T15:26:48.591201+08:00 2024-11-04 15:26:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:26:58.589079+08:00 2024-11-04 15:26:58.763 INFO 21062 --- [ocp-async-3,c63e1ea3b5bda5b6,fed6ec30c88e8366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:26:59.526 INFO 21062 --- [ocp-schedules-16,ef4048b99fbb8f2e,0ec6fd9eff32bee6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:26:59.576 WARN 21062 --- [ocp-async-1,d5a562991c422291,29f095a20c0451ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:26:59.580 INFO 21062 --- [ocp-async-2,db908e31e9deddbf,2e2aa07fa3c9d68b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:26:59.983 INFO 21062 --- [prometheus-minute-schedule-1,10cdf4930fee237b,81c83d15d0bc85ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705220, interval=60 2024-11-04 15:27:00.114 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:00.115 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:01.576 WARN 21062 --- [ocp-async-6,f37cca2776303988,4caf9927bb21ebbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:01.580 INFO 21062 --- [ocp-async-5,4b7bc00fbc4f4374,7e87ff2dd71d7c3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:01.695 INFO 21062 --- [ocp-schedules-11,828d498add7a39bf,1cae20da15e847c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:27:02.983 INFO 21062 --- [prometheus-minute-schedule-1,640e27bfecb39bfc,745729ad76ddfadb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:03.115 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:03.116 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:03.576 WARN 21062 --- [ocp-async-9,26b20e79787f6b50,72f16d68e717435e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:03.580 INFO 21062 --- [ocp-async-10,b7d7d5b1a0f56297,e9968110ce512014] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:05.576 WARN 21062 --- [ocp-async-4,b95910ab01ba4eab,654ab1ab796b5ca9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:05.580 INFO 21062 --- [ocp-async-3,6524489a2df8e8f9,2e850c62baf00711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:05.983 INFO 21062 --- [prometheus-minute-schedule-1,1c8e1b6f2f2f3a58,535eebce7a1140c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:06.116 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:06.116 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:06.801 INFO 21062 --- [ocp-schedules-19,d2ee1e194335fa7e,b5aaf01292a04cb2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:27:07.577 WARN 21062 --- [ocp-async-2,4a0491f58cab0dd2,c63ba6f653935da4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:07.580 INFO 21062 --- [ocp-async-8,34dce8c8f3a4c3c1,94501215a4f9ad5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:08.506 INFO 21062 --- [ocp-schedules-28,4030f843b4c20e09,bed47b49804a8bf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:27:08.506 INFO 21062 --- [ocp-schedules-2,f53e0a45d0c33ef8,d6fc5944321bc6e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:27:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:27:08.590570+08:00, lastProcessTime=2024-11-04T15:26:58.589079+08:00 2024-11-04 15:27:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:27:08.590570+08:00 2024-11-04 15:27:08.764 INFO 21062 --- [ocp-async-7,5ff46fe7d9e9b554,ee3348f79b24c990] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:27:08.983 INFO 21062 --- [prometheus-minute-schedule-1,f488e3e4bd49449e,c3fc25dd5dfb8428] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:09.117 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:09.117 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:09.577 WARN 21062 --- [ocp-async-9,9b4ef3a132eb0706,984ece13ec0e60a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:09.581 INFO 21062 --- [ocp-async-10,f541695fa677a60e,072852f0eb1e6b06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:27:10.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:27:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:27:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:27:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:27:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:27:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:27:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:27:10.447 INFO 21062 --- [ocp-schedules-12,60d25adcd2ee2fb2,c05f9fbc18e1092c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:27:10.447 INFO 21062 --- [ocp-schedules-12,8d63e0ad10175d3b,8bb358a75f95d80b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:27:10.451 INFO 21062 --- [ocp-schedules-12,8d63e0ad10175d3b,8bb358a75f95d80b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:10.480 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:27:10.480 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:27:11.577 WARN 21062 --- [ocp-async-4,b0c18d0c896ca11a,f65820c621c52664] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:11.581 INFO 21062 --- [ocp-async-3,2c3d3fe6f67f99d5,36f22bfa4df539cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:11.698 INFO 21062 --- [ocp-schedules-27,2fcc13cdde0de3ea,0f641337f402ce6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=21, affectRows=21 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=21, validCount=21 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=19, affectRows=19 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=18, validCount=18 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=48, affectRows=48 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=46, affectRows=46 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=38, affectRows=38 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=31, affectRows=31 2024-11-04 15:27:11.835 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=56, affectRows=56 2024-11-04 15:27:11.836 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=48, validCount=48 2024-11-04 15:27:11.836 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=31, validCount=31 2024-11-04 15:27:11.836 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=38, validCount=38 2024-11-04 15:27:11.836 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=46, validCount=46 2024-11-04 15:27:11.836 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=56, validCount=56 2024-11-04 15:27:11.836 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=73, affectRows=73 2024-11-04 15:27:11.836 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705160, nodeCount=90, affectRows=90 2024-11-04 15:27:11.837 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=73, validCount=73 2024-11-04 15:27:11.837 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705160, nodeCount=88, validCount=88 2024-11-04 15:27:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6a4ebf389d438b77,23d49defd5d28c8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:12.117 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:12.118 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:13.075 INFO 21062 --- [ocp-schedules-17,404ea5c5c92df95d,0616b153a5273e96] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:27:13.080 INFO 21062 --- [ocp-schedules-17,404ea5c5c92df95d,0616b153a5273e96] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:27:13.080 INFO 21062 --- [ocp-schedules-17,404ea5c5c92df95d,0616b153a5273e96] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:27:13.577 WARN 21062 --- [ocp-async-2,0c3d320e27977e3c,00fc8879b0dd1980] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:13.581 INFO 21062 --- [ocp-async-8,17908e69cee6c413,1dd93868f4c643b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:13.984 INFO 21062 --- [prometheus-cache-monitor1,8e05a77de502f024,0f10f454128d075c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:27:13.984 INFO 21062 --- [prometheus-cache-monitor1,8e05a77de502f024,0f10f454128d075c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:27:14.983 INFO 21062 --- [prometheus-minute-schedule-1,240a01242fe2100e,68d1b500b9fcca78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:15.118 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:15.119 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:15.577 WARN 21062 --- [ocp-async-5,0834c41ad8f28ac8,b8e3c461f6ffa2e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:15.581 INFO 21062 --- [ocp-async-7,a05dd2f0224f96cc,e4097fcecf1b7bbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:15.637 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:23:15.619, end=2024-11-04T15:25:15.634} 2024-11-04 15:27:15.646 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:27:15.646 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:27:16.841 INFO 21062 --- [ocp-schedules-4,1f6928d561247862,57670899331c4570] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:27:17.577 WARN 21062 --- [ocp-async-10,d779120a80631805,e696b01a87d3597e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:17.581 INFO 21062 --- [ocp-async-1,402595fe9da147ac,a0efdedc7d625be5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:17.983 INFO 21062 --- [prometheus-minute-schedule-1,ce63c8f05a61678e,160c04e06e83da66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:18.119 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:18.120 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:27:18.590134+08:00, lastProcessTime=2024-11-04T15:27:08.590570+08:00 2024-11-04 15:27:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:27:18.590134+08:00 2024-11-04 15:27:18.764 INFO 21062 --- [ocp-async-6,22756ceb4ab907c3,8b3a3983f374b0dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:27:19.324 INFO 21062 --- [ocp-schedules-28,9320c11c59f65625,7871ca5618b66fa8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:27:19.327 INFO 21062 --- [ocp-schedules-28,9320c11c59f65625,7871ca5618b66fa8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:19.328 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:27:19.577 WARN 21062 --- [ocp-async-2,34071d422d6fc340,d8ad8f0d8f9113e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:19.581 INFO 21062 --- [ocp-async-8,e202e5cb48fc12d6,c34712d1b23713c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:20.983 INFO 21062 --- [prometheus-minute-schedule-1,39c427f583622084,fc20410dac8c926e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:21.120 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:21.121 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:21.578 WARN 21062 --- [ocp-async-5,a193b090e11faea1,56238083e66c146f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:21.582 INFO 21062 --- [ocp-async-7,dff27b50df757cad,c425c4602a265ea9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:21.701 INFO 21062 --- [ocp-schedules-27,651ec08a41611878,497c28522c8e0348] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:27:23.578 WARN 21062 --- [ocp-async-10,95c1db5622cef598,53aba47ccde7a9d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:23.582 INFO 21062 --- [ocp-async-1,023752aa642affa5,b06c602451da4b1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:23.983 INFO 21062 --- [prometheus-minute-schedule-1,7a0537e618e10955,9fa6dffb803474b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:24.121 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:24.121 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:25.578 WARN 21062 --- [ocp-async-3,e9f2d075b7465790,25f14fb61dbd411e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:25.582 INFO 21062 --- [ocp-async-6,0f417aa36df061a0,192489b5df7164b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:26.867 INFO 21062 --- [ocp-schedules-20,39207c9ff559847a,aeb73be7da371d06] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:27:26.983 INFO 21062 --- [prometheus-minute-schedule-1,2c8fd59fe5c642bd,08ffa30b111e1ec7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ea0206d6f7e47a21,7b793c95cbd1164c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:27:27.122 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:27.122 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:27.578 WARN 21062 --- [ocp-async-8,0fb8d5a43db7f14a,85b5e1773f861a92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:27.582 INFO 21062 --- [ocp-async-9,7ebedfce54cbdc5d,112562324dd3f29a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:28.498 INFO 21062 --- [ocp-schedules-20,96a0c6ea3455c53f,6d3a9cb2c276008d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:27:28.499 INFO 21062 --- [ocp-schedules-27,f51315df1dbd8b2d,0e209fc11f5332c8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:27:28.500 INFO 21062 --- [ocp-schedules-5,bf2916d3b7c76a8f,1e8040405ad2c790] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:27:28.500 INFO 21062 --- [ocp-schedules-16,656ff57a7b64ce2e,9f307d7a30892abf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:27:28.502 INFO 21062 --- [ocp-schedules-17,07f80d65e235651e,dbd7cb4e46e70370] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:27:28.503 INFO 21062 --- [ocp-schedules-20,96a0c6ea3455c53f,6d3a9cb2c276008d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:27:28.503 INFO 21062 --- [ocp-schedules-27,f51315df1dbd8b2d,0e209fc11f5332c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:27:28.504 INFO 21062 --- [ocp-schedules-9,228482c9c65acf48,d1d09b905ee0b7aa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:27:28.504 INFO 21062 --- [ocp-schedules-4,ef345c79c31a5a6a,a06071e7b70a633b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:27:28.504 INFO 21062 --- [ocp-schedules-16,656ff57a7b64ce2e,9f307d7a30892abf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:28.505 INFO 21062 --- [ocp-schedules-5,bf2916d3b7c76a8f,1e8040405ad2c790] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:27:28.507 INFO 21062 --- [ocp-schedules-9,228482c9c65acf48,d1d09b905ee0b7aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:28.509 INFO 21062 --- [ocp-schedules-3,6b488c7d5851d6b7,d31fef88c9ef2a37] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:27:28.509 INFO 21062 --- [ocp-schedules-15,e05e2f3322c0fddc,539fa92de4c2bec5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:27:28.509 INFO 21062 --- [ocp-schedules-3,6b488c7d5851d6b7,d31fef88c9ef2a37] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:27:28.509 INFO 21062 --- [ocp-schedules-15,e05e2f3322c0fddc,539fa92de4c2bec5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:27:28.509 INFO 21062 --- [ocp-schedules-23,dd32c4de768534c5,2531dd3f0f7eaa8f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:27:28.510 INFO 21062 --- [ocp-schedules-23,dd32c4de768534c5,2531dd3f0f7eaa8f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:27:28.542 INFO 21062 --- [ocp-schedules-2,ef3c8b13e4c5d653,7b23b735e4feae69] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:27:28.542 INFO 21062 --- [ocp-schedules-2,ef3c8b13e4c5d653,7b23b735e4feae69] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:27:28.581 INFO 21062 --- [pool-12-thread-16,7b302725adb11958,bc4aef941e6b1901] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:27:28.581 INFO 21062 --- [pool-12-thread-13,7b302725adb11958,e96e58d726cdbfc7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:27:28.581 INFO 21062 --- [pool-12-thread-14,7b302725adb11958,90891edaf0945eb9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:27:28.581 INFO 21062 --- [pool-12-thread-13,7b302725adb11958,e96e58d726cdbfc7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:27:28.581 INFO 21062 --- [pool-12-thread-15,7b302725adb11958,e5ce8eb5104305ac] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:27:28.585 INFO 21062 --- [pool-12-thread-13,7b302725adb11958,e96e58d726cdbfc7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:27:28.585 INFO 21062 --- [pool-12-thread-13,7b302725adb11958,e96e58d726cdbfc7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:27:28.588 INFO 21062 --- [ForkJoinPool-11-worker-6,7b302725adb11958,7c76159a777942ca] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:27:28.588 INFO 21062 --- [ForkJoinPool-11-worker-6,7b302725adb11958,7c76159a777942ca] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:27:28.588 INFO 21062 --- [ForkJoinPool-11-worker-13,7b302725adb11958,4971ff72269074a1] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:27:28.588 INFO 21062 --- [pool-12-thread-14,7b302725adb11958,90891edaf0945eb9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:27:28.588 INFO 21062 --- [ForkJoinPool-11-worker-6,7b302725adb11958,7c76159a777942ca] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:27:28.588 INFO 21062 --- [pool-12-thread-14,7b302725adb11958,90891edaf0945eb9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:27:28.588 INFO 21062 --- [pool-12-thread-16,7b302725adb11958,bc4aef941e6b1901] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:27:28.588 INFO 21062 --- [pool-12-thread-15,7b302725adb11958,e5ce8eb5104305ac] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:27:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:27:28.589948+08:00, lastProcessTime=2024-11-04T15:27:18.590134+08:00 2024-11-04 15:27:28.666 INFO 21062 --- [ocp-schedules-14,8bbcc8fd4f2af90f,2f6638315fa0c9e4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:27:28.667 INFO 21062 --- [ocp-schedules-14,8bbcc8fd4f2af90f,2f6638315fa0c9e4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:27:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:27:28.589948+08:00 2024-11-04 15:27:28.764 INFO 21062 --- [ocp-async-1,8f5fd2fc3e9a444d,69874673b9d00835] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:27:28.992 INFO 21062 --- [ocp-schedules-3,2d3b40324d6564a0,ed64170e74057711] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:27:29.328 INFO 21062 --- [ocp-schedules-28,9320c11c59f65625,7871ca5618b66fa8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:27:29.388 INFO 21062 --- [ocp-schedules-29,92db14836e63112e,4f027be1033cb9ad] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:27:29.389 INFO 21062 --- [ocp-schedules-29,92db14836e63112e,4f027be1033cb9ad] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:27:29.506 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:27:29.522 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:27:29.524 WARN 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:27:29.524 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:27:29.526 WARN 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:27:29.528 INFO 21062 --- [ocp-schedules-14,42ef82e1c7fddcc6,3e9c3a34f412d949] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:27:29.532 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:27:29.533 WARN 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:27:29.534 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:27:29.535 WARN 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:27:29.541 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:27:29.542 WARN 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:27:29.543 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:27:29.544 WARN 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:27:29.549 INFO 21062 --- [ocp-schedules-18,db59571305e14a3b,8a56c1dcff1ab236] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:27:29.549 INFO 21062 --- [ocp-schedules-13,54e9fef863e92064,5bdc1b9526fa017f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:27:29.554 INFO 21062 --- [ocp-schedules-13,54e9fef863e92064,5bdc1b9526fa017f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:27:29.554 INFO 21062 --- [ocp-schedules-13,54e9fef863e92064,5bdc1b9526fa017f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:27:29.578 WARN 21062 --- [ocp-async-1,adcc0621037f94e0,a30c814ccf5cafd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:29.582 INFO 21062 --- [ocp-async-2,92eab183edd72c54,229772487a3fe550] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:29.983 INFO 21062 --- [prometheus-minute-schedule-1,4b26a8cb0db93c8d,5ae3f01a65170584] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:30.028 INFO 21062 --- [ocp-schedules-17,4d22645e16912a5a,6fd2bdf68048ed34] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:27:30.028 INFO 21062 --- [ocp-schedules-17,4d22645e16912a5a,6fd2bdf68048ed34] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:27:30.029 INFO 21062 --- [ocp-schedules-17,4d22645e16912a5a,6fd2bdf68048ed34] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:27:30.029 INFO 21062 --- [ocp-schedules-17,4d22645e16912a5a,6fd2bdf68048ed34] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:27:30.030 INFO 21062 --- [ocp-schedules-17,4d22645e16912a5a,6fd2bdf68048ed34] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:27:30.030 INFO 21062 --- [ocp-schedules-17,4d22645e16912a5a,6fd2bdf68048ed34] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:27:30.122 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:30.123 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:27:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:27:30.487 INFO 21062 --- [ocp-schedules-2,beb4dc6a48146be4,d2d9b5b496c79597] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:27:30.487 INFO 21062 --- [ocp-schedules-2,beb4dc6a48146be4,d2d9b5b496c79597] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:27:30.515 INFO 21062 --- [pool-54-thread-1,beb4dc6a48146be4,68f286fe9b208b3a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:27:30.545 WARN 21062 --- [ocp-schedules-2,beb4dc6a48146be4,d2d9b5b496c79597] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:27:30.545 INFO 21062 --- [ocp-schedules-2,beb4dc6a48146be4,d2d9b5b496c79597] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:27:31.578 WARN 21062 --- [ocp-async-2,cd9f67428432513f,cbff5099875e85e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:31.583 INFO 21062 --- [ocp-async-5,d127e28a318328e6,92aa3d4546de38c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:31.705 INFO 21062 --- [ocp-schedules-2,6ede48ca2a7344f2,a6aa83346901ddf9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:27:32.983 INFO 21062 --- [prometheus-minute-schedule-1,77bf2b82ebe9c0a9,0cb999544772b9ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:33.082 INFO 21062 --- [ocp-schedules-29,7b6e793f69277ce8,a9ad9e64c6facc38] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:27:33.087 INFO 21062 --- [ocp-schedules-29,7b6e793f69277ce8,a9ad9e64c6facc38] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:27:33.087 INFO 21062 --- [ocp-schedules-29,7b6e793f69277ce8,a9ad9e64c6facc38] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:27:33.123 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:33.124 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:33.319 INFO 21062 --- [ocp-schedules-26,0682354fd49a19af,6a5b3d79ae460be3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:27:33.319 INFO 21062 --- [ocp-schedules-26,0682354fd49a19af,6a5b3d79ae460be3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:27:33.346 INFO 21062 --- [ocp-schedules-26,0682354fd49a19af,6a5b3d79ae460be3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:27:33.346 INFO 21062 --- [ocp-schedules-26,0682354fd49a19af,6a5b3d79ae460be3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:27:33.346 INFO 21062 --- [ocp-schedules-5,575ad425543d5967,6cbb9da87bb09718] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:27:33.555 INFO 21062 --- [ocp-schedules-1,cd0c50ea1cc2b503,285d6951ceed8135] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:27:33.555+08:00, clusters=[1] 2024-11-04 15:27:33.556 INFO 21062 --- [ocp-schedules-1,cd0c50ea1cc2b503,285d6951ceed8135] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:27:33.578 WARN 21062 --- [ocp-async-5,a1a7204036ac462c,35e66e8bbab12845] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:33.583 INFO 21062 --- [ocp-async-9,d32ff490de3480f9,11903337f2c202c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:33.865 INFO 21062 --- [ocp-schedules-6,a2a050637e182e0f,cf234d9155c38fe2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:27:33.865+08:00, clusters=[1] 2024-11-04 15:27:33.869 INFO 21062 --- [ocp-schedules-6,a2a050637e182e0f,cf234d9155c38fe2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:27:35.579 WARN 21062 --- [ocp-async-9,1b25b969efa543bf,50fad782734329a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:35.583 INFO 21062 --- [ocp-async-1,8951f6a36c5b1a6f,c6b47e5c055ac6f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:35.710 INFO 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:27:35.710 INFO 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:27:35.710 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:27:35.710 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:27:35.719 INFO 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:27:35.720 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:27:35.720 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:27:35.721 WARN 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:27:35.721 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:27:35.721 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:27:35.722 INFO 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:27:35.722 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:27:35.722 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:27:35.723 WARN 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:27:35.723 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:27:35.723 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:27:35.729 INFO 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:27:35.730 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:27:35.730 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:27:35.741 ERROR 21062 --- [ocp-schedules-1,112b8761a1e2f412,261709a62ccd6577] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:27:35.983 INFO 21062 --- [prometheus-minute-schedule-1,737507258667302a,7d2ee94478b3d14d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:36.124 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:36.125 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:36.911 INFO 21062 --- [ocp-schedules-7,6e4a386d2d0b3400,73ecf7f0fdb2fa8e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:27:37.579 WARN 21062 --- [ocp-async-1,8311e822313bad63,296cb80517333ab5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:37.583 INFO 21062 --- [ocp-async-8,3596a13b9fd06b21,cf606f6650fd695b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:38.504 INFO 21062 --- [ocp-schedules-27,f51315df1dbd8b2d,0e209fc11f5332c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:27:38.505 INFO 21062 --- [ocp-schedules-16,656ff57a7b64ce2e,9f307d7a30892abf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:27:38.507 INFO 21062 --- [ocp-schedules-9,228482c9c65acf48,d1d09b905ee0b7aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:27:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:27:38.589901+08:00, lastProcessTime=2024-11-04T15:27:28.589948+08:00 2024-11-04 15:27:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:27:38.589901+08:00 2024-11-04 15:27:38.764 INFO 21062 --- [ocp-async-7,7b43019e86454444,1ea670fc0a833e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:27:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3604bb086e4a22cd,4997e1f9db34dd18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:39.125 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:39.126 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:39.579 WARN 21062 --- [ocp-async-10,35295aa667c58eba,3461d39d09d39c86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:39.583 INFO 21062 --- [ocp-async-9,0da4728efab53573,1053a7d009c82e1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:27:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:27:40.480 INFO 21062 --- [ocp-schedules-12,8d63e0ad10175d3b,8bb358a75f95d80b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:27:40.481 INFO 21062 --- [ocp-schedules-12,fc42ccd592cf9505,106c618fb8b9b360] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:27:40.484 INFO 21062 --- [ocp-schedules-12,fc42ccd592cf9505,106c618fb8b9b360] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:40.513 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:27:40.513 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:27:41.579 WARN 21062 --- [ocp-async-4,b2030e4bd24c5eb7,96689a00fb95f875] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:41.584 INFO 21062 --- [ocp-async-1,ba6355bc310ad855,b1f0a681a1e34e03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:41.707 INFO 21062 --- [ocp-schedules-23,d50697ee2a3a81dc,66fe249a28927bba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:27:41.983 INFO 21062 --- [prometheus-minute-schedule-1,63ef97c76c53383e,1873b295f82cdc06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:42.126 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:42.127 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:43.579 WARN 21062 --- [ocp-async-2,77619179ca714246,8cd92e8a84bbf153] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:43.584 INFO 21062 --- [ocp-async-8,acfabb6c9863b66a,ef3cecd28ec41240] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:44.704 INFO 21062 --- [ocp-schedules-29,d5795be8193b06ad,09c6502e07d386ab] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:27:44.708 INFO 21062 --- [ocp-schedules-29,d5795be8193b06ad,09c6502e07d386ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:44.794 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:27:44.796 ERROR 21062 --- [ocp-schedules-29,d5795be8193b06ad,09c6502e07d386ab] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:27:44.796 INFO 21062 --- [ocp-schedules-29,d5795be8193b06ad,09c6502e07d386ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:27:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b3a5c340f4de7c6e,f975bf0e12dd02cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:45.127 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:45.128 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:45.580 WARN 21062 --- [ocp-async-5,86b2774ae2a59090,6ebcae0051d2557a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:45.584 INFO 21062 --- [ocp-async-7,0872af61ce5c078e,fe4ad92cfd06bd6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:46.938 INFO 21062 --- [ocp-schedules-27,4ae7899e0800354f,ece6d46f4828af2d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:27:47.580 WARN 21062 --- [ocp-async-9,4642c05452b5e9b2,8a6573839e27e2c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:47.584 INFO 21062 --- [ocp-async-3,6024d68441fbf5da,0fd75233804b0aed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:47.983 INFO 21062 --- [prometheus-minute-schedule-1,38028f09e4208ca6,a7ffad6b62a92adf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:48.128 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:48.129 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:27:48.589920+08:00, lastProcessTime=2024-11-04T15:27:38.589901+08:00 2024-11-04 15:27:48.764 INFO 21062 --- [ocp-async-6,6392dad77a626ed5,b76108c3e81f1580] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:27:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:27:48.589920+08:00 2024-11-04 15:27:49.580 WARN 21062 --- [ocp-async-2,26da03d42ada7787,cc2eb508530264da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:49.584 INFO 21062 --- [ocp-async-8,1078c04874cb9ad5,eb3b3ffaaa0fd4da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:50.983 INFO 21062 --- [prometheus-minute-schedule-1,04629bc88fbd3a23,a5ad04213daa6496] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:51.129 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:51.130 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:51.580 WARN 21062 --- [ocp-async-5,9727f0c192bb1e96,81d9c59e772accd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:51.585 INFO 21062 --- [ocp-async-7,d4671d58c8577f71,86578b3226bbccff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:51.710 INFO 21062 --- [ocp-schedules-26,fe9f4724a648aa4b,338e88799fcd0487] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:27:53.090 INFO 21062 --- [ocp-schedules-27,3c0c9f7e20b938a1,88a40fca6bb915a6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:27:53.097 INFO 21062 --- [ocp-schedules-27,3c0c9f7e20b938a1,88a40fca6bb915a6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:27:53.097 INFO 21062 --- [ocp-schedules-27,3c0c9f7e20b938a1,88a40fca6bb915a6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:27:53.580 WARN 21062 --- [ocp-async-9,29a761a62d80c286,be0d4dc73a43d3e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:53.585 INFO 21062 --- [ocp-async-3,125b3a5420fa0fec,76f0e345afad8de0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:53.983 INFO 21062 --- [prometheus-minute-schedule-1,f4a4f3e7555ed465,7d88e50b634f16d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:54.130 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:54.131 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:55.581 WARN 21062 --- [ocp-async-1,d2c8b343f4751386,5d3c20b9f5324233] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:55.585 INFO 21062 --- [ocp-async-6,2521754b09bc55b3,e24643af08acc2c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:56.980 INFO 21062 --- [ocp-schedules-7,31d08dbde0d3969b,ccecedc3be6310f7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:27:56.983 INFO 21062 --- [prometheus-minute-schedule-1,2159014feac478ba,73b5833323682d78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:27:57.131 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:27:57.132 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:27:57.581 WARN 21062 --- [ocp-async-8,ab14f1a252cf3002,74cb4bb6087c6a2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:57.585 INFO 21062 --- [ocp-async-10,1c2163869fecf3e0,ea2a01dd51cbb2dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:58.500 INFO 21062 --- [ocp-schedules-21,0f8fab2019579af3,eeee38c914156ac2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:27:58.500 INFO 21062 --- [ocp-schedules-2,efa77f620daafba8,51b94ea5d6e5f53c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:27:58.502 INFO 21062 --- [ocp-schedules-30,e47615cdd2570dbf,0b04452dbe20ea16] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:27:58.504 INFO 21062 --- [ocp-schedules-21,0f8fab2019579af3,eeee38c914156ac2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:58.504 INFO 21062 --- [ocp-schedules-2,efa77f620daafba8,51b94ea5d6e5f53c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:27:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:27:58.505 INFO 21062 --- [ocp-schedules-8,f4c9d776a8d69da2,fee5dbc2ba4e46e1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:27:58.510 INFO 21062 --- [ocp-schedules-1,7a6c279d7354a05f,8ff7af021e658233] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:27:58.510 INFO 21062 --- [ocp-schedules-23,67f6e36fc25548af,31f336f8e9f95a8f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:27:58.510 INFO 21062 --- [ocp-schedules-1,7a6c279d7354a05f,8ff7af021e658233] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:27:58.510 INFO 21062 --- [ocp-schedules-23,67f6e36fc25548af,31f336f8e9f95a8f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:27:58.510 INFO 21062 --- [ocp-schedules-25,cd33d12c71706fcb,cc8f5adec112c2dd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:27:58.510 INFO 21062 --- [ocp-schedules-25,cd33d12c71706fcb,cc8f5adec112c2dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:27:58.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:27:58.592508+08:00, lastProcessTime=2024-11-04T15:27:48.589920+08:00 2024-11-04 15:27:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:27:58.592508+08:00 2024-11-04 15:27:58.764 INFO 21062 --- [ocp-async-4,aaf297b1d6dec1f3,8363335f0afad0f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:27:59.531 INFO 21062 --- [ocp-schedules-18,cd5f7b5321b94669,d4adddea09caaac4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:27:59.581 WARN 21062 --- [ocp-async-9,8b385f07f2f01993,a75c33b2a5b45190] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:27:59.585 INFO 21062 --- [ocp-async-3,f4e10c2ea26afdf3,91756728ab71db0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:27:59.983 INFO 21062 --- [prometheus-minute-schedule-1,8ad913fa5c816d19,5a2c78445b864895] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705280, interval=60 2024-11-04 15:28:00.132 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:00.133 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:01.581 WARN 21062 --- [ocp-async-1,ed00532e602cb7b2,e36bf78c805f6ccd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:01.586 INFO 21062 --- [ocp-async-6,44852debadcb0a5f,d2bdb6dc43f61ec4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:01.713 INFO 21062 --- [ocp-schedules-8,bfbabb987f4c8476,c9e524acc0fc7a5d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:28:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d2f1e5b3a9b2a64d,d7289b6706784fe0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:03.133 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:03.133 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:03.581 WARN 21062 --- [ocp-async-8,95634d08711aa311,944b84a5dce19257] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:03.586 INFO 21062 --- [ocp-async-10,fabd747d669fce2f,5ac6a3d9dce6ca91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:05.582 WARN 21062 --- [ocp-async-7,8d1ab598ce0275de,04a46c3060933c0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:05.586 INFO 21062 --- [ocp-async-4,4fadfb9bd03bb03e,66235ab617a0e325] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:05.983 INFO 21062 --- [prometheus-minute-schedule-1,7b9428bd576fe549,135fea68b918442d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:06.133 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:06.134 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:07.005 INFO 21062 --- [ocp-schedules-20,d231af71a74081c1,41fb97ecba1ebe04] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:28:07.582 WARN 21062 --- [ocp-async-3,9e379734e9a8a32b,f1ab1e69cfd05ca7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:07.586 INFO 21062 --- [ocp-async-2,4f2bab1ee898ce4c,5d0f9c02eed03fbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:08.505 INFO 21062 --- [ocp-schedules-2,efa77f620daafba8,51b94ea5d6e5f53c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:28:08.505 INFO 21062 --- [ocp-schedules-21,0f8fab2019579af3,eeee38c914156ac2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:28:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:28:08.591621+08:00, lastProcessTime=2024-11-04T15:27:58.592508+08:00 2024-11-04 15:28:08.765 INFO 21062 --- [ocp-async-5,7d936f9b930f5c38,d7872639fe5a00e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:28:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:28:08.591621+08:00 2024-11-04 15:28:08.983 INFO 21062 --- [prometheus-minute-schedule-1,77f39bf2079938b1,78a8c6105ac2a410] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:09.134 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:09.135 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:09.582 WARN 21062 --- [ocp-async-8,efa8ab35b9b5a9d9,62e88bb06baa7e5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:09.586 INFO 21062 --- [ocp-async-10,daf3ac5dcff4e2f6,1679512bf581aec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:28:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:28:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:28:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:28:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:28:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:28:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:28:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:28:10.513 INFO 21062 --- [ocp-schedules-12,fc42ccd592cf9505,106c618fb8b9b360] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:28:10.513 INFO 21062 --- [ocp-schedules-12,3f18ef0c7534d5b7,3708269b43f5dee2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:28:10.518 INFO 21062 --- [ocp-schedules-12,3f18ef0c7534d5b7,3708269b43f5dee2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:10.548 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:28:10.548 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:28:11.582 WARN 21062 --- [ocp-async-7,34d3d3956af355bb,4c1a13a4f331b2ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:11.587 INFO 21062 --- [ocp-async-4,2d8781decb2c36e4,7589e788380ec2e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:11.715 INFO 21062 --- [ocp-schedules-1,76543ede12261dbf,6a249b6e4681b6c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:28:11.841 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=18, affectRows=18 2024-11-04 15:28:11.842 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=9, affectRows=9 2024-11-04 15:28:11.842 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=18, validCount=18 2024-11-04 15:28:11.842 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=9, validCount=9 2024-11-04 15:28:11.842 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=42, affectRows=42 2024-11-04 15:28:11.842 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=42, validCount=42 2024-11-04 15:28:11.843 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=66, affectRows=66 2024-11-04 15:28:11.843 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=66, validCount=66 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=4, affectRows=4 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=52, affectRows=52 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=13, affectRows=13 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=4, validCount=4 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=13, validCount=13 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=22, affectRows=22 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=52, validCount=52 2024-11-04 15:28:11.844 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=21, validCount=21 2024-11-04 15:28:11.845 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=57, affectRows=57 2024-11-04 15:28:11.845 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=32, affectRows=32 2024-11-04 15:28:11.845 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=32, validCount=32 2024-11-04 15:28:11.845 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=57, validCount=57 2024-11-04 15:28:11.846 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=58, affectRows=58 2024-11-04 15:28:11.846 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705220, nodeCount=49, affectRows=49 2024-11-04 15:28:11.846 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=57, validCount=57 2024-11-04 15:28:11.846 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705220, nodeCount=48, validCount=48 2024-11-04 15:28:11.983 INFO 21062 --- [prometheus-minute-schedule-1,a2baf77452eefd41,880cbf20aca7be0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:12.135 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:12.136 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:13.100 INFO 21062 --- [ocp-schedules-22,0f879f2d83f159fa,9905da9d2809011b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:28:13.105 INFO 21062 --- [ocp-schedules-22,0f879f2d83f159fa,9905da9d2809011b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:28:13.105 INFO 21062 --- [ocp-schedules-22,0f879f2d83f159fa,9905da9d2809011b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:28:13.582 WARN 21062 --- [ocp-async-3,c1e02b2435bb939b,a9f8a96912250b3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:13.587 INFO 21062 --- [ocp-async-2,fd7c8e645784ecb7,f4a63327909b72c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:13.984 INFO 21062 --- [prometheus-cache-monitor1,2bba7d861a501033,5cf780e6ae439050] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:28:13.984 INFO 21062 --- [prometheus-cache-monitor1,2bba7d861a501033,5cf780e6ae439050] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:28:14.485 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:22:14.462, end=2024-11-04T15:27:14.483} 2024-11-04 15:28:14.486 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:22:14.463, end=2024-11-04T15:27:14.484} 2024-11-04 15:28:14.490 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:28:14.490 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:28:14.490 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:28:14.490 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:28:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f5b9144bb1cefb4a,00d9e3eba70ec3c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:15.136 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:15.137 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:15.582 WARN 21062 --- [ocp-async-6,0633a9acbc2bc57e,fdcbfeb7e908563f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:15.587 INFO 21062 --- [ocp-async-5,6e6b381f3c886fd4,55ec6bb845c7ddb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:17.084 INFO 21062 --- [ocp-schedules-9,16d8e4a8f4cb74b0,5b8362b4d3895b60] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:28:17.583 WARN 21062 --- [ocp-async-10,8161255135d4c390,1b6bc4b094e25ed8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:17.587 INFO 21062 --- [ocp-async-9,7e237c19da9e134a,0a2a9f04be721079] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:17.983 INFO 21062 --- [prometheus-minute-schedule-1,f8659fa81ab8d38f,68c37a0a2565bf94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:18.137 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:18.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:28:18.590350+08:00, lastProcessTime=2024-11-04T15:28:08.591621+08:00 2024-11-04 15:28:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:28:18.590350+08:00 2024-11-04 15:28:18.765 INFO 21062 --- [ocp-async-3,9a139add06e5ae77,1b358a33dc69aada] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:28:19.583 WARN 21062 --- [ocp-async-1,ca8a7396cfe6bf62,cc65ae9111439604] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:19.587 INFO 21062 --- [ocp-async-2,3f4ff8d15d05db6b,1ba98e0749afa304] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ccbac103e6da2c93,db16a4cfb3860809] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:21.138 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:21.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:21.583 WARN 21062 --- [ocp-async-6,c8917325599e4ef7,489d7b95d6487e5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:21.587 INFO 21062 --- [ocp-async-5,71ff0b6eab72380a,5881945c990d85be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:21.719 INFO 21062 --- [ocp-schedules-5,d2a8debccf6a8ec6,e8043fd1a905a71d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:28:23.583 WARN 21062 --- [ocp-async-5,de4b3874813a7080,28a2a2a2cc862a0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:23.588 INFO 21062 --- [ocp-async-9,3742eb4a5594753b,f1a7b70ff712430f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:23.983 INFO 21062 --- [prometheus-minute-schedule-1,697b6011c365b2fa,76d874dfe6aaf3b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:24.139 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:24.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:24.974 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:28:24.974 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:28:24.983 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:28:24.983 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:28:24.983 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:28:24.983 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:28:24.989 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:28:24.989 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:28:24.989 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:28:24.989 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:28:25.028 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:28:25.037 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:28:25.037 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:28:25.038 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:28:25.038 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:28:25.038 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:28:25.047 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:28:25.576 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:28:25.583 WARN 21062 --- [ocp-async-9,d5845cce5dd1231c,8a3c31d7671d4e7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:25.588 INFO 21062 --- [ocp-async-1,afcef821afc0740c,78ce349723b6f8dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:26.147 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.158787064S 2024-11-04 15:28:26.148 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:28:26.245 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:28:26.246 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:28:26.270 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:28:26.327 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:28:26.327 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:28:26.327 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:28:26.327 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:28:26.327 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:28:26.328 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:28:26.338 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:28:26.523 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] o.s.boot.SpringApplication : Started application in 1.54 seconds (JVM running for 10720.203) 2024-11-04 15:28:26.524 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.540844875S 2024-11-04 15:28:26.945 INFO 21062 --- [ocp-schedules-3,040b0f38c4db7e6e,f7c7abb4cb8f1e3e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:28:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f7a707cb79cb00ac,56b0fff878b65083] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:26.984 INFO 21062 --- [prometheus-hour-schedule-1,36d612db5318377c,e18af5ed653a6807] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:28:27.110 INFO 21062 --- [ocp-schedules-25,71803d6127b2963d,c98f24fa7254736f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:28:27.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:27.140 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:27.583 WARN 21062 --- [ocp-async-1,fbcac943ac405743,484827e3c316cc3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:27.588 INFO 21062 --- [ocp-async-6,981e8ea03741e307,fd5db60ca154462c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:28.498 INFO 21062 --- [ocp-schedules-15,68b68a7e319fa052,4a87ece54babb36a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:28:28.499 INFO 21062 --- [ocp-schedules-30,af37719eeeab1d0f,f6e2cc0cca89db35] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:28:28.499 INFO 21062 --- [ocp-schedules-16,f1f70653d37b52ec,3f4ad2a7a15cda3f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:28:28.500 INFO 21062 --- [ocp-schedules-5,0a568d16783a7d17,061e22a55e5009c2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:28:28.502 INFO 21062 --- [ocp-schedules-15,68b68a7e319fa052,4a87ece54babb36a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:28:28.502 INFO 21062 --- [ocp-schedules-14,68694243906bf0d7,4dc85e60b7128e25] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:28:28.503 INFO 21062 --- [ocp-schedules-16,f1f70653d37b52ec,3f4ad2a7a15cda3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:28:28.504 INFO 21062 --- [ocp-schedules-30,af37719eeeab1d0f,f6e2cc0cca89db35] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:28:28.504 INFO 21062 --- [ocp-schedules-5,0a568d16783a7d17,061e22a55e5009c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:28.504 INFO 21062 --- [ocp-schedules-27,3a4a881e2d68a78f,00f42182bb180749] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:28:28.509 INFO 21062 --- [ocp-schedules-29,90673dfcd1eb7ce9,303d811d5a89f0e9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:28:28.509 INFO 21062 --- [ocp-schedules-29,90673dfcd1eb7ce9,303d811d5a89f0e9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:28:28.509 INFO 21062 --- [ocp-schedules-10,80695251fdc292ef,3267ec2583d5b03a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:28:28.509 INFO 21062 --- [ocp-schedules-10,80695251fdc292ef,3267ec2583d5b03a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:28:28.509 INFO 21062 --- [ocp-schedules-28,cad925e6b7c9da30,f4d28f1fa01b9d76] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:28:28.509 INFO 21062 --- [ocp-schedules-28,cad925e6b7c9da30,f4d28f1fa01b9d76] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:28:28.589 INFO 21062 --- [ocp-schedules-23,f94348c80a51e614,f8deee173ce7494f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:28:28.589 INFO 21062 --- [ocp-schedules-23,f94348c80a51e614,f8deee173ce7494f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:28:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:28:28.588565+08:00, lastProcessTime=2024-11-04T15:28:18.590350+08:00 2024-11-04 15:28:28.742 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:28:28.588565+08:00 2024-11-04 15:28:28.765 INFO 21062 --- [ocp-async-9,1c1e740fe225063c,1d6c9a3ec6121c7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:28:28.996 INFO 21062 --- [ocp-schedules-13,ebe59771e90a2729,49ef41b6b3854c2f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:28:29.329 INFO 21062 --- [ocp-schedules-27,1439bd9bfdb85d7f,b6d2bb1814aeca74] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:28:29.333 INFO 21062 --- [ocp-schedules-27,1439bd9bfdb85d7f,b6d2bb1814aeca74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:29.333 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:28:29.393 INFO 21062 --- [ocp-schedules-4,8727535583e62c45,bbb610c6cfeae092] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:28:29.393 INFO 21062 --- [ocp-schedules-4,8727535583e62c45,bbb610c6cfeae092] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:28:29.533 INFO 21062 --- [ocp-schedules-17,659d1f2c9fb526d4,42d77b5ac6f1d38e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:28:29.554 INFO 21062 --- [ocp-schedules-2,b3129ae94ab12bcf,f0ca94647c64bc72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:28:29.560 INFO 21062 --- [ocp-schedules-2,b3129ae94ab12bcf,f0ca94647c64bc72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:28:29.560 INFO 21062 --- [ocp-schedules-2,b3129ae94ab12bcf,f0ca94647c64bc72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:28:29.584 WARN 21062 --- [ocp-async-10,dbefaa9e20728819,d565ac1669f8e0f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:29.588 INFO 21062 --- [ocp-async-3,07c1ab3c531a6c8a,75a76a1dbd861f75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:29.856 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:28:29.856 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:28:29.865 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:28:29.865 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:28:29.865 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:28:29.876 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:28:29.876 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:28:29.876 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:28:29.880 INFO 21062 --- [ocp-schedules-8,4453b9170d71a452,05864be8968cbc0f] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:28:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7cad4a7365ccf9ee,61441b4d1109fe53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:30.033 INFO 21062 --- [ocp-schedules-22,73d7a875ae2fca64,e4eff9ace2ab0f5d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:28:30.033 INFO 21062 --- [ocp-schedules-22,73d7a875ae2fca64,e4eff9ace2ab0f5d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:28:30.038 INFO 21062 --- [ocp-schedules-22,73d7a875ae2fca64,e4eff9ace2ab0f5d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:28:30.038 INFO 21062 --- [ocp-schedules-22,73d7a875ae2fca64,e4eff9ace2ab0f5d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:28:30.042 INFO 21062 --- [ocp-schedules-22,73d7a875ae2fca64,e4eff9ace2ab0f5d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:28:30.042 INFO 21062 --- [ocp-schedules-22,73d7a875ae2fca64,e4eff9ace2ab0f5d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:28:30.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:30.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:28:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:28:31.584 WARN 21062 --- [ocp-async-3,73bebf0c4f2efa19,14875da2dd8cc5f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:31.588 INFO 21062 --- [ocp-async-2,cef4192c6d4aec31,6443b6c1abea6f78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:31.722 INFO 21062 --- [ocp-schedules-17,97cece06712fa5ae,00636ab41f0d6666] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:28:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8bacd0f984c37dbe,6a5292289975071f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:33.108 INFO 21062 --- [ocp-schedules-1,7ee307a9a261da90,6035c3b3cd037d01] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:28:33.113 INFO 21062 --- [ocp-schedules-1,7ee307a9a261da90,6035c3b3cd037d01] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:28:33.113 INFO 21062 --- [ocp-schedules-1,7ee307a9a261da90,6035c3b3cd037d01] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:28:33.141 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:33.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:33.346 INFO 21062 --- [ocp-schedules-6,508ff06dafb249b7,1a1432ebb34ce22c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:28:33.346 INFO 21062 --- [ocp-schedules-6,508ff06dafb249b7,1a1432ebb34ce22c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:28:33.371 INFO 21062 --- [ocp-schedules-6,508ff06dafb249b7,1a1432ebb34ce22c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:28:33.371 INFO 21062 --- [ocp-schedules-6,508ff06dafb249b7,1a1432ebb34ce22c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:28:33.372 INFO 21062 --- [ocp-schedules-19,98b020d8b3c25dd9,fe4f7559e95637cf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:28:33.581 INFO 21062 --- [ocp-schedules-26,a4fae3bc74da3b93,e498b4af0f53949d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:28:33.581+08:00, clusters=[1] 2024-11-04 15:28:33.583 INFO 21062 --- [ocp-schedules-26,a4fae3bc74da3b93,e498b4af0f53949d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:28:33.584 WARN 21062 --- [ocp-async-2,79358b1e4acd8f69,1d582d0da948841d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:33.588 INFO 21062 --- [ocp-async-6,b105796881e6c25a,d4807f42756aa104] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:33.896 INFO 21062 --- [ocp-schedules-18,ab5c0fc2378bc9b2,7cb92f0b7a5ac942] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:28:33.896+08:00, clusters=[1] 2024-11-04 15:28:33.900 INFO 21062 --- [ocp-schedules-18,ab5c0fc2378bc9b2,7cb92f0b7a5ac942] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:28:33.936 INFO 21062 --- [ocp-schedules-23,647ea05551f6f903,1989e18efe5af3e8] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:28:33.939 INFO 21062 --- [ocp-schedules-23,647ea05551f6f903,1989e18efe5af3e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:33.996 ERROR 21062 --- [ocp-schedules-23,647ea05551f6f903,1989e18efe5af3e8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:28:33.996 INFO 21062 --- [ocp-schedules-23,647ea05551f6f903,1989e18efe5af3e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:28:35.584 WARN 21062 --- [ocp-async-7,53774a1917eb03fc,70f4d54588a12ee1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:35.589 INFO 21062 --- [ocp-async-9,dddc13a1860cff33,19bbefd7fe8a203f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:35.751 INFO 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:28:35.751 INFO 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:28:35.751 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:28:35.751 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:28:35.760 INFO 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:28:35.761 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:28:35.761 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:28:35.762 WARN 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:28:35.762 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:28:35.762 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:28:35.763 INFO 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:28:35.763 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:28:35.763 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:28:35.764 WARN 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:28:35.764 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:28:35.764 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:28:35.771 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:28:35.771 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:28:35.771 INFO 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:28:35.782 ERROR 21062 --- [ocp-schedules-13,24c5471e6e870e0a,60789f868c5315f4] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:28:35.983 INFO 21062 --- [prometheus-minute-schedule-1,6f5af4e393f570fa,2af3db2b50ac1f93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:36.142 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:36.143 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:37.140 INFO 21062 --- [ocp-schedules-23,ba4a12a296e349ec,2cbd3f1cc48d15a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:28:37.584 WARN 21062 --- [ocp-async-4,157f0171b1814821,02c41b90f6c921ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:37.589 INFO 21062 --- [ocp-async-1,5a76d44b75098a66,b251b95d96b4ac19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:38.504 INFO 21062 --- [ocp-schedules-16,f1f70653d37b52ec,3f4ad2a7a15cda3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:28:38.504 INFO 21062 --- [ocp-schedules-5,0a568d16783a7d17,061e22a55e5009c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:28:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:28:38.590350+08:00, lastProcessTime=2024-11-04T15:28:28.588565+08:00 2024-11-04 15:28:38.745 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:28:38.590350+08:00 2024-11-04 15:28:38.765 INFO 21062 --- [ocp-async-6,bf02bce29ab87b5f,f0d698a54de627bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:28:38.983 INFO 21062 --- [prometheus-minute-schedule-1,f0988709a7787eaa,7d4804dcf549b1db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:39.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:39.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:39.333 INFO 21062 --- [ocp-schedules-27,1439bd9bfdb85d7f,b6d2bb1814aeca74] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:28:39.584 WARN 21062 --- [ocp-async-5,e2269e2f5227d7f0,5ee2e6fad63ded5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:39.589 INFO 21062 --- [ocp-async-7,322d5a02b781bc02,bca2f57c528702ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:40.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:28:40.003 INFO 21062 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:28:40.548 INFO 21062 --- [ocp-schedules-12,3f18ef0c7534d5b7,3708269b43f5dee2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:28:40.548 INFO 21062 --- [ocp-schedules-12,a5685d8c676df799,c4922835929f337b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:28:40.552 INFO 21062 --- [ocp-schedules-12,a5685d8c676df799,c4922835929f337b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:40.579 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:28:40.580 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:28:41.585 WARN 21062 --- [ocp-async-10,a3f317488e16ff3a,35d6baaafe148f85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:41.589 INFO 21062 --- [ocp-async-4,ecc3870259993d7e,72b96974404dc9f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:41.724 INFO 21062 --- [ocp-schedules-29,d774b4f02a034aac,01ca9aca7594d82d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:28:41.821 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:28:41.822 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:28:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:28:41.823 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:28:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:28:41.849 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:28:41.849 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=25ms, period=120000ms 2024-11-04 15:28:41.983 INFO 21062 --- [prometheus-minute-schedule-1,30f833ea6a94305c,b43429d53ec7eb5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:42.144 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:42.145 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:43.585 WARN 21062 --- [ocp-async-3,c780898966c68fa9,4cb843c63f8eadfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:43.589 INFO 21062 --- [ocp-async-8,d88f3c6461e7330c,40cad741b36f8ee2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:44.796 INFO 21062 --- [ocp-schedules-1,0c6438efa5dd40a4,631e34733412d7bd] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:28:44.801 INFO 21062 --- [ocp-schedules-1,0c6438efa5dd40a4,631e34733412d7bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:44.884 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:28:44.886 ERROR 21062 --- [ocp-schedules-1,0c6438efa5dd40a4,631e34733412d7bd] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:28:44.886 INFO 21062 --- [ocp-schedules-1,0c6438efa5dd40a4,631e34733412d7bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:28:44.983 INFO 21062 --- [prometheus-minute-schedule-1,fb74f49f806ac8c3,636ff58425155a49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:45.145 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:45.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:45.585 WARN 21062 --- [ocp-async-2,f41d7775516427ce,a2ab4c68247b7cd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:45.590 INFO 21062 --- [ocp-async-6,ba0805459ba1d424,331108c304abf8ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:47.165 INFO 21062 --- [ocp-schedules-4,f2288fb6603c8664,8965532d2f2f2a5d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:28:47.585 WARN 21062 --- [ocp-async-7,4f14828cbf132369,6fa295b3d5edc3dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:47.590 INFO 21062 --- [ocp-async-9,76925179d2ce0367,a9cb95d9a9716cf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:47.983 INFO 21062 --- [prometheus-minute-schedule-1,56c01b35b074fe7d,4ff039b53e034f24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:48.146 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:48.147 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:28:48.590848+08:00, lastProcessTime=2024-11-04T15:28:38.590350+08:00 2024-11-04 15:28:48.751 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:28:48.590848+08:00 2024-11-04 15:28:48.765 INFO 21062 --- [ocp-async-1,d3050afb55affa65,46df75ac6eea0358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:28:49.585 WARN 21062 --- [ocp-async-3,c1efc68fe4ef4b54,f9b3f7860939f383] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:49.590 INFO 21062 --- [ocp-async-8,4f11174d56ac39f1,7249828607329313] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:50.983 INFO 21062 --- [prometheus-minute-schedule-1,09a10a8fda57a342,972bdf8890fce6df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:51.147 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:51.148 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:51.585 WARN 21062 --- [ocp-async-2,6a28e8a44c8c74d7,86f29f2ab432b468] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:51.590 INFO 21062 --- [ocp-async-6,ca572e086b37bd2d,dd442e2af5a17a67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:51.727 INFO 21062 --- [ocp-schedules-22,7bd281886641148d,506354cbee0dd20f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:28:53.116 INFO 21062 --- [ocp-schedules-3,879e0bf473f93475,c63fb05e87b27f28] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:28:53.121 INFO 21062 --- [ocp-schedules-3,879e0bf473f93475,c63fb05e87b27f28] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:28:53.121 INFO 21062 --- [ocp-schedules-3,879e0bf473f93475,c63fb05e87b27f28] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:28:53.586 WARN 21062 --- [ocp-async-7,8dd13fdc12cb2564,a925db45d415423d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:53.590 INFO 21062 --- [ocp-async-9,cd831ffd211fe67b,4d3acb35d43e2c48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:53.983 INFO 21062 --- [prometheus-minute-schedule-1,780b1db288131cf0,bfd7aee1099a89d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:54.148 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:54.149 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:55.586 WARN 21062 --- [ocp-async-4,be7d7e82f0c91376,9de54242c4822ab1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:55.590 INFO 21062 --- [ocp-async-1,b1623b2fe5af14dc,b32ab46ff20227d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:56.983 INFO 21062 --- [prometheus-minute-schedule-1,cb7426b0654c8daf,dd6322f04ebacf95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:28:57.149 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:28:57.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:28:57.191 INFO 21062 --- [ocp-schedules-6,e481a7b7e8402ea4,f89387f944932f30] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:28:57.586 WARN 21062 --- [ocp-async-8,a65b029aaa77b215,a9581c4ed986f29b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:57.591 INFO 21062 --- [ocp-async-5,c9785c7a4db8ad4f,754951c790fda03f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:58.500 INFO 21062 --- [ocp-schedules-22,7dd166d4da0b4d26,a13b6af3fe5f1357] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:28:58.500 INFO 21062 --- [ocp-schedules-14,ef866ab0cd7dc573,3aeadb1a6dc4535f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:28:58.502 INFO 21062 --- [ocp-schedules-27,8f3212e5bcc85997,9979b4b75306c1ae] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:28:58.504 INFO 21062 --- [ocp-schedules-22,7dd166d4da0b4d26,a13b6af3fe5f1357] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:58.504 INFO 21062 --- [ocp-schedules-14,ef866ab0cd7dc573,3aeadb1a6dc4535f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:28:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:28:58.505 INFO 21062 --- [ocp-schedules-19,38952c9803eaf872,bf18a84162145563] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:28:58.509 INFO 21062 --- [ocp-schedules-6,7fff7b949585e73a,0f911dcaab67647b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:28:58.510 INFO 21062 --- [ocp-schedules-6,7fff7b949585e73a,0f911dcaab67647b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:28:58.510 INFO 21062 --- [ocp-schedules-5,e5cad0bd85e7c4c6,703d68234fca875d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:28:58.510 INFO 21062 --- [ocp-schedules-17,cdf275e02ab12d6c,bbcb8ce76b1bad9c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:28:58.510 INFO 21062 --- [ocp-schedules-5,e5cad0bd85e7c4c6,703d68234fca875d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:28:58.510 INFO 21062 --- [ocp-schedules-17,cdf275e02ab12d6c,bbcb8ce76b1bad9c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:28:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:28:58.589944+08:00, lastProcessTime=2024-11-04T15:28:48.590848+08:00 2024-11-04 15:28:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:28:58.589944+08:00 2024-11-04 15:28:58.765 INFO 21062 --- [ocp-async-10,18512729c1984d36,3d990e906266371a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:28:59.536 INFO 21062 --- [ocp-schedules-30,30e021378efff8dd,693485daa7326200] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:28:59.586 WARN 21062 --- [ocp-async-7,6b64f9e28ce78bf3,4e5b11b0043f5cf7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:28:59.591 INFO 21062 --- [ocp-async-9,0db14a364fd7b08e,0306d3b2e61e28ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:28:59.983 INFO 21062 --- [prometheus-minute-schedule-1,7ad37f5c698da812,4b8f45a2f194103d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705340, interval=60 2024-11-04 15:29:00.150 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:00.151 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:01.586 WARN 21062 --- [ocp-async-4,5438f1f717f19470,aa5ef42546c6baac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:01.591 INFO 21062 --- [ocp-async-1,eb714964c238f8c5,6dca3b7a0245fa32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:01.731 INFO 21062 --- [ocp-schedules-15,18be1b3e1018d01d,1fb95dafb4545089] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:29:02.983 INFO 21062 --- [prometheus-minute-schedule-1,8577f105c370424d,3722cde366c7fc71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:03.151 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:03.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:03.587 WARN 21062 --- [ocp-async-8,b12a632b2ce0b75b,cf8d75d4f9d3a6c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:03.591 INFO 21062 --- [ocp-async-5,52269aa5f890c166,c93a6d9c171a2141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:05.587 WARN 21062 --- [ocp-async-6,ee2bc43da851cfe3,0a7ef660cb6ea9f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:05.591 INFO 21062 --- [ocp-async-10,9ffb075c511f47ad,737b8e9e1d32ceb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2eda336d2a579e80,60cf4120e1ae19b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:06.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:06.152 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:07.218 INFO 21062 --- [ocp-schedules-16,411e6a3eb05d3880,df4e4ece6ac95ac9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:29:07.587 WARN 21062 --- [ocp-async-9,4722673c651f38a1,383c9ffd2f49f223] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:07.591 INFO 21062 --- [ocp-async-3,99f6247d1caa3090,7445cb8a3bc67a81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:08.505 INFO 21062 --- [ocp-schedules-22,7dd166d4da0b4d26,a13b6af3fe5f1357] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:29:08.505 INFO 21062 --- [ocp-schedules-14,ef866ab0cd7dc573,3aeadb1a6dc4535f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:29:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:29:08.589726+08:00, lastProcessTime=2024-11-04T15:28:58.589944+08:00 2024-11-04 15:29:08.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:29:08.589726+08:00 2024-11-04 15:29:08.766 INFO 21062 --- [ocp-async-2,c368b0e84c2a6a48,48e1f7d5b36bc6f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:29:08.983 INFO 21062 --- [prometheus-minute-schedule-1,aec6f3ecfa284c41,c06c8ab7897fea94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:09.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:09.153 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:09.587 WARN 21062 --- [ocp-async-8,3af20c97ffbaeeb9,180f55b6b3366b8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:09.592 INFO 21062 --- [ocp-async-5,3142fd5e41694380,c139728ff83c0ae1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:10.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:29:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:29:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:29:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:29:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:29:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:29:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:29:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:29:10.580 INFO 21062 --- [ocp-schedules-12,a5685d8c676df799,c4922835929f337b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:29:10.580 INFO 21062 --- [ocp-schedules-12,10566332f4299f7b,f6ac61688b9ee7b3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:29:10.584 INFO 21062 --- [ocp-schedules-12,10566332f4299f7b,f6ac61688b9ee7b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:10.614 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:29:10.614 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:29:11.587 WARN 21062 --- [ocp-async-6,bce61e7c29a128f8,52fa46eee0eebe05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:11.592 INFO 21062 --- [ocp-async-10,946135036bd1ca3c,5ec401f9bef96b42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:11.734 INFO 21062 --- [ocp-schedules-11,f6de47f5f347e0b4,1af05a161f384e46] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:29:11.845 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=3, affectRows=3 2024-11-04 15:29:11.845 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=2, validCount=2 2024-11-04 15:29:11.845 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=5, affectRows=5 2024-11-04 15:29:11.845 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=10, affectRows=10 2024-11-04 15:29:11.845 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=5, validCount=5 2024-11-04 15:29:11.845 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=10, validCount=10 2024-11-04 15:29:11.846 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=21, affectRows=21 2024-11-04 15:29:11.846 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=23, affectRows=23 2024-11-04 15:29:11.846 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=21, validCount=21 2024-11-04 15:29:11.846 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=23, validCount=23 2024-11-04 15:29:11.846 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=57, affectRows=57 2024-11-04 15:29:11.847 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=57, validCount=57 2024-11-04 15:29:11.847 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=36, affectRows=36 2024-11-04 15:29:11.847 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=46, affectRows=46 2024-11-04 15:29:11.847 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=49, affectRows=49 2024-11-04 15:29:11.847 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=36, validCount=36 2024-11-04 15:29:11.847 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=46, validCount=46 2024-11-04 15:29:11.847 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=49, validCount=49 2024-11-04 15:29:11.848 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=81, affectRows=81 2024-11-04 15:29:11.848 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705280, nodeCount=91, affectRows=91 2024-11-04 15:29:11.848 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=81, validCount=81 2024-11-04 15:29:11.848 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705280, nodeCount=89, validCount=89 2024-11-04 15:29:11.983 INFO 21062 --- [prometheus-minute-schedule-1,4f6ae21ea6f04874,d4c5473ebb1d04e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:12.154 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:12.154 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:13.123 INFO 21062 --- [ocp-schedules-20,b6fd45bc17c0390e,2a359aa21abb86be] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:29:13.129 INFO 21062 --- [ocp-schedules-20,b6fd45bc17c0390e,2a359aa21abb86be] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:29:13.129 INFO 21062 --- [ocp-schedules-20,b6fd45bc17c0390e,2a359aa21abb86be] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:29:13.588 WARN 21062 --- [ocp-async-9,28bdd4aa652c68de,e2b3013632374d05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:13.592 INFO 21062 --- [ocp-async-3,8b4fe864114a97fc,f5ff42ffb3d595fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:13.984 INFO 21062 --- [prometheus-cache-monitor1,ecb93c13584fcf38,2927cbfa1e71d400] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:29:13.985 INFO 21062 --- [prometheus-cache-monitor1,ecb93c13584fcf38,2927cbfa1e71d400] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:29:14.983 INFO 21062 --- [prometheus-minute-schedule-1,b3b434f75249df8d,bfa2258aa7163775] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:15.154 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:15.155 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:15.588 WARN 21062 --- [ocp-async-1,7ad986ef72831b1a,2d3235272fcf997e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:15.592 INFO 21062 --- [ocp-async-2,953d2ad2239c4ab6,42dc063db34c8c3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:15.657 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:25:15.634, end=2024-11-04T15:27:15.654} 2024-11-04 15:29:15.665 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:29:15.665 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:29:17.275 INFO 21062 --- [ocp-schedules-22,0056e0ada9a3cf44,4e8d325ce2ff40ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:29:17.588 WARN 21062 --- [ocp-async-5,d7bc2d361d0c1dcf,ecfa67682783bfab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:17.592 INFO 21062 --- [ocp-async-7,9684c5b5d9638de9,b402463f84b7d271] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:17.983 INFO 21062 --- [prometheus-minute-schedule-1,cad7c73071cb2099,0ae65cffbc9298c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:18.156 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:18.156 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:29:18.590102+08:00, lastProcessTime=2024-11-04T15:29:08.589726+08:00 2024-11-04 15:29:18.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:29:18.590102+08:00 2024-11-04 15:29:18.766 INFO 21062 --- [ocp-async-9,76831c36e55100f1,176885bd8d9cb309] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:29:19.588 WARN 21062 --- [ocp-async-9,de3f3a22afdfb353,b06210911a6b6238] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:19.593 INFO 21062 --- [ocp-async-3,7eec7516be125f24,134a5a8956b10afb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f25e37451202a494,68bbd0aee531efca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:21.157 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:21.157 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:21.588 WARN 21062 --- [ocp-async-1,8c8c7bd8f5eeaddb,09b77a5004e9641a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:21.593 INFO 21062 --- [ocp-async-2,33092d7643f1fc39,cdce8bba574dadba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:21.737 INFO 21062 --- [ocp-schedules-29,b425e391595bbd19,3ac946c054c00e83] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:29:23.588 WARN 21062 --- [ocp-async-2,160c21baee62310c,77a8f0587a5a96ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:23.593 INFO 21062 --- [ocp-async-7,96695f1a05f3f68c,cab9f46eaf6963ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:23.983 INFO 21062 --- [prometheus-minute-schedule-1,efa21fc2cbcdd92f,d282f13d2c3ed59d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:24.157 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:24.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:25.589 WARN 21062 --- [ocp-async-7,e91165a1144e32bd,b151865bca48684e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:25.593 INFO 21062 --- [ocp-async-4,abff17e95e7a6fa4,027affd73434531d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:26.983 INFO 21062 --- [prometheus-minute-schedule-1,b984aca2526cde4a,936dc485c78f7149] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:26.984 INFO 21062 --- [prometheus-hour-schedule-1,90685b57d5e4db5e,dfab6fdf10d91016] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:29:27.158 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:27.159 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:27.310 INFO 21062 --- [ocp-schedules-10,2c4cd41e6d086694,a392ea369349d8fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:29:27.589 WARN 21062 --- [ocp-async-4,b25f2b536a3e1d94,45dd6fcf58120782] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:27.593 INFO 21062 --- [ocp-async-1,10a11b4f13825125,4db0f2558873f5fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:28.498 INFO 21062 --- [ocp-schedules-4,12d36415da701b02,3c8f03098a71041e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:29:28.499 INFO 21062 --- [ocp-schedules-18,6ae8cf2a28e60223,da6a69b5e9485ae5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:29:28.500 INFO 21062 --- [ocp-schedules-22,f6ea060fc2a6e86e,7797677acd6e9fb9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:29:28.500 INFO 21062 --- [ocp-schedules-2,265f8b8162d75d00,eb1281667b470442] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:29:28.502 INFO 21062 --- [ocp-schedules-4,12d36415da701b02,3c8f03098a71041e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:29:28.502 INFO 21062 --- [ocp-schedules-6,4d1163bc64debe63,7031b5f09fec5c6c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:29:28.503 INFO 21062 --- [ocp-schedules-22,f6ea060fc2a6e86e,7797677acd6e9fb9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:28.504 INFO 21062 --- [ocp-schedules-2,265f8b8162d75d00,eb1281667b470442] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:29:28.505 INFO 21062 --- [ocp-schedules-14,826127126aa6a45b,04275e572cc98877] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:29:28.505 INFO 21062 --- [ocp-schedules-18,6ae8cf2a28e60223,da6a69b5e9485ae5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:29:28.510 INFO 21062 --- [ocp-schedules-10,1c571c0fef252a81,a2ada1f115292d66] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:29:28.510 INFO 21062 --- [ocp-schedules-10,1c571c0fef252a81,a2ada1f115292d66] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:29:28.510 INFO 21062 --- [ocp-schedules-5,de84ed8316d37956,82c88180585323c4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:29:28.510 INFO 21062 --- [ocp-schedules-5,de84ed8316d37956,82c88180585323c4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:29:28.510 INFO 21062 --- [ocp-schedules-1,fc1130de38d0f941,fe1593d3df12f973] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:29:28.510 INFO 21062 --- [ocp-schedules-1,fc1130de38d0f941,fe1593d3df12f973] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:29:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:29:28.589261+08:00, lastProcessTime=2024-11-04T15:29:18.590102+08:00 2024-11-04 15:29:28.622 INFO 21062 --- [ocp-schedules-9,a03a1e591ae7e009,ee3e9cad63102a19] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:29:28.622+08:00, tenantSize=0 2024-11-04 15:29:28.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:29:28.589261+08:00 2024-11-04 15:29:28.766 INFO 21062 --- [ocp-async-7,3748631cc88d853e,4371f01656ab091e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:29:28.998 INFO 21062 --- [ocp-schedules-13,78caf422873062ad,2826b3f3ad3d8895] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:29:29.398 INFO 21062 --- [ocp-schedules-4,d77326fc54bb6f1d,da510819903fd55c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:29:29.398 INFO 21062 --- [ocp-schedules-4,d77326fc54bb6f1d,da510819903fd55c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:29:29.539 INFO 21062 --- [ocp-schedules-28,f6c7393238410722,7535eb9db87159f8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:29:29.560 INFO 21062 --- [ocp-schedules-30,3099b42005eed31e,cfc1ae8a09870212] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:29:29.566 INFO 21062 --- [ocp-schedules-30,3099b42005eed31e,cfc1ae8a09870212] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:29:29.566 INFO 21062 --- [ocp-schedules-30,3099b42005eed31e,cfc1ae8a09870212] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:29:29.589 WARN 21062 --- [ocp-async-5,8323bda2400951b0,d68545b5d4200a21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:29.594 INFO 21062 --- [ocp-async-9,08fe2b4d8e2d8cb8,bbdb362bf29fe60f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:29.983 INFO 21062 --- [prometheus-minute-schedule-1,72df7222e3940df5,2790b09da6a42039] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:30.047 INFO 21062 --- [ocp-schedules-24,0b0a674ccfc5dfbd,f2a8e7406fbeb53c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:29:30.048 INFO 21062 --- [ocp-schedules-24,0b0a674ccfc5dfbd,f2a8e7406fbeb53c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:29:30.050 INFO 21062 --- [ocp-schedules-24,0b0a674ccfc5dfbd,f2a8e7406fbeb53c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:29:30.050 INFO 21062 --- [ocp-schedules-24,0b0a674ccfc5dfbd,f2a8e7406fbeb53c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:29:30.051 INFO 21062 --- [ocp-schedules-24,0b0a674ccfc5dfbd,f2a8e7406fbeb53c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:29:30.051 INFO 21062 --- [ocp-schedules-24,0b0a674ccfc5dfbd,f2a8e7406fbeb53c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:29:30.159 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:30.160 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:29:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:29:31.589 WARN 21062 --- [ocp-async-9,29bb26c411143b5f,4d22f3cc595ef65a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:31.594 INFO 21062 --- [ocp-async-3,3e11e7d93a369d8c,f22fc40e98f17e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:31.740 INFO 21062 --- [ocp-schedules-8,9f69e720e389f38a,3abfc1f5e8a8dad0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:29:32.983 INFO 21062 --- [prometheus-minute-schedule-1,52c86c2bc25fcca6,dc4f09d412fb5fb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:33.131 INFO 21062 --- [ocp-schedules-21,6d62bbed7e805a99,c052f84556f0324c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:29:33.136 INFO 21062 --- [ocp-schedules-21,6d62bbed7e805a99,c052f84556f0324c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:29:33.136 INFO 21062 --- [ocp-schedules-21,6d62bbed7e805a99,c052f84556f0324c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:29:33.160 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:33.161 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:33.371 INFO 21062 --- [ocp-schedules-7,2b84ac1c14573c28,ca3022ac7b42e42a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:29:33.371 INFO 21062 --- [ocp-schedules-7,2b84ac1c14573c28,ca3022ac7b42e42a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:29:33.398 INFO 21062 --- [ocp-schedules-7,2b84ac1c14573c28,ca3022ac7b42e42a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:29:33.398 INFO 21062 --- [ocp-schedules-7,2b84ac1c14573c28,ca3022ac7b42e42a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:29:33.399 INFO 21062 --- [ocp-schedules-10,86a024bf47706d44,94fa7b8955c69bd5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:29:33.589 WARN 21062 --- [ocp-async-3,a7ba6b05417caa81,e930d0fbf2ceb022] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:33.594 INFO 21062 --- [ocp-async-1,4399a22d6a15db56,f88bb5a81f71945d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:33.611 INFO 21062 --- [ocp-schedules-11,a5b5974fa05b4eb2,d711bfc49bbc3e20] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:29:33.611+08:00, clusters=[1] 2024-11-04 15:29:33.612 INFO 21062 --- [ocp-schedules-11,a5b5974fa05b4eb2,d711bfc49bbc3e20] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:29:33.929 INFO 21062 --- [ocp-schedules-1,5319c5b44230a6e2,2242cc9e0c9b954e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:29:33.929+08:00, clusters=[1] 2024-11-04 15:29:33.933 INFO 21062 --- [ocp-schedules-1,5319c5b44230a6e2,2242cc9e0c9b954e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:29:35.589 WARN 21062 --- [ocp-async-6,a4108cf7e62816ea,23e5a25a05200d0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:35.594 INFO 21062 --- [ocp-async-7,eaf16485f5f2e1b7,84fcee08b70a914e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:35.793 INFO 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:29:35.793 INFO 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:29:35.793 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:29:35.793 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:29:35.803 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:29:35.804 INFO 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:29:35.804 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:29:35.805 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:29:35.806 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:29:35.806 WARN 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:29:35.806 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:29:35.806 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:29:35.806 INFO 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:29:35.807 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:29:35.807 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:29:35.807 WARN 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:29:35.816 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:29:35.816 INFO 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:29:35.816 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:29:35.829 ERROR 21062 --- [ocp-schedules-21,c5274ff224237d6d,e58752cb69b68480] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:29:35.983 INFO 21062 --- [prometheus-minute-schedule-1,9c587deaae6d1d06,65dbb4b770362af0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:36.161 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:36.162 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:37.358 INFO 21062 --- [ocp-schedules-27,3644e5252fb46303,e04cbff3dd03e5bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:29:37.590 WARN 21062 --- [ocp-async-10,2608b96ecd4beaac,fb15c1d17647b34a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:37.594 INFO 21062 --- [ocp-async-4,fe25cdd39c20a123,130764282502163a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:38.504 INFO 21062 --- [ocp-schedules-2,265f8b8162d75d00,eb1281667b470442] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:29:38.504 INFO 21062 --- [ocp-schedules-22,f6ea060fc2a6e86e,7797677acd6e9fb9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:29:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:29:38.588457+08:00, lastProcessTime=2024-11-04T15:29:28.589261+08:00 2024-11-04 15:29:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:29:38.588457+08:00 2024-11-04 15:29:38.766 INFO 21062 --- [ocp-async-1,1e1c5541243a61be,4580b60546a5f5d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:29:38.983 INFO 21062 --- [prometheus-minute-schedule-1,6b24dbb0800038fa,4f8e9662b3b21842] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:39.162 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:39.163 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:39.334 INFO 21062 --- [ocp-schedules-16,f77cea4b66d31b10,1c386bddcb5e58bd] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:29:39.337 INFO 21062 --- [ocp-schedules-16,f77cea4b66d31b10,1c386bddcb5e58bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:39.337 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:29:39.590 WARN 21062 --- [ocp-async-2,1821c6a761bbd9e9,a95febcb39daab87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:39.595 INFO 21062 --- [ocp-async-6,9bb27c17d46b851f,72269a5f3e1f3153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:29:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:29:40.615 INFO 21062 --- [ocp-schedules-12,10566332f4299f7b,f6ac61688b9ee7b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:29:40.615 INFO 21062 --- [ocp-schedules-12,e09bb3790cd5d6af,0b2d50e9e28ac79d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:29:40.619 INFO 21062 --- [ocp-schedules-12,e09bb3790cd5d6af,0b2d50e9e28ac79d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:40.648 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:29:40.648 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:29:41.590 WARN 21062 --- [ocp-async-5,d572639097921e60,05d04e17ac48532e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:41.595 INFO 21062 --- [ocp-async-10,0e9be1c67e1fac43,2ee36c5bb41434b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:41.742 INFO 21062 --- [ocp-schedules-4,50415b6ddb10e99e,6bdf285ccffba046] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:29:41.983 INFO 21062 --- [prometheus-minute-schedule-1,39b75f7bcb1f90ff,9aa3649b9832e640] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:42.163 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:42.164 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:43.590 WARN 21062 --- [ocp-async-9,5b061ff0997de035,54d26be752a736aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:43.595 INFO 21062 --- [ocp-async-8,4433df0bfdc07c00,63b244d0008f1fde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:44.886 INFO 21062 --- [ocp-schedules-18,4f13445ec039fbd7,1542c83b46c9f1ce] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:29:44.891 INFO 21062 --- [ocp-schedules-18,4f13445ec039fbd7,1542c83b46c9f1ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:44.978 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:29:44.980 ERROR 21062 --- [ocp-schedules-18,4f13445ec039fbd7,1542c83b46c9f1ce] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:29:44.981 INFO 21062 --- [ocp-schedules-18,4f13445ec039fbd7,1542c83b46c9f1ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:29:44.983 INFO 21062 --- [prometheus-minute-schedule-1,c2aee2f204ecf332,eb37510aead57269] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:45.164 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:45.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:45.590 WARN 21062 --- [ocp-async-3,8c92fbb0b30713c6,eff83762ac7d2f27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:45.595 INFO 21062 --- [ocp-async-1,12c226c181ed809c,e508f575f2d63654] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:47.395 INFO 21062 --- [ocp-schedules-17,7ce0204e33f3b682,315c520505c43840] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:29:47.591 WARN 21062 --- [ocp-async-6,fa197dad1ecdc9e1,1bba8f48c33b55af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:47.595 INFO 21062 --- [ocp-async-7,dac8c9cd2aa0eb56,b2a081ee3bf74202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:47.983 INFO 21062 --- [prometheus-minute-schedule-1,79a06ffc1faef56d,855b1fc3db9a5395] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:48.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:48.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:29:48.590148+08:00, lastProcessTime=2024-11-04T15:29:38.588457+08:00 2024-11-04 15:29:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:29:48.590148+08:00 2024-11-04 15:29:48.766 INFO 21062 --- [ocp-async-4,4a7c0f6948380c03,6bebb622c831f253] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:29:49.338 INFO 21062 --- [ocp-schedules-16,f77cea4b66d31b10,1c386bddcb5e58bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:29:49.591 WARN 21062 --- [ocp-async-9,8f896169bb32f095,310562b9226322ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:49.596 INFO 21062 --- [ocp-async-8,c6e12ee3edef2e21,f7d9c85046b26991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a2e3743eb793647d,741dbe236a27000d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:51.165 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:51.166 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:51.591 WARN 21062 --- [ocp-async-3,74cd3777e69948f3,070891ef4bec77b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:51.596 INFO 21062 --- [ocp-async-1,3f33dcf3a8132f53,378e8eacf19c50c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:51.745 INFO 21062 --- [ocp-schedules-17,e0814fe3b6171538,fae2f363dd4b7a29] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:29:53.138 INFO 21062 --- [ocp-schedules-15,c2cf3ce32e2237aa,6b399a02e216b231] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:29:53.144 INFO 21062 --- [ocp-schedules-15,c2cf3ce32e2237aa,6b399a02e216b231] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:29:53.144 INFO 21062 --- [ocp-schedules-15,c2cf3ce32e2237aa,6b399a02e216b231] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:29:53.591 WARN 21062 --- [ocp-async-6,ef24ba974a9b2b00,158b0c86850ffe9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:53.596 INFO 21062 --- [ocp-async-7,1b6dcbfa552dab6b,2eee6c5da9645e00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4f3240196b38c9fe,b3bae715a9b1d09b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:54.166 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:54.167 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:55.591 WARN 21062 --- [ocp-async-10,397ada1101e37317,721a78de45285380] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:55.596 INFO 21062 --- [ocp-async-4,f8dfec9da835c0b7,234f570aef7d7868] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:56.983 INFO 21062 --- [prometheus-minute-schedule-1,686c83014aeed6ac,1f5ee2eb174083a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:29:57.167 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:29:57.168 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:29:57.439 INFO 21062 --- [ocp-schedules-7,1428f95e84bb82f6,756d29560ce803e1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:29:57.591 WARN 21062 --- [ocp-async-8,1b8a02798d0e7857,c85f02be24a216c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:57.596 INFO 21062 --- [ocp-async-2,222ff7f937aa4307,62c05aa8d747e9f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:58.500 INFO 21062 --- [ocp-schedules-2,caa180abe842790b,fdb3079675987d3d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:29:58.500 INFO 21062 --- [ocp-schedules-17,0f02a0e16a160669,92e402638ba058c9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:29:58.502 INFO 21062 --- [ocp-schedules-7,2cff049405eb8d98,ff67d04ecc10390e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:29:58.504 INFO 21062 --- [ocp-schedules-2,caa180abe842790b,fdb3079675987d3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:58.504 INFO 21062 --- [ocp-schedules-17,0f02a0e16a160669,92e402638ba058c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:29:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:29:58.504 INFO 21062 --- [ocp-schedules-22,1c78d49f9847b40b,510fb469088670d9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:29:58.509 INFO 21062 --- [ocp-schedules-1,8bfde571c7a8389b,6aa78e504712d604] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:29:58.509 INFO 21062 --- [ocp-schedules-26,f258d4a49be3b4f0,775267c8360dca11] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:29:58.510 INFO 21062 --- [ocp-schedules-1,8bfde571c7a8389b,6aa78e504712d604] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:29:58.510 INFO 21062 --- [ocp-schedules-26,f258d4a49be3b4f0,775267c8360dca11] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:29:58.510 INFO 21062 --- [ocp-schedules-11,0ebbb0ec88e399e9,b9c37722d29acbd5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:29:58.510 INFO 21062 --- [ocp-schedules-11,0ebbb0ec88e399e9,b9c37722d29acbd5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:29:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:29:58.589025+08:00, lastProcessTime=2024-11-04T15:29:48.590148+08:00 2024-11-04 15:29:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:29:58.589025+08:00 2024-11-04 15:29:58.767 INFO 21062 --- [ocp-async-5,69084b9d57971113,338370f3021b8d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:29:59.542 INFO 21062 --- [ocp-schedules-5,4a66ea8423c86516,2225e1f715503539] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:29:59.592 WARN 21062 --- [ocp-async-6,0cd09cb8589f6fc1,2a88fd9b01352243] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:29:59.596 INFO 21062 --- [ocp-async-7,e498750b61eb2d1a,33dc3bd1b76db739] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:29:59.983 INFO 21062 --- [prometheus-minute-schedule-1,93e8ffa697591d77,432f766cea2242d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705400, interval=60 2024-11-04 15:30:00.168 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:00.169 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:00.479 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,77152afb05dcdd60] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:25:01.0, nextRun=2024-11-04 15:30:00.0, currentDate=Mon Nov 04 15:30:00 CST 2024 2024-11-04 15:30:00.479 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,77152afb05dcdd60] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:25:01.0, nextRunTime=2024-11-04 15:30:00.0} 2024-11-04 15:30:00.505 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,77152afb05dcdd60] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:30:00.509 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,77152afb05dcdd60] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:30:00.509 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,77152afb05dcdd60] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:30:01.592 WARN 21062 --- [ocp-async-10,71d74f8f3738a214,ff6dbba26083bee6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:01.597 INFO 21062 --- [ocp-async-4,3d9b84f96be001bd,ec1333477f463afb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:01.748 INFO 21062 --- [ocp-schedules-18,ac8473a30a04a00c,a9fa5ffee9c1cb78] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:30:02.983 INFO 21062 --- [prometheus-minute-schedule-1,0aacb9de88c7c11c,d04f29856b10d173] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:03.169 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:03.170 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:03.592 WARN 21062 --- [ocp-async-8,5d38ef418785de1b,9f5e07d6a7c76026] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:03.597 INFO 21062 --- [ocp-async-2,259b63825d04f9af,89efb176d0330d43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:04.578 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,8a1eea2544755d47] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:20:04.0, nextRun=2024-11-04 15:30:04.0, currentDate=Mon Nov 04 15:30:04 CST 2024 2024-11-04 15:30:04.578 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,8a1eea2544755d47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 15:20:04.0, nextRunTime=2024-11-04 15:30:04.0} 2024-11-04 15:30:04.584 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,8a1eea2544755d47] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 15:30:04.584 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,8a1eea2544755d47] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T15:30:04.584+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 15:30:05.246 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,79793fd335d5f92d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 88, operation:EXECUTE, state: RUNNING 2024-11-04 15:30:05.266 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=88, context=Context{parallelIdx=-1, stringMap={task_instance_id=25, task_operation=execute, latest_execution_start_time=2024-11-04T15:30:05.238+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=88}, listMap={}}, executor=10.46.128.24 2024-11-04 15:30:05.271 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:30:05.271 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:30:05.273 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.273 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:30:05.274 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.274 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.275 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.275 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:30:05.291 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.291 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.292 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.292 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:30:05.306 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.306 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.307 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.307 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:30:05.322 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.322 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.323 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.323 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:30:05.337 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.337 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.338 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.338 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:30:05.344 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.344 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.345 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.345 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:30:05.350 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.350 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.351 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.351 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:30:05.355 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.356 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.356 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.357 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:30:05.363 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.363 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.364 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.364 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:30:05.370 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.370 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.371 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.371 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:30:05.377 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.377 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.378 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.378 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:30:05.387 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.388 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.389 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.389 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:30:05.402 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.403 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.403 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.404 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:30:05.409 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.409 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.410 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.410 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:30:05.415 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.416 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.416 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.417 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:30:05.423 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.423 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.424 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.424 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:30:05.429 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.429 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.430 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.430 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:30:05.434 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.435 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.435 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.436 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:30:05.440 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.441 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.441 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.442 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:30:05.446 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.447 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.447 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.448 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:30:05.452 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.452 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.452 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.453 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:30:05.457 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.457 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.458 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.458 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:30:05.462 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.463 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.463 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.464 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:30:05.468 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.469 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.469 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.469 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:30:05.474 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.474 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.474 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.475 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:30:05.479 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.479 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.480 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.480 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:30:05.484 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.485 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.485 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.485 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:30:05.490 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.490 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.491 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.491 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:30:05.495 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.496 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.496 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.496 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:30:05.501 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.502 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.502 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.502 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:30:05.507 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.507 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.508 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.508 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:30:05.512 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.513 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.513 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.513 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:30:05.518 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.518 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.519 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.519 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:30:05.523 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.524 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.524 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.525 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:30:05.532 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.532 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.533 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.533 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:30:05.539 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.540 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.540 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.541 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:30:05.546 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.547 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.548 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.548 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:30:05.553 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.553 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.554 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.554 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:30:05.558 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.558 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.559 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.559 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:30:05.563 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.563 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.564 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.564 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:30:05.568 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.568 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.569 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.569 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:30:05.573 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.573 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.574 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.574 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:30:05.583 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.584 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.584 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.584 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:30:05.592 WARN 21062 --- [ocp-async-1,bbf619844b9511ee,9bcd970a346c830d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:05.597 INFO 21062 --- [ocp-async-5,f72a1d33ed3cf1b1,65b00ab06778a494] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:05.605 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.605 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.606 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.606 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:30:05.612 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.612 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.613 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.613 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:30:05.619 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.619 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.620 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.620 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:30:05.627 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.627 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.628 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.628 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:30:05.632 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.633 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.633 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.633 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:30:05.637 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.638 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.638 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.639 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:30:05.643 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:30:05.643 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:30:05.644 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.644 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:30:05.645 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.646 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.646 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.647 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:30:05.652 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.653 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:30:05.653 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:30:05.653 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:30:05.657 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:30:05.664 INFO 21062 --- [subtask-executor21,dd193f8f67a5a423,90fe84a1a2537a18] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 88, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 15:30:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0748e83a6a5d5f3f,c1f3e37df5b23ec7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:06.170 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:06.171 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:06.226 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:25, status:SUCCESS 2024-11-04 15:30:06.227 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:25, status:SUCCESS 2024-11-04 15:30:07.470 INFO 21062 --- [ocp-schedules-21,28288fd11665ca13,0989bd2520d6ae31] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:30:07.592 WARN 21062 --- [ocp-async-7,e1ea85853bf9f4ba,43c7f9f94f2bea4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:07.597 INFO 21062 --- [ocp-async-9,fbd596ea5937e7d5,374e4bdd40d0dcb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:08.505 INFO 21062 --- [ocp-schedules-17,0f02a0e16a160669,92e402638ba058c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:30:08.505 INFO 21062 --- [ocp-schedules-2,caa180abe842790b,fdb3079675987d3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:30:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:30:08.590521+08:00, lastProcessTime=2024-11-04T15:29:58.589025+08:00 2024-11-04 15:30:08.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:30:08.590521+08:00 2024-11-04 15:30:08.767 INFO 21062 --- [ocp-async-8,938c8a7f268aecb0,d6b57e8b5655d6fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:30:08.983 INFO 21062 --- [prometheus-minute-schedule-1,25afebbc77c8a039,9a7d8e330fb18d14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:09.171 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:09.172 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:09.593 WARN 21062 --- [ocp-async-3,352dd2be7f9c3c4e,a19292b0e55e357a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:09.597 INFO 21062 --- [ocp-async-2,02028cfdffe18288,bfc74e7fddd8576a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:10.004 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:30:10.004 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:30:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:30:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:30:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:30:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:30:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:30:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:30:10.648 INFO 21062 --- [ocp-schedules-12,e09bb3790cd5d6af,0b2d50e9e28ac79d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:30:10.648 INFO 21062 --- [ocp-schedules-12,41a66a178ce20fe8,1ae6245894ae45d3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:30:10.652 INFO 21062 --- [ocp-schedules-12,41a66a178ce20fe8,1ae6245894ae45d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:10.680 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:30:10.680 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:30:11.593 WARN 21062 --- [ocp-async-1,ad2f299f9f4f3742,1c199f123093665c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:11.598 INFO 21062 --- [ocp-async-5,1134957832e56ff6,82890854df84af76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:11.751 INFO 21062 --- [ocp-schedules-10,bfca09a7c570d0eb,e892dc71fbb7ad88] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:30:11.848 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=5, affectRows=5 2024-11-04 15:30:11.848 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=5, validCount=5 2024-11-04 15:30:11.848 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=13, affectRows=13 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=13, validCount=13 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=23, affectRows=23 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=53, affectRows=53 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=23, validCount=23 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=52, validCount=52 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=49, affectRows=49 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=36, affectRows=36 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=66, affectRows=66 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=34, affectRows=34 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=49, validCount=49 2024-11-04 15:30:11.849 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=36, validCount=36 2024-11-04 15:30:11.850 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=66, validCount=66 2024-11-04 15:30:11.850 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=66, affectRows=66 2024-11-04 15:30:11.850 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=34, validCount=34 2024-11-04 15:30:11.850 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705340, nodeCount=77, affectRows=77 2024-11-04 15:30:11.850 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=65, validCount=65 2024-11-04 15:30:11.850 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705340, nodeCount=76, validCount=76 2024-11-04 15:30:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6251474ece013d37,1fd4da7a5d4741ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:12.172 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:12.173 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:13.146 INFO 21062 --- [ocp-schedules-9,cf2374f15e4d4c4b,efaea917658bb5d2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:30:13.151 INFO 21062 --- [ocp-schedules-9,cf2374f15e4d4c4b,efaea917658bb5d2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:30:13.151 INFO 21062 --- [ocp-schedules-9,cf2374f15e4d4c4b,efaea917658bb5d2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:30:13.593 WARN 21062 --- [ocp-async-7,e263ecc6d5b0e593,2e097e37bd1810f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:13.598 INFO 21062 --- [ocp-async-9,9c1e4269ba7d7852,b4f96beb2fbb02da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:13.984 INFO 21062 --- [prometheus-cache-monitor1,a4f165d1fc62cfb9,385cf429cc673bb0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:30:13.984 INFO 21062 --- [prometheus-cache-monitor1,a4f165d1fc62cfb9,385cf429cc673bb0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:30:14.983 INFO 21062 --- [prometheus-minute-schedule-1,bdcb2db4581d3f9d,dc5156b4f18da5af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:15.173 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:15.174 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:15.593 WARN 21062 --- [ocp-async-4,c84d158e0a37e0a1,6f878c18c4d17f3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:15.598 INFO 21062 --- [ocp-async-8,2de694004d96b3f2,603993f5ee1050fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:17.512 INFO 21062 --- [ocp-schedules-17,9ed8b36e6f6bc51c,8bf7a4651a0e8c25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:30:17.593 WARN 21062 --- [ocp-async-2,2643d135108a908d,2206eb1d40152232] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:17.598 INFO 21062 --- [ocp-async-6,91985196e9ed0816,0a3b3bbc42617912] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:17.983 INFO 21062 --- [prometheus-minute-schedule-1,87beb37032168858,e485067dba40139c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:18.174 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:18.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:30:18.589921+08:00, lastProcessTime=2024-11-04T15:30:08.590521+08:00 2024-11-04 15:30:18.767 INFO 21062 --- [ocp-async-7,5b547764dd5a0078,363d03772a0c26c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:30:18.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:30:18.589921+08:00 2024-11-04 15:30:19.593 WARN 21062 --- [ocp-async-7,49b7c6cf7c43178f,9f2ea3fe6f9f07ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:19.598 INFO 21062 --- [ocp-async-9,433dc5274c5528c4,122a9745bdca8c59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5326e54d50f0a84f,efa53d7ed472a472] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:21.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:21.175 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:21.594 WARN 21062 --- [ocp-async-4,a86471d63b4992dc,a81235dcefd253de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:21.598 INFO 21062 --- [ocp-async-8,dd2d5d84f8a3a467,8c9285943441f190] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:21.754 INFO 21062 --- [ocp-schedules-22,fd8a7180d1c933d0,60e223cb883a2655] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:30:23.594 WARN 21062 --- [ocp-async-8,020ab6203474ccc2,b5ea5b7ef3e9876d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:23.599 INFO 21062 --- [ocp-async-6,540e42f41071346e,7301f80d38372785] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:23.983 INFO 21062 --- [prometheus-minute-schedule-1,5298f4fec51219c7,e973de0e29fb7cb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:24.176 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:24.176 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:25.594 WARN 21062 --- [ocp-async-6,ba32a25691aa6c05,77e4f5bc63b6f4a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:25.599 INFO 21062 --- [ocp-async-7,2b0e0b6b503d7a4b,36e3586f4f5ae1a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e927fc8cbfd031fa,c366f3008fde2eb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5bb5099568c04886,87f950dd732b9903] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:30:27.177 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:27.177 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:27.544 INFO 21062 --- [ocp-schedules-7,d63708af1b880a0f,721caf5ef451617a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:30:27.594 WARN 21062 --- [ocp-async-7,e8ec231725b382cb,b6dd406327a83842] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:27.599 INFO 21062 --- [ocp-async-4,6842686ab8179db9,667dbe9a11ecdb41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:28.498 INFO 21062 --- [ocp-schedules-15,f10cfaf875f85905,b7463d83ebcd30e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:30:28.499 INFO 21062 --- [ocp-schedules-20,91cd0da457f12a95,ea9f159eeff86992] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:30:28.499 INFO 21062 --- [ocp-schedules-28,ab029f2710308ef7,3dce3152c22fa87b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:30:28.500 INFO 21062 --- [ocp-schedules-9,dd2ab7231415b583,ff4261735e9aed6e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:30:28.502 INFO 21062 --- [ocp-schedules-15,f10cfaf875f85905,b7463d83ebcd30e6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:30:28.502 INFO 21062 --- [ocp-schedules-7,7b0add2b1116af1d,02c07d0dffb9fbe7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:30:28.503 INFO 21062 --- [ocp-schedules-28,ab029f2710308ef7,3dce3152c22fa87b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:30:28.504 INFO 21062 --- [ocp-schedules-9,dd2ab7231415b583,ff4261735e9aed6e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:28.504 INFO 21062 --- [ocp-schedules-3,548ef3024d009d92,13297a40c4e99be8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:30:28.506 INFO 21062 --- [ocp-schedules-20,91cd0da457f12a95,ea9f159eeff86992] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:30:28.509 INFO 21062 --- [ocp-schedules-2,c2dd17691b9c0bb7,334df6b9b20747b3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:30:28.509 INFO 21062 --- [ocp-schedules-2,c2dd17691b9c0bb7,334df6b9b20747b3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:30:28.509 INFO 21062 --- [ocp-schedules-19,8817496575d4e152,1f29a762e6ecf3be] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:30:28.509 INFO 21062 --- [ocp-schedules-19,8817496575d4e152,1f29a762e6ecf3be] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:30:28.510 INFO 21062 --- [ocp-schedules-6,c71f1666a8dfa040,ffbdc375af9669f3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:30:28.510 INFO 21062 --- [ocp-schedules-6,c71f1666a8dfa040,ffbdc375af9669f3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:30:28.589 INFO 21062 --- [ocp-schedules-23,397f1fad78b5aab8,a21091ca481d404a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:30:28.589 INFO 21062 --- [ocp-schedules-23,397f1fad78b5aab8,a21091ca481d404a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:30:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:30:28.588893+08:00, lastProcessTime=2024-11-04T15:30:18.589921+08:00 2024-11-04 15:30:28.668 INFO 21062 --- [ocp-schedules-18,f165629796d69081,6a1b6bd6c50c2134] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:30:28.669 INFO 21062 --- [ocp-schedules-18,f165629796d69081,6a1b6bd6c50c2134] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:30:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:30:28.588893+08:00 2024-11-04 15:30:28.767 INFO 21062 --- [ocp-async-6,4b90908b5d8fa255,1c396d4472bf2442] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:30:28.986 INFO 21062 --- [ocp-schedules-16,8057405f4738487d,65e3cf101a2a3239] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:30:28.986 INFO 21062 --- [ocp-schedules-16,8057405f4738487d,65e3cf101a2a3239] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:30:29.001 INFO 21062 --- [ocp-schedules-20,82eead8c5719fdbe,d390138056dd2313] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:30:29.402 INFO 21062 --- [ocp-schedules-7,28ff78bd192b702a,dcfc7e1c34e4aeff] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:30:29.402 INFO 21062 --- [ocp-schedules-7,28ff78bd192b702a,dcfc7e1c34e4aeff] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:30:29.544 INFO 21062 --- [ocp-schedules-22,380c3091ff2ef247,80f0f26b5baf4a35] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:30:29.566 INFO 21062 --- [ocp-schedules-18,b1d1ca44d9346bc5,771cb49f111effd8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:30:29.571 INFO 21062 --- [ocp-schedules-18,b1d1ca44d9346bc5,771cb49f111effd8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:30:29.571 INFO 21062 --- [ocp-schedules-18,b1d1ca44d9346bc5,771cb49f111effd8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:30:29.595 WARN 21062 --- [ocp-async-2,699544cc98044a2b,35ff0988283f6ae7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:29.599 INFO 21062 --- [ocp-async-10,0640ed87025a9d2f,02eb028da6b79f68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:29.983 INFO 21062 --- [prometheus-minute-schedule-1,02849c82b326b3b9,f17b61959efb0712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:30.052 INFO 21062 --- [ocp-schedules-4,c5bb9995815f6937,565b8f92c74655e8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:30:30.052 INFO 21062 --- [ocp-schedules-4,c5bb9995815f6937,565b8f92c74655e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:30:30.053 INFO 21062 --- [ocp-schedules-4,c5bb9995815f6937,565b8f92c74655e8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:30:30.053 INFO 21062 --- [ocp-schedules-4,c5bb9995815f6937,565b8f92c74655e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:30:30.054 INFO 21062 --- [ocp-schedules-4,c5bb9995815f6937,565b8f92c74655e8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:30:30.054 INFO 21062 --- [ocp-schedules-4,c5bb9995815f6937,565b8f92c74655e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:30:30.177 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:30.178 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:30:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:30:31.595 WARN 21062 --- [ocp-async-10,829a34e21275e9e9,06bd07d64e27370e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:31.599 INFO 21062 --- [ocp-async-9,676ff79d5e5bb7b7,5af04fe54c18d9d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:31.757 INFO 21062 --- [ocp-schedules-18,ae5c42e98e6e76b9,18f4ca943cea751a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:30:32.983 INFO 21062 --- [prometheus-minute-schedule-1,8c804db3ffec512f,f5b409b138d0aece] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:33.153 INFO 21062 --- [ocp-schedules-29,8e35d9614bcc69f1,15bbe793faf8f743] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:30:33.158 INFO 21062 --- [ocp-schedules-29,8e35d9614bcc69f1,15bbe793faf8f743] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:30:33.158 INFO 21062 --- [ocp-schedules-29,8e35d9614bcc69f1,15bbe793faf8f743] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:30:33.178 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:33.179 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:33.398 INFO 21062 --- [ocp-schedules-26,fb1eef962577c64c,791649ade4b3963a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:30:33.399 INFO 21062 --- [ocp-schedules-26,fb1eef962577c64c,791649ade4b3963a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:30:33.425 INFO 21062 --- [ocp-schedules-26,fb1eef962577c64c,791649ade4b3963a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:30:33.425 INFO 21062 --- [ocp-schedules-26,fb1eef962577c64c,791649ade4b3963a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:30:33.425 INFO 21062 --- [ocp-schedules-27,04255a212ef22d47,2b95c7922937b3e4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:30:33.595 WARN 21062 --- [ocp-async-9,9a502d1543eec9db,bf67b6ca011e80f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:33.600 INFO 21062 --- [ocp-async-8,188237dab8e17471,0702de873e81d1d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:33.638 INFO 21062 --- [ocp-schedules-8,15455cf8db916703,0af5ccab1cb2a098] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:30:33.638+08:00, clusters=[1] 2024-11-04 15:30:33.640 INFO 21062 --- [ocp-schedules-8,15455cf8db916703,0af5ccab1cb2a098] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:30:33.959 INFO 21062 --- [ocp-schedules-21,d8e5cf926bac858e,f917ac9f657026a2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:30:33.959+08:00, clusters=[1] 2024-11-04 15:30:33.963 INFO 21062 --- [ocp-schedules-21,d8e5cf926bac858e,f917ac9f657026a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:30:33.997 INFO 21062 --- [ocp-schedules-4,726f8afce29724e5,32a7860d67ee11e4] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:30:34.000 INFO 21062 --- [ocp-schedules-4,726f8afce29724e5,32a7860d67ee11e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:34.056 ERROR 21062 --- [ocp-schedules-4,726f8afce29724e5,32a7860d67ee11e4] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:30:34.056 INFO 21062 --- [ocp-schedules-4,726f8afce29724e5,32a7860d67ee11e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:30:35.595 WARN 21062 --- [ocp-async-1,3a2b3670b9cca889,f8ad865a42f4b122] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:35.600 INFO 21062 --- [ocp-async-6,59a52e3af658e625,c47f401c4f1988b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:35.839 INFO 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:30:35.839 INFO 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:30:35.839 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:30:35.839 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:30:35.848 INFO 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:30:35.848 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:30:35.849 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:30:35.849 WARN 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:30:35.850 INFO 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:30:35.850 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:30:35.850 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:30:35.851 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:30:35.851 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:30:35.851 WARN 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:30:35.852 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:30:35.852 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:30:35.859 INFO 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:30:35.859 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:30:35.859 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:30:35.884 ERROR 21062 --- [ocp-schedules-19,422c11272ec32508,fc2f82ee61522d90] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:30:35.983 INFO 21062 --- [prometheus-minute-schedule-1,a567c1679231f044,a8b95445767985bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:36.179 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:36.179 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:37.583 INFO 21062 --- [ocp-schedules-7,56a4cbf6fe9bab7c,c15938cbe4096260] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:30:37.595 WARN 21062 --- [ocp-async-5,5e7e568bab562f75,5ea6ffce82a165a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:37.600 INFO 21062 --- [ocp-async-7,e87573948e864f2b,7d11828f9eb6faf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:38.504 INFO 21062 --- [ocp-schedules-28,ab029f2710308ef7,3dce3152c22fa87b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:30:38.504 INFO 21062 --- [ocp-schedules-9,dd2ab7231415b583,ff4261735e9aed6e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:30:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:30:38.589297+08:00, lastProcessTime=2024-11-04T15:30:28.588893+08:00 2024-11-04 15:30:38.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:30:38.589297+08:00 2024-11-04 15:30:38.767 INFO 21062 --- [ocp-async-8,07c3655f1e60b4ef,3db55f002f473052] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:30:38.983 INFO 21062 --- [prometheus-minute-schedule-1,30a2adad692b9a87,8eada117c55e0518] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:39.179 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:39.180 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:39.595 WARN 21062 --- [ocp-async-3,9b7a9c3d143e4263,0d9b5feac0801449] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:39.600 INFO 21062 --- [ocp-async-1,98767ce229de6c95,ec4e627b1d91b33c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:40.003 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:30:40.003 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:30:40.681 INFO 21062 --- [ocp-schedules-12,41a66a178ce20fe8,1ae6245894ae45d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:30:40.681 INFO 21062 --- [ocp-schedules-12,7a8ff677df278a46,eb27b5a0ec4a50eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:30:40.685 INFO 21062 --- [ocp-schedules-12,7a8ff677df278a46,eb27b5a0ec4a50eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:40.714 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:30:40.714 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:30:41.596 WARN 21062 --- [ocp-async-2,a47a0e421790b465,d211143b70577b8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:41.600 INFO 21062 --- [ocp-async-5,a6e07c88284a4f51,94852455bb53913e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:41.761 INFO 21062 --- [ocp-schedules-5,8330e094def26bc7,e1a2af4d6c53d4b1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:30:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:30:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:30:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:30:41.983 INFO 21062 --- [prometheus-minute-schedule-1,81c3b8f808d539e5,acaeda7b28949a44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:42.180 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:42.181 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:43.596 WARN 21062 --- [ocp-async-10,05ff3c7649fd6519,0c7640317332fc9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:43.600 INFO 21062 --- [ocp-async-4,ddc8f4783e161215,4549da9472bbd35f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:44.981 INFO 21062 --- [ocp-schedules-25,aaf6b2277abf6ced,ebae43ad0b63cc9b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:30:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b779ff88f89910d9,de28b8ea4eab4c7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:44.984 INFO 21062 --- [ocp-schedules-25,aaf6b2277abf6ced,ebae43ad0b63cc9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:45.068 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:30:45.070 ERROR 21062 --- [ocp-schedules-25,aaf6b2277abf6ced,ebae43ad0b63cc9b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:30:45.071 INFO 21062 --- [ocp-schedules-25,aaf6b2277abf6ced,ebae43ad0b63cc9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:30:45.181 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:45.182 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:45.596 WARN 21062 --- [ocp-async-9,5f67d3826e3c49d9,09651a7a5fc72e0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:45.601 INFO 21062 --- [ocp-async-8,9be05928e341c32f,db64ff2ee05508b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:47.596 WARN 21062 --- [ocp-async-1,7476aa57e6b135a0,707f0bb526f97bc7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:47.601 INFO 21062 --- [ocp-async-6,f62c11c6a5a2fb29,6daa5c0f2a9cd08f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:47.612 INFO 21062 --- [ocp-schedules-15,1c2938b545c60069,196e17988e02e85c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:30:47.983 INFO 21062 --- [prometheus-minute-schedule-1,81261a6fd56d5867,57fa012be35da99f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:48.182 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:48.184 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:30:48.589529+08:00, lastProcessTime=2024-11-04T15:30:38.589297+08:00 2024-11-04 15:30:48.767 INFO 21062 --- [ocp-async-7,1ded8beac472eca5,91b4bca336cd47bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:30:48.806 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:30:48.589529+08:00 2024-11-04 15:30:49.338 INFO 21062 --- [ocp-schedules-6,4760bc4e111d778c,35cb4585c4d4cf1c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:30:49.342 INFO 21062 --- [ocp-schedules-6,4760bc4e111d778c,35cb4585c4d4cf1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:49.343 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:30:49.596 WARN 21062 --- [ocp-async-10,68ac9c132e5e0dad,69012a866ce72629] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:49.601 INFO 21062 --- [ocp-async-4,49d50d27d7ffbe64,5c20860e8e6353ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:50.983 INFO 21062 --- [prometheus-minute-schedule-1,a1c2ca964a125dd6,1e9ab7e7e24d46c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:51.184 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:51.184 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:51.596 WARN 21062 --- [ocp-async-9,844de749d7fa8932,ead7a391919301c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:51.601 INFO 21062 --- [ocp-async-8,eb5c03ce8692c2a3,3ebb72f5ea46ede7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:51.763 INFO 21062 --- [ocp-schedules-5,3c381f74b4ebc33b,13a19e1ce7733142] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:30:53.160 INFO 21062 --- [ocp-schedules-24,d5ac24cc28c4e423,088e77a16d8fe02a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:30:53.165 INFO 21062 --- [ocp-schedules-24,d5ac24cc28c4e423,088e77a16d8fe02a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:30:53.165 INFO 21062 --- [ocp-schedules-24,d5ac24cc28c4e423,088e77a16d8fe02a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:30:53.597 WARN 21062 --- [ocp-async-1,cd85b9cb05b4cd3c,d939758d45bb53b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:53.601 INFO 21062 --- [ocp-async-6,b0f1e8efe1e36d2d,e91d96e1fb7f8e75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:53.983 INFO 21062 --- [prometheus-minute-schedule-1,2dc05320481fda60,30a6d8a7a33fe6bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:54.185 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:54.186 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:55.597 WARN 21062 --- [ocp-async-5,d972cef45f9be3df,1cc69adf103d5e4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:55.602 INFO 21062 --- [ocp-async-7,382b2e311739de55,e24c7ccf2d63c157] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c716e0e0b0255a07,634a2558a98656f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:30:57.186 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:30:57.187 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:30:57.597 WARN 21062 --- [ocp-async-4,0783beef14259490,4b05f18ce42c8917] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:57.602 INFO 21062 --- [ocp-async-3,eb6532a5ada77606,6ef650cb89aec03f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:57.653 INFO 21062 --- [ocp-schedules-4,2ba4fa506b50450a,afd60d97caf0e672] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:30:58.500 INFO 21062 --- [ocp-schedules-22,a36582c866cc3bbd,6ae4e8cc1f3cefe4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:30:58.500 INFO 21062 --- [ocp-schedules-29,ee537cf69fb0cb79,b742dbdb7526045c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:30:58.503 INFO 21062 --- [ocp-schedules-24,c21b9ba26607afc0,61030875c921a2b1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:30:58.504 INFO 21062 --- [ocp-schedules-22,a36582c866cc3bbd,6ae4e8cc1f3cefe4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:58.504 INFO 21062 --- [ocp-schedules-29,ee537cf69fb0cb79,b742dbdb7526045c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:30:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:30:58.505 INFO 21062 --- [ocp-schedules-9,7ca66ae82447ee16,ef70de74d7f1b37a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:30:58.509 INFO 21062 --- [ocp-schedules-16,e040193b60ec3be4,bddaaeabbb436a7b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:30:58.509 INFO 21062 --- [ocp-schedules-16,e040193b60ec3be4,bddaaeabbb436a7b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:30:58.509 INFO 21062 --- [ocp-schedules-18,49487ccd20af3787,76b39d65bff44c7a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:30:58.509 INFO 21062 --- [ocp-schedules-18,49487ccd20af3787,76b39d65bff44c7a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:30:58.509 INFO 21062 --- [ocp-schedules-2,7f5927dae96b3936,4058a238cd20a4d5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:30:58.509 INFO 21062 --- [ocp-schedules-2,7f5927dae96b3936,4058a238cd20a4d5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:30:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:30:58.589513+08:00, lastProcessTime=2024-11-04T15:30:48.589529+08:00 2024-11-04 15:30:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:30:58.589513+08:00 2024-11-04 15:30:58.768 INFO 21062 --- [ocp-async-2,a44f3d5be5bafef2,43f4399b0a0178ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:30:59.343 INFO 21062 --- [ocp-schedules-6,4760bc4e111d778c,35cb4585c4d4cf1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:30:59.546 INFO 21062 --- [ocp-schedules-13,f9b34c3176b79da7,83d8a6f3bea29693] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:30:59.597 WARN 21062 --- [ocp-async-1,aa082147bef05829,2640c6f5e9d4d922] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:30:59.602 INFO 21062 --- [ocp-async-6,55b0d354e7983342,8e688c11b88db564] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:30:59.983 INFO 21062 --- [prometheus-minute-schedule-1,0c29cd9bb34126d9,4403b94af38daa98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705460, interval=60 2024-11-04 15:31:00.187 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:00.188 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:01.597 WARN 21062 --- [ocp-async-5,b2a739605e97bf1c,88472340f8fc7506] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:01.602 INFO 21062 --- [ocp-async-7,3be523f3d4fc8276,c8ee31c525bdf631] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:01.766 INFO 21062 --- [ocp-schedules-9,b3290ccbf97ed5e9,04a52b6b13fc8df6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:31:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a4e0e8e6f2da48a4,8b5cf28cefcc6be7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:03.188 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:03.189 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:03.597 WARN 21062 --- [ocp-async-4,99410e4c93a4c695,f03d6052c0345d64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:03.602 INFO 21062 --- [ocp-async-3,3404c824dd00c018,4db2b7b84db671ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:05.598 WARN 21062 --- [ocp-async-8,0b389ffd980bad35,47153d3fdb2aa7b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:05.602 INFO 21062 --- [ocp-async-2,9acc39e6e990c5b7,563c75ac5a6c3d1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:05.983 INFO 21062 --- [prometheus-minute-schedule-1,7f3313edfa32de18,981dd273fe231f1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:06.190 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:06.190 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:07.598 WARN 21062 --- [ocp-async-6,7a25b7d44686b426,41e0cb88df0d94ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:07.603 INFO 21062 --- [ocp-async-10,5635e7fa42e5cf24,eeca61afa325b565] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:07.679 INFO 21062 --- [ocp-schedules-7,7ef06d4429c5a662,e0c8130f192d4d86] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:31:08.505 INFO 21062 --- [ocp-schedules-29,ee537cf69fb0cb79,b742dbdb7526045c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:31:08.505 INFO 21062 --- [ocp-schedules-22,a36582c866cc3bbd,6ae4e8cc1f3cefe4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:31:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:31:08.588467+08:00, lastProcessTime=2024-11-04T15:30:58.589513+08:00 2024-11-04 15:31:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:31:08.588467+08:00 2024-11-04 15:31:08.768 INFO 21062 --- [ocp-async-4,b5f41bbad0a405fa,e8ca83069151b6d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:31:08.983 INFO 21062 --- [prometheus-minute-schedule-1,439e1b5288d0e6b1,4331b05cf69d3a86] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:09.190 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:09.191 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:09.598 WARN 21062 --- [ocp-async-4,a43ad26a396a7bf1,86db383584fed9ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:09.603 INFO 21062 --- [ocp-async-3,152d83b253caf91a,47355d11dd8d34a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:10.002 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:31:10.002 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:31:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:31:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:31:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:31:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:31:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:31:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:31:10.715 INFO 21062 --- [ocp-schedules-12,7a8ff677df278a46,eb27b5a0ec4a50eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:31:10.715 INFO 21062 --- [ocp-schedules-12,5a824f9d98b2bffa,eee6894a2f05cafb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:31:10.718 INFO 21062 --- [ocp-schedules-12,5a824f9d98b2bffa,eee6894a2f05cafb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:10.746 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:31:10.746 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:31:11.598 WARN 21062 --- [ocp-async-8,3f848c6d919577dd,4aaa0b387e2ac6f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:11.603 INFO 21062 --- [ocp-async-2,ee36412210c06929,3cd477c18e6b52df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:11.769 INFO 21062 --- [ocp-schedules-8,b99d660f539d09de,97e3060f2fcf52f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:31:11.852 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=1, affectRows=1 2024-11-04 15:31:11.852 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=1, validCount=1 2024-11-04 15:31:11.853 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=20, affectRows=20 2024-11-04 15:31:11.853 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=20, validCount=20 2024-11-04 15:31:11.853 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=47, affectRows=47 2024-11-04 15:31:11.853 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=38, affectRows=38 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=47, validCount=47 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=38, validCount=38 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=87, affectRows=87 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=53, affectRows=53 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=87, validCount=87 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=53, validCount=53 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=79, affectRows=79 2024-11-04 15:31:11.854 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705400, nodeCount=97, affectRows=97 2024-11-04 15:31:11.855 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=76, validCount=76 2024-11-04 15:31:11.855 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705400, nodeCount=97, validCount=97 2024-11-04 15:31:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8c83cc85793af5b4,119d76e303dcee5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:12.191 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:12.192 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:13.167 INFO 21062 --- [ocp-schedules-16,6754aaa4c4b5f31f,c150d988d8330753] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:31:13.173 INFO 21062 --- [ocp-schedules-16,6754aaa4c4b5f31f,c150d988d8330753] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:31:13.173 INFO 21062 --- [ocp-schedules-16,6754aaa4c4b5f31f,c150d988d8330753] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:31:13.598 WARN 21062 --- [ocp-async-6,27d5e32568a0d225,f71d47108cee2390] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:13.603 INFO 21062 --- [ocp-async-10,7b4b64eacd44cb30,dd9637370e5afa37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:13.984 INFO 21062 --- [prometheus-cache-monitor1,65b9d2a0ce30998c,a3cee734aed6fed9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:31:13.984 INFO 21062 --- [prometheus-cache-monitor1,65b9d2a0ce30998c,a3cee734aed6fed9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:31:14.983 INFO 21062 --- [prometheus-minute-schedule-1,fbe97a80460d5026,a9a600498fcebb02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:15.192 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:15.194 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:15.598 WARN 21062 --- [ocp-async-7,c4fd2604ebd31cbd,046de562fe269d7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:15.603 INFO 21062 --- [ocp-async-9,1a5c7b3f8b3d30b4,8d0143024ec7cc50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:15.677 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:27:15.654, end=2024-11-04T15:29:15.674} 2024-11-04 15:31:15.681 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:31:15.682 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:31:17.599 WARN 21062 --- [ocp-async-3,f1d1842bc227732b,6f751a8566bcc2df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:17.604 INFO 21062 --- [ocp-async-1,e85524d6be7cc0d2,ee786e97e0e53093] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:17.720 INFO 21062 --- [ocp-schedules-29,a4b73370bed9a9b2,f7344e773280c9e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:31:17.983 INFO 21062 --- [prometheus-minute-schedule-1,4dc4ff2a7d93144d,e5b15211ab758c24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:18.194 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:18.195 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:31:18.590363+08:00, lastProcessTime=2024-11-04T15:31:08.588467+08:00 2024-11-04 15:31:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:31:18.590363+08:00 2024-11-04 15:31:18.768 INFO 21062 --- [ocp-async-6,1eac0f0111b6c091,b8134c0e3edb519e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:31:19.599 WARN 21062 --- [ocp-async-6,39ebfcc914036ab0,c5f453b9ed41d79a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:19.604 INFO 21062 --- [ocp-async-10,229501f37c32d0f8,333dc87d7d0a7110] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:20.983 INFO 21062 --- [prometheus-minute-schedule-1,7ed60b91cb9f35b3,806b83e0a7c8cc8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:21.195 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:21.196 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:21.599 WARN 21062 --- [ocp-async-10,a0ca6d3d785a553c,f3c243042c219b53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:21.604 INFO 21062 --- [ocp-async-9,c0d3e46151314b3c,d42dc2d63e06e023] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:21.772 INFO 21062 --- [ocp-schedules-8,67134de69e5cb53f,4840c26ea7471db5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:31:23.599 WARN 21062 --- [ocp-async-9,e00bfaef4543705e,7ea1ae067eaa1818] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:23.604 INFO 21062 --- [ocp-async-8,3be9a24cfe40369d,3f5dc206b84d86db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:23.983 INFO 21062 --- [prometheus-minute-schedule-1,3992b18267450cee,ab2aedd8b223e48d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:24.196 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:24.197 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:25.599 WARN 21062 --- [ocp-async-8,6ed51bd46de368cf,df455774a17b712c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:25.605 INFO 21062 --- [ocp-async-6,83bd1f1cdc2bbe7a,7ce14fc575dea112] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:26.945 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:31:26.945 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:31:26.954 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:31:26.954 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:31:26.954 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:31:26.954 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:31:26.960 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:31:26.960 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:31:26.960 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:31:26.960 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:31:26.983 INFO 21062 --- [prometheus-minute-schedule-1,206fe2814238817d,196a0b34d9dce485] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f41aaa7ae8e2ba8f,fd650e1782012f5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:31:26.997 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:31:27.005 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:31:27.005 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:31:27.005 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:31:27.005 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:31:27.005 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:31:27.014 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:31:27.197 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:27.198 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:27.533 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:31:27.600 WARN 21062 --- [ocp-async-6,ccd515241e87cca9,7ec1bab236b27c9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:27.605 INFO 21062 --- [ocp-async-4,ea011c7abf3ea5cf,8f9b899ac145d50a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:27.746 INFO 21062 --- [ocp-schedules-20,74e34f52e71891d4,aa27aed6b0503ca2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:31:28.157 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.197635729S 2024-11-04 15:31:28.158 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:31:28.257 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:31:28.257 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:31:28.285 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:31:28.342 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:31:28.342 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:31:28.342 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:31:28.342 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:31:28.342 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:31:28.342 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:31:28.353 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:31:28.498 INFO 21062 --- [ocp-schedules-11,821f898afe28b0ca,bb3acad50b10d842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:31:28.499 INFO 21062 --- [ocp-schedules-26,702fbb146da705b6,44b7f4630e4f39b0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:31:28.500 INFO 21062 --- [ocp-schedules-15,b3cf7f81019abdc9,7bf76d66bbb85853] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:31:28.500 INFO 21062 --- [ocp-schedules-22,ea2c062a08899b0f,940473f32d4e4ef3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:31:28.502 INFO 21062 --- [ocp-schedules-11,821f898afe28b0ca,bb3acad50b10d842] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:31:28.502 INFO 21062 --- [ocp-schedules-14,bd17dfbaf48fc4eb,6fce68e47abc6db4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:31:28.503 INFO 21062 --- [ocp-schedules-26,702fbb146da705b6,44b7f4630e4f39b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:28.503 INFO 21062 --- [ocp-schedules-15,b3cf7f81019abdc9,7bf76d66bbb85853] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:31:28.505 INFO 21062 --- [ocp-schedules-27,6e4c8b209ee8118e,d3ad3ecdfa52002a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:31:28.506 INFO 21062 --- [ocp-schedules-22,ea2c062a08899b0f,940473f32d4e4ef3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:31:28.509 INFO 21062 --- [ocp-schedules-7,17b5411402114e61,f91b9c972b8074c8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:31:28.509 INFO 21062 --- [ocp-schedules-7,17b5411402114e61,f91b9c972b8074c8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:31:28.509 INFO 21062 --- [ocp-schedules-21,f5f0c8a9db1b1e06,631db1a7f2428838] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:31:28.509 INFO 21062 --- [ocp-schedules-21,f5f0c8a9db1b1e06,631db1a7f2428838] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:31:28.509 INFO 21062 --- [ocp-schedules-17,887d118f2d07043d,9ecb5e5cc7072c19] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:31:28.509 INFO 21062 --- [ocp-schedules-17,887d118f2d07043d,9ecb5e5cc7072c19] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:31:28.527 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] o.s.boot.SpringApplication : Started application in 1.573 seconds (JVM running for 10902.206) 2024-11-04 15:31:28.527 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.573578606S 2024-11-04 15:31:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:31:28.589943+08:00, lastProcessTime=2024-11-04T15:31:18.590363+08:00 2024-11-04 15:31:28.768 INFO 21062 --- [ocp-async-8,a7e9218ed440af9c,43ef72a7c93ce23a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:31:28.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:31:28.589943+08:00 2024-11-04 15:31:28.967 INFO 21062 --- [ocp-schedules-30,dc2fb2b38e47ac9b,4a4b6b1793d1c23a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:31:29.004 INFO 21062 --- [ocp-schedules-18,dcf9029160c4e2bc,d5307c201b94432e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:31:29.406 INFO 21062 --- [ocp-schedules-8,32b9ac4df49e70c3,41c656e0b1eef407] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:31:29.406 INFO 21062 --- [ocp-schedules-8,32b9ac4df49e70c3,41c656e0b1eef407] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:31:29.548 INFO 21062 --- [ocp-schedules-17,fefd23f8e7c6d5f5,3437494bd2d2b604] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:31:29.571 INFO 21062 --- [ocp-schedules-20,5323e7aafeadad3e,05fb5ff272685e80] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:31:29.576 INFO 21062 --- [ocp-schedules-20,5323e7aafeadad3e,05fb5ff272685e80] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:31:29.576 INFO 21062 --- [ocp-schedules-20,5323e7aafeadad3e,05fb5ff272685e80] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:31:29.600 WARN 21062 --- [ocp-async-1,e71ff02d3d576497,eb94cef58e5d0285] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:29.605 INFO 21062 --- [ocp-async-2,d1fdedb014f388e0,e761f9313157053f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:29.880 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:31:29.880 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:31:29.889 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:31:29.889 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:31:29.889 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:31:29.902 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:31:29.902 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:31:29.902 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:31:29.906 INFO 21062 --- [ocp-schedules-24,8d15f1ac36f272d4,88dcb85b90da3736] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:31:29.983 INFO 21062 --- [prometheus-minute-schedule-1,d1a33519cec4d78c,409779a89aa6bb1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:30.055 INFO 21062 --- [ocp-schedules-30,f9a719e6c22121c9,b62d9228a1e40597] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:31:30.055 INFO 21062 --- [ocp-schedules-30,f9a719e6c22121c9,b62d9228a1e40597] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:31:30.061 INFO 21062 --- [ocp-schedules-30,f9a719e6c22121c9,b62d9228a1e40597] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:31:30.061 INFO 21062 --- [ocp-schedules-30,f9a719e6c22121c9,b62d9228a1e40597] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:31:30.065 INFO 21062 --- [ocp-schedules-30,f9a719e6c22121c9,b62d9228a1e40597] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:31:30.065 INFO 21062 --- [ocp-schedules-30,f9a719e6c22121c9,b62d9228a1e40597] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:31:30.198 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:30.199 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:31:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:31:31.600 WARN 21062 --- [ocp-async-2,ca65026e3c4db3c6,f1b7cfd827c2cc28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:31.605 INFO 21062 --- [ocp-async-10,424ae12703dc2b49,edd8fc2456a8ca16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:31.775 INFO 21062 --- [ocp-schedules-6,1f0c4cc9e4d4fe62,0847b08753dc7ec5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:31:32.983 INFO 21062 --- [prometheus-minute-schedule-1,207656c78e0a0755,3ae3330204b9c694] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:33.176 INFO 21062 --- [ocp-schedules-23,af78ae84c5bbbe2e,a1791f3fb8642a81] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:31:33.181 INFO 21062 --- [ocp-schedules-23,af78ae84c5bbbe2e,a1791f3fb8642a81] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:31:33.181 INFO 21062 --- [ocp-schedules-23,af78ae84c5bbbe2e,a1791f3fb8642a81] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:31:33.199 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:33.199 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:33.425 INFO 21062 --- [ocp-schedules-30,c9fa83190357a1cd,1e2b1823b7501635] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:31:33.425 INFO 21062 --- [ocp-schedules-30,c9fa83190357a1cd,1e2b1823b7501635] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:31:33.451 INFO 21062 --- [ocp-schedules-30,c9fa83190357a1cd,1e2b1823b7501635] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:31:33.451 INFO 21062 --- [ocp-schedules-30,c9fa83190357a1cd,1e2b1823b7501635] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:31:33.452 INFO 21062 --- [ocp-schedules-9,e1d414a869e8c546,92d8c3e128210af6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:31:33.600 WARN 21062 --- [ocp-async-4,1967f73800bea9fb,b83985c06f12265d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:33.605 INFO 21062 --- [ocp-async-9,271416480b58f276,1eb34759c0026a9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:33.665 INFO 21062 --- [ocp-schedules-19,bd5d0efae2f066de,ef131dd5c2543a39] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:31:33.665+08:00, clusters=[1] 2024-11-04 15:31:33.667 INFO 21062 --- [ocp-schedules-19,bd5d0efae2f066de,ef131dd5c2543a39] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:31:33.989 INFO 21062 --- [ocp-schedules-6,dcb5c95f3a2801a2,1df1bac35d009104] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:31:33.989+08:00, clusters=[1] 2024-11-04 15:31:33.992 INFO 21062 --- [ocp-schedules-6,dcb5c95f3a2801a2,1df1bac35d009104] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:31:35.600 WARN 21062 --- [ocp-async-3,4f82f40a2eafeb04,f6512f79d9c84a63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:35.605 INFO 21062 --- [ocp-async-8,db913069b3c7dea4,03261d483ca02d0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:35.893 INFO 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:31:35.894 INFO 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:31:35.894 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:31:35.894 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:31:35.903 INFO 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:31:35.903 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:31:35.904 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:31:35.905 WARN 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:31:35.905 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:31:35.905 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:31:35.905 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:31:35.906 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:31:35.906 INFO 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:31:35.906 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:31:35.906 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:31:35.907 WARN 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:31:35.913 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:31:35.913 INFO 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:31:35.913 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:31:35.923 ERROR 21062 --- [ocp-schedules-30,11ff1f1adf3395e6,823ae2bb6fa249e3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:31:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0a4ad50b2cbb6d9c,3579ee4c8a2dca10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:36.199 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:36.200 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:37.601 WARN 21062 --- [ocp-async-5,55144f9ce678c4b4,9ce8be011ffb32c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:37.606 INFO 21062 --- [ocp-async-6,3d141205b4d67a54,f30a3966b065f037] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:37.791 INFO 21062 --- [ocp-schedules-18,d0e75627072c19bf,dee1a4b2daeb3a3b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:31:38.504 INFO 21062 --- [ocp-schedules-15,b3cf7f81019abdc9,7bf76d66bbb85853] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:31:38.504 INFO 21062 --- [ocp-schedules-26,702fbb146da705b6,44b7f4630e4f39b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:31:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:31:38.589481+08:00, lastProcessTime=2024-11-04T15:31:28.589943+08:00 2024-11-04 15:31:38.746 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:31:38.589481+08:00 2024-11-04 15:31:38.768 INFO 21062 --- [ocp-async-9,66b50cb24e6dc0a4,8f879a7c8231b14e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:31:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5da589c10a345b0e,3125cb39ae430e36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:39.200 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:39.201 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:39.601 WARN 21062 --- [ocp-async-7,f25936a472237c92,b92ac6dd892dc79a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:39.606 INFO 21062 --- [ocp-async-3,0018bf3c45502e17,0641a77c7a452a20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:40.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:31:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:31:40.746 INFO 21062 --- [ocp-schedules-12,5a824f9d98b2bffa,eee6894a2f05cafb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:31:40.746 INFO 21062 --- [ocp-schedules-12,2adbdc635adad973,38601c747a914886] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:31:40.750 INFO 21062 --- [ocp-schedules-12,2adbdc635adad973,38601c747a914886] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:40.779 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:31:40.779 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:31:41.601 WARN 21062 --- [ocp-async-1,ef01bb1efcb336c6,01441007308128c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:41.606 INFO 21062 --- [ocp-async-5,2aea1b86084821ac,026a40abbabffa46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:41.778 INFO 21062 --- [ocp-schedules-16,950868458a437abc,d75bdbf08c976d07] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:31:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6e40dda7116d9f18,a83d1157b14841cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:42.201 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:42.202 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:43.601 WARN 21062 --- [ocp-async-2,4a694d48e304bd1a,032a49281c0b3e64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:43.606 INFO 21062 --- [ocp-async-10,8b13ac36e0a0040c,e8dca4ed32f72683] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:44.983 INFO 21062 --- [prometheus-minute-schedule-1,ae43c7e9c2905c31,f40cd50b4677fe41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:45.071 INFO 21062 --- [ocp-schedules-1,f14243ae189153aa,1d2901f3dd2c87b2] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:31:45.075 INFO 21062 --- [ocp-schedules-1,f14243ae189153aa,1d2901f3dd2c87b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:45.154 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:31:45.156 ERROR 21062 --- [ocp-schedules-1,f14243ae189153aa,1d2901f3dd2c87b2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:31:45.156 INFO 21062 --- [ocp-schedules-1,f14243ae189153aa,1d2901f3dd2c87b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:31:45.202 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:45.202 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:45.601 WARN 21062 --- [ocp-async-4,5850ff3c2e492339,58845741636d5375] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:45.606 INFO 21062 --- [ocp-async-9,78cf8d2a76ff66f0,e79f3bb5a101fa8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:47.602 WARN 21062 --- [ocp-async-3,822b0ed89f6f5331,bdd747d03a200b0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:47.607 INFO 21062 --- [ocp-async-8,418f1abf712287a1,73a63c2dce14b6a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:47.813 INFO 21062 --- [ocp-schedules-24,d531fac30c6fafeb,90f4d46f6d3b2327] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:31:47.983 INFO 21062 --- [prometheus-minute-schedule-1,498ef097c4b526e3,93d07ee0d5e4ce8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:48.203 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:48.203 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:31:48.591620+08:00, lastProcessTime=2024-11-04T15:31:38.589481+08:00 2024-11-04 15:31:48.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:31:48.591620+08:00 2024-11-04 15:31:48.769 INFO 21062 --- [ocp-async-6,6391955c9d12f126,d342c43955b19058] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:31:49.602 WARN 21062 --- [ocp-async-2,f0f111b2d6c57f42,5c3530ad76fa5b7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:49.607 INFO 21062 --- [ocp-async-10,3f7070d76edae326,66da64357434e914] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ccb5b3c40180ba94,f131826896f0ae87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:51.203 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:51.204 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:51.602 WARN 21062 --- [ocp-async-4,42954820f759d25e,a51605f352012f51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:51.607 INFO 21062 --- [ocp-async-9,5cb259d2f33cb2e0,208627b966140f0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:51.781 INFO 21062 --- [ocp-schedules-19,af745aea59c12213,34f57ee36b46b77f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:31:53.184 INFO 21062 --- [ocp-schedules-7,affa81dfb3b77858,4cf476249b87204c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:31:53.188 INFO 21062 --- [ocp-schedules-7,affa81dfb3b77858,4cf476249b87204c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:31:53.188 INFO 21062 --- [ocp-schedules-7,affa81dfb3b77858,4cf476249b87204c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:31:53.602 WARN 21062 --- [ocp-async-3,a9ec8bc0d0f42872,f1135dfd2f8d4724] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:53.607 INFO 21062 --- [ocp-async-8,25b9e2ef38d3dac0,fe02f81bc6dae7ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a254939ba57c90ce,1735d364f04cb8f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:54.204 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:54.205 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:55.602 WARN 21062 --- [ocp-async-5,50d129e93e345c78,20e51bed20856cf3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:55.607 INFO 21062 --- [ocp-async-6,ec4a6e3b6ecd3767,7218e3d68882b7e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d95ef8c6817711f9,9debc15f99d5f7cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:31:57.205 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:31:57.206 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:31:57.602 WARN 21062 --- [ocp-async-10,68aafc3f6ee887f8,c578a33dbcba82f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:57.608 INFO 21062 --- [ocp-async-7,d9ca08c4a885a533,b60a8f89a0887b1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:57.841 INFO 21062 --- [ocp-schedules-6,935407d9ef78f7f4,62fb12690100b915] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:31:58.500 INFO 21062 --- [ocp-schedules-28,732de1f2697a9d1c,58b3547394243d21] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:31:58.500 INFO 21062 --- [ocp-schedules-18,a9dac53743265935,2905c39d5c2b598b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:31:58.502 INFO 21062 --- [ocp-schedules-13,3c3be2ecc88cc490,193a84e254d241c8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:31:58.504 INFO 21062 --- [ocp-schedules-28,732de1f2697a9d1c,58b3547394243d21] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:58.504 INFO 21062 --- [ocp-schedules-18,a9dac53743265935,2905c39d5c2b598b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:31:58.504 INFO 21062 --- [ocp-schedules-17,f5d894bffd27da6f,555882b7a2efbe71] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:31:58.510 INFO 21062 --- [ocp-schedules-21,d2755b2dbc3ed771,3fa9c5e53fc94e07] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:31:58.510 INFO 21062 --- [ocp-schedules-30,afe8afa3684ad55d,e72362be5258c806] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:31:58.510 INFO 21062 --- [ocp-schedules-21,d2755b2dbc3ed771,3fa9c5e53fc94e07] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:31:58.510 INFO 21062 --- [ocp-schedules-30,afe8afa3684ad55d,e72362be5258c806] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:31:58.510 INFO 21062 --- [ocp-schedules-10,4c66a3e2f4b20c02,13b2b73f5ceaa223] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:31:58.510 INFO 21062 --- [ocp-schedules-10,4c66a3e2f4b20c02,13b2b73f5ceaa223] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:31:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:31:58.589310+08:00, lastProcessTime=2024-11-04T15:31:48.591620+08:00 2024-11-04 15:31:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:31:58.589310+08:00 2024-11-04 15:31:58.769 INFO 21062 --- [ocp-async-1,d96015c0e8c5de9f,1d1a67cf001173a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:31:59.343 INFO 21062 --- [ocp-schedules-20,aaf7db9e3103401e,8e86397f6a186450] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:31:59.347 INFO 21062 --- [ocp-schedules-20,aaf7db9e3103401e,8e86397f6a186450] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:31:59.348 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:31:59.552 INFO 21062 --- [ocp-schedules-21,a9be3807f473eef2,d5d83a793a30809c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:31:59.602 WARN 21062 --- [ocp-async-3,0162bc6bbab4249d,a5c157de87555d00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:31:59.608 INFO 21062 --- [ocp-async-8,5509835ec81cf4a4,9eaf52a612679939] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:31:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d0e27b350dbb6cf6,0bd75b0014600db8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705520, interval=60 2024-11-04 15:32:00.206 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:00.207 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:01.603 WARN 21062 --- [ocp-async-5,70898ab2b408f7c1,66e2ecfd69a083c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:01.608 INFO 21062 --- [ocp-async-6,100dadc6f37fd038,1a86a0d7903e5b7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:01.784 INFO 21062 --- [ocp-schedules-10,2f859a0b9f3dbf4b,98a338ec5987e66a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:32:02.983 INFO 21062 --- [prometheus-minute-schedule-1,916d5004516fe979,63d94ba634efc38e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:03.207 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:03.208 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:03.603 WARN 21062 --- [ocp-async-10,bea2f9718ad091d4,b3754ac71af9f67b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:03.608 INFO 21062 --- [ocp-async-7,c8ef2479d95c6104,fa4ef00e2e90924d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:05.603 WARN 21062 --- [ocp-async-9,6af78e29806fa221,91869ebdcd9989f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:05.608 INFO 21062 --- [ocp-async-1,f31efd72d60f5a7f,e2e0f2c9f9687312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:05.983 INFO 21062 --- [prometheus-minute-schedule-1,ad9fb17aea721255,fba9dfad9fec7dff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:06.209 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:06.210 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:07.603 WARN 21062 --- [ocp-async-8,b64cabb816bd46e8,741fa88fedd9238b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:07.608 INFO 21062 --- [ocp-async-2,996cea8e0d480f67,943647b685163236] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:07.873 INFO 21062 --- [ocp-schedules-21,95eb5453b06a7b47,94ce0252f7aac611] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:32:08.505 INFO 21062 --- [ocp-schedules-28,732de1f2697a9d1c,58b3547394243d21] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:32:08.505 INFO 21062 --- [ocp-schedules-18,a9dac53743265935,2905c39d5c2b598b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:32:08.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:32:08.591520+08:00, lastProcessTime=2024-11-04T15:31:58.589310+08:00 2024-11-04 15:32:08.769 INFO 21062 --- [ocp-async-10,dec0928f3f4e9549,a83ac071775cee59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:32:08.802 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:32:08.591520+08:00 2024-11-04 15:32:08.983 INFO 21062 --- [prometheus-minute-schedule-1,224afd85f961751f,1405c06c0e23658e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:09.210 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:09.211 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:09.348 INFO 21062 --- [ocp-schedules-20,aaf7db9e3103401e,8e86397f6a186450] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:32:09.603 WARN 21062 --- [ocp-async-10,d2cb46debc1a333e,56e6d7853f957302] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:09.609 INFO 21062 --- [ocp-async-7,b2d4d3cab8747c61,0e1c98c9805e5282] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:32:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:32:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:32:10.303 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:32:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:32:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:32:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:32:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:32:10.779 INFO 21062 --- [ocp-schedules-12,2adbdc635adad973,38601c747a914886] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:32:10.779 INFO 21062 --- [ocp-schedules-12,6a55f9301b4d340d,1fb81260c31f680c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:32:10.783 INFO 21062 --- [ocp-schedules-12,6a55f9301b4d340d,1fb81260c31f680c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:10.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:32:10.811 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:32:11.604 WARN 21062 --- [ocp-async-9,a876315c28680001,992b7a646bcf6fa3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:11.609 INFO 21062 --- [ocp-async-1,c1028780e571fe23,8b616da01ab98cfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:11.786 INFO 21062 --- [ocp-schedules-27,b4f1e1126936d611,cc9d1b85a6f160d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:32:11.855 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=1, affectRows=1 2024-11-04 15:32:11.855 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=1, validCount=1 2024-11-04 15:32:11.856 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=2, affectRows=2 2024-11-04 15:32:11.856 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=2, validCount=2 2024-11-04 15:32:11.856 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=7, affectRows=7 2024-11-04 15:32:11.856 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=7, validCount=7 2024-11-04 15:32:11.856 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=17, affectRows=17 2024-11-04 15:32:11.856 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=17, validCount=17 2024-11-04 15:32:11.856 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=16, affectRows=16 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=16, validCount=16 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=43, affectRows=43 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=41, affectRows=41 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=47, affectRows=47 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=41, validCount=41 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=43, validCount=43 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=46, validCount=46 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=29, affectRows=29 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=73, affectRows=73 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=29, validCount=29 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=71, validCount=71 2024-11-04 15:32:11.857 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=72, affectRows=72 2024-11-04 15:32:11.858 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705460, nodeCount=74, affectRows=74 2024-11-04 15:32:11.858 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=72, validCount=72 2024-11-04 15:32:11.858 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705460, nodeCount=74, validCount=74 2024-11-04 15:32:11.983 INFO 21062 --- [prometheus-minute-schedule-1,8cae015272c3bdcc,89f45a396321cc6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:12.211 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:12.212 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:13.191 INFO 21062 --- [ocp-schedules-11,16c4d1ecf158262f,680a19f207b94ca8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:32:13.196 INFO 21062 --- [ocp-schedules-11,16c4d1ecf158262f,680a19f207b94ca8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:32:13.196 INFO 21062 --- [ocp-schedules-11,16c4d1ecf158262f,680a19f207b94ca8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:32:13.604 WARN 21062 --- [ocp-async-8,40f363e18a863c1c,eb1acf654cd126e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:13.609 INFO 21062 --- [ocp-async-2,3b2787eb7423bb9e,5a525d324a049662] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:13.984 INFO 21062 --- [prometheus-cache-monitor1,c29d7e956c7be268,651df9358107451e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:32:13.984 INFO 21062 --- [prometheus-cache-monitor1,c29d7e956c7be268,651df9358107451e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:32:14.983 INFO 21062 --- [prometheus-minute-schedule-1,617d0a9cd8f70b4b,9c7661e962053935] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:15.212 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:15.213 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:15.604 WARN 21062 --- [ocp-async-6,3c6d0dd88d5eaa5e,5b8497e47c928668] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:15.609 INFO 21062 --- [ocp-async-4,a6e26c9e978f8e3e,03e9bfb3ceb0c10c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:17.604 WARN 21062 --- [ocp-async-7,797a88b0bf71536c,0fb4feadd5f0af4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:17.609 INFO 21062 --- [ocp-async-3,92e0451817d9fd67,998cdc603e281cc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:17.931 INFO 21062 --- [ocp-schedules-27,9745a5d4983dcafa,798811a62c95128c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:32:17.983 INFO 21062 --- [prometheus-minute-schedule-1,e663ccbcbbe4f183,589d4ca1ce501208] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:18.213 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:18.214 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:32:18.590139+08:00, lastProcessTime=2024-11-04T15:32:08.591520+08:00 2024-11-04 15:32:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:32:18.590139+08:00 2024-11-04 15:32:18.769 INFO 21062 --- [ocp-async-8,3d9f923fad1d1cfe,43ed81c0f5ba6088] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:32:19.604 WARN 21062 --- [ocp-async-8,33b7d4d7576c3de4,74d51d69ce28beed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:19.609 INFO 21062 --- [ocp-async-2,bf8f6fec38f908f8,b5fefc443eda2ae2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:20.983 INFO 21062 --- [prometheus-minute-schedule-1,fbe19c0cff7f76db,60e827542a85c8ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:21.214 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:21.215 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:21.604 WARN 21062 --- [ocp-async-2,8450730f91aeac6e,225536543f9d3fdc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:21.610 INFO 21062 --- [ocp-async-4,c35dc0849a2562aa,88d10c90c534fb0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:21.789 INFO 21062 --- [ocp-schedules-4,91571242b036c198,efc141282907d966] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:32:23.605 WARN 21062 --- [ocp-async-4,0f6c960d87c57a62,76f7401cff81f22a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:23.610 INFO 21062 --- [ocp-async-9,a542614b3de13b54,7193e8c6eacb9594] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:23.983 INFO 21062 --- [prometheus-minute-schedule-1,e7eaedd0ab8d7b6c,fdf9f3c331eb9d2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:24.215 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:24.216 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:25.605 WARN 21062 --- [ocp-async-9,e7d4cb0bc8571c8f,d188a41b1205cc16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:25.610 INFO 21062 --- [ocp-async-8,56a28e2c1bbb448c,6d20e9e37f0093e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:26.983 INFO 21062 --- [prometheus-minute-schedule-1,7a3d2c56c639c567,aab46a0fefeae8c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:26.984 INFO 21062 --- [prometheus-hour-schedule-1,0a5b3f27fe945f3a,d83f76e352ce0fd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:32:27.216 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:27.218 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:27.605 WARN 21062 --- [ocp-async-8,63abad59560fbdc8,fc6eb9fd611e1797] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:27.610 INFO 21062 --- [ocp-async-10,e4003d73778764a8,f0df1f3c919a45e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:27.963 INFO 21062 --- [ocp-schedules-7,b02e72a87a813ad3,ec5025d04ac292fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:32:28.498 INFO 21062 --- [ocp-schedules-13,1b775fe37676ee9f,0f977c9f159b2654] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:32:28.500 INFO 21062 --- [ocp-schedules-28,e4b0454acbe937d2,3b21c9e7faa79173] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:32:28.500 INFO 21062 --- [ocp-schedules-1,5292433fa2aeac40,73500fb849592a7c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:32:28.501 INFO 21062 --- [ocp-schedules-8,0ddd588154513f18,72300d94792941bd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:32:28.502 INFO 21062 --- [ocp-schedules-13,1b775fe37676ee9f,0f977c9f159b2654] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:32:28.502 INFO 21062 --- [ocp-schedules-29,63810c5d2c07d684,d13fd80316d6af7c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:32:28.503 INFO 21062 --- [ocp-schedules-28,e4b0454acbe937d2,3b21c9e7faa79173] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:28.504 INFO 21062 --- [ocp-schedules-1,5292433fa2aeac40,73500fb849592a7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:32:28.504 INFO 21062 --- [ocp-schedules-16,40f46b53fe1f01be,f8f8844f48304101] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:32:28.505 INFO 21062 --- [ocp-schedules-23,f368f08ca03543b2,eac4bf1d0e0173d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:32:28.506 INFO 21062 --- [ocp-schedules-8,0ddd588154513f18,72300d94792941bd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:32:28.507 INFO 21062 --- [ocp-schedules-23,f368f08ca03543b2,eac4bf1d0e0173d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:28.510 INFO 21062 --- [ocp-schedules-10,8efa3d601ab9d7d5,fe65b6aaa64f9b88] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:32:28.511 INFO 21062 --- [ocp-schedules-5,975d42c5d28a3f10,f2d648461592c501] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:32:28.511 INFO 21062 --- [ocp-schedules-10,8efa3d601ab9d7d5,fe65b6aaa64f9b88] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:32:28.511 INFO 21062 --- [ocp-schedules-5,975d42c5d28a3f10,f2d648461592c501] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:32:28.511 INFO 21062 --- [ocp-schedules-18,698271f81a29f20d,2bdd37130b10f3c6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:32:28.511 INFO 21062 --- [ocp-schedules-18,698271f81a29f20d,2bdd37130b10f3c6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:32:28.542 INFO 21062 --- [ocp-schedules-21,ac61c3c1a2b5ff09,f3c426115cd78cbe] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:32:28.542 INFO 21062 --- [ocp-schedules-21,ac61c3c1a2b5ff09,f3c426115cd78cbe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:32:28.581 INFO 21062 --- [pool-12-thread-3,30022a60e4c36dd6,e12cfb7ad9cba12d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:32:28.581 INFO 21062 --- [pool-12-thread-4,30022a60e4c36dd6,e352a14b3e0c58f1] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:32:28.582 INFO 21062 --- [pool-12-thread-2,30022a60e4c36dd6,d6e0cfa2eef12bc0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:32:28.582 INFO 21062 --- [pool-12-thread-1,30022a60e4c36dd6,084907bbab2e5222] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:32:28.586 INFO 21062 --- [pool-12-thread-4,30022a60e4c36dd6,e352a14b3e0c58f1] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:32:28.586 INFO 21062 --- [pool-12-thread-4,30022a60e4c36dd6,e352a14b3e0c58f1] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:32:28.587 INFO 21062 --- [ForkJoinPool-11-worker-15,30022a60e4c36dd6,09770f9462242465] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:32:28.587 INFO 21062 --- [ForkJoinPool-11-worker-15,30022a60e4c36dd6,09770f9462242465] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:32:28.587 INFO 21062 --- [ForkJoinPool-11-worker-15,30022a60e4c36dd6,09770f9462242465] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:32:28.587 INFO 21062 --- [pool-12-thread-1,30022a60e4c36dd6,084907bbab2e5222] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:32:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:32:28.587781+08:00, lastProcessTime=2024-11-04T15:32:18.590139+08:00 2024-11-04 15:32:28.589 INFO 21062 --- [ocp-schedules-19,afe9fc4945a19e4d,eca794a6aa0514e5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:32:28.589 INFO 21062 --- [ocp-schedules-19,afe9fc4945a19e4d,eca794a6aa0514e5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:32:28.607 INFO 21062 --- [pool-12-thread-3,30022a60e4c36dd6,e12cfb7ad9cba12d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:32:28.608 INFO 21062 --- [pool-12-thread-3,30022a60e4c36dd6,e12cfb7ad9cba12d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:32:28.609 INFO 21062 --- [pool-12-thread-3,30022a60e4c36dd6,e12cfb7ad9cba12d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:32:28.612 INFO 21062 --- [ForkJoinPool-11-worker-15,30022a60e4c36dd6,28af728bfaeefa37] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:32:28.612 INFO 21062 --- [pool-12-thread-2,30022a60e4c36dd6,d6e0cfa2eef12bc0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:32:28.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:32:28.587781+08:00 2024-11-04 15:32:28.769 INFO 21062 --- [ocp-async-9,872d1cb814b1baf0,92bbbead91f75a43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:32:29.007 INFO 21062 --- [ocp-schedules-16,cb8c7a9eed7cdcbc,7b701af0765166ed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:32:29.020 INFO 21062 --- [ocp-schedules-5,02296fc72616dca7,239a207011272d02] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 15:32:29.021 INFO 21062 --- [ocp-schedules-5,02296fc72616dca7,239a207011272d02] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 15:32:29.021 INFO 21062 --- [ocp-schedules-5,02296fc72616dca7,239a207011272d02] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=422 2024-11-04 15:32:29.038 INFO 21062 --- [ocp-schedules-5,02296fc72616dca7,239a207011272d02] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=422 2024-11-04 15:32:29.350 INFO 21062 --- [ocp-schedules-26,678bc466a4d88c29,d34fdf65959a67fd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 15:32:29.354 INFO 21062 --- [ocp-schedules-26,678bc466a4d88c29,d34fdf65959a67fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:29.395 ERROR 21062 --- [ocp-schedules-26,678bc466a4d88c29,d34fdf65959a67fd] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.GeneratedMethodAccessor1314.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:32:29.395 INFO 21062 --- [ocp-schedules-26,678bc466a4d88c29,d34fdf65959a67fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 15:32:29.411 INFO 21062 --- [ocp-schedules-19,497d7f1fe76e86fd,466514c4cee791a6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:32:29.411 INFO 21062 --- [ocp-schedules-19,497d7f1fe76e86fd,466514c4cee791a6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:32:29.550 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:32:29.554 INFO 21062 --- [ocp-schedules-6,f0250a841eb66498,2758c4df24f1a499] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:32:29.566 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:32:29.567 WARN 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:32:29.568 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:32:29.570 WARN 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:32:29.576 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:32:29.576 INFO 21062 --- [ocp-schedules-22,197958eb76e07557,48a89f5a5b96f9bd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:32:29.577 WARN 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:32:29.578 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:32:29.579 WARN 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:32:29.580 INFO 21062 --- [ocp-schedules-22,197958eb76e07557,48a89f5a5b96f9bd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:32:29.580 INFO 21062 --- [ocp-schedules-22,197958eb76e07557,48a89f5a5b96f9bd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:32:29.585 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:32:29.587 WARN 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:32:29.587 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:32:29.588 WARN 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:32:29.594 INFO 21062 --- [ocp-schedules-7,91c6c03b0ebb0d50,c98cc07dc1d7b822] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:32:29.605 WARN 21062 --- [ocp-async-3,b3d134e222207b61,525e9b16ff22685e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:29.610 INFO 21062 --- [ocp-async-1,0caac9a58814b1b7,d088eecd58021197] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:29.983 INFO 21062 --- [prometheus-minute-schedule-1,45edc3ccbbe25fd0,e6ed88b9ba99c7d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:30.068 INFO 21062 --- [ocp-schedules-5,0bb5bbd6509e9377,a95c2314633c2b06] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:32:30.068 INFO 21062 --- [ocp-schedules-5,0bb5bbd6509e9377,a95c2314633c2b06] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:32:30.070 INFO 21062 --- [ocp-schedules-5,0bb5bbd6509e9377,a95c2314633c2b06] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:32:30.070 INFO 21062 --- [ocp-schedules-5,0bb5bbd6509e9377,a95c2314633c2b06] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:32:30.070 INFO 21062 --- [ocp-schedules-5,0bb5bbd6509e9377,a95c2314633c2b06] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:32:30.070 INFO 21062 --- [ocp-schedules-5,0bb5bbd6509e9377,a95c2314633c2b06] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:32:30.218 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:30.218 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:32:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:32:30.545 INFO 21062 --- [ocp-schedules-3,61c3270b374843b0,91f2f5ea2befbad7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:32:30.545 INFO 21062 --- [ocp-schedules-3,61c3270b374843b0,91f2f5ea2befbad7] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:32:30.574 INFO 21062 --- [pool-55-thread-1,61c3270b374843b0,d4cd7934e674566e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:32:30.602 WARN 21062 --- [ocp-schedules-3,61c3270b374843b0,91f2f5ea2befbad7] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:32:30.602 INFO 21062 --- [ocp-schedules-3,61c3270b374843b0,91f2f5ea2befbad7] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:32:31.605 WARN 21062 --- [ocp-async-5,6599d049d4584536,0fdb43a038a5994b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:31.611 INFO 21062 --- [ocp-async-2,2af394137a157a8f,7493f2c2be95ce9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:31.791 INFO 21062 --- [ocp-schedules-21,85feb36430615b08,428c80e91d679e20] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:32:32.983 INFO 21062 --- [prometheus-minute-schedule-1,df9395ec163fa18e,833bea804c38a7ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:33.198 INFO 21062 --- [ocp-schedules-14,e72a7a819897c58f,d48780aff9fa95b9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:32:33.203 INFO 21062 --- [ocp-schedules-14,e72a7a819897c58f,d48780aff9fa95b9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:32:33.203 INFO 21062 --- [ocp-schedules-14,e72a7a819897c58f,d48780aff9fa95b9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:32:33.219 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:33.219 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:33.451 INFO 21062 --- [ocp-schedules-25,c3bb7ac053b6e044,d694df9684233da1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:32:33.451 INFO 21062 --- [ocp-schedules-25,c3bb7ac053b6e044,d694df9684233da1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:32:33.478 INFO 21062 --- [ocp-schedules-25,c3bb7ac053b6e044,d694df9684233da1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:32:33.478 INFO 21062 --- [ocp-schedules-25,c3bb7ac053b6e044,d694df9684233da1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:32:33.480 INFO 21062 --- [ocp-schedules-5,6f63817f41a24274,c4cf0a4b977afbb5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:32:33.605 WARN 21062 --- [ocp-async-10,1f3cd19054b1e7fa,2f61a3401ea2ca21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:33.611 INFO 21062 --- [ocp-async-4,569160a3dd57d938,b0d6e5986406294a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:33.692 INFO 21062 --- [ocp-schedules-22,709f0a091b00d972,c526aab268061df2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:32:33.692+08:00, clusters=[1] 2024-11-04 15:32:33.693 INFO 21062 --- [ocp-schedules-22,709f0a091b00d972,c526aab268061df2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:32:34.019 INFO 21062 --- [ocp-schedules-30,5236b673ce5f6bf8,ecdf5303a6b2ccfc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:32:34.019+08:00, clusters=[1] 2024-11-04 15:32:34.022 INFO 21062 --- [ocp-schedules-30,5236b673ce5f6bf8,ecdf5303a6b2ccfc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:32:34.056 INFO 21062 --- [ocp-schedules-7,ef9846d381ea2da5,00d7b5a918897e2a] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:32:34.059 INFO 21062 --- [ocp-schedules-7,ef9846d381ea2da5,00d7b5a918897e2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:34.117 ERROR 21062 --- [ocp-schedules-7,ef9846d381ea2da5,00d7b5a918897e2a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:32:34.117 INFO 21062 --- [ocp-schedules-7,ef9846d381ea2da5,00d7b5a918897e2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:32:35.606 WARN 21062 --- [ocp-async-7,3d913456b64decac,f02e747f4b0a81e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:35.611 INFO 21062 --- [ocp-async-9,10cbe5595d009fc9,4cc42c4ec0770fec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:35.932 INFO 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:32:35.932 INFO 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:32:35.932 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:32:35.932 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:32:35.941 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:32:35.942 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:32:35.942 INFO 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:32:35.943 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:32:35.943 WARN 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:32:35.943 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:32:35.943 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:32:35.944 INFO 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:32:35.944 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:32:35.944 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:32:35.945 WARN 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:32:35.945 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:32:35.951 INFO 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:32:35.951 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:32:35.951 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:32:35.964 ERROR 21062 --- [ocp-schedules-4,9cf1a4d2877a3444,d8b8f39e665b95fb] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:32:35.983 INFO 21062 --- [prometheus-minute-schedule-1,84382c99732db026,5fac25f6956e91a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:36.219 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:36.220 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:37.606 WARN 21062 --- [ocp-async-1,c82ea839b02642bd,ad2a263034679cf1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:37.611 INFO 21062 --- [ocp-async-8,6fe56eb7ad03fc2c,caa36a488e7348c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:38.006 INFO 21062 --- [ocp-schedules-11,93727c632397ea69,77e016c04eeeda4a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:32:38.504 INFO 21062 --- [ocp-schedules-1,5292433fa2aeac40,73500fb849592a7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:32:38.504 INFO 21062 --- [ocp-schedules-28,e4b0454acbe937d2,3b21c9e7faa79173] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:32:38.508 INFO 21062 --- [ocp-schedules-23,f368f08ca03543b2,eac4bf1d0e0173d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:32:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:32:38.589267+08:00, lastProcessTime=2024-11-04T15:32:28.587781+08:00 2024-11-04 15:32:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:32:38.589267+08:00 2024-11-04 15:32:38.770 INFO 21062 --- [ocp-async-4,0a5ca36a7dfc5dc6,aa95987c71e70af3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:32:38.983 INFO 21062 --- [prometheus-minute-schedule-1,5f43ec6ffee39e56,45550f6d441d8dec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:39.220 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:39.221 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:39.606 WARN 21062 --- [ocp-async-6,02daf77ffac94123,b740490efe65b8c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:39.611 INFO 21062 --- [ocp-async-4,44c8aefe4f2181ca,8c6e9d3e64692d90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:40.002 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:32:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:32:40.812 INFO 21062 --- [ocp-schedules-12,6a55f9301b4d340d,1fb81260c31f680c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:32:40.812 INFO 21062 --- [ocp-schedules-12,990962be41501a40,a2d328d8db0c6dd0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:32:40.816 INFO 21062 --- [ocp-schedules-12,990962be41501a40,a2d328d8db0c6dd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:40.844 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:32:40.844 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:32:41.606 WARN 21062 --- [ocp-async-3,1d24da65c5753f4d,a7437dc7ee836658] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:41.612 INFO 21062 --- [ocp-async-1,8e4c5d1c40974904,6b8027fe5b110490] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:41.794 INFO 21062 --- [ocp-schedules-23,82fda09b8971c0ae,75a95902cc4a9973] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:32:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:32:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:32:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:32:41.801 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:32:41.802 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:32:41.828 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:32:41.828 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:32:41.983 INFO 21062 --- [prometheus-minute-schedule-1,174c31a42396a297,6060a4d482ef7da8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:42.221 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:42.222 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:43.606 WARN 21062 --- [ocp-async-5,2fecb9e83c78b12b,841bf8b2f5fc3717] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:43.612 INFO 21062 --- [ocp-async-2,5e69aefef245404b,1a270825bc4535db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:44.983 INFO 21062 --- [prometheus-minute-schedule-1,2c8414c53e98716f,d07de7e5eeb9f8a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:45.157 INFO 21062 --- [ocp-schedules-7,3c34c95d85b3cbe5,da6b0658f502caa1] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:32:45.160 INFO 21062 --- [ocp-schedules-7,3c34c95d85b3cbe5,da6b0658f502caa1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:45.223 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:45.223 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:45.241 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:32:45.243 ERROR 21062 --- [ocp-schedules-7,3c34c95d85b3cbe5,da6b0658f502caa1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:32:45.243 INFO 21062 --- [ocp-schedules-7,3c34c95d85b3cbe5,da6b0658f502caa1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:32:45.607 WARN 21062 --- [ocp-async-10,461a037246a10cf9,5dba077ed0c3c0ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:45.612 INFO 21062 --- [ocp-async-7,a94542af47e8d1ad,c2c8cfbc26a2b7ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:47.607 WARN 21062 --- [ocp-async-4,a7b2fa3528dea433,d98fc393c6781f4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:47.612 INFO 21062 --- [ocp-async-9,8403ad2ef1310551,a2b68f7da9b90cb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8d7052d13e1edcbe,10d88d3bc2dd1326] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:48.040 INFO 21062 --- [ocp-schedules-30,27d1287fb968f83f,54395c87be66d44c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:32:48.223 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:48.224 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:32:48.590936+08:00, lastProcessTime=2024-11-04T15:32:38.589267+08:00 2024-11-04 15:32:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:32:48.590936+08:00 2024-11-04 15:32:48.770 INFO 21062 --- [ocp-async-8,27006b7b6f56eddf,c67ebfbfe7e11fac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:32:49.612 WARN 21062 --- [ocp-async-8,6843921a6f32cd18,aa195e89e80b4178] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:49.612 INFO 21062 --- [ocp-async-2,7c902bc2113f6493,7e0cae7b76dd5eaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:50.983 INFO 21062 --- [prometheus-minute-schedule-1,97d48c707218a7f2,d129a122515e1bc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:51.224 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:51.225 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:51.612 WARN 21062 --- [ocp-async-6,99ce0889518cdc2e,13ed93a411ab4182] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:51.612 INFO 21062 --- [ocp-async-7,3dbd4b3b3ce079e9,60f93ccf8ae6c56a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:51.800 INFO 21062 --- [ocp-schedules-9,05aebcb65c558915,42ecf657efd8ffbc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:32:53.206 INFO 21062 --- [ocp-schedules-16,c38f05e7dc2223b8,fde96354aee53348] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:32:53.212 INFO 21062 --- [ocp-schedules-16,c38f05e7dc2223b8,fde96354aee53348] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:32:53.212 INFO 21062 --- [ocp-schedules-16,c38f05e7dc2223b8,fde96354aee53348] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:32:53.612 WARN 21062 --- [ocp-async-3,5707cf11e8595801,428cecd9416ad44a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:53.612 INFO 21062 --- [ocp-async-9,0263c6f039759418,f9a190c4b32706f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d702bad676953253,45c66cb355a76668] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:54.225 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:54.226 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:55.613 WARN 21062 --- [ocp-async-8,fafd503596414c87,7a171975b7cd0ef6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:55.613 INFO 21062 --- [ocp-async-5,188586b600a7b5ed,b790431c2708a352] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:56.983 INFO 21062 --- [prometheus-minute-schedule-1,628de4d31ea11f22,5d8a7488843a55ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:32:57.226 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:32:57.227 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:32:57.613 WARN 21062 --- [ocp-async-6,c6e9aa4068fc6a9a,985a9c9a0da3068a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:57.613 INFO 21062 --- [ocp-async-10,faf177c219283b70,323e1f10963cc45d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:58.082 INFO 21062 --- [ocp-schedules-22,66d85b647b8e834f,ce530d6d0d61a4dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:32:58.500 INFO 21062 --- [ocp-schedules-15,4be87c590566684d,af9ed8eb9b0f540e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:32:58.500 INFO 21062 --- [ocp-schedules-27,d535f31b30926f57,51bac10bdad20b57] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:32:58.502 INFO 21062 --- [ocp-schedules-3,76421364a3f7bd7a,6c107149f742c397] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:32:58.503 INFO 21062 --- [ocp-schedules-15,4be87c590566684d,af9ed8eb9b0f540e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:32:58.504 INFO 21062 --- [ocp-schedules-27,d535f31b30926f57,51bac10bdad20b57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:32:58.504 INFO 21062 --- [ocp-schedules-8,87259573c701e354,d4370f728df6ba89] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:32:58.509 INFO 21062 --- [ocp-schedules-14,88a5a7e656c984e2,8e9bbfdeeceb4354] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:32:58.509 INFO 21062 --- [ocp-schedules-14,88a5a7e656c984e2,8e9bbfdeeceb4354] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:32:58.510 INFO 21062 --- [ocp-schedules-21,8a16a298672ee786,cd1c4f33535fe4b4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:32:58.510 INFO 21062 --- [ocp-schedules-21,8a16a298672ee786,cd1c4f33535fe4b4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:32:58.510 INFO 21062 --- [ocp-schedules-28,c1e1b47b25f64df9,f9c885adbc8b1b61] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:32:58.510 INFO 21062 --- [ocp-schedules-28,c1e1b47b25f64df9,f9c885adbc8b1b61] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:32:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:32:58.589242+08:00, lastProcessTime=2024-11-04T15:32:48.590936+08:00 2024-11-04 15:32:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:32:58.589242+08:00 2024-11-04 15:32:58.770 INFO 21062 --- [ocp-async-3,4dc02b24066e8fef,a4f9e3ca22f01516] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:32:59.557 INFO 21062 --- [ocp-schedules-26,6c1edd0469fd3bb0,99a13ed7038cc69e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:32:59.613 WARN 21062 --- [ocp-async-3,9e0e01700974a2d0,1a5fcce6f77470d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:32:59.613 INFO 21062 --- [ocp-async-9,75b0deabfad4783f,cb5c64ef6dabf4cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:32:59.983 INFO 21062 --- [prometheus-minute-schedule-1,a8e1fab162dd9960,0ce8c46071d1ee41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705580, interval=60 2024-11-04 15:33:00.227 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:00.229 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:01.613 WARN 21062 --- [ocp-async-1,17145c1be1ec3bc1,cb63ef29f9883bfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:01.613 INFO 21062 --- [ocp-async-1,0a33be659f4e7a47,142f76f9fc202ba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:01.802 INFO 21062 --- [ocp-schedules-3,d84c940c395bf717,b177d0015d54238f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:33:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a0341a46915e3857,da44200f13022910] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:03.229 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:03.230 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:03.613 WARN 21062 --- [ocp-async-2,c005ac7f7a76e220,3023e0eadd7d4431] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:03.613 INFO 21062 --- [ocp-async-10,1512cda2c3c3d9db,777676d908eede2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:05.613 WARN 21062 --- [ocp-async-3,11efe3958c418573,34dbf0bd8bb9aeae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:05.614 INFO 21062 --- [ocp-async-3,242ab19737760a3c,3b9ea6f27d76ee72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8b5b1d7fb2cfa0e8,5ff4896765c88011] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:06.231 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:06.231 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:07.614 WARN 21062 --- [ocp-async-8,7826932560fcfba3,dde8513fe214da7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:07.614 INFO 21062 --- [ocp-async-5,bd00c6af3c86f25d,12817387b8c5207e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:08.106 INFO 21062 --- [ocp-schedules-3,3a4cbdbbd73ff773,7b5df3412c37563d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:33:08.504 INFO 21062 --- [ocp-schedules-15,4be87c590566684d,af9ed8eb9b0f540e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:33:08.504 INFO 21062 --- [ocp-schedules-27,d535f31b30926f57,51bac10bdad20b57] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:33:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:33:08.588886+08:00, lastProcessTime=2024-11-04T15:32:58.589242+08:00 2024-11-04 15:33:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:33:08.588886+08:00 2024-11-04 15:33:08.770 INFO 21062 --- [ocp-async-2,5bfdfeff6c8b888f,3322a79a066c1f5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:33:08.983 INFO 21062 --- [prometheus-minute-schedule-1,7058d87e40a2440c,cfc523cb38da1786] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:09.231 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:09.232 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:09.349 INFO 21062 --- [ocp-schedules-7,50dffc5be1f85611,88c27e4fed3ebd9a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:33:09.352 INFO 21062 --- [ocp-schedules-7,50dffc5be1f85611,88c27e4fed3ebd9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:09.352 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:33:09.614 INFO 21062 --- [ocp-async-2,a3da0e5638cd0f08,df27310d96efe80b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:09.614 WARN 21062 --- [ocp-async-10,5d5307b540fb9f05,c596f929acf70152] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:33:10.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:33:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:33:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:33:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:33:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:33:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:33:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:33:10.845 INFO 21062 --- [ocp-schedules-12,990962be41501a40,a2d328d8db0c6dd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:33:10.845 INFO 21062 --- [ocp-schedules-12,658a031dd2d581b9,70172551175a9a1f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:33:10.849 INFO 21062 --- [ocp-schedules-12,658a031dd2d581b9,70172551175a9a1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:10.879 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:33:10.879 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:33:11.614 WARN 21062 --- [ocp-async-3,966cfed07ceeb79f,46e429e625f28918] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:11.614 INFO 21062 --- [ocp-async-7,4cd96d0c3de7ecd9,b587cae90306fe38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:11.805 INFO 21062 --- [ocp-schedules-20,06b5f9e63dddb873,f70761724e696b3c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:33:11.859 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=3, affectRows=3 2024-11-04 15:33:11.859 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=6, affectRows=6 2024-11-04 15:33:11.859 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=12, affectRows=12 2024-11-04 15:33:11.859 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=3, validCount=3 2024-11-04 15:33:11.859 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=5, validCount=5 2024-11-04 15:33:11.860 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=12, validCount=12 2024-11-04 15:33:11.860 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=11, affectRows=11 2024-11-04 15:33:11.860 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=11, validCount=11 2024-11-04 15:33:11.860 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=34, affectRows=34 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=34, validCount=34 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=42, affectRows=42 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=42, validCount=42 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=30, affectRows=30 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=50, affectRows=50 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=30, validCount=30 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=81, affectRows=81 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=73, affectRows=73 2024-11-04 15:33:11.861 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=50, validCount=50 2024-11-04 15:33:11.862 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=73, validCount=73 2024-11-04 15:33:11.862 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=81, validCount=81 2024-11-04 15:33:11.862 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705520, nodeCount=80, affectRows=80 2024-11-04 15:33:11.862 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705520, nodeCount=78, validCount=78 2024-11-04 15:33:11.983 INFO 21062 --- [prometheus-minute-schedule-1,2985bf0ce813d02f,8a5a3a2dab974fed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:12.233 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:12.233 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:13.214 INFO 21062 --- [ocp-schedules-20,cb181fd60c1a2f54,44ddab8c567f8ff8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:33:13.220 INFO 21062 --- [ocp-schedules-20,cb181fd60c1a2f54,44ddab8c567f8ff8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:33:13.220 INFO 21062 --- [ocp-schedules-20,cb181fd60c1a2f54,44ddab8c567f8ff8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:33:13.614 WARN 21062 --- [ocp-async-5,0213b5c05b9d0437,e35ff2014317d9b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:13.614 INFO 21062 --- [ocp-async-9,8128ea594ebd6c71,93ddafdba56a77bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:13.984 INFO 21062 --- [prometheus-cache-monitor1,edacdcf6c4f8eebf,af9652db6899f004] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:33:13.984 INFO 21062 --- [prometheus-cache-monitor1,edacdcf6c4f8eebf,af9652db6899f004] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:33:14.501 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:27:14.484, end=2024-11-04T15:32:14.498} 2024-11-04 15:33:14.501 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:27:14.483, end=2024-11-04T15:32:14.499} 2024-11-04 15:33:14.511 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:33:14.511 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:33:14.511 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:33:14.511 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:33:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9689321eeaa93604,e9ffc5010e96627e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:15.234 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:15.234 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:15.614 WARN 21062 --- [ocp-async-6,7822689cba1077ae,e1e5163f6bc1e540] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:15.614 INFO 21062 --- [ocp-async-1,95fdb9986f363fa3,fbb31c5d9a4d7efe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:15.691 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:29:15.674, end=2024-11-04T15:31:15.689} 2024-11-04 15:33:15.695 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:33:15.695 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:33:17.614 WARN 21062 --- [ocp-async-3,61fca6577597ad7c,fa42b178bc39df81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:17.615 INFO 21062 --- [ocp-async-4,87369c28cded8daf,e2567c9a37a61ced] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:17.983 INFO 21062 --- [prometheus-minute-schedule-1,6f394e327803ee53,a415cd4632ae6201] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:18.147 INFO 21062 --- [ocp-schedules-24,4b4017a2cfa7aaf0,4780ad1934d8d1e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:33:18.234 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:18.235 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:33:18.589304+08:00, lastProcessTime=2024-11-04T15:33:08.588886+08:00 2024-11-04 15:33:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:33:18.589304+08:00 2024-11-04 15:33:18.770 INFO 21062 --- [ocp-async-5,95888a4259ead75e,2ec55bd26a5c0f02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:33:19.353 INFO 21062 --- [ocp-schedules-7,50dffc5be1f85611,88c27e4fed3ebd9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:33:19.615 WARN 21062 --- [ocp-async-5,fee532917a27e78b,da020220be4f71a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:19.615 INFO 21062 --- [ocp-async-8,370960f7304f2870,30c9d2924442cffe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:20.983 INFO 21062 --- [prometheus-minute-schedule-1,00e5d2f8bc03369e,733cf6e2c3e595b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:21.235 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:21.236 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:21.615 INFO 21062 --- [ocp-async-6,40e4b417fd2a9f76,b453c8543d44a3f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:21.615 WARN 21062 --- [ocp-async-1,5e0e818221c4bc1e,48c356f21c0e94a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:21.808 INFO 21062 --- [ocp-schedules-3,422724ffd372a50e,1cc81c3f4c830be9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:33:23.615 WARN 21062 --- [ocp-async-3,c2349b1dce47a2ea,38c32b24e2114787] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:23.615 INFO 21062 --- [ocp-async-4,00ea1e3a9b4d348f,7428886abf7ce589] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:23.983 INFO 21062 --- [prometheus-minute-schedule-1,38f21995906ac8fe,1ff735ca1d8b21f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:24.236 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:24.237 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:25.615 WARN 21062 --- [ocp-async-7,5a98eda5d32bb1c2,a0372d8d5242c7a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:25.616 INFO 21062 --- [ocp-async-9,a226077a626d1a33,dc3fae04ea40ed71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f076a567147d107b,1cc671700a55f10c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:26.984 INFO 21062 --- [prometheus-hour-schedule-1,81a281d941b9f55b,0d5afd200ce7ff26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:33:27.237 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:27.238 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:27.616 WARN 21062 --- [ocp-async-8,1cce844a43b5a326,9322d82ee9493c7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:27.616 INFO 21062 --- [ocp-async-1,c70416bdb235b8dc,3885cc3f6aecc820] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:28.172 INFO 21062 --- [ocp-schedules-10,0995f8c7de45aac2,258ed35b525031b4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:33:28.498 INFO 21062 --- [ocp-schedules-4,7d32ee2bd591a827,6543eacdd93c8237] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:33:28.499 INFO 21062 --- [ocp-schedules-15,b176aaf3b073d633,bd7a718cf205bb01] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:33:28.499 INFO 21062 --- [ocp-schedules-16,1f1a4802b3d8bb52,8ce3eb80781f4b10] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:33:28.500 INFO 21062 --- [ocp-schedules-14,539dfeaef2b74560,deceaa426ec95769] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:33:28.502 INFO 21062 --- [ocp-schedules-4,7d32ee2bd591a827,6543eacdd93c8237] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:33:28.502 INFO 21062 --- [ocp-schedules-5,c9f265ff332ca9d4,87f7b1c146907b95] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:33:28.503 INFO 21062 --- [ocp-schedules-16,1f1a4802b3d8bb52,8ce3eb80781f4b10] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:33:28.504 INFO 21062 --- [ocp-schedules-14,539dfeaef2b74560,deceaa426ec95769] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:28.504 INFO 21062 --- [ocp-schedules-18,47afdffdb9840ca7,ebd4af51a813018e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:33:28.504 INFO 21062 --- [ocp-schedules-15,b176aaf3b073d633,bd7a718cf205bb01] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:33:28.510 INFO 21062 --- [ocp-schedules-30,f571b213ff8c8cd1,97ab61fb68f543e8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:33:28.510 INFO 21062 --- [ocp-schedules-22,0ead5ad69256d665,983c9da955b661db] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:33:28.510 INFO 21062 --- [ocp-schedules-22,0ead5ad69256d665,983c9da955b661db] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:33:28.510 INFO 21062 --- [ocp-schedules-30,f571b213ff8c8cd1,97ab61fb68f543e8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:33:28.510 INFO 21062 --- [ocp-schedules-24,114141ec864c04e5,0a88e36b49f19de5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:33:28.510 INFO 21062 --- [ocp-schedules-24,114141ec864c04e5,0a88e36b49f19de5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:33:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:33:28.589297+08:00, lastProcessTime=2024-11-04T15:33:18.589304+08:00 2024-11-04 15:33:28.605 INFO 21062 --- [ocp-schedules-2,59128b96d647bf75,5f7fd434d9c17783] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 15:33:28.605 INFO 21062 --- [ocp-schedules-2,59128b96d647bf75,5f7fd434d9c17783] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 15:33:28.670 INFO 21062 --- [ocp-schedules-23,18099c0fc6b842a8,9a4bb2e2d62912ce] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:33:28.671 INFO 21062 --- [ocp-schedules-23,18099c0fc6b842a8,9a4bb2e2d62912ce] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:33:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:33:28.589297+08:00 2024-11-04 15:33:28.770 INFO 21062 --- [ocp-async-7,b781c75e47025d77,48cc91f88ab7dd33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:33:29.010 INFO 21062 --- [ocp-schedules-9,655948100ea031a7,204f9d7321e7c268] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:33:29.414 INFO 21062 --- [ocp-schedules-2,9fb0daf15e3efae8,d96fcd4d3dedcb59] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:33:29.415 INFO 21062 --- [ocp-schedules-2,9fb0daf15e3efae8,d96fcd4d3dedcb59] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:33:29.561 INFO 21062 --- [ocp-schedules-23,dc9a7eee55e45d7e,aee603cd253ac1d7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:33:29.581 INFO 21062 --- [ocp-schedules-8,c768a9de569f308b,aea5a2228e8f5126] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:33:29.587 INFO 21062 --- [ocp-schedules-8,c768a9de569f308b,aea5a2228e8f5126] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:33:29.587 INFO 21062 --- [ocp-schedules-8,c768a9de569f308b,aea5a2228e8f5126] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:33:29.616 WARN 21062 --- [ocp-async-7,4566fb7567997d1e,7d06b815334f7c69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:29.616 INFO 21062 --- [ocp-async-7,89fff47bdb35182d,84ce1c9ea424414d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:29.983 INFO 21062 --- [prometheus-minute-schedule-1,fe7a7ac45d06eb8a,617908849abe73eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:30.071 INFO 21062 --- [ocp-schedules-26,ab3bad94ce0ad066,e2130e9c9ce135b4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:33:30.071 INFO 21062 --- [ocp-schedules-26,ab3bad94ce0ad066,e2130e9c9ce135b4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:33:30.072 INFO 21062 --- [ocp-schedules-26,ab3bad94ce0ad066,e2130e9c9ce135b4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:33:30.072 INFO 21062 --- [ocp-schedules-26,ab3bad94ce0ad066,e2130e9c9ce135b4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:33:30.073 INFO 21062 --- [ocp-schedules-26,ab3bad94ce0ad066,e2130e9c9ce135b4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:33:30.073 INFO 21062 --- [ocp-schedules-26,ab3bad94ce0ad066,e2130e9c9ce135b4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:33:30.238 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:30.239 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:33:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:33:31.616 WARN 21062 --- [ocp-async-8,f424cc9c26242cfd,e097af34cb48fdbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:31.616 INFO 21062 --- [ocp-async-9,10399b0c0cb011f4,27a7c9ea0cf5f73e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:31.810 INFO 21062 --- [ocp-schedules-2,2bf55811fddf18ce,a43d7ead26ca7e66] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:33:32.983 INFO 21062 --- [prometheus-minute-schedule-1,1cf5d5111e5dcedc,8538f137d59fd17e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:33.223 INFO 21062 --- [ocp-schedules-4,d8a47d7479822c8d,3fafff3f8a2c04de] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:33:33.227 INFO 21062 --- [ocp-schedules-4,d8a47d7479822c8d,3fafff3f8a2c04de] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:33:33.227 INFO 21062 --- [ocp-schedules-4,d8a47d7479822c8d,3fafff3f8a2c04de] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:33:33.239 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:33.240 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:33.478 INFO 21062 --- [ocp-schedules-1,e0b4ee499c189bae,63af6bbf37a3aad1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:33:33.478 INFO 21062 --- [ocp-schedules-1,e0b4ee499c189bae,63af6bbf37a3aad1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:33:33.507 INFO 21062 --- [ocp-schedules-10,f400969ccf89febf,e0b727cfc2e24d69] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:33:33.508 INFO 21062 --- [ocp-schedules-1,e0b4ee499c189bae,63af6bbf37a3aad1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:33:33.508 INFO 21062 --- [ocp-schedules-1,e0b4ee499c189bae,63af6bbf37a3aad1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:33:33.616 WARN 21062 --- [ocp-async-10,f571e00a19e39092,5a618d72ca40ed84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:33.616 INFO 21062 --- [ocp-async-2,58f4027c9749e885,e45c57c5ce569d21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:33.719 INFO 21062 --- [ocp-schedules-24,cf28209ee34190e1,89edef5a4d99d550] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:33:33.719+08:00, clusters=[1] 2024-11-04 15:33:33.720 INFO 21062 --- [ocp-schedules-24,cf28209ee34190e1,89edef5a4d99d550] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:33:34.051 INFO 21062 --- [ocp-schedules-5,72a94cbc468a39d1,53bce2cd0bae3b53] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:33:34.051+08:00, clusters=[1] 2024-11-04 15:33:34.055 INFO 21062 --- [ocp-schedules-5,72a94cbc468a39d1,53bce2cd0bae3b53] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:33:35.616 WARN 21062 --- [ocp-async-4,8924a03d71704d2f,7ee70d880ae99b6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:35.616 INFO 21062 --- [ocp-async-3,0da9ace003d42bb6,1a204f08431ce6e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:35.974 INFO 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:33:35.975 INFO 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:33:35.975 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:33:35.975 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:33:35.983 INFO 21062 --- [prometheus-minute-schedule-1,03e0693585447c22,b2995ceb0e01d14d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:35.984 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:33:35.984 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:33:35.984 INFO 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:33:35.986 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:33:35.986 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:33:35.986 WARN 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:33:35.986 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:33:35.987 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:33:35.987 INFO 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:33:35.987 WARN 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:33:35.988 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:33:35.988 WARN 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:33:35.995 INFO 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:33:35.995 INFO 21062 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:33:35.995 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:33:36.006 ERROR 21062 --- [ocp-schedules-24,e1093c6d97b41449,3a1b30ae28dc3092] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:33:36.240 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:36.241 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:37.617 WARN 21062 --- [ocp-async-8,340a84b843cee2ab,198c8d38448f6742] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:37.617 INFO 21062 --- [ocp-async-5,87b7a6370b0f71cc,ce3eb00356f7420f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:38.249 INFO 21062 --- [ocp-schedules-1,bc14cbb10065088e,2f6c2c98cdcddc77] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:33:38.504 INFO 21062 --- [ocp-schedules-16,1f1a4802b3d8bb52,8ce3eb80781f4b10] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:33:38.504 INFO 21062 --- [ocp-schedules-14,539dfeaef2b74560,deceaa426ec95769] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:33:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:33:38.589250+08:00, lastProcessTime=2024-11-04T15:33:28.589297+08:00 2024-11-04 15:33:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:33:38.589250+08:00 2024-11-04 15:33:38.771 INFO 21062 --- [ocp-async-2,ea534d37f5b55bfd,7f1380230ee118de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:33:38.983 INFO 21062 --- [prometheus-minute-schedule-1,76f23198ede87d28,f80a8d86506e2928] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:39.241 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:39.242 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:39.617 INFO 21062 --- [ocp-async-6,d6022ef11b12209f,c294aae89d89bc3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:39.617 WARN 21062 --- [ocp-async-2,c6622db973c4ca77,4f4eb256d05e2d0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:40.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:33:40.003 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:33:40.879 INFO 21062 --- [ocp-schedules-12,658a031dd2d581b9,70172551175a9a1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:33:40.879 INFO 21062 --- [ocp-schedules-12,dfd92c3a8eb33403,84d7120b176ac44b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:33:40.883 INFO 21062 --- [ocp-schedules-12,dfd92c3a8eb33403,84d7120b176ac44b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:40.912 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:33:40.913 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:33:41.617 INFO 21062 --- [ocp-async-3,b3f59f6a7b217e3d,262f7333d0e851f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:41.617 WARN 21062 --- [ocp-async-7,d18fbe170e1448b7,b0d17f3150f9d3a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:41.813 INFO 21062 --- [ocp-schedules-1,82b0b3bbd2bbc211,b3e0d7b914b62b43] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:33:41.983 INFO 21062 --- [prometheus-minute-schedule-1,246d84f93d2ad427,9fe272032485ad49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:42.242 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:42.243 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:43.617 INFO 21062 --- [ocp-async-9,5e02ab964eeefcc3,21feb8097ea494a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:43.617 WARN 21062 --- [ocp-async-5,6859a340600c66f8,96525338e1488608] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:44.983 INFO 21062 --- [prometheus-minute-schedule-1,dc06d36ac06e32af,bb933985dbdf9173] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:45.243 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:45.243 INFO 21062 --- [ocp-schedules-22,6ad89cfec16699dc,6f1f95c7b17d5f8c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:33:45.244 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:45.247 INFO 21062 --- [ocp-schedules-22,6ad89cfec16699dc,6f1f95c7b17d5f8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:45.333 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:33:45.337 ERROR 21062 --- [ocp-schedules-22,6ad89cfec16699dc,6f1f95c7b17d5f8c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:33:45.337 INFO 21062 --- [ocp-schedules-22,6ad89cfec16699dc,6f1f95c7b17d5f8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:33:45.617 INFO 21062 --- [ocp-async-2,67e831b7cf6aec3b,d2993b374ab712fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:45.617 WARN 21062 --- [ocp-async-1,de611ef7cdf60556,4eac080e6c77b41d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:47.618 WARN 21062 --- [ocp-async-4,d7d8b3cd9db11d0c,1cc423b7dfad76ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:47.618 INFO 21062 --- [ocp-async-7,68c3fe58ddd82c6f,ad05774ec0dab1e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:47.983 INFO 21062 --- [prometheus-minute-schedule-1,bc0dd33352981315,a4c92cd8ccdcbb7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:48.244 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:48.245 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:48.275 INFO 21062 --- [ocp-schedules-19,af41f414e65a551f,fd6424caf34425ad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:33:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:33:48.589691+08:00, lastProcessTime=2024-11-04T15:33:38.589250+08:00 2024-11-04 15:33:48.771 INFO 21062 --- [ocp-async-5,bc1948463a7fdfe5,e93c6b1a6a941509] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:33:48.773 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:33:48.589691+08:00 2024-11-04 15:33:49.618 WARN 21062 --- [ocp-async-5,824604a41d79ad88,fb827ecaa70694f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:49.618 INFO 21062 --- [ocp-async-9,91397ed1399b011b,a34dbbf96e61a69c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:50.983 INFO 21062 --- [prometheus-minute-schedule-1,146aa0cff248909c,d854f3c858b02e46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:51.245 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:51.246 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:51.618 WARN 21062 --- [ocp-async-10,6da8b73befd88155,5f8fccae284cc966] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:51.618 INFO 21062 --- [ocp-async-2,d077c09fc70ea0b2,9029e9af8c7f051f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:51.816 INFO 21062 --- [ocp-schedules-24,8e48079a75a0c9e7,65fdcadcc75f9d0b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:33:53.230 INFO 21062 --- [ocp-schedules-16,561d7d39a54118e9,05a90a2c7d9aa5c2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:33:53.236 INFO 21062 --- [ocp-schedules-16,561d7d39a54118e9,05a90a2c7d9aa5c2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:33:53.236 INFO 21062 --- [ocp-schedules-16,561d7d39a54118e9,05a90a2c7d9aa5c2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:33:53.618 WARN 21062 --- [ocp-async-4,5abc03cbdbb67373,e91a691abd80ee43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:53.618 INFO 21062 --- [ocp-async-6,8e664345f742277e,7aafae23f9b69533] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1c95d21303cb0cec,96f2b2a12ae2c01e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:54.247 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:54.248 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:55.618 WARN 21062 --- [ocp-async-5,bf6b56fd2ce1eb03,2f0db9820f95b8c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:55.619 INFO 21062 --- [ocp-async-8,500926d1a36135c9,b35785ac6e5d03a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:56.983 INFO 21062 --- [prometheus-minute-schedule-1,761ef08bc3068c81,a138ad75eb5adf78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:33:57.248 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:33:57.249 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:33:57.619 WARN 21062 --- [ocp-async-10,515940d6028d5281,cb0ff01e78ed7ed0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:57.619 INFO 21062 --- [ocp-async-10,8f65c8cdf9b705a1,88155138c7308df0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:58.315 INFO 21062 --- [ocp-schedules-22,2667578cd595ea8f,72f7f3dcc76a1234] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:33:58.500 INFO 21062 --- [ocp-schedules-14,3dbe92f7d739f166,7aee5e1dced4e6db] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:33:58.500 INFO 21062 --- [ocp-schedules-30,f26cbffa00dafda2,e0c1d2b3b6f6bd11] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:33:58.502 INFO 21062 --- [ocp-schedules-23,de775ad719c231ec,3c82f001b88dc993] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:33:58.503 INFO 21062 --- [ocp-schedules-14,3dbe92f7d739f166,7aee5e1dced4e6db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:58.503 INFO 21062 --- [ocp-schedules-30,f26cbffa00dafda2,e0c1d2b3b6f6bd11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:33:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:33:58.504 INFO 21062 --- [ocp-schedules-13,5114dfe7a9c8ed43,c7b2872e81f47f62] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:33:58.510 INFO 21062 --- [ocp-schedules-25,29c497e949ec192c,c88a40a5185482f4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:33:58.510 INFO 21062 --- [ocp-schedules-17,d7580d41054024df,12ead15fad5469ca] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:33:58.510 INFO 21062 --- [ocp-schedules-25,29c497e949ec192c,c88a40a5185482f4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:33:58.510 INFO 21062 --- [ocp-schedules-17,d7580d41054024df,12ead15fad5469ca] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:33:58.510 INFO 21062 --- [ocp-schedules-10,e5d1dc3e955ac5af,eb04bf421dac9991] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:33:58.510 INFO 21062 --- [ocp-schedules-10,e5d1dc3e955ac5af,eb04bf421dac9991] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:33:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:33:58.589239+08:00, lastProcessTime=2024-11-04T15:33:48.589691+08:00 2024-11-04 15:33:58.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:33:58.589239+08:00 2024-11-04 15:33:58.771 INFO 21062 --- [ocp-async-7,a8a74f5e9649109c,30a5a8ca7e418be7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:33:59.564 INFO 21062 --- [ocp-schedules-13,f928349cce32d4af,2e6c0ccec84619f8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:33:59.619 WARN 21062 --- [ocp-async-4,5e0dc23bbac931b4,d0cf5b7eec92c839] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:33:59.619 INFO 21062 --- [ocp-async-6,baf2a6e3111c235f,db385190d5bb46f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:33:59.983 INFO 21062 --- [prometheus-minute-schedule-1,bff5affb36d18264,cc9bffa51d58a206] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705640, interval=60 2024-11-04 15:34:00.249 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:00.250 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:01.619 WARN 21062 --- [ocp-async-3,f0be0b68267e329f,5fa6789c13943820] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:01.619 INFO 21062 --- [ocp-async-3,2ee13d94637a64d7,a6d0bcaa9daceaa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:01.819 INFO 21062 --- [ocp-schedules-19,e25dcaf76f4fc1ad,ac53ace6b1d20d7c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:34:02.983 INFO 21062 --- [prometheus-minute-schedule-1,c9d54131298cb7b2,452b390825446de7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:03.250 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:03.251 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:03.619 WARN 21062 --- [ocp-async-1,f89c708ccdf43bd3,15cdbbff4f7cae73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:03.619 INFO 21062 --- [ocp-async-10,300ffac3b638bd86,171f0c809c6e002d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:05.619 WARN 21062 --- [ocp-async-7,a90e082a223704e9,1b446ba391a4b4e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:05.619 INFO 21062 --- [ocp-async-4,4687670b54d5cac4,b854229238394715] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:05.983 INFO 21062 --- [prometheus-minute-schedule-1,0bbb7061b84813a7,cc454ee92c4614d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:06.252 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:06.252 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:07.619 WARN 21062 --- [ocp-async-5,5f94bebee6b0e4a1,67e56935c456c1bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:07.620 INFO 21062 --- [ocp-async-8,79a8b5893b066f9a,5d0aa2a51d09a9bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:08.340 INFO 21062 --- [ocp-schedules-25,9790ad010b017291,420981c2f196ebdb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:34:08.504 INFO 21062 --- [ocp-schedules-14,3dbe92f7d739f166,7aee5e1dced4e6db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:34:08.504 INFO 21062 --- [ocp-schedules-30,f26cbffa00dafda2,e0c1d2b3b6f6bd11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:34:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:34:08.588870+08:00, lastProcessTime=2024-11-04T15:33:58.589239+08:00 2024-11-04 15:34:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:34:08.588870+08:00 2024-11-04 15:34:08.771 INFO 21062 --- [ocp-async-1,e87ab6c31835a85e,c420c11091b253de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:34:08.983 INFO 21062 --- [prometheus-minute-schedule-1,7ba3882c2287d07a,d262462d3b84de36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:09.252 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:09.253 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:09.620 WARN 21062 --- [ocp-async-9,7f8cea0ea41e7c47,b03825416ea5837f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:09.620 INFO 21062 --- [ocp-async-10,2ad8d0821fd66138,4f2f2c675b949a5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:34:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:34:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:34:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:34:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:34:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:34:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:34:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:34:10.913 INFO 21062 --- [ocp-schedules-12,dfd92c3a8eb33403,84d7120b176ac44b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:34:10.913 INFO 21062 --- [ocp-schedules-12,e3fd2b169424631a,f06539004b6933ea] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:34:10.917 INFO 21062 --- [ocp-schedules-12,e3fd2b169424631a,f06539004b6933ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:10.947 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:34:10.947 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:34:11.620 WARN 21062 --- [ocp-async-2,1ae51557fbe63af0,5b08b748ca94287b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:11.620 INFO 21062 --- [ocp-async-4,643ea56936181330,6b2e686952091017] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:11.821 INFO 21062 --- [ocp-schedules-27,fd25d51533241586,1bc1376336367d1a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:34:11.863 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=12, affectRows=12 2024-11-04 15:34:11.863 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=12, validCount=12 2024-11-04 15:34:11.863 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=9, affectRows=9 2024-11-04 15:34:11.863 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=15, affectRows=15 2024-11-04 15:34:11.863 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=9, validCount=9 2024-11-04 15:34:11.864 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=15, validCount=15 2024-11-04 15:34:11.864 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=27, affectRows=27 2024-11-04 15:34:11.864 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=28, affectRows=28 2024-11-04 15:34:11.864 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=26, validCount=26 2024-11-04 15:34:11.864 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=28, validCount=28 2024-11-04 15:34:11.865 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=60, affectRows=60 2024-11-04 15:34:11.865 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=50, affectRows=50 2024-11-04 15:34:11.865 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=47, affectRows=47 2024-11-04 15:34:11.865 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=60, validCount=60 2024-11-04 15:34:11.865 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=47, validCount=47 2024-11-04 15:34:11.865 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=50, validCount=50 2024-11-04 15:34:11.866 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=87, affectRows=87 2024-11-04 15:34:11.866 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705580, nodeCount=87, affectRows=87 2024-11-04 15:34:11.866 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=87, validCount=87 2024-11-04 15:34:11.866 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705580, nodeCount=85, validCount=85 2024-11-04 15:34:11.983 INFO 21062 --- [prometheus-minute-schedule-1,658d99c540e23ea6,6e532cb7709ed814] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:12.253 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:12.255 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:13.239 INFO 21062 --- [ocp-schedules-15,f3f61385a1491ea9,b7ab367181ef9afa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:34:13.244 INFO 21062 --- [ocp-schedules-15,f3f61385a1491ea9,b7ab367181ef9afa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:34:13.244 INFO 21062 --- [ocp-schedules-15,f3f61385a1491ea9,b7ab367181ef9afa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:34:13.620 WARN 21062 --- [ocp-async-5,19f94b97524e6a24,0ed3de696e8e5f4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:13.620 INFO 21062 --- [ocp-async-5,a4804676b6676872,c7667ea63730ca60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:13.984 INFO 21062 --- [prometheus-cache-monitor1,87693b30b128973e,889e9fbfc66bc6d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:34:13.984 INFO 21062 --- [prometheus-cache-monitor1,87693b30b128973e,889e9fbfc66bc6d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:34:14.983 INFO 21062 --- [prometheus-minute-schedule-1,6682bd4f6682eeb5,0b9e682267c692ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:15.255 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:15.256 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:15.620 WARN 21062 --- [ocp-async-3,fc12adbee61d7410,430df8aa4b9dce4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:15.620 INFO 21062 --- [ocp-async-9,c9c4e0f78f44dbc9,767c7e8d336d1dc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:17.620 WARN 21062 --- [ocp-async-7,0766ebae37875096,b3c25b5a19bbbc2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:17.620 INFO 21062 --- [ocp-async-7,092b84f614e21b1e,6ec5f893c633f1e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5a6155be9f629886,fa3bc95a60a579a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:18.256 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:18.258 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:18.388 INFO 21062 --- [ocp-schedules-7,3b8c81e7197eea9a,427280bb0b62d34c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:34:18.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:34:18.591860+08:00, lastProcessTime=2024-11-04T15:34:08.588870+08:00 2024-11-04 15:34:18.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:34:18.591860+08:00 2024-11-04 15:34:18.771 INFO 21062 --- [ocp-async-8,165799cab3046073,7bdac765d9b7e4c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:34:19.353 INFO 21062 --- [ocp-schedules-20,71838ddd3083ca14,845cf3634f8e10cc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:34:19.357 INFO 21062 --- [ocp-schedules-20,71838ddd3083ca14,845cf3634f8e10cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:19.358 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:34:19.621 INFO 21062 --- [ocp-async-8,8b0242e38db53c61,0944c995c9c413b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:19.621 WARN 21062 --- [ocp-async-4,53d0fc3267f2e427,8542c19862aa16de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:20.983 INFO 21062 --- [prometheus-minute-schedule-1,04f8ebe40d315171,f4a8c25c822ada53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:21.258 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:21.259 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:21.621 WARN 21062 --- [ocp-async-1,4a1261a40223acee,87dfee54ff8bbd67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:21.621 INFO 21062 --- [ocp-async-5,2983420b122fa39d,5cae90093275ae7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:21.823 INFO 21062 --- [ocp-schedules-18,2827b9e1ade546b8,2cfe578b89ae3997] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:34:23.621 INFO 21062 --- [ocp-async-9,59a9e1edfd9b024b,d854a75764e6bdec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:23.621 WARN 21062 --- [ocp-async-10,127f1f511f4195fe,abb781b7c2b32b66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:23.983 INFO 21062 --- [prometheus-minute-schedule-1,da3c16cbbb83715b,be47a38417445014] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:24.260 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:24.261 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:25.621 INFO 21062 --- [ocp-async-4,cf68f3e49c9b2e2e,5a9700d7ad9b7d89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:25.621 WARN 21062 --- [ocp-async-6,b008fe231f117043,ffe04f0275d4c30f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:26.983 INFO 21062 --- [prometheus-minute-schedule-1,b1dab1fdf6c6aa34,24d4652b6c320fc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:26.984 INFO 21062 --- [prometheus-hour-schedule-1,c1213f8f9bf3a674,3811f00fbc79edb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:34:27.261 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:27.262 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:27.621 WARN 21062 --- [ocp-async-8,d81697859b48fd5f,4515cad01e9741c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:27.621 INFO 21062 --- [ocp-async-1,7a9158faf4db18a6,098bd422b21320d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:28.413 INFO 21062 --- [ocp-schedules-19,8f3ad2c40682a9a0,4e50c34f1ea80e78] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:34:28.498 INFO 21062 --- [ocp-schedules-25,54e65121ce57e35d,afd66d5d79745b67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:34:28.499 INFO 21062 --- [ocp-schedules-16,588d4223acf1a8c3,03923c60a9029340] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:34:28.500 INFO 21062 --- [ocp-schedules-26,5f2732014b6672db,7a28ba06bacbc64f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:34:28.500 INFO 21062 --- [ocp-schedules-27,649ee2b298e22d96,7bfacc94a14c8a5c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:34:28.501 INFO 21062 --- [ocp-schedules-25,54e65121ce57e35d,afd66d5d79745b67] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:34:28.502 INFO 21062 --- [ocp-schedules-28,bc1b13a3ce35ffbd,d5fd273984c0b49b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:34:28.503 INFO 21062 --- [ocp-schedules-27,649ee2b298e22d96,7bfacc94a14c8a5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:28.503 INFO 21062 --- [ocp-schedules-26,5f2732014b6672db,7a28ba06bacbc64f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:34:28.504 INFO 21062 --- [ocp-schedules-14,c1789f905b5e6056,ccb4d26b9b1d7ec6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:34:28.504 INFO 21062 --- [ocp-schedules-16,588d4223acf1a8c3,03923c60a9029340] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:34:28.509 INFO 21062 --- [ocp-schedules-22,305b6793b145c04c,aaaddd5414c861bc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:34:28.509 INFO 21062 --- [ocp-schedules-10,425f8258bd8bc104,036b76cfb4daf22c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:34:28.509 INFO 21062 --- [ocp-schedules-2,dcd5c5317c289e1c,40e4aa8a44e8129d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:34:28.509 INFO 21062 --- [ocp-schedules-22,305b6793b145c04c,aaaddd5414c861bc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:34:28.509 INFO 21062 --- [ocp-schedules-10,425f8258bd8bc104,036b76cfb4daf22c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:34:28.509 INFO 21062 --- [ocp-schedules-2,dcd5c5317c289e1c,40e4aa8a44e8129d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:34:28.590 INFO 21062 --- [ocp-schedules-21,0e791d069cc4554d,1fb71792e397f37f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:34:28.590 INFO 21062 --- [ocp-schedules-21,0e791d069cc4554d,1fb71792e397f37f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:34:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:34:28.589916+08:00, lastProcessTime=2024-11-04T15:34:18.591860+08:00 2024-11-04 15:34:28.625 INFO 21062 --- [ocp-schedules-1,bd2f00c0fd47b9ea,e344a76ecd34d848] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:34:28.625+08:00, tenantSize=0 2024-11-04 15:34:28.735 INFO 21062 --- [ocp-schedules-18,45b529d35c7bdf72,532f0e7283958067] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 15:34:28.735 INFO 21062 --- [ocp-schedules-18,45b529d35c7bdf72,532f0e7283958067] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 15:34:28.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:34:28.589916+08:00 2024-11-04 15:34:28.771 INFO 21062 --- [ocp-async-6,7198f1b77d2a8787,0dc7a82fbabff226] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:34:28.967 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:34:28.967 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:34:28.977 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:34:28.977 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:34:28.977 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:34:28.977 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:34:28.983 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:34:28.983 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:34:28.983 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:34:28.983 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:34:28.997 INFO 21062 --- [ocp-schedules-28,badf3bb3585ba1fb,09e9a4f13b23325f] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:34:28.997 INFO 21062 --- [ocp-schedules-28,badf3bb3585ba1fb,09e9a4f13b23325f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:34:29.012 INFO 21062 --- [ocp-schedules-2,f95000ea65c27ec0,397dc6f26f756023] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:34:29.021 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:34:29.030 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:34:29.030 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:34:29.030 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:34:29.030 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:34:29.030 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:34:29.038 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:34:29.358 INFO 21062 --- [ocp-schedules-20,71838ddd3083ca14,845cf3634f8e10cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:34:29.429 INFO 21062 --- [ocp-schedules-3,f7b84093f4ebd878,e96ce566e9828626] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:34:29.429 INFO 21062 --- [ocp-schedules-3,f7b84093f4ebd878,e96ce566e9828626] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:34:29.567 INFO 21062 --- [ocp-schedules-23,9e7c761d24d2c269,13b0226d3e3b20c5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:34:29.588 INFO 21062 --- [ocp-schedules-9,af491a6a276f2534,82368d6287b78d25] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:34:29.593 INFO 21062 --- [ocp-schedules-9,af491a6a276f2534,82368d6287b78d25] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:34:29.593 INFO 21062 --- [ocp-schedules-9,af491a6a276f2534,82368d6287b78d25] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:34:29.604 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:34:29.622 INFO 21062 --- [ocp-async-6,f77476508ba0394b,96e1a24fa123fc39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:29.622 WARN 21062 --- [ocp-async-7,669ec8314d3d15c8,e066f89168db96e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:29.906 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:34:29.906 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:34:29.915 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:34:29.915 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:34:29.915 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:34:29.926 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:34:29.926 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:34:29.926 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:34:29.929 INFO 21062 --- [ocp-schedules-16,f37743494963393f,931fd21b6b3561ab] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:34:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b98bf4d93e5e6402,36d8fa464cd6aa5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:30.074 INFO 21062 --- [ocp-schedules-22,191a732805bd4454,a000d505f0f25a2b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:34:30.074 INFO 21062 --- [ocp-schedules-22,191a732805bd4454,a000d505f0f25a2b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:34:30.076 INFO 21062 --- [ocp-schedules-22,191a732805bd4454,a000d505f0f25a2b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:34:30.076 INFO 21062 --- [ocp-schedules-22,191a732805bd4454,a000d505f0f25a2b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:34:30.076 INFO 21062 --- [ocp-schedules-22,191a732805bd4454,a000d505f0f25a2b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:34:30.077 INFO 21062 --- [ocp-schedules-22,191a732805bd4454,a000d505f0f25a2b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:34:30.163 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.179837075S 2024-11-04 15:34:30.163 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:34:30.261 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:34:30.262 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:34:30.262 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:30.263 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:34:30.289 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:34:30.356 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:34:30.356 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:34:30.356 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:34:30.356 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:34:30.356 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:34:30.356 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:34:30.366 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:34:30.545 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] o.s.boot.SpringApplication : Started application in 1.568 seconds (JVM running for 11084.224) 2024-11-04 15:34:30.545 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.568732671S 2024-11-04 15:34:30.997 INFO 21062 --- [ocp-schedules-10,88e5c1c487ce9904,368875de66caaae9] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:34:31.622 WARN 21062 --- [ocp-async-8,b1db3efaa394e45c,b7984d6d585c03dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:31.622 INFO 21062 --- [ocp-async-8,eddf44252fe77c78,5e659febbc4a1741] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:31.827 INFO 21062 --- [ocp-schedules-24,09853915592c91a4,305a309e39ac6ecb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:34:32.983 INFO 21062 --- [prometheus-minute-schedule-1,6293a4f74ccb0c98,b88dc4abbd9eaf5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:33.247 INFO 21062 --- [ocp-schedules-10,ccbbb300583eb69a,7aaa803dfa955def] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:34:33.252 INFO 21062 --- [ocp-schedules-10,ccbbb300583eb69a,7aaa803dfa955def] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:34:33.252 INFO 21062 --- [ocp-schedules-10,ccbbb300583eb69a,7aaa803dfa955def] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:34:33.263 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:33.264 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:33.508 INFO 21062 --- [ocp-schedules-13,a33d022077188ee8,c116c220e45bc05a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:34:33.508 INFO 21062 --- [ocp-schedules-13,a33d022077188ee8,c116c220e45bc05a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:34:33.535 INFO 21062 --- [ocp-schedules-13,a33d022077188ee8,c116c220e45bc05a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:34:33.535 INFO 21062 --- [ocp-schedules-13,a33d022077188ee8,c116c220e45bc05a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:34:33.536 INFO 21062 --- [ocp-schedules-14,f40959fc5e8307a5,7ef8047c6e4197f0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:34:33.622 WARN 21062 --- [ocp-async-1,5e752d01bf189ffc,05c5587976c61e27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:33.622 INFO 21062 --- [ocp-async-10,e6c2204a11a3edfc,9e4e3e8bc5c76b0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:33.744 INFO 21062 --- [ocp-schedules-11,ee022c2131f35ac9,e0e3ff0d7e57c134] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:34:33.744+08:00, clusters=[1] 2024-11-04 15:34:33.746 INFO 21062 --- [ocp-schedules-11,ee022c2131f35ac9,e0e3ff0d7e57c134] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:34:34.080 INFO 21062 --- [ocp-schedules-17,49899e4d4f1b35be,f637888ba35ce05b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:34:34.080+08:00, clusters=[1] 2024-11-04 15:34:34.083 INFO 21062 --- [ocp-schedules-17,49899e4d4f1b35be,f637888ba35ce05b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:34:34.117 INFO 21062 --- [ocp-schedules-23,30f6fbae4b6ec513,e88df22a2ef38a93] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:34:34.121 INFO 21062 --- [ocp-schedules-23,30f6fbae4b6ec513,e88df22a2ef38a93] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:34.179 ERROR 21062 --- [ocp-schedules-23,30f6fbae4b6ec513,e88df22a2ef38a93] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:34:34.180 INFO 21062 --- [ocp-schedules-23,30f6fbae4b6ec513,e88df22a2ef38a93] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:34:35.622 WARN 21062 --- [ocp-async-7,1c0b20386718b69b,d7a2905dc1f721b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:35.622 INFO 21062 --- [ocp-async-9,874b3af3631ceaad,9facc69d4f4719e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:35.983 INFO 21062 --- [prometheus-minute-schedule-1,ca7b190b2abe840e,c3aa8b8586f30a50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:36.017 INFO 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:34:36.018 INFO 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:34:36.018 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:34:36.018 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:34:36.027 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:34:36.027 INFO 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:34:36.028 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:34:36.029 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:34:36.029 WARN 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:34:36.029 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:34:36.029 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:34:36.029 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:34:36.030 INFO 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:34:36.030 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:34:36.031 WARN 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:34:36.031 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:34:36.038 INFO 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:34:36.038 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:34:36.038 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:34:36.049 ERROR 21062 --- [ocp-schedules-19,2664d60de2cd6202,13de97cd80e7a693] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:34:36.264 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:36.265 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:37.622 WARN 21062 --- [ocp-async-3,2d8608acf922410c,23ed58be8dd744bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:37.622 INFO 21062 --- [ocp-async-4,39cbc6112e8c15e7,21796d66694e94a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:38.455 INFO 21062 --- [ocp-schedules-17,5f8324ed2094ee47,38e160e626edcb1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:34:38.504 INFO 21062 --- [ocp-schedules-27,649ee2b298e22d96,7bfacc94a14c8a5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:34:38.504 INFO 21062 --- [ocp-schedules-26,5f2732014b6672db,7a28ba06bacbc64f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:34:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:34:38.590323+08:00, lastProcessTime=2024-11-04T15:34:28.589916+08:00 2024-11-04 15:34:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:34:38.590323+08:00 2024-11-04 15:34:38.772 INFO 21062 --- [ocp-async-10,0ff44e9a4dc26cac,402f79fb78998097] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:34:38.983 INFO 21062 --- [prometheus-minute-schedule-1,bc70f560ba42b4d7,0f1a943bd8b08559] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:39.265 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:39.266 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:39.622 WARN 21062 --- [ocp-async-10,616e9f05a35148eb,bd142add1f773448] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:39.622 INFO 21062 --- [ocp-async-7,e8c6ce9a5ee0a63a,9364956e243b454b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:40.002 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:34:40.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:34:40.947 INFO 21062 --- [ocp-schedules-12,e3fd2b169424631a,f06539004b6933ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:34:40.947 INFO 21062 --- [ocp-schedules-12,b8ef82c3c635b764,d76f7d01c9e085da] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:34:40.952 INFO 21062 --- [ocp-schedules-12,b8ef82c3c635b764,d76f7d01c9e085da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:40.982 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:34:40.982 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:34:41.623 WARN 21062 --- [ocp-async-9,67651fb8124e9d00,b68c94f0dc42f998] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:41.623 INFO 21062 --- [ocp-async-3,79693d3495f3c9af,1cfba7038180a64f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:34:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:34:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:34:41.830 INFO 21062 --- [ocp-schedules-17,0ce64d49a59be241,8a2617a9d35cab08] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:34:41.852 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:34:41.852 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=25ms, period=120000ms 2024-11-04 15:34:41.983 INFO 21062 --- [prometheus-minute-schedule-1,271c5469af9a6ce2,fbcd02a1fbdd1bcd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:42.266 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:42.267 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:43.623 WARN 21062 --- [ocp-async-4,7a70d42d8e32b785,19199e56cf9f77df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:43.623 INFO 21062 --- [ocp-async-8,6e9a4c4e971e6f17,9477946477322038] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:44.983 INFO 21062 --- [prometheus-minute-schedule-1,80e182cd8be8f9c9,d17e2a7a5bda9e91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:45.267 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:45.268 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:45.279 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:34:45.279 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:34:45.280 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.281 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:34:45.282 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.282 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.283 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.283 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:34:45.298 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.298 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.299 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.299 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:34:45.313 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.313 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.314 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.314 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:34:45.328 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.329 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.330 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.330 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:34:45.337 INFO 21062 --- [ocp-schedules-20,d3aa89f12d0e6a2c,d75c2a12fe9a3eea] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:34:45.341 INFO 21062 --- [ocp-schedules-20,d3aa89f12d0e6a2c,d75c2a12fe9a3eea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:45.343 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.343 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.344 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.344 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:34:45.350 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.351 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.351 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.352 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:34:45.356 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.357 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.357 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.357 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:34:45.362 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.362 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.363 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.363 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:34:45.370 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.370 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.371 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.371 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:34:45.377 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.377 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.378 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.378 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:34:45.383 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.384 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.384 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.385 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:34:45.393 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.393 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.394 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.394 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:34:45.408 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.408 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.409 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.409 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:34:45.413 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.414 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.414 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.415 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:34:45.420 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.421 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.421 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.422 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:34:45.428 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.428 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.429 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:34:45.429 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.429 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:34:45.432 ERROR 21062 --- [ocp-schedules-20,d3aa89f12d0e6a2c,d75c2a12fe9a3eea] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:34:45.432 INFO 21062 --- [ocp-schedules-20,d3aa89f12d0e6a2c,d75c2a12fe9a3eea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:34:45.434 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.434 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.434 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.435 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:34:45.439 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.440 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.440 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.440 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:34:45.445 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.445 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.446 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.446 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:34:45.451 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.452 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.452 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.453 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:34:45.456 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.457 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.458 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.458 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:34:45.462 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.463 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.463 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.464 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:34:45.468 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.468 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.469 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.469 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:34:45.474 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.474 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.475 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.475 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:34:45.479 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.480 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.480 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.481 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:34:45.485 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.485 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.486 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.486 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:34:45.491 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.491 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.492 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.492 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:34:45.496 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.497 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.497 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.497 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:34:45.502 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.502 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.502 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.503 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:34:45.507 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.507 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.508 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.508 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:34:45.512 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.512 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.513 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.513 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:34:45.517 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.517 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.518 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.518 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:34:45.522 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.523 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.523 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.524 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:34:45.528 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.528 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.529 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.529 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:34:45.536 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.537 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.537 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.538 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:34:45.544 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.544 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.545 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.545 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:34:45.550 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.551 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.551 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.552 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:34:45.558 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.558 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.558 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.559 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:34:45.562 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.562 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.563 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.563 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:34:45.567 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.567 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.568 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.568 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:34:45.571 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.572 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.572 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.573 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:34:45.576 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.576 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.577 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.577 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:34:45.585 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.586 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.586 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.587 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:34:45.606 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.606 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.607 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.608 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:34:45.613 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.613 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.614 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.614 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:34:45.620 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.620 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.620 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.621 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:34:45.623 WARN 21062 --- [ocp-async-10,22f587ca68af42f4,7c070a6892bfa8e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:45.623 INFO 21062 --- [ocp-async-5,37957c02cdc5a0ed,128bd47efe823c76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:45.626 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.626 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.627 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.627 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:34:45.631 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.632 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.632 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.632 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:34:45.636 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.636 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.637 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.637 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:34:45.641 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:34:45.641 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:34:45.642 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.643 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:34:45.644 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.644 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.645 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.645 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:34:45.650 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.651 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:34:45.651 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:34:45.651 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:34:45.656 INFO 21062 --- [ocp-schedules-1,7e7f2415a7245b47,add546d3634b7ea1] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:34:47.623 WARN 21062 --- [ocp-async-9,95e22dc8d9d87610,ebfc6675e36e92e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:47.623 INFO 21062 --- [ocp-async-2,2a52e1e798cff1c6,35f388783d283a22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e39745973235d8cc,a691045bb0087182] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:48.269 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:48.270 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:48.482 INFO 21062 --- [ocp-schedules-13,1e4335418efd6e86,4487566cfb2943a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:34:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:34:48.589858+08:00, lastProcessTime=2024-11-04T15:34:38.590323+08:00 2024-11-04 15:34:48.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:34:48.589858+08:00 2024-11-04 15:34:48.772 INFO 21062 --- [ocp-async-4,4bb8a2682e73b02b,58b9420183e7db62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:34:49.623 WARN 21062 --- [ocp-async-4,a3fe9d378f689842,f08197b4adecdbc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:49.623 INFO 21062 --- [ocp-async-4,780b9d51bbe8af06,e136efcff437d765] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:50.983 INFO 21062 --- [prometheus-minute-schedule-1,3c4b746b6442eea5,e2338904d71f3749] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:51.270 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:51.271 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:51.623 WARN 21062 --- [ocp-async-1,9c9526d9f5f9ca1a,6f80434823d9c0b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:51.624 INFO 21062 --- [ocp-async-5,be3b6f2182935a48,8c7b258eb340c559] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:51.832 INFO 21062 --- [ocp-schedules-1,e5f5b25f8f77aa01,b9bc0d0f506eaec0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:34:53.255 INFO 21062 --- [ocp-schedules-18,d34a41ea47cbfd92,9afc5b3fd65a5a27] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:34:53.261 INFO 21062 --- [ocp-schedules-18,d34a41ea47cbfd92,9afc5b3fd65a5a27] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:34:53.261 INFO 21062 --- [ocp-schedules-18,d34a41ea47cbfd92,9afc5b3fd65a5a27] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:34:53.624 WARN 21062 --- [ocp-async-7,25ddd6735b8aa8fa,b188f6b8e7b4abd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:53.624 INFO 21062 --- [ocp-async-2,7510674081544a77,a29c4f7714b1b52b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:53.983 INFO 21062 --- [prometheus-minute-schedule-1,01e35d79a568d884,7f82c3a646de4e56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:54.271 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:54.272 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:55.624 WARN 21062 --- [ocp-async-8,c4eff525cab6e7b7,0e564b9237e9d040] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:55.624 INFO 21062 --- [ocp-async-6,64f151b5f01cbef2,34d9a56224ec33ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7bf8753a19cd9048,52f1fecddc86e516] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:34:57.272 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:34:57.273 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:34:57.624 WARN 21062 --- [ocp-async-1,0e5aee4372f8403b,b4beaa3606caac56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:57.624 INFO 21062 --- [ocp-async-1,cddb628dcc5a9d49,4f4fe97b89c0dce9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:58.500 INFO 21062 --- [ocp-schedules-24,ab55ebd15a1b0066,3b93bd35c407aadf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:34:58.500 INFO 21062 --- [ocp-schedules-8,2e8ec5352b8b9477,01231a0297213f3f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:34:58.502 INFO 21062 --- [ocp-schedules-21,032ed6860ffac8cb,d6799cf981c64f73] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:34:58.503 INFO 21062 --- [ocp-schedules-24,ab55ebd15a1b0066,3b93bd35c407aadf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:34:58.504 INFO 21062 --- [ocp-schedules-2,07fc82dccd6fc571,0469ab479463856f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:34:58.505 INFO 21062 --- [ocp-schedules-8,2e8ec5352b8b9477,01231a0297213f3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:34:58.509 INFO 21062 --- [ocp-schedules-30,82d6ad5c196e5d7a,68d33cb4a9ad9d7b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:34:58.509 INFO 21062 --- [ocp-schedules-6,fc298adf73215114,448981e80694da2e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:34:58.509 INFO 21062 --- [ocp-schedules-30,82d6ad5c196e5d7a,68d33cb4a9ad9d7b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:34:58.509 INFO 21062 --- [ocp-schedules-6,fc298adf73215114,448981e80694da2e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:34:58.510 INFO 21062 --- [ocp-schedules-7,3536b1e50a10ae62,f2b60f1c642cddcf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:34:58.510 INFO 21062 --- [ocp-schedules-7,3536b1e50a10ae62,f2b60f1c642cddcf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:34:58.528 INFO 21062 --- [ocp-schedules-11,4b0b91f68ce34402,96e2fdbacc408dad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:34:58.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:34:58.588022+08:00, lastProcessTime=2024-11-04T15:34:48.589858+08:00 2024-11-04 15:34:58.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:34:58.588022+08:00 2024-11-04 15:34:58.774 INFO 21062 --- [ocp-async-7,353cab736686e0a4,1d938b1baa40f391] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:34:59.570 INFO 21062 --- [ocp-schedules-19,9b4a9697eac018de,c421380816d8faca] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:34:59.624 WARN 21062 --- [ocp-async-9,4299d8edaccabdfa,36a9887d9356268f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:34:59.624 INFO 21062 --- [ocp-async-2,99d398708fa03121,715ed5f692bc12b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:34:59.983 INFO 21062 --- [prometheus-minute-schedule-1,890bb0a26be8687a,e5c74b9235f039e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705700, interval=60 2024-11-04 15:35:00.151 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecaac18a6620cfc3] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:30:00.0, nextRun=2024-11-04 15:35:00.0, currentDate=Mon Nov 04 15:35:00 CST 2024 2024-11-04 15:35:00.151 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecaac18a6620cfc3] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:30:00.0, nextRunTime=2024-11-04 15:35:00.0} 2024-11-04 15:35:00.177 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecaac18a6620cfc3] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:35:00.182 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecaac18a6620cfc3] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:35:00.182 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,ecaac18a6620cfc3] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:35:00.274 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:00.274 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:01.624 WARN 21062 --- [ocp-async-3,2fcb01d54300e14e,dd91fede32ef5a09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:01.624 INFO 21062 --- [ocp-async-6,6a5a50b29a48d944,a7b5252e69b32e3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:01.835 INFO 21062 --- [ocp-schedules-23,b223b02db9e9c441,62415c4a12c2afc9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:35:02.983 INFO 21062 --- [prometheus-minute-schedule-1,64dbc658ede1711c,2cd7c766857cff01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:03.275 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:03.276 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:03.625 WARN 21062 --- [ocp-async-4,8a0b417281768eaa,b459026febde15f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:03.625 INFO 21062 --- [ocp-async-4,3475553c9bea0b91,4f34988b4ddbc895] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:05.625 WARN 21062 --- [ocp-async-9,6482f0c7b9e4dc12,8ac30275ab3c3d35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:05.625 INFO 21062 --- [ocp-async-7,d55d6599bd41688b,c5a5c62fe55a3744] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c449472113fca386,61aaec726ae23e8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:06.276 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:06.278 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:07.625 WARN 21062 --- [ocp-async-3,ad1e491ef1007b69,74713f746f8e6f73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:07.625 INFO 21062 --- [ocp-async-8,a306c4bd4f5f8e06,b8faac28fe6f8d9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:08.504 INFO 21062 --- [ocp-schedules-24,ab55ebd15a1b0066,3b93bd35c407aadf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:35:08.506 INFO 21062 --- [ocp-schedules-8,2e8ec5352b8b9477,01231a0297213f3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:35:08.554 INFO 21062 --- [ocp-schedules-26,f5c0d60174479e5b,7e74f1e47459a0da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:35:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:35:08.588908+08:00, lastProcessTime=2024-11-04T15:34:58.588022+08:00 2024-11-04 15:35:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:35:08.588908+08:00 2024-11-04 15:35:08.774 INFO 21062 --- [ocp-async-1,2923d4a7eb5092a2,b20fc3033b2679bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:35:08.983 INFO 21062 --- [prometheus-minute-schedule-1,1975c4bc942d986c,22bccb6682255c23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:09.278 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:09.278 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:09.625 WARN 21062 --- [ocp-async-4,083bf78b97b872ec,5e8051f07eef5d1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:09.625 INFO 21062 --- [ocp-async-10,4b16265e91939de4,ed4a7d189068df04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:10.003 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:35:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:35:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:35:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:35:10.307 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:35:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:35:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:35:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:35:10.983 INFO 21062 --- [ocp-schedules-12,b8ef82c3c635b764,d76f7d01c9e085da] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:35:10.983 INFO 21062 --- [ocp-schedules-12,fa02951d5ec5d9e3,aa60378963d58852] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:35:10.986 INFO 21062 --- [ocp-schedules-12,fa02951d5ec5d9e3,aa60378963d58852] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:11.014 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:35:11.014 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:35:11.625 INFO 21062 --- [ocp-async-5,01e74fdbbfe875f5,08469e452f5278e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:11.625 WARN 21062 --- [ocp-async-7,05d7a2cfe9b57dfb,0ad92c942fdd542d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:11.838 INFO 21062 --- [ocp-schedules-29,ee76aeff193a0f12,1cec45849cbdca70] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:35:11.870 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=1, affectRows=1 2024-11-04 15:35:11.870 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=1, validCount=1 2024-11-04 15:35:11.873 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=14, affectRows=14 2024-11-04 15:35:11.873 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=14, validCount=14 2024-11-04 15:35:11.873 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=3, affectRows=3 2024-11-04 15:35:11.873 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=3, validCount=3 2024-11-04 15:35:11.873 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=24, affectRows=24 2024-11-04 15:35:11.873 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=24, validCount=24 2024-11-04 15:35:11.873 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=33, affectRows=33 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=33, validCount=33 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=57, affectRows=57 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=30, affectRows=30 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=47, affectRows=47 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=59, affectRows=59 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=30, validCount=30 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=56, validCount=56 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=47, validCount=47 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=76, affectRows=76 2024-11-04 15:35:11.874 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=59, validCount=59 2024-11-04 15:35:11.875 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705640, nodeCount=78, affectRows=78 2024-11-04 15:35:11.875 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=76, validCount=76 2024-11-04 15:35:11.875 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705640, nodeCount=76, validCount=76 2024-11-04 15:35:11.983 INFO 21062 --- [prometheus-minute-schedule-1,ccbc534b7dc7f36e,2821a15f0ef10c80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:12.279 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:12.280 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:13.264 INFO 21062 --- [ocp-schedules-26,a63f5b836e8a5566,fbf90fc50b81f58f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:35:13.269 INFO 21062 --- [ocp-schedules-26,a63f5b836e8a5566,fbf90fc50b81f58f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:35:13.269 INFO 21062 --- [ocp-schedules-26,a63f5b836e8a5566,fbf90fc50b81f58f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:35:13.625 INFO 21062 --- [ocp-async-3,4144e9dc997e7262,fdddfb87fd62e8fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:13.625 WARN 21062 --- [ocp-async-2,d5e32edbe8875339,d521fd57be1baeea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:13.984 INFO 21062 --- [prometheus-cache-monitor1,122c83598acdac55,bd5ed82b2f64ab59] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:35:13.984 INFO 21062 --- [prometheus-cache-monitor1,122c83598acdac55,bd5ed82b2f64ab59] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:35:14.983 INFO 21062 --- [prometheus-minute-schedule-1,36e7d8d10b8d17a3,bb3dcf3a140dc42c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:15.280 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:15.281 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:15.625 INFO 21062 --- [ocp-async-6,616085f911250bee,d2d54391d684c3ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:15.625 WARN 21062 --- [ocp-async-4,b06aa51dade36655,8cf8edf81ad09770] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:15.706 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:31:15.689, end=2024-11-04T15:33:15.703} 2024-11-04 15:35:15.717 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:35:15.717 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:35:17.626 INFO 21062 --- [ocp-async-10,18d8640e782e1fbf,06de4d71b177cb5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:17.626 WARN 21062 --- [ocp-async-7,d342b51cb3c6f890,f0042d4a427803f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7f20f8519b5da554,1d012546e3d379f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:18.281 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:18.283 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:35:18.590683+08:00, lastProcessTime=2024-11-04T15:35:08.588908+08:00 2024-11-04 15:35:18.594 INFO 21062 --- [ocp-schedules-15,998dbda17590b8a2,8962d8f652f2715a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:35:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:35:18.590683+08:00 2024-11-04 15:35:18.774 INFO 21062 --- [ocp-async-8,93bba9c3b7e08de2,fe8db159682e718e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:35:19.626 WARN 21062 --- [ocp-async-5,eab246733a736d93,b6f326a5629180be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:19.626 INFO 21062 --- [ocp-async-8,dde2f17b0d93698b,49c38ef49cedfe5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ed0886a17af63c9d,fed88ac2312902aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:21.283 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:21.284 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:21.626 INFO 21062 --- [ocp-async-3,725c48a815ed26a8,36db444ba56d7782] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:21.626 WARN 21062 --- [ocp-async-1,1d612dc9de8f0bb3,5b548d371d76d49c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:21.841 INFO 21062 --- [ocp-schedules-5,7ce2f5212a259892,f10beb6017436845] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:35:23.626 INFO 21062 --- [ocp-async-6,bc9285908569dd95,00f76000eec34cda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:23.626 WARN 21062 --- [ocp-async-9,a3552b8bfeb4b01d,41e0d683820aacf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:23.983 INFO 21062 --- [prometheus-minute-schedule-1,cc2ec459abd46901,ee8b40bbf72a2a77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:24.284 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:24.285 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:25.626 WARN 21062 --- [ocp-async-2,5d538d5ab3043cfa,6e51b1c2e083afb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:25.626 INFO 21062 --- [ocp-async-5,99eaf05fb5f14400,c9730e2122744152] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ec62f80992861ba6,a916785ae42fc0f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:26.984 INFO 21062 --- [prometheus-hour-schedule-1,0f9ffd6adcf00335,7a5e6496917920f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:35:27.285 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:27.286 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:27.627 WARN 21062 --- [ocp-async-8,0486ae525e9b7906,fa24ff79a335ae53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:27.627 INFO 21062 --- [ocp-async-1,fc39a5d4939a8528,0560769154a6bdbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:28.498 INFO 21062 --- [ocp-schedules-20,fb088d862b821c2b,156fe72b631a6806] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:35:28.500 INFO 21062 --- [ocp-schedules-8,0040152b678cc9b3,a4aa555a94ba67a2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:35:28.500 INFO 21062 --- [ocp-schedules-11,bca6ff2ae02be3d7,ae4e4610a70d9551] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:35:28.500 INFO 21062 --- [ocp-schedules-18,18ae34eaa55f3ff9,e4a4361628bc0210] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:35:28.502 INFO 21062 --- [ocp-schedules-23,d716cdd1df4ae97e,1f7815c77d5c809d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:35:28.503 INFO 21062 --- [ocp-schedules-20,fb088d862b821c2b,156fe72b631a6806] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:35:28.504 INFO 21062 --- [ocp-schedules-8,0040152b678cc9b3,a4aa555a94ba67a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:28.504 INFO 21062 --- [ocp-schedules-18,18ae34eaa55f3ff9,e4a4361628bc0210] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:35:28.505 INFO 21062 --- [ocp-schedules-24,56f7055ea999f3a9,f67fd7ec68859214] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:35:28.506 INFO 21062 --- [ocp-schedules-11,bca6ff2ae02be3d7,ae4e4610a70d9551] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:35:28.510 INFO 21062 --- [ocp-schedules-15,74da69ec2b4609e6,7017dfd97c03da56] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:35:28.510 INFO 21062 --- [ocp-schedules-15,74da69ec2b4609e6,7017dfd97c03da56] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:35:28.511 INFO 21062 --- [ocp-schedules-14,cde9b201ded52550,04ec9cd4cad99ed8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:35:28.511 INFO 21062 --- [ocp-schedules-14,cde9b201ded52550,04ec9cd4cad99ed8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:35:28.511 INFO 21062 --- [ocp-schedules-9,1b9d8626d01dc0d5,89bbf6d67dff438e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:35:28.511 INFO 21062 --- [ocp-schedules-9,1b9d8626d01dc0d5,89bbf6d67dff438e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:35:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:35:28.589080+08:00, lastProcessTime=2024-11-04T15:35:18.590683+08:00 2024-11-04 15:35:28.619 INFO 21062 --- [ocp-schedules-5,3e1a1b3cf94ff8a2,7a68f0d6d5844820] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:35:28.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:35:28.589080+08:00 2024-11-04 15:35:28.774 INFO 21062 --- [ocp-async-2,13e43081d72944b3,c7a9971cd22c6d90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:35:29.015 INFO 21062 --- [ocp-schedules-22,06040307dc732bef,9e6917b4df974c2e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:35:29.359 INFO 21062 --- [ocp-schedules-26,25558afcc8faae48,9ae773d2a4a44087] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:35:29.363 INFO 21062 --- [ocp-schedules-26,25558afcc8faae48,9ae773d2a4a44087] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:29.364 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:35:29.434 INFO 21062 --- [ocp-schedules-2,02da0036a40c2902,9a4156bbc381f560] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:35:29.434 INFO 21062 --- [ocp-schedules-2,02da0036a40c2902,9a4156bbc381f560] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:35:29.573 INFO 21062 --- [ocp-schedules-4,924c22128d51e07e,6ee9d8e68578edda] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:35:29.593 INFO 21062 --- [ocp-schedules-29,60bc0c0e78fec25c,5e413b2e6cb22834] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:35:29.598 INFO 21062 --- [ocp-schedules-29,60bc0c0e78fec25c,5e413b2e6cb22834] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:35:29.598 INFO 21062 --- [ocp-schedules-29,60bc0c0e78fec25c,5e413b2e6cb22834] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:35:29.627 WARN 21062 --- [ocp-async-10,c561194c957ff7ed,0d14564aa481a97d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:29.627 INFO 21062 --- [ocp-async-2,48e9b61c78ef3521,07400e79ede559b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3a28d16d8871eb6c,cf2a960dc3a5894e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:30.079 INFO 21062 --- [ocp-schedules-3,d5b9a89fe5ca633f,7edcf35286332d50] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:35:30.079 INFO 21062 --- [ocp-schedules-3,d5b9a89fe5ca633f,7edcf35286332d50] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:35:30.087 INFO 21062 --- [ocp-schedules-3,d5b9a89fe5ca633f,7edcf35286332d50] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:35:30.087 INFO 21062 --- [ocp-schedules-3,d5b9a89fe5ca633f,7edcf35286332d50] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:35:30.092 INFO 21062 --- [ocp-schedules-3,d5b9a89fe5ca633f,7edcf35286332d50] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:35:30.092 INFO 21062 --- [ocp-schedules-3,d5b9a89fe5ca633f,7edcf35286332d50] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:35:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:35:30.286 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:30.287 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:35:31.627 INFO 21062 --- [ocp-async-8,2158419d9cb1ea53,f2558caa9b3855f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:31.627 WARN 21062 --- [ocp-async-4,5b58d8eaf02a7587,f7a37ea9303b06eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:31.844 INFO 21062 --- [ocp-schedules-28,250051aaaf47f8a1,c2f157aa1ae03786] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:35:32.983 INFO 21062 --- [prometheus-minute-schedule-1,0f16d1075d66bf80,27539ad9e8d752b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:33.271 INFO 21062 --- [ocp-schedules-9,c807675967d4bdcc,8403058aab34b392] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:35:33.276 INFO 21062 --- [ocp-schedules-9,c807675967d4bdcc,8403058aab34b392] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:35:33.276 INFO 21062 --- [ocp-schedules-9,c807675967d4bdcc,8403058aab34b392] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:35:33.287 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:33.289 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:33.535 INFO 21062 --- [ocp-schedules-24,4f968438a6b7a0a9,bdec00f35feebd33] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:35:33.535 INFO 21062 --- [ocp-schedules-24,4f968438a6b7a0a9,bdec00f35feebd33] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:35:33.563 INFO 21062 --- [ocp-schedules-24,4f968438a6b7a0a9,bdec00f35feebd33] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:35:33.563 INFO 21062 --- [ocp-schedules-24,4f968438a6b7a0a9,bdec00f35feebd33] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:35:33.564 INFO 21062 --- [ocp-schedules-3,242cde42c9142448,6018fa72fa172514] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:35:33.627 INFO 21062 --- [ocp-async-1,3050e796edd38e87,124fcc428ad3be39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:33.627 WARN 21062 --- [ocp-async-9,ce09c507b8d76007,e3cd6dcbfe245f01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:33.772 INFO 21062 --- [ocp-schedules-15,8d9878c7be0a6acb,5888323cdbb80887] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:35:33.772+08:00, clusters=[1] 2024-11-04 15:35:33.773 INFO 21062 --- [ocp-schedules-15,8d9878c7be0a6acb,5888323cdbb80887] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:35:34.109 INFO 21062 --- [ocp-schedules-16,c04639b0dec0a155,7c873316f3babdc6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:35:34.109+08:00, clusters=[1] 2024-11-04 15:35:34.113 INFO 21062 --- [ocp-schedules-16,c04639b0dec0a155,7c873316f3babdc6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:35:35.627 WARN 21062 --- [ocp-async-10,fb1872739ca6a66b,8666950086df396a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:35.627 INFO 21062 --- [ocp-async-6,d71db80cea4a2932,2d2cdf44fa41f054] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:35.983 INFO 21062 --- [prometheus-minute-schedule-1,3601e10541c41059,106a3d62e62ffdb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:36.058 INFO 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:35:36.058 INFO 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:35:36.058 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:35:36.058 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:35:36.069 INFO 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:35:36.069 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:35:36.069 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:35:36.071 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:35:36.071 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:35:36.071 WARN 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:35:36.072 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:35:36.072 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:35:36.072 INFO 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:35:36.073 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:35:36.073 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:35:36.073 WARN 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:35:36.081 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:35:36.081 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:35:36.081 INFO 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:35:36.095 ERROR 21062 --- [ocp-schedules-19,330deedf3dff41d4,3a43d7c3374b7f54] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:35:36.289 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:36.290 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:37.628 WARN 21062 --- [ocp-async-4,b0835522344fd763,faf4167a66037f15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:37.628 INFO 21062 --- [ocp-async-4,2e786889e2a54711,ffce3bd80529ce2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:38.505 INFO 21062 --- [ocp-schedules-18,18ae34eaa55f3ff9,e4a4361628bc0210] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:35:38.505 INFO 21062 --- [ocp-schedules-8,0040152b678cc9b3,a4aa555a94ba67a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:35:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:35:38.590133+08:00, lastProcessTime=2024-11-04T15:35:28.589080+08:00 2024-11-04 15:35:38.661 INFO 21062 --- [ocp-schedules-19,fff3ce32e647046b,e359d93c0553dd50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:35:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:35:38.590133+08:00 2024-11-04 15:35:38.774 INFO 21062 --- [ocp-async-1,9a9552c990cdc59f,f64ee933ed2a32fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:35:38.983 INFO 21062 --- [prometheus-minute-schedule-1,cb31337f1785f714,fd07922a33856b50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:39.290 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:39.291 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:39.364 INFO 21062 --- [ocp-schedules-26,25558afcc8faae48,9ae773d2a4a44087] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:35:39.628 WARN 21062 --- [ocp-async-1,94ced9da73078d8c,9c5e6e4d1366c8cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:39.628 INFO 21062 --- [ocp-async-10,cd250dd2397fb38a,f3620fe2b17881ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:35:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:35:41.014 INFO 21062 --- [ocp-schedules-12,fa02951d5ec5d9e3,aa60378963d58852] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:35:41.015 INFO 21062 --- [ocp-schedules-12,21ad0f20376ff223,c89174feee61bce2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:35:41.021 INFO 21062 --- [ocp-schedules-12,21ad0f20376ff223,c89174feee61bce2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:41.051 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:35:41.051 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:35:41.628 WARN 21062 --- [ocp-async-6,84bed4e8987178bd,88d04cad82c460cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:41.628 INFO 21062 --- [ocp-async-5,78400411147244c7,4d96c450243bc7b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:41.847 INFO 21062 --- [ocp-schedules-14,81bb6ac0ffd1b30c,a513a75ffd6091a9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:35:41.983 INFO 21062 --- [prometheus-minute-schedule-1,bd2f8358440b07e7,8b3288ffcf46a36a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:42.291 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:42.292 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:43.628 WARN 21062 --- [ocp-async-4,355415a85cd82f3c,b4d0824d6c031080] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:43.628 INFO 21062 --- [ocp-async-8,194bcba63e9ad21e,b03eb03a819d6276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:44.983 INFO 21062 --- [prometheus-minute-schedule-1,d5e4314fe58e5229,9c90a10280410fde] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:45.292 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:45.294 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:45.432 INFO 21062 --- [ocp-schedules-21,79deed80f9513cad,ea2b57342354dc3d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:35:45.437 INFO 21062 --- [ocp-schedules-21,79deed80f9513cad,ea2b57342354dc3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:45.525 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:35:45.527 ERROR 21062 --- [ocp-schedules-21,79deed80f9513cad,ea2b57342354dc3d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:35:45.527 INFO 21062 --- [ocp-schedules-21,79deed80f9513cad,ea2b57342354dc3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:35:45.628 WARN 21062 --- [ocp-async-1,5e24abe5efeaaf4b,668dfac0e9af5d79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:45.628 INFO 21062 --- [ocp-async-3,d060094744114d73,2303c1ddf5ba101b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:47.628 WARN 21062 --- [ocp-async-6,de7535dc431777ac,266e645cd64264d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:47.629 INFO 21062 --- [ocp-async-7,7228f654d29a3f04,854076cd26847381] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:47.983 INFO 21062 --- [prometheus-minute-schedule-1,44348fe984affde3,4832953eab2ae992] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:48.294 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:48.295 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:35:48.591143+08:00, lastProcessTime=2024-11-04T15:35:38.590133+08:00 2024-11-04 15:35:48.685 INFO 21062 --- [ocp-schedules-15,eb91662bbfd88bbd,74ef3a47d1058175] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:35:48.775 INFO 21062 --- [ocp-async-4,d4c8abcd5f44fb30,dc6d2776077382ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:35:48.779 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:35:48.591143+08:00 2024-11-04 15:35:49.629 WARN 21062 --- [ocp-async-2,aaebb4d888fc4c65,e164b10d71addebf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:49.629 INFO 21062 --- [ocp-async-8,216c69dd41227067,3c6fb9f11efc131c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:50.983 INFO 21062 --- [prometheus-minute-schedule-1,2c0cf338f3b67a7e,1e8515fa0aacd843] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:51.295 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:51.296 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:51.629 WARN 21062 --- [ocp-async-9,43d2672a531b80aa,0f8b536e1b0b495a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:51.629 INFO 21062 --- [ocp-async-3,dc06982be8ee1ec3,e4ad842b86e0efc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:51.849 INFO 21062 --- [ocp-schedules-23,8d7462b6dfd7b627,8bb0a6ab902957fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:35:53.279 INFO 21062 --- [ocp-schedules-11,6a3022165dff7683,a6d13a5bb964df3d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:35:53.285 INFO 21062 --- [ocp-schedules-11,6a3022165dff7683,a6d13a5bb964df3d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:35:53.285 INFO 21062 --- [ocp-schedules-11,6a3022165dff7683,a6d13a5bb964df3d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:35:53.629 WARN 21062 --- [ocp-async-10,bbd21e49e38928a5,0ee0a48bc093b6ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:53.629 INFO 21062 --- [ocp-async-7,bc7ae61ab51735dd,85400b68f79977d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:53.983 INFO 21062 --- [prometheus-minute-schedule-1,f06831c86d6c94b7,9024b3019a0d829f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:54.296 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:54.297 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:55.629 WARN 21062 --- [ocp-async-2,e6f47a3dd1bbb473,432f56bf56fe3258] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:55.629 INFO 21062 --- [ocp-async-4,129c5fd6ffcc31ab,f9e19f006f12e752] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:56.983 INFO 21062 --- [prometheus-minute-schedule-1,3a05b22b6d1f9c9e,dc57f45a5080ae21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:35:57.297 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:35:57.298 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:35:57.629 WARN 21062 --- [ocp-async-9,46b49d6fccec78f2,0394ccf6799db93e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:57.629 INFO 21062 --- [ocp-async-1,f7c02fc61d108950,81c87bc2998b380a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:58.500 INFO 21062 --- [ocp-schedules-10,da24e57963e57766,7a6dbfe5cdf06247] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:35:58.500 INFO 21062 --- [ocp-schedules-23,d0babc488a08ffa6,bfff420ef40385cd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:35:58.502 INFO 21062 --- [ocp-schedules-26,4aba560333f50fd9,c6e14f439ee85a31] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:35:58.503 INFO 21062 --- [ocp-schedules-10,da24e57963e57766,7a6dbfe5cdf06247] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:35:58.505 INFO 21062 --- [ocp-schedules-4,095701db51c3cc01,84c9c66fe8510f1d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:35:58.505 INFO 21062 --- [ocp-schedules-23,d0babc488a08ffa6,bfff420ef40385cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:35:58.511 INFO 21062 --- [ocp-schedules-8,0101e4c17a8e1545,9e12ef86777a0567] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:35:58.511 INFO 21062 --- [ocp-schedules-18,27402f964d195487,14e809d8c2ea899f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:35:58.511 INFO 21062 --- [ocp-schedules-8,0101e4c17a8e1545,9e12ef86777a0567] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:35:58.511 INFO 21062 --- [ocp-schedules-18,27402f964d195487,14e809d8c2ea899f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:35:58.511 INFO 21062 --- [ocp-schedules-2,028acf459f83bb1a,c4a2970008739b53] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:35:58.511 INFO 21062 --- [ocp-schedules-2,028acf459f83bb1a,c4a2970008739b53] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:35:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:35:58.589736+08:00, lastProcessTime=2024-11-04T15:35:48.591143+08:00 2024-11-04 15:35:58.727 INFO 21062 --- [ocp-schedules-15,648d56d0d49ce7d2,e994e91e5700e046] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:35:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:35:58.589736+08:00 2024-11-04 15:35:58.775 INFO 21062 --- [ocp-async-10,bf5e6c7bb7976169,f3dc5b9ca23ab444] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:35:59.576 INFO 21062 --- [ocp-schedules-2,db684fa464d8c7fc,3eb3057fa2691c36] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:35:59.630 WARN 21062 --- [ocp-async-6,f6c3f8971a4e5b98,f73fa1b414b97115] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:35:59.630 INFO 21062 --- [ocp-async-7,a44e89cdc9371d64,f4867be33f0f1b63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:35:59.983 INFO 21062 --- [prometheus-minute-schedule-1,f0d900edd99fc47c,0bbca7c36c6e010b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705760, interval=60 2024-11-04 15:36:00.298 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:00.299 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:01.630 WARN 21062 --- [ocp-async-5,59f706170120d82c,a939775f1cd69459] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:01.630 INFO 21062 --- [ocp-async-5,729c06021f7158c7,2d579d9ebf769d75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:01.853 INFO 21062 --- [ocp-schedules-2,3a24d7efab34d476,6d33535349f373e0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:36:02.983 INFO 21062 --- [prometheus-minute-schedule-1,d55465468db60bd6,86a64302f1cd1572] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:03.299 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:03.300 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:03.630 WARN 21062 --- [ocp-async-8,4620f2b2848173c6,7c77e415d50fa3b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:03.630 INFO 21062 --- [ocp-async-1,c5ba561f26a4b596,002a985aa5fde7d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:05.630 WARN 21062 --- [ocp-async-6,8c76f810e1db0dea,f515371f95ce062a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:05.630 INFO 21062 --- [ocp-async-10,baa01cdda56fb9cb,20f9f9b0900d9b56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:05.983 INFO 21062 --- [prometheus-minute-schedule-1,89c423ab86c856b4,0621ec6f135453b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:06.300 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:06.301 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:07.630 WARN 21062 --- [ocp-async-4,7a82571be86f90a9,c1bbb4521564d2d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:07.630 INFO 21062 --- [ocp-async-4,7a1ae25fb5006b82,2abf22cefcf63daf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:08.505 INFO 21062 --- [ocp-schedules-10,da24e57963e57766,7a6dbfe5cdf06247] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:36:08.506 INFO 21062 --- [ocp-schedules-23,d0babc488a08ffa6,bfff420ef40385cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:36:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:36:08.590134+08:00, lastProcessTime=2024-11-04T15:35:58.589736+08:00 2024-11-04 15:36:08.753 INFO 21062 --- [ocp-schedules-20,bce5ed9fcfabddd7,d0287ef9e76c6dad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:36:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:36:08.590134+08:00 2024-11-04 15:36:08.809 INFO 21062 --- [ocp-async-8,0598633159efc9e1,93ac4506ae9ca02f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:36:08.983 INFO 21062 --- [prometheus-minute-schedule-1,6fe6a7b8724753c0,78409e4c39f72fee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:09.301 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:09.302 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:09.630 WARN 21062 --- [ocp-async-9,3b0ecce45e735a14,ac40839f7fec2a37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:09.630 INFO 21062 --- [ocp-async-1,ab19ae5dbde3f399,b2ba76485bde6c13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:10.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:36:10.004 INFO 21062 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:36:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:36:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:36:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:36:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:36:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:36:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:36:11.051 INFO 21062 --- [ocp-schedules-12,21ad0f20376ff223,c89174feee61bce2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:36:11.051 INFO 21062 --- [ocp-schedules-12,2919f430fd1bd4c7,a77607ce6ea81440] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:36:11.055 INFO 21062 --- [ocp-schedules-12,2919f430fd1bd4c7,a77607ce6ea81440] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:11.083 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:36:11.083 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:36:11.631 WARN 21062 --- [ocp-async-3,0cf5e6ac7e4d9736,2bd97f1d1a3950bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:11.631 INFO 21062 --- [ocp-async-10,bbada8b602374914,83ff44d6938690fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:11.856 INFO 21062 --- [ocp-schedules-1,3e39e6df3fe44d50,f27b022bceb33e25] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:36:11.869 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=1, affectRows=1 2024-11-04 15:36:11.870 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=1, validCount=1 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=19, affectRows=19 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=35, affectRows=35 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=39, affectRows=39 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=19, validCount=19 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=35, validCount=35 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=39, validCount=39 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=16, affectRows=16 2024-11-04 15:36:11.871 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=50, affectRows=50 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=16, validCount=16 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=64, affectRows=64 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=45, affectRows=45 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=47, affectRows=47 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=49, validCount=49 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=30, affectRows=30 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=44, validCount=44 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=47, validCount=47 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=30, validCount=30 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=64, validCount=64 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705700, nodeCount=76, affectRows=76 2024-11-04 15:36:11.872 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705700, nodeCount=75, validCount=75 2024-11-04 15:36:11.983 INFO 21062 --- [prometheus-minute-schedule-1,eeeb7898dd150edc,b7b38fd84220dc48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:12.302 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:12.304 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:13.288 INFO 21062 --- [ocp-schedules-15,21b80d3179c0fa5a,75dc72927b3146f7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:36:13.293 INFO 21062 --- [ocp-schedules-15,21b80d3179c0fa5a,75dc72927b3146f7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:36:13.293 INFO 21062 --- [ocp-schedules-15,21b80d3179c0fa5a,75dc72927b3146f7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:36:13.631 WARN 21062 --- [ocp-async-7,ddccb32a11e896fe,405b1d3cf1c5fec8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:13.631 INFO 21062 --- [ocp-async-4,059e49bc00069f16,66ee9798a14d715f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:13.984 INFO 21062 --- [prometheus-cache-monitor1,8d23494b39b6ddbd,a58d24b6a87958c7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:36:13.984 INFO 21062 --- [prometheus-cache-monitor1,8d23494b39b6ddbd,a58d24b6a87958c7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:36:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9c3fb51950951e72,e72e0a6873756344] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:15.304 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:15.305 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:15.631 WARN 21062 --- [ocp-async-5,9e4a4bd4624b98e2,b5bcf97b7fa768b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:15.631 INFO 21062 --- [ocp-async-9,04d2266414695da8,17f65b1a4bf503bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:17.631 WARN 21062 --- [ocp-async-3,100dc98a0c9a56f5,13996d7631ab16e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:17.631 INFO 21062 --- [ocp-async-1,778be9458fbc7c8d,cea0be50e57c59f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:17.983 INFO 21062 --- [prometheus-minute-schedule-1,f1b0d445d7eedc07,f6e2fb446370c055] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:18.305 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:18.306 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:36:18.589294+08:00, lastProcessTime=2024-11-04T15:36:08.590134+08:00 2024-11-04 15:36:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:36:18.589294+08:00 2024-11-04 15:36:18.809 INFO 21062 --- [ocp-async-7,e9b7f7313a360f26,49c6b8caa228e0e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:36:18.832 INFO 21062 --- [ocp-schedules-16,197887307bc69465,5e7ac68dbd2c2a2a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:36:19.631 INFO 21062 --- [ocp-async-7,19d2aa5ef38790fe,4edb69b3a4b5c385] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:19.631 WARN 21062 --- [ocp-async-10,6024bd16f25a2ebd,41bdaf876c70ce87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:20.983 INFO 21062 --- [prometheus-minute-schedule-1,d18a879aa17c28b0,9b661195f5e89572] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:21.306 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:21.307 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:21.631 INFO 21062 --- [ocp-async-8,c29660a031060542,f60600be9a5c5d81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:21.631 WARN 21062 --- [ocp-async-4,3268ee42ae637a2b,28ad8bc122772221] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:21.859 INFO 21062 --- [ocp-schedules-16,ddec74cd4f41773b,370600e136cc32c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:36:23.632 WARN 21062 --- [ocp-async-6,7f2a5a10b73e8c36,d5898401bdb491ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:23.632 INFO 21062 --- [ocp-async-9,7fcab7aa92af1cde,44396896777c4602] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6e74cb84191916c1,3622481a8375d86e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:24.307 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:24.308 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:25.632 INFO 21062 --- [ocp-async-10,fef95929da3db734,d92bfcb00a525a04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:25.632 WARN 21062 --- [ocp-async-2,785b480d643767a2,8e0bd64f9e73e2fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:26.983 INFO 21062 --- [prometheus-minute-schedule-1,9558481e03a3db8f,a550b50f04b8d472] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a9e288dcc5f825a5,2df40d7b69d34f2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:36:27.308 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:27.309 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:27.632 INFO 21062 --- [ocp-async-4,0b4f16c95b12640f,288ce3ee1e1bfea7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:27.632 WARN 21062 --- [ocp-async-7,2d61b2a253a67ec9,60c95ab47dd6e6c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:28.498 INFO 21062 --- [ocp-schedules-10,fead680a16b8fb6b,d2b7cc73e8877771] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:36:28.500 INFO 21062 --- [ocp-schedules-19,0274740d45f6cab2,f0a91400e2ed8004] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:36:28.500 INFO 21062 --- [ocp-schedules-20,9b818669936b1298,0a9e5c1af86188a2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:36:28.500 INFO 21062 --- [ocp-schedules-21,94d7746dae5eb7b8,1fd72e246eef84c9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:36:28.502 INFO 21062 --- [ocp-schedules-10,fead680a16b8fb6b,d2b7cc73e8877771] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:36:28.502 INFO 21062 --- [ocp-schedules-17,66c2ba682ba84a22,c9e2e1cad597f5b5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:36:28.504 INFO 21062 --- [ocp-schedules-21,94d7746dae5eb7b8,1fd72e246eef84c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:28.504 INFO 21062 --- [ocp-schedules-3,eae35c52474b25f0,2745d2710f37e14b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:36:28.504 INFO 21062 --- [ocp-schedules-19,0274740d45f6cab2,f0a91400e2ed8004] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:36:28.506 INFO 21062 --- [ocp-schedules-20,9b818669936b1298,0a9e5c1af86188a2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:36:28.509 INFO 21062 --- [ocp-schedules-28,394622a17f0926cf,d1a5697a5398c5b5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:36:28.509 INFO 21062 --- [ocp-schedules-14,efc8d31fbf88c9c0,8fe41b2aff1212b2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:36:28.509 INFO 21062 --- [ocp-schedules-5,a2853839615119ef,996816fcc5784dea] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:36:28.509 INFO 21062 --- [ocp-schedules-14,efc8d31fbf88c9c0,8fe41b2aff1212b2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:36:28.509 INFO 21062 --- [ocp-schedules-28,394622a17f0926cf,d1a5697a5398c5b5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:36:28.509 INFO 21062 --- [ocp-schedules-5,a2853839615119ef,996816fcc5784dea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:36:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:36:28.589056+08:00, lastProcessTime=2024-11-04T15:36:18.589294+08:00 2024-11-04 15:36:28.590 INFO 21062 --- [ocp-schedules-30,2bb5a0f6ecb7187a,e8b5ab786553b450] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:36:28.590 INFO 21062 --- [ocp-schedules-30,2bb5a0f6ecb7187a,e8b5ab786553b450] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:36:28.672 INFO 21062 --- [ocp-schedules-29,1c7b51c434446374,e1fbd1c8b09a9dd1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:36:28.673 INFO 21062 --- [ocp-schedules-29,1c7b51c434446374,e1fbd1c8b09a9dd1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:36:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:36:28.589056+08:00 2024-11-04 15:36:28.810 INFO 21062 --- [ocp-async-2,3fb8c57036356685,f2324b9eb2d60eff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:36:28.856 INFO 21062 --- [ocp-schedules-18,1835dadf3a54dedf,c47b07bb76d55706] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:36:29.017 INFO 21062 --- [ocp-schedules-20,7d59653ae77042b4,cc92bf5867585578] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:36:29.439 INFO 21062 --- [ocp-schedules-10,7b4bccffbc219594,d1fe2d4514a8e427] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:36:29.439 INFO 21062 --- [ocp-schedules-10,7b4bccffbc219594,d1fe2d4514a8e427] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:36:29.579 INFO 21062 --- [ocp-schedules-24,4850bef588d68071,dd7d828c43e5e31a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:36:29.598 INFO 21062 --- [ocp-schedules-2,5ed825c2d6607320,1d5c735ddc480cc5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:36:29.603 INFO 21062 --- [ocp-schedules-2,5ed825c2d6607320,1d5c735ddc480cc5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:36:29.603 INFO 21062 --- [ocp-schedules-2,5ed825c2d6607320,1d5c735ddc480cc5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:36:29.632 INFO 21062 --- [ocp-async-2,01c00ff637187f90,b29f019bf526205b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:29.632 WARN 21062 --- [ocp-async-1,db3f9bb8546712e5,9b1add0e6aa2dd0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:29.983 INFO 21062 --- [prometheus-minute-schedule-1,c78310b4e408b253,5a42f258c7a2847e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:30.097 INFO 21062 --- [ocp-schedules-23,952c71977c0bb00e,204af29bfe4e0d5b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:36:30.097 INFO 21062 --- [ocp-schedules-23,952c71977c0bb00e,204af29bfe4e0d5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:36:30.100 INFO 21062 --- [ocp-schedules-23,952c71977c0bb00e,204af29bfe4e0d5b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:36:30.100 INFO 21062 --- [ocp-schedules-23,952c71977c0bb00e,204af29bfe4e0d5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:36:30.101 INFO 21062 --- [ocp-schedules-23,952c71977c0bb00e,204af29bfe4e0d5b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:36:30.101 INFO 21062 --- [ocp-schedules-23,952c71977c0bb00e,204af29bfe4e0d5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:36:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:36:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:36:30.309 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:30.310 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:31.632 WARN 21062 --- [ocp-async-7,23df3fdd04b82f27,9319ee01d9be0efc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:31.632 INFO 21062 --- [ocp-async-5,3fbe6f782db63914,2a49a84c7d7f4dea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:31.862 INFO 21062 --- [ocp-schedules-2,b8d8298c9d2a637c,dc25fceddfaedcb2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:36:32.983 INFO 21062 --- [prometheus-minute-schedule-1,442a28142506fdbb,41dc5dfc2faf7817] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:33.296 INFO 21062 --- [ocp-schedules-14,7fb59f3255153654,b414e68f6e2d1861] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:36:33.303 INFO 21062 --- [ocp-schedules-14,7fb59f3255153654,b414e68f6e2d1861] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:36:33.303 INFO 21062 --- [ocp-schedules-14,7fb59f3255153654,b414e68f6e2d1861] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:36:33.310 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:33.311 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:33.563 INFO 21062 --- [ocp-schedules-22,fbd6c7cd08763356,1153ebf670803538] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:36:33.563 INFO 21062 --- [ocp-schedules-22,fbd6c7cd08763356,1153ebf670803538] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:36:33.591 INFO 21062 --- [ocp-schedules-22,fbd6c7cd08763356,1153ebf670803538] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:36:33.591 INFO 21062 --- [ocp-schedules-22,fbd6c7cd08763356,1153ebf670803538] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:36:33.591 INFO 21062 --- [ocp-schedules-23,6adecc6a227b5654,b7ed9937dd0f5d70] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:36:33.632 WARN 21062 --- [ocp-async-6,2df18a1f3ac949b7,d3112dd8fe4fa051] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:33.633 INFO 21062 --- [ocp-async-4,62b1ab908de73e3e,e4a09aa5a6667d0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:33.800 INFO 21062 --- [ocp-schedules-15,f355d859bdb05380,3241f5fc5b37b21a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:36:33.800+08:00, clusters=[1] 2024-11-04 15:36:33.801 INFO 21062 --- [ocp-schedules-15,f355d859bdb05380,3241f5fc5b37b21a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:36:34.139 INFO 21062 --- [ocp-schedules-10,3d8ae3157b47afbc,bc749847773a80cd] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:36:34.139+08:00, clusters=[1] 2024-11-04 15:36:34.143 INFO 21062 --- [ocp-schedules-10,3d8ae3157b47afbc,bc749847773a80cd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:36:34.180 INFO 21062 --- [ocp-schedules-3,88dab461047dd6b6,cf66c16e236f201d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:36:34.184 INFO 21062 --- [ocp-schedules-3,88dab461047dd6b6,cf66c16e236f201d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:34.244 ERROR 21062 --- [ocp-schedules-3,88dab461047dd6b6,cf66c16e236f201d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:36:34.244 INFO 21062 --- [ocp-schedules-3,88dab461047dd6b6,cf66c16e236f201d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:36:35.633 WARN 21062 --- [ocp-async-9,d097b487d727366b,f4f057e6a64734c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:35.633 INFO 21062 --- [ocp-async-1,ff784f475f264256,6d22dc6332537e33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:35.983 INFO 21062 --- [prometheus-minute-schedule-1,b5da265dbf19cc14,d2d3dab4604bfc88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:36.105 INFO 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:36:36.105 INFO 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:36:36.105 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:36:36.105 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:36:36.115 INFO 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:36:36.115 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:36:36.116 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:36:36.117 WARN 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:36:36.117 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:36:36.117 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:36:36.118 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:36:36.118 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:36:36.118 INFO 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:36:36.119 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:36:36.119 WARN 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:36:36.119 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:36:36.126 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:36:36.126 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:36:36.126 INFO 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:36:36.138 ERROR 21062 --- [ocp-schedules-24,750240f322885f8a,9ca8644f91f876e3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:36:36.311 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:36.312 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:37.633 INFO 21062 --- [ocp-async-7,e368c97c60224c26,821124e03122499c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:37.633 WARN 21062 --- [ocp-async-10,b80fa9bc9194eaf0,8bc49876c1dea126] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:38.504 INFO 21062 --- [ocp-schedules-21,94d7746dae5eb7b8,1fd72e246eef84c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:36:38.505 INFO 21062 --- [ocp-schedules-19,0274740d45f6cab2,f0a91400e2ed8004] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:36:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:36:38.589295+08:00, lastProcessTime=2024-11-04T15:36:28.589056+08:00 2024-11-04 15:36:38.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:36:38.589295+08:00 2024-11-04 15:36:38.810 INFO 21062 --- [ocp-async-4,409ebef74405f026,979e41b31cd5f186] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:36:38.897 INFO 21062 --- [ocp-schedules-7,291a055869c520ee,a64504b5975d7ace] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:36:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ebe06056cd31dd0e,8ef00cf90a743463] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:39.312 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:39.313 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:39.364 INFO 21062 --- [ocp-schedules-16,3c92ac1db96a92c1,2837a30cccb3be8e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:36:39.367 INFO 21062 --- [ocp-schedules-16,3c92ac1db96a92c1,2837a30cccb3be8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:39.368 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:36:39.633 INFO 21062 --- [ocp-async-4,e41e7765868270f6,7d65265d8a3e9fcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:39.633 WARN 21062 --- [ocp-async-9,60382f89e5a57e8a,833d8c954e5a2527] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:40.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:36:40.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:36:41.084 INFO 21062 --- [ocp-schedules-12,2919f430fd1bd4c7,a77607ce6ea81440] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:36:41.084 INFO 21062 --- [ocp-schedules-12,9c7219e674fc3bca,2fbe6f785231a328] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:36:41.088 INFO 21062 --- [ocp-schedules-12,9c7219e674fc3bca,2fbe6f785231a328] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:41.117 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:36:41.117 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:36:41.633 INFO 21062 --- [ocp-async-1,0a4b1124b97ff00c,5dd573b09f7328c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:41.633 WARN 21062 --- [ocp-async-10,d24db2ca12f4354b,5730fd18f3c53b91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:36:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:36:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:36:41.865 INFO 21062 --- [ocp-schedules-2,d9a9b103c5cd50e1,2340b2102aac70e5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:36:41.983 INFO 21062 --- [prometheus-minute-schedule-1,f79d98aa86aa682f,27e10a63d18e7c89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:42.313 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:42.314 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:43.633 WARN 21062 --- [ocp-async-7,b5c1b9208d608757,a470b609048a34f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:43.633 INFO 21062 --- [ocp-async-5,ff7f6c73ccb05103,f59aa52b70e156c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:44.983 INFO 21062 --- [prometheus-minute-schedule-1,73d3edddace3ae0a,500c2b6734f32a98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:45.314 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:45.315 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:45.527 INFO 21062 --- [ocp-schedules-20,3e12be16779578e5,0556cae0bf57b47b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:36:45.532 INFO 21062 --- [ocp-schedules-20,3e12be16779578e5,0556cae0bf57b47b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:45.618 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:36:45.627 ERROR 21062 --- [ocp-schedules-20,3e12be16779578e5,0556cae0bf57b47b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:36:45.627 INFO 21062 --- [ocp-schedules-20,3e12be16779578e5,0556cae0bf57b47b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:36:45.634 WARN 21062 --- [ocp-async-9,1a53baffcb5979d3,156596c280141d7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:45.634 INFO 21062 --- [ocp-async-8,bb2ff9df310b6d39,7fe305ecaef16c4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:47.634 WARN 21062 --- [ocp-async-10,19317d092bec0bda,7bbb3b6ff7c69d16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:47.634 INFO 21062 --- [ocp-async-3,82a6e58982d21dcf,8c255eb71260e088] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:47.983 INFO 21062 --- [prometheus-minute-schedule-1,93e678a205d67b14,74300525432ac34c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:48.315 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:48.316 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:36:48.590133+08:00, lastProcessTime=2024-11-04T15:36:38.589295+08:00 2024-11-04 15:36:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:36:48.590133+08:00 2024-11-04 15:36:48.810 INFO 21062 --- [ocp-async-2,a97c8337094da06b,f198595145655e88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:36:48.921 INFO 21062 --- [ocp-schedules-13,3885857178f9735a,3c3c79ab88019d85] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:36:49.368 INFO 21062 --- [ocp-schedules-16,3c92ac1db96a92c1,2837a30cccb3be8e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:36:49.634 WARN 21062 --- [ocp-async-2,150b12870fe87271,d7e0e6068020c81a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:49.634 INFO 21062 --- [ocp-async-5,a8244634fc25fcc6,aab1b498b7067f60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:50.983 INFO 21062 --- [prometheus-minute-schedule-1,179ecdc78b25a11c,addb28c48cd6eeb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:51.316 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:51.317 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:51.634 WARN 21062 --- [ocp-async-6,c381b517e6078396,6ba4af5367e39cb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:51.634 INFO 21062 --- [ocp-async-6,241b1b537a0b38fa,b9e6c3f48610545f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:51.868 INFO 21062 --- [ocp-schedules-23,0a159356a520a68e,9a88cb5c60cc806e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:36:53.305 INFO 21062 --- [ocp-schedules-6,517ff35f60197400,381dacaa5c2e02c9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:36:53.312 INFO 21062 --- [ocp-schedules-6,517ff35f60197400,381dacaa5c2e02c9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:36:53.312 INFO 21062 --- [ocp-schedules-6,517ff35f60197400,381dacaa5c2e02c9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:36:53.634 WARN 21062 --- [ocp-async-4,d17633975145cc9d,2656b322025a7bed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:53.634 INFO 21062 --- [ocp-async-3,b95278e20fe28890,506074f1ad2d1f95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a4f030ea4062d92f,718c1e6a509a3883] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:54.317 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:54.318 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:55.634 WARN 21062 --- [ocp-async-2,942c037abb078e49,0dc7b4cf6fd31ec4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:55.634 INFO 21062 --- [ocp-async-7,784ce40afb7d34cc,a8b04cbd4f045793] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:56.983 INFO 21062 --- [prometheus-minute-schedule-1,6f898d22e9d736b5,e2167f92b24de0a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:36:57.318 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:36:57.319 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:36:57.635 WARN 21062 --- [ocp-async-8,4752b862936925ff,5a24c18ea9bd6db1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:57.635 INFO 21062 --- [ocp-async-9,6d0fbcd6b7734a78,f6262e9588bf3219] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:58.500 INFO 21062 --- [ocp-schedules-19,910b93139a857a58,f4aed9d4257d6756] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:36:58.500 INFO 21062 --- [ocp-schedules-3,4468588409ce41a0,c1ccef00d4edd3b0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:36:58.502 INFO 21062 --- [ocp-schedules-28,7e0dc5da96f523aa,a5ad6fadd330f337] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:36:58.504 INFO 21062 --- [ocp-schedules-19,910b93139a857a58,f4aed9d4257d6756] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:58.504 INFO 21062 --- [ocp-schedules-3,4468588409ce41a0,c1ccef00d4edd3b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:36:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:36:58.504 INFO 21062 --- [ocp-schedules-20,91f2deb8aae92dc6,430262bef7af9024] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:36:58.510 INFO 21062 --- [ocp-schedules-1,42bbb12a61759c6a,b213cb49ca20ed10] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:36:58.510 INFO 21062 --- [ocp-schedules-1,42bbb12a61759c6a,b213cb49ca20ed10] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:36:58.510 INFO 21062 --- [ocp-schedules-13,87f8643de53aef0a,9e8ee60d0b394bc0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:36:58.510 INFO 21062 --- [ocp-schedules-13,87f8643de53aef0a,9e8ee60d0b394bc0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:36:58.511 INFO 21062 --- [ocp-schedules-22,d4fa6665c407fdc6,d4a0a5817cfec4ee] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:36:58.511 INFO 21062 --- [ocp-schedules-22,d4fa6665c407fdc6,d4a0a5817cfec4ee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:36:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:36:58.589514+08:00, lastProcessTime=2024-11-04T15:36:48.590133+08:00 2024-11-04 15:36:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:36:58.589514+08:00 2024-11-04 15:36:58.810 INFO 21062 --- [ocp-async-10,363d08c9b6a93ccb,2018d1cb3f13b3e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:36:58.963 INFO 21062 --- [ocp-schedules-6,dfa9d90205bb1f9e,3b9be22778b03c07] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:36:59.583 INFO 21062 --- [ocp-schedules-22,48daeda883e2c065,53443142948ab2a9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:36:59.635 WARN 21062 --- [ocp-async-10,e0d96748299b0be0,f56fc1269b9ade47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:36:59.635 INFO 21062 --- [ocp-async-3,fb83dfb2747336f7,965ec457d5d5032c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:36:59.983 INFO 21062 --- [prometheus-minute-schedule-1,d95d4090e0a49ff5,df57f9b152d2f996] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705820, interval=60 2024-11-04 15:37:00.319 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:00.320 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:01.635 WARN 21062 --- [ocp-async-1,171666dd1d2a2c32,b82f8ad316d57b13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:01.635 INFO 21062 --- [ocp-async-7,7bd1122fffe5b2d8,5b850c687599332a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:01.871 INFO 21062 --- [ocp-schedules-7,4ad922c25132158d,61defa63acf46cfd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:37:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4f709d06e5a901ec,30cfdb6bb711950d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:03.320 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:03.321 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:03.635 WARN 21062 --- [ocp-async-5,3525573261c2e570,d1db99475caad4d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:03.635 INFO 21062 --- [ocp-async-9,9349df3a0e8bcdc5,78efc3a96c59f6ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:05.636 INFO 21062 --- [ocp-async-10,f7eb0cf655f6f2b7,9adb9ff1b126dca1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:05.636 WARN 21062 --- [ocp-async-4,298291e71523b08c,9433a5b52a9337b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:05.983 INFO 21062 --- [prometheus-minute-schedule-1,7671dfc5887d7f3c,ca7c8f8c24689225] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:06.321 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:06.322 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:07.636 INFO 21062 --- [ocp-async-1,6b3a2e3dd1aa054b,2de0325f7ca77b0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:07.636 WARN 21062 --- [ocp-async-2,a48579951a74178f,2ed953242ec609e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:08.504 INFO 21062 --- [ocp-schedules-19,910b93139a857a58,f4aed9d4257d6756] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:37:08.505 INFO 21062 --- [ocp-schedules-3,4468588409ce41a0,c1ccef00d4edd3b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:37:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:37:08.590114+08:00, lastProcessTime=2024-11-04T15:36:58.589514+08:00 2024-11-04 15:37:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:37:08.590114+08:00 2024-11-04 15:37:08.810 INFO 21062 --- [ocp-async-5,1dd514328460f4a2,790d968338b1408d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:37:08.983 INFO 21062 --- [prometheus-minute-schedule-1,1b8eb59723046a12,3c1ac7497a6c845a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:08.987 INFO 21062 --- [ocp-schedules-11,a165dfa0acd7e932,bfc49e0764cb0bc6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:37:09.322 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:09.323 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:09.636 WARN 21062 --- [ocp-async-9,f2ef466824cb8658,6014ba7185126e5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:09.636 INFO 21062 --- [ocp-async-8,c005dba7f6aa810c,2b58ff1f0309e2aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:10.002 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:37:10.002 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:37:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:37:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:37:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:37:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:37:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:37:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:37:11.118 INFO 21062 --- [ocp-schedules-12,9c7219e674fc3bca,2fbe6f785231a328] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:37:11.118 INFO 21062 --- [ocp-schedules-12,192d0de2211f081f,cb6053b27ba95d46] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:37:11.121 INFO 21062 --- [ocp-schedules-12,192d0de2211f081f,cb6053b27ba95d46] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:11.150 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:37:11.150 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:37:11.636 WARN 21062 --- [ocp-async-10,5f77fd1baf7b509d,7a70bec8f12613ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:11.636 INFO 21062 --- [ocp-async-6,45349edab4ea9970,c497720f70bda187] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:11.874 INFO 21062 --- [ocp-schedules-27,420badc7fc00b601,bd34aaefcd6b7d40] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:37:11.878 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705760, nodeCount=29, affectRows=29 2024-11-04 15:37:11.879 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705760, nodeCount=48, affectRows=48 2024-11-04 15:37:11.879 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705760, nodeCount=28, validCount=28 2024-11-04 15:37:11.879 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705760, nodeCount=35, affectRows=35 2024-11-04 15:37:11.879 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705760, nodeCount=48, validCount=48 2024-11-04 15:37:11.879 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705760, nodeCount=35, validCount=35 2024-11-04 15:37:11.879 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705760, nodeCount=105, affectRows=105 2024-11-04 15:37:11.880 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705760, nodeCount=105, validCount=105 2024-11-04 15:37:11.881 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705760, nodeCount=205, affectRows=205 2024-11-04 15:37:11.883 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705760, nodeCount=203, validCount=203 2024-11-04 15:37:11.983 INFO 21062 --- [prometheus-minute-schedule-1,87139fa55af2a685,bb2c6ac53b029959] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:12.324 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:12.325 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:13.314 INFO 21062 --- [ocp-schedules-10,2692a623fd7de8dc,4d6d19bd61d92d49] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:37:13.319 INFO 21062 --- [ocp-schedules-10,2692a623fd7de8dc,4d6d19bd61d92d49] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:37:13.319 INFO 21062 --- [ocp-schedules-10,2692a623fd7de8dc,4d6d19bd61d92d49] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:37:13.636 WARN 21062 --- [ocp-async-1,d7e28266f71cbc80,6796359d113975e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:13.636 INFO 21062 --- [ocp-async-3,2957094c366661b9,297ddcb398c0374a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:13.985 INFO 21062 --- [prometheus-cache-monitor1,28ff89192be21999,610f5bf4f42c3eed] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:37:13.985 INFO 21062 --- [prometheus-cache-monitor1,28ff89192be21999,610f5bf4f42c3eed] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:37:14.983 INFO 21062 --- [prometheus-minute-schedule-1,822e969ab5431b76,3e758c07cdd8b5d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:15.325 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:15.325 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:15.637 INFO 21062 --- [ocp-async-7,cf370a7666850ae2,e2cbd02a2413a54f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:15.637 WARN 21062 --- [ocp-async-9,1e4efc72343da761,26d2d89620650b0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:15.729 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:33:15.703, end=2024-11-04T15:35:15.727} 2024-11-04 15:37:15.734 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:37:15.734 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:37:17.637 INFO 21062 --- [ocp-async-8,7853d24d43a82669,41d0901bc07044f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:17.637 WARN 21062 --- [ocp-async-10,0adaba3fe2d351e9,1b78da820e0c57ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:17.983 INFO 21062 --- [prometheus-minute-schedule-1,3aad4d58cf8b3561,f39b62d5a4636a43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:18.325 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:18.327 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:37:18.589726+08:00, lastProcessTime=2024-11-04T15:37:08.590114+08:00 2024-11-04 15:37:18.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:37:18.589726+08:00 2024-11-04 15:37:18.810 INFO 21062 --- [ocp-async-1,85966edebb833d2d,3b72c60fb70d3739] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:37:19.028 INFO 21062 --- [ocp-schedules-13,79329db42e88f473,4bd74dea7f3527ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:37:19.637 INFO 21062 --- [ocp-async-1,53e4fea83b8c23ca,27d27304fc2ee135] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:19.637 WARN 21062 --- [ocp-async-6,d26908ce784ea37e,24dc24da06eb7d36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:20.983 INFO 21062 --- [prometheus-minute-schedule-1,c0b1c8cfb9aad639,301f2b69b19faa61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:21.327 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:21.328 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:21.637 WARN 21062 --- [ocp-async-5,00aabd6627cad6c1,f9a0681380770e30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:21.637 INFO 21062 --- [ocp-async-3,fa2ca3e77ad0ddf0,376e2dc4d05f6fbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:21.877 INFO 21062 --- [ocp-schedules-16,e92bcd20ed134a5b,671bb49b3296ea2a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:37:23.637 WARN 21062 --- [ocp-async-7,4f4b3871c53d37cf,5faea0d23aecc0f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:23.637 INFO 21062 --- [ocp-async-4,fc3b06f46c634607,0bd9c431ec88384d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:23.983 INFO 21062 --- [prometheus-minute-schedule-1,d910c66bd10a8a9b,04959971ffda7fab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:24.328 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:24.329 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:25.638 WARN 21062 --- [ocp-async-6,c128430807559994,73c6a5cba111bc9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:25.638 INFO 21062 --- [ocp-async-6,f216b39d227122f6,ff0c74130114e595] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:26.983 INFO 21062 --- [prometheus-minute-schedule-1,67f3e3206bb2a82b,cb9a20342294352a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:26.984 INFO 21062 --- [prometheus-hour-schedule-1,612a14d876080239,8b0fd7a79a9a9353] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:37:27.329 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:27.331 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:27.638 WARN 21062 --- [ocp-async-5,1761910804dfec1e,9a1b831f61f42b46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:27.638 INFO 21062 --- [ocp-async-5,6968d42737475a8f,d0021901976bd717] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:28.498 INFO 21062 --- [ocp-schedules-17,07d1c716a9090ecd,99c988a266b8b3cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:37:28.499 INFO 21062 --- [ocp-schedules-22,89a752b75377bad9,13601c2314ce1fa4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:37:28.500 INFO 21062 --- [ocp-schedules-9,c51895a451cb7f9d,ea8adb896d0fb0c0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:37:28.500 INFO 21062 --- [ocp-schedules-18,e8dc155094913105,611bb84d72cff24f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:37:28.502 INFO 21062 --- [ocp-schedules-17,07d1c716a9090ecd,99c988a266b8b3cb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:37:28.502 INFO 21062 --- [ocp-schedules-7,2df51cb794f30c0c,8477bc8bfdfc5584] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:37:28.503 INFO 21062 --- [ocp-schedules-9,c51895a451cb7f9d,ea8adb896d0fb0c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:28.504 INFO 21062 --- [ocp-schedules-18,e8dc155094913105,611bb84d72cff24f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:37:28.504 INFO 21062 --- [ocp-schedules-5,9c1206967ac0463f,676aaae3dcb2a06f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:37:28.504 INFO 21062 --- [ocp-schedules-29,5253f4e3dea68315,b404db6386cbdb90] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:37:28.505 INFO 21062 --- [ocp-schedules-22,89a752b75377bad9,13601c2314ce1fa4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:37:28.507 INFO 21062 --- [ocp-schedules-5,9c1206967ac0463f,676aaae3dcb2a06f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:28.510 INFO 21062 --- [ocp-schedules-3,755bac3ceaf6298d,82434a837829382f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:37:28.510 INFO 21062 --- [ocp-schedules-14,2d141530a172820b,356ebdd8d88a4303] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:37:28.510 INFO 21062 --- [ocp-schedules-2,ffd0895466ee269e,18c92ccd2983f9e7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:37:28.510 INFO 21062 --- [ocp-schedules-3,755bac3ceaf6298d,82434a837829382f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:37:28.510 INFO 21062 --- [ocp-schedules-14,2d141530a172820b,356ebdd8d88a4303] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:37:28.510 INFO 21062 --- [ocp-schedules-2,ffd0895466ee269e,18c92ccd2983f9e7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:37:28.542 INFO 21062 --- [ocp-schedules-6,0f861dbfa2721029,7e3ec917ff29c288] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:37:28.542 INFO 21062 --- [ocp-schedules-6,0f861dbfa2721029,7e3ec917ff29c288] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:37:28.583 INFO 21062 --- [pool-12-thread-6,9b7b8394af53657e,f2434286679fcd0e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:37:28.583 INFO 21062 --- [pool-12-thread-7,9b7b8394af53657e,ef19ead3f5662bef] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:37:28.583 INFO 21062 --- [pool-12-thread-7,9b7b8394af53657e,ef19ead3f5662bef] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:37:28.584 INFO 21062 --- [pool-12-thread-8,9b7b8394af53657e,40c3fa0fe0ed001e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:37:28.584 INFO 21062 --- [pool-12-thread-5,9b7b8394af53657e,0c59a4c762bcfc63] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:37:28.586 INFO 21062 --- [pool-12-thread-7,9b7b8394af53657e,ef19ead3f5662bef] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:37:28.587 INFO 21062 --- [pool-12-thread-7,9b7b8394af53657e,ef19ead3f5662bef] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:37:28.587 INFO 21062 --- [pool-12-thread-6,9b7b8394af53657e,f2434286679fcd0e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:37:28.587 INFO 21062 --- [pool-12-thread-6,9b7b8394af53657e,f2434286679fcd0e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:37:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:37:28.588007+08:00, lastProcessTime=2024-11-04T15:37:18.589726+08:00 2024-11-04 15:37:28.589 INFO 21062 --- [ForkJoinPool-11-worker-8,9b7b8394af53657e,0072ae541e8c5ce7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:37:28.589 INFO 21062 --- [ForkJoinPool-11-worker-8,9b7b8394af53657e,0072ae541e8c5ce7] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:37:28.589 INFO 21062 --- [ForkJoinPool-11-worker-1,9b7b8394af53657e,1f13c6a2c49c66df] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:37:28.589 INFO 21062 --- [ForkJoinPool-11-worker-8,9b7b8394af53657e,0072ae541e8c5ce7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:37:28.590 INFO 21062 --- [pool-12-thread-5,9b7b8394af53657e,0c59a4c762bcfc63] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:37:28.590 INFO 21062 --- [pool-12-thread-8,9b7b8394af53657e,40c3fa0fe0ed001e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:37:28.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:37:28.588007+08:00 2024-11-04 15:37:28.811 INFO 21062 --- [ocp-async-2,36e8d20edd7549e3,0174a5ec488bd781] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:37:29.020 INFO 21062 --- [ocp-schedules-3,7c16778230ddfdbb,382a19919c43571f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:37:29.058 INFO 21062 --- [ocp-schedules-14,61b669783ad1baeb,e7401a2346de6c58] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:37:29.444 INFO 21062 --- [ocp-schedules-1,8fd55c78990112cf,e407778e80ff648f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:37:29.444 INFO 21062 --- [ocp-schedules-1,8fd55c78990112cf,e407778e80ff648f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:37:29.586 INFO 21062 --- [ocp-schedules-23,30521cc531bed15e,601ef64c6068b9cf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:37:29.594 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:37:29.603 INFO 21062 --- [ocp-schedules-26,dd536ff743aa473c,b93616f4f8dcda27] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:37:29.608 INFO 21062 --- [ocp-schedules-26,dd536ff743aa473c,b93616f4f8dcda27] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:37:29.608 INFO 21062 --- [ocp-schedules-26,dd536ff743aa473c,b93616f4f8dcda27] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:37:29.610 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:37:29.612 WARN 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:37:29.612 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:37:29.613 WARN 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:37:29.620 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:37:29.621 WARN 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:37:29.622 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:37:29.623 WARN 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:37:29.629 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:37:29.630 WARN 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:37:29.631 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:37:29.632 WARN 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:37:29.637 INFO 21062 --- [ocp-schedules-15,5c7ed00f7a298457,5dcec333b6b58a7d] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:37:29.638 WARN 21062 --- [ocp-async-2,e5bf66a8678f941f,5c78ca5649d260c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:29.638 INFO 21062 --- [ocp-async-8,c496d445c82f8288,cc7bcf6a49535944] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:29.929 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:37:29.929 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:37:29.939 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:37:29.939 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:37:29.939 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:37:29.950 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:37:29.950 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:37:29.950 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:37:29.954 INFO 21062 --- [ocp-schedules-29,ed4964c67599db34,908e92a035223ee4] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:37:29.983 INFO 21062 --- [prometheus-minute-schedule-1,69799f6f29016303,b3086d9ccef8886d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:30.104 INFO 21062 --- [ocp-schedules-7,697cc3401e43ae74,0a66a9633cc5e41c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:37:30.104 INFO 21062 --- [ocp-schedules-7,697cc3401e43ae74,0a66a9633cc5e41c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:37:30.105 INFO 21062 --- [ocp-schedules-7,697cc3401e43ae74,0a66a9633cc5e41c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:37:30.105 INFO 21062 --- [ocp-schedules-7,697cc3401e43ae74,0a66a9633cc5e41c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:37:30.106 INFO 21062 --- [ocp-schedules-7,697cc3401e43ae74,0a66a9633cc5e41c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:37:30.106 INFO 21062 --- [ocp-schedules-7,697cc3401e43ae74,0a66a9633cc5e41c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:37:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:37:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:37:30.332 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:30.333 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:30.602 INFO 21062 --- [ocp-schedules-6,6ff89eb74088b227,789fb73e00a353fb] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:37:30.602 INFO 21062 --- [ocp-schedules-6,6ff89eb74088b227,789fb73e00a353fb] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:37:30.631 INFO 21062 --- [pool-56-thread-1,6ff89eb74088b227,d4407386bf7cdb48] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:37:30.660 WARN 21062 --- [ocp-schedules-6,6ff89eb74088b227,789fb73e00a353fb] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:37:30.660 INFO 21062 --- [ocp-schedules-6,6ff89eb74088b227,789fb73e00a353fb] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:37:30.997 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:37:30.997 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:37:31.005 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:37:31.005 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:37:31.005 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:37:31.006 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:37:31.012 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:37:31.012 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:37:31.012 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:37:31.012 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:37:31.047 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:37:31.056 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:37:31.056 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:37:31.056 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:37:31.056 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:37:31.056 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:37:31.064 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:37:31.601 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:37:31.638 INFO 21062 --- [ocp-async-1,72d8bb4185ccd353,b8978cde18be91f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:31.638 WARN 21062 --- [ocp-async-9,285363ce8c3a061e,bddd88d95e875cf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:31.880 INFO 21062 --- [ocp-schedules-28,4518c0d6ce59a263,b82d3e97336fe128] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:37:32.164 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.15177513S 2024-11-04 15:37:32.164 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:37:32.263 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:37:32.263 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:37:32.289 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:37:32.346 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:37:32.346 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:37:32.346 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:37:32.346 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:37:32.346 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:37:32.347 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:37:32.358 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:37:32.542 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] o.s.boot.SpringApplication : Started application in 1.536 seconds (JVM running for 11266.221) 2024-11-04 15:37:32.542 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.536956842S 2024-11-04 15:37:32.983 INFO 21062 --- [prometheus-minute-schedule-1,ff3c37bef0ddaad8,1bcbfe7dbe5fdfaa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:33.012 INFO 21062 --- [ocp-schedules-30,963ae5dfe36d5ce0,addcdea86c924a9d] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:37:33.322 INFO 21062 --- [ocp-schedules-15,1db4d0e55496a919,d45f17f59b968fbd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:37:33.328 INFO 21062 --- [ocp-schedules-15,1db4d0e55496a919,d45f17f59b968fbd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:37:33.328 INFO 21062 --- [ocp-schedules-15,1db4d0e55496a919,d45f17f59b968fbd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:37:33.333 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:33.335 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:33.592 INFO 21062 --- [ocp-schedules-23,20723a97742a460b,fb17053f7b361736] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:37:33.592 INFO 21062 --- [ocp-schedules-23,20723a97742a460b,fb17053f7b361736] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:37:33.619 INFO 21062 --- [ocp-schedules-23,20723a97742a460b,fb17053f7b361736] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:37:33.619 INFO 21062 --- [ocp-schedules-23,20723a97742a460b,fb17053f7b361736] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:37:33.620 INFO 21062 --- [ocp-schedules-4,2769eb6e25e4cd73,c915510f25cc803e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:37:33.638 WARN 21062 --- [ocp-async-5,c2ee44c7a69ca9ee,83d1d987a0404739] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:33.638 INFO 21062 --- [ocp-async-7,abceb9eb723b4c7c,79e0350a334d9b82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:33.827 INFO 21062 --- [ocp-schedules-28,03b2ff16f123c36e,f9f66eb37a12783d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:37:33.827+08:00, clusters=[1] 2024-11-04 15:37:33.829 INFO 21062 --- [ocp-schedules-28,03b2ff16f123c36e,f9f66eb37a12783d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:37:34.169 INFO 21062 --- [ocp-schedules-7,e9de451b382fa227,944a62308d8c1121] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:37:34.169+08:00, clusters=[1] 2024-11-04 15:37:34.172 INFO 21062 --- [ocp-schedules-7,e9de451b382fa227,944a62308d8c1121] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:37:35.638 WARN 21062 --- [ocp-async-2,9d6ccb30c5217aea,0ce0dcf8d6218c5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:35.638 INFO 21062 --- [ocp-async-4,8c69ced1090a4ab7,91b8c55baec7349d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c46ea4489730f4ee,be461e32c197cd26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:36.147 INFO 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:37:36.147 INFO 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:37:36.147 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:37:36.147 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:37:36.192 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:37:36.192 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:37:36.192 INFO 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:37:36.194 WARN 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:37:36.194 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:37:36.194 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:37:36.195 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:37:36.195 INFO 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:37:36.195 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:37:36.196 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:37:36.196 WARN 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:37:36.197 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:37:36.204 INFO 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:37:36.204 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:37:36.204 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:37:36.216 ERROR 21062 --- [ocp-schedules-21,7cdf617dcb671642,4ee6c4c1f839f25a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:37:36.335 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:36.336 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:37.639 WARN 21062 --- [ocp-async-9,5bc0e9c2bb8dd562,ba6c9e854512c7c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:37.639 INFO 21062 --- [ocp-async-9,4bc9a6bd228fa8e1,ed5e7830b87ca5a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:38.504 INFO 21062 --- [ocp-schedules-18,e8dc155094913105,611bb84d72cff24f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:37:38.504 INFO 21062 --- [ocp-schedules-9,c51895a451cb7f9d,ea8adb896d0fb0c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:37:38.507 INFO 21062 --- [ocp-schedules-5,9c1206967ac0463f,676aaae3dcb2a06f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:37:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:37:38.590135+08:00, lastProcessTime=2024-11-04T15:37:28.588007+08:00 2024-11-04 15:37:38.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:37:38.590135+08:00 2024-11-04 15:37:38.811 INFO 21062 --- [ocp-async-7,40531451af6d147f,932e695529aee7a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:37:38.983 INFO 21062 --- [prometheus-minute-schedule-1,b2b666fe42e91113,977adbea1b6f3e9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:39.142 INFO 21062 --- [ocp-schedules-13,c5087fc1fe2ec8bb,f9741f98b8c25747] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:37:39.336 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:39.336 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:39.639 WARN 21062 --- [ocp-async-7,409b16fa0eb4510e,0e61870cf0942280] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:39.639 INFO 21062 --- [ocp-async-2,dda049cbd91f953a,9f6f90fdf3b59eb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:40.002 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:37:40.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:37:41.151 INFO 21062 --- [ocp-schedules-12,192d0de2211f081f,cb6053b27ba95d46] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:37:41.151 INFO 21062 --- [ocp-schedules-12,919dbe89ae6c484c,17b61cad27446bd5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:37:41.155 INFO 21062 --- [ocp-schedules-12,919dbe89ae6c484c,17b61cad27446bd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:41.182 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:37:41.182 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:37:41.639 WARN 21062 --- [ocp-async-4,eacefe61ac28cb80,67348992062f0993] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:41.639 INFO 21062 --- [ocp-async-6,b50d2e8e967ecc14,8ee23e600ae81ff5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:41.883 INFO 21062 --- [ocp-schedules-16,8e9b192bc3fd38c7,691da935fda61d51] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:37:41.983 INFO 21062 --- [prometheus-minute-schedule-1,7ec76e4e0450b39d,6e31d89d36c46c20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:42.337 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:42.339 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:43.639 WARN 21062 --- [ocp-async-9,dfc7667a5ef85c92,43a20784499e6fb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:43.639 INFO 21062 --- [ocp-async-9,5ffc7b59edcacc59,c91e6575009bfb9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:44.983 INFO 21062 --- [prometheus-minute-schedule-1,261f668e1742f9e4,f5ca8275d59530d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:45.339 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:45.340 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:45.627 INFO 21062 --- [ocp-schedules-23,b81e916d076be6aa,faca84b6a0674769] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:37:45.631 INFO 21062 --- [ocp-schedules-23,b81e916d076be6aa,faca84b6a0674769] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:45.639 WARN 21062 --- [ocp-async-7,84f968f8788a6752,bfc32d24337332e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:45.639 INFO 21062 --- [ocp-async-3,44488577f6369360,028618150b11ca5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:45.716 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:37:45.718 ERROR 21062 --- [ocp-schedules-23,b81e916d076be6aa,faca84b6a0674769] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:37:45.718 INFO 21062 --- [ocp-schedules-23,b81e916d076be6aa,faca84b6a0674769] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:37:47.639 WARN 21062 --- [ocp-async-4,2cbda79a419a5f8d,1451c8d5fe464cc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:47.639 INFO 21062 --- [ocp-async-10,6375aec93c5fff6f,1fabcdba4e139dd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b197a9b1a6e5c802,2fc3aaa194cc179a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:48.340 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:48.341 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:37:48.590373+08:00, lastProcessTime=2024-11-04T15:37:38.590135+08:00 2024-11-04 15:37:48.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:37:48.590373+08:00 2024-11-04 15:37:48.811 INFO 21062 --- [ocp-async-8,2cf8796ba78cf285,33f791ef36d75885] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:37:49.164 INFO 21062 --- [ocp-schedules-29,c861111e4e049fa0,a7737995dcdb2373] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:37:49.369 INFO 21062 --- [ocp-schedules-2,6c9aafe969784b2c,7f1fc67222d52d6f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:37:49.372 INFO 21062 --- [ocp-schedules-2,6c9aafe969784b2c,7f1fc67222d52d6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:49.373 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:37:49.639 WARN 21062 --- [ocp-async-8,b5d548f4281d1079,6e3b1d285be015a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:49.640 INFO 21062 --- [ocp-async-9,8ff6fa7b9c26685e,e331111c8b4dec16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:50.983 INFO 21062 --- [prometheus-minute-schedule-1,98cf27fb537d55e2,1d791604a3d3e851] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:51.342 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:51.343 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:51.640 WARN 21062 --- [ocp-async-5,bbf6494cdda65096,3cf8ca0f9ce54c33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:51.640 INFO 21062 --- [ocp-async-3,9ff1180d9e8e7613,ccb88317f4d0c323] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:51.886 INFO 21062 --- [ocp-schedules-19,2c9a86266f542b4c,740793854861e8de] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:37:53.330 INFO 21062 --- [ocp-schedules-29,9002fd5306819194,b781975be1325477] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:37:53.336 INFO 21062 --- [ocp-schedules-29,9002fd5306819194,b781975be1325477] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:37:53.336 INFO 21062 --- [ocp-schedules-29,9002fd5306819194,b781975be1325477] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:37:53.640 WARN 21062 --- [ocp-async-2,a5e042bdc6709eac,db94715222bf708a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:53.640 INFO 21062 --- [ocp-async-10,18024f9966243da2,1e25eb3e1e9b62a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:53.983 INFO 21062 --- [prometheus-minute-schedule-1,4ebb3ad53c4c8f3b,8780767a85d6275f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:54.343 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:54.344 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:55.640 INFO 21062 --- [ocp-async-8,4f319e0a361708c9,0c0f735bde318111] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:55.640 WARN 21062 --- [ocp-async-1,fe19babccb4a294e,4cd60424f436f7ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:56.983 INFO 21062 --- [prometheus-minute-schedule-1,95130bf5f95e23d0,cfedf18d3e48676a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:37:57.344 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:37:57.345 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:37:57.640 INFO 21062 --- [ocp-async-5,315363c36bfc94b1,021c8bd07a8897d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:57.640 WARN 21062 --- [ocp-async-7,801aac4ec483aa5f,99b7ef36d4ae45ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:58.500 INFO 21062 --- [ocp-schedules-29,b2998bfc9e5ef6f8,84d0713d02c230f4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:37:58.500 INFO 21062 --- [ocp-schedules-15,1df2be04d6b33c62,021fef04de4c4ef4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:37:58.502 INFO 21062 --- [ocp-schedules-8,c43dde6e50f0f1d4,888fa9985ccedcf1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:37:58.504 INFO 21062 --- [ocp-schedules-29,b2998bfc9e5ef6f8,84d0713d02c230f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:58.504 INFO 21062 --- [ocp-schedules-15,1df2be04d6b33c62,021fef04de4c4ef4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:37:58.504 INFO 21062 --- [ocp-schedules-18,3ad3de2430a34fb2,be177d55c55ada34] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:37:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:37:58.509 INFO 21062 --- [ocp-schedules-24,e61e58bb92eaa3b0,9973cea71dadf163] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:37:58.509 INFO 21062 --- [ocp-schedules-24,e61e58bb92eaa3b0,9973cea71dadf163] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:37:58.509 INFO 21062 --- [ocp-schedules-17,30928694ebbaac56,e0ff803ccf368ff8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:37:58.509 INFO 21062 --- [ocp-schedules-17,30928694ebbaac56,e0ff803ccf368ff8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:37:58.511 INFO 21062 --- [ocp-schedules-3,eb82c0570bede1db,56f9e5f5fe83d91a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:37:58.511 INFO 21062 --- [ocp-schedules-3,eb82c0570bede1db,56f9e5f5fe83d91a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:37:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:37:58.589039+08:00, lastProcessTime=2024-11-04T15:37:48.590373+08:00 2024-11-04 15:37:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:37:58.589039+08:00 2024-11-04 15:37:58.811 INFO 21062 --- [ocp-async-2,1f27942b08e88ae4,6b22cbc9fe5bd147] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:37:59.204 INFO 21062 --- [ocp-schedules-30,dee0c544dcc1a51c,03efc323f67aaece] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:37:59.373 INFO 21062 --- [ocp-schedules-2,6c9aafe969784b2c,7f1fc67222d52d6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:37:59.588 INFO 21062 --- [ocp-schedules-19,dfec510f9e84e70d,130d27b35faccb58] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:37:59.640 INFO 21062 --- [ocp-async-4,f8585a8e78e4186f,e548d0b59a382cac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:37:59.640 WARN 21062 --- [ocp-async-10,f287c0a198e469c8,61ecf512eb840c1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:37:59.983 INFO 21062 --- [prometheus-minute-schedule-1,e7101b6052dfd923,9c949e1b0de236a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705880, interval=60 2024-11-04 15:38:00.345 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:00.346 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:01.641 INFO 21062 --- [ocp-async-6,9b52612c11fa9012,002ca07f36600672] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:01.641 WARN 21062 --- [ocp-async-8,0fa3548a5f9afb49,d9733b86d6038e81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:01.888 INFO 21062 --- [ocp-schedules-14,7c5ded124728b0e8,7b336e56f694714f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:38:02.983 INFO 21062 --- [prometheus-minute-schedule-1,037fe272b5a6e695,7ea33ab3f439a3d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:03.346 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:03.347 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:03.641 INFO 21062 --- [ocp-async-9,f2219980c43e5449,987290bb392e51e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:03.641 WARN 21062 --- [ocp-async-5,4f193ee4db804f3b,d55eaeb250b81ff4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:05.641 INFO 21062 --- [ocp-async-10,6fc1ff6c9ef06302,84ee2cec6cdaad0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:05.641 WARN 21062 --- [ocp-async-2,b9e17cc18328dad8,82abf699e1317c3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8ba03e956f9514a0,7f3427facf4bb6f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:06.347 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:06.348 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:07.641 WARN 21062 --- [ocp-async-8,d97ddbd093c5e0e9,c72fa22b2fbe1b51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:07.641 INFO 21062 --- [ocp-async-1,2e5e675ce4b5c884,9dea92cb272b16e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:08.504 INFO 21062 --- [ocp-schedules-15,1df2be04d6b33c62,021fef04de4c4ef4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:38:08.505 INFO 21062 --- [ocp-schedules-29,b2998bfc9e5ef6f8,84d0713d02c230f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:38:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:38:08.589512+08:00, lastProcessTime=2024-11-04T15:37:58.589039+08:00 2024-11-04 15:38:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:38:08.589512+08:00 2024-11-04 15:38:08.811 INFO 21062 --- [ocp-async-5,95d8e14a4ee365aa,3b1ba6caa99dc2d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:38:08.983 INFO 21062 --- [prometheus-minute-schedule-1,082d8b169dab35dd,498b33ecd8ea62bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:09.230 INFO 21062 --- [ocp-schedules-7,2df69963fec9e94e,4569e022dbd7684a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:38:09.348 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:09.348 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:09.641 INFO 21062 --- [ocp-async-5,bf1990c9682449a1,e07a523c3b566cca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:09.641 WARN 21062 --- [ocp-async-7,6fd7f3fa58ab7426,5d006de67a147ea7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:38:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:38:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:38:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:38:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:38:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:38:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:38:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:38:11.183 INFO 21062 --- [ocp-schedules-12,919dbe89ae6c484c,17b61cad27446bd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:38:11.183 INFO 21062 --- [ocp-schedules-12,d3f0f7f028c513bc,c2cd3ff05ba922ae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:38:11.186 INFO 21062 --- [ocp-schedules-12,d3f0f7f028c513bc,c2cd3ff05ba922ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:11.216 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:38:11.216 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:38:11.641 INFO 21062 --- [ocp-async-2,723df63ef611892f,8c9dad5ba8315e90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:11.641 WARN 21062 --- [ocp-async-3,8b3a3d3cfa163d5e,62993762b0a918be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:11.883 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=3, affectRows=3 2024-11-04 15:38:11.883 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=3, validCount=3 2024-11-04 15:38:11.885 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=5, affectRows=5 2024-11-04 15:38:11.885 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=5, validCount=5 2024-11-04 15:38:11.886 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=8, affectRows=8 2024-11-04 15:38:11.886 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=8, validCount=8 2024-11-04 15:38:11.886 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=6, affectRows=6 2024-11-04 15:38:11.886 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=30, affectRows=30 2024-11-04 15:38:11.886 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=29, affectRows=29 2024-11-04 15:38:11.886 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=6, validCount=6 2024-11-04 15:38:11.887 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=28, validCount=28 2024-11-04 15:38:11.887 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=24, affectRows=24 2024-11-04 15:38:11.887 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=30, validCount=30 2024-11-04 15:38:11.887 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=24, validCount=24 2024-11-04 15:38:11.887 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=54, affectRows=54 2024-11-04 15:38:11.887 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=35, affectRows=35 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=70, affectRows=70 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=35, validCount=35 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=54, validCount=54 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=70, validCount=70 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=77, affectRows=77 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705820, nodeCount=81, affectRows=81 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=75, validCount=75 2024-11-04 15:38:11.888 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705820, nodeCount=81, validCount=81 2024-11-04 15:38:11.892 INFO 21062 --- [ocp-schedules-29,6bc628141411d174,e1ff48043eb27a38] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:38:11.983 INFO 21062 --- [prometheus-minute-schedule-1,0bd6ae9701a77f38,96292f5f0307f516] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:12.349 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:12.349 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:13.339 INFO 21062 --- [ocp-schedules-14,e9200d795ab9eab5,6c53c4581e90fa30] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:38:13.345 INFO 21062 --- [ocp-schedules-14,e9200d795ab9eab5,6c53c4581e90fa30] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:38:13.345 INFO 21062 --- [ocp-schedules-14,e9200d795ab9eab5,6c53c4581e90fa30] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:38:13.642 INFO 21062 --- [ocp-async-8,78aa0dc21787b056,e59d7e667e437d5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:13.642 WARN 21062 --- [ocp-async-10,7329643ff7cb0ca2,818932eef9f64fcf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:13.984 INFO 21062 --- [prometheus-cache-monitor1,d26c19e2b5397512,e94869f4778465e0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:38:13.985 INFO 21062 --- [prometheus-cache-monitor1,d26c19e2b5397512,e94869f4778465e0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:38:14.522 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:32:14.498, end=2024-11-04T15:37:14.519} 2024-11-04 15:38:14.522 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:32:14.499, end=2024-11-04T15:37:14.519} 2024-11-04 15:38:14.532 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:38:14.532 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:38:14.532 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:38:14.532 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:38:14.983 INFO 21062 --- [prometheus-minute-schedule-1,43a035277b91f253,6ae7be131ceccc22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:15.350 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:15.350 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:15.642 INFO 21062 --- [ocp-async-7,54ba8c42693f216a,4d019054ac2163e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:15.642 WARN 21062 --- [ocp-async-9,649f75812e5485cb,ccfca6e511b68ccc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:17.642 WARN 21062 --- [ocp-async-3,97c7eded3735d063,597d097966c1d82c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:17.642 INFO 21062 --- [ocp-async-5,fe5d7dfb6cfa53fb,07beb4346e6f26ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:17.983 INFO 21062 --- [prometheus-minute-schedule-1,28cf30ecc177edf3,12243736d00914b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:18.350 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:18.351 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:38:18.589972+08:00, lastProcessTime=2024-11-04T15:38:08.589512+08:00 2024-11-04 15:38:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:38:18.589972+08:00 2024-11-04 15:38:18.812 INFO 21062 --- [ocp-async-2,b1bf7644ab7899d3,7988ae3ca86326b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:38:19.269 INFO 21062 --- [ocp-schedules-8,7ebab5ed23c9534c,d04888caa36db34b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:38:19.642 WARN 21062 --- [ocp-async-2,ea7e2a4d117cc3b5,dbffde19ce158c8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:19.642 INFO 21062 --- [ocp-async-1,a80d4e97609fd1b6,98fb538b35fd4015] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f516588bfed49f29,d28779c18df913fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:21.351 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:21.352 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:21.643 WARN 21062 --- [ocp-async-8,c2ca987ebbd4ec62,440ff20b04e64b1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:21.643 INFO 21062 --- [ocp-async-6,9d74a77e7528a320,ce501a7b7affd5df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:21.895 INFO 21062 --- [ocp-schedules-20,94944df7a4ee0931,e79eec6a1214b98b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:38:23.643 WARN 21062 --- [ocp-async-7,3f060d4411a5563c,1ef02fb21b1c108e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:23.643 INFO 21062 --- [ocp-async-5,23418ed0508dabd9,fbbab40131423b32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:23.983 INFO 21062 --- [prometheus-minute-schedule-1,181748a1fdf6a0d7,b7250e1144d3691d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:24.352 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:24.353 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:25.643 WARN 21062 --- [ocp-async-2,8ccd2ab362c551e7,12db227944bdae33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:25.643 INFO 21062 --- [ocp-async-10,42174d47306721d9,4dd70191184d9fd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:26.983 INFO 21062 --- [prometheus-minute-schedule-1,084c7121585fb98c,e442044948ff821d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f88e3d2d0a8bd303,f42c6339933d7eca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:38:27.353 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:27.353 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:27.643 WARN 21062 --- [ocp-async-8,7adc7c6d08e18fc3,fb29af8bdfd4d29e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:27.643 INFO 21062 --- [ocp-async-9,5ef12692854973c9,7c8638dc40aaac96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:28.498 INFO 21062 --- [ocp-schedules-17,711c1324e48b4eed,505cf3fb40f8be9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:38:28.499 INFO 21062 --- [ocp-schedules-7,b62adf2432ed7aa2,37f9dc5f96a8d8df] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:38:28.499 INFO 21062 --- [ocp-schedules-24,d2a9e1f983029bef,99548803e41e45af] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:38:28.500 INFO 21062 --- [ocp-schedules-4,925ac1d45b17300a,af575e9c88f55402] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:38:28.502 INFO 21062 --- [ocp-schedules-17,711c1324e48b4eed,505cf3fb40f8be9a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:38:28.502 INFO 21062 --- [ocp-schedules-27,d006f9d2169cd1d7,b86bda49a4f69b9e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:38:28.503 INFO 21062 --- [ocp-schedules-24,d2a9e1f983029bef,99548803e41e45af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:38:28.503 INFO 21062 --- [ocp-schedules-4,925ac1d45b17300a,af575e9c88f55402] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:28.504 INFO 21062 --- [ocp-schedules-15,e05e240c516ef295,986c714787578b02] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:38:28.505 INFO 21062 --- [ocp-schedules-7,b62adf2432ed7aa2,37f9dc5f96a8d8df] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:38:28.509 INFO 21062 --- [ocp-schedules-21,bf0e54f71b2b2168,9a7a937f63d3c487] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:38:28.509 INFO 21062 --- [ocp-schedules-6,d05162c105869f13,c9cb753879fb18a9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:38:28.509 INFO 21062 --- [ocp-schedules-6,d05162c105869f13,c9cb753879fb18a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:38:28.509 INFO 21062 --- [ocp-schedules-21,bf0e54f71b2b2168,9a7a937f63d3c487] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:38:28.509 INFO 21062 --- [ocp-schedules-30,659bc051601dc000,82eab5a479e1c649] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:38:28.509 INFO 21062 --- [ocp-schedules-30,659bc051601dc000,82eab5a479e1c649] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:38:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:38:28.588244+08:00, lastProcessTime=2024-11-04T15:38:18.589972+08:00 2024-11-04 15:38:28.590 INFO 21062 --- [ocp-schedules-8,69b1c607137ba822,cb56b9756cc21244] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:38:28.590 INFO 21062 --- [ocp-schedules-8,69b1c607137ba822,cb56b9756cc21244] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:38:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:38:28.588244+08:00 2024-11-04 15:38:28.812 INFO 21062 --- [ocp-async-2,36a9010e912d5d48,2c04c535a2494388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:38:29.009 INFO 21062 --- [ocp-schedules-22,995b83f39d361bcc,c2af6d50b7bc351e] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:38:29.009 INFO 21062 --- [ocp-schedules-22,995b83f39d361bcc,c2af6d50b7bc351e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:38:29.023 INFO 21062 --- [ocp-schedules-20,5de2d4024fac9a12,9de4a18a1be1b6ae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:38:29.293 INFO 21062 --- [ocp-schedules-6,a9e5e6e96f7956d0,01ad062ffc241181] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:38:29.448 INFO 21062 --- [ocp-schedules-17,2ba5a09fbdf569fb,c50e698a75e50183] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:38:29.448 INFO 21062 --- [ocp-schedules-17,2ba5a09fbdf569fb,c50e698a75e50183] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:38:29.591 INFO 21062 --- [ocp-schedules-13,81c16c875b217e34,5361630b01a2ff00] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:38:29.608 INFO 21062 --- [ocp-schedules-5,c89c0498ec6282b2,8aa31aaa04137af6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:38:29.613 INFO 21062 --- [ocp-schedules-5,c89c0498ec6282b2,8aa31aaa04137af6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:38:29.613 INFO 21062 --- [ocp-schedules-5,c89c0498ec6282b2,8aa31aaa04137af6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:38:29.643 WARN 21062 --- [ocp-async-2,a4f46f5a01c37aba,3aaa5fc3a1f719fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:29.643 INFO 21062 --- [ocp-async-4,2203b1ebbff5d798,07f31d47d652f06d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:29.983 INFO 21062 --- [prometheus-minute-schedule-1,19d6f80d873fdb19,ecded05442eb8a88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:30.108 INFO 21062 --- [ocp-schedules-18,b8333a2743f1dd99,d2b10c276c2427b7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:38:30.108 INFO 21062 --- [ocp-schedules-18,b8333a2743f1dd99,d2b10c276c2427b7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:38:30.115 INFO 21062 --- [ocp-schedules-18,b8333a2743f1dd99,d2b10c276c2427b7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:38:30.115 INFO 21062 --- [ocp-schedules-18,b8333a2743f1dd99,d2b10c276c2427b7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:38:30.119 INFO 21062 --- [ocp-schedules-18,b8333a2743f1dd99,d2b10c276c2427b7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:38:30.119 INFO 21062 --- [ocp-schedules-18,b8333a2743f1dd99,d2b10c276c2427b7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:38:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:38:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:38:30.354 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:30.354 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:31.644 WARN 21062 --- [ocp-async-8,cda57e1ffe5c8269,9e919f97cba6e186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:31.644 INFO 21062 --- [ocp-async-1,a6e117b0ed87aafe,c95e272e2106f3ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:31.897 INFO 21062 --- [ocp-schedules-13,d344ee6017354df8,e67edd6c7a37fc08] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:38:32.983 INFO 21062 --- [prometheus-minute-schedule-1,205b55c8bb1a6507,8accdaead82fa62f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:33.347 INFO 21062 --- [ocp-schedules-6,89efd29b27ed0106,0f766ccddf4bcc53] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:38:33.352 INFO 21062 --- [ocp-schedules-6,89efd29b27ed0106,0f766ccddf4bcc53] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:38:33.352 INFO 21062 --- [ocp-schedules-6,89efd29b27ed0106,0f766ccddf4bcc53] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:38:33.355 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:33.355 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:33.619 INFO 21062 --- [ocp-schedules-22,faf4118e08a7052f,e99b2ffe0acb2c87] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:38:33.619 INFO 21062 --- [ocp-schedules-22,faf4118e08a7052f,e99b2ffe0acb2c87] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:38:33.644 WARN 21062 --- [ocp-async-9,a2b1c13fd0863e6a,c6c8874a611f9ff5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:33.644 INFO 21062 --- [ocp-async-7,a5d1fd2e40f57c68,0df4361c6094df6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:33.647 INFO 21062 --- [ocp-schedules-22,faf4118e08a7052f,e99b2ffe0acb2c87] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:38:33.647 INFO 21062 --- [ocp-schedules-22,faf4118e08a7052f,e99b2ffe0acb2c87] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:38:33.647 INFO 21062 --- [ocp-schedules-28,23cf771087fc6da4,6a3c6637177b8ba2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:38:33.855 INFO 21062 --- [ocp-schedules-30,29e4c0487a4b2624,6e39fa9f21b7cd2e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:38:33.855+08:00, clusters=[1] 2024-11-04 15:38:33.856 INFO 21062 --- [ocp-schedules-30,29e4c0487a4b2624,6e39fa9f21b7cd2e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:38:34.199 INFO 21062 --- [ocp-schedules-10,007459c1cd646f92,54a8ded3c13b23ac] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:38:34.199+08:00, clusters=[1] 2024-11-04 15:38:34.202 INFO 21062 --- [ocp-schedules-10,007459c1cd646f92,54a8ded3c13b23ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:38:34.245 INFO 21062 --- [ocp-schedules-29,728a56c1d131b106,e1b41463beaa09d7] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:38:34.248 INFO 21062 --- [ocp-schedules-29,728a56c1d131b106,e1b41463beaa09d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:34.306 ERROR 21062 --- [ocp-schedules-29,728a56c1d131b106,e1b41463beaa09d7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:38:34.306 INFO 21062 --- [ocp-schedules-29,728a56c1d131b106,e1b41463beaa09d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:38:35.644 WARN 21062 --- [ocp-async-2,ea6bb7781faa52df,57a37f8f65f08366] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:35.644 INFO 21062 --- [ocp-async-2,f4f3eda875ae07f9,932324d5d9081e47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:35.983 INFO 21062 --- [prometheus-minute-schedule-1,e19bf3a7fbf2145d,69e64d46ad06d34b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:36.225 INFO 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:38:36.225 INFO 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:38:36.225 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:38:36.225 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:38:36.234 INFO 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:38:36.235 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:38:36.235 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:38:36.236 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:38:36.236 WARN 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:38:36.236 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:38:36.237 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:38:36.237 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:38:36.237 INFO 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:38:36.238 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:38:36.238 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:38:36.238 WARN 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:38:36.246 INFO 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:38:36.246 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:38:36.246 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:38:36.258 ERROR 21062 --- [ocp-schedules-21,4edba8daf086ec6c,b9e5366a3051bb23] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:38:36.356 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:36.356 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:37.644 WARN 21062 --- [ocp-async-8,41cd8adce4abf000,67f240d7251a61ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:37.644 INFO 21062 --- [ocp-async-10,cef01f67c512efee,979a23c9287c2495] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:38.504 INFO 21062 --- [ocp-schedules-24,d2a9e1f983029bef,99548803e41e45af] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:38:38.504 INFO 21062 --- [ocp-schedules-4,925ac1d45b17300a,af575e9c88f55402] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:38:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:38:38.588673+08:00, lastProcessTime=2024-11-04T15:38:28.588244+08:00 2024-11-04 15:38:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:38:38.588673+08:00 2024-11-04 15:38:38.812 INFO 21062 --- [ocp-async-7,af5f913c7d49a43d,732abcce1839d093] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:38:38.983 INFO 21062 --- [prometheus-minute-schedule-1,f377a2d3b64c1735,2d79af496af48096] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:39.334 INFO 21062 --- [ocp-schedules-3,dd9e5f201114ee8b,51252021252e98bf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:38:39.357 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:39.357 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:39.644 WARN 21062 --- [ocp-async-7,457b10c78b2c1f05,a91ffa92f1aa6bd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:39.644 INFO 21062 --- [ocp-async-5,9816d87ef76c0e51,1b6be7a7eeb4ec8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:38:40.004 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:38:41.217 INFO 21062 --- [ocp-schedules-12,d3f0f7f028c513bc,c2cd3ff05ba922ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:38:41.217 INFO 21062 --- [ocp-schedules-12,c4337de81ef6f871,6e65f6e528cb66a3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:38:41.220 INFO 21062 --- [ocp-schedules-12,c4337de81ef6f871,6e65f6e528cb66a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:41.249 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:38:41.249 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:38:41.644 WARN 21062 --- [ocp-async-2,24069732a4c0d748,acf3e4e613c67afa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:41.645 INFO 21062 --- [ocp-async-8,c8d2716aaacaf9a6,e243331e079c7ff7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:38:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:38:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:38:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:38:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:38:41.853 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:38:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:38:41.901 INFO 21062 --- [ocp-schedules-10,32e6aca6548dec5c,3d532f971750f778] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:38:41.983 INFO 21062 --- [prometheus-minute-schedule-1,6ff8da0cb1734ab0,8ae762070ba4da9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:42.357 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:42.358 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:43.645 WARN 21062 --- [ocp-async-10,d39fe673dedfa366,02e6b21ce2fe3ea3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:43.645 INFO 21062 --- [ocp-async-1,8d4f9ad79adecd14,684fbde70e4e3f49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a35cf2c0e4eff9a6,293b0633240d1086] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:45.358 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:45.359 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:45.645 WARN 21062 --- [ocp-async-7,9fbdabe180a2f71c,84482155498c7055] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:45.645 INFO 21062 --- [ocp-async-7,7a7c97d55810b76e,020376497076c8b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:45.718 INFO 21062 --- [ocp-schedules-5,023e06239a55f47e,1c5c87cf1e7ae8ad] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:38:45.722 INFO 21062 --- [ocp-schedules-5,023e06239a55f47e,1c5c87cf1e7ae8ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:45.807 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:38:45.810 ERROR 21062 --- [ocp-schedules-5,023e06239a55f47e,1c5c87cf1e7ae8ad] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:38:45.810 INFO 21062 --- [ocp-schedules-5,023e06239a55f47e,1c5c87cf1e7ae8ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:38:47.645 WARN 21062 --- [ocp-async-2,a1b385e292e5271a,4d3a26892ac38ab6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:47.645 INFO 21062 --- [ocp-async-3,6b76608e9a1fe489,3de815bdc78b6475] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:47.983 INFO 21062 --- [prometheus-minute-schedule-1,bbb7cf4e7a36d99c,14fec0a3a4251fd4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:48.360 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:48.361 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:38:48.591304+08:00, lastProcessTime=2024-11-04T15:38:38.588673+08:00 2024-11-04 15:38:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:38:48.591304+08:00 2024-11-04 15:38:48.812 INFO 21062 --- [ocp-async-10,ded6b6829f8a4aeb,c1bc289f5a8dd152] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:38:49.359 INFO 21062 --- [ocp-schedules-17,6263e86c355b3a0c,31fc35484b73f94e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:38:49.645 WARN 21062 --- [ocp-async-10,84dff4eddd2ee87f,5f258023a3e907f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:49.645 INFO 21062 --- [ocp-async-10,a803b46964fd1d7a,09ab44edd19916e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8655a131b6b13fcb,61facbed68012bad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:51.361 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:51.363 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:51.645 WARN 21062 --- [ocp-async-9,dd19e1545b00561d,2aff474676f2f2e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:51.645 INFO 21062 --- [ocp-async-7,80ee3bb4250f8908,475aa8c03ff1dfd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:51.903 INFO 21062 --- [ocp-schedules-28,ad47d9a6a39dabc8,4a434c53a3f14f4a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:38:53.355 INFO 21062 --- [ocp-schedules-16,b422a9f2a32495d9,c083a9c36d381206] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:38:53.361 INFO 21062 --- [ocp-schedules-16,b422a9f2a32495d9,c083a9c36d381206] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:38:53.361 INFO 21062 --- [ocp-schedules-16,b422a9f2a32495d9,c083a9c36d381206] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:38:53.645 WARN 21062 --- [ocp-async-5,dae9c5dbdac81f75,ff6fb2724cc2d82e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:53.645 INFO 21062 --- [ocp-async-5,a691094a1e979850,55d1deac38d06bef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:53.983 INFO 21062 --- [prometheus-minute-schedule-1,d40b6d86326ea8ed,f2ffee213c2af57b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:54.363 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:54.364 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:55.646 INFO 21062 --- [ocp-async-4,901499b4b63ef167,7360eafef418cf9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:55.646 WARN 21062 --- [ocp-async-1,3a8268a2fab7b2e3,316ddc818c08e41c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:56.983 INFO 21062 --- [prometheus-minute-schedule-1,3178b54f96ada03c,54cd54b4e3733a48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:38:57.364 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:38:57.365 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:38:57.646 WARN 21062 --- [ocp-async-9,21fc63e0ead3633e,3f44f71dd99670b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:57.646 INFO 21062 --- [ocp-async-6,f7fcff5b476b316d,454c8cb0202b8a90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:58.500 INFO 21062 --- [ocp-schedules-1,16ecf8457b909730,8e63589acaf8ef77] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:38:58.500 INFO 21062 --- [ocp-schedules-8,b595679244a4553e,f3ddf50298bf6817] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:38:58.503 INFO 21062 --- [ocp-schedules-21,f25ff0060b2c15e4,2425923e6070ee11] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:38:58.504 INFO 21062 --- [ocp-schedules-1,16ecf8457b909730,8e63589acaf8ef77] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:58.504 INFO 21062 --- [ocp-schedules-8,b595679244a4553e,f3ddf50298bf6817] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:58.504 INFO 21062 --- [ocp-schedules-26,6bfa821b17e889c5,bc387094cfb757cc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:38:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:38:58.510 INFO 21062 --- [ocp-schedules-7,737e77fbd2a3d1cf,0626280786fdce56] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:38:58.510 INFO 21062 --- [ocp-schedules-15,3e558de13a514319,6c7fa6cfe5361871] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:38:58.510 INFO 21062 --- [ocp-schedules-7,737e77fbd2a3d1cf,0626280786fdce56] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:38:58.510 INFO 21062 --- [ocp-schedules-15,3e558de13a514319,6c7fa6cfe5361871] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:38:58.510 INFO 21062 --- [ocp-schedules-17,b87992f7843baf18,c7b56a9cbb04d4d2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:38:58.510 INFO 21062 --- [ocp-schedules-17,b87992f7843baf18,c7b56a9cbb04d4d2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:38:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:38:58.589729+08:00, lastProcessTime=2024-11-04T15:38:48.591304+08:00 2024-11-04 15:38:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:38:58.589729+08:00 2024-11-04 15:38:58.812 INFO 21062 --- [ocp-async-3,3d0528ab064337eb,ff486c74d901f2dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:38:59.373 INFO 21062 --- [ocp-schedules-10,9ecacb91c9977eb3,c679690626f90f6f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:38:59.378 INFO 21062 --- [ocp-schedules-10,9ecacb91c9977eb3,c679690626f90f6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:38:59.378 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:38:59.401 INFO 21062 --- [ocp-schedules-21,928dfa6b3e6413ce,50d250bd9ebddd21] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:38:59.594 INFO 21062 --- [ocp-schedules-15,00b66a54e8e95aca,3bf22ea957045718] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:38:59.646 WARN 21062 --- [ocp-async-2,0b7b75b6cffb9aa5,38b80d6f15898bc5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:38:59.646 INFO 21062 --- [ocp-async-5,4bd74bb279009955,408437e4b75fd0aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:38:59.983 INFO 21062 --- [prometheus-minute-schedule-1,28e3340f133dea9f,64797d6dea315c31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730705940, interval=60 2024-11-04 15:39:00.365 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:00.366 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:01.646 WARN 21062 --- [ocp-async-8,1c136474484d4482,c742c55b154fb311] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:01.646 INFO 21062 --- [ocp-async-4,3e5b8252116ce234,2f389701e6c96e3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:01.906 INFO 21062 --- [ocp-schedules-15,8da2981b626a752b,26a69be5b1ae36b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:39:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4d5efff590610040,9258cdc04eaa7346] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:03.367 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:03.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:03.646 WARN 21062 --- [ocp-async-10,f7a156b86a9d8fc2,8d1280bf58ea9e5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:03.646 INFO 21062 --- [ocp-async-6,b973a0283cdd6a52,3c31a393e0c9c8a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:05.647 WARN 21062 --- [ocp-async-7,be5ad2965094ea52,3433ed3fbfd46358] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:05.647 INFO 21062 --- [ocp-async-2,a7affb0b84ab99e4,6fe61edfbed3dfb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:05.983 INFO 21062 --- [prometheus-minute-schedule-1,bf6baacb93a42f54,dec1144d552b0c0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:06.368 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:06.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:07.647 WARN 21062 --- [ocp-async-5,f5ed2bb05b279c9a,1e004ad96719c3c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:07.647 INFO 21062 --- [ocp-async-8,7aea97570ea12ca1,47fcd646b59c4b58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:08.505 INFO 21062 --- [ocp-schedules-8,b595679244a4553e,f3ddf50298bf6817] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:39:08.505 INFO 21062 --- [ocp-schedules-1,16ecf8457b909730,8e63589acaf8ef77] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:39:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:39:08.590133+08:00, lastProcessTime=2024-11-04T15:38:58.589729+08:00 2024-11-04 15:39:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:39:08.590133+08:00 2024-11-04 15:39:08.812 INFO 21062 --- [ocp-async-10,29badf57513d43f2,6d10bcd948577611] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:39:08.983 INFO 21062 --- [prometheus-minute-schedule-1,c52f337d8c84c83c,a61cff7fe611c02b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:09.369 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:09.370 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:09.378 INFO 21062 --- [ocp-schedules-10,9ecacb91c9977eb3,c679690626f90f6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:39:09.427 INFO 21062 --- [ocp-schedules-8,d09c29b6aab453b1,aaa628acdaf3e8e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:39:09.647 WARN 21062 --- [ocp-async-10,7ba838e9e88b3cf5,3f83a30a0767e202] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:09.647 INFO 21062 --- [ocp-async-4,0325301189f98c71,2aba69c7e1dcdd8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:10.003 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:39:10.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:39:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:39:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:39:10.296 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:39:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:39:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:39:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:39:11.249 INFO 21062 --- [ocp-schedules-12,c4337de81ef6f871,6e65f6e528cb66a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:39:11.250 INFO 21062 --- [ocp-schedules-23,24d182f8dddc25a2,40c067f3f91ace80] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:39:11.253 INFO 21062 --- [ocp-schedules-23,24d182f8dddc25a2,40c067f3f91ace80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:39:11.281 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:39:11.282 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:39:11.647 WARN 21062 --- [ocp-async-6,1d270b394fcec7e6,c93dfc6bf124c392] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:11.647 INFO 21062 --- [ocp-async-6,809945d41d5d8313,729eb9b62e4cc301] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:11.893 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=6, affectRows=6 2024-11-04 15:39:11.893 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=5, affectRows=5 2024-11-04 15:39:11.893 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=1, affectRows=1 2024-11-04 15:39:11.893 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=5, validCount=5 2024-11-04 15:39:11.893 INFO 21062 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=5, validCount=5 2024-11-04 15:39:11.893 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=1, validCount=1 2024-11-04 15:39:11.894 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=10, affectRows=10 2024-11-04 15:39:11.895 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=10, validCount=10 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=10, affectRows=10 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=10, validCount=10 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=17, affectRows=17 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=17, affectRows=17 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=16, affectRows=16 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=17, validCount=17 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=17, validCount=17 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=16, validCount=16 2024-11-04 15:39:11.897 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=31, affectRows=31 2024-11-04 15:39:11.898 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=31, validCount=31 2024-11-04 15:39:11.898 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=38, affectRows=38 2024-11-04 15:39:11.898 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=38, validCount=38 2024-11-04 15:39:11.899 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=71, affectRows=71 2024-11-04 15:39:11.899 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=71, validCount=71 2024-11-04 15:39:11.899 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=61, affectRows=61 2024-11-04 15:39:11.899 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=61, validCount=61 2024-11-04 15:39:11.899 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=70, affectRows=70 2024-11-04 15:39:11.899 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705880, nodeCount=69, affectRows=69 2024-11-04 15:39:11.900 INFO 21062 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=67, validCount=67 2024-11-04 15:39:11.900 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705880, nodeCount=70, validCount=70 2024-11-04 15:39:11.909 INFO 21062 --- [ocp-schedules-3,cf275f9dffdbc151,74143ba2f77d8254] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:39:11.983 INFO 21062 --- [prometheus-minute-schedule-1,2866ac572100f16f,a1a97b672c2ab320] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:12.371 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:12.372 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:13.362 INFO 21062 --- [ocp-schedules-24,3c6c73eb18d09142,a2e2b7a2ff698686] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:39:13.368 INFO 21062 --- [ocp-schedules-24,3c6c73eb18d09142,a2e2b7a2ff698686] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:39:13.368 INFO 21062 --- [ocp-schedules-24,3c6c73eb18d09142,a2e2b7a2ff698686] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:39:13.647 WARN 21062 --- [ocp-async-2,8349916972b58c98,92f406307e04c694] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:13.647 INFO 21062 --- [ocp-async-1,3b4d3a6a8766c83d,17b99056bcdeed3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,2aef65df28b72042,87d1665864ffe91d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:39:13.984 INFO 21062 --- [prometheus-cache-monitor1,2aef65df28b72042,87d1665864ffe91d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:39:14.983 INFO 21062 --- [prometheus-minute-schedule-1,b221566e6b64b400,e7fb8eac7feb3777] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:15.372 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:15.373 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:15.647 WARN 21062 --- [ocp-async-10,81364d7db1caffb7,1f61e966054ddb91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:15.648 INFO 21062 --- [ocp-async-10,f0369f2807e68185,6403fa67b49f19f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:15.745 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:35:15.727, end=2024-11-04T15:37:15.743} 2024-11-04 15:39:15.750 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:39:15.750 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:39:17.648 INFO 21062 --- [ocp-async-3,8a3304e5263b56ea,43895e815838230d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:17.648 WARN 21062 --- [ocp-async-4,baf49d3db864947d,819df909788619fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:17.983 INFO 21062 --- [prometheus-minute-schedule-1,74311efa71df9d2e,26bfb43caf7fe40d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:18.374 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:18.375 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:39:18.590136+08:00, lastProcessTime=2024-11-04T15:39:08.590133+08:00 2024-11-04 15:39:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:39:18.590136+08:00 2024-11-04 15:39:18.813 INFO 21062 --- [ocp-async-5,1e1d1d025d754dd5,d9f5ed4723eb9ed9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:39:19.470 INFO 21062 --- [ocp-schedules-10,f819ec66e35f31a4,0f1f0c7f4b563676] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:39:19.648 INFO 21062 --- [ocp-async-5,d9d4c962cd67ba11,a488d393b96a98fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:19.648 WARN 21062 --- [ocp-async-1,aef602b92e616fd9,7eab46cb9388b0d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8fb3cf59f2e0722e,8ff5adc9b427a250] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:21.375 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:21.376 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:21.648 INFO 21062 --- [ocp-async-8,83950ad8bf5f2073,9c403d3e39089a26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:21.648 WARN 21062 --- [ocp-async-10,3e9f9d9110333a74,bb36ca37a7c0913a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:21.911 INFO 21062 --- [ocp-schedules-15,50ca49d96c4850df,a3704542d2c0b461] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:39:23.648 INFO 21062 --- [ocp-async-7,fc93abbd7794b0c2,a2a0aca8bebc02d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:23.648 WARN 21062 --- [ocp-async-3,683a50844bc92768,359c4f6e09e87cd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:23.983 INFO 21062 --- [prometheus-minute-schedule-1,914283fa57ae21f8,24a8f67ecd982a39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:24.376 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:24.378 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:25.648 WARN 21062 --- [ocp-async-2,f7aface05a392dfa,2426e4ee335075c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:25.649 INFO 21062 --- [ocp-async-1,e85436e1b28ae2f5,50d6d310213e3714] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3b35a0abc8ca50f7,4354f24f1af91720] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b666a0566f5d890f,e5ac999eb23342c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:39:27.378 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:27.380 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:27.649 WARN 21062 --- [ocp-async-9,93b10658d579bc40,0567e4fe506bc8fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:27.649 INFO 21062 --- [ocp-async-10,66485024384f481f,34771e71a41dcdd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:28.498 INFO 21062 --- [ocp-schedules-18,67be57fffbb622f4,90f4c7a54b34c9af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:39:28.500 INFO 21062 --- [ocp-schedules-19,52f5b834dc505582,a122a34596770cd0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:39:28.500 INFO 21062 --- [ocp-schedules-6,58d960d2e84973a3,db37c4c545647b8d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:39:28.500 INFO 21062 --- [ocp-schedules-26,a232ed5d53e80267,64630feb50d39172] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:39:28.502 INFO 21062 --- [ocp-schedules-17,e6f77dcf28d11942,7ab528259fbf42c5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:39:28.503 INFO 21062 --- [ocp-schedules-18,67be57fffbb622f4,90f4c7a54b34c9af] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:39:28.504 INFO 21062 --- [ocp-schedules-26,a232ed5d53e80267,64630feb50d39172] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:39:28.504 INFO 21062 --- [ocp-schedules-7,fc2f38d1804f883d,b6f8897af39dfc58] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:39:28.504 INFO 21062 --- [ocp-schedules-19,52f5b834dc505582,a122a34596770cd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:39:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:39:28.505 INFO 21062 --- [ocp-schedules-6,58d960d2e84973a3,db37c4c545647b8d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:39:28.510 INFO 21062 --- [ocp-schedules-14,a68ad383cf4512c1,90785b2a2efa601a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:39:28.510 INFO 21062 --- [ocp-schedules-15,6dcd4aff5a15258c,fb16e19f96a025f9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:39:28.510 INFO 21062 --- [ocp-schedules-14,a68ad383cf4512c1,90785b2a2efa601a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:39:28.510 INFO 21062 --- [ocp-schedules-15,6dcd4aff5a15258c,fb16e19f96a025f9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:39:28.511 INFO 21062 --- [ocp-schedules-1,1313232a0ba2de16,9c637251c64705cc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:39:28.511 INFO 21062 --- [ocp-schedules-1,1313232a0ba2de16,9c637251c64705cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:39:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:39:28.589733+08:00, lastProcessTime=2024-11-04T15:39:18.590136+08:00 2024-11-04 15:39:28.627 INFO 21062 --- [ocp-schedules-3,4dec00a42db75b7b,8ca94a42f28ce728] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:39:28.627+08:00, tenantSize=0 2024-11-04 15:39:28.674 INFO 21062 --- [ocp-schedules-8,a4f1e32007630f59,71aa0f4fa35cac77] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:39:28.675 INFO 21062 --- [ocp-schedules-8,a4f1e32007630f59,71aa0f4fa35cac77] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:39:28.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:39:28.589733+08:00 2024-11-04 15:39:28.813 INFO 21062 --- [ocp-async-2,ffa18272c16a66a4,6a042e52ecdb78e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:39:29.026 INFO 21062 --- [ocp-schedules-6,b2ed5b357d436586,3ea1ee102378444b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:39:29.452 INFO 21062 --- [ocp-schedules-12,9dab38398231a432,5884b76210cf423a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:39:29.452 INFO 21062 --- [ocp-schedules-12,9dab38398231a432,5884b76210cf423a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:39:29.495 INFO 21062 --- [ocp-schedules-3,b19ccf2d20aca92e,1d61235f203c80fb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:39:29.596 INFO 21062 --- [ocp-schedules-27,02d62a3745497ba1,b17df7c065f244a6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:39:29.613 INFO 21062 --- [ocp-schedules-24,44de2d8427fa8a83,1c380d555f558e71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:39:29.618 INFO 21062 --- [ocp-schedules-24,44de2d8427fa8a83,1c380d555f558e71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:39:29.618 INFO 21062 --- [ocp-schedules-24,44de2d8427fa8a83,1c380d555f558e71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:39:29.649 INFO 21062 --- [ocp-async-2,e9e17e8fd3c9a148,0e36f97ef4a8be19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:29.649 WARN 21062 --- [ocp-async-6,8075a525065f7046,ece78ccd749f7353] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:29.983 INFO 21062 --- [prometheus-minute-schedule-1,8ecbd2c93cb659d8,417169e879cdb970] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:30.123 INFO 21062 --- [ocp-schedules-2,a591ff1e6c8aa236,c8acbfb3234378c2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:39:30.123 INFO 21062 --- [ocp-schedules-2,a591ff1e6c8aa236,c8acbfb3234378c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:39:30.125 INFO 21062 --- [ocp-schedules-2,a591ff1e6c8aa236,c8acbfb3234378c2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:39:30.125 INFO 21062 --- [ocp-schedules-2,a591ff1e6c8aa236,c8acbfb3234378c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:39:30.125 INFO 21062 --- [ocp-schedules-2,a591ff1e6c8aa236,c8acbfb3234378c2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:39:30.125 INFO 21062 --- [ocp-schedules-2,a591ff1e6c8aa236,c8acbfb3234378c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:39:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:39:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:39:30.380 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:30.381 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:31.649 WARN 21062 --- [ocp-async-5,06224f4dbf3f2f89,c5130f5550d68028] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:31.649 INFO 21062 --- [ocp-async-9,9688019f284f2481,032a96e9f09b6a1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:31.914 INFO 21062 --- [ocp-schedules-5,81dbc5d34a352a6f,b87e12c3205e3c92] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:39:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f6fbbd7d52d01b6d,05cb1a277be3765e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:33.371 INFO 21062 --- [ocp-schedules-22,5df3ddb20fe6a8e2,c48117b83202672f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:39:33.378 INFO 21062 --- [ocp-schedules-22,5df3ddb20fe6a8e2,c48117b83202672f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:39:33.378 INFO 21062 --- [ocp-schedules-22,5df3ddb20fe6a8e2,c48117b83202672f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:39:33.381 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:33.382 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:33.647 INFO 21062 --- [ocp-schedules-12,83ff140f89ecfe63,66876888df09f922] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:39:33.647 INFO 21062 --- [ocp-schedules-12,83ff140f89ecfe63,66876888df09f922] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:39:33.649 INFO 21062 --- [ocp-async-10,a7b3f1a380b3e19a,bac00a51a8b85aaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:33.649 WARN 21062 --- [ocp-async-3,11e53f5df5272bee,7820a60571a1ea5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:33.673 INFO 21062 --- [ocp-schedules-12,83ff140f89ecfe63,66876888df09f922] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:39:33.673 INFO 21062 --- [ocp-schedules-12,83ff140f89ecfe63,66876888df09f922] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:39:33.674 INFO 21062 --- [ocp-schedules-25,89eb79e10575b4df,3aae1eb3dba44501] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:39:33.883 INFO 21062 --- [ocp-schedules-6,440640e3ef7bfbe2,672b26dbaaeccbf2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:39:33.883+08:00, clusters=[1] 2024-11-04 15:39:33.885 INFO 21062 --- [ocp-schedules-6,440640e3ef7bfbe2,672b26dbaaeccbf2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:39:34.228 INFO 21062 --- [ocp-schedules-1,1e0bd37fd1bc7374,913a213616859eab] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:39:34.228+08:00, clusters=[1] 2024-11-04 15:39:34.232 INFO 21062 --- [ocp-schedules-1,1e0bd37fd1bc7374,913a213616859eab] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:39:35.649 WARN 21062 --- [ocp-async-6,39a0b7172478fad7,d4cd26338f62c256] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:35.649 INFO 21062 --- [ocp-async-7,a4987351a48d1ed5,43ee373dd84a2eec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:35.983 INFO 21062 --- [prometheus-minute-schedule-1,d2634d2ad488676c,d8e30c5644a13309] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:36.269 INFO 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:39:36.269 INFO 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:39:36.269 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:39:36.269 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:39:36.278 INFO 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:39:36.279 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:39:36.279 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:39:36.280 WARN 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:39:36.280 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:39:36.280 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:39:36.281 INFO 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:39:36.281 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:39:36.281 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:39:36.282 WARN 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:39:36.282 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:39:36.282 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:39:36.289 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:39:36.289 INFO 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:39:36.289 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:39:36.302 ERROR 21062 --- [ocp-schedules-10,432e1492637219ea,2956499a25034ff3] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:39:36.382 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:36.384 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:37.650 WARN 21062 --- [ocp-async-5,8c0c532955376f9c,ee114c637ad7dc5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:37.650 INFO 21062 --- [ocp-async-1,dac4c3b677e88abe,507b7d8d7d446206] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:38.504 INFO 21062 --- [ocp-schedules-26,a232ed5d53e80267,64630feb50d39172] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:39:38.505 INFO 21062 --- [ocp-schedules-19,52f5b834dc505582,a122a34596770cd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:39:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:39:38.590793+08:00, lastProcessTime=2024-11-04T15:39:28.589733+08:00 2024-11-04 15:39:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:39:38.590793+08:00 2024-11-04 15:39:38.813 INFO 21062 --- [ocp-async-10,8830aa191fc3e8d9,4421b9640e3d7fb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:39:38.983 INFO 21062 --- [prometheus-minute-schedule-1,710486a38cce53d5,57cee089fc375014] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:39.384 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:39.385 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:39.540 INFO 21062 --- [ocp-schedules-15,c33b1e08dc0f8921,58aea59c3a6af7c9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:39:39.650 INFO 21062 --- [ocp-async-10,4457c4027d3a9939,7a63a2406570fc3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:39.650 WARN 21062 --- [ocp-async-6,ce39be4166ae83db,e840a0dc7c74d45b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:39:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:39:41.282 INFO 21062 --- [ocp-schedules-23,24d182f8dddc25a2,40c067f3f91ace80] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:39:41.282 INFO 21062 --- [ocp-schedules-23,1046e95bab36c11f,e581b8696ea2511c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:39:41.286 INFO 21062 --- [ocp-schedules-23,1046e95bab36c11f,e581b8696ea2511c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:39:41.316 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:39:41.316 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:39:41.650 WARN 21062 --- [ocp-async-7,f3b3702577572097,86b6d4fc80938bd9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:41.650 INFO 21062 --- [ocp-async-5,dc29247636939ba4,eb6f1d3c6c28237c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:41.917 INFO 21062 --- [ocp-schedules-15,60eb80378ce11497,8c9023f32bdf2e7a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:39:41.983 INFO 21062 --- [prometheus-minute-schedule-1,947cdc49f6f1c0d2,4f633be5de20d6b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:42.385 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:42.387 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:43.650 WARN 21062 --- [ocp-async-1,7baf94667cb0ca99,412c95179cbae885] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:43.650 INFO 21062 --- [ocp-async-1,2c196075f6f57eff,76db8d88498e3d51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:44.983 INFO 21062 --- [prometheus-minute-schedule-1,bf5200d667c684a2,c652d4b9fd59e69a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:45.387 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:45.388 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:45.650 WARN 21062 --- [ocp-async-6,5910fd8e1b8648eb,4f234a08a4fabe36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:45.650 INFO 21062 --- [ocp-async-8,fdf5ed6b040d2ee4,959f9b9c45ac6273] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:45.811 INFO 21062 --- [ocp-schedules-19,2769d1dbb14ee125,c51bf9fdc00b38df] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:39:45.814 INFO 21062 --- [ocp-schedules-19,2769d1dbb14ee125,c51bf9fdc00b38df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:39:45.899 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:39:45.902 ERROR 21062 --- [ocp-schedules-19,2769d1dbb14ee125,c51bf9fdc00b38df] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:39:45.902 INFO 21062 --- [ocp-schedules-19,2769d1dbb14ee125,c51bf9fdc00b38df] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:39:47.650 WARN 21062 --- [ocp-async-7,14e6d772a9cee1b7,f01fb0710533a702] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:47.651 INFO 21062 --- [ocp-async-4,7b76f53a6ba862cc,e9a6c035c1a49bff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:47.983 INFO 21062 --- [prometheus-minute-schedule-1,7f27c4924f42c94e,b3a9964a7cfc5791] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:48.388 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:48.390 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:39:48.591514+08:00, lastProcessTime=2024-11-04T15:39:38.590793+08:00 2024-11-04 15:39:48.777 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:39:48.591514+08:00 2024-11-04 15:39:48.813 INFO 21062 --- [ocp-async-9,b36ac46be55481bb,0f5b3212b29f826d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:39:49.565 INFO 21062 --- [ocp-schedules-19,28a84526f6497883,7e94f3906bac76ef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:39:49.651 INFO 21062 --- [ocp-async-2,fe43b1408b713990,0a65d0d67167df0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:49.651 WARN 21062 --- [ocp-async-1,afc5e7402474c543,895f73d687d03126] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:50.983 INFO 21062 --- [prometheus-minute-schedule-1,64a18d1f383d0220,e4bbecc99a4fd76b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:51.390 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:51.391 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:51.651 INFO 21062 --- [ocp-async-3,98bbb3a58ed58be2,c39784122a15bd8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:51.651 WARN 21062 --- [ocp-async-8,a1fe958600ab6afc,1014ebf17c683d56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:51.921 INFO 21062 --- [ocp-schedules-14,4fdd9210d794a2eb,a756705680f69bb4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:39:53.381 INFO 21062 --- [ocp-schedules-18,cdb3163db805d27b,5fc9b9d131b157ab] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:39:53.387 INFO 21062 --- [ocp-schedules-18,cdb3163db805d27b,5fc9b9d131b157ab] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:39:53.387 INFO 21062 --- [ocp-schedules-18,cdb3163db805d27b,5fc9b9d131b157ab] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:39:53.651 INFO 21062 --- [ocp-async-10,95ad5df9c4a9a4b6,55abe1285d27f5c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:53.651 WARN 21062 --- [ocp-async-4,ab24e67bf8d81c3d,fd8089a61b35dc3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:53.983 INFO 21062 --- [prometheus-minute-schedule-1,27c2c2391859687b,4d9de03ca845ef36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:54.391 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:54.393 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:55.651 INFO 21062 --- [ocp-async-1,2ae56163f3874639,1e9756b54e48c3f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:55.651 WARN 21062 --- [ocp-async-9,90b536cfe0d4ee6d,892c1d0bafddaf5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:56.983 INFO 21062 --- [prometheus-minute-schedule-1,42862168850186ca,6a9cd0d5b018173e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:39:57.393 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:39:57.394 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:39:57.651 INFO 21062 --- [ocp-async-8,b6cff2904c1c52f2,be7209b1d49621b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:57.651 WARN 21062 --- [ocp-async-6,d45ef915bfe76c7a,135ebc54f538f619] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:58.500 INFO 21062 --- [ocp-schedules-30,6312446d4894efaf,2276ecef9f3c2c83] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:39:58.500 INFO 21062 --- [ocp-schedules-19,16d4234b222c80a2,462d742fc0d990e2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:39:58.502 INFO 21062 --- [ocp-schedules-8,ef44d3e83267c9fe,1bf3b64376f280d0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:39:58.504 INFO 21062 --- [ocp-schedules-17,308405dbcfdca408,56693d53f3f20122] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:39:58.505 INFO 21062 --- [ocp-schedules-30,6312446d4894efaf,2276ecef9f3c2c83] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:39:58.505 INFO 21062 --- [ocp-schedules-19,16d4234b222c80a2,462d742fc0d990e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:39:58.506 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:39:58.509 INFO 21062 --- [ocp-schedules-18,31218455568a6024,85294c1afa6f1573] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:39:58.509 INFO 21062 --- [ocp-schedules-22,2efdfbdc954a9acc,5f9b6d191c1fd365] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:39:58.509 INFO 21062 --- [ocp-schedules-2,8e97286ade39cfd3,3b7ff071ffe82657] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:39:58.509 INFO 21062 --- [ocp-schedules-18,31218455568a6024,85294c1afa6f1573] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:39:58.509 INFO 21062 --- [ocp-schedules-22,2efdfbdc954a9acc,5f9b6d191c1fd365] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:39:58.509 INFO 21062 --- [ocp-schedules-2,8e97286ade39cfd3,3b7ff071ffe82657] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:39:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:39:58.590137+08:00, lastProcessTime=2024-11-04T15:39:48.591514+08:00 2024-11-04 15:39:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:39:58.590137+08:00 2024-11-04 15:39:58.814 INFO 21062 --- [ocp-async-4,acac2f349c99a8ad,3432f2240b2be1f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:39:59.599 INFO 21062 --- [ocp-schedules-29,bf7f7916bdc7d4e5,7279f8d014083aa3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:39:59.614 INFO 21062 --- [ocp-schedules-18,45fe34053cddb098,1c64929b056f8a20] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:39:59.651 INFO 21062 --- [ocp-async-7,b1a97c22f2da0136,ac657a9201d2bf01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:39:59.652 WARN 21062 --- [ocp-async-10,43c4180b138541f3,8233c635b084fc38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:39:59.983 INFO 21062 --- [prometheus-minute-schedule-1,37962a6c581353f2,a5e5962007166725] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706000, interval=60 2024-11-04 15:40:00.394 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:00.395 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:00.904 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,e6b678a07a2ab038] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:35:00.0, nextRun=2024-11-04 15:40:00.0, currentDate=Mon Nov 04 15:40:00 CST 2024 2024-11-04 15:40:00.904 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,e6b678a07a2ab038] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:35:00.0, nextRunTime=2024-11-04 15:40:00.0} 2024-11-04 15:40:00.930 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,e6b678a07a2ab038] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:40:00.935 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,e6b678a07a2ab038] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:40:00.935 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,e6b678a07a2ab038] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:40:01.652 WARN 21062 --- [ocp-async-1,8881dd4508d5efde,091c339b995ed90e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:01.652 INFO 21062 --- [ocp-async-5,a8aa63bfd6416cac,7987a0c560dfe219] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:01.924 INFO 21062 --- [ocp-schedules-26,6ef9dd38c8b4266b,cbf156391cdaa6ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:40:02.983 INFO 21062 --- [prometheus-minute-schedule-1,609b7a1b6ea4fd54,5fda3dd668d8be19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:03.395 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:03.397 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:03.652 WARN 21062 --- [ocp-async-2,97f031927cfd56df,c21497ba91bd9a4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:03.652 INFO 21062 --- [ocp-async-2,5e6ef52569e2543a,a020b8d4d7ba633b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:05.002 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,91ce078ac19565db] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:30:05.0, nextRun=2024-11-04 15:40:04.0, currentDate=Mon Nov 04 15:40:05 CST 2024 2024-11-04 15:40:05.002 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,91ce078ac19565db] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 15:30:05.0, nextRunTime=2024-11-04 15:40:04.0} 2024-11-04 15:40:05.008 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,91ce078ac19565db] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 15:40:05.008 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,91ce078ac19565db] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T15:40:05.008+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 15:40:05.652 INFO 21062 --- [ocp-async-3,3b1ac42e2fd86448,ac5e484cc1a9b4de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:05.652 WARN 21062 --- [ocp-async-10,cde882edb93c4102,18eb180792d4d0eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:05.664 INFO 21062 --- [subtask-coordinator2,dd193f8f67a5a423,622381bfa005202e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 89, operation:EXECUTE, state: RUNNING 2024-11-04 15:40:05.683 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=89, context=Context{parallelIdx=-1, stringMap={task_instance_id=26, task_operation=execute, latest_execution_start_time=2024-11-04T15:40:05.656+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=89}, listMap={}}, executor=10.46.128.24 2024-11-04 15:40:05.688 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:40:05.688 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:40:05.689 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.690 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:40:05.691 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.691 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.692 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.692 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:40:05.708 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.708 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.709 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.709 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:40:05.722 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.722 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.723 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.723 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:40:05.737 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.737 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.738 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.738 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:40:05.752 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.752 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.753 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.754 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:40:05.759 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.760 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.760 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.761 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:40:05.766 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.766 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.767 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.767 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:40:05.773 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.773 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.774 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.774 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:40:05.780 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.781 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.781 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.781 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:40:05.786 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.787 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.788 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.788 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:40:05.793 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.794 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.795 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.795 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:40:05.804 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.804 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.805 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.805 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:40:05.818 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.818 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.819 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.819 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:40:05.823 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.824 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.824 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.825 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:40:05.830 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.831 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.831 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.832 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:40:05.838 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.838 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.839 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.839 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:40:05.843 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.844 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.844 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.845 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:40:05.849 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.850 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.850 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.851 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:40:05.855 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.855 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.856 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.857 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:40:05.862 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.862 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.863 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.863 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:40:05.867 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.867 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.868 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.868 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:40:05.873 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.873 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.874 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.874 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:40:05.878 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.878 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.879 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.880 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:40:05.884 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.884 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.885 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.885 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:40:05.889 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.890 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.891 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.891 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:40:05.895 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.895 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.896 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.896 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:40:05.901 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.901 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.902 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.902 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:40:05.906 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.907 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.907 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.908 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:40:05.912 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.912 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.913 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.913 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:40:05.917 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.918 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.918 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.919 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:40:05.922 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.923 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.924 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.924 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:40:05.928 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.928 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.929 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.929 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:40:05.933 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.934 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.934 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.935 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:40:05.939 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.940 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.940 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.940 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:40:05.947 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.948 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.948 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.949 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:40:05.955 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.955 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.956 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.956 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:40:05.962 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.962 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.963 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.963 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:40:05.968 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.968 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.969 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.969 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:40:05.973 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.973 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.974 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.974 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:40:05.977 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.978 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.979 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.979 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:40:05.982 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.983 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.983 INFO 21062 --- [prometheus-minute-schedule-1,edc7bd70ce123e6e,3f22f6cd1c752b21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:05.983 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.984 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:40:05.987 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.987 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.988 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.988 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:40:05.997 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.998 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:05.998 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:05.998 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:40:06.019 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.019 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.020 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.020 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:40:06.025 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.026 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.027 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.027 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:40:06.033 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.033 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.034 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.034 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:40:06.040 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.040 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.040 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.041 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:40:06.045 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.045 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.046 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.046 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:40:06.050 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.051 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.051 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.052 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:40:06.056 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:40:06.056 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:40:06.057 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.057 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:40:06.058 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.059 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.059 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.060 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:40:06.065 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.065 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:40:06.066 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:06.066 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:40:06.070 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:40:06.078 INFO 21062 --- [subtask-executor22,dd193f8f67a5a423,dae05c0cd9a5602a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 89, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 15:40:06.397 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:06.398 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:06.976 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:26, status:SUCCESS 2024-11-04 15:40:06.977 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:26, status:SUCCESS 2024-11-04 15:40:07.652 INFO 21062 --- [ocp-async-7,d243cb324966b39f,66ba1ea3455037e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:07.652 WARN 21062 --- [ocp-async-1,92b137418025cb36,ff0d55896ef79548] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:08.506 INFO 21062 --- [ocp-schedules-19,16d4234b222c80a2,462d742fc0d990e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:40:08.506 INFO 21062 --- [ocp-schedules-30,6312446d4894efaf,2276ecef9f3c2c83] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:40:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:40:08.589730+08:00, lastProcessTime=2024-11-04T15:39:58.590137+08:00 2024-11-04 15:40:08.806 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:40:08.589730+08:00 2024-11-04 15:40:08.814 INFO 21062 --- [ocp-async-8,43d68cad8c52a5d7,2f408476960b5a48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:40:08.983 INFO 21062 --- [prometheus-minute-schedule-1,c58acbe865b91b8a,4de26c8aa16cb8e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:09.379 INFO 21062 --- [ocp-schedules-1,65ebfb57d654a99f,4b69473fffeef1c4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:40:09.383 INFO 21062 --- [ocp-schedules-1,65ebfb57d654a99f,4b69473fffeef1c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:09.384 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:40:09.398 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:09.399 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:09.641 INFO 21062 --- [ocp-schedules-30,908d0668b2550f1a,cd26c597ffd1adc5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:40:09.652 INFO 21062 --- [ocp-async-8,27fcf73f78f37a80,f685aea4a355bb3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:09.653 WARN 21062 --- [ocp-async-5,452931b62bd27017,61f957b06245a649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:40:10.005 INFO 21062 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:40:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:40:10.296 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:40:10.296 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:40:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:40:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:40:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:40:11.316 INFO 21062 --- [ocp-schedules-23,1046e95bab36c11f,e581b8696ea2511c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:40:11.316 INFO 21062 --- [ocp-schedules-23,b7091706ddeb4a77,022c3f3fa4c27df6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:40:11.322 INFO 21062 --- [ocp-schedules-23,b7091706ddeb4a77,022c3f3fa4c27df6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:11.352 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:40:11.352 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:40:11.653 INFO 21062 --- [ocp-async-4,7507c3a7e1dda3fd,7981e1a44c64705f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:11.653 WARN 21062 --- [ocp-async-2,309c8e3415e42c8f,32d027867b41d75c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:11.891 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=5, affectRows=5 2024-11-04 15:40:11.891 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=5, validCount=5 2024-11-04 15:40:11.891 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=24, affectRows=24 2024-11-04 15:40:11.892 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=23, validCount=23 2024-11-04 15:40:11.892 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=40, affectRows=40 2024-11-04 15:40:11.892 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=31, affectRows=31 2024-11-04 15:40:11.892 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=17, affectRows=17 2024-11-04 15:40:11.892 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=17, validCount=17 2024-11-04 15:40:11.892 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=31, validCount=31 2024-11-04 15:40:11.892 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=40, validCount=40 2024-11-04 15:40:11.893 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=62, affectRows=62 2024-11-04 15:40:11.893 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=74, affectRows=74 2024-11-04 15:40:11.893 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=62, validCount=62 2024-11-04 15:40:11.893 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=74, validCount=74 2024-11-04 15:40:11.893 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=84, affectRows=84 2024-11-04 15:40:11.894 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730705940, nodeCount=85, affectRows=85 2024-11-04 15:40:11.894 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=82, validCount=82 2024-11-04 15:40:11.894 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730705940, nodeCount=85, validCount=85 2024-11-04 15:40:11.927 INFO 21062 --- [ocp-schedules-27,e68fce24fcba75b0,d07e34e09aeb2de3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:40:11.983 INFO 21062 --- [prometheus-minute-schedule-1,2e3b8ba3d0e2029f,ab7df364aacaa838] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:12.400 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:12.401 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:13.389 INFO 21062 --- [ocp-schedules-17,961cb3fd92de9861,6bd486c456169f04] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:40:13.394 INFO 21062 --- [ocp-schedules-17,961cb3fd92de9861,6bd486c456169f04] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:40:13.394 INFO 21062 --- [ocp-schedules-17,961cb3fd92de9861,6bd486c456169f04] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:40:13.653 INFO 21062 --- [ocp-async-3,76bbe205e9a772ee,bb5b4be824970ed0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:13.653 WARN 21062 --- [ocp-async-9,41322d6038608722,658a44ba84bf67c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,3ec03b0a8e13b42a,a9774148161247d6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:40:13.984 INFO 21062 --- [prometheus-cache-monitor1,3ec03b0a8e13b42a,a9774148161247d6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:40:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9d4651a4b424d933,c8437710ee49c88d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:15.401 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:15.402 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:15.653 INFO 21062 --- [ocp-async-5,310120c2e80f3fd0,97fbc7fc1d47935e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:15.653 WARN 21062 --- [ocp-async-6,9dfd4ffea7793e9c,08b3142a2eab0b34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:17.653 INFO 21062 --- [ocp-async-2,6e0a638fade02967,319dfb134855eb97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:17.653 WARN 21062 --- [ocp-async-10,eafe476ee42ab5db,ab89600a22754375] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:17.983 INFO 21062 --- [prometheus-minute-schedule-1,5c7c538475e263f9,74b2e1f4735651da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:18.403 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:18.404 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:40:18.589728+08:00, lastProcessTime=2024-11-04T15:40:08.589730+08:00 2024-11-04 15:40:18.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:40:18.589728+08:00 2024-11-04 15:40:18.814 INFO 21062 --- [ocp-async-1,4e5ebfc015a9b877,245e73537fdb7d4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:40:19.384 INFO 21062 --- [ocp-schedules-1,65ebfb57d654a99f,4b69473fffeef1c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:40:19.653 INFO 21062 --- [ocp-async-1,fd68ecea1633fbe3,64f3feea0e81ee0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:19.653 WARN 21062 --- [ocp-async-3,9d404fbf0079124c,51da18f07f5fb717] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:19.685 INFO 21062 --- [ocp-schedules-7,4ce92faefb16230f,9335e51daa74630c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:40:20.983 INFO 21062 --- [prometheus-minute-schedule-1,38510229c4125196,5e94bedc53885da5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:21.404 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:21.406 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:21.653 WARN 21062 --- [ocp-async-5,3ed738852baa2065,68971f02c607361c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:21.654 INFO 21062 --- [ocp-async-7,921b82455970cdeb,1838f29728dae08e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:21.929 INFO 21062 --- [ocp-schedules-22,713de9ff2beceb4d,72b2be20b30f464a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:40:23.654 INFO 21062 --- [ocp-async-8,1b7e5eba14e5f63d,d9fe7263f6b1933b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:23.654 WARN 21062 --- [ocp-async-2,5a9429af31e658cc,ba7b9ab6b73b0c7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:23.983 INFO 21062 --- [prometheus-minute-schedule-1,13a15207d70a7f1e,dff22f2e28732399] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:24.406 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:24.407 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:25.654 INFO 21062 --- [ocp-async-3,7c233f3a299f7cc9,bb5e9893f8d57216] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:25.654 WARN 21062 --- [ocp-async-9,865a24b27f28f252,665768d3352a28db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e7c6cad0dae13591,3530d6a75ca4018f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:26.984 INFO 21062 --- [prometheus-hour-schedule-1,f5386ef9b0839f1c,316c683b14ab8819] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:40:27.407 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:27.408 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:27.654 WARN 21062 --- [ocp-async-6,6593af0fba77d059,e9d5cda3fe80d607] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:27.654 INFO 21062 --- [ocp-async-5,6305cd3dbe71b0b2,420890398dfb87fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:28.498 INFO 21062 --- [ocp-schedules-12,609ce99b90e72413,5ee81c4017113363] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:40:28.499 INFO 21062 --- [ocp-schedules-13,313163b360fdbaf8,c4e72c4cd2d915e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:40:28.499 INFO 21062 --- [ocp-schedules-14,f75f498a3c1f946c,704a67ca91833859] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:40:28.500 INFO 21062 --- [ocp-schedules-25,7781cb28b881a442,9c101a2e766967b0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:40:28.502 INFO 21062 --- [ocp-schedules-12,609ce99b90e72413,5ee81c4017113363] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:40:28.502 INFO 21062 --- [ocp-schedules-15,f12bd1e0b7933fa4,5cb20f2752a730de] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:40:28.503 INFO 21062 --- [ocp-schedules-14,f75f498a3c1f946c,704a67ca91833859] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:28.504 INFO 21062 --- [ocp-schedules-25,7781cb28b881a442,9c101a2e766967b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:40:28.504 INFO 21062 --- [ocp-schedules-5,d64303548a090ba0,a5c86d82a1a11a0e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:40:28.506 INFO 21062 --- [ocp-schedules-13,313163b360fdbaf8,c4e72c4cd2d915e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:40:28.511 INFO 21062 --- [ocp-schedules-28,b81b395f9fb4f169,e713eff63a56579c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:40:28.511 INFO 21062 --- [ocp-schedules-27,029a79b7ee4fbd3c,67a543e0d73475d3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:40:28.511 INFO 21062 --- [ocp-schedules-22,82a6d19d9f3a57b2,686be06b87bbb97f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:40:28.511 INFO 21062 --- [ocp-schedules-28,b81b395f9fb4f169,e713eff63a56579c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:40:28.511 INFO 21062 --- [ocp-schedules-27,029a79b7ee4fbd3c,67a543e0d73475d3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:40:28.511 INFO 21062 --- [ocp-schedules-22,82a6d19d9f3a57b2,686be06b87bbb97f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:40:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:40:28.588198+08:00, lastProcessTime=2024-11-04T15:40:18.589728+08:00 2024-11-04 15:40:28.591 INFO 21062 --- [ocp-schedules-8,fae9af90b4c3d694,ec8994ae4bfcadc8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:40:28.591 INFO 21062 --- [ocp-schedules-8,fae9af90b4c3d694,ec8994ae4bfcadc8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:40:28.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:40:28.588198+08:00 2024-11-04 15:40:28.814 INFO 21062 --- [ocp-async-9,ad5f149e5ca8611b,3333e87850ab9187] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:40:29.028 INFO 21062 --- [ocp-schedules-30,977b01b7b1f21bd5,4afa4e186ac8cdc4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:40:29.457 INFO 21062 --- [ocp-schedules-12,d25ffb066af125be,9f52184e6b100d29] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:40:29.457 INFO 21062 --- [ocp-schedules-12,d25ffb066af125be,9f52184e6b100d29] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:40:29.602 INFO 21062 --- [ocp-schedules-24,393fb6cfdbe2b687,80630b18f7846db5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:40:29.619 INFO 21062 --- [ocp-schedules-18,fdb005785ab7bfb9,4eb4fa7e885f5e4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:40:29.624 INFO 21062 --- [ocp-schedules-18,fdb005785ab7bfb9,4eb4fa7e885f5e4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:40:29.624 INFO 21062 --- [ocp-schedules-18,fdb005785ab7bfb9,4eb4fa7e885f5e4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:40:29.654 INFO 21062 --- [ocp-async-9,2d2d9232a05243df,e26c4f7674d06447] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:29.654 WARN 21062 --- [ocp-async-4,7ba80dcf1cf31503,e2f740e64bc0b17d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:29.711 INFO 21062 --- [ocp-schedules-19,8ca31303792c2c5b,f0bfb108dac411d4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:40:29.954 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:40:29.954 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:40:29.963 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:40:29.963 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:40:29.963 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:40:29.973 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:40:29.973 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:40:29.973 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:40:29.976 INFO 21062 --- [ocp-schedules-11,a4c0a1b111a15a00,3e53ea8198dacb5b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:40:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b90d01213385ae31,84f638f3de66f7ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:30.128 INFO 21062 --- [ocp-schedules-16,084313fd7591b137,49546a7c23df4629] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:40:30.128 INFO 21062 --- [ocp-schedules-16,084313fd7591b137,49546a7c23df4629] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:40:30.131 INFO 21062 --- [ocp-schedules-16,084313fd7591b137,49546a7c23df4629] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:40:30.131 INFO 21062 --- [ocp-schedules-16,084313fd7591b137,49546a7c23df4629] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:40:30.132 INFO 21062 --- [ocp-schedules-16,084313fd7591b137,49546a7c23df4629] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:40:30.132 INFO 21062 --- [ocp-schedules-16,084313fd7591b137,49546a7c23df4629] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:40:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:40:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:40:30.408 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:30.409 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:31.654 INFO 21062 --- [ocp-async-6,806b463bd88758a3,a7a55e8670f0b2f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:31.655 WARN 21062 --- [ocp-async-5,6c03a6c4e90a2439,63dc84f8daf27578] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:31.932 INFO 21062 --- [ocp-schedules-24,136f33f8bb1bfc5d,a7afda52b2ee3dc8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:40:32.983 INFO 21062 --- [prometheus-minute-schedule-1,da57c2eb0ffc0860,c0087cc139062fe0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:33.012 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:40:33.012 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:40:33.020 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:40:33.020 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:40:33.020 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:40:33.021 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:40:33.027 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:40:33.027 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:40:33.027 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:40:33.027 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:40:33.064 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:40:33.073 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:40:33.073 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:40:33.073 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:40:33.073 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:40:33.073 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:40:33.081 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:40:33.397 INFO 21062 --- [ocp-schedules-18,0504f48abaa447cc,72862d699a159f3a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:40:33.402 INFO 21062 --- [ocp-schedules-18,0504f48abaa447cc,72862d699a159f3a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:40:33.402 INFO 21062 --- [ocp-schedules-18,0504f48abaa447cc,72862d699a159f3a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:40:33.410 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:33.411 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:33.597 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:40:33.655 INFO 21062 --- [ocp-async-1,7777ddc8110a91e9,14dbc773801d5cb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:33.655 WARN 21062 --- [ocp-async-2,9e76071073c58c14,dbf4f69731ff02f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:33.673 INFO 21062 --- [ocp-schedules-28,59684979699a2df2,71363bffdb4bd0c8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:40:33.673 INFO 21062 --- [ocp-schedules-28,59684979699a2df2,71363bffdb4bd0c8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:40:33.700 INFO 21062 --- [ocp-schedules-28,59684979699a2df2,71363bffdb4bd0c8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:40:33.700 INFO 21062 --- [ocp-schedules-28,59684979699a2df2,71363bffdb4bd0c8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:40:33.700 INFO 21062 --- [ocp-schedules-30,8f998793a968d536,44e1a2bf8c5f19fe] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:40:33.911 INFO 21062 --- [ocp-schedules-27,26122f7ec80d993a,d0939f5366cd4a24] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:40:33.911+08:00, clusters=[1] 2024-11-04 15:40:33.913 INFO 21062 --- [ocp-schedules-27,26122f7ec80d993a,d0939f5366cd4a24] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:40:34.157 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.130337261S 2024-11-04 15:40:34.157 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:40:34.253 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:40:34.253 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:40:34.258 INFO 21062 --- [ocp-schedules-7,77e4ca493c9c4265,d210bdd00249dd10] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:40:34.258+08:00, clusters=[1] 2024-11-04 15:40:34.262 INFO 21062 --- [ocp-schedules-7,77e4ca493c9c4265,d210bdd00249dd10] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:40:34.281 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:40:34.338 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:40:34.338 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:40:34.338 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:40:34.338 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:40:34.338 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:40:34.338 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:40:34.349 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:40:34.425 ERROR 21062 --- [ocp-schedules-1,37a318e9c9eeb209,f12bdb4564917800] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:40:34.425 INFO 21062 --- [ocp-schedules-1,37a318e9c9eeb209,f12bdb4564917800] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:40:34.562 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] o.s.boot.SpringApplication : Started application in 1.542 seconds (JVM running for 11448.242) 2024-11-04 15:40:34.562 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.542291807S 2024-11-04 15:40:35.021 INFO 21062 --- [ocp-schedules-9,48fbcdd08a0f7bd5,73c73c6c16d38265] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:40:35.655 WARN 21062 --- [ocp-async-8,940b19fa3473afd6,92f8556990815812] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:35.655 INFO 21062 --- [ocp-async-4,0606fc267f77c8e2,1621fe78184e6a55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:35.983 INFO 21062 --- [prometheus-minute-schedule-1,9cd9664de3b0a871,cf8e1ff62db28e9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:36.311 INFO 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:40:36.312 INFO 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:40:36.312 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:40:36.312 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:40:36.320 INFO 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:40:36.321 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:40:36.321 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:40:36.322 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:40:36.322 WARN 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:40:36.323 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:40:36.323 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:40:36.323 INFO 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:40:36.323 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:40:36.324 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:40:36.324 WARN 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:40:36.325 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:40:36.331 INFO 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:40:36.331 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:40:36.331 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:40:36.341 ERROR 21062 --- [ocp-schedules-17,41facc5f7099d1ec,8901a4d71f261720] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:40:36.411 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:36.412 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:37.655 WARN 21062 --- [ocp-async-5,648855af8e87ac9c,3540990090443484] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:37.655 INFO 21062 --- [ocp-async-3,4cb98b4f5ee6d748,f767f19f097f32c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:38.504 INFO 21062 --- [ocp-schedules-25,7781cb28b881a442,9c101a2e766967b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:40:38.504 INFO 21062 --- [ocp-schedules-14,f75f498a3c1f946c,704a67ca91833859] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:40:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:40:38.589711+08:00, lastProcessTime=2024-11-04T15:40:28.588198+08:00 2024-11-04 15:40:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:40:38.589711+08:00 2024-11-04 15:40:38.814 INFO 21062 --- [ocp-async-7,f79fda66d9b78d1d,311f7bccaa552e0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:40:38.983 INFO 21062 --- [prometheus-minute-schedule-1,aef464681e3f2494,665cc2493da0fc3b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:39.412 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:39.414 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:39.655 WARN 21062 --- [ocp-async-7,a6f5df5370f4d195,c3240dbf61422a62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:39.655 INFO 21062 --- [ocp-async-7,133803605952462c,fb5d3ad8e889a326] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:39.753 INFO 21062 --- [ocp-schedules-21,4c9519f190caf096,e1f61f2811baea52] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:40:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:40:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:40:41.353 INFO 21062 --- [ocp-schedules-23,b7091706ddeb4a77,022c3f3fa4c27df6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:40:41.353 INFO 21062 --- [ocp-schedules-23,9aad26ecb6e8e070,2daf0d8a9f2526f9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:40:41.357 INFO 21062 --- [ocp-schedules-23,9aad26ecb6e8e070,2daf0d8a9f2526f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:41.387 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:40:41.387 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:40:41.656 WARN 21062 --- [ocp-async-4,53813c5a530d87e5,0236d3b8e14982b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:41.656 INFO 21062 --- [ocp-async-5,9f08c3c045c716d5,4fe4bb4dc8eef558] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:40:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:40:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:40:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:40:41.853 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:40:41.853 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:40:41.935 INFO 21062 --- [ocp-schedules-4,57a1d0e1ec132e28,ed0201147ecf0803] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:40:41.983 INFO 21062 --- [prometheus-minute-schedule-1,761a2541fa24c6b4,43c9c9783ed1b4a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:42.414 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:42.416 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:43.656 WARN 21062 --- [ocp-async-3,b1b10c4c10fef7a6,56faf0645403438d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:43.656 INFO 21062 --- [ocp-async-6,a714849e92e8fecc,95c741a5242374a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:44.983 INFO 21062 --- [prometheus-minute-schedule-1,5cfd851b88c0d56b,7451da0d9d9bf982] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:45.416 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:45.417 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:45.656 WARN 21062 --- [ocp-async-8,7302cbd230dbd63d,e3bd982df93d81ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:45.656 INFO 21062 --- [ocp-async-1,bdb6ef428d7b4bf0,b6a1b16e6e72b33e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:45.903 INFO 21062 --- [ocp-schedules-20,cd8d0f399c427100,4f42ab3d0d65f319] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:40:45.907 INFO 21062 --- [ocp-schedules-20,cd8d0f399c427100,4f42ab3d0d65f319] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:45.996 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:40:45.999 ERROR 21062 --- [ocp-schedules-20,cd8d0f399c427100,4f42ab3d0d65f319] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:40:45.999 INFO 21062 --- [ocp-schedules-20,cd8d0f399c427100,4f42ab3d0d65f319] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:40:47.656 WARN 21062 --- [ocp-async-4,4d8bfd8533487230,09ad9bfdb5f5f37d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:47.656 INFO 21062 --- [ocp-async-10,8ea7d4844de2ea2e,f0afe4c8d234eedb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:47.983 INFO 21062 --- [prometheus-minute-schedule-1,e2118d1725910624,51624379bed1d32f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:48.417 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:48.419 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:40:48.591397+08:00, lastProcessTime=2024-11-04T15:40:38.589711+08:00 2024-11-04 15:40:48.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:40:48.591397+08:00 2024-11-04 15:40:48.814 INFO 21062 --- [ocp-async-3,dfa14a57da999f2d,c5666a28cc71e905] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:40:49.656 WARN 21062 --- [ocp-async-6,6af709dcf8f52e70,1263e7513711dbbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:49.656 INFO 21062 --- [ocp-async-9,8373b86bf0a7a2ed,111d91134e9f0994] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:49.779 INFO 21062 --- [ocp-schedules-27,1cb35e592033c7d9,c8eaa314d890cf15] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:40:50.983 INFO 21062 --- [prometheus-minute-schedule-1,81f9b594a1e46885,d7e4ae80535ea6cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:51.419 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:51.421 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:51.657 WARN 21062 --- [ocp-async-1,52f34ddbf31ca673,bf02601a61fcec38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:51.657 INFO 21062 --- [ocp-async-2,8eb615f77b465929,30b2796146450e52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:51.939 INFO 21062 --- [ocp-schedules-14,47fec2ac698fef27,202c3c3b84ec973f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:40:53.404 INFO 21062 --- [ocp-schedules-3,75bfc9ad074156bc,ac3d08c4e3153e96] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:40:53.410 INFO 21062 --- [ocp-schedules-3,75bfc9ad074156bc,ac3d08c4e3153e96] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:40:53.410 INFO 21062 --- [ocp-schedules-3,75bfc9ad074156bc,ac3d08c4e3153e96] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:40:53.657 INFO 21062 --- [ocp-async-7,749e425b1322ec8b,c999623f1e28ff95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:53.657 WARN 21062 --- [ocp-async-10,a791d9c52686a791,07480ee357f68b4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:53.983 INFO 21062 --- [prometheus-minute-schedule-1,a847b91629754f21,adb88cb82fbbe648] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:54.421 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:54.422 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:55.657 INFO 21062 --- [ocp-async-6,bb625cf02d1722f3,ac7d76d7865c1834] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:55.657 WARN 21062 --- [ocp-async-3,e7eefbea600ac448,b6f4ae8c25c484be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c080bcf245e7f322,88b34030b8b47766] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:40:57.422 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:40:57.424 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:40:57.657 INFO 21062 --- [ocp-async-1,3b794cfea52f5563,70d3054a6bfe46d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:57.657 WARN 21062 --- [ocp-async-8,01397667bba9b4c5,90b017a60aaf4b83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:58.500 INFO 21062 --- [ocp-schedules-17,2e9b090c7803f44b,9a472f7d9485f798] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:40:58.500 INFO 21062 --- [ocp-schedules-5,ce58d01e673d5db2,cf347673c0799cc4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:40:58.502 INFO 21062 --- [ocp-schedules-1,204c3b51bb91b021,df067c81c3490ec8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:40:58.503 INFO 21062 --- [ocp-schedules-17,2e9b090c7803f44b,9a472f7d9485f798] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:40:58.504 INFO 21062 --- [ocp-schedules-5,ce58d01e673d5db2,cf347673c0799cc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:40:58.505 INFO 21062 --- [ocp-schedules-19,386bc2e021204c50,7396f00f7cdec15c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:40:58.510 INFO 21062 --- [ocp-schedules-22,fde5c05445662acf,06a9684de8de3cff] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:40:58.510 INFO 21062 --- [ocp-schedules-22,fde5c05445662acf,06a9684de8de3cff] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:40:58.510 INFO 21062 --- [ocp-schedules-24,50b55df6da61066a,e49f3c99e21f4f23] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:40:58.510 INFO 21062 --- [ocp-schedules-24,50b55df6da61066a,e49f3c99e21f4f23] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:40:58.511 INFO 21062 --- [ocp-schedules-16,d8e5fae5bcfc232a,b20292e9047c59e8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:40:58.511 INFO 21062 --- [ocp-schedules-16,d8e5fae5bcfc232a,b20292e9047c59e8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:40:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:40:58.588462+08:00, lastProcessTime=2024-11-04T15:40:48.591397+08:00 2024-11-04 15:40:58.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:40:58.588462+08:00 2024-11-04 15:40:58.814 INFO 21062 --- [ocp-async-10,7efdee0dc7a050f5,3fddf3ce1dcf3dc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:40:59.606 INFO 21062 --- [ocp-schedules-9,ae6d886b8c9e038e,df08a2de0bf24965] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:40:59.657 INFO 21062 --- [ocp-async-4,aa3eb224acdfc839,fbcddbce0c39f13a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:40:59.657 WARN 21062 --- [ocp-async-7,1bd94aef48755dac,108263a15b72390e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:40:59.815 INFO 21062 --- [ocp-schedules-18,9a4f28ad001efdeb,8d174fa6f867e17c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:40:59.983 INFO 21062 --- [prometheus-minute-schedule-1,b5d984b774c583ad,dcc4a86bf77bcae3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706060, interval=60 2024-11-04 15:41:00.424 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:00.425 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:01.658 INFO 21062 --- [ocp-async-3,fef2d89e3b29bc8c,ed9eb7833fe8551c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:01.658 WARN 21062 --- [ocp-async-5,323c1313e2ee66c1,6631f23fabb96204] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:01.941 INFO 21062 --- [ocp-schedules-30,0cc2c8fed0c87596,ce19690c726e05d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:41:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f05e3f3f7a578f70,514f2918e314195c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:03.425 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:03.427 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:03.658 WARN 21062 --- [ocp-async-9,1443fb6dfe3a979b,44a0ff87188548b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:03.658 INFO 21062 --- [ocp-async-8,97ae2ee40db0be35,e2d46b4ce4fd1b5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:05.658 INFO 21062 --- [ocp-async-2,1af4ba8a93877869,799339517107dbe6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:05.658 WARN 21062 --- [ocp-async-7,20e56a9783a69db4,0f0dfd6a5b44bcbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:05.983 INFO 21062 --- [prometheus-minute-schedule-1,3eea082fe4944ca8,de45c09be1e8ca7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:06.427 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:06.429 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:07.658 WARN 21062 --- [ocp-async-6,7e23e76bede6dc17,cbec562153ef9cea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:07.658 INFO 21062 --- [ocp-async-5,ce705393a6cd2db5,3678171d4c1edc07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:08.504 INFO 21062 --- [ocp-schedules-17,2e9b090c7803f44b,9a472f7d9485f798] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:41:08.505 INFO 21062 --- [ocp-schedules-5,ce58d01e673d5db2,cf347673c0799cc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:41:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:41:08.590570+08:00, lastProcessTime=2024-11-04T15:40:58.588462+08:00 2024-11-04 15:41:08.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:41:08.590570+08:00 2024-11-04 15:41:08.815 INFO 21062 --- [ocp-async-3,b0231f2bf02c9d59,54e06ead70bb3212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:41:08.983 INFO 21062 --- [prometheus-minute-schedule-1,b6887c8c78ec15d6,354d8947d663cfe7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:09.429 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:09.430 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:09.658 INFO 21062 --- [ocp-async-3,f7ba012b40988d42,fce54aae22a4972f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:09.658 WARN 21062 --- [ocp-async-1,455efe210c956707,ab55574f5ebe21d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:09.839 INFO 21062 --- [ocp-schedules-30,7056def840a2a71d,e33aff95fed3cfc3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:41:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:41:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:41:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:41:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:41:10.305 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:41:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:41:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:41:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:41:11.387 INFO 21062 --- [ocp-schedules-23,9aad26ecb6e8e070,2daf0d8a9f2526f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:41:11.387 INFO 21062 --- [ocp-schedules-23,45ffffb4bc1a33e8,82c49510eb702f26] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:41:11.391 INFO 21062 --- [ocp-schedules-23,45ffffb4bc1a33e8,82c49510eb702f26] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:11.421 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:41:11.421 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:41:11.659 INFO 21062 --- [ocp-async-8,abc481fff18703a7,45b4f254eec12f62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:11.659 WARN 21062 --- [ocp-async-10,e1a85472ab1ad069,4e84cef1f4f650ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706000, nodeCount=53, affectRows=53 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706000, nodeCount=78, affectRows=78 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706000, nodeCount=53, validCount=53 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706000, nodeCount=66, affectRows=66 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706000, nodeCount=46, affectRows=46 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706000, nodeCount=78, validCount=78 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706000, nodeCount=46, validCount=46 2024-11-04 15:41:11.894 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706000, nodeCount=66, validCount=66 2024-11-04 15:41:11.895 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706000, nodeCount=99, affectRows=99 2024-11-04 15:41:11.895 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706000, nodeCount=80, affectRows=80 2024-11-04 15:41:11.895 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706000, nodeCount=80, validCount=80 2024-11-04 15:41:11.895 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706000, nodeCount=96, validCount=96 2024-11-04 15:41:11.945 INFO 21062 --- [ocp-schedules-15,28805290d463efe6,68e0910ea20743a7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:41:11.983 INFO 21062 --- [prometheus-minute-schedule-1,fcd6996edc75ea8e,996301504f759804] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:12.431 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:12.432 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:13.412 INFO 21062 --- [ocp-schedules-13,9dbf07c025c8de1b,f197e43daf9814d8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:41:13.417 INFO 21062 --- [ocp-schedules-13,9dbf07c025c8de1b,f197e43daf9814d8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:41:13.417 INFO 21062 --- [ocp-schedules-13,9dbf07c025c8de1b,f197e43daf9814d8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:41:13.658 INFO 21062 --- [ocp-async-2,e5ae4e722228a438,406eae8f2ab4297f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:13.659 WARN 21062 --- [ocp-async-5,9223cfdd27c56909,442757c30bd6906c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:13.984 INFO 21062 --- [prometheus-cache-monitor1,75dc4d7dbdb290d4,1810fd073b3b26ba] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:41:13.984 INFO 21062 --- [prometheus-cache-monitor1,75dc4d7dbdb290d4,1810fd073b3b26ba] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:41:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d3a06cebcd7e5e71,1fb4756ba3497025] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:15.432 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:15.434 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:15.659 INFO 21062 --- [ocp-async-1,4a50782a8c63f7a1,0d7475bf33584f5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:15.659 WARN 21062 --- [ocp-async-9,6ea4fd70d199e998,3e736f2513dbedd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:15.762 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:37:15.743, end=2024-11-04T15:39:15.76} 2024-11-04 15:41:15.772 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:41:15.772 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:41:17.659 INFO 21062 --- [ocp-async-10,fc7f10384cbf3a9f,97445f2ac73d2568] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:17.659 WARN 21062 --- [ocp-async-7,4bb8c06fe54b9661,0c007044623176b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:17.983 INFO 21062 --- [prometheus-minute-schedule-1,661c2af0714ea9c8,a78a0eff9b97fedc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:18.434 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:18.435 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:41:18.589511+08:00, lastProcessTime=2024-11-04T15:41:08.590570+08:00 2024-11-04 15:41:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:41:18.589511+08:00 2024-11-04 15:41:18.815 INFO 21062 --- [ocp-async-6,870ce97928e933dc,cca4c3a20857f2fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:41:19.385 INFO 21062 --- [ocp-schedules-22,6a34bed97de662ff,7ecbe9d11178814d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:41:19.389 INFO 21062 --- [ocp-schedules-22,6a34bed97de662ff,7ecbe9d11178814d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:19.390 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:41:19.659 INFO 21062 --- [ocp-async-6,3685fe20d79b9e6a,a3f174642e94b235] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:19.659 WARN 21062 --- [ocp-async-2,0cf37b915fee05de,241dea3f1d42fb7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:19.880 INFO 21062 --- [ocp-schedules-21,9ae8c0f55ea4089b,d3419c8a251078f3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:41:20.983 INFO 21062 --- [prometheus-minute-schedule-1,1d4d9635fcf75304,e68dd2b8df8f7771] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:21.435 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:21.436 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:21.659 INFO 21062 --- [ocp-async-4,ec85c44d686873fa,40c0d42488d80ae7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:21.660 WARN 21062 --- [ocp-async-1,8ee203e5389ed5cc,3c13f706962daf05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:21.948 INFO 21062 --- [ocp-schedules-11,26f80359fc5f608b,f485065ee7a8e9c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:41:23.659 INFO 21062 --- [ocp-async-3,daa7c8e62315179f,22bc6434ed06e500] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:23.660 WARN 21062 --- [ocp-async-10,80ece154c971f977,45cc976b9fecb90b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:23.983 INFO 21062 --- [prometheus-minute-schedule-1,86e510bad8f13075,f00cda05cbe3bcb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:24.436 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:24.438 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:25.660 INFO 21062 --- [ocp-async-2,c4a3d6b3263b051d,981dedf93b7558d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:25.660 WARN 21062 --- [ocp-async-5,0f5685605a17ba15,5cadb701bf5fcbb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:26.983 INFO 21062 --- [prometheus-minute-schedule-1,7d9352f919cf6e75,3ef041a560b2a384] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:26.984 INFO 21062 --- [prometheus-hour-schedule-1,1c48f9e4f94be468,f8f4c03f793376f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:41:27.438 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:27.439 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:27.660 INFO 21062 --- [ocp-async-1,b8731f2d84e4a00b,cae27d691e2e663c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:27.660 WARN 21062 --- [ocp-async-9,1e3de473113fdea3,07d69ebd3d1685b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:28.498 INFO 21062 --- [ocp-schedules-25,d3b5cf078cd91888,03bdc5cc5c2e89e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:41:28.500 INFO 21062 --- [ocp-schedules-8,488fbb3798ea21d9,841febd42b92a175] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:41:28.500 INFO 21062 --- [ocp-schedules-28,3f67a9182ef94063,dfa57201131beb98] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:41:28.500 INFO 21062 --- [ocp-schedules-9,ec2a3afa905b0448,5845995951c2fa31] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:41:28.502 INFO 21062 --- [ocp-schedules-2,45f52c2603491805,d3080d4f8e3a5f32] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:41:28.503 INFO 21062 --- [ocp-schedules-25,d3b5cf078cd91888,03bdc5cc5c2e89e2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:41:28.504 INFO 21062 --- [ocp-schedules-8,488fbb3798ea21d9,841febd42b92a175] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:28.504 INFO 21062 --- [ocp-schedules-9,ec2a3afa905b0448,5845995951c2fa31] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:41:28.504 INFO 21062 --- [ocp-schedules-29,05275e6c4e1be3ea,1b1436ffb96ed726] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:41:28.506 INFO 21062 --- [ocp-schedules-28,3f67a9182ef94063,dfa57201131beb98] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:41:28.509 INFO 21062 --- [ocp-schedules-30,3992c4c681af9a73,024781c2908d8dd4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:41:28.509 INFO 21062 --- [ocp-schedules-30,3992c4c681af9a73,024781c2908d8dd4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:41:28.509 INFO 21062 --- [ocp-schedules-6,7a22111e101e6de6,7d0743bf66d8845b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:41:28.509 INFO 21062 --- [ocp-schedules-6,7a22111e101e6de6,7d0743bf66d8845b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:41:28.509 INFO 21062 --- [ocp-schedules-5,a53d3c804813ae10,a14e6b665acae53a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:41:28.509 INFO 21062 --- [ocp-schedules-5,a53d3c804813ae10,a14e6b665acae53a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:41:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:41:28.589078+08:00, lastProcessTime=2024-11-04T15:41:18.589511+08:00 2024-11-04 15:41:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:41:28.589078+08:00 2024-11-04 15:41:28.815 INFO 21062 --- [ocp-async-5,55361ba8c9b8aa74,0018b72414b5f1f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:41:29.031 INFO 21062 --- [ocp-schedules-2,50922bfa2a3caf56,99cdb3ea49ee2cba] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:41:29.390 INFO 21062 --- [ocp-schedules-22,6a34bed97de662ff,7ecbe9d11178814d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:41:29.461 INFO 21062 --- [ocp-schedules-6,8ad84db8ec119983,5536f9eb2ef0dc43] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:41:29.461 INFO 21062 --- [ocp-schedules-6,8ad84db8ec119983,5536f9eb2ef0dc43] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:41:29.608 INFO 21062 --- [ocp-schedules-14,12a3d70dcbec4190,5210ffda3845cd12] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:41:29.624 INFO 21062 --- [ocp-schedules-16,9acfd12c9d3c7183,0b66d690f73cb5a8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:41:29.630 INFO 21062 --- [ocp-schedules-16,9acfd12c9d3c7183,0b66d690f73cb5a8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:41:29.630 INFO 21062 --- [ocp-schedules-16,9acfd12c9d3c7183,0b66d690f73cb5a8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:41:29.660 INFO 21062 --- [ocp-async-5,e4820a34a2eeeb0b,3dc0ea0782bed4dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:29.660 WARN 21062 --- [ocp-async-8,d448098980c884ca,74956db74289f33a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:29.904 INFO 21062 --- [ocp-schedules-3,2fd1f42ab8f20ee0,db9dcd4103b418e4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:41:29.983 INFO 21062 --- [prometheus-minute-schedule-1,8f734119bdd09e52,a58fb8b5e5ab63d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:30.135 INFO 21062 --- [ocp-schedules-26,d3dce66caf22673f,df23425ff451ef32] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:41:30.135 INFO 21062 --- [ocp-schedules-26,d3dce66caf22673f,df23425ff451ef32] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:41:30.143 INFO 21062 --- [ocp-schedules-26,d3dce66caf22673f,df23425ff451ef32] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:41:30.143 INFO 21062 --- [ocp-schedules-26,d3dce66caf22673f,df23425ff451ef32] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:41:30.148 INFO 21062 --- [ocp-schedules-26,d3dce66caf22673f,df23425ff451ef32] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:41:30.148 INFO 21062 --- [ocp-schedules-26,d3dce66caf22673f,df23425ff451ef32] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:41:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:41:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:41:30.439 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:30.441 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:31.660 INFO 21062 --- [ocp-async-9,e2e6c1cd3770f775,34022e29c4a3b89e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:31.660 WARN 21062 --- [ocp-async-1,efb75188ad2747d3,5ce303c707e15c15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:31.951 INFO 21062 --- [ocp-schedules-11,5d2927a92aae9338,edde354f3d3f591f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:41:32.983 INFO 21062 --- [prometheus-minute-schedule-1,bd08fab2a260e896,1eaef0222dfc69dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:33.420 INFO 21062 --- [ocp-schedules-20,791af26ef50b27f6,d996e2a35a4f1d0f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:41:33.425 INFO 21062 --- [ocp-schedules-20,791af26ef50b27f6,d996e2a35a4f1d0f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:41:33.425 INFO 21062 --- [ocp-schedules-20,791af26ef50b27f6,d996e2a35a4f1d0f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:41:33.441 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:33.442 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:33.660 INFO 21062 --- [ocp-async-4,fe7c93a4a2bbb684,605a697e2b592bb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:33.661 WARN 21062 --- [ocp-async-10,5b5e94990c18185f,a857d70c2563b946] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:33.700 INFO 21062 --- [ocp-schedules-29,f0d33d8781354b19,9899ef4075fcaaae] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:41:33.700 INFO 21062 --- [ocp-schedules-29,f0d33d8781354b19,9899ef4075fcaaae] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:41:33.765 INFO 21062 --- [ocp-schedules-29,f0d33d8781354b19,9899ef4075fcaaae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:41:33.765 INFO 21062 --- [ocp-schedules-29,f0d33d8781354b19,9899ef4075fcaaae] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:41:33.768 INFO 21062 --- [ocp-schedules-7,61560f8c8fcba567,bd177a9eed902067] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:41:33.938 INFO 21062 --- [ocp-schedules-30,79bd0cf1a2c8a0a6,3678b7befdcaf7aa] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:41:33.938+08:00, clusters=[1] 2024-11-04 15:41:33.940 INFO 21062 --- [ocp-schedules-30,79bd0cf1a2c8a0a6,3678b7befdcaf7aa] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:41:34.289 INFO 21062 --- [ocp-schedules-25,78bb7374e3e662aa,c7e6274ae1d29963] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:41:34.289+08:00, clusters=[1] 2024-11-04 15:41:34.293 INFO 21062 --- [ocp-schedules-25,78bb7374e3e662aa,c7e6274ae1d29963] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:41:35.660 INFO 21062 --- [ocp-async-8,dd61c6b45f80ba26,9b121f5f67e66262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:35.661 WARN 21062 --- [ocp-async-3,6a63827e74563a8c,a2fcfe630d121a31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:35.983 INFO 21062 --- [prometheus-minute-schedule-1,7feae8d9bdafa1ee,00dd68f48086b170] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:36.349 INFO 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:41:36.350 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:41:36.350 INFO 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:41:36.350 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:41:36.359 INFO 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:41:36.359 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:41:36.360 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:41:36.361 WARN 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:41:36.361 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:41:36.361 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:41:36.362 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:41:36.362 INFO 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:41:36.362 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:41:36.363 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:41:36.363 WARN 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:41:36.363 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:41:36.370 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:41:36.370 INFO 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:41:36.370 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:41:36.379 ERROR 21062 --- [ocp-schedules-29,309f6856006b73cd,137c362559f06a6e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:41:36.442 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:36.443 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:37.661 INFO 21062 --- [ocp-async-1,af90cdcb80470f4e,22ffddc8df6f7217] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:37.661 WARN 21062 --- [ocp-async-2,1a6c12ff670807fb,eb2e3fe48a36f4d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:38.505 INFO 21062 --- [ocp-schedules-9,ec2a3afa905b0448,5845995951c2fa31] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:41:38.505 INFO 21062 --- [ocp-schedules-8,488fbb3798ea21d9,841febd42b92a175] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:41:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:41:38.588121+08:00, lastProcessTime=2024-11-04T15:41:28.589078+08:00 2024-11-04 15:41:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:41:38.588121+08:00 2024-11-04 15:41:38.816 INFO 21062 --- [ocp-async-6,fa8b24ab57e0b187,9459ac892fa859e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:41:38.983 INFO 21062 --- [prometheus-minute-schedule-1,7723dc47fec6198f,7001e035fe8f6b97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:39.443 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:39.444 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:39.661 INFO 21062 --- [ocp-async-4,a41bdd51f477a8b9,6219b9b53a026ba5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:39.661 WARN 21062 --- [ocp-async-3,7f028d0654d9bc16,598129dc9593e2b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:39.946 INFO 21062 --- [ocp-schedules-16,e0a0b77ae8164f93,1c3a03b86eecd390] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:41:40.002 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:41:40.003 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:41:41.421 INFO 21062 --- [ocp-schedules-23,45ffffb4bc1a33e8,82c49510eb702f26] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:41:41.421 INFO 21062 --- [ocp-schedules-23,dce26261dd2e483f,20a127e7a0311e33] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:41:41.425 INFO 21062 --- [ocp-schedules-23,dce26261dd2e483f,20a127e7a0311e33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:41.455 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:41:41.455 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:41:41.661 INFO 21062 --- [ocp-async-8,24d9eeecdc869887,c1b94c42a190f9cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:41.661 WARN 21062 --- [ocp-async-2,bb1e41bca5a21950,8ec7b96956f628ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:41.953 INFO 21062 --- [ocp-schedules-5,b5051d2a16669107,675d5a6c88e0a745] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:41:41.983 INFO 21062 --- [prometheus-minute-schedule-1,efe1191b79696f5c,dd83bc331542d2ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:42.444 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:42.445 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:43.661 INFO 21062 --- [ocp-async-1,7c0d1a2126bf0170,f1011de17ca9bbef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:43.661 WARN 21062 --- [ocp-async-5,54c5a5e403e995eb,ac4a0b1b1a76cc35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:44.983 INFO 21062 --- [prometheus-minute-schedule-1,4f0c71a3eb3f1715,c11cc103218ab05f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:45.445 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:45.447 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:45.661 INFO 21062 --- [ocp-async-10,3a05a2705dcd4d0f,18d8a9580b4e005a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:45.662 WARN 21062 --- [ocp-async-3,250149afe0d2463d,abc97fa9bc951a3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:45.999 INFO 21062 --- [ocp-schedules-30,4b169e307e905235,d77aec97408ec33d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:41:46.003 INFO 21062 --- [ocp-schedules-30,4b169e307e905235,d77aec97408ec33d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:46.089 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:41:46.093 ERROR 21062 --- [ocp-schedules-30,4b169e307e905235,d77aec97408ec33d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:41:46.093 INFO 21062 --- [ocp-schedules-30,4b169e307e905235,d77aec97408ec33d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:41:47.662 INFO 21062 --- [ocp-async-7,7e713e7a134120db,f1babf844cadb708] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:47.662 WARN 21062 --- [ocp-async-2,8765a353f4940a18,38266f313a31db6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:47.983 INFO 21062 --- [prometheus-minute-schedule-1,fdd82267c0898b6a,262683cc49d613fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:48.447 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:48.448 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:41:48.589864+08:00, lastProcessTime=2024-11-04T15:41:38.588121+08:00 2024-11-04 15:41:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:41:48.589864+08:00 2024-11-04 15:41:48.816 INFO 21062 --- [ocp-async-9,54e7e89ebc63a4e5,f2b33f65eed4133b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:41:49.662 INFO 21062 --- [ocp-async-9,bb4c9291080d5491,53804b9087e5f6f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:49.662 WARN 21062 --- [ocp-async-8,3b5a37e3e65365d4,6c5f33da0045ddec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:49.974 INFO 21062 --- [ocp-schedules-18,fa174e4e0b405111,c6501d298445bc8a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:41:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b0039c78e6031aca,f1ea0645f6ee7ba3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:51.448 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:51.450 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:51.662 INFO 21062 --- [ocp-async-1,a76f62a30c56d328,940cd82ad7f45ba3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:51.662 WARN 21062 --- [ocp-async-6,d7ddddb3b7bfe5b9,fcaadd5d6492df49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:51.956 INFO 21062 --- [ocp-schedules-3,53a6999fe43a3283,b58188e9c36182e8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:41:53.427 INFO 21062 --- [ocp-schedules-5,bd193846778a175d,2f2a4f48a7efd763] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:41:53.433 INFO 21062 --- [ocp-schedules-5,bd193846778a175d,2f2a4f48a7efd763] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:41:53.433 INFO 21062 --- [ocp-schedules-5,bd193846778a175d,2f2a4f48a7efd763] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:41:53.662 INFO 21062 --- [ocp-async-10,16bf7e484f2d3926,13eeb681f2c0ede4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:53.662 WARN 21062 --- [ocp-async-4,2d05106e77486e36,8d59296205e07e1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:53.983 INFO 21062 --- [prometheus-minute-schedule-1,ea08a1fb48870085,5507120f02706c8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:54.450 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:54.451 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:55.662 INFO 21062 --- [ocp-async-8,eabc1e14b81be5c3,aac1f3e5fd4bd211] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:55.662 WARN 21062 --- [ocp-async-5,efd18f3409b2fca9,f51958941437dd38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:56.983 INFO 21062 --- [prometheus-minute-schedule-1,03fdc9be1d79caa6,9cfd03fe597cdaee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:41:57.451 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:41:57.452 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:41:57.662 INFO 21062 --- [ocp-async-6,8faa60f9fbafd8a5,60c690d6ae160dca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:57.663 WARN 21062 --- [ocp-async-9,f54d43d309e5dc68,31b14a27458cf57b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:58.500 INFO 21062 --- [ocp-schedules-11,fb1f555e3d5786a8,32ec88a171a4d64e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:41:58.500 INFO 21062 --- [ocp-schedules-5,c1af8a10f58a7a34,8251b806946cd47e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:41:58.502 INFO 21062 --- [ocp-schedules-6,399f25f89f699a72,42c0124902b88ad8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:41:58.503 INFO 21062 --- [ocp-schedules-11,fb1f555e3d5786a8,32ec88a171a4d64e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:58.504 INFO 21062 --- [ocp-schedules-5,c1af8a10f58a7a34,8251b806946cd47e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:41:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:41:58.505 INFO 21062 --- [ocp-schedules-22,d25f89412622fe57,189cdec3afd191c6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:41:58.510 INFO 21062 --- [ocp-schedules-30,4cd79b722d2a9dc8,0a25a7bf67f1b19f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:41:58.510 INFO 21062 --- [ocp-schedules-20,e3d4931289ccf91b,8b8b2aaa0e345d9b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:41:58.510 INFO 21062 --- [ocp-schedules-21,ec1a0466d4df0b51,d1317b2f6f3f6dd4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:41:58.510 INFO 21062 --- [ocp-schedules-20,e3d4931289ccf91b,8b8b2aaa0e345d9b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:41:58.510 INFO 21062 --- [ocp-schedules-30,4cd79b722d2a9dc8,0a25a7bf67f1b19f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:41:58.510 INFO 21062 --- [ocp-schedules-21,ec1a0466d4df0b51,d1317b2f6f3f6dd4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:41:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:41:58.588197+08:00, lastProcessTime=2024-11-04T15:41:48.589864+08:00 2024-11-04 15:41:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:41:58.588197+08:00 2024-11-04 15:41:58.816 INFO 21062 --- [ocp-async-2,b799e7af96699b34,d49151b1e4505ed9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:41:59.611 INFO 21062 --- [ocp-schedules-6,4bbfe280efa759df,034c74644ab425a1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:41:59.663 INFO 21062 --- [ocp-async-2,180a4d2c2914507e,01ba7ecd4f685214] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:41:59.663 WARN 21062 --- [ocp-async-10,273806a0fadc7aab,d2318cf10e7d385f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:41:59.983 INFO 21062 --- [prometheus-minute-schedule-1,c330d3e288c00e36,5399634e47bfe754] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706120, interval=60 2024-11-04 15:42:00.015 INFO 21062 --- [ocp-schedules-24,9af6a7ffaa770f91,a1c07505efe545d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:42:00.452 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:00.453 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:01.663 INFO 21062 --- [ocp-async-7,46676e000fbaf5af,03fad25bbe171212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:01.663 WARN 21062 --- [ocp-async-8,57e4cc489920302e,f6088ee5508b629a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:01.959 INFO 21062 --- [ocp-schedules-18,0598c384de150dfd,52c24f958c9d5cb5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:42:02.983 INFO 21062 --- [prometheus-minute-schedule-1,4c3f08987ee91d64,2d7a0b2ae9c733c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:03.453 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:03.454 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:03.663 WARN 21062 --- [ocp-async-6,43c2d992b35e2707,b0853e61b2be0691] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:03.663 INFO 21062 --- [ocp-async-3,010031125b815e0a,75422d225d4cc576] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:05.663 INFO 21062 --- [ocp-async-10,6847cc483c437360,d384090ee92955a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:05.663 WARN 21062 --- [ocp-async-4,8b922dc959522d31,29d333feac3df42f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:05.983 INFO 21062 --- [prometheus-minute-schedule-1,74cb59476f2dfd8c,c2dfd750644b7647] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:06.455 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:06.456 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:07.663 INFO 21062 --- [ocp-async-8,1dbe5ab07a8b71a5,2504c2e0446b58a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:07.663 WARN 21062 --- [ocp-async-5,50612f9d37dec35f,f49bfad2726da479] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:08.505 INFO 21062 --- [ocp-schedules-11,fb1f555e3d5786a8,32ec88a171a4d64e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:42:08.505 INFO 21062 --- [ocp-schedules-5,c1af8a10f58a7a34,8251b806946cd47e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:42:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:42:08.588893+08:00, lastProcessTime=2024-11-04T15:41:58.588197+08:00 2024-11-04 15:42:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:42:08.588893+08:00 2024-11-04 15:42:08.816 INFO 21062 --- [ocp-async-9,dcceb975f36a47c5,02ad2e500cf9ac15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:42:08.983 INFO 21062 --- [prometheus-minute-schedule-1,34edce8e2ae26d4e,7ad4b6e76c615dc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:09.456 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:09.457 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:09.664 INFO 21062 --- [ocp-async-9,a6a8c031620eba1f,85241d96068ffc28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:09.664 WARN 21062 --- [ocp-async-3,79dd67b4c0d5234b,b00252b228bd98a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:10.003 INFO 21062 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:42:10.003 INFO 21062 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:42:10.041 INFO 21062 --- [ocp-schedules-14,8e24e6857966b77b,2758ca2324d941d0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:42:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:42:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:42:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:42:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:42:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:42:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:42:11.456 INFO 21062 --- [ocp-schedules-23,dce26261dd2e483f,20a127e7a0311e33] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:42:11.456 INFO 21062 --- [ocp-schedules-23,e9921d7ef952318d,8d5fcc5893aa1cf2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:42:11.460 INFO 21062 --- [ocp-schedules-23,e9921d7ef952318d,8d5fcc5893aa1cf2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:11.489 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:42:11.490 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:42:11.664 WARN 21062 --- [ocp-async-10,a9129a73ca43f673,34c116b4ef8d0985] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:11.664 INFO 21062 --- [ocp-async-1,c82d87e7214ea7db,bfa99a938c0b46e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=14, affectRows=14 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=14, validCount=14 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=23, affectRows=23 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=43, affectRows=43 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=23, validCount=23 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=43, validCount=43 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=33, affectRows=33 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=45, affectRows=45 2024-11-04 15:42:11.901 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=41, affectRows=41 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=33, validCount=33 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=45, validCount=45 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=58, affectRows=58 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=41, validCount=41 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=58, validCount=58 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=82, affectRows=82 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706060, nodeCount=83, affectRows=83 2024-11-04 15:42:11.902 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=81, validCount=81 2024-11-04 15:42:11.903 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706060, nodeCount=81, validCount=81 2024-11-04 15:42:11.962 INFO 21062 --- [ocp-schedules-3,f2eee3a6997e50a6,bfd4826b6c9deac9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:42:11.983 INFO 21062 --- [prometheus-minute-schedule-1,72d6a5357df6f90d,20ea685b41a508ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:12.458 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:12.459 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:13.435 INFO 21062 --- [ocp-schedules-8,de2b6108d331a00c,f9741e2d08318e58] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:42:13.441 INFO 21062 --- [ocp-schedules-8,de2b6108d331a00c,f9741e2d08318e58] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:42:13.441 INFO 21062 --- [ocp-schedules-8,de2b6108d331a00c,f9741e2d08318e58] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:42:13.664 INFO 21062 --- [ocp-async-2,d9376a98777cdc79,f709869150bf7e6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:13.664 WARN 21062 --- [ocp-async-8,8c6e54f5782fbf75,48847d5921f88d72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:13.984 INFO 21062 --- [prometheus-cache-monitor1,dc74f2b74e358b5b,d1ac1c1e1adb08a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:42:13.984 INFO 21062 --- [prometheus-cache-monitor1,dc74f2b74e358b5b,d1ac1c1e1adb08a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:42:14.983 INFO 21062 --- [prometheus-minute-schedule-1,efdcff217e0ebf2e,6a5e06b6d0b08a2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:15.459 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:15.460 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:15.664 INFO 21062 --- [ocp-async-3,ea0bd984df38372b,f6f33ee6397f9f8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:15.664 WARN 21062 --- [ocp-async-6,93685d075b6fb1af,34918ccceaa209ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:17.664 INFO 21062 --- [ocp-async-10,b435851ef14d1aab,062c7362d1c26d22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:17.664 WARN 21062 --- [ocp-async-4,8fbfddf820ca24b6,8b4520525531f674] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:17.983 INFO 21062 --- [prometheus-minute-schedule-1,36ec3557edb548d1,4d60a87a0af727a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:18.460 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:18.461 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:42:18.589519+08:00, lastProcessTime=2024-11-04T15:42:08.588893+08:00 2024-11-04 15:42:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:42:18.589519+08:00 2024-11-04 15:42:18.816 INFO 21062 --- [ocp-async-8,ab6c45c5992744c7,f38b506898247a5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:42:19.664 INFO 21062 --- [ocp-async-8,ba39918f2cb321e1,dd504d5fcfacc4bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:19.664 WARN 21062 --- [ocp-async-2,4869eca8f9b83067,f44ae35032b962f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:20.081 INFO 21062 --- [ocp-schedules-11,086ce26f008c9c40,45e37fb6a58a512c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:42:20.983 INFO 21062 --- [prometheus-minute-schedule-1,c783108eaf28e1b0,6f406d3ed81fe07b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:21.462 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:21.462 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:21.665 WARN 21062 --- [ocp-async-7,98440fd92272432f,506c4e5e72d4a37c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:21.665 INFO 21062 --- [ocp-async-3,43c32c3d6415e9a1,0e78e4f93f5dd98d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:21.965 INFO 21062 --- [ocp-schedules-4,0eab4f6615d7098a,6d3e2de6de4642dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:42:23.665 WARN 21062 --- [ocp-async-9,81cdc67ad8b6ecf3,9f59d9a0cd2ede08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:23.665 INFO 21062 --- [ocp-async-10,cf8b4c31d5f47404,71b917d436b712d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:23.983 INFO 21062 --- [prometheus-minute-schedule-1,e1355a2fccd4b764,71bfb5bfe4921d2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:24.463 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:24.464 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:25.665 WARN 21062 --- [ocp-async-2,6fb160b9e3ea0e3b,a47bc992255bdc76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:25.665 INFO 21062 --- [ocp-async-5,947a909652bf0cf4,03bbb6d1b76c53f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:26.983 INFO 21062 --- [prometheus-minute-schedule-1,07cdbcd7db7df210,f424ec746c493866] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:26.984 INFO 21062 --- [prometheus-hour-schedule-1,8109ad04a5b8e093,854352df28e329aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:42:27.464 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:27.464 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:27.665 WARN 21062 --- [ocp-async-7,0efab4e98f7df1cf,468a5653cb77705a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:27.665 INFO 21062 --- [ocp-async-6,5503815968772f79,6b5e7d7605a90621] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:28.498 INFO 21062 --- [ocp-schedules-13,b82dc8b3146cd257,9603f1c879de22cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:42:28.499 INFO 21062 --- [ocp-schedules-10,4a1e87c904b32028,95d8e999509099fb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:42:28.500 INFO 21062 --- [ocp-schedules-28,7ac43a69e09dd64c,66a9f4f7e4a28b99] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:42:28.500 INFO 21062 --- [ocp-schedules-26,f7c4297a7c804790,5343e78b65d578f5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:42:28.502 INFO 21062 --- [ocp-schedules-13,b82dc8b3146cd257,9603f1c879de22cb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:42:28.502 INFO 21062 --- [ocp-schedules-25,e6dde51bd36811a5,fd5c1ceb6330f711] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:42:28.503 INFO 21062 --- [ocp-schedules-28,7ac43a69e09dd64c,66a9f4f7e4a28b99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:42:28.504 INFO 21062 --- [ocp-schedules-26,f7c4297a7c804790,5343e78b65d578f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:28.504 INFO 21062 --- [ocp-schedules-6,05c25dcc38c6e5b5,12d7903d0edf3075] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:42:28.505 INFO 21062 --- [ocp-schedules-10,4a1e87c904b32028,95d8e999509099fb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:42:28.505 INFO 21062 --- [ocp-schedules-29,68d1fd5451efe97b,14de321dd19f1739] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:42:28.507 INFO 21062 --- [ocp-schedules-6,05c25dcc38c6e5b5,12d7903d0edf3075] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:28.509 INFO 21062 --- [ocp-schedules-19,6cc202a58672c589,6ef0679eb8b0980a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:42:28.509 INFO 21062 --- [ocp-schedules-20,175afd6193a6fce1,cde42642b70e1c55] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:42:28.509 INFO 21062 --- [ocp-schedules-20,175afd6193a6fce1,cde42642b70e1c55] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:42:28.509 INFO 21062 --- [ocp-schedules-19,6cc202a58672c589,6ef0679eb8b0980a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:42:28.510 INFO 21062 --- [ocp-schedules-2,9d9ddcb5aceddda2,02622c4ed6b17c3c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:42:28.511 INFO 21062 --- [ocp-schedules-2,9d9ddcb5aceddda2,02622c4ed6b17c3c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:42:28.543 INFO 21062 --- [ocp-schedules-14,3263f10f03925d08,71aacb14b6b1ee39] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:42:28.543 INFO 21062 --- [ocp-schedules-14,3263f10f03925d08,71aacb14b6b1ee39] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:42:28.581 INFO 21062 --- [pool-12-thread-9,f74ccb72e24f3a59,d1480e2d8675d973] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:42:28.582 INFO 21062 --- [pool-12-thread-11,f74ccb72e24f3a59,574fbe1edd6dbaec] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:42:28.582 INFO 21062 --- [pool-12-thread-12,f74ccb72e24f3a59,ebbe7e2122020235] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:42:28.582 INFO 21062 --- [pool-12-thread-11,f74ccb72e24f3a59,574fbe1edd6dbaec] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:42:28.582 INFO 21062 --- [pool-12-thread-10,f74ccb72e24f3a59,b1f24f2202ef385f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:42:28.585 INFO 21062 --- [pool-12-thread-11,f74ccb72e24f3a59,574fbe1edd6dbaec] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:42:28.586 INFO 21062 --- [pool-12-thread-11,f74ccb72e24f3a59,574fbe1edd6dbaec] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:42:28.587 INFO 21062 --- [pool-12-thread-9,f74ccb72e24f3a59,d1480e2d8675d973] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:42:28.587 INFO 21062 --- [pool-12-thread-9,f74ccb72e24f3a59,d1480e2d8675d973] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:42:28.587 INFO 21062 --- [ForkJoinPool-11-worker-10,f74ccb72e24f3a59,a019149c462e3832] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:42:28.587 INFO 21062 --- [pool-12-thread-10,f74ccb72e24f3a59,b1f24f2202ef385f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:42:28.587 INFO 21062 --- [ForkJoinPool-11-worker-3,f74ccb72e24f3a59,99807a1887d5ca08] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:42:28.587 INFO 21062 --- [ForkJoinPool-11-worker-3,f74ccb72e24f3a59,99807a1887d5ca08] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:42:28.587 INFO 21062 --- [ForkJoinPool-11-worker-3,f74ccb72e24f3a59,99807a1887d5ca08] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:42:28.587 INFO 21062 --- [pool-12-thread-12,f74ccb72e24f3a59,ebbe7e2122020235] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:42:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:42:28.589053+08:00, lastProcessTime=2024-11-04T15:42:18.589519+08:00 2024-11-04 15:42:28.591 INFO 21062 --- [ocp-schedules-15,6bcc878000918ad4,aa3825b7d89aa5f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:42:28.591 INFO 21062 --- [ocp-schedules-15,6bcc878000918ad4,aa3825b7d89aa5f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:42:28.676 INFO 21062 --- [ocp-schedules-17,65e462b75b662ab0,71ae43d1ff7f966e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:42:28.677 INFO 21062 --- [ocp-schedules-17,65e462b75b662ab0,71ae43d1ff7f966e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:42:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:42:28.589053+08:00 2024-11-04 15:42:28.816 INFO 21062 --- [ocp-async-10,e932c83477137272,870fd413f37c3a6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:42:29.019 INFO 21062 --- [ocp-schedules-2,73efb6bc94ebb5c2,c11501df59fba1e1] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:42:29.019 INFO 21062 --- [ocp-schedules-2,73efb6bc94ebb5c2,c11501df59fba1e1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:42:29.034 INFO 21062 --- [ocp-schedules-20,25aa127ec50f8d92,e5ed72acf5266198] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:42:29.043 INFO 21062 --- [ocp-schedules-5,1b578c16bbca3c76,387f52b62884be00] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 15:42:29.043 INFO 21062 --- [ocp-schedules-5,1b578c16bbca3c76,387f52b62884be00] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 15:42:29.043 INFO 21062 --- [ocp-schedules-5,1b578c16bbca3c76,387f52b62884be00] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=422 2024-11-04 15:42:29.062 INFO 21062 --- [ocp-schedules-5,1b578c16bbca3c76,387f52b62884be00] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=422 2024-11-04 15:42:29.391 INFO 21062 --- [ocp-schedules-21,64bdc5447b3bd651,3246c0af094687cd] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:42:29.394 INFO 21062 --- [ocp-schedules-21,64bdc5447b3bd651,3246c0af094687cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:29.395 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:42:29.395 INFO 21062 --- [ocp-schedules-8,e5b339c975697948,96dd0af93846f05c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 15:42:29.398 INFO 21062 --- [ocp-schedules-8,e5b339c975697948,96dd0af93846f05c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:29.436 ERROR 21062 --- [ocp-schedules-8,e5b339c975697948,96dd0af93846f05c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.GeneratedMethodAccessor1314.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:42:29.436 INFO 21062 --- [ocp-schedules-8,e5b339c975697948,96dd0af93846f05c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 15:42:29.466 INFO 21062 --- [ocp-schedules-27,6d7ddc3a7ce04e6b,3dea8873bbe637ee] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:42:29.466 INFO 21062 --- [ocp-schedules-27,6d7ddc3a7ce04e6b,3dea8873bbe637ee] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:42:29.614 INFO 21062 --- [ocp-schedules-24,a62c542b390309d2,0956d144d84df1c6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:42:29.630 INFO 21062 --- [ocp-schedules-11,4b218dfec4a02c73,172eb034b368a229] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:42:29.635 INFO 21062 --- [ocp-schedules-11,4b218dfec4a02c73,172eb034b368a229] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:42:29.635 INFO 21062 --- [ocp-schedules-11,4b218dfec4a02c73,172eb034b368a229] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:42:29.638 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:42:29.654 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:42:29.656 WARN 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:42:29.656 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:42:29.657 WARN 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:42:29.664 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:42:29.665 WARN 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:42:29.665 WARN 21062 --- [ocp-async-2,432ae3b90c95bbc8,55c6e5428b6b809a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:29.666 INFO 21062 --- [ocp-async-1,7dcb83a5c0c14216,17971b1cceea6369] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:29.666 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:42:29.668 WARN 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:42:29.674 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:42:29.675 WARN 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:42:29.676 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:42:29.677 WARN 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:42:29.683 INFO 21062 --- [ocp-schedules-22,7b0ef76dc1698e23,37f203e263a10996] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:42:29.983 INFO 21062 --- [prometheus-minute-schedule-1,dc916ea753deedea,751db3af64aa54b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:30.106 INFO 21062 --- [ocp-schedules-7,2262eb3901feb005,d3eb615cff6d5d13] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:42:30.154 INFO 21062 --- [ocp-schedules-15,156e247dfe63aa44,24e3980c3797b73a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:42:30.154 INFO 21062 --- [ocp-schedules-15,156e247dfe63aa44,24e3980c3797b73a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:42:30.155 INFO 21062 --- [ocp-schedules-15,156e247dfe63aa44,24e3980c3797b73a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:42:30.155 INFO 21062 --- [ocp-schedules-15,156e247dfe63aa44,24e3980c3797b73a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:42:30.156 INFO 21062 --- [ocp-schedules-15,156e247dfe63aa44,24e3980c3797b73a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:42:30.156 INFO 21062 --- [ocp-schedules-15,156e247dfe63aa44,24e3980c3797b73a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:42:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:42:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:42:30.465 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:30.465 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:30.660 INFO 21062 --- [ocp-schedules-4,fed6480d5153f7f2,9b32ba33a819bb1a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:42:30.660 INFO 21062 --- [ocp-schedules-4,fed6480d5153f7f2,9b32ba33a819bb1a] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:42:30.689 INFO 21062 --- [pool-57-thread-1,fed6480d5153f7f2,4263d794699e5ffe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:42:30.717 WARN 21062 --- [ocp-schedules-4,fed6480d5153f7f2,9b32ba33a819bb1a] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:42:30.717 INFO 21062 --- [ocp-schedules-4,fed6480d5153f7f2,9b32ba33a819bb1a] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:42:31.666 WARN 21062 --- [ocp-async-7,391ba8f3727a89c5,d601ec3c2d5ddbb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:31.666 INFO 21062 --- [ocp-async-6,29d11093f69909a3,80ad79665528c3a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:31.967 INFO 21062 --- [ocp-schedules-3,db5da4564486d592,929adb0b545a5c99] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:42:32.983 INFO 21062 --- [prometheus-minute-schedule-1,ebd91cb989f917c7,8794581f841c2973] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:33.444 INFO 21062 --- [ocp-schedules-29,06e57678aa6a62c5,a426c01bd59f7db2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:42:33.448 INFO 21062 --- [ocp-schedules-29,06e57678aa6a62c5,a426c01bd59f7db2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:42:33.448 INFO 21062 --- [ocp-schedules-29,06e57678aa6a62c5,a426c01bd59f7db2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:42:33.465 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:33.466 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:33.666 WARN 21062 --- [ocp-async-3,daf1f3cbaa3c5a54,ff8895cc7e67e994] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:33.666 INFO 21062 --- [ocp-async-9,5690e4f8b4bb8d23,257cde8f303df3ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:33.765 INFO 21062 --- [ocp-schedules-24,86f9c230f0ce5ff7,f3fd1960f057d96a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:42:33.765 INFO 21062 --- [ocp-schedules-24,86f9c230f0ce5ff7,f3fd1960f057d96a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:42:33.792 INFO 21062 --- [ocp-schedules-24,86f9c230f0ce5ff7,f3fd1960f057d96a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:42:33.792 INFO 21062 --- [ocp-schedules-24,86f9c230f0ce5ff7,f3fd1960f057d96a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:42:33.793 INFO 21062 --- [ocp-schedules-12,468b3c39af671678,d970942fda9243e5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:42:33.965 INFO 21062 --- [ocp-schedules-14,9bd1c4dc657662d4,24f34c237749d867] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:42:33.965+08:00, clusters=[1] 2024-11-04 15:42:33.967 INFO 21062 --- [ocp-schedules-14,9bd1c4dc657662d4,24f34c237749d867] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:42:34.318 INFO 21062 --- [ocp-schedules-25,6bc66f7bdebb8dbb,a59b1c15b6198e55] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:42:34.318+08:00, clusters=[1] 2024-11-04 15:42:34.321 INFO 21062 --- [ocp-schedules-25,6bc66f7bdebb8dbb,a59b1c15b6198e55] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:42:34.425 INFO 21062 --- [ocp-schedules-4,69e1ed57ca832d0a,3a98f80144b470ab] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:42:34.428 INFO 21062 --- [ocp-schedules-4,69e1ed57ca832d0a,3a98f80144b470ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:34.485 ERROR 21062 --- [ocp-schedules-4,69e1ed57ca832d0a,3a98f80144b470ab] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:42:34.485 INFO 21062 --- [ocp-schedules-4,69e1ed57ca832d0a,3a98f80144b470ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:42:35.666 WARN 21062 --- [ocp-async-2,8b147d56c54cb4f0,c2a8a562e7bc3b2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:35.666 INFO 21062 --- [ocp-async-10,57493b880f380cfc,507458c781e51db5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:35.983 INFO 21062 --- [prometheus-minute-schedule-1,446bd743df985779,ff6594e5fc6ee1a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:36.390 INFO 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:42:36.390 INFO 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:42:36.390 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:42:36.390 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:42:36.400 INFO 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:42:36.400 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:42:36.400 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:42:36.401 WARN 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:42:36.402 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:42:36.402 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:42:36.402 INFO 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:42:36.403 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:42:36.403 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:42:36.404 WARN 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:42:36.404 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:42:36.404 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:42:36.410 INFO 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:42:36.410 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:42:36.410 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:42:36.422 ERROR 21062 --- [ocp-schedules-29,8da92791616d3f7e,bc0a39baa3558fcd] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:42:36.466 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:36.467 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:37.666 WARN 21062 --- [ocp-async-7,ad71d22669969a1b,aefa30d3f61cc244] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:37.666 INFO 21062 --- [ocp-async-5,ca63ed60e2ca1fc8,1652f5ff24a0ef62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:38.504 INFO 21062 --- [ocp-schedules-28,7ac43a69e09dd64c,66a9f4f7e4a28b99] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:42:38.504 INFO 21062 --- [ocp-schedules-26,f7c4297a7c804790,5343e78b65d578f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:42:38.508 INFO 21062 --- [ocp-schedules-6,05c25dcc38c6e5b5,12d7903d0edf3075] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:42:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:42:38.589081+08:00, lastProcessTime=2024-11-04T15:42:28.589053+08:00 2024-11-04 15:42:38.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:42:38.589081+08:00 2024-11-04 15:42:38.816 INFO 21062 --- [ocp-async-8,08cb2517b31810b2,dc5e6358a0e0530b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:42:38.983 INFO 21062 --- [prometheus-minute-schedule-1,1cc1cd7553ca4cf4,73466afb83af9bae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:39.395 INFO 21062 --- [ocp-schedules-21,64bdc5447b3bd651,3246c0af094687cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:42:39.467 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:39.468 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:39.666 WARN 21062 --- [ocp-async-9,27cf0b9c8cbe9b24,e03a10f3c0741ff0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:39.666 INFO 21062 --- [ocp-async-2,c38d617959a27c29,086064d7e56007d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:40.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:42:40.004 INFO 21062 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:42:40.148 INFO 21062 --- [ocp-schedules-1,787b108bd1bb0129,372b20e6adc16986] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:42:41.490 INFO 21062 --- [ocp-schedules-23,e9921d7ef952318d,8d5fcc5893aa1cf2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:42:41.490 INFO 21062 --- [ocp-schedules-23,b9549ab263c5e447,40fa2d7e67de6590] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:42:41.494 INFO 21062 --- [ocp-schedules-23,b9549ab263c5e447,40fa2d7e67de6590] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:41.524 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:42:41.524 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:42:41.666 WARN 21062 --- [ocp-async-10,3c101110d951cea4,866bba52d460c163] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:41.666 INFO 21062 --- [ocp-async-7,2e4689de699558ac,54d5163e5d4cef6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:42:41.824 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:42:41.824 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:42:41.970 INFO 21062 --- [ocp-schedules-10,df9a03a23786517d,0ad3c3e35f01e3a2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:42:41.983 INFO 21062 --- [prometheus-minute-schedule-1,56d52217856ae72c,049db6dc41901a63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:42.468 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:42.469 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:43.667 WARN 21062 --- [ocp-async-1,8aeb30519736fe13,9cefabe6f5a854f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:43.667 INFO 21062 --- [ocp-async-6,a83ffa0831ab6c6c,af0c779acb948b78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:44.983 INFO 21062 --- [prometheus-minute-schedule-1,8df64df0d63e79a7,d204ce2707531444] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:45.469 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:45.470 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:45.667 WARN 21062 --- [ocp-async-3,11c32aafeb254bf1,30de2ecf4a101854] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:45.667 INFO 21062 --- [ocp-async-3,fb6c413c5bfa73db,e33c1f7fa7987357] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:46.093 INFO 21062 --- [ocp-schedules-1,396e53da9a5b1086,545bd3239e49ce2c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:42:46.097 INFO 21062 --- [ocp-schedules-1,396e53da9a5b1086,545bd3239e49ce2c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:46.179 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:42:46.182 ERROR 21062 --- [ocp-schedules-1,396e53da9a5b1086,545bd3239e49ce2c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:42:46.182 INFO 21062 --- [ocp-schedules-1,396e53da9a5b1086,545bd3239e49ce2c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:42:47.667 WARN 21062 --- [ocp-async-2,4731491dea3d483d,364e3bb379c567fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:47.667 INFO 21062 --- [ocp-async-10,61101ee1381693aa,f4c26c4b6f5dcfa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:47.983 INFO 21062 --- [prometheus-minute-schedule-1,d7628d482939161c,5943c4d6c4e62457] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:48.470 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:48.471 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:48.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:42:48.591411+08:00, lastProcessTime=2024-11-04T15:42:38.589081+08:00 2024-11-04 15:42:48.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:42:48.591411+08:00 2024-11-04 15:42:48.817 INFO 21062 --- [ocp-async-1,c4f82b4d2af41356,fb111f5f53150049] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:42:49.667 WARN 21062 --- [ocp-async-1,cf3fafc671c2d04a,9c9eb2c0e904cfa9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:49.667 INFO 21062 --- [ocp-async-7,5e8d55f309e84d99,3e41cc10aaa54216] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:50.175 INFO 21062 --- [ocp-schedules-10,fdb41711f1359680,89801ade7b452e5b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:42:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ecf202611b86dbc9,9a1fedfb03861f39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:51.471 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:51.471 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:51.667 WARN 21062 --- [ocp-async-8,35eb94486706fb65,9c4f937f9c76121a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:51.667 INFO 21062 --- [ocp-async-6,6ecfffa8b7f21d3f,7dee7051fad766eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:51.974 INFO 21062 --- [ocp-schedules-28,87e983cbfaea7154,91eb1cd94229bdbf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:42:53.450 INFO 21062 --- [ocp-schedules-3,0795d9ad38911e2f,68f7c1290230a129] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:42:53.456 INFO 21062 --- [ocp-schedules-3,0795d9ad38911e2f,68f7c1290230a129] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:42:53.456 INFO 21062 --- [ocp-schedules-3,0795d9ad38911e2f,68f7c1290230a129] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:42:53.668 WARN 21062 --- [ocp-async-3,1dff2775f3ae926a,082b786893d564b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:53.668 INFO 21062 --- [ocp-async-4,dade4ac9bb16544e,c26b658e4cecb387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:53.983 INFO 21062 --- [prometheus-minute-schedule-1,9af85950cd694938,b75bfa9616b5725e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:54.472 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:54.473 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:55.668 WARN 21062 --- [ocp-async-1,c4e135565b3b0476,493ccd6acec1f658] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:55.668 INFO 21062 --- [ocp-async-5,7add686f830d094f,421b79370f76cc5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:56.983 INFO 21062 --- [prometheus-minute-schedule-1,de85cdc36b9b4431,ca3e4a05460d1541] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:42:57.473 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:42:57.474 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:42:57.668 WARN 21062 --- [ocp-async-8,7fd92adb84f6f803,37bfeed50360f73b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:57.668 INFO 21062 --- [ocp-async-8,bb8fed2627c2873d,69bdf1805d85d6ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:58.499 INFO 21062 --- [ocp-schedules-14,7893aee2f6ea3a81,86e02fdbb62159dd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:42:58.500 INFO 21062 --- [ocp-schedules-16,f921b9dadc480c79,9e00b0a8edaed2f9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:42:58.502 INFO 21062 --- [ocp-schedules-25,268d3a2d60e26c36,4c2d67d8c477dfdd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:42:58.503 INFO 21062 --- [ocp-schedules-14,7893aee2f6ea3a81,86e02fdbb62159dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:42:58.504 INFO 21062 --- [ocp-schedules-16,f921b9dadc480c79,9e00b0a8edaed2f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:42:58.507 INFO 21062 --- [ocp-schedules-21,2f8f9ef0fd93485b,7a1575c70fa0a231] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:42:58.509 INFO 21062 --- [ocp-schedules-8,caf6f97bf3e0c584,47e16389b6ad8f14] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:42:58.509 INFO 21062 --- [ocp-schedules-11,9e1d06f0c64ca354,ff697ca00c3e2cb5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:42:58.509 INFO 21062 --- [ocp-schedules-15,7068e9d922a68c42,d107f48b8e99bc4e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:42:58.509 INFO 21062 --- [ocp-schedules-11,9e1d06f0c64ca354,ff697ca00c3e2cb5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:42:58.509 INFO 21062 --- [ocp-schedules-8,caf6f97bf3e0c584,47e16389b6ad8f14] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:42:58.509 INFO 21062 --- [ocp-schedules-15,7068e9d922a68c42,d107f48b8e99bc4e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:42:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:42:58.590352+08:00, lastProcessTime=2024-11-04T15:42:48.591411+08:00 2024-11-04 15:42:58.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:42:58.590352+08:00 2024-11-04 15:42:58.817 INFO 21062 --- [ocp-async-2,62ec7a971e0b984c,068af7bad3437887] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:42:59.617 INFO 21062 --- [ocp-schedules-9,919a0da76a47fe3e,e10deefda08b6ce9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:42:59.668 WARN 21062 --- [ocp-async-2,b69e33b055b0bc38,b52aca89373969d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:42:59.668 INFO 21062 --- [ocp-async-4,3550e9f80117f179,f7efa810c13212b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:42:59.983 INFO 21062 --- [prometheus-minute-schedule-1,51ba28260ff9c0ba,cb3414ce44f74162] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706180, interval=60 2024-11-04 15:43:00.252 INFO 21062 --- [ocp-schedules-22,c80d9c6188ec5f4d,2077708727dfca70] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:43:00.475 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:00.475 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:01.668 WARN 21062 --- [ocp-async-10,04cbd15d417a4688,164bb5c2afa5777d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:01.668 INFO 21062 --- [ocp-async-5,168fd3ac2623baf1,d3eff9ca80d5ec77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:01.977 INFO 21062 --- [ocp-schedules-30,ff2065b7b4167993,f61071c6687eea0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:43:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2cc4b8fa384a7c62,55fc0483cf8a961a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:03.475 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:03.476 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:03.669 WARN 21062 --- [ocp-async-9,288fbcafd110dbb6,1dcaaa482655fc8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:03.669 INFO 21062 --- [ocp-async-9,ba2621e2f727cff3,a9cf6f8801fa45a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:05.669 WARN 21062 --- [ocp-async-2,a82cd141a0aae75a,4c4eecf6987750a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:05.669 INFO 21062 --- [ocp-async-3,d5eb1ac53f8b69ba,d2b757ac85dc19f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:05.983 INFO 21062 --- [prometheus-minute-schedule-1,3fb29e1bf9c8448d,aab5b8808e85dcb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:06.476 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:06.477 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:07.669 WARN 21062 --- [ocp-async-10,684ec1c519288e15,f7651e1188c6c4e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:07.669 INFO 21062 --- [ocp-async-1,667271b788ca9c2e,85cb44ac4e27f848] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:08.504 INFO 21062 --- [ocp-schedules-14,7893aee2f6ea3a81,86e02fdbb62159dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:43:08.507 INFO 21062 --- [ocp-schedules-16,f921b9dadc480c79,9e00b0a8edaed2f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:43:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:43:08.588730+08:00, lastProcessTime=2024-11-04T15:42:58.590352+08:00 2024-11-04 15:43:08.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:43:08.588730+08:00 2024-11-04 15:43:08.817 INFO 21062 --- [ocp-async-7,8ab64fc7b0b28b1f,b73777f996093986] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:43:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a498d81d1b50a7fe,84d87965ece05fa1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:09.478 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:09.479 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:09.669 WARN 21062 --- [ocp-async-7,ad6ccd5ef2ae4bd5,298c7a1bfcd7f7c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:09.669 INFO 21062 --- [ocp-async-7,0fc9e2749dbf2b83,809fc5af51b585cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:10.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:43:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:43:10.281 INFO 21062 --- [ocp-schedules-20,f2c394f4c87abcd6,708c11cc688244fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:43:10.307 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:43:10.307 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:43:10.308 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:43:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:43:10.326 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:43:10.326 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:43:11.524 INFO 21062 --- [ocp-schedules-23,b9549ab263c5e447,40fa2d7e67de6590] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:43:11.525 INFO 21062 --- [ocp-schedules-23,fd03df9a8180bfea,ba093a852cd92333] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:43:11.529 INFO 21062 --- [ocp-schedules-23,fd03df9a8180bfea,ba093a852cd92333] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:11.559 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:43:11.559 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:43:11.669 WARN 21062 --- [ocp-async-6,b7e8cc0c65845319,5188350e4f018d5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:11.670 INFO 21062 --- [ocp-async-6,5bf878e727f108d3,89e6f9fae3a9bd7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=4, affectRows=4 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=4, validCount=4 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=18, affectRows=18 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=18, validCount=18 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=19, affectRows=19 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=20, affectRows=20 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=22, affectRows=22 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-49,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=19, validCount=19 2024-11-04 15:43:11.911 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=20, validCount=20 2024-11-04 15:43:11.912 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=22, validCount=22 2024-11-04 15:43:11.912 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=43, affectRows=43 2024-11-04 15:43:11.912 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=54, affectRows=54 2024-11-04 15:43:11.912 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=43, validCount=43 2024-11-04 15:43:11.912 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=77, affectRows=77 2024-11-04 15:43:11.912 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=54, validCount=54 2024-11-04 15:43:11.912 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=68, affectRows=68 2024-11-04 15:43:11.913 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=77, validCount=77 2024-11-04 15:43:11.913 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706120, nodeCount=97, affectRows=97 2024-11-04 15:43:11.913 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=68, validCount=68 2024-11-04 15:43:11.913 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706120, nodeCount=94, validCount=94 2024-11-04 15:43:11.980 INFO 21062 --- [ocp-schedules-1,afa3323a7989e029,01edeaaa95382824] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:43:11.983 INFO 21062 --- [prometheus-minute-schedule-1,ba072da5879b7111,cc2c33a2762f13ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:12.479 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:12.480 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:13.458 INFO 21062 --- [ocp-schedules-3,9798cd164e54d927,950fdf4d999b0483] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:43:13.464 INFO 21062 --- [ocp-schedules-3,9798cd164e54d927,950fdf4d999b0483] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:43:13.464 INFO 21062 --- [ocp-schedules-3,9798cd164e54d927,950fdf4d999b0483] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:43:13.670 WARN 21062 --- [ocp-async-4,2051c09341a8960d,cff878dfeda9fe7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:13.670 INFO 21062 --- [ocp-async-1,74e0a638c2dcb62c,690788b5f059ed26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:13.984 INFO 21062 --- [prometheus-cache-monitor1,b1f703b91ca08355,befd7595ab329674] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:43:13.985 INFO 21062 --- [prometheus-cache-monitor1,b1f703b91ca08355,befd7595ab329674] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:43:14.543 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:37:14.519, end=2024-11-04T15:42:14.541} 2024-11-04 15:43:14.544 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:37:14.519, end=2024-11-04T15:42:14.541} 2024-11-04 15:43:14.548 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:43:14.548 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:43:14.548 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:43:14.548 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:43:14.983 INFO 21062 --- [prometheus-minute-schedule-1,e70427f358ad8b39,61e8beea470e74d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:15.480 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:15.481 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:15.670 WARN 21062 --- [ocp-async-9,923e1dea33ea49cb,2e3449c3791f0c2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:15.670 INFO 21062 --- [ocp-async-8,a06e77183ecb471a,b9f184d8b1de2e9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:15.783 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:39:15.76, end=2024-11-04T15:41:15.78} 2024-11-04 15:43:15.787 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:43:15.787 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:43:17.670 WARN 21062 --- [ocp-async-3,175d37c966f17310,da8f79c1a4e72086] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:17.670 INFO 21062 --- [ocp-async-2,659e798481fc7591,e89ce1ca34c59692] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:17.983 INFO 21062 --- [prometheus-minute-schedule-1,457499651cd98652,32e59ccd79c1671b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:18.481 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:18.482 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:43:18.589513+08:00, lastProcessTime=2024-11-04T15:43:08.588730+08:00 2024-11-04 15:43:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:43:18.589513+08:00 2024-11-04 15:43:18.817 INFO 21062 --- [ocp-async-4,278fd1feffbd15ff,c4e50ac9231b56e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:43:19.670 WARN 21062 --- [ocp-async-4,4189baebf1756fd7,f88237e15af03e9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:19.670 INFO 21062 --- [ocp-async-1,517eb14d3e813d58,1e111b8afdebfa2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:20.325 INFO 21062 --- [ocp-schedules-26,eb7f957b1149dd34,6512043560501c96] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:43:20.983 INFO 21062 --- [prometheus-minute-schedule-1,377a07119dc6f4b7,ce9920224c789a5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:21.482 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:21.483 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:21.671 WARN 21062 --- [ocp-async-5,df8006dc0c27eb2b,8b7ab9176ce5efd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:21.671 INFO 21062 --- [ocp-async-5,1b0611c97bee08ba,32becf7583327998] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:21.982 INFO 21062 --- [ocp-schedules-8,8aba215345aad0ec,d7ae45f08c1be602] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:43:23.671 WARN 21062 --- [ocp-async-7,5215ebc51ea0a692,4f4420b4a75725a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:23.671 INFO 21062 --- [ocp-async-2,946b4c0a7d39ffc3,cdaecf4a8913a464] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:23.983 INFO 21062 --- [prometheus-minute-schedule-1,4ae6f962ae4e986f,53b5c3d888cb5d0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:24.483 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:24.484 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:25.671 WARN 21062 --- [ocp-async-4,3f653745af980926,3ca07daa783bce9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:25.671 INFO 21062 --- [ocp-async-4,240fe4db723ebd7f,eb312f80ff97c4a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:26.983 INFO 21062 --- [prometheus-minute-schedule-1,1e6b2816c9ecd256,05634b08b4b23ea3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4d63dab852dd85e8,f82ba3bb65a6f779] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:43:27.484 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:27.485 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:27.671 WARN 21062 --- [ocp-async-8,171cad211d546e8a,bf50d71ee5ed3608] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:27.671 INFO 21062 --- [ocp-async-9,1e7c3e9a64e5676e,c09b5dc28a6dec60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:28.498 INFO 21062 --- [ocp-schedules-2,55c98fb68b64c834,ce305e2229b98e98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:43:28.500 INFO 21062 --- [ocp-schedules-25,b7afc921da6c345d,fad1a277dd8eadce] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:43:28.500 INFO 21062 --- [ocp-schedules-14,58243d934075e463,624adea5964f3295] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:43:28.500 INFO 21062 --- [ocp-schedules-16,d312aa025394996c,83196c94e3c68e3e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:43:28.502 INFO 21062 --- [ocp-schedules-2,55c98fb68b64c834,ce305e2229b98e98] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:43:28.502 INFO 21062 --- [ocp-schedules-7,a5c4bfa0db5f95a8,028a2da83bce5836] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:43:28.503 INFO 21062 --- [ocp-schedules-14,58243d934075e463,624adea5964f3295] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:28.504 INFO 21062 --- [ocp-schedules-16,d312aa025394996c,83196c94e3c68e3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:43:28.504 INFO 21062 --- [ocp-schedules-28,d6c52577406e3188,4fea669d1a16ca03] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:43:28.505 INFO 21062 --- [ocp-schedules-25,b7afc921da6c345d,fad1a277dd8eadce] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:43:28.510 INFO 21062 --- [ocp-schedules-24,678f72beb7e899ea,f664e5df3b1c9ce3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:43:28.510 INFO 21062 --- [ocp-schedules-24,678f72beb7e899ea,f664e5df3b1c9ce3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:43:28.510 INFO 21062 --- [ocp-schedules-4,9d8ec0422456442c,52d2f948836095a5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:43:28.510 INFO 21062 --- [ocp-schedules-18,d60699b84282f26d,40795e9303c9e740] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:43:28.510 INFO 21062 --- [ocp-schedules-18,d60699b84282f26d,40795e9303c9e740] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:43:28.510 INFO 21062 --- [ocp-schedules-4,9d8ec0422456442c,52d2f948836095a5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:43:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:43:28.589080+08:00, lastProcessTime=2024-11-04T15:43:18.589513+08:00 2024-11-04 15:43:28.608 INFO 21062 --- [ocp-schedules-30,ed17c099a66fe9e2,165b1e587534e498] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 15:43:28.608 INFO 21062 --- [ocp-schedules-30,ed17c099a66fe9e2,165b1e587534e498] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 15:43:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:43:28.589080+08:00 2024-11-04 15:43:28.817 INFO 21062 --- [ocp-async-2,75033e1ebb80226b,44332bdbaa868d97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:43:29.036 INFO 21062 --- [ocp-schedules-25,cc02f0fce87ddd9f,0c77a124f44df6db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:43:29.470 INFO 21062 --- [ocp-schedules-7,eab811b5b1faa8c4,6a5ef27ff5c30fcf] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:43:29.470 INFO 21062 --- [ocp-schedules-7,eab811b5b1faa8c4,6a5ef27ff5c30fcf] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:43:29.620 INFO 21062 --- [ocp-schedules-1,ac9a3e1bab1492f9,2352cad29ce52cf5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:43:29.635 INFO 21062 --- [ocp-schedules-19,f24f7d8afbadff02,2b8c0a5d399fee47] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:43:29.640 INFO 21062 --- [ocp-schedules-19,f24f7d8afbadff02,2b8c0a5d399fee47] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:43:29.640 INFO 21062 --- [ocp-schedules-19,f24f7d8afbadff02,2b8c0a5d399fee47] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:43:29.671 WARN 21062 --- [ocp-async-10,00f7480fcb19f5d1,e2c986a458765dad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:29.671 INFO 21062 --- [ocp-async-6,8949ad1a7f82c87b,9c00c2e0905a32d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:29.976 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:43:29.976 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:43:29.983 INFO 21062 --- [prometheus-minute-schedule-1,1ee1963a4916b411,e592689b9df633bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:29.985 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:43:29.985 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:43:29.985 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:43:29.996 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:43:29.996 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:43:29.996 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:43:29.999 INFO 21062 --- [ocp-schedules-15,4e007d50740edb8f,f2cca0017c9505d1] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:43:30.157 INFO 21062 --- [ocp-schedules-28,2b2845c858a3ea03,47f22b6756a86edf] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:43:30.157 INFO 21062 --- [ocp-schedules-28,2b2845c858a3ea03,47f22b6756a86edf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:43:30.158 INFO 21062 --- [ocp-schedules-28,2b2845c858a3ea03,47f22b6756a86edf] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:43:30.158 INFO 21062 --- [ocp-schedules-28,2b2845c858a3ea03,47f22b6756a86edf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:43:30.158 INFO 21062 --- [ocp-schedules-28,2b2845c858a3ea03,47f22b6756a86edf] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:43:30.158 INFO 21062 --- [ocp-schedules-28,2b2845c858a3ea03,47f22b6756a86edf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:43:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:43:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:43:30.355 INFO 21062 --- [ocp-schedules-24,b8287ae0da7c2076,33bf3cde9b07e522] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:43:30.485 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:30.486 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:31.672 WARN 21062 --- [ocp-async-9,c8a2ebab983f32e8,034f0ad4fa70ce74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:31.672 INFO 21062 --- [ocp-async-8,8d510d4411b569ae,3ec20f4c92f0483e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:31.986 INFO 21062 --- [ocp-schedules-29,aef876334d9d3b47,ceab4f6e364964b8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:43:32.983 INFO 21062 --- [prometheus-minute-schedule-1,060cf98fc3d54fd3,08605793890b3bbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:33.466 INFO 21062 --- [ocp-schedules-25,daf4c72c65102be7,ced775e01a265af2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:43:33.471 INFO 21062 --- [ocp-schedules-25,daf4c72c65102be7,ced775e01a265af2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:43:33.471 INFO 21062 --- [ocp-schedules-25,daf4c72c65102be7,ced775e01a265af2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:43:33.486 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:33.487 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:33.672 WARN 21062 --- [ocp-async-7,0a0fe1e70f998a99,c5c6e12577668f53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:33.672 INFO 21062 --- [ocp-async-5,7a0e2bdb4c2e4b65,a99b9d7c76cff23e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:33.792 INFO 21062 --- [ocp-schedules-28,bef58be2d49de877,28e484cadb413fce] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:43:33.792 INFO 21062 --- [ocp-schedules-28,bef58be2d49de877,28e484cadb413fce] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:43:33.819 INFO 21062 --- [ocp-schedules-28,bef58be2d49de877,28e484cadb413fce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:43:33.819 INFO 21062 --- [ocp-schedules-28,bef58be2d49de877,28e484cadb413fce] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:43:33.819 INFO 21062 --- [ocp-schedules-19,a0f195cff26dc39e,008524e58e684627] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:43:33.995 INFO 21062 --- [ocp-schedules-7,632b743b95e046d8,c19381a543c9c614] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:43:33.995+08:00, clusters=[1] 2024-11-04 15:43:33.997 INFO 21062 --- [ocp-schedules-7,632b743b95e046d8,c19381a543c9c614] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:43:34.346 INFO 21062 --- [ocp-schedules-11,66adbfc8e8d58948,bc00f92ed25ac803] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:43:34.346+08:00, clusters=[1] 2024-11-04 15:43:34.349 INFO 21062 --- [ocp-schedules-11,66adbfc8e8d58948,bc00f92ed25ac803] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:43:35.021 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:43:35.021 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:43:35.030 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:43:35.030 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:43:35.030 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:43:35.030 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:43:35.036 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:43:35.036 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:43:35.036 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:43:35.036 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:43:35.072 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:43:35.081 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:43:35.081 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:43:35.081 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:43:35.081 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:43:35.081 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:43:35.089 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:43:35.605 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:43:35.672 WARN 21062 --- [ocp-async-2,7b539eda8103a955,5a704aa855d9f5cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:35.672 INFO 21062 --- [ocp-async-10,026f056cb423e43f,e11c684d0580c1fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:35.983 INFO 21062 --- [prometheus-minute-schedule-1,be2c5ac110f1ce11,85cbee2a484d6530] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:36.184 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.14830202S 2024-11-04 15:43:36.185 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:43:36.277 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:43:36.277 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:43:36.306 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:43:36.362 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:43:36.362 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:43:36.362 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:43:36.362 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:43:36.362 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:43:36.362 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:43:36.373 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:43:36.431 INFO 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:43:36.431 INFO 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:43:36.431 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:43:36.431 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:43:36.441 INFO 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:43:36.441 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:43:36.441 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:43:36.442 WARN 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:43:36.443 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:43:36.443 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:43:36.443 INFO 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:43:36.444 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:43:36.444 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:43:36.444 WARN 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:43:36.445 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:43:36.445 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:43:36.451 INFO 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:43:36.451 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:43:36.451 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:43:36.462 ERROR 21062 --- [ocp-schedules-17,511bc678deaa5f63,9b2f4a2e3f6309c6] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:43:36.487 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:36.488 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:36.552 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] o.s.boot.SpringApplication : Started application in 1.522 seconds (JVM running for 11630.231) 2024-11-04 15:43:36.552 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.522570924S 2024-11-04 15:43:36.947 INFO 21062 --- [ocp-schedules-22,d208d01b0ace2514,030b571beff80830] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:43:37.672 INFO 21062 --- [ocp-async-9,b3389b8564ac36a8,c3d9b7c60315d93c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:37.672 WARN 21062 --- [ocp-async-4,6df8887d5d4b414d,a39ea84c401b4911] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:38.504 INFO 21062 --- [ocp-schedules-14,58243d934075e463,624adea5964f3295] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:43:38.504 INFO 21062 --- [ocp-schedules-16,d312aa025394996c,83196c94e3c68e3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:43:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:43:38.588024+08:00, lastProcessTime=2024-11-04T15:43:28.589080+08:00 2024-11-04 15:43:38.742 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:43:38.588024+08:00 2024-11-04 15:43:38.818 INFO 21062 --- [ocp-async-1,71fe499f2f14ccbb,9349a6905fddbc6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:43:38.983 INFO 21062 --- [prometheus-minute-schedule-1,94e9a60b178054f7,e21c1f646cae6a7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:39.395 INFO 21062 --- [ocp-schedules-6,c5b5637d3af62e50,9725d90f38ca71bf] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:43:39.399 INFO 21062 --- [ocp-schedules-6,c5b5637d3af62e50,9725d90f38ca71bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:39.399 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:43:39.488 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:39.488 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:39.672 INFO 21062 --- [ocp-async-5,3ffa022c371696b9,13919149365bcde6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:39.672 WARN 21062 --- [ocp-async-2,e4308a8dfdfabf12,834d735c176c667a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:43:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:43:40.397 INFO 21062 --- [ocp-schedules-30,43729afedba239f5,4bbd6674f5c9726e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:43:41.559 INFO 21062 --- [ocp-schedules-23,fd03df9a8180bfea,ba093a852cd92333] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:43:41.560 INFO 21062 --- [ocp-schedules-23,66eba2b9915fd39b,c45c6a39030bca49] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:43:41.564 INFO 21062 --- [ocp-schedules-23,66eba2b9915fd39b,c45c6a39030bca49] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:41.592 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:43:41.592 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:43:41.673 WARN 21062 --- [ocp-async-4,17c82a9a68c525cd,4b82d5a8a0d5e473] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:41.673 INFO 21062 --- [ocp-async-10,b4274e4826c76dfd,49e24a78143c455e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:41.983 INFO 21062 --- [prometheus-minute-schedule-1,0e9cacc84beaca1a,1a4ec4c9368179b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:41.989 INFO 21062 --- [ocp-schedules-11,4cd6cf15cd0b35ae,3388f6438a03da8e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:43:42.489 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:42.489 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:43.673 WARN 21062 --- [ocp-async-9,04da224eebf881bc,3d6d6d722256c58e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:43.673 INFO 21062 --- [ocp-async-6,f573691d9af3eae5,baa564eb2b3c2396] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e7ae423c435a0902,415039ec9e74b4d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:45.489 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:45.491 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:45.673 WARN 21062 --- [ocp-async-7,7ae5cd38dbc38a20,9e5c6e24f1f91ea2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:45.673 INFO 21062 --- [ocp-async-2,aa4778e98824f54c,7718852893ecad8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:46.182 INFO 21062 --- [ocp-schedules-28,ae6b22a8542ecf79,45116a9c864bb0a3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:43:46.186 INFO 21062 --- [ocp-schedules-28,ae6b22a8542ecf79,45116a9c864bb0a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:46.269 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:43:46.271 ERROR 21062 --- [ocp-schedules-28,ae6b22a8542ecf79,45116a9c864bb0a3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:43:46.271 INFO 21062 --- [ocp-schedules-28,ae6b22a8542ecf79,45116a9c864bb0a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:43:47.673 WARN 21062 --- [ocp-async-3,d114eff003e38f5d,5d1a810a54d43d49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:47.673 INFO 21062 --- [ocp-async-4,7e949b8797c56d3c,832a82c05d860d24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:47.983 INFO 21062 --- [prometheus-minute-schedule-1,c25e141a9812218b,472c37cdacd391e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:48.491 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:48.492 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:43:48.589637+08:00, lastProcessTime=2024-11-04T15:43:38.588024+08:00 2024-11-04 15:43:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:43:48.589637+08:00 2024-11-04 15:43:48.818 INFO 21062 --- [ocp-async-8,31bd0ec6d5b9c9b7,4142fa0afdd59ace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:43:49.399 INFO 21062 --- [ocp-schedules-6,c5b5637d3af62e50,9725d90f38ca71bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:43:49.673 WARN 21062 --- [ocp-async-8,bbe5d465012ad4b6,908002bc56c7e807] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:49.673 INFO 21062 --- [ocp-async-10,ddc02ca7eb5ca4e3,715cae81d690f900] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:50.431 INFO 21062 --- [ocp-schedules-24,480c02f4032e8487,d3ed63318d9521af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:43:50.983 INFO 21062 --- [prometheus-minute-schedule-1,2ca52f6ac8059ca6,f8474666ef5c2137] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:51.492 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:51.494 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:51.673 WARN 21062 --- [ocp-async-6,3e091bc31d07e7c0,b01c12aab0f12032] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:51.674 INFO 21062 --- [ocp-async-6,537f7e8f53419f2f,68bb9e69ec6bb5d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:51.991 INFO 21062 --- [ocp-schedules-2,8481a7d731b50e72,3248d38981bc60f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:43:53.474 INFO 21062 --- [ocp-schedules-5,4b0ea96f0d4f7ecb,69dc6d5f0646691c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:43:53.480 INFO 21062 --- [ocp-schedules-5,4b0ea96f0d4f7ecb,69dc6d5f0646691c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:43:53.480 INFO 21062 --- [ocp-schedules-5,4b0ea96f0d4f7ecb,69dc6d5f0646691c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:43:53.674 WARN 21062 --- [ocp-async-2,1c0fb2cc76e3fc78,d3d290c72ae51d31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:53.674 INFO 21062 --- [ocp-async-5,5389d411dd1f2e4f,7605ba90a1b469be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:53.983 INFO 21062 --- [prometheus-minute-schedule-1,aa9c9bee6780d790,94e6a3983068dc70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:54.494 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:54.495 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:55.674 WARN 21062 --- [ocp-async-8,2879ca6092e9ccc3,3de657c23ff3974b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:55.674 INFO 21062 --- [ocp-async-8,0106d863eb28c2a0,63d71f0c91fa59f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:56.983 INFO 21062 --- [prometheus-minute-schedule-1,e6d6732997e383a2,24827721d3bcd016] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:43:57.495 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:43:57.496 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:43:57.674 WARN 21062 --- [ocp-async-1,0f9be387e52f4712,f8831c8efc84dc8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:57.674 INFO 21062 --- [ocp-async-10,d2c9b64ba84e180c,599e5638441d2edc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:58.500 INFO 21062 --- [ocp-schedules-27,004d3ef8e05dd23d,dd634332ccaa95d8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:43:58.500 INFO 21062 --- [ocp-schedules-20,088aa2bbae033848,9202f63fb42facb1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:43:58.504 INFO 21062 --- [ocp-schedules-27,004d3ef8e05dd23d,dd634332ccaa95d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:58.504 INFO 21062 --- [ocp-schedules-20,088aa2bbae033848,9202f63fb42facb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:43:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:43:58.506 INFO 21062 --- [ocp-schedules-29,6c859ef6fc0ff457,061c54480b70e520] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:43:58.506 INFO 21062 --- [ocp-schedules-18,d903e92a1942a686,e4241752e2f718fd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:43:58.510 INFO 21062 --- [ocp-schedules-1,b0383edecc20acfd,74b44978e15d07c2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:43:58.510 INFO 21062 --- [ocp-schedules-14,003815d81098d9c3,8ad84ecb99d89b0d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:43:58.510 INFO 21062 --- [ocp-schedules-1,b0383edecc20acfd,74b44978e15d07c2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:43:58.510 INFO 21062 --- [ocp-schedules-14,003815d81098d9c3,8ad84ecb99d89b0d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:43:58.513 INFO 21062 --- [ocp-schedules-28,0989e69dfa60e68f,d688f5ec029c9f97] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:43:58.513 INFO 21062 --- [ocp-schedules-28,0989e69dfa60e68f,d688f5ec029c9f97] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:43:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:43:58.588025+08:00, lastProcessTime=2024-11-04T15:43:48.589637+08:00 2024-11-04 15:43:58.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:43:58.588025+08:00 2024-11-04 15:43:58.818 INFO 21062 --- [ocp-async-3,bea34ef71191e379,c09ec6caa6fe96c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:43:59.624 INFO 21062 --- [ocp-schedules-7,b0f5a0d823deaa75,2c4a592273340ad4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:43:59.674 WARN 21062 --- [ocp-async-3,e12d2f06adcc89d6,780ace10c9427c8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:43:59.674 INFO 21062 --- [ocp-async-3,aca45d6cae8a323a,43cb596c28c96dac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:43:59.983 INFO 21062 --- [prometheus-minute-schedule-1,be1137a041ca26c6,dba4841502242eee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706240, interval=60 2024-11-04 15:44:00.471 INFO 21062 --- [ocp-schedules-1,038006412a4c210e,7041156dc4523513] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:44:00.496 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:00.496 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:01.674 WARN 21062 --- [ocp-async-4,30b69052494b8e78,ed9afaa5d5385eec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:01.674 INFO 21062 --- [ocp-async-8,3ba80f0e15edd6c3,69d350b7dbc56f02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:01.994 INFO 21062 --- [ocp-schedules-18,ec3fee031dcedbdd,ea0a17b0bee230fd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:44:02.983 INFO 21062 --- [prometheus-minute-schedule-1,770b94b5b5377a09,58b3fad76f8efbc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:03.496 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:03.497 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:03.674 WARN 21062 --- [ocp-async-7,45fbad1b870d3543,b3a1bc7e5a51c554] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:03.675 INFO 21062 --- [ocp-async-10,5149b36c569b61cd,018971688e69b6a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:05.675 WARN 21062 --- [ocp-async-5,359a755f2427f2ae,be5d86d2f1fa5420] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:05.675 INFO 21062 --- [ocp-async-5,aa9bd1c4fc7c7898,8438726ce56b0154] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d647d2e035939780,cd123d1ea5096649] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:06.497 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:06.498 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:07.675 WARN 21062 --- [ocp-async-4,7cee84aea2e3b554,2fcff17078d9407d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:07.675 INFO 21062 --- [ocp-async-9,fe7251d17a315a68,f58691579223bc14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:08.505 INFO 21062 --- [ocp-schedules-20,088aa2bbae033848,9202f63fb42facb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:44:08.505 INFO 21062 --- [ocp-schedules-27,004d3ef8e05dd23d,dd634332ccaa95d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:44:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:44:08.588460+08:00, lastProcessTime=2024-11-04T15:43:58.588025+08:00 2024-11-04 15:44:08.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:44:08.588460+08:00 2024-11-04 15:44:08.818 INFO 21062 --- [ocp-async-1,2928c7873bf62660,e5beeee7b921b3cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:44:08.983 INFO 21062 --- [prometheus-minute-schedule-1,960e60f9cf118856,e3206bbdbcd65e88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:09.498 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:09.499 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:09.675 WARN 21062 --- [ocp-async-1,cf8bf306d105de39,ae053a17f79282ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:09.675 INFO 21062 --- [ocp-async-10,a9e02f34b764ef31,7e94b370c4795c43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:10.003 INFO 21062 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:44:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:44:10.305 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:44:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:44:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:44:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:44:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:44:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:44:10.498 INFO 21062 --- [ocp-schedules-30,550d951c9f6852b0,369aae771ba40858] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:44:11.593 INFO 21062 --- [ocp-schedules-23,66eba2b9915fd39b,c45c6a39030bca49] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:44:11.593 INFO 21062 --- [ocp-schedules-23,9d2426473aef335e,13045b86dac21ddc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:44:11.597 INFO 21062 --- [ocp-schedules-23,9d2426473aef335e,13045b86dac21ddc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:11.625 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:44:11.625 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:44:11.675 WARN 21062 --- [ocp-async-6,e39aa68d433d7878,89a15ad82228ac5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:11.676 INFO 21062 --- [ocp-async-5,228af92b092ad8ff,c619c05a010c5b87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:11.929 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=6, affectRows=6 2024-11-04 15:44:11.929 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=6, validCount=6 2024-11-04 15:44:11.930 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=22, affectRows=22 2024-11-04 15:44:11.930 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=1, affectRows=1 2024-11-04 15:44:11.930 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=1, validCount=1 2024-11-04 15:44:11.930 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=22, validCount=22 2024-11-04 15:44:11.930 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=25, affectRows=25 2024-11-04 15:44:11.930 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=24, validCount=24 2024-11-04 15:44:11.930 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=20, affectRows=20 2024-11-04 15:44:11.931 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=35, affectRows=35 2024-11-04 15:44:11.931 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=20, validCount=20 2024-11-04 15:44:11.931 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=35, validCount=35 2024-11-04 15:44:11.931 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=23, affectRows=23 2024-11-04 15:44:11.931 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=23, validCount=23 2024-11-04 15:44:11.931 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=58, affectRows=58 2024-11-04 15:44:11.931 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=68, affectRows=68 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=46, affectRows=46 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=63, affectRows=63 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=57, validCount=57 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=67, validCount=67 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=46, validCount=46 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706180, nodeCount=55, affectRows=55 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=63, validCount=63 2024-11-04 15:44:11.932 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706180, nodeCount=55, validCount=55 2024-11-04 15:44:11.983 INFO 21062 --- [prometheus-minute-schedule-1,c96ec44702fe6177,74799e922c9e66b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:11.997 INFO 21062 --- [ocp-schedules-30,7318b322f95bcf1b,99f5d9120d261115] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:44:12.499 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:12.500 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:13.483 INFO 21062 --- [ocp-schedules-22,c20bb370a5cc4412,3a5939070675ec3c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:44:13.490 INFO 21062 --- [ocp-schedules-22,c20bb370a5cc4412,3a5939070675ec3c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:44:13.490 INFO 21062 --- [ocp-schedules-22,c20bb370a5cc4412,3a5939070675ec3c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:44:13.676 WARN 21062 --- [ocp-async-3,b8f9fe9bedbb1596,56b65f9b9088f58b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:13.676 INFO 21062 --- [ocp-async-3,3cde852bc24b8a8e,6d76f9d155ec6e6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:13.984 INFO 21062 --- [prometheus-cache-monitor1,505d624e418f0b66,c149e3a545daa507] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:44:13.984 INFO 21062 --- [prometheus-cache-monitor1,505d624e418f0b66,c149e3a545daa507] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:44:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d6b02da4f9ebcffb,62c19516378c5408] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:15.500 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:15.501 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:15.676 WARN 21062 --- [ocp-async-1,eee78a98cba9adf5,59b8ccf150a0302b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:15.676 INFO 21062 --- [ocp-async-7,083e26a556943777,1cf8518dad4466b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:17.676 WARN 21062 --- [ocp-async-6,d534d034631f997b,f4224d33a2ded6a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:17.676 INFO 21062 --- [ocp-async-2,10c0142b7e4afcd0,a98e6c6463d58022] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:17.983 INFO 21062 --- [prometheus-minute-schedule-1,cc9d77ef6a875b11,df16c43ba7be9fca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:18.501 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:18.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:44:18.589511+08:00, lastProcessTime=2024-11-04T15:44:08.588460+08:00 2024-11-04 15:44:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:44:18.589511+08:00 2024-11-04 15:44:18.818 INFO 21062 --- [ocp-async-9,2b09cb4a10880ba2,c17449be6cbdceef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:44:19.676 WARN 21062 --- [ocp-async-4,c3538fcb90e789bc,b7295efb72b2ad29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:19.676 INFO 21062 --- [ocp-async-3,bada5c3b88e38057,1d45772a9a8b21e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:20.604 INFO 21062 --- [ocp-schedules-18,0462dfc1680bf2de,c173f0f6d87149f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:44:20.983 INFO 21062 --- [prometheus-minute-schedule-1,4203568806924c5b,599b475a2eddd064] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:21.502 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:21.503 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:21.676 WARN 21062 --- [ocp-async-8,bd934d1102484969,f1889308225de68d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:21.677 INFO 21062 --- [ocp-async-7,88d468abdf2668f8,56cd39a76954c6a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:21.999 INFO 21062 --- [ocp-schedules-18,d4cdda8da5a53957,6564ad450a3f7122] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:44:23.677 WARN 21062 --- [ocp-async-10,1c9440fdd568ebfb,30134a9cec5537f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:23.677 INFO 21062 --- [ocp-async-2,4e70db7d033d799d,2bf0f4d28758ce70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:23.983 INFO 21062 --- [prometheus-minute-schedule-1,b473042265f48414,85d5953db6bd3f1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:24.503 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:24.504 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:25.677 WARN 21062 --- [ocp-async-4,c9cbecfdbdef8946,ec180c6e7e13af11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:25.677 INFO 21062 --- [ocp-async-4,666e02aff88299a0,6c1f00d9f0bef046] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:26.983 INFO 21062 --- [prometheus-minute-schedule-1,c4876acd0bd74991,6b1cf28c33f90bc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ef0c9ce6ff55bb3d,6bfb998800e5961e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:44:27.504 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:27.505 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:27.677 WARN 21062 --- [ocp-async-8,bd66c571017d3414,39f14193bcf090a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:27.677 INFO 21062 --- [ocp-async-1,919dd3cb93359e9d,a990ec85bf4ed607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:28.498 INFO 21062 --- [ocp-schedules-2,7d8efc40ea725d39,c1eaff2a7ef4c327] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:44:28.499 INFO 21062 --- [ocp-schedules-15,926fd57e1ce04fb9,6731420d1ef3c657] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:44:28.500 INFO 21062 --- [ocp-schedules-11,3578700ddeb3b996,2e88be6bde7d0aaf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:44:28.500 INFO 21062 --- [ocp-schedules-4,d09775100812ca56,b993fbefb2b09490] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:44:28.502 INFO 21062 --- [ocp-schedules-2,7d8efc40ea725d39,c1eaff2a7ef4c327] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:44:28.502 INFO 21062 --- [ocp-schedules-29,b7fbded549a67889,d029305c2c63c5e4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:44:28.503 INFO 21062 --- [ocp-schedules-11,3578700ddeb3b996,2e88be6bde7d0aaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:44:28.504 INFO 21062 --- [ocp-schedules-4,d09775100812ca56,b993fbefb2b09490] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:28.505 INFO 21062 --- [ocp-schedules-28,95773820578c6d86,485f23e75d1b6389] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:44:28.505 INFO 21062 --- [ocp-schedules-15,926fd57e1ce04fb9,6731420d1ef3c657] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:44:28.510 INFO 21062 --- [ocp-schedules-14,63293d072af41d97,0fb4983e9240c46b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:44:28.511 INFO 21062 --- [ocp-schedules-24,e4f9ab0c4469d816,3766490b48f710fd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:44:28.511 INFO 21062 --- [ocp-schedules-26,2b851e18c52962d3,be050f7e68565b36] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:44:28.511 INFO 21062 --- [ocp-schedules-14,63293d072af41d97,0fb4983e9240c46b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:44:28.511 INFO 21062 --- [ocp-schedules-24,e4f9ab0c4469d816,3766490b48f710fd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:44:28.511 INFO 21062 --- [ocp-schedules-26,2b851e18c52962d3,be050f7e68565b36] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:44:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:44:28.588889+08:00, lastProcessTime=2024-11-04T15:44:18.589511+08:00 2024-11-04 15:44:28.591 INFO 21062 --- [ocp-schedules-27,28e6bcb9733399b2,1896fbab30457e40] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:44:28.591 INFO 21062 --- [ocp-schedules-27,28e6bcb9733399b2,1896fbab30457e40] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:44:28.630 INFO 21062 --- [ocp-schedules-16,58af63a8e76a49f9,4629c5acbe27f664] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:44:28.630+08:00, tenantSize=0 2024-11-04 15:44:28.745 INFO 21062 --- [ocp-schedules-18,3ecca7df9abe628d,dc48901109fc3b03] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 15:44:28.745 INFO 21062 --- [ocp-schedules-18,3ecca7df9abe628d,dc48901109fc3b03] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 15:44:28.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:44:28.588889+08:00 2024-11-04 15:44:28.819 INFO 21062 --- [ocp-async-2,0710ca969b2c4a5b,7c6ac9c832f8ba8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:44:29.039 INFO 21062 --- [ocp-schedules-29,ccbfa3e7027dcf84,821ec9b9e2e3877c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:44:29.475 INFO 21062 --- [ocp-schedules-19,7e29613d107873f2,b352f921f976aafe] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:44:29.475 INFO 21062 --- [ocp-schedules-19,7e29613d107873f2,b352f921f976aafe] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:44:29.627 INFO 21062 --- [ocp-schedules-20,8ec581e1979a63a9,3bfb22b8a232066a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:44:29.640 INFO 21062 --- [ocp-schedules-9,3efc5b578f622cae,78db297c89ad1604] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:44:29.645 INFO 21062 --- [ocp-schedules-9,3efc5b578f622cae,78db297c89ad1604] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:44:29.645 INFO 21062 --- [ocp-schedules-9,3efc5b578f622cae,78db297c89ad1604] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:44:29.677 WARN 21062 --- [ocp-async-9,0a5fed2f87d0c716,3fbf333417c73d8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:29.677 INFO 21062 --- [ocp-async-9,644c281db972c783,a4bf8c635c47b7a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:29.983 INFO 21062 --- [prometheus-minute-schedule-1,bf88a520064ee85e,32ac68426a70a5b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:30.161 INFO 21062 --- [ocp-schedules-28,0b0ecc2d57c692fb,7883ab4641cb41b9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:44:30.161 INFO 21062 --- [ocp-schedules-28,0b0ecc2d57c692fb,7883ab4641cb41b9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:44:30.169 INFO 21062 --- [ocp-schedules-28,0b0ecc2d57c692fb,7883ab4641cb41b9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:44:30.169 INFO 21062 --- [ocp-schedules-28,0b0ecc2d57c692fb,7883ab4641cb41b9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:44:30.173 INFO 21062 --- [ocp-schedules-28,0b0ecc2d57c692fb,7883ab4641cb41b9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:44:30.173 INFO 21062 --- [ocp-schedules-28,0b0ecc2d57c692fb,7883ab4641cb41b9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:44:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:44:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:44:30.505 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:30.506 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:30.630 INFO 21062 --- [ocp-schedules-27,c35e7d33c41a74b8,e09b8b4df6d81096] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:44:31.677 WARN 21062 --- [ocp-async-8,99f4e13000142a73,2d56818352ccf089] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:31.677 INFO 21062 --- [ocp-async-1,ccb9793f567e3ae2,ee0aed10e209575c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:32.003 INFO 21062 --- [ocp-schedules-19,0243a86853228239,7e5bca0957b17671] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:44:32.983 INFO 21062 --- [prometheus-minute-schedule-1,1996a579db92aa98,3f2f1ca1fcd2a5d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:33.494 INFO 21062 --- [ocp-schedules-3,13cc3f2278499463,fb829ec519734dc7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:44:33.500 INFO 21062 --- [ocp-schedules-3,13cc3f2278499463,fb829ec519734dc7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:44:33.500 INFO 21062 --- [ocp-schedules-3,13cc3f2278499463,fb829ec519734dc7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:44:33.506 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:33.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:33.677 WARN 21062 --- [ocp-async-7,7a77b2cf2b835395,8ce2fc0dbd1fab34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:33.678 INFO 21062 --- [ocp-async-10,a943b2208d84f1ef,4e7ab5a61022ff3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:33.819 INFO 21062 --- [ocp-schedules-5,fefe7c6417e3d757,96f20acba94cdbf8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:44:33.819 INFO 21062 --- [ocp-schedules-5,fefe7c6417e3d757,96f20acba94cdbf8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:44:33.847 INFO 21062 --- [ocp-schedules-5,fefe7c6417e3d757,96f20acba94cdbf8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:44:33.847 INFO 21062 --- [ocp-schedules-5,fefe7c6417e3d757,96f20acba94cdbf8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:44:33.849 INFO 21062 --- [ocp-schedules-17,2a63e1157c2925c9,d401b1260b810fce] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:44:34.022 INFO 21062 --- [ocp-schedules-22,155b2f86a06776ec,22b4bca2f2b14fcd] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:44:34.022+08:00, clusters=[1] 2024-11-04 15:44:34.024 INFO 21062 --- [ocp-schedules-22,155b2f86a06776ec,22b4bca2f2b14fcd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:44:34.377 INFO 21062 --- [ocp-schedules-1,9ce8a36fc7c8508e,213c83bb384d2a38] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:44:34.377+08:00, clusters=[1] 2024-11-04 15:44:34.383 INFO 21062 --- [ocp-schedules-1,9ce8a36fc7c8508e,213c83bb384d2a38] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:44:34.485 INFO 21062 --- [ocp-schedules-28,f3eb87e03118bd55,5902af42447fb23e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:44:34.489 INFO 21062 --- [ocp-schedules-28,f3eb87e03118bd55,5902af42447fb23e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:34.548 ERROR 21062 --- [ocp-schedules-28,f3eb87e03118bd55,5902af42447fb23e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:44:34.548 INFO 21062 --- [ocp-schedules-28,f3eb87e03118bd55,5902af42447fb23e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:44:35.678 WARN 21062 --- [ocp-async-5,5c8ded569fce2aec,e06974f12d881986] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:35.678 INFO 21062 --- [ocp-async-2,cf16811b3a0d776d,c92bb7cb26ba82f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:35.983 INFO 21062 --- [prometheus-minute-schedule-1,dd8fdaaa5e685e6c,a58bc221e7c92017] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:36.471 INFO 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:44:36.471 INFO 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:44:36.471 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:44:36.471 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:44:36.481 INFO 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:44:36.482 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:44:36.482 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:44:36.483 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:44:36.483 WARN 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:44:36.483 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:44:36.484 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:44:36.484 INFO 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:44:36.484 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:44:36.485 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:44:36.485 WARN 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:44:36.485 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:44:36.492 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:44:36.493 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:44:36.493 INFO 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:44:36.503 ERROR 21062 --- [ocp-schedules-28,f9c49ae84e3f1d26,6c7a408f2511608b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:44:36.507 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:36.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:37.678 WARN 21062 --- [ocp-async-8,f14a241f2f038bae,70ade8794bfe816d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:37.678 INFO 21062 --- [ocp-async-4,f7eb8c1120de2dd2,747579377d202925] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:38.504 INFO 21062 --- [ocp-schedules-11,3578700ddeb3b996,2e88be6bde7d0aaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:44:38.505 INFO 21062 --- [ocp-schedules-4,d09775100812ca56,b993fbefb2b09490] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:44:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:44:38.589087+08:00, lastProcessTime=2024-11-04T15:44:28.588889+08:00 2024-11-04 15:44:38.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:44:38.589087+08:00 2024-11-04 15:44:38.819 INFO 21062 --- [ocp-async-3,c7c2548212b04594,0600b42a307d12ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:44:38.983 INFO 21062 --- [prometheus-minute-schedule-1,940698a542e95937,d05df291c6582c31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:39.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:39.508 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:39.678 WARN 21062 --- [ocp-async-10,6ebf98ee83a2161f,2a26567a288baef1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:39.678 INFO 21062 --- [ocp-async-5,779dee584bf94898,6eff0ae883a255d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:40.003 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:44:40.003 INFO 21062 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:44:40.670 INFO 21062 --- [ocp-schedules-28,359f64a0087fe7db,7637d9dc413f2ca3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:44:41.626 INFO 21062 --- [ocp-schedules-23,9d2426473aef335e,13045b86dac21ddc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:44:41.626 INFO 21062 --- [ocp-schedules-23,ca46988d72899134,f5e2955d55c2e71d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:44:41.630 INFO 21062 --- [ocp-schedules-23,ca46988d72899134,f5e2955d55c2e71d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:41.662 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:44:41.662 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:44:41.678 WARN 21062 --- [ocp-async-2,6e5e952ad4a67b34,5b7e21d771a566c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:41.678 INFO 21062 --- [ocp-async-8,73c90dcc85553033,f07113adb13bf4a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:41.827 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:44:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:44:41.830 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:44:41.855 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:44:41.855 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=25ms, period=120000ms 2024-11-04 15:44:41.983 INFO 21062 --- [prometheus-minute-schedule-1,c4c59338171679b8,40aaa93c2f44fb4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:42.006 INFO 21062 --- [ocp-schedules-14,3d113f4bebf16aa2,67bfdeac745bda95] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:44:42.509 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:42.510 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:43.679 WARN 21062 --- [ocp-async-4,e2d8f7c6d3222204,199969a0f822235d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:43.679 INFO 21062 --- [ocp-async-1,59d0797ce8893a85,85435d5d860fd50a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:44.983 INFO 21062 --- [prometheus-minute-schedule-1,dc8ff2105c7a60cd,4007eca7b5c2742b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:45.510 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:45.511 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:45.663 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:44:45.663 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:44:45.664 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.665 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:44:45.666 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.666 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.667 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.667 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:44:45.679 WARN 21062 --- [ocp-async-7,eb29837259e959aa,9c42e6c9828ba146] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:45.679 INFO 21062 --- [ocp-async-10,9ba0394e598eb375,c5d8201fd0f1f575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:45.685 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.685 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.686 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.686 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:44:45.700 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.700 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.701 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.701 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:44:45.715 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.715 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.716 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.716 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:44:45.730 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.730 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.730 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.731 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:44:45.736 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.737 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.737 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.738 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:44:45.742 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.742 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.742 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.743 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:44:45.747 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.747 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.748 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.748 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:44:45.754 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.755 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.755 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.756 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:44:45.761 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.761 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.761 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.762 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:44:45.767 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.767 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.768 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.768 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:44:45.776 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.777 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.777 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.778 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:44:45.791 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.791 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.792 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.792 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:44:45.797 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.797 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.797 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.798 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:44:45.803 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.804 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.804 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.805 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:44:45.810 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.810 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.811 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.811 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:44:45.815 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.816 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.816 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.816 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:44:45.820 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.821 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.821 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.822 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:44:45.826 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.827 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.827 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.828 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:44:45.833 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.833 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.834 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.835 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:44:45.839 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.840 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.840 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.841 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:44:45.845 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.846 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.846 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.847 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:44:45.850 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.851 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.851 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.852 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:44:45.856 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.856 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.857 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.857 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:44:45.861 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.862 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.862 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.862 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:44:45.866 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.867 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.867 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.868 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:44:45.872 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.873 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.873 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.874 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:44:45.878 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.878 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.879 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.879 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:44:45.883 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.884 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.884 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.885 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:44:45.889 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.889 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.890 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.890 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:44:45.894 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.895 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.895 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.895 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:44:45.899 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.900 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.900 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.901 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:44:45.905 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.905 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.906 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.906 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:44:45.911 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.911 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.911 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.912 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:44:45.919 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.920 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.920 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.921 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:44:45.927 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.927 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.928 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.928 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:44:45.934 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.934 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.935 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.935 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:44:45.940 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.941 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.941 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.942 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:44:45.945 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.945 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.946 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.946 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:44:45.949 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.950 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.950 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.951 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:44:45.955 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.955 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.956 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.956 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:44:45.959 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.960 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.960 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.961 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:44:45.970 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.970 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.971 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.971 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:44:45.992 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.992 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:45.994 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:45.994 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:44:46.000 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.000 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:46.001 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.001 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:44:46.008 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.008 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:46.009 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.009 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:44:46.015 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.016 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:46.016 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.017 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:44:46.021 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.021 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:46.022 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.022 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:44:46.026 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.026 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:46.027 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.027 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:44:46.032 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:44:46.032 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:44:46.033 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.033 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:44:46.034 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.035 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:46.035 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.035 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:44:46.041 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.041 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:44:46.041 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:44:46.042 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:44:46.045 INFO 21062 --- [ocp-schedules-2,ec4007fc8270f5ca,43bc84af479e82b3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:44:46.272 INFO 21062 --- [ocp-schedules-7,89d138f0ec0f628e,24e7c0b0aa20c4ba] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:44:46.276 INFO 21062 --- [ocp-schedules-7,89d138f0ec0f628e,24e7c0b0aa20c4ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:46.364 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:44:46.366 ERROR 21062 --- [ocp-schedules-7,89d138f0ec0f628e,24e7c0b0aa20c4ba] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:44:46.367 INFO 21062 --- [ocp-schedules-7,89d138f0ec0f628e,24e7c0b0aa20c4ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:44:47.679 WARN 21062 --- [ocp-async-5,00695e95f29f6b2e,8d0969d45ba39fdf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:47.679 INFO 21062 --- [ocp-async-2,95e8a31759d4d57b,cb56d1bc6a1fad0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b2109e81e69ffbb5,eca8a8f518fb8c6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:48.511 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:48.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:44:48.589078+08:00, lastProcessTime=2024-11-04T15:44:38.589087+08:00 2024-11-04 15:44:48.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:44:48.589078+08:00 2024-11-04 15:44:48.819 INFO 21062 --- [ocp-async-4,615a59c9d600fb8e,f94e8c28cb48ef2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:44:49.399 INFO 21062 --- [ocp-schedules-15,a744c04606107020,cfdd690737c41449] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:44:49.404 INFO 21062 --- [ocp-schedules-15,a744c04606107020,cfdd690737c41449] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:49.404 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:44:49.679 WARN 21062 --- [ocp-async-4,5bd4e3395727507e,c664d5625e0f9b18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:49.679 INFO 21062 --- [ocp-async-8,a9412590a45af1eb,c2d05b164df0385b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:50.697 INFO 21062 --- [ocp-schedules-19,71c480311b7498c7,df364f20911e975b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:44:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ecf613937b7d0983,870c9adf53dd7db1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:51.512 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:51.513 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:51.679 WARN 21062 --- [ocp-async-1,559a0f9003c96a2a,27464658dfd9363c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:51.679 INFO 21062 --- [ocp-async-3,49d186930002a786,528d21b9d9750e07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:52.009 INFO 21062 --- [ocp-schedules-25,59d95232b4d98218,bcef6acf891f1373] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:44:53.503 INFO 21062 --- [ocp-schedules-24,f0619a1f39406478,04f0a4b76391cb4f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:44:53.510 INFO 21062 --- [ocp-schedules-24,f0619a1f39406478,04f0a4b76391cb4f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:44:53.510 INFO 21062 --- [ocp-schedules-24,f0619a1f39406478,04f0a4b76391cb4f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:44:53.679 WARN 21062 --- [ocp-async-10,d1b004f0a6325fc7,6ce2431c1c95ddd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:53.680 INFO 21062 --- [ocp-async-10,c25bc606657cd126,24dc52019de580f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1cead7eb01ea217d,5721d5f17de0fd83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:54.513 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:54.514 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:55.680 WARN 21062 --- [ocp-async-4,e07b160064741f27,8df1d2297815c327] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:55.680 INFO 21062 --- [ocp-async-9,ea1f2f97ee3f00ab,0a1f8a147f4641d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:56.983 INFO 21062 --- [prometheus-minute-schedule-1,764feba78c5579b9,d6d80c6eb0c1731c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:44:57.514 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:44:57.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:44:57.680 WARN 21062 --- [ocp-async-1,ca937fce6088c0f2,e6786ba0030d22aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:57.680 INFO 21062 --- [ocp-async-8,05670c8d9e9b76b6,5f6a4169a210d57d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:58.500 INFO 21062 --- [ocp-schedules-25,170d2fc112651ba9,cb283e75d7836d2a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:44:58.500 INFO 21062 --- [ocp-schedules-22,8be639120aacbaa0,c2568b960e1ca6ad] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:44:58.502 INFO 21062 --- [ocp-schedules-17,5af92255474690d3,dac9673d68d65972] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:44:58.503 INFO 21062 --- [ocp-schedules-25,170d2fc112651ba9,cb283e75d7836d2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:58.504 INFO 21062 --- [ocp-schedules-22,8be639120aacbaa0,c2568b960e1ca6ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:44:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:44:58.505 INFO 21062 --- [ocp-schedules-1,12b3d3770a363170,800b6e214ea3e3bd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:44:58.510 INFO 21062 --- [ocp-schedules-20,78abe0f0ca50032e,61c51fa942b7bc89] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:44:58.510 INFO 21062 --- [ocp-schedules-16,8cce882aa271afe2,c07b679656f7e316] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:44:58.510 INFO 21062 --- [ocp-schedules-20,78abe0f0ca50032e,61c51fa942b7bc89] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:44:58.510 INFO 21062 --- [ocp-schedules-16,8cce882aa271afe2,c07b679656f7e316] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:44:58.511 INFO 21062 --- [ocp-schedules-28,382df146fa7c92f4,e3885e552e33ee11] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:44:58.511 INFO 21062 --- [ocp-schedules-28,382df146fa7c92f4,e3885e552e33ee11] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:44:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:44:58.590303+08:00, lastProcessTime=2024-11-04T15:44:48.589078+08:00 2024-11-04 15:44:58.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:44:58.590303+08:00 2024-11-04 15:44:58.820 INFO 21062 --- [ocp-async-5,0150872816aeed7b,2292fc386c64f1e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:44:59.404 INFO 21062 --- [ocp-schedules-15,a744c04606107020,cfdd690737c41449] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:44:59.630 INFO 21062 --- [ocp-schedules-1,deb313a927560286,7582c4754a4e6d6e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:44:59.680 WARN 21062 --- [ocp-async-5,25bf7503b91c75ac,b639644573581c1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:44:59.680 INFO 21062 --- [ocp-async-10,c434d91266e9a487,a3e1540f230d7ae8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:44:59.983 INFO 21062 --- [prometheus-minute-schedule-1,08f0cf2a3cc0a29e,f0ed7dd60b757182] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706300, interval=60 2024-11-04 15:45:00.516 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:00.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:00.700 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d24d133cb38abe72] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:40:01.0, nextRun=2024-11-04 15:45:00.0, currentDate=Mon Nov 04 15:45:00 CST 2024 2024-11-04 15:45:00.701 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d24d133cb38abe72] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:40:01.0, nextRunTime=2024-11-04 15:45:00.0} 2024-11-04 15:45:00.728 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d24d133cb38abe72] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:45:00.732 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d24d133cb38abe72] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:45:00.732 INFO 21062 --- [task-coordinator1,dd193f8f67a5a423,d24d133cb38abe72] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:45:00.737 INFO 21062 --- [ocp-schedules-18,ed93df56cb57f0c0,20592c10ba7dc89d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:45:01.680 WARN 21062 --- [ocp-async-2,3d0d2f888e9da8b2,e0036d16e940b9d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:01.680 INFO 21062 --- [ocp-async-2,4403efab77be5857,a79a9522f2f2773c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:02.013 INFO 21062 --- [ocp-schedules-30,f9e1d9497a32a8d7,2073b9b25f82d522] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:45:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f7d7b7ea4f76a580,2328f49b7aa2eff4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:03.517 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:03.518 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:03.680 WARN 21062 --- [ocp-async-8,d5f234adb96fd229,b1cc248a7def5a21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:03.680 INFO 21062 --- [ocp-async-7,4b93b5b2ad556430,570e24aecd8a8a5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:05.681 WARN 21062 --- [ocp-async-6,7e0f54ff76b2ca07,54e5eb019f9de336] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:05.681 INFO 21062 --- [ocp-async-5,dc604b1fc0976b81,278ffd6fdf650800] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:05.983 INFO 21062 --- [prometheus-minute-schedule-1,19d61f9410e56468,20648e78ef02ac4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:06.518 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:06.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:07.681 INFO 21062 --- [ocp-async-9,9092312663a1175e,37a9f2b5a77d7311] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:07.681 WARN 21062 --- [ocp-async-4,d1bcb1d22897defd,c20cc2c3151a5e7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:08.504 INFO 21062 --- [ocp-schedules-22,8be639120aacbaa0,c2568b960e1ca6ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:45:08.505 INFO 21062 --- [ocp-schedules-25,170d2fc112651ba9,cb283e75d7836d2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:45:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:45:08.588465+08:00, lastProcessTime=2024-11-04T15:44:58.590303+08:00 2024-11-04 15:45:08.754 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:45:08.588465+08:00 2024-11-04 15:45:08.820 INFO 21062 --- [ocp-async-8,12e37647169d74ee,94fd022fd27dfb53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:45:08.983 INFO 21062 --- [prometheus-minute-schedule-1,ca2dcb3da4dd66b9,ef4a528747ecaf69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:09.519 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:09.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:09.681 INFO 21062 --- [ocp-async-8,64863e237061bb71,305184694ff63e6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:09.681 WARN 21062 --- [ocp-async-7,29c04d0b5c38a4ca,234692e61f22289e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:45:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:45:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:45:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:45:10.301 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:45:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:45:10.321 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:45:10.321 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:45:10.763 INFO 21062 --- [ocp-schedules-3,9a459d15ec065e3d,e012319d09e811d4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:45:11.662 INFO 21062 --- [ocp-schedules-23,ca46988d72899134,f5e2955d55c2e71d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:45:11.663 INFO 21062 --- [ocp-schedules-23,bc92eef96091bce9,96a70c4c1d8d4650] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:45:11.667 INFO 21062 --- [ocp-schedules-23,bc92eef96091bce9,96a70c4c1d8d4650] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:11.681 INFO 21062 --- [ocp-async-3,4f5b9ed9e0033f22,451a112824b36766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:11.681 WARN 21062 --- [ocp-async-5,ea327df84d833ea3,8e5547eabdb66f29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:11.696 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:45:11.696 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:45:11.936 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=1, affectRows=1 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=16, affectRows=16 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-44,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=1, validCount=1 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=23, affectRows=23 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=23, affectRows=23 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=16, validCount=16 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=12, affectRows=12 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=6, affectRows=6 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=3, affectRows=3 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=6, validCount=6 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=23, validCount=23 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-47,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=12, validCount=12 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=3, validCount=3 2024-11-04 15:45:11.937 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=22, validCount=22 2024-11-04 15:45:11.938 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=8, affectRows=8 2024-11-04 15:45:11.938 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=26, affectRows=26 2024-11-04 15:45:11.938 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=8, validCount=8 2024-11-04 15:45:11.938 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=19, affectRows=19 2024-11-04 15:45:11.938 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=19, validCount=19 2024-11-04 15:45:11.938 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=26, validCount=26 2024-11-04 15:45:11.939 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=38, affectRows=38 2024-11-04 15:45:11.939 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=40, affectRows=40 2024-11-04 15:45:11.939 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=14, affectRows=14 2024-11-04 15:45:11.939 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=38, validCount=38 2024-11-04 15:45:11.939 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=40, validCount=40 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=30, affectRows=30 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=41, affectRows=41 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=30, validCount=30 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=36, affectRows=36 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=41, validCount=41 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=41, affectRows=41 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=36, validCount=36 2024-11-04 15:45:11.940 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=39, validCount=39 2024-11-04 15:45:11.941 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706240, nodeCount=45, affectRows=45 2024-11-04 15:45:11.941 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=45, validCount=45 2024-11-04 15:45:11.941 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706240, nodeCount=14, validCount=14 2024-11-04 15:45:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b4a86bc2ef4f8d17,59bffcf4a7e39bca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:12.016 INFO 21062 --- [ocp-schedules-26,33d75f873c59446c,d994c5e1b487c1da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:45:12.520 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:12.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:13.513 INFO 21062 --- [ocp-schedules-7,9554ed737bb0b50d,b0f1c26c0e5cc0b0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:45:13.519 INFO 21062 --- [ocp-schedules-7,9554ed737bb0b50d,b0f1c26c0e5cc0b0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:45:13.519 INFO 21062 --- [ocp-schedules-7,9554ed737bb0b50d,b0f1c26c0e5cc0b0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:45:13.682 WARN 21062 --- [ocp-async-9,002d22b3dfe14666,5550e5192029eb93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:13.682 INFO 21062 --- [ocp-async-10,041fef2d86bbdbbf,089e06885d18bf3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:13.984 INFO 21062 --- [prometheus-cache-monitor1,8b59b24f48102e45,5873a5f70ddd9206] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:45:13.984 INFO 21062 --- [prometheus-cache-monitor1,8b59b24f48102e45,5873a5f70ddd9206] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:45:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a450fac1433a67ed,51dd62ace3450113] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:15.522 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:15.523 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:15.682 WARN 21062 --- [ocp-async-7,5964d173f6476404,caa8b0779e597d5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:15.682 INFO 21062 --- [ocp-async-1,b24b9a4596139cf6,f155e7d43b063254] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:15.798 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:41:15.78, end=2024-11-04T15:43:15.795} 2024-11-04 15:45:15.806 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:45:15.806 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:45:17.682 WARN 21062 --- [ocp-async-5,44e7be8c781eb320,0b8818b91f157870] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:17.682 INFO 21062 --- [ocp-async-5,a47e4f65d6977841,98d9e6ad016eba93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:17.983 INFO 21062 --- [prometheus-minute-schedule-1,fc9c979cb8fe36b2,6c91b4cacb8e0706] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:18.523 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:18.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:45:18.589948+08:00, lastProcessTime=2024-11-04T15:45:08.588465+08:00 2024-11-04 15:45:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:45:18.589948+08:00 2024-11-04 15:45:18.820 INFO 21062 --- [ocp-async-9,b183309596b9c6a0,d783870d5796c078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:45:19.682 WARN 21062 --- [ocp-async-4,bd1e59ee1a2bf20d,3590f2777cdd310e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:19.682 INFO 21062 --- [ocp-async-10,d9dca21f81fa6141,406394e1293f666b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:20.807 INFO 21062 --- [ocp-schedules-24,04c958131e3dd39a,6e1ce7dfc31497a0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:45:20.983 INFO 21062 --- [prometheus-minute-schedule-1,b2439b3c46f0e124,6b140c140e4cad2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:21.524 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:21.525 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:21.682 WARN 21062 --- [ocp-async-1,516d38a18bf08987,f3d6725e857515d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:21.682 INFO 21062 --- [ocp-async-2,c7303055c79fb3dc,8849c771eec22cd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:22.019 INFO 21062 --- [ocp-schedules-25,5a8b85dc9dcf5355,cd401b8fc6b9ccb7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:45:23.683 WARN 21062 --- [ocp-async-8,8d5099d9763d0d82,b72214031ac2e8e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:23.683 INFO 21062 --- [ocp-async-5,caa4cb0e402d03d2,b60acb24efe5a1dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c20a22e46a6229f5,ada6e03dc7d5376a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:24.525 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:24.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:25.683 WARN 21062 --- [ocp-async-4,721444e6c2ce3033,3f6ff041d084bc66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:25.683 INFO 21062 --- [ocp-async-9,c643c3bbacc808e5,52a97694be031e61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:26.983 INFO 21062 --- [prometheus-minute-schedule-1,7fd69ed9f8392d75,abb9c620f00f7481] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b1b4ba96cc6b12ac,e8a416190492be78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:45:27.526 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:27.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:27.683 WARN 21062 --- [ocp-async-1,728cccee1ff173b6,d8d25528d1c4aa2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:27.683 INFO 21062 --- [ocp-async-1,84b5b55afc4cebf7,6dd40fd81f208b4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:28.498 INFO 21062 --- [ocp-schedules-29,3db4018c8cfd6c68,d0e2671528f7d0d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:45:28.499 INFO 21062 --- [ocp-schedules-30,542327a977b3c4d2,39fec112906a8f52] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:45:28.500 INFO 21062 --- [ocp-schedules-6,329cb13d199ac2ca,44b1b1d28592c39a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:45:28.500 INFO 21062 --- [ocp-schedules-11,73d6fc55bddf8202,e70aa5c8590d2fbc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:45:28.502 INFO 21062 --- [ocp-schedules-29,3db4018c8cfd6c68,d0e2671528f7d0d7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:45:28.502 INFO 21062 --- [ocp-schedules-2,3db1882679be6c9d,48d0f7c653fbb948] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:45:28.503 INFO 21062 --- [ocp-schedules-6,329cb13d199ac2ca,44b1b1d28592c39a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:45:28.504 INFO 21062 --- [ocp-schedules-11,73d6fc55bddf8202,e70aa5c8590d2fbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:28.505 INFO 21062 --- [ocp-schedules-30,542327a977b3c4d2,39fec112906a8f52] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:45:28.505 INFO 21062 --- [ocp-schedules-21,d2e14c0c72017024,7affbdc283df98b2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:45:28.511 INFO 21062 --- [ocp-schedules-7,281987083b962f42,6a4e58b0ae7ac723] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:45:28.511 INFO 21062 --- [ocp-schedules-7,281987083b962f42,6a4e58b0ae7ac723] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:45:28.511 INFO 21062 --- [ocp-schedules-3,51f98e370171edda,dc6c180a8f95a95e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:45:28.511 INFO 21062 --- [ocp-schedules-3,51f98e370171edda,dc6c180a8f95a95e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:45:28.511 INFO 21062 --- [ocp-schedules-28,ed2546e928cd6ccf,4dc59fe78a48997f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:45:28.511 INFO 21062 --- [ocp-schedules-28,ed2546e928cd6ccf,4dc59fe78a48997f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:45:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:45:28.588893+08:00, lastProcessTime=2024-11-04T15:45:18.589948+08:00 2024-11-04 15:45:28.677 INFO 21062 --- [ocp-schedules-19,033716ee3496589d,785fd00589e920c8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:45:28.678 INFO 21062 --- [ocp-schedules-19,033716ee3496589d,785fd00589e920c8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:45:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:45:28.588893+08:00 2024-11-04 15:45:28.820 INFO 21062 --- [ocp-async-5,4c23793f9de73846,4e4c1c87a08eafc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:45:29.042 INFO 21062 --- [ocp-schedules-27,6a412dfdafbf8358,284474b697fd2279] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:45:29.480 INFO 21062 --- [ocp-schedules-14,031b3a7eac64ba1f,b7b6cb29bc342589] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:45:29.480 INFO 21062 --- [ocp-schedules-14,031b3a7eac64ba1f,b7b6cb29bc342589] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:45:29.632 INFO 21062 --- [ocp-schedules-19,3f8a5c1fe298b3b1,5840a7bf0a7440db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:45:29.645 INFO 21062 --- [ocp-schedules-13,f2b391b6ff2ee6f6,7e256fa8d1046ffc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:45:29.650 INFO 21062 --- [ocp-schedules-13,f2b391b6ff2ee6f6,7e256fa8d1046ffc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:45:29.651 INFO 21062 --- [ocp-schedules-13,f2b391b6ff2ee6f6,7e256fa8d1046ffc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:45:29.683 WARN 21062 --- [ocp-async-4,1d2998ca1a7eb3b9,cade876d409dd4c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:29.683 INFO 21062 --- [ocp-async-3,6d6776b99319c97b,0fdca8553128c2d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:29.983 INFO 21062 --- [prometheus-minute-schedule-1,47fee68b3c56672d,d3544f5946702cce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:30.178 INFO 21062 --- [ocp-schedules-20,e8a3a8bd41e5f6ea,560607f7c7439d10] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:45:30.178 INFO 21062 --- [ocp-schedules-20,e8a3a8bd41e5f6ea,560607f7c7439d10] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:45:30.180 INFO 21062 --- [ocp-schedules-20,e8a3a8bd41e5f6ea,560607f7c7439d10] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:45:30.180 INFO 21062 --- [ocp-schedules-20,e8a3a8bd41e5f6ea,560607f7c7439d10] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:45:30.180 INFO 21062 --- [ocp-schedules-20,e8a3a8bd41e5f6ea,560607f7c7439d10] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:45:30.180 INFO 21062 --- [ocp-schedules-20,e8a3a8bd41e5f6ea,560607f7c7439d10] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:45:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:45:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:45:30.527 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:30.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:30.833 INFO 21062 --- [ocp-schedules-2,2fa71df2eec468d1,6a3489e4c81ceb90] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:45:31.683 WARN 21062 --- [ocp-async-9,52b57409a5d07011,1372100c7b6d7408] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:31.683 INFO 21062 --- [ocp-async-7,1cb39c8c97640fe8,723a6b87fb85165a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:32.021 INFO 21062 --- [ocp-schedules-14,98444b13aeb1d110,7d8ea67ddeb4a0d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:45:32.983 INFO 21062 --- [prometheus-minute-schedule-1,0fc69dcdceba285b,71870cc6d3efe2d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:33.522 INFO 21062 --- [ocp-schedules-15,cb9577d11e9479a7,58d183860cb1a56c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:45:33.528 INFO 21062 --- [ocp-schedules-15,cb9577d11e9479a7,58d183860cb1a56c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:45:33.528 INFO 21062 --- [ocp-schedules-15,cb9577d11e9479a7,58d183860cb1a56c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:45:33.528 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:33.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:33.683 WARN 21062 --- [ocp-async-10,b14f466772600cb2,8bd9932001350b47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:33.683 INFO 21062 --- [ocp-async-2,232dac617a48a5d5,5e0778e9aa75d7ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:33.848 INFO 21062 --- [ocp-schedules-5,96867667686f11fe,d4233847d3634992] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:45:33.848 INFO 21062 --- [ocp-schedules-5,96867667686f11fe,d4233847d3634992] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:45:33.876 INFO 21062 --- [ocp-schedules-5,96867667686f11fe,d4233847d3634992] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:45:33.876 INFO 21062 --- [ocp-schedules-5,96867667686f11fe,d4233847d3634992] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:45:33.876 INFO 21062 --- [ocp-schedules-17,8b66c9fc69b9f4f2,cb27c8c775a26328] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:45:34.051 INFO 21062 --- [ocp-schedules-12,f9fbeed1e0e2c860,90935e758923c0af] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:45:34.051+08:00, clusters=[1] 2024-11-04 15:45:34.052 INFO 21062 --- [ocp-schedules-12,f9fbeed1e0e2c860,90935e758923c0af] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:45:34.408 INFO 21062 --- [ocp-schedules-25,efa43193c9d50bfc,e2a8c43aba77e375] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:45:34.408+08:00, clusters=[1] 2024-11-04 15:45:34.412 INFO 21062 --- [ocp-schedules-25,efa43193c9d50bfc,e2a8c43aba77e375] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:45:35.683 WARN 21062 --- [ocp-async-6,5544f0882abf88f0,6950e5fa0e79ab09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:35.684 INFO 21062 --- [ocp-async-4,d0164765ca6617ac,2464feaad7e85c08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:35.983 INFO 21062 --- [prometheus-minute-schedule-1,29666a022a06ca36,0e558aaa583a6635] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:36.515 INFO 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:45:36.515 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:45:36.515 INFO 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:45:36.515 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:45:36.525 INFO 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:45:36.525 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:45:36.526 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:45:36.527 WARN 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:45:36.527 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:45:36.527 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:45:36.527 INFO 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:45:36.528 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:45:36.528 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:45:36.529 WARN 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:45:36.529 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:45:36.529 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:36.529 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:45:36.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:36.536 INFO 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:45:36.536 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:45:36.536 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:45:36.549 ERROR 21062 --- [ocp-schedules-1,fa05214ee1afbd42,accccb4845b102c2] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:45:37.684 WARN 21062 --- [ocp-async-9,e3cf8575f3f03026,9f525f4c6cdc20bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:37.684 INFO 21062 --- [ocp-async-3,4a43d3ec5e6cfe3e,0f915b4251be8d18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:38.504 INFO 21062 --- [ocp-schedules-11,73d6fc55bddf8202,e70aa5c8590d2fbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:45:38.504 INFO 21062 --- [ocp-schedules-6,329cb13d199ac2ca,44b1b1d28592c39a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:45:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:45:38.588026+08:00, lastProcessTime=2024-11-04T15:45:28.588893+08:00 2024-11-04 15:45:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:45:38.588026+08:00 2024-11-04 15:45:38.820 INFO 21062 --- [ocp-async-2,3542e0e576d75c03,6764b45405a19460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:45:38.983 INFO 21062 --- [prometheus-minute-schedule-1,c3a0f798bf931347,12295da4a511deed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:39.530 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:39.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:39.684 WARN 21062 --- [ocp-async-2,4f638080b6c2f027,753ee5e654dd0d3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:39.684 INFO 21062 --- [ocp-async-6,4b594715bf8d9b22,258d8492b348e16e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:40.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:45:40.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:45:40.914 INFO 21062 --- [ocp-schedules-28,a3ba3a1644fc3ba0,8054a7039758b427] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:45:41.684 WARN 21062 --- [ocp-async-4,bbf4033862534203,eb037ee8301df94e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:41.684 INFO 21062 --- [ocp-async-9,972b38f451e6593a,2732f15e7c86d590] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:41.697 INFO 21062 --- [ocp-schedules-23,bc92eef96091bce9,96a70c4c1d8d4650] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:45:41.697 INFO 21062 --- [ocp-schedules-23,579d3f7f3fe20991,2b96df4888214bc9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:45:41.700 INFO 21062 --- [ocp-schedules-23,579d3f7f3fe20991,2b96df4888214bc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:41.729 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:45:41.729 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:45:41.983 INFO 21062 --- [prometheus-minute-schedule-1,ed6180608d950960,93719d5fc225ea71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:42.024 INFO 21062 --- [ocp-schedules-7,a586ceb124ba429c,385771e723eb0b51] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:45:42.531 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:42.532 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:43.684 WARN 21062 --- [ocp-async-3,32ae0a1198edd90d,ee019b1fd3acecac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:43.684 INFO 21062 --- [ocp-async-7,ffcceb9000eb277c,e9f303c0a106826b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e9e1c8eba2276dcd,7a33d2e1b6d10580] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:45.532 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:45.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:45.684 WARN 21062 --- [ocp-async-2,254eefcff9f0a41d,081bed214a93a4b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:45.684 INFO 21062 --- [ocp-async-2,0e730638d73a5320,2dd34f9c8d460acf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:46.367 INFO 21062 --- [ocp-schedules-18,81491222e9546797,910f553649eb2b2f] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:45:46.371 INFO 21062 --- [ocp-schedules-18,81491222e9546797,910f553649eb2b2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:46.456 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:45:46.464 ERROR 21062 --- [ocp-schedules-18,81491222e9546797,910f553649eb2b2f] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:45:46.464 INFO 21062 --- [ocp-schedules-18,81491222e9546797,910f553649eb2b2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:45:47.685 WARN 21062 --- [ocp-async-4,3ba1d5cfcc27b6bb,925fee0a00266056] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:47.685 INFO 21062 --- [ocp-async-8,abafac0ea6a97583,51785472fca0f151] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:47.983 INFO 21062 --- [prometheus-minute-schedule-1,83a16941049b23d3,016482c4092de8d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:48.533 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:48.534 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:45:48.590410+08:00, lastProcessTime=2024-11-04T15:45:38.588026+08:00 2024-11-04 15:45:48.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:45:48.590410+08:00 2024-11-04 15:45:48.821 INFO 21062 --- [ocp-async-3,a99b7aa11d799c99,929c3fd49c9a2155] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:45:49.685 INFO 21062 --- [ocp-async-3,3b3eea76874f817b,186376f760ac258b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:49.685 WARN 21062 --- [ocp-async-7,9f1c0a287afc5209,e489e07096aef41d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:50.941 INFO 21062 --- [ocp-schedules-8,15b8cbe17741e4ae,6d2afba9f68cb681] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:45:50.983 INFO 21062 --- [prometheus-minute-schedule-1,cb627ea5f207229a,cccf119c4ec92f24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:51.534 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:51.535 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:51.685 INFO 21062 --- [ocp-async-10,b3a7471d8e564036,3a5d6d8cffba20f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:51.685 WARN 21062 --- [ocp-async-2,21070c6a0944df58,dab149ff9c0ce24c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:52.028 INFO 21062 --- [ocp-schedules-24,294d17357cb62201,7c89e748f8ad1537] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:45:53.531 INFO 21062 --- [ocp-schedules-14,90095ae7481dc3fb,beff2bde9515231a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:45:53.537 INFO 21062 --- [ocp-schedules-14,90095ae7481dc3fb,beff2bde9515231a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:45:53.537 INFO 21062 --- [ocp-schedules-14,90095ae7481dc3fb,beff2bde9515231a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:45:53.685 INFO 21062 --- [ocp-async-6,76608c710a38fc7b,7d5e3cf52abed4b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:53.685 WARN 21062 --- [ocp-async-8,57ccb36718fb0ba3,d2159b7e2d6fe576] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:53.983 INFO 21062 --- [prometheus-minute-schedule-1,c60e201371d4828b,1330f3013daede41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:54.535 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:54.536 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:55.685 WARN 21062 --- [ocp-async-5,22b5821c88d129b1,951a479a48217c96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:55.685 INFO 21062 --- [ocp-async-7,28805617bc3446aa,ebf95d86ec1464b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:56.983 INFO 21062 --- [prometheus-minute-schedule-1,abc6ff0364d3db16,1ab413d0370b7e5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:45:57.536 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:45:57.537 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:45:57.685 INFO 21062 --- [ocp-async-2,5f0aa911d931726f,a055bd64f26d68a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:57.685 WARN 21062 --- [ocp-async-1,7702b48df4c30e7a,ec0d0f2b8be2affe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:58.499 INFO 21062 --- [ocp-schedules-16,1108f60c46473d1e,cd2ab8531488a7a9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:45:58.500 INFO 21062 --- [ocp-schedules-27,14f0ef376181b72b,54cdb1f4deab3565] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:45:58.502 INFO 21062 --- [ocp-schedules-9,0e3dd4784ad7c611,4d243b08ad487338] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:45:58.503 INFO 21062 --- [ocp-schedules-16,1108f60c46473d1e,cd2ab8531488a7a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:45:58.504 INFO 21062 --- [ocp-schedules-2,4ac3304bf0dc243c,750aec50d9d7aae2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:45:58.505 INFO 21062 --- [ocp-schedules-27,14f0ef376181b72b,54cdb1f4deab3565] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:58.509 INFO 21062 --- [ocp-schedules-24,b3bcead7b48d8969,d74ad8424ce8e811] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:45:58.509 INFO 21062 --- [ocp-schedules-21,4086f9c81318bda0,b5eb202a62ddab35] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:45:58.509 INFO 21062 --- [ocp-schedules-24,b3bcead7b48d8969,d74ad8424ce8e811] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:45:58.509 INFO 21062 --- [ocp-schedules-21,4086f9c81318bda0,b5eb202a62ddab35] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:45:58.510 INFO 21062 --- [ocp-schedules-13,741a3a12a2915eed,ff281de204dfe020] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:45:58.510 INFO 21062 --- [ocp-schedules-13,741a3a12a2915eed,ff281de204dfe020] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:45:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:45:58.589731+08:00, lastProcessTime=2024-11-04T15:45:48.590410+08:00 2024-11-04 15:45:58.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:45:58.589731+08:00 2024-11-04 15:45:58.821 INFO 21062 --- [ocp-async-8,26101d6c5c1605c2,d81b067160283fdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:45:59.405 INFO 21062 --- [ocp-schedules-1,3b68db7785890c09,a6a53ee81a8028d7] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:45:59.409 INFO 21062 --- [ocp-schedules-1,3b68db7785890c09,a6a53ee81a8028d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:45:59.409 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:45:59.636 INFO 21062 --- [ocp-schedules-13,609bb1f78d7d8920,1bd8af4d2840e2fb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:45:59.686 WARN 21062 --- [ocp-async-6,5f27fc35071ccc3b,59f914355cb2937f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:45:59.685 INFO 21062 --- [ocp-async-4,cd8e507f65174b06,8dc1dd28e4110b03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:45:59.983 INFO 21062 --- [prometheus-minute-schedule-1,042ba573587b8215,6e817b8943b82514] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706360, interval=60 2024-11-04 15:46:00.537 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:00.538 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:00.982 INFO 21062 --- [ocp-schedules-15,33a4ba4cdeee0fb0,96210bc2e2a75a47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:46:01.686 INFO 21062 --- [ocp-async-9,47ea537497038dd7,50abeedd088c5389] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:01.686 WARN 21062 --- [ocp-async-7,9ede5118b69aa012,acd48031067d449d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:02.031 INFO 21062 --- [ocp-schedules-21,ff8595b45640b4fb,f2acc58740d6983d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:46:02.983 INFO 21062 --- [prometheus-minute-schedule-1,fb0f4a2a1e4dd8c2,cdcc184d0d287fe7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:03.538 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:03.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:03.686 INFO 21062 --- [ocp-async-3,84845c1a8314118d,0f5676f61ef52abc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:03.686 WARN 21062 --- [ocp-async-2,72ec29e5bcd85fe5,b9891a36705956bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:05.686 WARN 21062 --- [ocp-async-8,9d60395375cc721c,1418c5311b6a6b68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:05.686 INFO 21062 --- [ocp-async-6,3b54bb577e045c9f,c168a1bcad9d0537] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:05.983 INFO 21062 --- [prometheus-minute-schedule-1,dfd8ae468de90236,d4d61d1fa102d491] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:06.539 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:06.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:07.686 INFO 21062 --- [ocp-async-7,b37b253610fbbd6a,ca8b8a38dbc726a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:07.686 WARN 21062 --- [ocp-async-5,4def9ba1453426b3,68059219a58de41a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:08.504 INFO 21062 --- [ocp-schedules-16,1108f60c46473d1e,cd2ab8531488a7a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:46:08.505 INFO 21062 --- [ocp-schedules-27,14f0ef376181b72b,54cdb1f4deab3565] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:46:08.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:46:08.587400+08:00, lastProcessTime=2024-11-04T15:45:58.589731+08:00 2024-11-04 15:46:08.752 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:46:08.587400+08:00 2024-11-04 15:46:08.821 INFO 21062 --- [ocp-async-2,fa907c43bcc6d5f3,2341f590b0cdef50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:46:08.983 INFO 21062 --- [prometheus-minute-schedule-1,3a4ddc3b9b7608e8,640dbbb775a67e60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:09.410 INFO 21062 --- [ocp-schedules-1,3b68db7785890c09,a6a53ee81a8028d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:46:09.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:09.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:09.686 WARN 21062 --- [ocp-async-1,8d804619b7d8c04a,098d702279f6ae80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:09.686 INFO 21062 --- [ocp-async-3,fc439ac757f442c9,188d422ab46db66c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:10.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:46:10.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:46:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:46:10.300 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:46:10.300 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:46:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:46:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:46:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:46:11.007 INFO 21062 --- [ocp-schedules-7,c9e7f62f7382f0a1,d3106d714a42091f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:46:11.687 WARN 21062 --- [ocp-async-10,4bc856f24ae9945b,04f36a36011b695b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:11.687 INFO 21062 --- [ocp-async-6,1e9e403036393bfe,082e61a9418c50c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:11.730 INFO 21062 --- [ocp-schedules-23,579d3f7f3fe20991,2b96df4888214bc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:46:11.730 INFO 21062 --- [ocp-schedules-23,9aa5e6ee91dcbcf1,ebb2d80bce1d4bdc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:46:11.734 INFO 21062 --- [ocp-schedules-23,9aa5e6ee91dcbcf1,ebb2d80bce1d4bdc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:11.764 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:46:11.764 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:46:11.930 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=2, affectRows=2 2024-11-04 15:46:11.930 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=2, validCount=2 2024-11-04 15:46:11.931 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=10, affectRows=10 2024-11-04 15:46:11.931 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=17, affectRows=17 2024-11-04 15:46:11.931 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=10, validCount=10 2024-11-04 15:46:11.931 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=17, validCount=17 2024-11-04 15:46:11.931 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=12, affectRows=12 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=12, validCount=12 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=19, affectRows=19 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=18, validCount=18 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=31, affectRows=31 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=25, affectRows=25 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-52,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=31, validCount=31 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=25, validCount=25 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=51, affectRows=51 2024-11-04 15:46:11.932 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=46, affectRows=46 2024-11-04 15:46:11.933 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=57, affectRows=57 2024-11-04 15:46:11.933 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=51, validCount=51 2024-11-04 15:46:11.933 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=46, validCount=46 2024-11-04 15:46:11.933 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=57, validCount=57 2024-11-04 15:46:11.933 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=76, affectRows=76 2024-11-04 15:46:11.933 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706300, nodeCount=76, affectRows=76 2024-11-04 15:46:11.934 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=75, validCount=75 2024-11-04 15:46:11.934 INFO 21062 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706300, nodeCount=75, validCount=75 2024-11-04 15:46:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9475a16b617e2fb8,66e84525afd71848] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:12.034 INFO 21062 --- [ocp-schedules-3,909ad6ace2041e01,8e04cdafdd001b74] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:46:12.540 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:12.541 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:13.540 INFO 21062 --- [ocp-schedules-4,a8510a2623065e66,42c4e9a11035f330] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:46:13.546 INFO 21062 --- [ocp-schedules-4,a8510a2623065e66,42c4e9a11035f330] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:46:13.546 INFO 21062 --- [ocp-schedules-4,a8510a2623065e66,42c4e9a11035f330] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:46:13.687 WARN 21062 --- [ocp-async-4,22c799e04e0a60a0,f73d5b02177e3eb8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:13.687 INFO 21062 --- [ocp-async-7,c5218372d8d1fc55,730ddf974e764b5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,7e447bb58ee37de1,807146132ee7649f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:46:13.984 INFO 21062 --- [prometheus-cache-monitor1,7e447bb58ee37de1,807146132ee7649f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:46:14.983 INFO 21062 --- [prometheus-minute-schedule-1,f461ef1c1f7ce9e9,e8f4abfdd6fdc646] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:15.541 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:15.543 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:15.687 WARN 21062 --- [ocp-async-1,ff3f6e2eaa04303e,03e8b5bce7843e87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:15.687 INFO 21062 --- [ocp-async-2,329dbd3eaf4a40ed,e7e9400038a2844d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:17.687 WARN 21062 --- [ocp-async-10,1488449688d9a899,dbad2fde55e836c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:17.687 INFO 21062 --- [ocp-async-8,1b644257544ef958,05766fdd5467fde3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:17.983 INFO 21062 --- [prometheus-minute-schedule-1,27f408b80817870e,ff5155340407e259] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:18.543 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:18.543 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:46:18.589083+08:00, lastProcessTime=2024-11-04T15:46:08.587400+08:00 2024-11-04 15:46:18.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:46:18.589083+08:00 2024-11-04 15:46:18.821 INFO 21062 --- [ocp-async-4,2dd4d315637eeecb,aa119c6485ddca7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:46:19.687 WARN 21062 --- [ocp-async-5,16abe18a6ebf0b0f,c5bb086a1486f573] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:19.687 INFO 21062 --- [ocp-async-7,4070f30e6339a294,1d4d999ddc5820c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:20.983 INFO 21062 --- [prometheus-minute-schedule-1,284507afc28ff7bd,7c47e8095e070817] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:21.050 INFO 21062 --- [ocp-schedules-30,4f835deffb9e788a,b3c87c918ada58a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:46:21.544 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:21.544 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:21.687 WARN 21062 --- [ocp-async-9,dd98777694a20478,6a0e85b3f9137337] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:21.688 INFO 21062 --- [ocp-async-2,88532deb284c1ac6,5e625b15556c2c62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:22.036 INFO 21062 --- [ocp-schedules-10,39f3cf53874d1987,f3d7d2fbeae33103] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:46:23.688 WARN 21062 --- [ocp-async-10,5766aaae960b701a,f043ea162095dffc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:23.688 INFO 21062 --- [ocp-async-10,77f209addb51ae55,412379fab1be3932] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:23.983 INFO 21062 --- [prometheus-minute-schedule-1,f0ba04b0a605fc9c,384b421ad517e971] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:24.545 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:24.546 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:25.688 WARN 21062 --- [ocp-async-6,a9727820c6c12950,1689da68a38a3c18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:25.688 INFO 21062 --- [ocp-async-5,c3e6b126dd09e70d,12070551c98edb5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:26.983 INFO 21062 --- [prometheus-minute-schedule-1,bfdaf4852604c5d3,b2042cd96cdfd903] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:26.984 INFO 21062 --- [prometheus-hour-schedule-1,923a74713ad72312,c9c7259a4a4c2569] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:46:27.546 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:27.547 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:27.688 WARN 21062 --- [ocp-async-1,bc7544682e6162f7,a4a4016726386a24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:27.688 INFO 21062 --- [ocp-async-9,46302d926e27ea09,cc2d5b3ff7643567] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:28.498 INFO 21062 --- [ocp-schedules-7,a8e2cb2eb5350715,77ed0932c4dd9947] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:46:28.500 INFO 21062 --- [ocp-schedules-2,88ef90ff6d7bc04c,a0469cdf5d36ccf1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:46:28.500 INFO 21062 --- [ocp-schedules-22,f4284b1307cb9dd4,80f883394a51a036] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:46:28.500 INFO 21062 --- [ocp-schedules-28,1889ef1d1cb697af,44cd34e49432d66d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:46:28.502 INFO 21062 --- [ocp-schedules-7,a8e2cb2eb5350715,77ed0932c4dd9947] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:46:28.502 INFO 21062 --- [ocp-schedules-24,449f68cd017ecc83,b876f14327a03cad] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:46:28.503 INFO 21062 --- [ocp-schedules-2,88ef90ff6d7bc04c,a0469cdf5d36ccf1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:28.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:46:28.504 INFO 21062 --- [ocp-schedules-28,1889ef1d1cb697af,44cd34e49432d66d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:28.504 INFO 21062 --- [ocp-schedules-12,683a51738a4ec7ea,c0175d2bb09d0d9d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:46:28.506 INFO 21062 --- [ocp-schedules-22,f4284b1307cb9dd4,80f883394a51a036] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:46:28.511 INFO 21062 --- [ocp-schedules-4,7ef29ccce9be5cc1,2a0677388bde0001] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:46:28.511 INFO 21062 --- [ocp-schedules-14,a5c0b4c0a0531665,a373fd2becee8cee] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:46:28.511 INFO 21062 --- [ocp-schedules-14,a5c0b4c0a0531665,a373fd2becee8cee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:46:28.511 INFO 21062 --- [ocp-schedules-4,7ef29ccce9be5cc1,2a0677388bde0001] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:46:28.512 INFO 21062 --- [ocp-schedules-30,d4b929e6a91bb80e,8b7898590ec20fde] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:46:28.512 INFO 21062 --- [ocp-schedules-30,d4b929e6a91bb80e,8b7898590ec20fde] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:46:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:46:28.589726+08:00, lastProcessTime=2024-11-04T15:46:18.589083+08:00 2024-11-04 15:46:28.592 INFO 21062 --- [ocp-schedules-21,e398e97f14e35394,563f479e90ea0350] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:46:28.592 INFO 21062 --- [ocp-schedules-21,e398e97f14e35394,563f479e90ea0350] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:46:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:46:28.589726+08:00 2024-11-04 15:46:28.821 INFO 21062 --- [ocp-async-6,7c8efef831d4b661,825f4d8e9b2068e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:46:29.029 INFO 21062 --- [ocp-schedules-26,eb5c73427230995c,c8d0678473a4e00b] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:46:29.030 INFO 21062 --- [ocp-schedules-26,eb5c73427230995c,c8d0678473a4e00b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:46:29.045 INFO 21062 --- [ocp-schedules-22,13d39a559ae3b70e,eeb5f23d1678294c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:46:29.484 INFO 21062 --- [ocp-schedules-5,53783038fb9ce569,3b21275b29b76df8] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:46:29.484 INFO 21062 --- [ocp-schedules-5,53783038fb9ce569,3b21275b29b76df8] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:46:29.639 INFO 21062 --- [ocp-schedules-13,cc50033acddae23e,03b51cb67c121fb7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:46:29.651 INFO 21062 --- [ocp-schedules-25,9c999da02091c84f,749a7a85b146b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:46:29.656 INFO 21062 --- [ocp-schedules-25,9c999da02091c84f,749a7a85b146b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:46:29.656 INFO 21062 --- [ocp-schedules-25,9c999da02091c84f,749a7a85b146b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:46:29.688 WARN 21062 --- [ocp-async-6,2197bb74be9bfa27,f0e6ee57e7d3bc56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:29.688 INFO 21062 --- [ocp-async-10,5644ae1bf138d57d,fd4b82bdb095d772] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:29.983 INFO 21062 --- [prometheus-minute-schedule-1,f3fff4ed8b0ef37c,4397232ffb40282c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:30.000 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:46:30.000 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:46:30.010 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:46:30.010 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:46:30.010 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:46:30.020 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:46:30.020 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:46:30.020 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:46:30.024 INFO 21062 --- [ocp-schedules-16,b3eae5caedcf7e2f,e5b465df00aefdfa] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:46:30.181 INFO 21062 --- [ocp-schedules-6,ea998111b44ed4c7,86eb1ab7c255ca01] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:46:30.181 INFO 21062 --- [ocp-schedules-6,ea998111b44ed4c7,86eb1ab7c255ca01] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:46:30.182 INFO 21062 --- [ocp-schedules-6,ea998111b44ed4c7,86eb1ab7c255ca01] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:46:30.183 INFO 21062 --- [ocp-schedules-6,ea998111b44ed4c7,86eb1ab7c255ca01] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:46:30.184 INFO 21062 --- [ocp-schedules-6,ea998111b44ed4c7,86eb1ab7c255ca01] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:46:30.184 INFO 21062 --- [ocp-schedules-6,ea998111b44ed4c7,86eb1ab7c255ca01] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:46:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:46:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:46:30.547 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:30.548 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:31.077 INFO 21062 --- [ocp-schedules-20,0208fbb9d4785cf6,5ab283a0cdf80af0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:46:31.689 WARN 21062 --- [ocp-async-1,f78590b0df5882a5,32f11520043043e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:31.689 INFO 21062 --- [ocp-async-5,53ff921872d7bf57,8a72f02237e38066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:32.040 INFO 21062 --- [ocp-schedules-4,1ec5adb2c71a03d2,159426ccfacac5a0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:46:32.983 INFO 21062 --- [prometheus-minute-schedule-1,d521324afb566602,772448af56e0d9d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:33.548 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:33.548 INFO 21062 --- [ocp-schedules-26,3a5d35606da3f818,3b2a96c7c5d6d118] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:46:33.549 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:33.555 INFO 21062 --- [ocp-schedules-26,3a5d35606da3f818,3b2a96c7c5d6d118] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:46:33.555 INFO 21062 --- [ocp-schedules-26,3a5d35606da3f818,3b2a96c7c5d6d118] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:46:33.689 WARN 21062 --- [ocp-async-7,573c9817c9c1faa9,4de652aea68a8a42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:33.689 INFO 21062 --- [ocp-async-7,7d2728021da20f44,a81a02dac8b93d2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:33.876 INFO 21062 --- [ocp-schedules-3,f3b40df6737c07ee,345694fdeee8961d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:46:33.876 INFO 21062 --- [ocp-schedules-3,f3b40df6737c07ee,345694fdeee8961d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:46:33.903 INFO 21062 --- [ocp-schedules-3,f3b40df6737c07ee,345694fdeee8961d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:46:33.903 INFO 21062 --- [ocp-schedules-3,f3b40df6737c07ee,345694fdeee8961d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:46:33.903 INFO 21062 --- [ocp-schedules-12,db9b3dd8b17da841,831516029c3b0b8d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:46:34.079 INFO 21062 --- [ocp-schedules-14,e5ce98a881887416,22be6a809bf52990] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:46:34.079+08:00, clusters=[1] 2024-11-04 15:46:34.082 INFO 21062 --- [ocp-schedules-14,e5ce98a881887416,22be6a809bf52990] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:46:34.438 INFO 21062 --- [ocp-schedules-27,dbadcbb84e9d9665,34fed99d385c0090] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:46:34.438+08:00, clusters=[1] 2024-11-04 15:46:34.442 INFO 21062 --- [ocp-schedules-27,dbadcbb84e9d9665,34fed99d385c0090] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:46:34.548 INFO 21062 --- [ocp-schedules-8,251e9a38ec1ca4f1,c7bc839f20c56146] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:46:34.552 INFO 21062 --- [ocp-schedules-8,251e9a38ec1ca4f1,c7bc839f20c56146] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:34.610 ERROR 21062 --- [ocp-schedules-8,251e9a38ec1ca4f1,c7bc839f20c56146] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:46:34.611 INFO 21062 --- [ocp-schedules-8,251e9a38ec1ca4f1,c7bc839f20c56146] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:46:35.689 WARN 21062 --- [ocp-async-6,43834d75e5ba4543,8670da472ad5e005] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:35.689 INFO 21062 --- [ocp-async-2,6ec7324650619de3,e6a1a38865ffcc50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c8c66dc6c21b176d,c4dae9e3fcc239e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:36.549 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:36.549 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:36.558 INFO 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:46:36.558 INFO 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:46:36.558 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:46:36.558 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:46:36.567 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:46:36.567 INFO 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:46:36.567 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:46:36.569 WARN 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:46:36.569 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:46:36.569 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:46:36.570 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:46:36.570 INFO 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:46:36.570 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:46:36.571 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:46:36.571 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:46:36.571 WARN 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:46:36.579 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:46:36.579 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:46:36.579 INFO 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:46:36.589 ERROR 21062 --- [ocp-schedules-13,67c774f399a89428,89cdb76e074dd020] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:46:36.947 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:46:36.947 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:46:36.955 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:46:36.955 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:46:36.956 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:46:36.956 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:46:36.962 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:46:36.962 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:46:36.962 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:46:36.962 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:46:36.997 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:46:37.005 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:46:37.005 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:46:37.006 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:46:37.006 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:46:37.006 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:46:37.013 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:46:37.588 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:46:37.689 WARN 21062 --- [ocp-async-1,1b195921e4d9e8f4,e92af5ea884714ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:37.689 INFO 21062 --- [ocp-async-10,cfaf68ec03be7496,a131236310f0ea56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:38.159 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.196537893S 2024-11-04 15:46:38.159 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:46:38.258 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:46:38.258 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:46:38.287 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:46:38.346 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:46:38.346 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:46:38.346 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:46:38.346 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:46:38.346 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:46:38.347 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:46:38.358 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:46:38.504 INFO 21062 --- [ocp-schedules-2,88ef90ff6d7bc04c,a0469cdf5d36ccf1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:46:38.504 INFO 21062 --- [ocp-schedules-28,1889ef1d1cb697af,44cd34e49432d66d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:46:38.540 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] o.s.boot.SpringApplication : Started application in 1.584 seconds (JVM running for 11812.219) 2024-11-04 15:46:38.540 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.584823752S 2024-11-04 15:46:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:46:38.589494+08:00, lastProcessTime=2024-11-04T15:46:28.589726+08:00 2024-11-04 15:46:38.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:46:38.589494+08:00 2024-11-04 15:46:38.822 INFO 21062 --- [ocp-async-7,9dfbeb4a0dda88f6,7e2885570d3a489f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:46:38.983 INFO 21062 --- [prometheus-minute-schedule-1,fbb47faeb89f56dc,7a6479a941679236] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:39.009 INFO 21062 --- [ocp-schedules-10,830de02dc334cea3,1ff8dec9334b5c4c] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:46:39.550 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:39.551 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:39.689 WARN 21062 --- [ocp-async-7,1fb2d5072c8cb8e7,49a4644496ffa545] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:39.689 INFO 21062 --- [ocp-async-6,e35f1c7e8ba5d715,35d5726227b932a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:40.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:46:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:46:41.119 INFO 21062 --- [ocp-schedules-2,45a7194d6bf5d1d1,cadc44442517e46d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:46:41.690 WARN 21062 --- [ocp-async-2,f888e4a90a7e5e82,4e2f34fb8fbf1b84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:41.690 INFO 21062 --- [ocp-async-2,7a1e3725954a5445,ff7fe0f6f0d298e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:41.765 INFO 21062 --- [ocp-schedules-23,9aa5e6ee91dcbcf1,ebb2d80bce1d4bdc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:46:41.765 INFO 21062 --- [ocp-schedules-23,9af9e2b51feb1d9d,a20c8204a51702cd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:46:41.769 INFO 21062 --- [ocp-schedules-23,9af9e2b51feb1d9d,a20c8204a51702cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:41.799 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:46:41.799 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:46:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:46:41.824 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:46:41.825 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:46:41.826 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:46:41.826 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:46:41.851 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:46:41.851 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=25ms, period=120000ms 2024-11-04 15:46:41.983 INFO 21062 --- [prometheus-minute-schedule-1,cf3b203223b4c9b6,d589f3a09b6434cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:42.044 INFO 21062 --- [ocp-schedules-3,842bf1ae3edb472a,3649e7dd33816d40] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:46:42.551 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:42.552 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:43.690 WARN 21062 --- [ocp-async-10,cab1ee7a3d0806fc,9f290a7c79961991] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:43.690 INFO 21062 --- [ocp-async-10,3aad4386c71474ed,9d6c72c88cf14cb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:44.983 INFO 21062 --- [prometheus-minute-schedule-1,a8e484222c413270,51b0d5c955a23b2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:45.552 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:45.553 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:45.690 WARN 21062 --- [ocp-async-7,8eb3ecf6cec9c4ef,124d01f5c38ae5bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:45.690 INFO 21062 --- [ocp-async-9,d1976c8417f820f2,dd1ef9abfb38f41b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:46.465 INFO 21062 --- [ocp-schedules-8,ef0e0e19c5606303,a089bd4b5e32d3d0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:46:46.468 INFO 21062 --- [ocp-schedules-8,ef0e0e19c5606303,a089bd4b5e32d3d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:46.553 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:46:46.555 ERROR 21062 --- [ocp-schedules-8,ef0e0e19c5606303,a089bd4b5e32d3d0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:46:46.556 INFO 21062 --- [ocp-schedules-8,ef0e0e19c5606303,a089bd4b5e32d3d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:46:47.690 WARN 21062 --- [ocp-async-1,b90e88295df10e5f,1ff459d1f98a0f8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:47.690 INFO 21062 --- [ocp-async-8,756bcf316b4cad83,f26a4d7ce0e55e35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:47.983 INFO 21062 --- [prometheus-minute-schedule-1,decc0b6c4013e094,96316a136f25608d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:48.553 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:48.554 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:46:48.588913+08:00, lastProcessTime=2024-11-04T15:46:38.589494+08:00 2024-11-04 15:46:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:46:48.588913+08:00 2024-11-04 15:46:48.822 INFO 21062 --- [ocp-async-5,2d07fd26587e73e0,377c423a3a3ba6aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:46:49.690 WARN 21062 --- [ocp-async-4,0fefe7638f96188c,caa14030a60ab742] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:49.691 INFO 21062 --- [ocp-async-10,448ca0b6303fe3c4,edfd4d603b7e4282] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:50.983 INFO 21062 --- [prometheus-minute-schedule-1,8aaed76a86c3364d,a632c10a5a056e63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:51.145 INFO 21062 --- [ocp-schedules-22,eb43b6f985adcfd1,177b8f69062f708d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:46:51.554 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:51.555 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:51.691 WARN 21062 --- [ocp-async-3,508be562e1653199,8f924ede060d86e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:51.691 INFO 21062 --- [ocp-async-9,a21d85cb53503794,de208b1d9a4b7ec5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:52.046 INFO 21062 --- [ocp-schedules-1,da4228acb3abf166,71d2ac664b63b336] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:46:53.557 INFO 21062 --- [ocp-schedules-6,898bb629faf5f706,52127e36ede8f5b4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:46:53.562 INFO 21062 --- [ocp-schedules-6,898bb629faf5f706,52127e36ede8f5b4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:46:53.562 INFO 21062 --- [ocp-schedules-6,898bb629faf5f706,52127e36ede8f5b4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:46:53.691 WARN 21062 --- [ocp-async-6,4c22bf0217343bee,158840cda05e8071] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:53.691 INFO 21062 --- [ocp-async-8,f4d19d9c14cdd136,832fe703bc4d2617] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:53.983 INFO 21062 --- [prometheus-minute-schedule-1,06aa25f4fa1b1391,3786660bb7abe349] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:54.555 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:54.556 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:55.691 WARN 21062 --- [ocp-async-4,b94b2236792ec11d,bd91a016b6b9216e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:55.691 INFO 21062 --- [ocp-async-5,8bd3eb818cb042c8,c99e49151c8286e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:56.983 INFO 21062 --- [prometheus-minute-schedule-1,03805c65a3a8043a,99fc62dfb6aa5262] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:46:57.556 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:46:57.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:46:57.691 WARN 21062 --- [ocp-async-3,83e0f4e05eb59620,cd9002177aab171a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:57.691 INFO 21062 --- [ocp-async-7,23e1e6bcb563e84d,b9bf7116e0c95308] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:58.500 INFO 21062 --- [ocp-schedules-22,2c6d2e2ce5239f2b,1c56dba82167474f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:46:58.500 INFO 21062 --- [ocp-schedules-27,af05039939caa18f,65f50ce772a50e6d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:46:58.502 INFO 21062 --- [ocp-schedules-13,421ad6aa7a634fd5,0f2bed9062b28344] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:46:58.504 INFO 21062 --- [ocp-schedules-27,af05039939caa18f,65f50ce772a50e6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:58.504 INFO 21062 --- [ocp-schedules-22,2c6d2e2ce5239f2b,1c56dba82167474f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:46:58.505 INFO 21062 --- [ocp-schedules-21,ae37bb52cc08087f,f8776df254038ace] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:46:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:46:58.509 INFO 21062 --- [ocp-schedules-30,3fa009458b92e8f1,0b404fbc3b226e99] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:46:58.509 INFO 21062 --- [ocp-schedules-30,3fa009458b92e8f1,0b404fbc3b226e99] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:46:58.510 INFO 21062 --- [ocp-schedules-25,eae0769ef0da4f38,326752382741df86] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:46:58.510 INFO 21062 --- [ocp-schedules-25,eae0769ef0da4f38,326752382741df86] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:46:58.510 INFO 21062 --- [ocp-schedules-11,ad676a6c4a0695c7,9a2645a0c44c1a3e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:46:58.510 INFO 21062 --- [ocp-schedules-11,ad676a6c4a0695c7,9a2645a0c44c1a3e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:46:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:46:58.588681+08:00, lastProcessTime=2024-11-04T15:46:48.588913+08:00 2024-11-04 15:46:58.797 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:46:58.588681+08:00 2024-11-04 15:46:58.822 INFO 21062 --- [ocp-async-6,d8e9248f07e86023,b8686aeb4e3eb64b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:46:59.641 INFO 21062 --- [ocp-schedules-25,677d8f7ecc3265c2,b3ac6debb41108b5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:46:59.691 WARN 21062 --- [ocp-async-1,76eac42b9d1252d6,d89485e27cacac8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:46:59.691 INFO 21062 --- [ocp-async-8,e1b885aafadf7473,d878570a59ae9991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:46:59.983 INFO 21062 --- [prometheus-minute-schedule-1,8f76fa600a1d9d93,54f7b5b8e14994d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706420, interval=60 2024-11-04 15:47:00.557 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:00.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:01.189 INFO 21062 --- [ocp-schedules-5,7e6d12052dfa2cac,8add59945709b98e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:47:01.691 WARN 21062 --- [ocp-async-2,09b2b6eb6d888300,103c6a7ab812dd7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:01.691 INFO 21062 --- [ocp-async-5,68ef60393be5508f,6f9d4e25aa30d415] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:02.051 INFO 21062 --- [ocp-schedules-10,2e28271b7295172e,57f4ce69f43cceba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:47:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5f81d68258e00fa0,77bdade865dfd1ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:03.559 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:03.560 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:03.692 WARN 21062 --- [ocp-async-3,9f5b2288fbb00823,93888e14ec31a2d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:03.692 INFO 21062 --- [ocp-async-3,41147cd21bb8afa1,33d01e373ccc8323] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:05.692 WARN 21062 --- [ocp-async-1,f0230936e1925018,4b34d6c15e8f4ce6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:05.692 INFO 21062 --- [ocp-async-1,03f88a9cbba0a4c1,39c0df92a99593c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:05.983 INFO 21062 --- [prometheus-minute-schedule-1,853a71a6b921aa45,bb53f2ada679b9d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:06.560 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:06.561 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:07.692 WARN 21062 --- [ocp-async-2,10547d2c04d51f81,323f3b210a74b633] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:07.692 INFO 21062 --- [ocp-async-2,d445e22123ea41b7,269b655cbb769ed3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:08.505 INFO 21062 --- [ocp-schedules-27,af05039939caa18f,65f50ce772a50e6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:47:08.505 INFO 21062 --- [ocp-schedules-22,2c6d2e2ce5239f2b,1c56dba82167474f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:47:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:47:08.589957+08:00, lastProcessTime=2024-11-04T15:46:58.588681+08:00 2024-11-04 15:47:08.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:47:08.589957+08:00 2024-11-04 15:47:08.822 INFO 21062 --- [ocp-async-7,d7d837cb762c333b,c98513e87f406c90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:47:08.983 INFO 21062 --- [prometheus-minute-schedule-1,5f63bf1f9e809241,43ec6ffd1c40c75d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:09.410 INFO 21062 --- [ocp-schedules-26,07353cbbbcf87536,efb144b7b8d2bf42] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:47:09.414 INFO 21062 --- [ocp-schedules-26,07353cbbbcf87536,efb144b7b8d2bf42] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:09.414 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:47:09.561 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:09.562 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:09.692 WARN 21062 --- [ocp-async-10,4dcf941128a87702,73a1ac3facaa99ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:09.692 INFO 21062 --- [ocp-async-3,9d90d0127192feac,1392602ca32490d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:10.002 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:47:10.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:47:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:47:10.306 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:47:10.306 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:47:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:47:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:47:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:47:11.215 INFO 21062 --- [ocp-schedules-15,5562e005017462cb,23ec411e2b1bba86] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:47:11.692 WARN 21062 --- [ocp-async-9,74ce1d5236b59d43,4ddd0ce8df8d3868] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:11.692 INFO 21062 --- [ocp-async-9,bcff170a27846156,75c0bcc56a6ce1ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:11.799 INFO 21062 --- [ocp-schedules-23,9af9e2b51feb1d9d,a20c8204a51702cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:47:11.800 INFO 21062 --- [ocp-schedules-23,17c26b9058190385,13518b6681880afb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:47:11.803 INFO 21062 --- [ocp-schedules-23,17c26b9058190385,13518b6681880afb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:11.833 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:47:11.833 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:47:11.937 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706360, nodeCount=87, affectRows=87 2024-11-04 15:47:11.937 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706360, nodeCount=79, affectRows=79 2024-11-04 15:47:11.937 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706360, nodeCount=68, affectRows=68 2024-11-04 15:47:11.938 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706360, nodeCount=109, affectRows=109 2024-11-04 15:47:11.938 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706360, nodeCount=87, validCount=87 2024-11-04 15:47:11.938 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706360, nodeCount=67, validCount=67 2024-11-04 15:47:11.938 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706360, nodeCount=79, validCount=79 2024-11-04 15:47:11.938 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706360, nodeCount=79, affectRows=79 2024-11-04 15:47:11.938 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706360, nodeCount=107, validCount=107 2024-11-04 15:47:11.939 INFO 21062 --- [metric-archive-second-46,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706360, nodeCount=79, validCount=79 2024-11-04 15:47:11.983 INFO 21062 --- [prometheus-minute-schedule-1,bee89bab1ae21741,a4623d3a1bc1617a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:12.054 INFO 21062 --- [ocp-schedules-7,416a26b04a6e446e,677ba1304be28827] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:47:12.562 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:12.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:13.564 INFO 21062 --- [ocp-schedules-30,de4522a0c7abd6a3,b664ae1679700b16] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:47:13.569 INFO 21062 --- [ocp-schedules-30,de4522a0c7abd6a3,b664ae1679700b16] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:47:13.569 INFO 21062 --- [ocp-schedules-30,de4522a0c7abd6a3,b664ae1679700b16] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:47:13.692 INFO 21062 --- [ocp-async-8,e1c475c2b1681f95,1ad6146239cac0e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:13.692 WARN 21062 --- [ocp-async-2,9415d30b0d34d04b,7ce376924735e7f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:13.984 INFO 21062 --- [prometheus-cache-monitor1,99082a56f3d0aa4d,bd4a3ca1ec2b8311] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:47:13.984 INFO 21062 --- [prometheus-cache-monitor1,99082a56f3d0aa4d,bd4a3ca1ec2b8311] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:47:14.983 INFO 21062 --- [prometheus-minute-schedule-1,9a6f9ebd0bd4cdd6,c25fa5ae6d3c2d51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:15.563 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:15.564 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:15.693 INFO 21062 --- [ocp-async-10,a2a25a87d08618f7,b7e9c35a230e7165] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:15.693 WARN 21062 --- [ocp-async-7,295cdf1fefcf5b39,d48340e7fe0bdf0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:15.816 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:43:15.795, end=2024-11-04T15:45:15.813} 2024-11-04 15:47:15.824 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:47:15.824 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:47:17.693 INFO 21062 --- [ocp-async-1,42c257155123b349,6dd5a229b6fce2a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:17.693 WARN 21062 --- [ocp-async-6,509c4385b02cd0fd,0a4ecb4653a4f859] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:17.983 INFO 21062 --- [prometheus-minute-schedule-1,96018b1b59cebc33,ea9146c7905790bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:18.564 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:18.565 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:47:18.588897+08:00, lastProcessTime=2024-11-04T15:47:08.589957+08:00 2024-11-04 15:47:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:47:18.588897+08:00 2024-11-04 15:47:18.823 INFO 21062 --- [ocp-async-2,42cd259253341b57,519a79058fa10f25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:47:19.414 INFO 21062 --- [ocp-schedules-26,07353cbbbcf87536,efb144b7b8d2bf42] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:47:19.693 INFO 21062 --- [ocp-async-4,b124f9f90a2a0978,38c005d39949993e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:19.693 WARN 21062 --- [ocp-async-8,d6c1190f67464942,7934acf38010623d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:20.983 INFO 21062 --- [prometheus-minute-schedule-1,f7d928ef81e65fa7,f4e37b9b54245378] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:21.256 INFO 21062 --- [ocp-schedules-3,f75aebd362111375,4edcd4ba7d471cd4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:47:21.565 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:21.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:21.693 WARN 21062 --- [ocp-async-5,8c7636645f755a0a,8b34d7baf7041292] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:21.693 INFO 21062 --- [ocp-async-7,584a03367a1a9bc2,a8ce7260d9570b0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:22.057 INFO 21062 --- [ocp-schedules-28,cf20093ff14299f8,4f73c5af24db8b52] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:47:23.693 INFO 21062 --- [ocp-async-6,98377ead3411463d,bcb70895939a63a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:23.693 WARN 21062 --- [ocp-async-3,a65d814a8ea509c3,f3533a0914d74f2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:23.983 INFO 21062 --- [prometheus-minute-schedule-1,8aa5b464e26f0d86,59a8fec2dafd1013] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:24.566 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:24.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:25.693 WARN 21062 --- [ocp-async-8,42e5e30a07972ef7,1c1752e4dabc8e2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:25.693 INFO 21062 --- [ocp-async-2,5dbdf03c5c3c843a,824bdb4871b633d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:26.983 INFO 21062 --- [prometheus-minute-schedule-1,bd1f346d1715df69,ca7e5b266112ab9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a1e5f08f6819380c,bbf11dd29207a778] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:47:27.567 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:27.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:27.694 INFO 21062 --- [ocp-async-5,440aef8adc5859a6,ebc040d618c0b832] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:27.694 WARN 21062 --- [ocp-async-4,8044c7f5d81f8672,388ba23acc335a43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:28.498 INFO 21062 --- [ocp-schedules-25,dcf2103b5712e7f6,141265a6b2b7f870] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:47:28.500 INFO 21062 --- [ocp-schedules-16,da99d712d792821a,e804787e53e65303] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:47:28.500 INFO 21062 --- [ocp-schedules-2,32546f9dc02bfbb3,b0b35e2bfaa5747c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:47:28.500 INFO 21062 --- [ocp-schedules-27,8c14c862c0f83fed,3a0cd949c0014ab0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:47:28.502 INFO 21062 --- [ocp-schedules-25,dcf2103b5712e7f6,141265a6b2b7f870] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:47:28.502 INFO 21062 --- [ocp-schedules-17,85b77a55fd1c125c,4c40e8fc3e7ddc2a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:47:28.503 INFO 21062 --- [ocp-schedules-16,da99d712d792821a,e804787e53e65303] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:47:28.504 INFO 21062 --- [ocp-schedules-27,8c14c862c0f83fed,3a0cd949c0014ab0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:28.504 INFO 21062 --- [ocp-schedules-1,e0d1e7fce5420877,ff573e1708ef21a1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:47:28.504 INFO 21062 --- [ocp-schedules-22,efe55589ed0a502a,aac2a5236b54e4c8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:47:28.505 INFO 21062 --- [ocp-schedules-2,32546f9dc02bfbb3,b0b35e2bfaa5747c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:47:28.507 INFO 21062 --- [ocp-schedules-22,efe55589ed0a502a,aac2a5236b54e4c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:28.509 INFO 21062 --- [ocp-schedules-5,9012d3b62a51cac0,fa2f2773e53b2071] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:47:28.509 INFO 21062 --- [ocp-schedules-5,9012d3b62a51cac0,fa2f2773e53b2071] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:47:28.509 INFO 21062 --- [ocp-schedules-11,60be2f8179e264f9,449b1fcf0e3ef8bb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:47:28.509 INFO 21062 --- [ocp-schedules-11,60be2f8179e264f9,449b1fcf0e3ef8bb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:47:28.509 INFO 21062 --- [ocp-schedules-3,1d073279d75c8ab4,945e0ca45510e6dc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:47:28.509 INFO 21062 --- [ocp-schedules-3,1d073279d75c8ab4,945e0ca45510e6dc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:47:28.543 INFO 21062 --- [ocp-schedules-24,c130a8eb3ebb87ae,5b89ef390da26280] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:47:28.543 INFO 21062 --- [ocp-schedules-24,c130a8eb3ebb87ae,5b89ef390da26280] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:47:28.583 INFO 21062 --- [pool-12-thread-15,e810ed7c0d38f4a1,51cb9b568ac1868e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:47:28.583 INFO 21062 --- [pool-12-thread-14,e810ed7c0d38f4a1,d9461d5df1759175] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:47:28.583 INFO 21062 --- [pool-12-thread-13,e810ed7c0d38f4a1,39fba92457a086ba] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:47:28.583 INFO 21062 --- [pool-12-thread-13,e810ed7c0d38f4a1,39fba92457a086ba] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:47:28.584 INFO 21062 --- [pool-12-thread-16,e810ed7c0d38f4a1,4f42070cec70cd6a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:47:28.587 INFO 21062 --- [pool-12-thread-13,e810ed7c0d38f4a1,39fba92457a086ba] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:47:28.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:47:28.588449+08:00, lastProcessTime=2024-11-04T15:47:18.588897+08:00 2024-11-04 15:47:28.589 INFO 21062 --- [ForkJoinPool-11-worker-12,e810ed7c0d38f4a1,7b9f6c2df254a9f9] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:47:28.589 INFO 21062 --- [pool-12-thread-15,e810ed7c0d38f4a1,51cb9b568ac1868e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:47:28.589 INFO 21062 --- [pool-12-thread-13,e810ed7c0d38f4a1,39fba92457a086ba] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:47:28.589 INFO 21062 --- [ForkJoinPool-11-worker-5,e810ed7c0d38f4a1,3228b1b11ca86c9c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:47:28.589 INFO 21062 --- [ForkJoinPool-11-worker-5,e810ed7c0d38f4a1,3228b1b11ca86c9c] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:47:28.589 INFO 21062 --- [ForkJoinPool-11-worker-5,e810ed7c0d38f4a1,3228b1b11ca86c9c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:47:28.589 INFO 21062 --- [pool-12-thread-16,e810ed7c0d38f4a1,4f42070cec70cd6a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:47:28.591 INFO 21062 --- [pool-12-thread-14,e810ed7c0d38f4a1,d9461d5df1759175] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:47:28.591 INFO 21062 --- [pool-12-thread-14,e810ed7c0d38f4a1,d9461d5df1759175] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:47:28.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:47:28.588449+08:00 2024-11-04 15:47:28.823 INFO 21062 --- [ocp-async-8,fb6e7fdb9a77a168,01589baea0b2c66c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:47:29.048 INFO 21062 --- [ocp-schedules-11,7f6616523b02f1d5,533973e4083b6b65] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:47:29.488 INFO 21062 --- [ocp-schedules-21,42d7c023add94f6d,c85b4f5d54e33d5e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:47:29.488 INFO 21062 --- [ocp-schedules-21,42d7c023add94f6d,c85b4f5d54e33d5e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:47:29.644 INFO 21062 --- [ocp-schedules-15,4627a82dc36ef244,5a6be58c6e2bd9bf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:47:29.656 INFO 21062 --- [ocp-schedules-9,1484bf1c9534b639,3e78fd025f097ea8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:47:29.661 INFO 21062 --- [ocp-schedules-9,1484bf1c9534b639,3e78fd025f097ea8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:47:29.661 INFO 21062 --- [ocp-schedules-9,1484bf1c9534b639,3e78fd025f097ea8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:47:29.683 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:47:29.694 INFO 21062 --- [ocp-async-8,ee176376a4b4a253,c4bf6697e4fad474] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:29.694 WARN 21062 --- [ocp-async-6,ae6866eefb029195,9bb1147559dc99bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:29.699 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:47:29.701 WARN 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:47:29.701 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:47:29.702 WARN 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:47:29.709 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:47:29.710 WARN 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:47:29.711 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:47:29.712 WARN 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:47:29.718 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:47:29.719 WARN 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:47:29.720 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:47:29.721 WARN 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:47:29.726 INFO 21062 --- [ocp-schedules-30,852b9b8ca8a13f15,f4a9b935c22fc4d1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:47:29.983 INFO 21062 --- [prometheus-minute-schedule-1,bae59ea44f802719,23178f0d7b48753d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:30.185 INFO 21062 --- [ocp-schedules-11,82cd64fe1862a05e,cbc91b443cde8cd1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:47:30.185 INFO 21062 --- [ocp-schedules-11,82cd64fe1862a05e,cbc91b443cde8cd1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:47:30.193 INFO 21062 --- [ocp-schedules-11,82cd64fe1862a05e,cbc91b443cde8cd1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:47:30.193 INFO 21062 --- [ocp-schedules-11,82cd64fe1862a05e,cbc91b443cde8cd1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:47:30.198 INFO 21062 --- [ocp-schedules-11,82cd64fe1862a05e,cbc91b443cde8cd1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:47:30.198 INFO 21062 --- [ocp-schedules-11,82cd64fe1862a05e,cbc91b443cde8cd1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:47:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:47:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:47:30.568 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:30.569 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:30.717 INFO 21062 --- [ocp-schedules-21,3fffc9bd552f9119,873a874281b842d1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:47:30.717 INFO 21062 --- [ocp-schedules-21,3fffc9bd552f9119,873a874281b842d1] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:47:30.747 INFO 21062 --- [pool-58-thread-1,3fffc9bd552f9119,0bdbbe267ec49d77] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:47:30.776 WARN 21062 --- [ocp-schedules-21,3fffc9bd552f9119,873a874281b842d1] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:47:30.776 INFO 21062 --- [ocp-schedules-21,3fffc9bd552f9119,873a874281b842d1] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:47:31.282 INFO 21062 --- [ocp-schedules-2,a8546ef493310d96,7ab32be4f4a936dd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:47:31.694 WARN 21062 --- [ocp-async-5,f741102ee7e3c44d,1a7d39719528b5bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:31.694 INFO 21062 --- [ocp-async-4,c93b9a9ed8569b74,1b6bd15987fdbb4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:32.059 INFO 21062 --- [ocp-schedules-6,dc59b078ddc79889,b3448be542a67c38] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:47:32.983 INFO 21062 --- [prometheus-minute-schedule-1,af7329cadb4a8490,1faacb8303c8f3fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:33.569 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:33.570 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:33.572 INFO 21062 --- [ocp-schedules-14,e94f229ff7804152,a7fcefbccb0afc07] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:47:33.577 INFO 21062 --- [ocp-schedules-14,e94f229ff7804152,a7fcefbccb0afc07] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:47:33.577 INFO 21062 --- [ocp-schedules-14,e94f229ff7804152,a7fcefbccb0afc07] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:47:33.694 INFO 21062 --- [ocp-async-10,7e6c9f79ce37250d,2a72ba2f6087ec89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:33.694 WARN 21062 --- [ocp-async-3,68a41402fab96319,64ace83e6a8e47d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:33.903 INFO 21062 --- [ocp-schedules-25,d02644063f5ee531,50f774df928c5719] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:47:33.903 INFO 21062 --- [ocp-schedules-25,d02644063f5ee531,50f774df928c5719] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:47:33.929 INFO 21062 --- [ocp-schedules-25,d02644063f5ee531,50f774df928c5719] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:47:33.929 INFO 21062 --- [ocp-schedules-25,d02644063f5ee531,50f774df928c5719] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:47:33.931 INFO 21062 --- [ocp-schedules-21,44b53c8eaf52c7af,fdf2eb42451fc282] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:47:34.106 INFO 21062 --- [ocp-schedules-15,a4b04e88636ba9f6,973c2baa3eb85549] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:47:34.106+08:00, clusters=[1] 2024-11-04 15:47:34.108 INFO 21062 --- [ocp-schedules-15,a4b04e88636ba9f6,973c2baa3eb85549] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:47:34.468 INFO 21062 --- [ocp-schedules-8,0a99eaba85131aee,730cbc6be66513f0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:47:34.468+08:00, clusters=[1] 2024-11-04 15:47:34.472 INFO 21062 --- [ocp-schedules-8,0a99eaba85131aee,730cbc6be66513f0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:47:35.694 INFO 21062 --- [ocp-async-6,84ba1ed84c3331dc,8ef40b42e371f5a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:35.694 WARN 21062 --- [ocp-async-1,4e62ac7325ae87de,b800ba7ab5b6bac4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:35.983 INFO 21062 --- [prometheus-minute-schedule-1,fc4031148b2bd1d0,7be1dab1fb36972f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:36.571 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:36.572 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:36.599 INFO 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:47:36.599 INFO 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:47:36.599 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:47:36.599 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:47:36.608 INFO 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:47:36.609 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:47:36.610 WARN 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:47:36.610 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:47:36.610 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:47:36.610 INFO 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:47:36.611 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:47:36.611 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:47:36.611 WARN 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:47:36.612 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:47:36.612 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:47:36.613 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:47:36.618 INFO 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:47:36.618 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:47:36.619 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:47:36.630 ERROR 21062 --- [ocp-schedules-21,0c6c4c868b8f08d3,4e81a47e991c5be9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:47:37.695 WARN 21062 --- [ocp-async-9,6cb3e3601c00ee37,41dbcafd83dc65d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:37.695 INFO 21062 --- [ocp-async-5,5ccfdef8604fe661,4751334091383fc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:38.504 INFO 21062 --- [ocp-schedules-16,da99d712d792821a,e804787e53e65303] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:47:38.504 INFO 21062 --- [ocp-schedules-27,8c14c862c0f83fed,3a0cd949c0014ab0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:47:38.508 INFO 21062 --- [ocp-schedules-22,efe55589ed0a502a,aac2a5236b54e4c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:47:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:47:38.588397+08:00, lastProcessTime=2024-11-04T15:47:28.588449+08:00 2024-11-04 15:47:38.772 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:47:38.588397+08:00 2024-11-04 15:47:38.823 INFO 21062 --- [ocp-async-2,8d7d0efbbb66c038,a4d90aa9508994db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:47:38.983 INFO 21062 --- [prometheus-minute-schedule-1,3ea6796b9de0d83b,ae8a97af8cf32ee8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:39.572 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:39.573 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:39.695 WARN 21062 --- [ocp-async-2,d033635861fbd81b,8bfc71bbb56af148] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:39.695 INFO 21062 --- [ocp-async-1,c5971784b637f509,fce6023dc023dfd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:40.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:47:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:47:41.321 INFO 21062 --- [ocp-schedules-13,97f76f6a2160ded0,c2839a2b5b39e15d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:47:41.695 WARN 21062 --- [ocp-async-6,6e9c5106f3808584,cd238574707c660d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:41.695 INFO 21062 --- [ocp-async-9,0036f4ef3e09a9d4,cd15e9b275c17303] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:41.834 INFO 21062 --- [ocp-schedules-23,17c26b9058190385,13518b6681880afb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:47:41.834 INFO 21062 --- [ocp-schedules-23,c4761d2d7b861cdf,1fb843983f70ed68] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:47:41.838 INFO 21062 --- [ocp-schedules-23,c4761d2d7b861cdf,1fb843983f70ed68] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:41.868 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:47:41.868 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:47:41.983 INFO 21062 --- [prometheus-minute-schedule-1,9e3ad4310b38bc73,76674b5fcf39c3a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:42.063 INFO 21062 --- [ocp-schedules-20,7b0a69db9cc86fe0,9d9d9424ef7aebf6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:47:42.573 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:42.574 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:43.695 WARN 21062 --- [ocp-async-4,906d758f3266abe4,80450d8910205dc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:43.695 INFO 21062 --- [ocp-async-5,e4162092f61f13dd,e7e016556264ddd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:44.983 INFO 21062 --- [prometheus-minute-schedule-1,09f0fcdcef552992,16ed61f8d284d1f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:45.574 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:45.575 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:45.695 INFO 21062 --- [ocp-async-2,da6f8183a824ca62,cc80c1ab46b4e6fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:45.695 WARN 21062 --- [ocp-async-10,c8ecec8e43584f76,3c97092e54473712] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:46.556 INFO 21062 --- [ocp-schedules-21,c76f29a9882b53fb,3cada96f622bc08d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:47:46.559 INFO 21062 --- [ocp-schedules-21,c76f29a9882b53fb,3cada96f622bc08d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:46.643 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:47:46.646 ERROR 21062 --- [ocp-schedules-21,c76f29a9882b53fb,3cada96f622bc08d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:47:46.646 INFO 21062 --- [ocp-schedules-21,c76f29a9882b53fb,3cada96f622bc08d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:47:47.695 INFO 21062 --- [ocp-async-6,596c879f80b8dc21,8b3b8889e3a38489] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:47.695 WARN 21062 --- [ocp-async-7,11dbf3599b5b7703,e610fd5dcb47a0d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:47.983 INFO 21062 --- [prometheus-minute-schedule-1,9132a07f3bfde519,08a84cdbd263ded8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:48.575 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:48.576 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:47:48.588860+08:00, lastProcessTime=2024-11-04T15:47:38.588397+08:00 2024-11-04 15:47:48.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:47:48.588860+08:00 2024-11-04 15:47:48.823 INFO 21062 --- [ocp-async-4,6a508cc71c4dc58e,f35a5762c70437c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:47:49.696 WARN 21062 --- [ocp-async-8,416fa931d8d0a964,17e71bfb177039dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:49.696 INFO 21062 --- [ocp-async-5,85cf3563e2c4f22e,15d5094f3cc33891] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b8ad75e4c0cbf414,e34581894617fe44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:51.345 INFO 21062 --- [ocp-schedules-15,81735b2c115072fe,1057ff1b83c6f9e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:47:51.576 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:51.577 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:51.696 WARN 21062 --- [ocp-async-3,61683866eb133baa,dfafd30eab9a4f1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:51.696 INFO 21062 --- [ocp-async-2,6dc6250eabf7ac93,954c59cd5d5767ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:52.064 INFO 21062 --- [ocp-schedules-8,8a772d4838bebfa2,bd2eb89d181171f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:47:53.579 INFO 21062 --- [ocp-schedules-3,3229b34094a27632,66880ce0b04716d3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:47:53.584 INFO 21062 --- [ocp-schedules-3,3229b34094a27632,66880ce0b04716d3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:47:53.584 INFO 21062 --- [ocp-schedules-3,3229b34094a27632,66880ce0b04716d3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:47:53.696 WARN 21062 --- [ocp-async-1,85e10fe95f647305,c79cb9b7c4a06886] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:53.696 INFO 21062 --- [ocp-async-6,e9b819b8e500ada4,35cafb80541f4b15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:53.983 INFO 21062 --- [prometheus-minute-schedule-1,762194b0bf255163,9dd83c22d304bbe3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:54.577 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:54.578 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:55.696 WARN 21062 --- [ocp-async-8,0412ef2022f99aa4,f51401b5c70e3735] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:55.696 INFO 21062 --- [ocp-async-4,04d8f8a82d6d3614,88b0b3a9c30615e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:56.983 INFO 21062 --- [prometheus-minute-schedule-1,ba8b9914ef3953d4,7e4cd7faca92da69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:47:57.578 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:47:57.579 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:47:57.696 WARN 21062 --- [ocp-async-3,ab30aebb85daeb0a,60b6b9bf4571e711] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:57.696 INFO 21062 --- [ocp-async-10,8c1119b9fbbde584,ccd95bef73823aa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:58.500 INFO 21062 --- [ocp-schedules-27,d5feb36dca5a63b1,8bb78399ed33587c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:47:58.500 INFO 21062 --- [ocp-schedules-5,4ba92299b40c6d74,d409aa8895bedd76] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:47:58.502 INFO 21062 --- [ocp-schedules-8,b60e5404b5f8f22f,3f95a696c8492c8b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:47:58.504 INFO 21062 --- [ocp-schedules-27,d5feb36dca5a63b1,8bb78399ed33587c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:58.504 INFO 21062 --- [ocp-schedules-5,4ba92299b40c6d74,d409aa8895bedd76] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:47:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:47:58.505 INFO 21062 --- [ocp-schedules-21,1d69ba20cefc200b,5f359cf41c5ac40e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:47:58.510 INFO 21062 --- [ocp-schedules-13,6da2f3c8eeed9c0f,23d3a54e471ae20e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:47:58.510 INFO 21062 --- [ocp-schedules-13,6da2f3c8eeed9c0f,23d3a54e471ae20e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:47:58.510 INFO 21062 --- [ocp-schedules-22,d2d597aaa2287c3c,d0275d023e8bef57] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:47:58.510 INFO 21062 --- [ocp-schedules-12,637e6c74e8c53fb3,cdc2ef05f848b744] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:47:58.510 INFO 21062 --- [ocp-schedules-22,d2d597aaa2287c3c,d0275d023e8bef57] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:47:58.510 INFO 21062 --- [ocp-schedules-12,637e6c74e8c53fb3,cdc2ef05f848b744] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:47:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:47:58.588728+08:00, lastProcessTime=2024-11-04T15:47:48.588860+08:00 2024-11-04 15:47:58.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:47:58.588728+08:00 2024-11-04 15:47:58.823 INFO 21062 --- [ocp-async-1,ef5d88f51e55e926,97261ea98d3cf65f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:47:59.647 INFO 21062 --- [ocp-schedules-8,bd41001d0063bd88,60ee86ae6ad8590b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:47:59.697 WARN 21062 --- [ocp-async-7,45664ceba03cd951,72e7fb03ac2c2f1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:47:59.697 INFO 21062 --- [ocp-async-6,9059d1a55e72ab60,0b6d4e4db0762923] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:47:59.983 INFO 21062 --- [prometheus-minute-schedule-1,ed0a21fb63f393e2,294ff421ce1181bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706480, interval=60 2024-11-04 15:48:00.579 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:00.580 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:01.388 INFO 21062 --- [ocp-schedules-14,ca4547e5a4629e9c,67527f3a29e44d77] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:48:01.697 WARN 21062 --- [ocp-async-9,a039f39ad6494644,31b86c8389abf08e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:01.697 INFO 21062 --- [ocp-async-4,c31e3337bb0a6f65,cb17b2fc9aa35eb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:02.067 INFO 21062 --- [ocp-schedules-18,dc08217b9bb92627,0f21fecb4cbbc4c3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:48:02.983 INFO 21062 --- [prometheus-minute-schedule-1,2cb158d783a8e913,ba32f7248167aff4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:03.580 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:03.581 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:03.697 WARN 21062 --- [ocp-async-5,0d75b2e1540c9867,7550f7daa43f6273] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:03.697 INFO 21062 --- [ocp-async-10,9468b8315258f5b8,ad020827e40dd9d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:05.697 WARN 21062 --- [ocp-async-7,111d038f7970f574,13a0b74d56311f31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:05.697 INFO 21062 --- [ocp-async-7,dfd7d2c80384afb7,e287ed85594360ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:05.983 INFO 21062 --- [prometheus-minute-schedule-1,c9ccf91f27dfee19,88d9b892a367f006] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:06.581 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:06.582 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:07.697 WARN 21062 --- [ocp-async-9,d01927c0cae85775,82c9be541ac832e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:07.697 INFO 21062 --- [ocp-async-8,c2b9a9bc59705c01,40c84a2e4127204f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:08.505 INFO 21062 --- [ocp-schedules-5,4ba92299b40c6d74,d409aa8895bedd76] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:48:08.505 INFO 21062 --- [ocp-schedules-27,d5feb36dca5a63b1,8bb78399ed33587c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:48:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:48:08.589156+08:00, lastProcessTime=2024-11-04T15:47:58.588728+08:00 2024-11-04 15:48:08.755 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:48:08.589156+08:00 2024-11-04 15:48:08.823 INFO 21062 --- [ocp-async-5,0d1c491b4347e515,1ab78939b76864a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:48:08.983 INFO 21062 --- [prometheus-minute-schedule-1,7e0b6de8d4d7b840,0af5b0387ebb8d0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:09.582 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:09.583 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:09.697 WARN 21062 --- [ocp-async-3,ee637a22ff94ccd1,51954cc607ca1c4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:09.697 INFO 21062 --- [ocp-async-10,354d0a1a7fbdc5e1,536cdce70b320ab6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:10.003 INFO 21062 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:48:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:48:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:48:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:48:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:48:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:48:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:48:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:48:11.415 INFO 21062 --- [ocp-schedules-15,6bc58bdc963655f8,8bf95608a02a92b3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:48:11.698 WARN 21062 --- [ocp-async-2,59bcb47c28a4ebaa,8f9bc06693613686] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:11.698 INFO 21062 --- [ocp-async-2,ea4961c95af0cc57,2236916ee0af9e56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:11.868 INFO 21062 --- [ocp-schedules-23,c4761d2d7b861cdf,1fb843983f70ed68] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:48:11.868 INFO 21062 --- [ocp-schedules-23,62a9a24c2fe86597,53a102d57118a756] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:48:11.872 INFO 21062 --- [ocp-schedules-23,62a9a24c2fe86597,53a102d57118a756] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:11.901 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:48:11.901 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:48:11.941 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706420, nodeCount=39, affectRows=39 2024-11-04 15:48:11.941 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706420, nodeCount=56, affectRows=56 2024-11-04 15:48:11.941 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706420, nodeCount=78, affectRows=78 2024-11-04 15:48:11.941 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706420, nodeCount=43, affectRows=43 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706420, nodeCount=39, validCount=39 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706420, nodeCount=68, affectRows=68 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706420, nodeCount=49, affectRows=49 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706420, nodeCount=43, validCount=43 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706420, nodeCount=56, validCount=56 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706420, nodeCount=89, affectRows=89 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706420, nodeCount=76, validCount=76 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706420, nodeCount=49, validCount=49 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706420, nodeCount=67, validCount=67 2024-11-04 15:48:11.942 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706420, nodeCount=89, validCount=89 2024-11-04 15:48:11.983 INFO 21062 --- [prometheus-minute-schedule-1,beac8a957b2efc5c,c95104786e3d9ba1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:12.071 INFO 21062 --- [ocp-schedules-29,ac5a501e838a70b1,2ad9939677ce0301] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:48:12.583 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:12.584 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:13.586 INFO 21062 --- [ocp-schedules-2,3d5a2939a6fd89bf,4cad3cd5d33e752c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:48:13.592 INFO 21062 --- [ocp-schedules-2,3d5a2939a6fd89bf,4cad3cd5d33e752c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:48:13.592 INFO 21062 --- [ocp-schedules-2,3d5a2939a6fd89bf,4cad3cd5d33e752c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:48:13.698 WARN 21062 --- [ocp-async-9,57badba289fbd3fe,253a8d50871240c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:13.698 INFO 21062 --- [ocp-async-8,7f5537f44b7c58c6,9d65df30966fe9f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:13.984 INFO 21062 --- [prometheus-cache-monitor1,6726785401f6b9b5,b07de8d75c14bc63] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:48:13.984 INFO 21062 --- [prometheus-cache-monitor1,6726785401f6b9b5,b07de8d75c14bc63] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:48:14.559 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:42:14.541, end=2024-11-04T15:47:14.557} 2024-11-04 15:48:14.559 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:42:14.541, end=2024-11-04T15:47:14.557} 2024-11-04 15:48:14.563 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:48:14.563 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:48:14.563 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:48:14.563 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:48:14.983 INFO 21062 --- [prometheus-minute-schedule-1,0e1890ea436c4b4a,9e07228436ed8c8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:15.584 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:15.585 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:15.698 WARN 21062 --- [ocp-async-3,8df9f4fb16007598,72fad9e0f3f2d41f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:15.698 INFO 21062 --- [ocp-async-5,e33617b4c779b3b8,6c1e368341ae590b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:16.732 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 15:48:16.732 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 15:48:16.732 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 15:48:17.497 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,febfd19ce998b35c,ee0dc7b604b3c79a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=febfd19ce998b35c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 15:48:17.497 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,febfd19ce998b35c,ee0dc7b604b3c79a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=febfd19ce998b35c, duration=6 ms] 2024-11-04 15:48:17.632 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,35b8dce5c12581ae,9676540bf4fc3c9c] c.c.s.h.OcpAuthenticationSuccessListener : User[name=admin, origin=LOCAL] authenticated successes. 2024-11-04 15:48:17.646 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,35b8dce5c12581ae,9676540bf4fc3c9c] c.s.h.CustomAuthenticationSuccessHandler : Authentication successful for /api/v2/iam/login 2024-11-04 15:48:17.646 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,35b8dce5c12581ae,9676540bf4fc3c9c] c.s.h.CustomAuthenticationSuccessHandler : Redirecting to saved url: / 2024-11-04 15:48:17.698 INFO 21062 --- [ocp-async-1,ac699d4b74c11812,7559a3b699efb514] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:17.698 WARN 21062 --- [ocp-async-5,c2748ace11867811,90d7cd0f2a5d8af2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:17.904 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f5fd0acdd627758f,96a6892ceb7a28d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.130.47.17, traceId=f5fd0acdd627758f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-11-04 15:48:17.904 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,f5fd0acdd627758f,96a6892ceb7a28d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.130.47.17, traceId=f5fd0acdd627758f, duration=6 ms] 2024-11-04 15:48:17.905 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,5fe6398630ee5196,46f2b1e0234f432e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=5fe6398630ee5196, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:48:17.905 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9da153b8f88fb2e9,9f18a46c0b73e1d1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=9da153b8f88fb2e9, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 15:48:17.905 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,e8edac553a976959,29ce1905bc11e513] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.130.47.17, traceId=e8edac553a976959, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-11-04 15:48:17.906 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,9da153b8f88fb2e9,9f18a46c0b73e1d1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=9da153b8f88fb2e9, duration=7 ms] 2024-11-04 15:48:17.908 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6342aaef03f7ab82,c9ccc665d65a3d02] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.130.47.17, traceId=6342aaef03f7ab82, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-11-04 15:48:17.910 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,e8edac553a976959,29ce1905bc11e513] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.130.47.17, traceId=e8edac553a976959, duration=10 ms] 2024-11-04 15:48:17.931 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,5fe6398630ee5196,46f2b1e0234f432e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=5fe6398630ee5196, duration=34 ms] 2024-11-04 15:48:17.983 INFO 21062 --- [prometheus-minute-schedule-1,7edaf10c825d23cc,87cd568b7f6bd4d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:18.067 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,49def276fe3dfb3f,5dfa746e0c26da8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.130.47.17, traceId=49def276fe3dfb3f, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-11-04 15:48:18.072 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,49def276fe3dfb3f,5dfa746e0c26da8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.130.47.17, traceId=49def276fe3dfb3f, duration=10 ms] 2024-11-04 15:48:18.084 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,6342aaef03f7ab82,c9ccc665d65a3d02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.130.47.17, traceId=6342aaef03f7ab82, duration=183 ms] 2024-11-04 15:48:18.586 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:18.587 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:18.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:48:18.590349+08:00, lastProcessTime=2024-11-04T15:48:08.589156+08:00 2024-11-04 15:48:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:48:18.590349+08:00 2024-11-04 15:48:18.824 INFO 21062 --- [ocp-async-9,922b75197260e352,e4e6144fefdf4f89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:48:19.223 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,1816fcbd292fd2a0,392bdfc8aea6b340] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=1816fcbd292fd2a0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 15:48:19.232 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,1816fcbd292fd2a0,392bdfc8aea6b340] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=1816fcbd292fd2a0, duration=15 ms] 2024-11-04 15:48:19.415 INFO 21062 --- [ocp-schedules-11,2ddb9e1925334be5,c88ac38f88adcab3] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:48:19.419 INFO 21062 --- [ocp-schedules-11,2ddb9e1925334be5,c88ac38f88adcab3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:19.419 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:48:19.698 INFO 21062 --- [ocp-async-6,92e94438b9cca652,65b1dbf15301a55f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:19.698 WARN 21062 --- [ocp-async-8,0aef799e816dd7cd,e7fe5988f4a7ccb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:19.899 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f9517283d52f2b80,f7141b1e2c469609] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.130.47.17, traceId=f9517283d52f2b80, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-11-04 15:48:19.899 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4107edf210358a8f,e019f1f7274fd8d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.130.47.17, traceId=4107edf210358a8f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-11-04 15:48:19.899 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,f9517283d52f2b80,f7141b1e2c469609] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.130.47.17, traceId=f9517283d52f2b80, duration=5 ms] 2024-11-04 15:48:19.899 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,4107edf210358a8f,e019f1f7274fd8d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.130.47.17, traceId=4107edf210358a8f, duration=6 ms] 2024-11-04 15:48:19.902 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,0864fe940970b9ed,86e12938877eb88f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.130.47.17, traceId=0864fe940970b9ed, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-11-04 15:48:19.902 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,0864fe940970b9ed,86e12938877eb88f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.130.47.17, traceId=0864fe940970b9ed, duration=4 ms] 2024-11-04 15:48:19.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 15:48:19.916 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.130.47.17, client=GET, traceId=] 2024-11-04 15:48:19.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 15:48:19.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.130.47.17, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-11-04 15:48:19.916 WARN 21062 --- [http-nio-0.0.0.0-8090-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.130.47.17, client=GET, traceId=] 2024-11-04 15:48:19.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.130.47.17, traceId=, duration=0 ms] 2024-11-04 15:48:20.983 INFO 21062 --- [prometheus-minute-schedule-1,e69688944d4012af,8c85bce634d801b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:21.456 INFO 21062 --- [ocp-schedules-19,4fedef309283668f,942d1ad4edad39fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:48:21.587 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:21.588 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:21.699 WARN 21062 --- [ocp-async-3,019b975bdf66cd00,da46928d3e3c1e4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:21.699 INFO 21062 --- [ocp-async-4,79a2943dc5f82f44,f69de5f7ccc292e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:22.074 INFO 21062 --- [ocp-schedules-29,9b22f5cfa52af2c5,4ef70f80bb1abda2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:48:23.621 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d7e6cb8f33b65b06,f6ef277afe64d176] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/downloadDiagnosis, client 10.130.47.17, traceId d7e6cb8f33b65b06] 2024-11-04 15:48:23.626 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d7e6cb8f33b65b06,f6ef277afe64d176] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/downloadDiagnosis?taskInstanceId=2, client=10.130.47.17, traceId=d7e6cb8f33b65b06, method=ResponseEntity com.oceanbase.ocp.server.common.controller.TaskController.downloadTaskDiagnosis(Long), args=2,] 2024-11-04 15:48:23.654 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,d7e6cb8f33b65b06,f6ef277afe64d176] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/downloadDiagnosis client=10.130.47.17, traceId=d7e6cb8f33b65b06, duration=37 ms] 2024-11-04 15:48:23.699 WARN 21062 --- [ocp-async-5,7adac5d4f76efdb8,8b8779f68c390dc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:23.699 INFO 21062 --- [ocp-async-7,6238faa6d6c8c357,981de8ff79aa02c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:23.983 INFO 21062 --- [prometheus-minute-schedule-1,fc84258c723a7278,d85f3065fa5319f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:24.588 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:24.590 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:25.699 WARN 21062 --- [ocp-async-9,2f9ec2bd2bd1fa15,2ab8ea4921c13f02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:25.699 INFO 21062 --- [ocp-async-9,06b418fa7bd8c6b8,aaf41d3116adcd4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:26.983 INFO 21062 --- [prometheus-minute-schedule-1,ee352f0a1d0e6cce,0b06aa68f5435770] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:26.984 INFO 21062 --- [prometheus-hour-schedule-1,34455d92e44419ba,03b4fca8db86bf3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:48:27.590 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:27.591 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:27.699 WARN 21062 --- [ocp-async-6,0dfe01bcb25bf689,54f1d173519ed2fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:27.699 INFO 21062 --- [ocp-async-6,9c587564f6e5cca6,c4672c944c422637] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:28.498 INFO 21062 --- [ocp-schedules-15,ade1e8f35bf7f51d,409b1f99b799c13e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:48:28.499 INFO 21062 --- [ocp-schedules-21,bed9c39102389313,e0a9459f2b43e9c3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:48:28.500 INFO 21062 --- [ocp-schedules-17,86cc8399a46a9ac5,3b70d106b683f5dd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:48:28.500 INFO 21062 --- [ocp-schedules-12,61680cd1e01f66f7,2cb9f8c75ca48eab] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:48:28.502 INFO 21062 --- [ocp-schedules-15,ade1e8f35bf7f51d,409b1f99b799c13e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:48:28.502 INFO 21062 --- [ocp-schedules-22,2f5dbfea91af04e4,f72f97b9f113b16e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:48:28.503 INFO 21062 --- [ocp-schedules-17,86cc8399a46a9ac5,3b70d106b683f5dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:48:28.504 INFO 21062 --- [ocp-schedules-12,61680cd1e01f66f7,2cb9f8c75ca48eab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:28.504 INFO 21062 --- [ocp-schedules-27,cc7c0621313b2d8f,e0c4d44a6d2f8b7e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:48:28.506 INFO 21062 --- [ocp-schedules-21,bed9c39102389313,e0a9459f2b43e9c3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:48:28.510 INFO 21062 --- [ocp-schedules-8,2271b85fd802b308,2c6523db578191ec] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:48:28.510 INFO 21062 --- [ocp-schedules-10,c07f1499951ed234,5b337019137ea042] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:48:28.510 INFO 21062 --- [ocp-schedules-5,4bd5bbb7190296da,2b5256bb4a5c949a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:48:28.510 INFO 21062 --- [ocp-schedules-5,4bd5bbb7190296da,2b5256bb4a5c949a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:48:28.510 INFO 21062 --- [ocp-schedules-8,2271b85fd802b308,2c6523db578191ec] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:48:28.510 INFO 21062 --- [ocp-schedules-10,c07f1499951ed234,5b337019137ea042] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:48:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:48:28.589241+08:00, lastProcessTime=2024-11-04T15:48:18.590349+08:00 2024-11-04 15:48:28.592 INFO 21062 --- [ocp-schedules-26,24c4cf6605d12081,372067af54f7e5c2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":421,"seriesIdKeySize":422,"secondWriteQueue":0,"minuteWriteCacheSize":419} 2024-11-04 15:48:28.592 INFO 21062 --- [ocp-schedules-26,24c4cf6605d12081,372067af54f7e5c2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=66, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=66, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:48:28.679 INFO 21062 --- [ocp-schedules-19,baf99ec9fe6eedca,2c243ce5f0fa42e4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:48:28.680 INFO 21062 --- [ocp-schedules-19,baf99ec9fe6eedca,2c243ce5f0fa42e4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:48:28.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:48:28.589241+08:00 2024-11-04 15:48:28.824 INFO 21062 --- [ocp-async-8,a251691dc4cbb63f,1b5917fd3835213f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:48:29.050 INFO 21062 --- [ocp-schedules-27,2cf42e89650f18a5,dd4a3fdc8e4d6497] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:48:29.420 INFO 21062 --- [ocp-schedules-11,2ddb9e1925334be5,c88ac38f88adcab3] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:48:29.493 INFO 21062 --- [ocp-schedules-8,16ce088129321ac2,3438524193d6736c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:48:29.493 INFO 21062 --- [ocp-schedules-8,16ce088129321ac2,3438524193d6736c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:48:29.649 INFO 21062 --- [ocp-schedules-25,73ce5df7784bae66,4c33269935a4ef5c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:48:29.662 INFO 21062 --- [ocp-schedules-20,886cecb955d7f7d8,cf8284df7b91741d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:48:29.668 INFO 21062 --- [ocp-schedules-20,886cecb955d7f7d8,cf8284df7b91741d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:48:29.668 INFO 21062 --- [ocp-schedules-20,886cecb955d7f7d8,cf8284df7b91741d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:48:29.699 WARN 21062 --- [ocp-async-8,27ce0b768297051b,f048c7f3b69558fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:29.699 INFO 21062 --- [ocp-async-7,8c6ff25c2ba49752,aa35633a0ebc40ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:29.983 INFO 21062 --- [prometheus-minute-schedule-1,b5564e447494074c,b6022a6b960b398c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:30.204 INFO 21062 --- [ocp-schedules-1,a9420deb2524032e,ac8efef1bf5dea62] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:48:30.204 INFO 21062 --- [ocp-schedules-1,a9420deb2524032e,ac8efef1bf5dea62] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:48:30.205 INFO 21062 --- [ocp-schedules-1,a9420deb2524032e,ac8efef1bf5dea62] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:48:30.205 INFO 21062 --- [ocp-schedules-1,a9420deb2524032e,ac8efef1bf5dea62] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:48:30.206 INFO 21062 --- [ocp-schedules-1,a9420deb2524032e,ac8efef1bf5dea62] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:48:30.206 INFO 21062 --- [ocp-schedules-1,a9420deb2524032e,ac8efef1bf5dea62] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709939979026190336 2024-11-04 15:48:30.207 INFO 21062 --- [ocp-schedules-1,a9420deb2524032e,ac8efef1bf5dea62] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=709957217611436032 2024-11-04 15:48:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:48:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:48:30.592 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:30.594 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:31.480 INFO 21062 --- [ocp-schedules-21,c6245c437a94e359,ab8fa19ae9bc5d78] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:48:31.700 WARN 21062 --- [ocp-async-3,bd82d7988eb3a595,33ade291d234f275] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:31.700 INFO 21062 --- [ocp-async-6,b5093dcdf179a4b7,003e244544560ac9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:32.077 INFO 21062 --- [ocp-schedules-16,ee15b053963b8950,e8423565394ed69f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:48:32.983 INFO 21062 --- [prometheus-minute-schedule-1,7126ff98ba0f66e3,efa8f07d9ce30512] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:33.594 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:33.595 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:33.596 INFO 21062 --- [ocp-schedules-11,50460f49cec31e3b,7e2e3acf84e2d09e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:48:33.602 INFO 21062 --- [ocp-schedules-11,50460f49cec31e3b,7e2e3acf84e2d09e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:48:33.602 INFO 21062 --- [ocp-schedules-11,50460f49cec31e3b,7e2e3acf84e2d09e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:48:33.700 INFO 21062 --- [ocp-async-10,625ce58ab831f417,a207aca5dc1e73fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:33.700 WARN 21062 --- [ocp-async-5,3383ab66c82826ac,9878a104db30f4d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:33.929 INFO 21062 --- [ocp-schedules-1,a555c13c49b03eef,b52f9f22f880a46f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:48:33.929 INFO 21062 --- [ocp-schedules-1,a555c13c49b03eef,b52f9f22f880a46f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:48:33.957 INFO 21062 --- [ocp-schedules-1,a555c13c49b03eef,b52f9f22f880a46f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:48:33.957 INFO 21062 --- [ocp-schedules-1,a555c13c49b03eef,b52f9f22f880a46f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:48:33.957 INFO 21062 --- [ocp-schedules-18,06d991e58cc44393,6133ccc58c80c682] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:48:34.134 INFO 21062 --- [ocp-schedules-2,e0b151bb6736e215,20ef07804f7c1f2a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:48:34.134+08:00, clusters=[1] 2024-11-04 15:48:34.137 INFO 21062 --- [ocp-schedules-2,e0b151bb6736e215,20ef07804f7c1f2a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:48:34.501 INFO 21062 --- [ocp-schedules-4,e0d064595bd074fb,8b2b8bd682949db8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:48:34.501+08:00, clusters=[1] 2024-11-04 15:48:34.504 INFO 21062 --- [ocp-schedules-4,e0d064595bd074fb,8b2b8bd682949db8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:48:34.611 INFO 21062 --- [ocp-schedules-10,f2ecfd427da1d5ed,8e8ddfdfa933770e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:48:34.614 INFO 21062 --- [ocp-schedules-10,f2ecfd427da1d5ed,8e8ddfdfa933770e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:34.672 ERROR 21062 --- [ocp-schedules-10,f2ecfd427da1d5ed,8e8ddfdfa933770e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:48:34.672 INFO 21062 --- [ocp-schedules-10,f2ecfd427da1d5ed,8e8ddfdfa933770e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:48:35.700 INFO 21062 --- [ocp-async-1,abbd19ce836e3793,6001d59478904b2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:35.700 WARN 21062 --- [ocp-async-8,d818e9bf07ea31ea,ff5c249854e91cb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:35.983 INFO 21062 --- [prometheus-minute-schedule-1,c708737bf75e3f36,231e0ae0331d892b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:36.595 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:36.596 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:36.639 INFO 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:48:36.640 INFO 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:48:36.640 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:48:36.640 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:48:36.649 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:48:36.650 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:48:36.650 INFO 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:48:36.651 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:48:36.651 WARN 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:48:36.651 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:48:36.652 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:48:36.652 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:48:36.652 INFO 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:48:36.653 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:48:36.653 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:48:36.653 WARN 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:48:36.660 INFO 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:48:36.660 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:48:36.660 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:48:36.671 ERROR 21062 --- [ocp-schedules-21,034dd4d13b4986e1,ecd4b002fafec480] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:48:37.700 WARN 21062 --- [ocp-async-3,8c4f6a91c6a6b5e0,9c0d7cf2f725c52e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:37.700 INFO 21062 --- [ocp-async-2,ab404883c063bd63,1c0e63539f1b9b22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:38.504 INFO 21062 --- [ocp-schedules-17,86cc8399a46a9ac5,3b70d106b683f5dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:48:38.505 INFO 21062 --- [ocp-schedules-12,61680cd1e01f66f7,2cb9f8c75ca48eab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:48:38.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:48:38.590737+08:00, lastProcessTime=2024-11-04T15:48:28.589241+08:00 2024-11-04 15:48:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:48:38.590737+08:00 2024-11-04 15:48:38.824 INFO 21062 --- [ocp-async-9,11d9c46a5268d9d1,1376ad6f0fb6c647] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:48:38.983 INFO 21062 --- [prometheus-minute-schedule-1,19d7a710cc9d37e8,efb40272f2f2f4d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:39.597 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:39.598 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:39.700 WARN 21062 --- [ocp-async-9,481253ba5bdee7df,e09c51919c95fc41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:39.700 INFO 21062 --- [ocp-async-8,52088ccca14bbc25,6034b1779b52c379] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:40.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:48:40.003 INFO 21062 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:48:41.517 INFO 21062 --- [ocp-schedules-21,53110facadccf103,0516c3a4169b2a95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:48:41.700 WARN 21062 --- [ocp-async-1,acbeacd4e430d0e7,02d0610a56f08da6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:41.700 INFO 21062 --- [ocp-async-1,7dd1bf90f81eb158,c6ad411e2fab9668] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:48:41.823 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:48:41.823 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:48:41.902 INFO 21062 --- [ocp-schedules-23,62a9a24c2fe86597,53a102d57118a756] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:48:41.902 INFO 21062 --- [ocp-schedules-23,cddc7ec4fc89aa1b,2312ae1cc48f0669] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:48:41.906 INFO 21062 --- [ocp-schedules-23,cddc7ec4fc89aa1b,2312ae1cc48f0669] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:41.933 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:48:41.933 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:48:41.983 INFO 21062 --- [prometheus-minute-schedule-1,be1308ffd91494c5,34d471b6a1878f5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:42.080 INFO 21062 --- [ocp-schedules-12,ffd999e50fc15ef0,be339b9ba13cda4f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:48:42.598 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:42.599 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:43.701 WARN 21062 --- [ocp-async-2,a0fa3e8b18c61048,67db64b962404437] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:43.701 INFO 21062 --- [ocp-async-6,aa8c13542ebcd95e,84274c2eac4ecfcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:44.983 INFO 21062 --- [prometheus-minute-schedule-1,b286407e50375323,66eb4fe6745a9bd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:45.600 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:45.601 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:45.701 WARN 21062 --- [ocp-async-9,af0d8b6143514ba0,2daffc5cd7b2b2f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:45.701 INFO 21062 --- [ocp-async-10,468f6911065c8425,cd7582a1733ae5ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:46.646 INFO 21062 --- [ocp-schedules-14,e39a0d2d3fa15f35,0445abca0f3226f9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:48:46.650 INFO 21062 --- [ocp-schedules-14,e39a0d2d3fa15f35,0445abca0f3226f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:46.736 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:48:46.740 ERROR 21062 --- [ocp-schedules-14,e39a0d2d3fa15f35,0445abca0f3226f9] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:48:46.740 INFO 21062 --- [ocp-schedules-14,e39a0d2d3fa15f35,0445abca0f3226f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:48:47.701 WARN 21062 --- [ocp-async-3,d3d50aaf7175dcfb,ba5db4620e489301] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:47.701 INFO 21062 --- [ocp-async-4,ee74616e0db27d1a,8fa94d24904dde3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:47.983 INFO 21062 --- [prometheus-minute-schedule-1,0a6f646afb524cc2,32705650632249b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:48:48.590598+08:00, lastProcessTime=2024-11-04T15:48:38.590737+08:00 2024-11-04 15:48:48.602 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:48.603 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:48.768 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:48:48.590598+08:00 2024-11-04 15:48:48.824 INFO 21062 --- [ocp-async-2,9b9e1f6d289a72bb,08ebce9484d9e65e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:48:49.701 WARN 21062 --- [ocp-async-7,384ffa86dadb2524,175c30f9f4401312] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:49.701 INFO 21062 --- [ocp-async-7,9ed97bd2e60576a8,475f63bca7fd8947] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:50.983 INFO 21062 --- [prometheus-minute-schedule-1,59c1a04706c9f491,abb345c2ec77b18a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:51.542 INFO 21062 --- [ocp-schedules-3,e890277adf5e8437,0d106943d0a29301] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:48:51.603 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:51.604 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:51.701 WARN 21062 --- [ocp-async-9,949b74e808edbdf7,34894eef3d3e13a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:51.702 INFO 21062 --- [ocp-async-5,f399bcd9e94eca8c,47ddcb7f53d0cf94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:52.083 INFO 21062 --- [ocp-schedules-19,72d683e48206b7e0,2600ab49518f071a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:48:53.605 INFO 21062 --- [ocp-schedules-24,3eda7d1aedaab74c,6eb9d313cae8a7c6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:48:53.611 INFO 21062 --- [ocp-schedules-24,3eda7d1aedaab74c,6eb9d313cae8a7c6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:48:53.611 INFO 21062 --- [ocp-schedules-24,3eda7d1aedaab74c,6eb9d313cae8a7c6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:48:53.701 WARN 21062 --- [ocp-async-8,b4cd8e6c23a1552c,428c7e5588ebb7a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:53.702 INFO 21062 --- [ocp-async-8,f4c2ad0a6a4d383d,8d4fee5a4a82d73e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:53.983 INFO 21062 --- [prometheus-minute-schedule-1,1a3feb25a031b884,2e6db2cbcd5eeefc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:54.604 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:54.605 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:55.702 WARN 21062 --- [ocp-async-6,124a18a6e7bad08a,dbccc7b294dd74bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:55.702 INFO 21062 --- [ocp-async-2,52696b832487fb0a,24fed83253359bf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:56.983 INFO 21062 --- [prometheus-minute-schedule-1,03f6b13790eb7cde,305711004b49ed38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:48:57.605 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:48:57.607 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:48:57.702 WARN 21062 --- [ocp-async-10,f6cbaf01394d8469,de6fcbce2987669d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:57.702 INFO 21062 --- [ocp-async-9,ad7fac092dc2526b,7c3f83a64f191cd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:58.500 INFO 21062 --- [ocp-schedules-18,b8bea0ee988775df,270af37d969ae99e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:48:58.500 INFO 21062 --- [ocp-schedules-2,0b8d8f9d45c7406c,3d11f4e36ef63a67] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:48:58.503 INFO 21062 --- [ocp-schedules-28,d16e3e0fe96ff496,c413a599a336e6c6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:48:58.504 INFO 21062 --- [ocp-schedules-18,b8bea0ee988775df,270af37d969ae99e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:48:58.504 INFO 21062 --- [ocp-schedules-2,0b8d8f9d45c7406c,3d11f4e36ef63a67] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:48:58.505 INFO 21062 --- [ocp-schedules-12,35cb8971f2e8176f,57d7527192f90206] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:48:58.509 INFO 21062 --- [ocp-schedules-4,f0947e977d7fdc43,164a224c49566039] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:48:58.510 INFO 21062 --- [ocp-schedules-4,f0947e977d7fdc43,164a224c49566039] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:48:58.510 INFO 21062 --- [ocp-schedules-20,10863adb63a72552,183e46e9a6ad1096] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:48:58.511 INFO 21062 --- [ocp-schedules-20,10863adb63a72552,183e46e9a6ad1096] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:48:58.511 INFO 21062 --- [ocp-schedules-27,1f3c25b14cb9865e,fc02c69ddb136536] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:48:58.511 INFO 21062 --- [ocp-schedules-27,1f3c25b14cb9865e,fc02c69ddb136536] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:48:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:48:58.588896+08:00, lastProcessTime=2024-11-04T15:48:48.590598+08:00 2024-11-04 15:48:58.758 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:48:58.588896+08:00 2024-11-04 15:48:58.825 INFO 21062 --- [ocp-async-3,47955bd1eb5f8814,02fc2be0f9a02983] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:48:59.652 INFO 21062 --- [ocp-schedules-5,f9d795c9b324fc1e,3bfe4ed9a1a4b2b6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:48:59.702 WARN 21062 --- [ocp-async-4,96100b39e4f89bf8,3e89440b239ed9c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:48:59.702 INFO 21062 --- [ocp-async-8,429967a0a33a1c05,041df61540fae4a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:48:59.983 INFO 21062 --- [prometheus-minute-schedule-1,2d401e6ffef035c2,40e7d9eac468be7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706540, interval=60 2024-11-04 15:49:00.607 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:00.608 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:01.586 INFO 21062 --- [ocp-schedules-22,3414c952d907651d,5ce63940a462b402] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:49:01.702 WARN 21062 --- [ocp-async-1,ee327c7feed37e7a,92b6b322c3b6e110] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:01.702 INFO 21062 --- [ocp-async-2,db335c67d238f093,857dce5f85321c57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:02.086 INFO 21062 --- [ocp-schedules-16,054e42f1d2980f0c,c90ed433d09d6a10] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:49:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1c80cff0aa1aba40,6d506336fef4a9cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:03.608 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:03.609 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:03.702 WARN 21062 --- [ocp-async-7,600e9d50d092a97a,535305fd63fcbb23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:03.702 INFO 21062 --- [ocp-async-9,c222d13900eba7b0,0b07e2af92d61f61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:05.703 WARN 21062 --- [ocp-async-3,47128fb095bbc942,1e934d5aa4809eac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:05.703 INFO 21062 --- [ocp-async-4,3933a165a946f566,bb839383c1ef94f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:05.983 INFO 21062 --- [prometheus-minute-schedule-1,b2e83606f140b326,e5852cfc5eb2724e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:06.609 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:06.610 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:07.703 WARN 21062 --- [ocp-async-6,0baf54c21512a096,8722782986070e68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:07.703 INFO 21062 --- [ocp-async-1,c34e8d4e0c5c9145,805b405c4253c912] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:08.504 INFO 21062 --- [ocp-schedules-18,b8bea0ee988775df,270af37d969ae99e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:49:08.504 INFO 21062 --- [ocp-schedules-2,0b8d8f9d45c7406c,3d11f4e36ef63a67] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:49:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:49:08.590345+08:00, lastProcessTime=2024-11-04T15:48:58.588896+08:00 2024-11-04 15:49:08.765 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:49:08.590345+08:00 2024-11-04 15:49:08.825 INFO 21062 --- [ocp-async-7,8858f0eb2b01215b,0bef16920961e4cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:49:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2302b4db36a23041,1cbe4fda4bbc4d15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:09.611 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:09.613 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:09.703 WARN 21062 --- [ocp-async-10,33b5e354bbec7288,26d93dd265757c1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:09.703 INFO 21062 --- [ocp-async-10,ad2cb1d3f47474f2,70706ce4241d6d29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:10.002 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:49:10.003 INFO 21062 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:49:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:49:10.299 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:49:10.299 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:49:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:49:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:49:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:49:11.613 INFO 21062 --- [ocp-schedules-12,2c5423eb01c9c52e,9d46e127a343faf1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:49:11.703 WARN 21062 --- [ocp-async-3,6f2ddde8408a1041,be4f0eddcf1b886d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:11.703 INFO 21062 --- [ocp-async-4,84d61c7c6b88e34f,409a973bf9103ec9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:11.934 INFO 21062 --- [ocp-schedules-23,cddc7ec4fc89aa1b,2312ae1cc48f0669] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:49:11.934 INFO 21062 --- [ocp-schedules-23,b5b472741753c5dc,d1f96a31fea6c2a3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:49:11.938 INFO 21062 --- [ocp-schedules-23,b5b472741753c5dc,d1f96a31fea6c2a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:11.946 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706480, nodeCount=71, affectRows=71 2024-11-04 15:49:11.946 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706480, nodeCount=66, affectRows=66 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706480, nodeCount=88, affectRows=88 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706480, nodeCount=71, validCount=71 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706480, nodeCount=74, affectRows=74 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706480, nodeCount=66, validCount=66 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706480, nodeCount=125, affectRows=125 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706480, nodeCount=88, validCount=88 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706480, nodeCount=74, validCount=74 2024-11-04 15:49:11.947 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706480, nodeCount=122, validCount=122 2024-11-04 15:49:11.966 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:49:11.966 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:49:11.983 INFO 21062 --- [prometheus-minute-schedule-1,03259440df117a20,4659c513b26c0b91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:12.089 INFO 21062 --- [ocp-schedules-29,cebc45564a2cbc50,caa579c7bacc03a4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:49:12.613 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:12.615 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:13.614 INFO 21062 --- [ocp-schedules-29,13685d8732dde7a3,09ae22fd92a93236] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:49:13.619 INFO 21062 --- [ocp-schedules-29,13685d8732dde7a3,09ae22fd92a93236] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:49:13.619 INFO 21062 --- [ocp-schedules-29,13685d8732dde7a3,09ae22fd92a93236] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:49:13.703 WARN 21062 --- [ocp-async-8,1070cfbb8f90513c,4254f20bd49f67d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:13.703 INFO 21062 --- [ocp-async-8,e5d61979bbe5e232,845cb910de5fd56a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:13.984 INFO 21062 --- [prometheus-cache-monitor1,0604fe718af3bc18,845dfc8662149ca1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1690,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:49:13.984 INFO 21062 --- [prometheus-cache-monitor1,0604fe718af3bc18,845dfc8662149ca1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:49:14.983 INFO 21062 --- [prometheus-minute-schedule-1,a469457e30c303f0,557af7c40a0daca0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:15.615 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:15.616 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:15.703 WARN 21062 --- [ocp-async-7,6bfa49907685565d,2a68208f5eae0a38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:15.703 INFO 21062 --- [ocp-async-9,cf2cd06f1e74bd38,1aa5683ab573f418] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:15.835 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:45:15.813, end=2024-11-04T15:47:15.832} 2024-11-04 15:49:15.839 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:49:15.839 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:49:17.704 INFO 21062 --- [ocp-async-10,d0dc518e2ab21903,ce6a6414ee0a221e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:17.704 WARN 21062 --- [ocp-async-5,572ef47b4fd3b554,8e39ba0507a409be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9948a7b5a018f473,a7ad8d4ce3dfd625] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:18.592 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:49:18.591621+08:00, lastProcessTime=2024-11-04T15:49:08.590345+08:00 2024-11-04 15:49:18.616 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:18.617 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:18.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:49:18.591621+08:00 2024-11-04 15:49:18.825 INFO 21062 --- [ocp-async-1,4dc72d1edec8317c,b5d5fd603a7fbc4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:49:19.704 INFO 21062 --- [ocp-async-1,874fbfecd8c9bb81,589d1c6e999d7b8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:19.704 WARN 21062 --- [ocp-async-6,fec32e9841a36267,2ef473e94c9bc6bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:20.983 INFO 21062 --- [prometheus-minute-schedule-1,dee0cc059c693e83,4cd3156cb526808c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:21.617 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:21.618 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:21.657 INFO 21062 --- [ocp-schedules-3,b1aa13dcf6eae607,b36085877e4e4a4f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:49:21.704 WARN 21062 --- [ocp-async-8,6bf6c3f10261c5da,b382dd271a2294f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:21.704 INFO 21062 --- [ocp-async-7,b589e79622f09713,fc0579cbd0fe93a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:22.091 INFO 21062 --- [ocp-schedules-20,c8c3ac505fa0dd21,9604fb6589f67ce7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:49:23.704 WARN 21062 --- [ocp-async-3,60a1a37d615d596c,43f3614b4c2618c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:23.704 INFO 21062 --- [ocp-async-5,fb1b964e17cbda33,3a27984b93c911db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:23.983 INFO 21062 --- [prometheus-minute-schedule-1,c7bf6711a8367277,2b4ef665b5dbfa3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:24.618 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:24.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:25.704 WARN 21062 --- [ocp-async-4,3190b46b2b168c45,15e7e309891b814d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:25.704 INFO 21062 --- [ocp-async-6,8267f66056ffe68d,24e05cf34067c368] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:26.983 INFO 21062 --- [prometheus-minute-schedule-1,b69ecfadae3fccef,0ecdb42b2c068a3b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:26.984 INFO 21062 --- [prometheus-hour-schedule-1,34fc92c33cc7bbe9,b000253404894555] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:49:27.620 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:27.621 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:27.705 INFO 21062 --- [ocp-async-1,0a6c2661df18b181,36dd4519a58a11eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:27.705 WARN 21062 --- [ocp-async-8,182d6bb670a6a2b3,493b1741bec2cf66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:28.498 INFO 21062 --- [ocp-schedules-24,5c4a7183e559880b,64c9ae597b5c8581] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:49:28.499 INFO 21062 --- [ocp-schedules-9,b182453571f0c1a1,2c65ad9efc42a4c3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:49:28.500 INFO 21062 --- [ocp-schedules-20,1827f92186e46d93,1c69078a2d5a8f09] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:49:28.500 INFO 21062 --- [ocp-schedules-27,c24e924c60414c5d,a3cc6bbb687e6b1f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:49:28.502 INFO 21062 --- [ocp-schedules-29,fcc2a2e7433efc77,8076bbef30c1d36c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:49:28.503 INFO 21062 --- [ocp-schedules-24,5c4a7183e559880b,64c9ae597b5c8581] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:49:28.503 INFO 21062 --- [ocp-schedules-20,1827f92186e46d93,1c69078a2d5a8f09] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:28.504 INFO 21062 --- [ocp-schedules-27,c24e924c60414c5d,a3cc6bbb687e6b1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:49:28.504 INFO 21062 --- [ocp-schedules-7,43ad492bb95309c7,e7e4257f37905861] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:49:28.505 INFO 21062 --- [ocp-schedules-9,b182453571f0c1a1,2c65ad9efc42a4c3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:49:28.511 INFO 21062 --- [ocp-schedules-10,1a05dcd520bb2db9,0c7a2a1ba3939116] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:49:28.511 INFO 21062 --- [ocp-schedules-10,1a05dcd520bb2db9,0c7a2a1ba3939116] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:49:28.511 INFO 21062 --- [ocp-schedules-11,289db8f94c8ff951,28bf04264130ffe6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:49:28.511 INFO 21062 --- [ocp-schedules-11,289db8f94c8ff951,28bf04264130ffe6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:49:28.511 INFO 21062 --- [ocp-schedules-14,eb8b178ecb6f8c7d,caf9a58fe357cbcf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:49:28.511 INFO 21062 --- [ocp-schedules-14,eb8b178ecb6f8c7d,caf9a58fe357cbcf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:49:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:49:28.589026+08:00, lastProcessTime=2024-11-04T15:49:18.591621+08:00 2024-11-04 15:49:28.633 INFO 21062 --- [ocp-schedules-21,7f8661d0eb14a38d,60b8927403bbf983] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:49:28.633+08:00, tenantSize=0 2024-11-04 15:49:28.804 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:49:28.589026+08:00 2024-11-04 15:49:28.825 INFO 21062 --- [ocp-async-5,b15e1439428a734b,a2284ef55c134105] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:49:29.053 INFO 21062 --- [ocp-schedules-9,8782a0441ab0c368,910745ab8c1de788] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:49:29.420 INFO 21062 --- [ocp-schedules-14,ec3b0c198a45e47d,d4aaf956bde9239a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:49:29.424 INFO 21062 --- [ocp-schedules-14,ec3b0c198a45e47d,d4aaf956bde9239a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:29.424 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:49:29.497 INFO 21062 --- [ocp-schedules-25,6331624deb6e71bb,00247d143551708a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:49:29.498 INFO 21062 --- [ocp-schedules-25,6331624deb6e71bb,00247d143551708a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:49:29.656 INFO 21062 --- [ocp-schedules-19,b78886081721bea4,f574197017d84cd8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:49:29.668 INFO 21062 --- [ocp-schedules-21,570ae52d7ab2739f,766ea637ad2126a1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:49:29.674 INFO 21062 --- [ocp-schedules-21,570ae52d7ab2739f,766ea637ad2126a1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:49:29.674 INFO 21062 --- [ocp-schedules-21,570ae52d7ab2739f,766ea637ad2126a1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:49:29.705 INFO 21062 --- [ocp-async-5,591440aa55ac6340,254464ecaa91ec79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:29.705 WARN 21062 --- [ocp-async-10,54e30bb31890152f,9cf7e070652c49d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:29.983 INFO 21062 --- [prometheus-minute-schedule-1,d58b3549b6272d14,025795df00e9c45f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:30.024 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:49:30.024 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:49:30.032 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:49:30.032 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:49:30.032 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:49:30.042 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:49:30.042 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:49:30.042 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:49:30.045 INFO 21062 --- [ocp-schedules-5,fa8de451dfcb2faa,e8029c03f501aa32] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:49:30.207 INFO 21062 --- [ocp-schedules-16,657cd9233abfc873,a4dcd4ce72624abd] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:49:30.207 INFO 21062 --- [ocp-schedules-16,657cd9233abfc873,a4dcd4ce72624abd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:49:30.208 INFO 21062 --- [ocp-schedules-16,657cd9233abfc873,a4dcd4ce72624abd] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:49:30.208 INFO 21062 --- [ocp-schedules-16,657cd9233abfc873,a4dcd4ce72624abd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:49:30.209 INFO 21062 --- [ocp-schedules-16,657cd9233abfc873,a4dcd4ce72624abd] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:49:30.209 INFO 21062 --- [ocp-schedules-16,657cd9233abfc873,a4dcd4ce72624abd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:49:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:49:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:49:30.622 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:30.623 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:31.683 INFO 21062 --- [ocp-schedules-13,7d2efac87a8e5571,7d5101eba90f6ac9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:49:31.705 INFO 21062 --- [ocp-async-8,cff0597dfc0711b8,5253f17926cf4a68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:31.705 WARN 21062 --- [ocp-async-1,41d3c2c6693eb03d,031ee4029815bef9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:32.094 INFO 21062 --- [ocp-schedules-18,fd34b61b800e26f8,9c31fb2eb1661174] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:49:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b20302bb7880a921,315644f53e9ca7a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:33.621 INFO 21062 --- [ocp-schedules-13,5a3907753b2907ff,ccfd39aff27a0adb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:49:33.623 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:33.625 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:33.627 INFO 21062 --- [ocp-schedules-13,5a3907753b2907ff,ccfd39aff27a0adb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:49:33.627 INFO 21062 --- [ocp-schedules-13,5a3907753b2907ff,ccfd39aff27a0adb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:49:33.705 INFO 21062 --- [ocp-async-9,cef7bd6843b8c10f,bb1637ace8442a1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:33.705 WARN 21062 --- [ocp-async-2,cebf85a7c727e1cf,1e3e78da59abbba7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:33.957 INFO 21062 --- [ocp-schedules-9,49db67b17eff8f30,a01df6de0d4f76ef] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:49:33.957 INFO 21062 --- [ocp-schedules-9,49db67b17eff8f30,a01df6de0d4f76ef] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:49:33.984 INFO 21062 --- [ocp-schedules-9,49db67b17eff8f30,a01df6de0d4f76ef] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:49:33.984 INFO 21062 --- [ocp-schedules-9,49db67b17eff8f30,a01df6de0d4f76ef] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:49:33.984 INFO 21062 --- [ocp-schedules-17,0f1a03756847ebc9,f42e5554421d57ec] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:49:34.163 INFO 21062 --- [ocp-schedules-18,fd0df8d94ef77db5,9b20b81b9570c11f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:49:34.163+08:00, clusters=[1] 2024-11-04 15:49:34.164 INFO 21062 --- [ocp-schedules-18,fd0df8d94ef77db5,9b20b81b9570c11f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:49:34.531 INFO 21062 --- [ocp-schedules-28,e6b1f5196d51dfad,b8fde866049c268d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:49:34.531+08:00, clusters=[1] 2024-11-04 15:49:34.535 INFO 21062 --- [ocp-schedules-28,e6b1f5196d51dfad,b8fde866049c268d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:49:35.705 INFO 21062 --- [ocp-async-10,de0ac75eeced3f56,07aad3f6afda8ecc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:35.705 WARN 21062 --- [ocp-async-4,27d7f9c4b7caef9d,dd6611baf1f79359] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:35.983 INFO 21062 --- [prometheus-minute-schedule-1,0ac6d3f762e58714,03080a17c0167c18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:36.625 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:36.626 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:36.681 INFO 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:49:36.681 INFO 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:49:36.681 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:49:36.681 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:49:36.691 INFO 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:49:36.691 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:49:36.691 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:49:36.693 WARN 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:49:36.693 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:49:36.693 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:49:36.693 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:49:36.693 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:49:36.693 INFO 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:49:36.694 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:49:36.694 WARN 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:49:36.695 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:49:36.701 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:49:36.701 INFO 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:49:36.701 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:49:36.712 ERROR 21062 --- [ocp-schedules-28,027d02a77f105fbf,0831a0e13891d0e4] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:49:37.705 INFO 21062 --- [ocp-async-6,089ce6510c96c061,7028a4959011fda8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:37.705 WARN 21062 --- [ocp-async-1,c91e6960df5dd43e,cc18e7f37943c356] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:38.504 INFO 21062 --- [ocp-schedules-27,c24e924c60414c5d,a3cc6bbb687e6b1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:49:38.504 INFO 21062 --- [ocp-schedules-20,1827f92186e46d93,1c69078a2d5a8f09] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:49:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:49:38.588679+08:00, lastProcessTime=2024-11-04T15:49:28.589026+08:00 2024-11-04 15:49:38.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:49:38.588679+08:00 2024-11-04 15:49:38.825 INFO 21062 --- [ocp-async-2,f23f9760b46f3db3,03443ea28bed1b86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:49:38.983 INFO 21062 --- [prometheus-minute-schedule-1,d554b8ef8ebde324,fa290ecb0956ce73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:39.010 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:49:39.010 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:49:39.017 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:49:39.017 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:49:39.017 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:49:39.018 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:49:39.022 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:49:39.022 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:49:39.022 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:49:39.023 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:49:39.059 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:49:39.067 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:49:39.068 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:49:39.068 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:49:39.068 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:49:39.068 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:49:39.076 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:49:39.425 INFO 21062 --- [ocp-schedules-14,ec3b0c198a45e47d,d4aaf956bde9239a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:49:39.604 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:49:39.626 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:39.628 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:39.706 INFO 21062 --- [ocp-async-9,a9ac554954f5e7cd,2d9a77df4d0616e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:39.706 WARN 21062 --- [ocp-async-4,2a644de439be0061,f8af49a109035f4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:40.003 INFO 21062 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:49:40.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:49:40.179 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.156342168S 2024-11-04 15:49:40.179 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:49:40.286 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:49:40.286 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:49:40.313 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:49:40.373 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:49:40.373 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:49:40.373 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:49:40.373 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:49:40.373 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:49:40.374 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:49:40.384 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:49:40.557 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] o.s.boot.SpringApplication : Started application in 1.539 seconds (JVM running for 11994.236) 2024-11-04 15:49:40.557 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.539926423S 2024-11-04 15:49:41.017 INFO 21062 --- [ocp-schedules-3,46d4b23d174f1409,93d57822f7aa9611] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:49:41.706 INFO 21062 --- [ocp-async-10,61c947bcd825fead,63db232191c04f75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:41.706 WARN 21062 --- [ocp-async-1,097a3a01b3f7b474,8f470e5b2365be4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:41.723 INFO 21062 --- [ocp-schedules-14,1b48a1a30dbd1eda,c2a6a04981ce8429] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:49:41.967 INFO 21062 --- [ocp-schedules-23,b5b472741753c5dc,d1f96a31fea6c2a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:49:41.967 INFO 21062 --- [ocp-schedules-23,387dc6dfddaded18,53fa6c5c18818692] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:49:41.970 INFO 21062 --- [ocp-schedules-23,387dc6dfddaded18,53fa6c5c18818692] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d369c715f62fc8f1,44853af9dc0d3af0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:41.997 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:49:41.997 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:49:42.097 INFO 21062 --- [ocp-schedules-1,cdd132e0045ad61a,a758833f88e624b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:49:42.628 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:42.629 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:43.706 WARN 21062 --- [ocp-async-8,3c70eca7a0a70186,52ea3471198b1cf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:43.706 INFO 21062 --- [ocp-async-6,277ab11b8a03e400,0b3719f14b913a26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:44.983 INFO 21062 --- [prometheus-minute-schedule-1,e53a13b5cdf33c4d,5d4d0e4ee49b87cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:45.630 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:45.631 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:45.706 INFO 21062 --- [ocp-async-4,39fc9744251bd4e7,61f4cac4f9d48a28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:45.706 WARN 21062 --- [ocp-async-2,85df367198ff100b,503814383033ba29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:46.740 INFO 21062 --- [ocp-schedules-7,7b08d974b6d5a0b1,8d693a537739040f] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:49:46.744 INFO 21062 --- [ocp-schedules-7,7b08d974b6d5a0b1,8d693a537739040f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:46.826 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:49:46.828 ERROR 21062 --- [ocp-schedules-7,7b08d974b6d5a0b1,8d693a537739040f] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:49:46.828 INFO 21062 --- [ocp-schedules-7,7b08d974b6d5a0b1,8d693a537739040f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:49:47.706 WARN 21062 --- [ocp-async-1,a753e62247f3cf3b,fb3880c3ccb9a197] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:47.707 INFO 21062 --- [ocp-async-7,170edf525c0b586b,a0b78502231a1157] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:47.983 INFO 21062 --- [prometheus-minute-schedule-1,67ba69f1c233f729,aebe583e33e542e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:48.595 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:49:48.594744+08:00, lastProcessTime=2024-11-04T15:49:38.588679+08:00 2024-11-04 15:49:48.631 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:48.633 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:48.756 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:49:48.594744+08:00 2024-11-04 15:49:48.825 INFO 21062 --- [ocp-async-8,926794abe4c0749b,975d2e69d4dd5188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:49:49.707 INFO 21062 --- [ocp-async-6,ca7bd5c4395239a9,ade2b12e21b8aa24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:49.707 WARN 21062 --- [ocp-async-5,446410892d59200b,23b73793c05fec2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:50.983 INFO 21062 --- [prometheus-minute-schedule-1,d64fa94c947bd7b3,86ec059ea5e854e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:51.633 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:51.634 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:51.707 WARN 21062 --- [ocp-async-4,bc4a42735a06fc27,02e7aca957821fbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:51.707 INFO 21062 --- [ocp-async-3,893781420639bdbc,27377d27c6a80594] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:51.750 INFO 21062 --- [ocp-schedules-3,527eeafb93262dc8,958cc8acb0bdc973] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:49:52.100 INFO 21062 --- [ocp-schedules-4,337adbf3efb98c2d,06336464e4874470] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:49:53.630 INFO 21062 --- [ocp-schedules-4,c00550389a8a610f,2ae2b18c652c90b8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:49:53.636 INFO 21062 --- [ocp-schedules-4,c00550389a8a610f,2ae2b18c652c90b8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:49:53.636 INFO 21062 --- [ocp-schedules-4,c00550389a8a610f,2ae2b18c652c90b8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:49:53.707 INFO 21062 --- [ocp-async-9,5e11721dc72f9ba5,ba979db9bb31d9d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:53.707 WARN 21062 --- [ocp-async-7,79ec084b428fd46f,b7335faab3436b78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:53.983 INFO 21062 --- [prometheus-minute-schedule-1,31deb50d30d4ac3c,38acc18eb0b301db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:54.634 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:54.636 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:55.707 WARN 21062 --- [ocp-async-8,cb041f64466fb3aa,267dfec90f244c09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:55.707 INFO 21062 --- [ocp-async-5,6314003998ed9cb3,f6562586f42873d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:56.983 INFO 21062 --- [prometheus-minute-schedule-1,d751e5c41cef70f4,1393fa04fcfce756] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:49:57.636 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:49:57.637 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:49:57.707 INFO 21062 --- [ocp-async-4,cf420915eb72a170,c85d9ea5ca8d1485] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:57.707 WARN 21062 --- [ocp-async-2,b3b4edc6212c0e84,f393c8b7e39d5116] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:58.500 INFO 21062 --- [ocp-schedules-28,20f500a894a495be,4333e20ffa5ada24] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:49:58.500 INFO 21062 --- [ocp-schedules-8,dabe1b4e17255979,2db5dfb7fd61dbbf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:49:58.502 INFO 21062 --- [ocp-schedules-14,7ae7708324cfe409,9b6f9a9c52f37b30] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:49:58.503 INFO 21062 --- [ocp-schedules-28,20f500a894a495be,4333e20ffa5ada24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:49:58.503 INFO 21062 --- [ocp-schedules-8,dabe1b4e17255979,2db5dfb7fd61dbbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:49:58.505 INFO 21062 --- [ocp-schedules-11,051ba10dd00cc9ea,18a3c7d580aa3b7a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:49:58.510 INFO 21062 --- [ocp-schedules-20,59ec9fd5d466dad3,34ef022222fddb97] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:49:58.510 INFO 21062 --- [ocp-schedules-20,59ec9fd5d466dad3,34ef022222fddb97] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:49:58.510 INFO 21062 --- [ocp-schedules-9,fd9cf2d93139f539,01c578d8f72ef937] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:49:58.510 INFO 21062 --- [ocp-schedules-12,a29f9e9a4ae31d32,4bcbbdcdac71ca3e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:49:58.510 INFO 21062 --- [ocp-schedules-9,fd9cf2d93139f539,01c578d8f72ef937] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:49:58.510 INFO 21062 --- [ocp-schedules-12,a29f9e9a4ae31d32,4bcbbdcdac71ca3e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:49:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:49:58.590247+08:00, lastProcessTime=2024-11-04T15:49:48.594744+08:00 2024-11-04 15:49:58.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:49:58.590247+08:00 2024-11-04 15:49:58.826 INFO 21062 --- [ocp-async-7,1fe26e0c2cc49d58,86a48047ea93e3e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:49:59.660 INFO 21062 --- [ocp-schedules-27,2d9d6118bd2695d9,91d658c5f97b246e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:49:59.708 WARN 21062 --- [ocp-async-9,915aac51c8fe0b53,b4e29c7903f51865] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:49:59.708 INFO 21062 --- [ocp-async-1,37b30b18c787b8b9,e0a2ac74e982cb6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:49:59.983 INFO 21062 --- [prometheus-minute-schedule-1,76960ef511883d73,4d1cd4ab44a1f9ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706600, interval=60 2024-11-04 15:50:00.483 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,b3e09f54fc441a84] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:45:01.0, nextRun=2024-11-04 15:50:00.0, currentDate=Mon Nov 04 15:50:00 CST 2024 2024-11-04 15:50:00.483 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,b3e09f54fc441a84] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:45:01.0, nextRunTime=2024-11-04 15:50:00.0} 2024-11-04 15:50:00.509 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,b3e09f54fc441a84] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:50:00.541 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,b3e09f54fc441a84] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:50:00.541 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,b3e09f54fc441a84] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:50:00.637 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:00.638 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:01.708 INFO 21062 --- [ocp-async-10,f6462971d81938e7,f296f890ad3aaebd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:01.708 WARN 21062 --- [ocp-async-5,abad610a1e4c25a6,b8409c1318ac8473] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:01.792 INFO 21062 --- [ocp-schedules-20,9b5b0af319082b8b,919f418c0f8cbd45] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:50:02.102 INFO 21062 --- [ocp-schedules-30,c51be75e54ef4d5e,cf897360bc3ba3fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:50:02.983 INFO 21062 --- [prometheus-minute-schedule-1,1ca633b0a9577a2a,df44f890c9bd508e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:03.638 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:03.640 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:03.708 INFO 21062 --- [ocp-async-6,cb7b39de2cba511f,1c34257dd90ebf6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:03.708 WARN 21062 --- [ocp-async-4,173bddd5f7180a23,a02414a47fb23c0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:04.612 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3d7b021c6c54e70d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:40:05.0, nextRun=2024-11-04 15:50:04.0, currentDate=Mon Nov 04 15:50:04 CST 2024 2024-11-04 15:50:04.612 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3d7b021c6c54e70d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-11-04 15:40:05.0, nextRunTime=2024-11-04 15:50:04.0} 2024-11-04 15:50:04.619 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3d7b021c6c54e70d] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-11-04 15:50:04.619 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,3d7b021c6c54e70d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-11-04T15:50:04.619+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='dd193f8f67a5a423', operation=EXECUTE} 2024-11-04 15:50:05.080 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,d944ae36f91ccb4f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 90, operation:EXECUTE, state: RUNNING 2024-11-04 15:50:05.101 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=90, context=Context{parallelIdx=-1, stringMap={task_instance_id=27, task_operation=execute, latest_execution_start_time=2024-11-04T15:50:05.071+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=90}, listMap={}}, executor=10.46.128.24 2024-11-04 15:50:05.106 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:50:05.106 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:50:05.107 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.108 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:50:05.109 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.109 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.110 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.110 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:50:05.125 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.126 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.126 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.126 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:50:05.139 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.140 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.141 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.141 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:50:05.155 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.156 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.156 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.156 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:50:05.169 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.169 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.170 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.170 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:50:05.176 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.176 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.177 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.177 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:50:05.182 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.183 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.183 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.183 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:50:05.187 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.188 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.189 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.189 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:50:05.195 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.195 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.196 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.196 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:50:05.202 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.202 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.203 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.203 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:50:05.208 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.209 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.209 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.209 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:50:05.218 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.218 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.219 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.219 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:50:05.232 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.233 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.234 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.234 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:50:05.239 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.239 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.240 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.240 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:50:05.246 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.246 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.246 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.247 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:50:05.252 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.252 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.253 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.253 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:50:05.257 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.258 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.258 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.258 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:50:05.263 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.263 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.264 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.264 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:50:05.269 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.269 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.269 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.270 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:50:05.275 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.275 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.276 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.276 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:50:05.280 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.280 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.281 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.281 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:50:05.286 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.287 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.287 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.288 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:50:05.291 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.292 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.292 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.293 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:50:05.297 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.298 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.298 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.299 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:50:05.303 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.303 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.304 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.304 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:50:05.308 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.308 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.309 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.309 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:50:05.314 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.314 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.315 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.315 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:50:05.321 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.321 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.322 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.322 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:50:05.327 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.327 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.328 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.329 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:50:05.332 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.333 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.333 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.334 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:50:05.338 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.338 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.339 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.339 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:50:05.344 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.344 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.345 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.345 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:50:05.350 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.351 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.351 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.352 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:50:05.356 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.356 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.357 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.357 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:50:05.364 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.365 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.365 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.366 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:50:05.373 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.373 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.374 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.374 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:50:05.379 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.380 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.380 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.381 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:50:05.386 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.386 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.387 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.387 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:50:05.391 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.392 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.392 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.393 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:50:05.396 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.396 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.397 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.397 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:50:05.401 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.402 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.402 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.402 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:50:05.406 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.406 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.407 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.407 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:50:05.416 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.417 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.417 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.418 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:50:05.438 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.439 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.439 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.440 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:50:05.446 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.447 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.447 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.448 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:50:05.453 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.454 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.454 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.455 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:50:05.460 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.461 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.461 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.462 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:50:05.466 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.466 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.467 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.467 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:50:05.471 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.471 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.472 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.472 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:50:05.477 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:50:05.477 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:50:05.478 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.478 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:50:05.480 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.480 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.481 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.481 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:50:05.487 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.487 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:50:05.488 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:05.488 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:50:05.492 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:50:05.501 INFO 21062 --- [subtask-executor23,dd193f8f67a5a423,b969f14630266bb4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 90, operation:EXECUTE, state: SUCCESSFUL 2024-11-04 15:50:05.708 INFO 21062 --- [ocp-async-9,307780917809453f,92aab2e38bf75a27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:05.708 WARN 21062 --- [ocp-async-7,dad02c1f44d8ad54,bed75c4d81ba8b42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:05.755 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:27, status:SUCCESS 2024-11-04 15:50:05.756 INFO 21062 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:27, status:SUCCESS 2024-11-04 15:50:05.983 INFO 21062 --- [prometheus-minute-schedule-1,08c77d4d42ccaf50,e709fb0b0c92e8be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:06.640 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:06.641 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:07.708 INFO 21062 --- [ocp-async-5,1a9ad8eeb427394c,183972908261e596] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:07.708 WARN 21062 --- [ocp-async-1,2bdac856df466b5e,4f0339a2ad072dd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:08.504 INFO 21062 --- [ocp-schedules-28,20f500a894a495be,4333e20ffa5ada24] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:50:08.504 INFO 21062 --- [ocp-schedules-8,dabe1b4e17255979,2db5dfb7fd61dbbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:50:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:50:08.589520+08:00, lastProcessTime=2024-11-04T15:49:58.590247+08:00 2024-11-04 15:50:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:50:08.589520+08:00 2024-11-04 15:50:08.826 INFO 21062 --- [ocp-async-4,20f840b2040fb525,c6b6a12707c3a2fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:50:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d9bb22090876f30a,d92b774f67e0e7b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:09.642 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:09.643 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:09.708 INFO 21062 --- [ocp-async-2,0c4492aabe240a29,692e003c273b0dd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:09.708 WARN 21062 --- [ocp-async-4,a65a24c4fe107800,45407e6b9b50381b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:10.003 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:50:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:50:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:50:10.304 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:50:10.304 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:50:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:50:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:50:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:50:11.708 INFO 21062 --- [ocp-async-3,376684f6b62bcaee,3ab8f66f0dacccfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:11.709 WARN 21062 --- [ocp-async-9,cb3d1fa03fe53be1,358855c80332f4e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:11.818 INFO 21062 --- [ocp-schedules-12,edb0cdb7ff761445,4fe94a93d8e899d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:50:11.954 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=2, affectRows=2 2024-11-04 15:50:11.954 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=2, validCount=2 2024-11-04 15:50:11.955 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=7, affectRows=7 2024-11-04 15:50:11.955 INFO 21062 --- [metric-archive-second-53,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=6, validCount=6 2024-11-04 15:50:11.955 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=16, affectRows=16 2024-11-04 15:50:11.955 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=16, validCount=16 2024-11-04 15:50:11.955 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=47, affectRows=47 2024-11-04 15:50:11.956 INFO 21062 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=47, validCount=47 2024-11-04 15:50:11.956 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=41, affectRows=41 2024-11-04 15:50:11.956 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=22, affectRows=22 2024-11-04 15:50:11.956 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=32, affectRows=32 2024-11-04 15:50:11.956 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=22, validCount=22 2024-11-04 15:50:11.956 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=41, validCount=41 2024-11-04 15:50:11.956 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=32, validCount=32 2024-11-04 15:50:11.957 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=60, affectRows=60 2024-11-04 15:50:11.957 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=53, affectRows=53 2024-11-04 15:50:11.957 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=60, validCount=60 2024-11-04 15:50:11.957 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=62, affectRows=62 2024-11-04 15:50:11.957 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=53, validCount=53 2024-11-04 15:50:11.958 INFO 21062 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=62, validCount=62 2024-11-04 15:50:11.958 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706540, nodeCount=82, affectRows=82 2024-11-04 15:50:11.958 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706540, nodeCount=80, validCount=80 2024-11-04 15:50:11.983 INFO 21062 --- [prometheus-minute-schedule-1,6796f66f67d30d65,fc742d7ea5d2ffda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:11.998 INFO 21062 --- [ocp-schedules-23,387dc6dfddaded18,53fa6c5c18818692] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:50:11.998 INFO 21062 --- [ocp-schedules-23,6cfa0e86ab042c94,e4c9a672bd3711c7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:50:12.002 INFO 21062 --- [ocp-schedules-23,6cfa0e86ab042c94,e4c9a672bd3711c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:12.030 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:50:12.030 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:50:12.106 INFO 21062 --- [ocp-schedules-29,cd757ed464f0c3fd,f3546f34f6cfc1d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:50:12.643 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:12.644 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:13.638 INFO 21062 --- [ocp-schedules-26,a274863203ee5ecb,2bfa54eecd95807b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:50:13.643 INFO 21062 --- [ocp-schedules-26,a274863203ee5ecb,2bfa54eecd95807b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:50:13.643 INFO 21062 --- [ocp-schedules-26,a274863203ee5ecb,2bfa54eecd95807b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:50:13.709 WARN 21062 --- [ocp-async-1,a9b152f5e0b5c7f5,689e317905178cef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:13.709 INFO 21062 --- [ocp-async-1,5dc80e8d585bdb58,d2b0d16c38a6ad5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:13.984 INFO 21062 --- [prometheus-cache-monitor1,6d713b68cb150544,c6a7e7a66873aeda] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1692,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:50:13.984 INFO 21062 --- [prometheus-cache-monitor1,6d713b68cb150544,c6a7e7a66873aeda] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:50:14.983 INFO 21062 --- [prometheus-minute-schedule-1,8c3e76148e360308,e245e0b9a279c0fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:15.645 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:15.646 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:15.709 WARN 21062 --- [ocp-async-10,4cf1286e28672c89,35c6d0673bccfe44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:15.709 INFO 21062 --- [ocp-async-6,28f9fb54b9c2f6ec,12748366a12705a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:17.709 WARN 21062 --- [ocp-async-2,396ec5d28b0332e0,27f879159dda57cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:17.709 INFO 21062 --- [ocp-async-2,c3d52bbcb25b4dde,555ecd1cd67d2b3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:17.983 INFO 21062 --- [prometheus-minute-schedule-1,6ac41a3984687430,d59828a1c2950e1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:50:18.589885+08:00, lastProcessTime=2024-11-04T15:50:08.589520+08:00 2024-11-04 15:50:18.646 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:18.647 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:18.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:50:18.589885+08:00 2024-11-04 15:50:18.826 INFO 21062 --- [ocp-async-5,dbe0817144bf39bb,3a72f22cbbdbef32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:50:19.709 WARN 21062 --- [ocp-async-5,27b6c3a47238b289,410afc08174de0b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:19.709 INFO 21062 --- [ocp-async-3,84e14f2b983131b8,51e800fda181bcba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:20.983 INFO 21062 --- [prometheus-minute-schedule-1,6bccaed87221b9d5,873a6542fccc3e99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:21.647 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:21.648 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:21.709 WARN 21062 --- [ocp-async-4,1e315e289b436214,575adb379c0f317a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:21.709 INFO 21062 --- [ocp-async-1,cef5dd241315e322,32767a3935d3e60b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:21.859 INFO 21062 --- [ocp-schedules-25,76f8a2ec6373e731,f3cc9e97be59cde3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:50:22.108 INFO 21062 --- [ocp-schedules-17,cd28a9ff4180bc5b,7240a2e1e49be520] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:50:23.709 WARN 21062 --- [ocp-async-6,cc7156da495b6032,2e045e1840d1c947] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:23.709 INFO 21062 --- [ocp-async-7,c33252721ac94dbc,5834c66eb9de3da7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:23.983 INFO 21062 --- [prometheus-minute-schedule-1,93093dca8b65c34a,f608f0e6de1d25b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:24.648 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:24.650 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:25.710 WARN 21062 --- [ocp-async-5,912474674576734a,bbf1bae8f278af54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:25.710 INFO 21062 --- [ocp-async-5,85943ef4e74e6a6a,b94e51924bb99ca2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:26.983 INFO 21062 --- [prometheus-minute-schedule-1,e960610e1735404b,1654cd92c6127320] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:26.984 INFO 21062 --- [prometheus-hour-schedule-1,b24f1a2ab7b53f5b,0dc5a5e4935f204a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:50:27.650 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:27.652 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:27.710 WARN 21062 --- [ocp-async-4,a7dddf29e0e34304,898c67592b5de235] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:27.710 INFO 21062 --- [ocp-async-3,edc38736f572f38d,e77738682c416b2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:28.498 INFO 21062 --- [ocp-schedules-15,d81d9c0f47b866ce,1ced2a9efb8f5872] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:50:28.499 INFO 21062 --- [ocp-schedules-30,e8bff671cc0054d1,73363d1df41a6310] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:50:28.500 INFO 21062 --- [ocp-schedules-24,d2343111c400a713,d5031e162d312599] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:50:28.500 INFO 21062 --- [ocp-schedules-5,3b36eb12e87d2455,85f1c4787f44b6e0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:50:28.502 INFO 21062 --- [ocp-schedules-15,d81d9c0f47b866ce,1ced2a9efb8f5872] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:50:28.502 INFO 21062 --- [ocp-schedules-9,95fe92b061bdde74,4b00cfd92a8e144b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:50:28.504 INFO 21062 --- [ocp-schedules-24,d2343111c400a713,d5031e162d312599] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:28.504 INFO 21062 --- [ocp-schedules-5,3b36eb12e87d2455,85f1c4787f44b6e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:28.504 INFO 21062 --- [ocp-schedules-14,f7a7cdf52888041f,f03c628505aabacc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:50:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:50:28.506 INFO 21062 --- [ocp-schedules-30,e8bff671cc0054d1,73363d1df41a6310] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:50:28.510 INFO 21062 --- [ocp-schedules-2,ef0b8384c43069aa,42049323ed7581c9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:50:28.510 INFO 21062 --- [ocp-schedules-2,ef0b8384c43069aa,42049323ed7581c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:50:28.510 INFO 21062 --- [ocp-schedules-10,f8aff4b4e7b572e5,2ef0791719fabc81] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:50:28.510 INFO 21062 --- [ocp-schedules-10,f8aff4b4e7b572e5,2ef0791719fabc81] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:50:28.510 INFO 21062 --- [ocp-schedules-20,2672176e61853734,d0a385aa474c8ddd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:50:28.510 INFO 21062 --- [ocp-schedules-20,2672176e61853734,d0a385aa474c8ddd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:50:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:50:28.588459+08:00, lastProcessTime=2024-11-04T15:50:18.589885+08:00 2024-11-04 15:50:28.592 INFO 21062 --- [ocp-schedules-8,91571e78bd671f84,415cc721a3d088b8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1692,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":423,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:50:28.592 INFO 21062 --- [ocp-schedules-8,91571e78bd671f84,415cc721a3d088b8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:50:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:50:28.588459+08:00 2024-11-04 15:50:28.826 INFO 21062 --- [ocp-async-7,1f41a043ea04244f,e84163c1414653ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:50:29.041 INFO 21062 --- [ocp-schedules-12,bfb0157839815f15,9657319916457345] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:50:29.041 INFO 21062 --- [ocp-schedules-12,bfb0157839815f15,9657319916457345] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:50:29.055 INFO 21062 --- [ocp-schedules-9,08ab6a49ce6691e8,aa6239f5b2a35bbb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:50:29.502 INFO 21062 --- [ocp-schedules-2,e8f530d523262767,f6d53c5fa0b50b97] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:50:29.502 INFO 21062 --- [ocp-schedules-2,e8f530d523262767,f6d53c5fa0b50b97] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:50:29.663 INFO 21062 --- [ocp-schedules-19,8ce08a2976c19dd4,38df28faaafec478] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:50:29.674 INFO 21062 --- [ocp-schedules-13,e35d5a62768998cf,d9440f98ef40da08] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:50:29.680 INFO 21062 --- [ocp-schedules-13,e35d5a62768998cf,d9440f98ef40da08] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:50:29.680 INFO 21062 --- [ocp-schedules-13,e35d5a62768998cf,d9440f98ef40da08] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:50:29.710 WARN 21062 --- [ocp-async-7,a50ef4bee8d0a066,4129516508d32cac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:29.710 INFO 21062 --- [ocp-async-2,fd92688a31ffd4a7,cbb2414947d3dffd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:29.983 INFO 21062 --- [prometheus-minute-schedule-1,2153d2fc48091ae1,cb8be261f78eecc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:30.211 INFO 21062 --- [ocp-schedules-3,f22010c7f737b243,53e9a9e79d5fefb5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:50:30.212 INFO 21062 --- [ocp-schedules-3,f22010c7f737b243,53e9a9e79d5fefb5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:50:30.219 INFO 21062 --- [ocp-schedules-3,f22010c7f737b243,53e9a9e79d5fefb5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:50:30.219 INFO 21062 --- [ocp-schedules-3,f22010c7f737b243,53e9a9e79d5fefb5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:50:30.223 INFO 21062 --- [ocp-schedules-3,f22010c7f737b243,53e9a9e79d5fefb5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:50:30.223 INFO 21062 --- [ocp-schedules-3,f22010c7f737b243,53e9a9e79d5fefb5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:50:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:50:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:50:30.652 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:30.654 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:31.710 WARN 21062 --- [ocp-async-4,6ab171b1738e5df3,fa39ba3d635f859a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:31.710 INFO 21062 --- [ocp-async-4,e10bfe820d534e21,25c85407868affdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:31.885 INFO 21062 --- [ocp-schedules-11,fb922f3f26f5a3a8,17398dcae1429a74] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:50:32.111 INFO 21062 --- [ocp-schedules-22,93d9ae6487f56c23,0e82cad9865f6045] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:50:32.983 INFO 21062 --- [prometheus-minute-schedule-1,e951d3819fd655ee,ea74dd5232144c82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:33.647 INFO 21062 --- [ocp-schedules-7,8cf8abc5092fba0c,01a29758f3b4f2af] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:50:33.654 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:33.654 INFO 21062 --- [ocp-schedules-7,8cf8abc5092fba0c,01a29758f3b4f2af] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:50:33.654 INFO 21062 --- [ocp-schedules-7,8cf8abc5092fba0c,01a29758f3b4f2af] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:50:33.655 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:33.710 WARN 21062 --- [ocp-async-9,c4bdd6ed6da47312,d72dce80e17f5dbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:33.710 INFO 21062 --- [ocp-async-6,da91944a95d2c6d3,0e532d5cb7273226] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:33.984 INFO 21062 --- [ocp-schedules-28,44b143d378427258,e0ac3a989e1b7d10] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:50:33.984 INFO 21062 --- [ocp-schedules-28,44b143d378427258,e0ac3a989e1b7d10] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:50:34.012 INFO 21062 --- [ocp-schedules-28,44b143d378427258,e0ac3a989e1b7d10] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:50:34.012 INFO 21062 --- [ocp-schedules-28,44b143d378427258,e0ac3a989e1b7d10] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:50:34.014 INFO 21062 --- [ocp-schedules-15,ff381c3e4e3d5f59,b7782f9422979563] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:50:34.190 INFO 21062 --- [ocp-schedules-27,bda31b8bf6ca5c20,5023bceb35618bab] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:50:34.190+08:00, clusters=[1] 2024-11-04 15:50:34.191 INFO 21062 --- [ocp-schedules-27,bda31b8bf6ca5c20,5023bceb35618bab] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:50:34.562 INFO 21062 --- [ocp-schedules-9,57d6e964a52cea12,3116fef7f2170401] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:50:34.562+08:00, clusters=[1] 2024-11-04 15:50:34.569 INFO 21062 --- [ocp-schedules-9,57d6e964a52cea12,3116fef7f2170401] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:50:34.672 INFO 21062 --- [ocp-schedules-11,0bf44d0faa18647c,cf32021eb25dd779] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:50:34.676 INFO 21062 --- [ocp-schedules-11,0bf44d0faa18647c,cf32021eb25dd779] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:34.732 ERROR 21062 --- [ocp-schedules-11,0bf44d0faa18647c,cf32021eb25dd779] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:50:34.732 INFO 21062 --- [ocp-schedules-11,0bf44d0faa18647c,cf32021eb25dd779] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:50:35.710 WARN 21062 --- [ocp-async-7,fffdd42537ba9236,31aeafac07632f48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:35.710 INFO 21062 --- [ocp-async-8,8ba4f496d199cbbf,ac235600a8e88427] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:35.983 INFO 21062 --- [prometheus-minute-schedule-1,54a23b3cb21c3ae8,8d82c3698d0b22cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:36.655 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:36.656 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:36.721 INFO 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:50:36.721 INFO 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:50:36.721 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:50:36.721 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:50:36.730 INFO 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:50:36.731 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:50:36.731 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:50:36.731 WARN 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:50:36.732 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:50:36.732 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:50:36.732 INFO 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:50:36.732 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:50:36.733 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:50:36.733 WARN 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:50:36.733 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:50:36.734 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:50:36.740 INFO 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:50:36.740 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:50:36.740 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:50:36.767 ERROR 21062 --- [ocp-schedules-15,581e31662e395069,4cd68d318609d0fb] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:50:37.711 WARN 21062 --- [ocp-async-3,7ee332cc69e23563,0f8ff29acca28532] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:37.711 INFO 21062 --- [ocp-async-5,0fb50519f5087b20,c232337998315006] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:38.504 INFO 21062 --- [ocp-schedules-5,3b36eb12e87d2455,85f1c4787f44b6e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:50:38.505 INFO 21062 --- [ocp-schedules-24,d2343111c400a713,d5031e162d312599] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:50:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:50:38.589008+08:00, lastProcessTime=2024-11-04T15:50:28.588459+08:00 2024-11-04 15:50:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:50:38.589008+08:00 2024-11-04 15:50:38.826 INFO 21062 --- [ocp-async-10,f689ae1dcdb81a84,9f4e244c5c9be7b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:50:38.983 INFO 21062 --- [prometheus-minute-schedule-1,863d4febdec34367,98952ce06cad27dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:39.425 INFO 21062 --- [ocp-schedules-26,ab8384ebf7533ebf,f172870b8e420efd] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:50:39.429 INFO 21062 --- [ocp-schedules-26,ab8384ebf7533ebf,f172870b8e420efd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:39.429 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:50:39.656 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:39.657 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:39.711 WARN 21062 --- [ocp-async-6,6e79962369b442ce,e32cbc5f6e264c48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:39.711 INFO 21062 --- [ocp-async-7,af3a7031795bf708,4415b31fe5af9257] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:50:40.004 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:50:41.711 WARN 21062 --- [ocp-async-8,9c382efd6932337d,573c40dc2b793c9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:41.711 INFO 21062 --- [ocp-async-3,0dd735397ab04aa8,54b6ac38cf88d07c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:41.798 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:50:41.799 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:50:41.800 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:50:41.801 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:50:41.827 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:50:41.827 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-11-04 15:50:41.924 INFO 21062 --- [ocp-schedules-19,162386488100dd16,579d7fc181cabd93] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:50:41.983 INFO 21062 --- [prometheus-minute-schedule-1,edd7e61bf20652cc,0e4af2db45429459] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:42.031 INFO 21062 --- [ocp-schedules-23,6cfa0e86ab042c94,e4c9a672bd3711c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:50:42.031 INFO 21062 --- [ocp-schedules-23,8e2773ceab8959ab,6dafc029b4530f74] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:50:42.034 INFO 21062 --- [ocp-schedules-23,8e2773ceab8959ab,6dafc029b4530f74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:42.061 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:50:42.061 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:50:42.113 INFO 21062 --- [ocp-schedules-13,bd7dd88a0c09f629,0932867e1fe27460] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:50:42.657 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:42.658 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:43.711 WARN 21062 --- [ocp-async-5,f94e2aff536af651,e70688a754b053d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:43.711 INFO 21062 --- [ocp-async-4,8f8b7960501330c1,378166acb17d2ff1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:44.983 INFO 21062 --- [prometheus-minute-schedule-1,89b0246a9bdbc7c4,88dbca4d4ea92240] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:45.658 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:45.659 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:45.711 WARN 21062 --- [ocp-async-6,2a6e955f9652f334,1b252b2852ffee98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:45.711 INFO 21062 --- [ocp-async-10,687c62e2873fd6b6,6cc27ebbaaf6a36a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:46.828 INFO 21062 --- [ocp-schedules-8,7cb9fd7d8ecee35d,810f5796b5be5618] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:50:46.832 INFO 21062 --- [ocp-schedules-8,7cb9fd7d8ecee35d,810f5796b5be5618] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:46.917 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:50:46.921 ERROR 21062 --- [ocp-schedules-8,7cb9fd7d8ecee35d,810f5796b5be5618] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:50:46.921 INFO 21062 --- [ocp-schedules-8,7cb9fd7d8ecee35d,810f5796b5be5618] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:50:47.711 WARN 21062 --- [ocp-async-8,d3acc96f057a02a0,4d0bb4e4b2841063] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:47.712 INFO 21062 --- [ocp-async-1,3116d635a552b7cd,562a26c1767836fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:47.983 INFO 21062 --- [prometheus-minute-schedule-1,b3b4d5c3f6fbf207,f8c286c666c82e4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:50:48.589689+08:00, lastProcessTime=2024-11-04T15:50:38.589008+08:00 2024-11-04 15:50:48.659 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:48.661 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:48.776 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:50:48.589689+08:00 2024-11-04 15:50:48.827 INFO 21062 --- [ocp-async-5,33165e886e1e72c0,899fab5d55a5cc36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:50:49.430 INFO 21062 --- [ocp-schedules-26,ab8384ebf7533ebf,f172870b8e420efd] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:50:49.712 WARN 21062 --- [ocp-async-2,54186c70f92145c1,b6fed31582789027] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:49.712 INFO 21062 --- [ocp-async-4,a2013b937d3c9811,5f4b15c2f740586d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:50.983 INFO 21062 --- [prometheus-minute-schedule-1,7a003f60438dcbd6,c08542fd3485d42c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:51.661 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:51.661 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:51.712 WARN 21062 --- [ocp-async-9,33761b33c104d43f,bc5cc395e5cd99cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:51.712 INFO 21062 --- [ocp-async-10,b976e6a768f593e6,c24d1d712aa1e10b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:51.951 INFO 21062 --- [ocp-schedules-17,7dcf8289cb8dd65a,2f88e58b842bd55e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:50:52.116 INFO 21062 --- [ocp-schedules-29,b3d94080ee594487,14839cbf462ab20b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:50:53.657 INFO 21062 --- [ocp-schedules-22,21e095e60c2cc5e3,42decc1b51fdfc5b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:50:53.663 INFO 21062 --- [ocp-schedules-22,21e095e60c2cc5e3,42decc1b51fdfc5b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:50:53.663 INFO 21062 --- [ocp-schedules-22,21e095e60c2cc5e3,42decc1b51fdfc5b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:50:53.712 WARN 21062 --- [ocp-async-8,0efa7bcf3e26f24c,ebd4d68a4be895a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:53.712 INFO 21062 --- [ocp-async-7,95c95c8db9ecb08f,1d6fd8bf1bc191f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:53.983 INFO 21062 --- [prometheus-minute-schedule-1,0e4e745a40da598f,3fcfe681690f1cb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:54.661 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:54.662 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:55.712 WARN 21062 --- [ocp-async-3,3e019732ccd8d96d,1a8a5cdaf5f60e26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:55.712 INFO 21062 --- [ocp-async-2,8b372567d62ac69c,861f93ecf64678ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:56.983 INFO 21062 --- [prometheus-minute-schedule-1,6930cd078f472b28,4df9ded8bddf237f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:50:57.662 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:50:57.663 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:50:57.712 WARN 21062 --- [ocp-async-4,e6ab8cb6229e7099,64d25ec326cafee8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:57.712 INFO 21062 --- [ocp-async-4,416e4f8a40e01ed2,3eaf8b5fa07a45a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:58.500 INFO 21062 --- [ocp-schedules-21,8eef097c94fdd366,1c9bd77c7814bff7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:50:58.500 INFO 21062 --- [ocp-schedules-8,724bad9c796509a0,9d42136afe201364] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:50:58.502 INFO 21062 --- [ocp-schedules-12,3f6f073cb023a318,a45d6056eb372a84] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:50:58.503 INFO 21062 --- [ocp-schedules-21,8eef097c94fdd366,1c9bd77c7814bff7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:50:58.504 INFO 21062 --- [ocp-schedules-8,724bad9c796509a0,9d42136afe201364] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:50:58.504 INFO 21062 --- [ocp-schedules-14,4e1e98c3deca91bb,51b8c7e143e24bb5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:50:58.509 INFO 21062 --- [ocp-schedules-20,1539b5c8a3984ef4,73890d61618996c9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:50:58.509 INFO 21062 --- [ocp-schedules-20,1539b5c8a3984ef4,73890d61618996c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:50:58.509 INFO 21062 --- [ocp-schedules-28,7e73a2a9d9163d44,1bc6fcd853771ed3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:50:58.509 INFO 21062 --- [ocp-schedules-28,7e73a2a9d9163d44,1bc6fcd853771ed3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:50:58.510 INFO 21062 --- [ocp-schedules-2,a0dc87c1b5d15d23,17007aa7d2a6563d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:50:58.510 INFO 21062 --- [ocp-schedules-2,a0dc87c1b5d15d23,17007aa7d2a6563d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:50:58.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:50:58.588850+08:00, lastProcessTime=2024-11-04T15:50:48.589689+08:00 2024-11-04 15:50:58.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:50:58.588850+08:00 2024-11-04 15:50:58.827 INFO 21062 --- [ocp-async-8,41a919bd2a2a7312,4319b6e01584284e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:50:59.666 INFO 21062 --- [ocp-schedules-2,6692489ee1514709,3cccb373c9baa99b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:50:59.712 WARN 21062 --- [ocp-async-8,5b8afca85f37a50d,ffc17d918b8e8c6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:50:59.713 INFO 21062 --- [ocp-async-8,02ea2811f4408d4b,e500baa5e26dff04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:50:59.983 INFO 21062 --- [prometheus-minute-schedule-1,89b68fd0350ddeba,b81315d3ca1cf94d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706660, interval=60 2024-11-04 15:51:00.663 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:00.664 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:01.713 WARN 21062 --- [ocp-async-7,a909047d7acfc76f,d250bf2a3f16c425] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:01.713 INFO 21062 --- [ocp-async-7,15733d2587f05bec,52c3c85e990af640] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:02.033 INFO 21062 --- [ocp-schedules-29,076db64b722400cd,71685e931ea86224] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:51:02.119 INFO 21062 --- [ocp-schedules-19,bb3670b6d4f0c2b6,2a89b00b4230b4ee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:51:02.983 INFO 21062 --- [prometheus-minute-schedule-1,5d002ac933b6b3ff,ad44490fc79fdf21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:03.664 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:03.665 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:03.713 WARN 21062 --- [ocp-async-2,8a1de8c9a25749fb,2a322359c405003b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:03.713 INFO 21062 --- [ocp-async-6,1ffc9305fb5036c4,059b2847bd499685] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:05.713 WARN 21062 --- [ocp-async-1,e1731e6f76850582,dee7c4963eb684fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:05.713 INFO 21062 --- [ocp-async-10,e3bde6665f59214b,f8a781ab5774ee20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d0ef2b9c82624da6,778c7f8c87e14f65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:06.665 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:06.666 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:07.713 INFO 21062 --- [ocp-async-5,e0e0256ecc245765,529743b67312748d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:07.713 WARN 21062 --- [ocp-async-8,387c9d6dbf4f8ed3,c44c2be04dda2005] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:08.504 INFO 21062 --- [ocp-schedules-21,8eef097c94fdd366,1c9bd77c7814bff7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:51:08.504 INFO 21062 --- [ocp-schedules-8,724bad9c796509a0,9d42136afe201364] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:51:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:51:08.590209+08:00, lastProcessTime=2024-11-04T15:50:58.588850+08:00 2024-11-04 15:51:08.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:51:08.590209+08:00 2024-11-04 15:51:08.827 INFO 21062 --- [ocp-async-2,593b9cb2960063ec,0543b7cf088b2b03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:51:08.983 INFO 21062 --- [prometheus-minute-schedule-1,79923dfcaac5af85,5d0579a3f8db5070] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:09.666 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:09.667 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:09.713 INFO 21062 --- [ocp-async-2,395c52ffee92a3d5,cd3d3fdd80b520f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:09.713 WARN 21062 --- [ocp-async-6,cb33e0d315dd120f,ba169a0893a3a0d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:10.003 INFO 21062 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:51:10.003 INFO 21062 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:51:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:51:10.297 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:51:10.297 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:51:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:51:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:51:10.325 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:51:11.713 INFO 21062 --- [ocp-async-4,5d8458c7d401e739,b41ab83ce133cf03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:11.713 WARN 21062 --- [ocp-async-10,819d98f9f79dff04,5824f9ba3ce5e784] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:11.953 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=20, affectRows=20 2024-11-04 15:51:11.953 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=20, validCount=20 2024-11-04 15:51:11.953 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=32, affectRows=32 2024-11-04 15:51:11.953 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=11, affectRows=11 2024-11-04 15:51:11.953 INFO 21062 --- [metric-archive-second-39,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=32, validCount=32 2024-11-04 15:51:11.953 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=40, affectRows=40 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=22, affectRows=22 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=11, validCount=11 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-37,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=22, validCount=22 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=28, affectRows=28 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=40, validCount=40 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=28, validCount=28 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=45, affectRows=45 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=78, affectRows=78 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=62, affectRows=62 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=45, validCount=45 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=62, validCount=62 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=77, validCount=77 2024-11-04 15:51:11.954 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706600, nodeCount=86, affectRows=86 2024-11-04 15:51:11.955 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706600, nodeCount=84, validCount=84 2024-11-04 15:51:11.983 INFO 21062 --- [prometheus-minute-schedule-1,7529beec8e2e8bed,4c8cce62bf09b893] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:12.056 INFO 21062 --- [ocp-schedules-12,340e96f72f16af94,1cd603235277afbb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:51:12.062 INFO 21062 --- [ocp-schedules-23,8e2773ceab8959ab,6dafc029b4530f74] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:51:12.062 INFO 21062 --- [ocp-schedules-23,583ec116b2973600,d55ff1ec2c8bb974] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:51:12.066 INFO 21062 --- [ocp-schedules-23,583ec116b2973600,d55ff1ec2c8bb974] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:12.095 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:51:12.095 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:51:12.122 INFO 21062 --- [ocp-schedules-5,2edfc79bb4dbcb77,a254516980260f3d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:51:12.667 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:12.668 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:13.665 INFO 21062 --- [ocp-schedules-5,4ca748b31fc8eac8,00229620f28cc163] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:51:13.669 INFO 21062 --- [ocp-schedules-5,4ca748b31fc8eac8,00229620f28cc163] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:51:13.669 INFO 21062 --- [ocp-schedules-5,4ca748b31fc8eac8,00229620f28cc163] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:51:13.713 WARN 21062 --- [ocp-async-5,c934221332fc1d96,3db946b54f4a91cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:13.713 INFO 21062 --- [ocp-async-3,a5525fa7e3512be4,493f857251dd3c1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:13.984 INFO 21062 --- [prometheus-cache-monitor1,74a32241deda3f2e,91a56baee8bc99b5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1694,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:51:13.985 INFO 21062 --- [prometheus-cache-monitor1,74a32241deda3f2e,91a56baee8bc99b5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:51:14.983 INFO 21062 --- [prometheus-minute-schedule-1,561076c66dac47ae,bd7eaa2b405b1be9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:15.668 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:15.669 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:15.714 INFO 21062 --- [ocp-async-6,c057ac6788dc6ed3,b3b9863ffbc861be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:15.714 WARN 21062 --- [ocp-async-9,79e45c41b29b2239,5648736bb08c09c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:15.850 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:47:15.832, end=2024-11-04T15:49:15.847} 2024-11-04 15:51:15.858 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:51:15.858 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:51:17.714 WARN 21062 --- [ocp-async-1,c31d3c67f48e61d0,9da6d2b7bfe0c4cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:17.714 INFO 21062 --- [ocp-async-10,1e80bc5a1323cf62,4e8fb59817830787] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:17.983 INFO 21062 --- [prometheus-minute-schedule-1,9b3d75deb51d3a56,cfa7e665c5182870] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:51:18.588894+08:00, lastProcessTime=2024-11-04T15:51:08.590209+08:00 2024-11-04 15:51:18.669 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:18.669 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:18.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:51:18.588894+08:00 2024-11-04 15:51:18.827 INFO 21062 --- [ocp-async-5,92c97837ab6f4c1c,58dbe99bc789acf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:51:19.714 WARN 21062 --- [ocp-async-8,4f0c5fafc818f2c4,76fa2ece7b422889] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:19.714 INFO 21062 --- [ocp-async-3,5e20b4c5fee49d76,53f417f5bd3de922] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:20.983 INFO 21062 --- [prometheus-minute-schedule-1,2f80bbcdb53311af,fb0f30371154d46a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:21.670 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:21.671 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:21.714 WARN 21062 --- [ocp-async-7,66784ce7dfa426b7,8735b38734ac7b60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:21.715 INFO 21062 --- [ocp-async-7,880fa1a123d8a8cc,3fc1ef58ed23e966] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:22.098 INFO 21062 --- [ocp-schedules-20,3893f64daf624e4e,6b62eabc5d62cb74] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:51:22.124 INFO 21062 --- [ocp-schedules-24,4bfe902c41810f1e,a283719ae9072e1e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:51:23.715 WARN 21062 --- [ocp-async-2,b31a6cbd31837863,7ba80389d9aba4f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:23.715 INFO 21062 --- [ocp-async-10,ddf4c75b79ee3b7d,9567723a787e63b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:23.983 INFO 21062 --- [prometheus-minute-schedule-1,a4f97089cd42b91c,4629c3bdb893f19e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:24.671 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:24.672 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:25.715 WARN 21062 --- [ocp-async-5,11b416993e716f32,f86c1835fcc46cd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:25.715 INFO 21062 --- [ocp-async-8,eb4ec531e1595c97,71d0a630f5337800] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:26.983 INFO 21062 --- [prometheus-minute-schedule-1,3f3f5b25520eb49b,18c923b4612d572a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:26.984 INFO 21062 --- [prometheus-hour-schedule-1,e19c3b8343820c8b,18cc547b2d1cc8cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:51:27.672 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:27.673 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:27.715 INFO 21062 --- [ocp-async-6,a765a451ee6844ab,01fdc64481dbd5b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:27.715 WARN 21062 --- [ocp-async-9,5bfc12bac9345297,79fd634b9681e1c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:28.498 INFO 21062 --- [ocp-schedules-21,8f4b0a80f2c5c5c1,12424f1aa17ba7a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:51:28.500 INFO 21062 --- [ocp-schedules-4,914db3245912d2cc,8e9fcb6f90c2a1a4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:51:28.500 INFO 21062 --- [ocp-schedules-8,0d3e2b5982949ef1,94714f1bb6fbf15d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:51:28.500 INFO 21062 --- [ocp-schedules-30,67b95d0aef7281be,8bbbc90e25b6d4e9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:51:28.503 INFO 21062 --- [ocp-schedules-19,ef73ab61cacd9e6f,3efe1666d9f1aef1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:51:28.504 INFO 21062 --- [ocp-schedules-21,8f4b0a80f2c5c5c1,12424f1aa17ba7a7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:51:28.504 INFO 21062 --- [ocp-schedules-30,67b95d0aef7281be,8bbbc90e25b6d4e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:28.505 INFO 21062 --- [ocp-schedules-4,914db3245912d2cc,8e9fcb6f90c2a1a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:28.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:51:28.506 INFO 21062 --- [ocp-schedules-24,15bf709103344503,f375498bd79ec9a9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:51:28.507 INFO 21062 --- [ocp-schedules-8,0d3e2b5982949ef1,94714f1bb6fbf15d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:51:28.511 INFO 21062 --- [ocp-schedules-15,9040ba26b2d368a3,81a37e7a1e076400] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:51:28.511 INFO 21062 --- [ocp-schedules-7,bbe409de88e8172e,93a35913177f384c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:51:28.511 INFO 21062 --- [ocp-schedules-28,5f92209c3656060d,9ac05e7defe4fa34] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:51:28.511 INFO 21062 --- [ocp-schedules-15,9040ba26b2d368a3,81a37e7a1e076400] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:51:28.511 INFO 21062 --- [ocp-schedules-7,bbe409de88e8172e,93a35913177f384c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:51:28.511 INFO 21062 --- [ocp-schedules-28,5f92209c3656060d,9ac05e7defe4fa34] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:51:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:51:28.588895+08:00, lastProcessTime=2024-11-04T15:51:18.588894+08:00 2024-11-04 15:51:28.681 INFO 21062 --- [ocp-schedules-10,026fc15915260ae7,eaa03c1e98ef5687] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:51:28.682 INFO 21062 --- [ocp-schedules-10,026fc15915260ae7,eaa03c1e98ef5687] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:51:28.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:51:28.588895+08:00 2024-11-04 15:51:28.827 INFO 21062 --- [ocp-async-10,79733ed8d46fcdba,5c3587cbbda34aab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:51:29.057 INFO 21062 --- [ocp-schedules-12,651f10207fc5b130,ee37d89141d0aa2d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:51:29.507 INFO 21062 --- [ocp-schedules-28,9f5f0354b6a06118,495dd1d4594dbc42] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:51:29.507 INFO 21062 --- [ocp-schedules-28,9f5f0354b6a06118,495dd1d4594dbc42] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:51:29.669 INFO 21062 --- [ocp-schedules-29,f488943854be422b,748cf6e0f84093c4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:51:29.680 INFO 21062 --- [ocp-schedules-25,813f68acc5b9b4b2,fcb62bcd3a89a7c8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:51:29.686 INFO 21062 --- [ocp-schedules-25,813f68acc5b9b4b2,fcb62bcd3a89a7c8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:51:29.686 INFO 21062 --- [ocp-schedules-25,813f68acc5b9b4b2,fcb62bcd3a89a7c8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:51:29.715 INFO 21062 --- [ocp-async-5,ef4d198132e91d54,96863c79d0cb9337] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:29.715 WARN 21062 --- [ocp-async-4,ae8173849d5e31a8,d165fe90556d565e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:29.983 INFO 21062 --- [prometheus-minute-schedule-1,655aa2a62ee35c3f,3b12392e127cbd18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:30.228 INFO 21062 --- [ocp-schedules-6,72caac1773e43444,289adcea60199391] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:51:30.228 INFO 21062 --- [ocp-schedules-6,72caac1773e43444,289adcea60199391] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:51:30.229 INFO 21062 --- [ocp-schedules-6,72caac1773e43444,289adcea60199391] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:51:30.229 INFO 21062 --- [ocp-schedules-6,72caac1773e43444,289adcea60199391] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:51:30.230 INFO 21062 --- [ocp-schedules-6,72caac1773e43444,289adcea60199391] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:51:30.230 INFO 21062 --- [ocp-schedules-6,72caac1773e43444,289adcea60199391] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:51:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:51:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:51:30.673 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:30.674 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:31.716 INFO 21062 --- [ocp-async-9,48c385ba6ce61e9e,59bbc9c743a5b01b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:31.716 WARN 21062 --- [ocp-async-6,e2780bfa1c7b0d79,9fa2d6ecf1294fb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:32.126 INFO 21062 --- [ocp-schedules-17,11cc16e411f7f0cc,29ea8d4aaa9d95ed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:51:32.129 INFO 21062 --- [ocp-schedules-13,c7d193d71d48af3b,5335a9ec9d4f1444] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:51:32.983 INFO 21062 --- [prometheus-minute-schedule-1,b4024bc3fb63bb16,3122b6ef503e5962] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:33.672 INFO 21062 --- [ocp-schedules-12,f16e32316cc20cdc,bae5114e9e136b28] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:51:33.674 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:33.675 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:33.678 INFO 21062 --- [ocp-schedules-12,f16e32316cc20cdc,bae5114e9e136b28] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:51:33.678 INFO 21062 --- [ocp-schedules-12,f16e32316cc20cdc,bae5114e9e136b28] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:51:33.716 WARN 21062 --- [ocp-async-7,3d18a72461dbca01,a72bce4b722c18c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:33.716 INFO 21062 --- [ocp-async-2,1fe6ee6c56605eda,830da31feea29e69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:34.013 INFO 21062 --- [ocp-schedules-27,d9db5c0f53102fc6,5c21660f5164beca] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:51:34.013 INFO 21062 --- [ocp-schedules-27,d9db5c0f53102fc6,5c21660f5164beca] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:51:34.040 INFO 21062 --- [ocp-schedules-27,d9db5c0f53102fc6,5c21660f5164beca] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:51:34.040 INFO 21062 --- [ocp-schedules-27,d9db5c0f53102fc6,5c21660f5164beca] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:51:34.040 INFO 21062 --- [ocp-schedules-24,c60e2667cce88783,553abbbb52a552cf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:51:34.218 INFO 21062 --- [ocp-schedules-9,294a4e88ac688d15,610a29474bec91f6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:51:34.218+08:00, clusters=[1] 2024-11-04 15:51:34.219 INFO 21062 --- [ocp-schedules-9,294a4e88ac688d15,610a29474bec91f6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:51:34.597 INFO 21062 --- [ocp-schedules-5,33af67c60810fdc8,4d2386f189f0d2d8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:51:34.597+08:00, clusters=[1] 2024-11-04 15:51:34.602 INFO 21062 --- [ocp-schedules-5,33af67c60810fdc8,4d2386f189f0d2d8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:51:35.716 WARN 21062 --- [ocp-async-4,63b1d5c346af677b,ebbcee53ea4ab1b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:35.716 INFO 21062 --- [ocp-async-10,082160d5706d0e28,d07fcd3504c323b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:35.983 INFO 21062 --- [prometheus-minute-schedule-1,9f7b38b1612f81b2,59e08076453d83a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:36.675 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:36.676 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:36.777 INFO 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:51:36.777 INFO 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:51:36.777 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:51:36.777 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:51:36.788 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:51:36.788 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:51:36.788 INFO 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:51:36.790 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:51:36.790 WARN 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:51:36.790 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:51:36.790 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:51:36.790 INFO 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:51:36.791 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:51:36.791 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:51:36.792 WARN 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:51:36.792 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:51:36.799 INFO 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:51:36.799 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:51:36.799 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:51:36.809 ERROR 21062 --- [ocp-schedules-1,8b8db6461b351a20,fe57fea3ff3764dc] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:51:37.716 WARN 21062 --- [ocp-async-6,9b2835657ae0703b,b1f4e36a18eee27d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:37.716 INFO 21062 --- [ocp-async-8,fba8c83b63f53eb8,db097e64ee92ddd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:38.505 INFO 21062 --- [ocp-schedules-30,67b95d0aef7281be,8bbbc90e25b6d4e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:51:38.506 INFO 21062 --- [ocp-schedules-4,914db3245912d2cc,8e9fcb6f90c2a1a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:51:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:51:38.589467+08:00, lastProcessTime=2024-11-04T15:51:28.588895+08:00 2024-11-04 15:51:38.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:51:38.589467+08:00 2024-11-04 15:51:38.827 INFO 21062 --- [ocp-async-3,624c49d376820d1f,36ad3bb9b6d33afb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:51:38.983 INFO 21062 --- [prometheus-minute-schedule-1,53bad167c2d87a9e,950795d6e7699909] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:39.676 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:39.676 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:39.716 WARN 21062 --- [ocp-async-2,e44d742da5c1139c,f6372707f37c0b46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:39.716 INFO 21062 --- [ocp-async-4,c561b212aa1a1d49,7fcf9ab8871320ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:40.003 INFO 21062 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:51:40.004 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:51:41.716 WARN 21062 --- [ocp-async-10,cfb249ab9b2119ed,9a91b64747021e64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:41.717 INFO 21062 --- [ocp-async-6,4a65b88a367c6531,8d33cfdab69613df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:41.983 INFO 21062 --- [prometheus-minute-schedule-1,2aa8b3061015c14d,e1729721dd918ac8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:42.095 INFO 21062 --- [ocp-schedules-23,583ec116b2973600,d55ff1ec2c8bb974] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:51:42.096 INFO 21062 --- [ocp-schedules-23,881deab416eca06b,cb135884baea94a0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:51:42.099 INFO 21062 --- [ocp-schedules-23,881deab416eca06b,cb135884baea94a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:42.130 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:51:42.130 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:51:42.130 INFO 21062 --- [ocp-schedules-21,5c2400fdd667b41e,97bfa362015e611a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:51:42.172 INFO 21062 --- [ocp-schedules-18,6ec98614324d9b96,f7e431918fa27292] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:51:42.677 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:42.678 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:43.717 WARN 21062 --- [ocp-async-8,99597a2ec0a0abfb,76c6e3a4be3f6ebe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:43.717 INFO 21062 --- [ocp-async-9,52394b2a12fa3065,ed362eef4f53b79c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:44.983 INFO 21062 --- [prometheus-minute-schedule-1,54a3df31aa04d2c6,dca20ab79b7792ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:45.678 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:45.679 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:45.717 WARN 21062 --- [ocp-async-2,0e66cd4d8b735088,1359ca363860d65a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:45.717 INFO 21062 --- [ocp-async-2,a84d1958f715d804,da1beb754e6c333c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:46.921 INFO 21062 --- [ocp-schedules-20,4e6bfd41bc5f3062,065e25e13f2f499b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:51:46.926 INFO 21062 --- [ocp-schedules-20,4e6bfd41bc5f3062,065e25e13f2f499b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:47.010 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:51:47.013 ERROR 21062 --- [ocp-schedules-20,4e6bfd41bc5f3062,065e25e13f2f499b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:51:47.013 INFO 21062 --- [ocp-schedules-20,4e6bfd41bc5f3062,065e25e13f2f499b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:51:47.717 WARN 21062 --- [ocp-async-10,e65968acd8655cc4,466dbc7c36ee0c3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:47.717 INFO 21062 --- [ocp-async-1,29a29c0695313b35,4552a3d51c309ea9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:47.983 INFO 21062 --- [prometheus-minute-schedule-1,d3ff124933967620,16aa7ce4914e06da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:51:48.590240+08:00, lastProcessTime=2024-11-04T15:51:38.589467+08:00 2024-11-04 15:51:48.679 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:48.680 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:51:48.590240+08:00 2024-11-04 15:51:48.827 INFO 21062 --- [ocp-async-8,fc86414f0399c54e,808f708b41b5db25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:51:49.430 INFO 21062 --- [ocp-schedules-22,8daea70741d98fbc,183fc7a7096a69e2] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:51:49.433 INFO 21062 --- [ocp-schedules-22,8daea70741d98fbc,183fc7a7096a69e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:49.434 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:51:49.717 WARN 21062 --- [ocp-async-5,7de46a01ed635ea4,8b41cdd0f21d228e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:49.717 INFO 21062 --- [ocp-async-5,4220cc297e6a9fff,6d98f597503a6396] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:50.983 INFO 21062 --- [prometheus-minute-schedule-1,0c925349b1f649b3,5922dc3bc560bdb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:51.680 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:51.681 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:51.717 WARN 21062 --- [ocp-async-7,7a5daad407131ec4,dd6c0b1d9cc41c16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:51.717 INFO 21062 --- [ocp-async-7,0600c9e660a1a5bb,f1079fab09a99b5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:52.132 INFO 21062 --- [ocp-schedules-12,3ac069eb915177b3,47f095cbdd75c9cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:51:52.196 INFO 21062 --- [ocp-schedules-9,26a24a9cb8275f66,14088a6b662caac4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:51:53.680 INFO 21062 --- [ocp-schedules-20,5144f64a0a8049a9,3abc00892695e997] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:51:53.686 INFO 21062 --- [ocp-schedules-20,5144f64a0a8049a9,3abc00892695e997] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:51:53.686 INFO 21062 --- [ocp-schedules-20,5144f64a0a8049a9,3abc00892695e997] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:51:53.718 WARN 21062 --- [ocp-async-4,c9284b403b313d48,b94fe5505b8e1c65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:53.718 INFO 21062 --- [ocp-async-1,8d3571a8e8e464e7,3679303bf776360f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:53.983 INFO 21062 --- [prometheus-minute-schedule-1,b59736d4e1d73e6d,a8c9d0460b9d6c1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:54.681 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:54.682 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:55.718 INFO 21062 --- [ocp-async-6,f3323c2f58c47af4,c106ac57115471fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:55.718 WARN 21062 --- [ocp-async-9,a908022578b57588,e7630d3ec2bc2a5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c5ff5348a1080ec9,6e87fb9b8aeeef11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:51:57.682 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:51:57.683 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:51:57.718 INFO 21062 --- [ocp-async-5,1ad9293bc0bb330e,9b6859b427529f65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:57.718 WARN 21062 --- [ocp-async-2,af46f2b318225ce9,0c37d19d6e88e5ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:58.500 INFO 21062 --- [ocp-schedules-19,90a8bd4ee84ea1f5,e01002efdbb87503] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:51:58.500 INFO 21062 --- [ocp-schedules-10,cf015b31d1bd4a52,24b096a33c78b325] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:51:58.503 INFO 21062 --- [ocp-schedules-1,5f1f799e7912d3e0,07cee65c3723befa] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:51:58.503 INFO 21062 --- [ocp-schedules-19,90a8bd4ee84ea1f5,e01002efdbb87503] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:51:58.504 INFO 21062 --- [ocp-schedules-10,cf015b31d1bd4a52,24b096a33c78b325] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:51:58.505 INFO 21062 --- [ocp-schedules-4,61a6293eb298f0fe,c179e02fb6fbb836] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:51:58.509 INFO 21062 --- [ocp-schedules-12,14bad40989ab8226,0457355ea88452af] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:51:58.509 INFO 21062 --- [ocp-schedules-12,14bad40989ab8226,0457355ea88452af] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:51:58.510 INFO 21062 --- [ocp-schedules-18,094be5fa08517243,ca4ebf4cc1ade7d2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:51:58.510 INFO 21062 --- [ocp-schedules-18,094be5fa08517243,ca4ebf4cc1ade7d2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:51:58.510 INFO 21062 --- [ocp-schedules-14,bd961ca99e40f2db,516c9dc91d6d8242] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:51:58.510 INFO 21062 --- [ocp-schedules-14,bd961ca99e40f2db,516c9dc91d6d8242] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:51:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:51:58.588562+08:00, lastProcessTime=2024-11-04T15:51:48.590240+08:00 2024-11-04 15:51:58.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:51:58.588562+08:00 2024-11-04 15:51:58.828 INFO 21062 --- [ocp-async-4,a5158beb2ee49e9b,76f160592fc67030] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:51:59.434 INFO 21062 --- [ocp-schedules-22,8daea70741d98fbc,183fc7a7096a69e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:51:59.674 INFO 21062 --- [ocp-schedules-18,be55ea2227153ab0,2c58faa3ea0d1349] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:51:59.718 INFO 21062 --- [ocp-async-4,e007b8f91bfa3fc5,bc2b7f92847f4f78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:51:59.718 WARN 21062 --- [ocp-async-7,d4d3783863f60b92,371a302a0d6f8e57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:51:59.983 INFO 21062 --- [prometheus-minute-schedule-1,fd5d285ceb4ceb9d,3d5519a19c7cfdb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706720, interval=60 2024-11-04 15:52:00.683 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:00.683 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:01.718 INFO 21062 --- [ocp-async-8,db561ffcf012fa42,5c626ccff900580b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:01.719 WARN 21062 --- [ocp-async-1,fff9e3e00a11cd93,8e8bff63cdd325bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:02.136 INFO 21062 --- [ocp-schedules-21,711e80e9f97542e4,f7ddfdb7f4e71d1c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:52:02.237 INFO 21062 --- [ocp-schedules-5,af7d8a2a9cf2281a,cdff5be852d8ce85] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:52:02.983 INFO 21062 --- [prometheus-minute-schedule-1,60ab7f439a218be8,1b716293842b987c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:03.684 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:03.685 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:03.719 INFO 21062 --- [ocp-async-6,443e36a3f9ec6b8c,8c0b7a36a055f4a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:03.719 WARN 21062 --- [ocp-async-3,192c91b96508ceea,361c2bfb86ee9916] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:05.719 INFO 21062 --- [ocp-async-7,f37a4c3a841e2c89,fa44bb2a7df0a2d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:05.719 WARN 21062 --- [ocp-async-10,31abedecfa86d307,dbda059ea63d53c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:05.983 INFO 21062 --- [prometheus-minute-schedule-1,8ea0af04315f905f,f5bd44351830a5fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:06.685 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:06.686 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:07.719 INFO 21062 --- [ocp-async-1,9c8ef53c71ea090b,6e990e6c015fcb64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:07.719 WARN 21062 --- [ocp-async-4,e2d1431573800ea1,f85e18af914053d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:08.504 INFO 21062 --- [ocp-schedules-19,90a8bd4ee84ea1f5,e01002efdbb87503] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:52:08.505 INFO 21062 --- [ocp-schedules-10,cf015b31d1bd4a52,24b096a33c78b325] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:52:08.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:52:08.589951+08:00, lastProcessTime=2024-11-04T15:51:58.588562+08:00 2024-11-04 15:52:08.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:52:08.589951+08:00 2024-11-04 15:52:08.828 INFO 21062 --- [ocp-async-3,f4989e9599132d9b,24fefb3259ad9775] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:52:08.983 INFO 21062 --- [prometheus-minute-schedule-1,c112db70c213cb3f,c78f6b1b8b798a30] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:09.686 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:09.687 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:09.719 INFO 21062 --- [ocp-async-2,058761771c262e93,ce81ccb4e3b1e9db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:09.719 WARN 21062 --- [ocp-async-6,dbb234f8471c1e01,7f49e7cbb8cc2ad7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:10.002 INFO 21062 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:52:10.003 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:52:10.295 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:52:10.295 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:52:10.296 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:52:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:52:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:52:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:52:11.719 INFO 21062 --- [ocp-async-5,4cdd2d0d234a4459,f20cc9b369bda7e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:11.719 WARN 21062 --- [ocp-async-7,9933b5648e311c58,2f63de24518710a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:11.960 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=1, affectRows=1 2024-11-04 15:52:11.960 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=1, validCount=1 2024-11-04 15:52:11.961 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=12, affectRows=12 2024-11-04 15:52:11.961 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=12, validCount=12 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=10, affectRows=10 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=15, affectRows=15 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=10, validCount=10 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=57, affectRows=57 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=24, affectRows=24 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-33,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=15, validCount=15 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=24, validCount=24 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=57, validCount=57 2024-11-04 15:52:11.962 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=39, affectRows=39 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=39, validCount=39 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=60, affectRows=60 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=75, affectRows=75 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=53, affectRows=53 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706660, nodeCount=78, affectRows=78 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-56,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=59, validCount=59 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-38,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=74, validCount=74 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-36,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=53, validCount=53 2024-11-04 15:52:11.963 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706660, nodeCount=77, validCount=77 2024-11-04 15:52:11.983 INFO 21062 --- [prometheus-minute-schedule-1,e449c17a2f82cf4f,050cc08e68f37c22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:12.130 INFO 21062 --- [ocp-schedules-23,881deab416eca06b,cb135884baea94a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:52:12.130 INFO 21062 --- [ocp-schedules-23,12f41b3595345359,0581e625aeae38d6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:52:12.134 INFO 21062 --- [ocp-schedules-23,12f41b3595345359,0581e625aeae38d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:12.139 INFO 21062 --- [ocp-schedules-26,6cb61f2fede3a360,b2f743e59f31c5b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:52:12.164 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:52:12.164 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:52:12.262 INFO 21062 --- [ocp-schedules-13,aa817c8b3413dc33,8ec961e140b93f2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:52:12.687 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:12.687 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:13.688 INFO 21062 --- [ocp-schedules-3,52735611bac15499,819d729d9f660ea4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:52:13.694 INFO 21062 --- [ocp-schedules-3,52735611bac15499,819d729d9f660ea4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:52:13.694 INFO 21062 --- [ocp-schedules-3,52735611bac15499,819d729d9f660ea4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:52:13.719 INFO 21062 --- [ocp-async-9,e782e051f71d650e,418c99a6485e8c7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:13.720 WARN 21062 --- [ocp-async-1,e89e22db74fb0807,a7d8cff9cdf895d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:13.984 INFO 21062 --- [prometheus-cache-monitor1,160208a5f5ccd05a,22a70f87cf581d7b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:52:13.984 INFO 21062 --- [prometheus-cache-monitor1,160208a5f5ccd05a,22a70f87cf581d7b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:52:14.983 INFO 21062 --- [prometheus-minute-schedule-1,99cc834ef083c6e9,07ac95849f20d29e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:15.688 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:15.688 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:15.719 INFO 21062 --- [ocp-async-3,ef6966e8ea7a4792,b711ad378927dcd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:15.720 WARN 21062 --- [ocp-async-6,b1ea639dd827053f,ba4c54a94c55e9ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:17.720 INFO 21062 --- [ocp-async-7,26efe70fe54d50da,124388bf10b3fd91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:17.720 WARN 21062 --- [ocp-async-10,fcec8c256dedba96,de4d9a534ed5c761] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:17.983 INFO 21062 --- [prometheus-minute-schedule-1,3d802d80d4fea53a,8cbb559e50cd4618] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:52:18.589079+08:00, lastProcessTime=2024-11-04T15:52:08.589951+08:00 2024-11-04 15:52:18.688 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:18.689 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:18.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:52:18.589079+08:00 2024-11-04 15:52:18.828 INFO 21062 --- [ocp-async-1,bb34bfdd1a0df661,1f84396be2534fc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:52:19.720 INFO 21062 --- [ocp-async-4,c19c64866d0aa6c7,b1c208a1fc27bcd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:19.720 WARN 21062 --- [ocp-async-9,acea8a4d4aac7009,e6055bff876e2c94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:20.983 INFO 21062 --- [prometheus-minute-schedule-1,8aac0c9d38db9bab,82d04cf4c86230a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:21.689 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:21.690 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:21.720 INFO 21062 --- [ocp-async-8,82f4ac435618cc12,1e6f5f0ba5532eff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:21.720 WARN 21062 --- [ocp-async-3,800c6fbf6f74a989,0c50dd6f5ec0f428] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:22.144 INFO 21062 --- [ocp-schedules-26,7263eeb3f056672f,5c54d5e909a8cd6d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:52:22.308 INFO 21062 --- [ocp-schedules-16,246876c3ad99dc1c,8694ac0bcc521d2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:52:23.720 INFO 21062 --- [ocp-async-2,fbca47a8058c6aa8,318fd981b31e8004] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:23.721 WARN 21062 --- [ocp-async-7,6a89ed088a8c5e71,f703bf21a0fb4a29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:23.983 INFO 21062 --- [prometheus-minute-schedule-1,fb48be2daca1e293,95fea278ebd259fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:24.690 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:24.691 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:25.720 INFO 21062 --- [ocp-async-9,bfa22c56e403fb0c,f52b5d765d756639] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:25.721 WARN 21062 --- [ocp-async-1,99f9837a81c5184b,1244f6c345ea8b90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:26.983 INFO 21062 --- [prometheus-minute-schedule-1,baed4a83bf9bb509,d022569fa23d096d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:26.984 INFO 21062 --- [prometheus-hour-schedule-1,cf279ae372e8f11a,db687558001fd6a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:52:27.691 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:27.692 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:27.721 INFO 21062 --- [ocp-async-3,cd02ae9714a6e363,7010d65ccafe0fc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:27.721 WARN 21062 --- [ocp-async-6,15f59b3121e24209,8c369b048a4137d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:28.498 INFO 21062 --- [ocp-schedules-18,514b20f7683af7a1,499295a523bec341] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:52:28.500 INFO 21062 --- [ocp-schedules-19,fa4c27208030f1e8,5e42be6a2a86f2a2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:52:28.500 INFO 21062 --- [ocp-schedules-12,b9e27137b9126244,2c3eca2724543713] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:52:28.500 INFO 21062 --- [ocp-schedules-17,52f7e611e4769763,8e0a1ed38bc4e46f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:52:28.502 INFO 21062 --- [ocp-schedules-6,8eebdc348e579c5c,936675834a69fae1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:52:28.502 INFO 21062 --- [ocp-schedules-18,514b20f7683af7a1,499295a523bec341] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:52:28.503 INFO 21062 --- [ocp-schedules-19,fa4c27208030f1e8,5e42be6a2a86f2a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:52:28.504 INFO 21062 --- [ocp-schedules-22,4b2c102fb15e4739,1c722b71c9a3cb23] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:52:28.504 INFO 21062 --- [ocp-schedules-17,52f7e611e4769763,8e0a1ed38bc4e46f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:28.504 INFO 21062 --- [ocp-schedules-2,b582b5be069c03fc,d65b6f0f1ce1176d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-11-04 15:52:28.505 INFO 21062 --- [ocp-schedules-12,b9e27137b9126244,2c3eca2724543713] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:52:28.507 INFO 21062 --- [ocp-schedules-2,b582b5be069c03fc,d65b6f0f1ce1176d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:28.510 INFO 21062 --- [ocp-schedules-14,565d4bb50957d2d1,dd7f849b16f1fcb9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:52:28.510 INFO 21062 --- [ocp-schedules-16,ffe5c3c2bfc3020e,94ef8d236e8c5961] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:52:28.510 INFO 21062 --- [ocp-schedules-14,565d4bb50957d2d1,dd7f849b16f1fcb9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:52:28.510 INFO 21062 --- [ocp-schedules-16,ffe5c3c2bfc3020e,94ef8d236e8c5961] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:52:28.510 INFO 21062 --- [ocp-schedules-25,d853dcbff58cbdc9,f454dd432d425bfb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:52:28.510 INFO 21062 --- [ocp-schedules-25,d853dcbff58cbdc9,f454dd432d425bfb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:52:28.544 INFO 21062 --- [ocp-schedules-26,6f0069184f8c2e07,b32d5cce057fd5b5] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-11-04 15:52:28.544 INFO 21062 --- [ocp-schedules-26,6f0069184f8c2e07,b32d5cce057fd5b5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-11-04 15:52:28.584 INFO 21062 --- [pool-12-thread-4,4c55c58a6904beaf,f96f9ca7136d2458] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-11-04 15:52:28.584 INFO 21062 --- [pool-12-thread-1,4c55c58a6904beaf,e4e5d5c41c122673] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-11-04 15:52:28.584 INFO 21062 --- [pool-12-thread-4,4c55c58a6904beaf,f96f9ca7136d2458] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-11-04 15:52:28.585 INFO 21062 --- [pool-12-thread-2,4c55c58a6904beaf,d39e993dea2acb22] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-11-04 15:52:28.586 INFO 21062 --- [pool-12-thread-3,4c55c58a6904beaf,891eb850a7be800e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-11-04 15:52:28.595 INFO 21062 --- [ocp-schedules-28,7b6e3f1acdfa3c0e,53a7feb1f074edd3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":423,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:52:28.595 INFO 21062 --- [pool-12-thread-4,4c55c58a6904beaf,f96f9ca7136d2458] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-11-04 15:52:28.595 INFO 21062 --- [ocp-schedules-28,7b6e3f1acdfa3c0e,53a7feb1f074edd3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:52:28.595 INFO 21062 --- [pool-12-thread-1,4c55c58a6904beaf,e4e5d5c41c122673] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-11-04 15:52:28.595 INFO 21062 --- [pool-12-thread-1,4c55c58a6904beaf,e4e5d5c41c122673] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-11-04 15:52:28.596 INFO 21062 --- [pool-12-thread-4,4c55c58a6904beaf,f96f9ca7136d2458] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-11-04 15:52:28.598 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:52:28.596904+08:00, lastProcessTime=2024-11-04T15:52:18.589079+08:00 2024-11-04 15:52:28.599 INFO 21062 --- [ForkJoinPool-11-worker-14,4c55c58a6904beaf,34a79449668e5bc1] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2024-11-04 15:52:28.600 INFO 21062 --- [pool-12-thread-2,4c55c58a6904beaf,d39e993dea2acb22] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-11-04 15:52:28.600 INFO 21062 --- [ForkJoinPool-11-worker-7,4c55c58a6904beaf,fe392c1d91c74f14] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of dba_test_ob 2024-11-04 15:52:28.600 INFO 21062 --- [ForkJoinPool-11-worker-7,4c55c58a6904beaf,fe392c1d91c74f14] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=dba_test_ob, obClusterId=1730694047, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2024-11-04 15:52:28.600 INFO 21062 --- [ForkJoinPool-11-worker-7,4c55c58a6904beaf,fe392c1d91c74f14] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of dba_test_ob 2024-11-04 15:52:28.600 INFO 21062 --- [pool-12-thread-3,4c55c58a6904beaf,891eb850a7be800e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-11-04 15:52:28.769 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:52:28.596904+08:00 2024-11-04 15:52:28.828 INFO 21062 --- [ocp-async-2,59c8cb81f22e5ae2,b60bc92623a00a38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:52:29.061 INFO 21062 --- [ocp-schedules-16,5cd6f628e55aad11,994ccee43168477c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:52:29.068 INFO 21062 --- [ocp-schedules-22,2cacb81936a57f3f,9e3393cf62fda593] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-11-04 15:52:29.068 INFO 21062 --- [ocp-schedules-22,2cacb81936a57f3f,9e3393cf62fda593] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-11-04 15:52:29.068 INFO 21062 --- [ocp-schedules-22,2cacb81936a57f3f,9e3393cf62fda593] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=424 2024-11-04 15:52:29.097 INFO 21062 --- [ocp-schedules-22,2cacb81936a57f3f,9e3393cf62fda593] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=424 2024-11-04 15:52:29.436 INFO 21062 --- [ocp-schedules-1,b3184617a3f9bf31,11952dd202998f3a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-11-04 15:52:29.440 INFO 21062 --- [ocp-schedules-1,b3184617a3f9bf31,11952dd202998f3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:29.476 ERROR 21062 --- [ocp-schedules-1,b3184617a3f9bf31,11952dd202998f3a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:93) at sun.reflect.GeneratedMethodAccessor1314.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:52:29.476 INFO 21062 --- [ocp-schedules-1,b3184617a3f9bf31,11952dd202998f3a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2024-11-04 15:52:29.512 INFO 21062 --- [ocp-schedules-7,b9b49a13c052b8cf,e27004586f219e32] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:52:29.512 INFO 21062 --- [ocp-schedules-7,b9b49a13c052b8cf,e27004586f219e32] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:52:29.677 INFO 21062 --- [ocp-schedules-27,3bfbfe0591e07da3,da3c2ceec2a541f7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:52:29.686 INFO 21062 --- [ocp-schedules-5,99f47a7a48379488,bc5b04b59a48677e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:52:29.691 INFO 21062 --- [ocp-schedules-5,99f47a7a48379488,bc5b04b59a48677e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:52:29.691 INFO 21062 --- [ocp-schedules-5,99f47a7a48379488,bc5b04b59a48677e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:52:29.721 INFO 21062 --- [ocp-async-1,e84e65ad11181a70,9fd1255c1476a327] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:29.721 WARN 21062 --- [ocp-async-5,5323122dad1c832e,bc44f602a56ead6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:29.726 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-11-04 15:52:29.741 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:52:29.743 WARN 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:52:29.744 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:52:29.745 WARN 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:52:29.752 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:52:29.753 WARN 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:52:29.754 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:52:29.755 WARN 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:52:29.761 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:52:29.763 WARN 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:52:29.764 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:52:29.765 WARN 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:52:29.770 INFO 21062 --- [ocp-schedules-8,a1049d65833db395,375f297466884886] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=3 2024-11-04 15:52:29.983 INFO 21062 --- [prometheus-minute-schedule-1,2e887498f8f8a9d4,38d1562096919700] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:30.046 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:52:30.046 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:52:30.053 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:52:30.053 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:52:30.053 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:52:30.063 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:52:30.063 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:52:30.063 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:52:30.067 INFO 21062 --- [ocp-schedules-21,30d523a3c9c8bfa1,4dbfd561aa02a2f3] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:52:30.231 INFO 21062 --- [ocp-schedules-3,b4ce969050eaa6a9,7629aea0c9254a4d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:52:30.231 INFO 21062 --- [ocp-schedules-3,b4ce969050eaa6a9,7629aea0c9254a4d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:52:30.232 INFO 21062 --- [ocp-schedules-3,b4ce969050eaa6a9,7629aea0c9254a4d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:52:30.232 INFO 21062 --- [ocp-schedules-3,b4ce969050eaa6a9,7629aea0c9254a4d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:52:30.233 INFO 21062 --- [ocp-schedules-3,b4ce969050eaa6a9,7629aea0c9254a4d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:52:30.233 INFO 21062 --- [ocp-schedules-3,b4ce969050eaa6a9,7629aea0c9254a4d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:52:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:52:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:52:30.692 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:30.693 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:30.777 INFO 21062 --- [ocp-schedules-20,5a883471533faa4d,702d03c76f9da813] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-11-04 15:52:30.777 INFO 21062 --- [ocp-schedules-20,5a883471533faa4d,702d03c76f9da813] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-11-04 15:52:30.805 INFO 21062 --- [pool-59-thread-1,5a883471533faa4d,362695fba14df44d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-11-04 15:52:30.833 WARN 21062 --- [ocp-schedules-20,5a883471533faa4d,702d03c76f9da813] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:124) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$d5b41fa1.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:52:30.833 INFO 21062 --- [ocp-schedules-20,5a883471533faa4d,702d03c76f9da813] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-11-04 15:52:31.721 INFO 21062 --- [ocp-async-6,9aec5418b762ac30,0af1bb5fe1c3afea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:31.721 WARN 21062 --- [ocp-async-3,5fee3b8b3d876538,1054d3b85eb60054] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:32.147 INFO 21062 --- [ocp-schedules-30,329bdf205f6d2299,e383cdf6c072f7d7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:52:32.335 INFO 21062 --- [ocp-schedules-9,36701942c839969c,e82af20af1ca6166] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:52:32.983 INFO 21062 --- [prometheus-minute-schedule-1,a8df9d0a9fd1d480,290321262f58755d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:33.693 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:33.694 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:33.695 INFO 21062 --- [ocp-schedules-21,f2f89e43ea4fdac0,c0a4f9b33f382fc4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:52:33.701 INFO 21062 --- [ocp-schedules-21,f2f89e43ea4fdac0,c0a4f9b33f382fc4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:52:33.701 INFO 21062 --- [ocp-schedules-21,f2f89e43ea4fdac0,c0a4f9b33f382fc4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:52:33.721 INFO 21062 --- [ocp-async-8,cabc92c18387a24c,f77e400fe5b28cf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:33.721 WARN 21062 --- [ocp-async-7,a77b316f4b1764ee,6e0efc4512d5a70e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:34.040 INFO 21062 --- [ocp-schedules-18,941f0450601aaf1c,20b2f97ab0d07362] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:52:34.040 INFO 21062 --- [ocp-schedules-18,941f0450601aaf1c,20b2f97ab0d07362] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:52:34.067 INFO 21062 --- [ocp-schedules-18,941f0450601aaf1c,20b2f97ab0d07362] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:52:34.067 INFO 21062 --- [ocp-schedules-18,941f0450601aaf1c,20b2f97ab0d07362] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:52:34.067 INFO 21062 --- [ocp-schedules-25,e01c953786bf2f10,155e382b80404b49] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:52:34.245 INFO 21062 --- [ocp-schedules-30,14f7a97d2393cf80,17d91a47be051e20] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:52:34.245+08:00, clusters=[1] 2024-11-04 15:52:34.246 INFO 21062 --- [ocp-schedules-30,14f7a97d2393cf80,17d91a47be051e20] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:52:34.629 INFO 21062 --- [ocp-schedules-7,cb9296012779f8d9,7bf63dc9f8115f1a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:52:34.629+08:00, clusters=[1] 2024-11-04 15:52:34.633 INFO 21062 --- [ocp-schedules-7,cb9296012779f8d9,7bf63dc9f8115f1a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:52:34.732 INFO 21062 --- [ocp-schedules-28,d38a54d644556d28,4b10975d0163e2bd] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:52:34.737 INFO 21062 --- [ocp-schedules-28,d38a54d644556d28,4b10975d0163e2bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:34.796 ERROR 21062 --- [ocp-schedules-28,d38a54d644556d28,4b10975d0163e2bd] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:52:34.796 INFO 21062 --- [ocp-schedules-28,d38a54d644556d28,4b10975d0163e2bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:52:35.721 INFO 21062 --- [ocp-async-5,5e16cf384a98cc29,597aa0218ce11852] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:35.722 WARN 21062 --- [ocp-async-2,4aaed42eb50468e6,de95f12288d02e3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:35.983 INFO 21062 --- [prometheus-minute-schedule-1,26de66f92893aae6,e94ace1ff6db25f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:36.694 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:36.695 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:36.818 INFO 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:52:36.818 INFO 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:52:36.818 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:52:36.818 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:52:36.828 INFO 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:52:36.828 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:52:36.829 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:52:36.829 WARN 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:52:36.830 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:52:36.830 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:52:36.830 INFO 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:52:36.830 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:52:36.830 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:52:36.831 WARN 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:52:36.831 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:52:36.832 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:52:36.838 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:52:36.838 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:52:36.838 INFO 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:52:36.850 ERROR 21062 --- [ocp-schedules-6,78090f04c0adb794,cfd01f93aa2d0188] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.11:62889 [/10.46.128.11] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:52:37.721 INFO 21062 --- [ocp-async-3,3020c9e3387ad7be,be88b37deda80602] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:37.722 WARN 21062 --- [ocp-async-9,e9da09f569aadcd7,8aff808b44a299e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:38.504 INFO 21062 --- [ocp-schedules-19,fa4c27208030f1e8,5e42be6a2a86f2a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:52:38.504 INFO 21062 --- [ocp-schedules-17,52f7e611e4769763,8e0a1ed38bc4e46f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:52:38.508 INFO 21062 --- [ocp-schedules-2,b582b5be069c03fc,d65b6f0f1ce1176d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2024-11-04 15:52:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:52:38.589469+08:00, lastProcessTime=2024-11-04T15:52:28.596904+08:00 2024-11-04 15:52:38.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:52:38.589469+08:00 2024-11-04 15:52:38.829 INFO 21062 --- [ocp-async-4,64c24938a3efabc2,a18c4685bfb48cd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:52:38.983 INFO 21062 --- [prometheus-minute-schedule-1,ac29bafb8234a49c,36eb32796f7a6811] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:39.695 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:39.696 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:39.722 INFO 21062 --- [ocp-async-8,8002de2d6d513dbf,ccfff52a4b86172c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:39.722 WARN 21062 --- [ocp-async-2,09cbd1ed808b5ff4,e49177cae35643f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:40.003 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:52:40.004 INFO 21062 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:52:41.017 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:52:41.017 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:52:41.025 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:52:41.025 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:52:41.025 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:52:41.025 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:52:41.030 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:52:41.030 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:52:41.030 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:52:41.030 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:52:41.067 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:52:41.076 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:52:41.076 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:52:41.076 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:52:41.076 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:52:41.076 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:52:41.084 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:52:41.614 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:52:41.722 INFO 21062 --- [ocp-async-5,3352147c379a73e1,e8cd7c36ddbd333e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:41.722 WARN 21062 --- [ocp-async-9,8bf9b1e90f2d6121,f140c9b357a6db51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:41.797 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:52:41.825 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:52:41.825 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=28ms, period=120000ms 2024-11-04 15:52:41.983 INFO 21062 --- [prometheus-minute-schedule-1,1e0de7941be8fd29,3170feb087dda5fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:42.150 INFO 21062 --- [ocp-schedules-19,abfc74081f8fadb9,c0ca281f982288aa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:52:42.164 INFO 21062 --- [ocp-schedules-23,12f41b3595345359,0581e625aeae38d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:52:42.165 INFO 21062 --- [ocp-schedules-23,adf7c3c86b4c003d,497435181ad6467c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:52:42.168 INFO 21062 --- [ocp-schedules-23,adf7c3c86b4c003d,497435181ad6467c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:42.231 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.200403338S 2024-11-04 15:52:42.231 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:52:42.235 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:52:42.235 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:52:42.332 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:52:42.332 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:52:42.363 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:52:42.376 INFO 21062 --- [ocp-schedules-24,b2e994c0ade0f009,daaea6ada0eefebe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:52:42.436 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:52:42.436 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:52:42.436 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:52:42.436 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:52:42.436 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:52:42.437 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:52:42.448 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:52:42.627 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] o.s.boot.SpringApplication : Started application in 1.602 seconds (JVM running for 12176.306) 2024-11-04 15:52:42.627 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.60268814S 2024-11-04 15:52:42.696 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:42.697 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:43.074 INFO 21062 --- [ocp-schedules-4,cd1b34ce73e7d781,fa6ff93818960b23] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:52:43.722 INFO 21062 --- [ocp-async-1,7211fcd2ac326781,187bf0c7abb2a57c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:43.722 WARN 21062 --- [ocp-async-3,c0ce9198fe710073,39d95f07dfe7c6d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:44.983 INFO 21062 --- [prometheus-minute-schedule-1,02f002f210da7977,530f4050d3df428a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:45.697 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:45.697 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:45.722 INFO 21062 --- [ocp-async-7,030aaa1201e178ad,8251fea0ceb0238e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:45.722 WARN 21062 --- [ocp-async-2,03cdb458acf78c30,a84c5218d857f9f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:47.013 INFO 21062 --- [ocp-schedules-6,b7649114287dcb22,c3b47dc4d0a1f30e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:52:47.016 INFO 21062 --- [ocp-schedules-6,b7649114287dcb22,c3b47dc4d0a1f30e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:47.100 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:52:47.102 ERROR 21062 --- [ocp-schedules-6,b7649114287dcb22,c3b47dc4d0a1f30e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:52:47.102 INFO 21062 --- [ocp-schedules-6,b7649114287dcb22,c3b47dc4d0a1f30e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:52:47.722 INFO 21062 --- [ocp-async-8,b9e4158f37b1ec04,6621c8ae1679f9b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:47.722 WARN 21062 --- [ocp-async-10,b916469c857d38a0,d8cb75e4a6d1e705] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:47.983 INFO 21062 --- [prometheus-minute-schedule-1,400440942edb495b,b2786e71413acc90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:52:48.590048+08:00, lastProcessTime=2024-11-04T15:52:38.589469+08:00 2024-11-04 15:52:48.698 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:48.698 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:48.748 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:52:48.590048+08:00 2024-11-04 15:52:48.829 INFO 21062 --- [ocp-async-3,4ce75c22c8fb7068,b1c57eb0a3feeed1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:52:49.722 INFO 21062 --- [ocp-async-3,9caf9f8d615a3d1c,1c4f1d215871fd05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:49.723 WARN 21062 --- [ocp-async-6,6bde9dade277a911,b29820147d3eab24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:50.983 INFO 21062 --- [prometheus-minute-schedule-1,b727191eba277366,40511f514794b53c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:51.698 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:51.699 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:51.723 INFO 21062 --- [ocp-async-1,95412e6f1018e515,8ebeb93c72ab2dea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:51.723 WARN 21062 --- [ocp-async-4,0357b7489a82fb17,c8ed921d83ff2f35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:52.152 INFO 21062 --- [ocp-schedules-2,4e51614db8bfabcf,be8b890d2648f3e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:52:52.413 INFO 21062 --- [ocp-schedules-30,db5294a1a73e5643,4838bb520473775a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:52:53.704 INFO 21062 --- [ocp-schedules-17,f279510f2424be33,947e1cf6fd6626fe] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:52:53.710 INFO 21062 --- [ocp-schedules-17,f279510f2424be33,947e1cf6fd6626fe] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:52:53.710 INFO 21062 --- [ocp-schedules-17,f279510f2424be33,947e1cf6fd6626fe] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:52:53.723 WARN 21062 --- [ocp-async-7,d35cf6b48b95354e,80cbce4ef2558167] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:53.723 INFO 21062 --- [ocp-async-9,c3fa3ed5b62bb5be,4f6b96fd4de9b208] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:53.983 INFO 21062 --- [prometheus-minute-schedule-1,bc30f6ce1bf2424d,2d39abd4507cd2ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:54.699 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:54.700 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:55.723 WARN 21062 --- [ocp-async-6,1a577c443ab2b531,5d3a36b63d520478] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:55.723 INFO 21062 --- [ocp-async-5,bcd068840cb19f4f,eb6b7c76356cf763] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:56.983 INFO 21062 --- [prometheus-minute-schedule-1,162720356f5b5d12,c8b4dc171efa3c8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:52:57.700 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:52:57.701 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:52:57.723 WARN 21062 --- [ocp-async-4,1df840a376260377,0709b6debd03b27b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:57.723 INFO 21062 --- [ocp-async-4,a08453adf2e04e34,c50a12c94c3f0ea8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:58.500 INFO 21062 --- [ocp-schedules-2,f6b07848dde04589,1b7fbeb7b31c5021] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:52:58.500 INFO 21062 --- [ocp-schedules-4,fa7c789c7d74f020,88adce23d1e77eac] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:52:58.502 INFO 21062 --- [ocp-schedules-14,d81f97ead3e28dfc,d2d91ebc92d2825d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:52:58.503 INFO 21062 --- [ocp-schedules-2,f6b07848dde04589,1b7fbeb7b31c5021] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:52:58.504 INFO 21062 --- [ocp-schedules-4,fa7c789c7d74f020,88adce23d1e77eac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:58.505 INFO 21062 --- [ocp-schedules-18,3eaee49c8f1f3307,273709a479995e70] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:52:58.509 INFO 21062 --- [ocp-schedules-7,6d1abe3891334f34,6e9587f676b6e737] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:52:58.509 INFO 21062 --- [ocp-schedules-7,6d1abe3891334f34,6e9587f676b6e737] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:52:58.509 INFO 21062 --- [ocp-schedules-11,1ad0328723e33f1f,d4653c5f572b8255] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:52:58.509 INFO 21062 --- [ocp-schedules-11,1ad0328723e33f1f,d4653c5f572b8255] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:52:58.510 INFO 21062 --- [ocp-schedules-15,2226780a651386f2,ae0a45348bb55963] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:52:58.510 INFO 21062 --- [ocp-schedules-15,2226780a651386f2,ae0a45348bb55963] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:52:58.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:52:58.590780+08:00, lastProcessTime=2024-11-04T15:52:48.590048+08:00 2024-11-04 15:52:58.748 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:52:58.590780+08:00 2024-11-04 15:52:58.829 INFO 21062 --- [ocp-async-7,506b0d5a066cd26c,ef1846e1a6bce706] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:52:59.435 INFO 21062 --- [ocp-schedules-6,678a55c61df77b00,e8015917985f8d92] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:52:59.439 INFO 21062 --- [ocp-schedules-6,678a55c61df77b00,e8015917985f8d92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:52:59.439 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:52:59.680 INFO 21062 --- [ocp-schedules-11,0d190599c54b3d28,ca9a8c69ffae7c86] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:52:59.723 WARN 21062 --- [ocp-async-10,459b1d0a43b0e993,3a21cb27cda32b84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:52:59.723 INFO 21062 --- [ocp-async-9,376db926bb995e93,6cc92ce8c6c2ed2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:52:59.983 INFO 21062 --- [prometheus-minute-schedule-1,44c0abae6f4b7ffe,d1f8737feb15382e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706780, interval=60 2024-11-04 15:53:00.701 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:00.701 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:01.723 WARN 21062 --- [ocp-async-8,d5f1323fef9fea0e,40a8a69cdd3c1080] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:01.723 INFO 21062 --- [ocp-async-5,220960b11ce4ae76,34f199138f4c3dde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:02.156 INFO 21062 --- [ocp-schedules-5,748df5590715aced,6286d62fa1649b09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:53:02.438 INFO 21062 --- [ocp-schedules-21,a7727f3f60a75ec3,665c5c8f8cc4240e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:53:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f2e2f181027a0c2b,75712acd427a7ccb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:03.702 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:03.702 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:03.724 INFO 21062 --- [ocp-async-4,0ff96f9b62430687,460e6e9efe88b340] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:03.724 WARN 21062 --- [ocp-async-3,e3b270b6e9d012c6,2702ebc883dff4d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:05.724 WARN 21062 --- [ocp-async-10,f1c5bb426de2fd7a,390e89ff0d24146c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:05.724 INFO 21062 --- [ocp-async-7,73e786525f81c45b,22424bbc45d46049] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:05.983 INFO 21062 --- [prometheus-minute-schedule-1,d227230996a790a3,4878e292895aa04a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:06.702 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:06.703 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:07.724 INFO 21062 --- [ocp-async-8,b01244fd34cb8895,9c24d0ed86138872] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:07.724 WARN 21062 --- [ocp-async-6,39a52de80254667f,227cefdee9dde1d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:08.504 INFO 21062 --- [ocp-schedules-2,f6b07848dde04589,1b7fbeb7b31c5021] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:53:08.505 INFO 21062 --- [ocp-schedules-4,fa7c789c7d74f020,88adce23d1e77eac] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:53:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:53:08.589129+08:00, lastProcessTime=2024-11-04T15:52:58.590780+08:00 2024-11-04 15:53:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:53:08.589129+08:00 2024-11-04 15:53:08.829 INFO 21062 --- [ocp-async-3,5118c3024304108d,eb73a537c8367f55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:53:08.983 INFO 21062 --- [prometheus-minute-schedule-1,a486cd4fe7586e59,0d8e0e6d16b8c4ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:09.440 INFO 21062 --- [ocp-schedules-6,678a55c61df77b00,e8015917985f8d92] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:53:09.703 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:09.704 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:09.724 INFO 21062 --- [ocp-async-2,0bd220d31736d472,11833f547ac940e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:09.724 WARN 21062 --- [ocp-async-4,f240ed3f08aa9bfa,f019d456f33f63c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:10.004 INFO 21062 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:53:10.005 INFO 21062 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:53:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:53:10.301 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:53:10.302 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:53:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:53:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:53:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:53:11.724 WARN 21062 --- [ocp-async-7,5f639c4cd97e4edd,1009f9224c6314ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:11.724 INFO 21062 --- [ocp-async-1,19e52b4151552722,1ddd0c8e649ae7d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:11.964 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=6, affectRows=6 2024-11-04 15:53:11.964 INFO 21062 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=6, validCount=6 2024-11-04 15:53:11.965 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=28, affectRows=28 2024-11-04 15:53:11.965 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=57, affectRows=57 2024-11-04 15:53:11.965 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=34, affectRows=34 2024-11-04 15:53:11.965 INFO 21062 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=28, validCount=28 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=15, affectRows=15 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=50, affectRows=50 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=34, validCount=34 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=15, validCount=15 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=57, validCount=57 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=73, affectRows=73 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=50, validCount=50 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=72, affectRows=72 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706720, nodeCount=89, affectRows=89 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-40,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=72, validCount=72 2024-11-04 15:53:11.966 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=71, validCount=71 2024-11-04 15:53:11.967 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706720, nodeCount=88, validCount=88 2024-11-04 15:53:11.983 INFO 21062 --- [prometheus-minute-schedule-1,313798dacd13d8b5,ba66d620f9dcd6b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:12.159 INFO 21062 --- [ocp-schedules-17,cb30abc2a61cb677,3692f33b6e8d3df0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:53:12.236 INFO 21062 --- [ocp-schedules-23,adf7c3c86b4c003d,497435181ad6467c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:53:12.236 INFO 21062 --- [ocp-schedules-23,eecb35b299104c3f,4f9c263a1deffdeb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:53:12.239 INFO 21062 --- [ocp-schedules-23,eecb35b299104c3f,4f9c263a1deffdeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:53:12.268 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:53:12.268 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:53:12.463 INFO 21062 --- [ocp-schedules-2,bd5b9aa097e8b2c4,8c6bb1e5689c68ed] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:53:12.704 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:12.705 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:13.713 INFO 21062 --- [ocp-schedules-4,42a1baadace5cec2,3fcf9e30b8a4f9a0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:53:13.719 INFO 21062 --- [ocp-schedules-4,42a1baadace5cec2,3fcf9e30b8a4f9a0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:53:13.719 INFO 21062 --- [ocp-schedules-4,42a1baadace5cec2,3fcf9e30b8a4f9a0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:53:13.724 INFO 21062 --- [ocp-async-9,a69b7d00b5175ab5,d6b065e38834ccb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:13.724 WARN 21062 --- [ocp-async-8,52cfa814a07f9e54,3bf59af528da634f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:13.984 INFO 21062 --- [prometheus-cache-monitor1,67efee8b59d42fcc,e56a2f4e5b8941ce] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:53:13.984 INFO 21062 --- [prometheus-cache-monitor1,67efee8b59d42fcc,e56a2f4e5b8941ce] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:53:14.575 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-11-04T15:47:14.557, end=2024-11-04T15:52:14.572} 2024-11-04 15:53:14.575 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-11-04T15:47:14.557, end=2024-11-04T15:52:14.573} 2024-11-04 15:53:14.585 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2024-11-04 15:53:14.585 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2024-11-04 15:53:14.585 INFO 21062 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:53:14.585 INFO 21062 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2024-11-04 15:53:14.983 INFO 21062 --- [prometheus-minute-schedule-1,28ac2dbcfc5dfdf3,46cab605a1cb15b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:15.705 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:15.706 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:15.725 INFO 21062 --- [ocp-async-4,1ebe1d9608e7b6e0,2c11823c0ab5d80d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:15.725 WARN 21062 --- [ocp-async-3,f30db3f54b280ea3,b5c8c9b19682f5e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:15.868 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:49:15.847, end=2024-11-04T15:51:15.866} 2024-11-04 15:53:15.872 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:53:15.872 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:53:17.725 INFO 21062 --- [ocp-async-7,a16b1680f1b2a6cc,12db0cd1d3ede16e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:17.725 WARN 21062 --- [ocp-async-10,b2fbe527643b3556,16492958d5821982] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:17.983 INFO 21062 --- [prometheus-minute-schedule-1,62d5339efb32d761,daca5699120aff9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:53:18.588557+08:00, lastProcessTime=2024-11-04T15:53:08.589129+08:00 2024-11-04 15:53:18.706 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:18.707 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:18.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:53:18.588557+08:00 2024-11-04 15:53:18.829 INFO 21062 --- [ocp-async-8,924f0ba041fbe10d,61759f99771e576c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:53:19.725 INFO 21062 --- [ocp-async-6,07dd386f6351372a,a3f06e55e296985d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:19.725 WARN 21062 --- [ocp-async-9,87ba952dcc5e89da,b1d88854c0a2ffed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:20.983 INFO 21062 --- [prometheus-minute-schedule-1,5a18e27a3e14726f,df375d7f268941d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:21.707 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:21.707 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:21.725 WARN 21062 --- [ocp-async-5,07f026288f99f6b9,8c1348500531a431] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:21.725 INFO 21062 --- [ocp-async-4,395ac93972f4e6d6,a2e50f1dc52dd4ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:22.162 INFO 21062 --- [ocp-schedules-27,fd08b4e9dfd6b67d,a580ff4223561e4b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:53:22.509 INFO 21062 --- [ocp-schedules-28,81d195d88830620d,5d43ba7cb3633e7e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:53:23.725 WARN 21062 --- [ocp-async-2,6155b822b4f0046c,2fa0158c25541862] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:23.725 INFO 21062 --- [ocp-async-7,2c0381a5cdf162dc,ff8a84329c0e3cd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:23.983 INFO 21062 --- [prometheus-minute-schedule-1,d2cdc86d74f6ceb5,9613c13cee709a29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:24.708 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:24.708 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:25.725 WARN 21062 --- [ocp-async-9,44c0ced446a1a48f,fbdb80571f32101a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:25.725 INFO 21062 --- [ocp-async-9,5a363452ccd498bf,d7b2b75aedbcb36b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:26.983 INFO 21062 --- [prometheus-minute-schedule-1,f3bd4f4a8c7c1d4c,d0c05f9288c74b18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:26.984 INFO 21062 --- [prometheus-hour-schedule-1,ac5db48a13cd3449,733b2010d1020c45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:53:27.709 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:27.709 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:27.725 WARN 21062 --- [ocp-async-5,5bb6f62e60e27eac,92d093558c017936] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:27.726 INFO 21062 --- [ocp-async-3,ffd7b049e4ab1b83,bba4ff3fa216e82a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:28.498 INFO 21062 --- [ocp-schedules-27,eddf1c23f760dc1d,d20ab8d36658d8d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:53:28.499 INFO 21062 --- [ocp-schedules-15,ae07fc04f49e7c4b,2999b9e7fa394514] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:53:28.499 INFO 21062 --- [ocp-schedules-7,f20af4a187d42ebb,042bd9bacec3b33d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:53:28.500 INFO 21062 --- [ocp-schedules-16,397fad23034291ab,063ed4420d001ee5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:53:28.502 INFO 21062 --- [ocp-schedules-20,6f88361464eb840b,d377875027a06525] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:53:28.503 INFO 21062 --- [ocp-schedules-27,eddf1c23f760dc1d,d20ab8d36658d8d0] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:53:28.503 INFO 21062 --- [ocp-schedules-15,ae07fc04f49e7c4b,2999b9e7fa394514] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:53:28.504 INFO 21062 --- [ocp-schedules-16,397fad23034291ab,063ed4420d001ee5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:53:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:53:28.504 INFO 21062 --- [ocp-schedules-4,568b328c92e7e279,a8c5630df0eea960] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:53:28.505 INFO 21062 --- [ocp-schedules-7,f20af4a187d42ebb,042bd9bacec3b33d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:53:28.509 INFO 21062 --- [ocp-schedules-18,391fac87c0895cd9,fd5a796d17ea454a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:53:28.509 INFO 21062 --- [ocp-schedules-24,61087b17ff776d6a,5bd4e641bbcd3e29] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:53:28.509 INFO 21062 --- [ocp-schedules-18,391fac87c0895cd9,fd5a796d17ea454a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:53:28.509 INFO 21062 --- [ocp-schedules-3,5718c0d2c13ab0f8,83cdeb4b4cfe4a42] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:53:28.509 INFO 21062 --- [ocp-schedules-3,5718c0d2c13ab0f8,83cdeb4b4cfe4a42] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:53:28.509 INFO 21062 --- [ocp-schedules-24,61087b17ff776d6a,5bd4e641bbcd3e29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:53:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:53:28.589036+08:00, lastProcessTime=2024-11-04T15:53:18.588557+08:00 2024-11-04 15:53:28.611 INFO 21062 --- [ocp-schedules-8,0af9b9206911308a,81048ccaffbd372a] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-11-04 15:53:28.611 INFO 21062 --- [ocp-schedules-8,0af9b9206911308a,81048ccaffbd372a] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-11-04 15:53:28.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:53:28.589036+08:00 2024-11-04 15:53:28.829 INFO 21062 --- [ocp-async-7,2641762b42736365,b788a823a6c3fdfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:53:29.063 INFO 21062 --- [ocp-schedules-7,3df8571777ebad02,ff7d195f0e67aab7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:53:29.516 INFO 21062 --- [ocp-schedules-27,271ab529005b665d,678537ec3af255be] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:53:29.516 INFO 21062 --- [ocp-schedules-27,271ab529005b665d,678537ec3af255be] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:53:29.683 INFO 21062 --- [ocp-schedules-17,a512053b6e11e43b,a49cb64836fb7a94] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:53:29.691 INFO 21062 --- [ocp-schedules-10,e38d874a46783618,654591b959282d82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:53:29.697 INFO 21062 --- [ocp-schedules-10,e38d874a46783618,654591b959282d82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:53:29.697 INFO 21062 --- [ocp-schedules-10,e38d874a46783618,654591b959282d82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:53:29.726 WARN 21062 --- [ocp-async-8,3c491f32a89ff4c5,b37313212957ad1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:29.726 INFO 21062 --- [ocp-async-1,f4f33b0d86db4ec9,762dc797b4d97ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:29.983 INFO 21062 --- [prometheus-minute-schedule-1,7e3acb714094f0cd,810ffd7984d86da4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:30.234 INFO 21062 --- [ocp-schedules-21,5b3448b82c0af5f0,98ea0109fc5902ad] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:53:30.234 INFO 21062 --- [ocp-schedules-21,5b3448b82c0af5f0,98ea0109fc5902ad] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:53:30.241 INFO 21062 --- [ocp-schedules-21,5b3448b82c0af5f0,98ea0109fc5902ad] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:53:30.241 INFO 21062 --- [ocp-schedules-21,5b3448b82c0af5f0,98ea0109fc5902ad] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:53:30.245 INFO 21062 --- [ocp-schedules-21,5b3448b82c0af5f0,98ea0109fc5902ad] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:53:30.245 INFO 21062 --- [ocp-schedules-21,5b3448b82c0af5f0,98ea0109fc5902ad] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:53:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:53:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:53:30.709 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:30.710 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:31.726 WARN 21062 --- [ocp-async-3,945e6c42baa03035,db0d6b4c72ff46f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:31.726 INFO 21062 --- [ocp-async-6,d4e19ec379a21cdb,b2d9ec609d321319] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:32.165 INFO 21062 --- [ocp-schedules-12,0bf52d3710e437b4,648c7d60e0563834] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:53:32.534 INFO 21062 --- [ocp-schedules-17,de8fa270756674c9,27df2560e71257e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:53:32.983 INFO 21062 --- [prometheus-minute-schedule-1,c8e755821e8e01ea,d7e525ef160134e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:33.710 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:33.711 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:33.721 INFO 21062 --- [ocp-schedules-10,a7696b4d4909b706,37d7235df2ce0e2a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:53:33.726 WARN 21062 --- [ocp-async-4,7b4707e98d748d72,84474d502c1e180e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:33.726 INFO 21062 --- [ocp-schedules-10,a7696b4d4909b706,37d7235df2ce0e2a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:53:33.726 INFO 21062 --- [ocp-schedules-10,a7696b4d4909b706,37d7235df2ce0e2a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:53:33.726 INFO 21062 --- [ocp-async-4,d1c04a96e46a6a93,a01db2b3715d6753] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:34.067 INFO 21062 --- [ocp-schedules-30,7cab720ca71f7a5d,8c3a707acb959566] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:53:34.067 INFO 21062 --- [ocp-schedules-30,7cab720ca71f7a5d,8c3a707acb959566] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:53:34.094 INFO 21062 --- [ocp-schedules-30,7cab720ca71f7a5d,8c3a707acb959566] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:53:34.094 INFO 21062 --- [ocp-schedules-30,7cab720ca71f7a5d,8c3a707acb959566] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:53:34.095 INFO 21062 --- [ocp-schedules-6,ff948bc0bdbee6b8,bcd293616b11e990] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:53:34.272 INFO 21062 --- [ocp-schedules-12,916200b20de64563,af80c34124b50a1d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:53:34.272+08:00, clusters=[1] 2024-11-04 15:53:34.274 INFO 21062 --- [ocp-schedules-12,916200b20de64563,af80c34124b50a1d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:53:34.660 INFO 21062 --- [ocp-schedules-20,f3a8f6f068f412f5,405eeb53bacedd5c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:53:34.660+08:00, clusters=[1] 2024-11-04 15:53:34.664 INFO 21062 --- [ocp-schedules-20,f3a8f6f068f412f5,405eeb53bacedd5c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:53:35.726 WARN 21062 --- [ocp-async-8,1ce5e6739127557b,f506e9bc7dec6a43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:35.726 INFO 21062 --- [ocp-async-7,308862fb760fb19a,8ec80ea9ab11c088] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:35.983 INFO 21062 --- [prometheus-minute-schedule-1,d9dfd8b93c002c3c,d83f05b9e56f1a7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:36.711 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:36.712 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:36.859 INFO 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:53:36.859 INFO 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:53:36.859 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:53:36.859 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:53:36.869 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:53:36.869 INFO 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:53:36.870 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:53:36.871 WARN 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:53:36.871 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:53:36.871 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:53:36.871 INFO 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:53:36.871 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:53:36.872 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:53:36.872 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:53:36.872 WARN 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:53:36.873 WARN 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:53:36.879 INFO 21062 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:53:36.879 INFO 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:53:36.879 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:53:36.894 ERROR 21062 --- [ocp-schedules-27,b8034173b01d59d0,3a29d9e4a1f39a60] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:53:37.726 INFO 21062 --- [ocp-async-9,a03d9155b60a7b32,252c8d021e2fcd2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:37.727 WARN 21062 --- [ocp-async-3,762ba29e10704037,bf7a0bf5228b527e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:38.504 INFO 21062 --- [ocp-schedules-15,ae07fc04f49e7c4b,2999b9e7fa394514] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:53:38.504 INFO 21062 --- [ocp-schedules-16,397fad23034291ab,063ed4420d001ee5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:53:38.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:53:38.588677+08:00, lastProcessTime=2024-11-04T15:53:28.589036+08:00 2024-11-04 15:53:38.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:53:38.588677+08:00 2024-11-04 15:53:38.830 INFO 21062 --- [ocp-async-5,ee5616821caacad6,56a214d5a13fe33f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:53:38.983 INFO 21062 --- [prometheus-minute-schedule-1,0f654546fd6dd370,447b41212ffb8a5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:39.712 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:39.713 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:39.727 WARN 21062 --- [ocp-async-4,6ce8a5ea2871dfe6,fb9350fd1dacc748] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:39.727 INFO 21062 --- [ocp-async-8,b99f816ec4c6fbcf,384281f6c717358a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:40.002 INFO 21062 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:53:40.007 INFO 21062 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:53:41.727 WARN 21062 --- [ocp-async-7,7d27a206c37a5a0e,6a667dfce377693c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:41.727 INFO 21062 --- [ocp-async-10,7ec7377707c56788,c7acf6ed1d9f5620] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:41.983 INFO 21062 --- [prometheus-minute-schedule-1,1086ea761fd9d6b0,8ec6a383d89529bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:42.168 INFO 21062 --- [ocp-schedules-18,b9ae2bcec4f2c93d,0c03741cc1262fbf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:53:42.269 INFO 21062 --- [ocp-schedules-23,eecb35b299104c3f,4f9c263a1deffdeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:53:42.269 INFO 21062 --- [ocp-schedules-23,d7af155521c1c956,d92f093cae0c618a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:53:42.273 INFO 21062 --- [ocp-schedules-23,d7af155521c1c956,d92f093cae0c618a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:53:42.303 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:53:42.303 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:53:42.614 INFO 21062 --- [ocp-schedules-28,12293eddb1ab4d43,63bfb0c1448f5825] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:53:42.713 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:42.713 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:43.727 INFO 21062 --- [ocp-async-1,73e176e9dea32b89,bdf4433b92f37c89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:43.727 WARN 21062 --- [ocp-async-9,b0eda10c62bd1fca,5f2c1400491a6c42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:44.983 INFO 21062 --- [prometheus-minute-schedule-1,f57cb14d2897f5b1,ad57d6fda5071bf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:45.714 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:45.714 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:45.727 WARN 21062 --- [ocp-async-5,2743f5fb38377154,d1d3a77e974d2863] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:45.727 INFO 21062 --- [ocp-async-4,06241d5da8f5a832,7686d612f4beddf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:47.102 INFO 21062 --- [ocp-schedules-4,a79a15673f91dbbb,f3fa874af83c336b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:53:47.106 INFO 21062 --- [ocp-schedules-4,a79a15673f91dbbb,f3fa874af83c336b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:53:47.191 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:53:47.193 ERROR 21062 --- [ocp-schedules-4,a79a15673f91dbbb,f3fa874af83c336b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:53:47.193 INFO 21062 --- [ocp-schedules-4,a79a15673f91dbbb,f3fa874af83c336b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:53:47.728 WARN 21062 --- [ocp-async-8,7885ce0dc4d34856,0ade80269475fe6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:47.728 INFO 21062 --- [ocp-async-7,7f2edd1b67b4a98c,91a584ec14c0dc3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:47.983 INFO 21062 --- [prometheus-minute-schedule-1,59ef79d9d86024b9,a319aea78917c210] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:48.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:53:48.589168+08:00, lastProcessTime=2024-11-04T15:53:38.588677+08:00 2024-11-04 15:53:48.714 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:48.715 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:48.770 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:53:48.589168+08:00 2024-11-04 15:53:48.830 INFO 21062 --- [ocp-async-9,79d7b73ecfc36e13,bdcb16ff8602cdc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:53:49.728 WARN 21062 --- [ocp-async-10,832aae3141fa68ad,5104816ff3aee0ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:49.728 INFO 21062 --- [ocp-async-6,7b959bfa17a8faed,db854338d253558e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:50.983 INFO 21062 --- [prometheus-minute-schedule-1,412305b300060b09,c80096d9dd1f9d2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:51.715 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:51.716 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:51.728 WARN 21062 --- [ocp-async-1,b1a3953c752baa98,98f6b2f2e32320f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:51.728 INFO 21062 --- [ocp-async-1,b16ffddcd993191a,eff20daccc6a4bed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:52.170 INFO 21062 --- [ocp-schedules-24,823b78574f0cd398,3cb26425a87adfa0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:53:52.639 INFO 21062 --- [ocp-schedules-22,8609f14200586482,9429f74eea5b8634] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:53:53.728 WARN 21062 --- [ocp-async-4,bd1bd82ada7e0b97,d62745a6963851a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:53.728 INFO 21062 --- [ocp-async-7,3b77e412f3bae8f9,48e69ba907d0212d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:53.729 INFO 21062 --- [ocp-schedules-4,22958135b065723b,2f52a1a04f4f3de6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:53:53.735 INFO 21062 --- [ocp-schedules-4,22958135b065723b,2f52a1a04f4f3de6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:53:53.735 INFO 21062 --- [ocp-schedules-4,22958135b065723b,2f52a1a04f4f3de6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:53:53.983 INFO 21062 --- [prometheus-minute-schedule-1,816b5638f54e8e5b,e802e19c428a38e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:54.716 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:54.717 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:55.728 WARN 21062 --- [ocp-async-10,3a3ca2e899823d9f,f4f16d3821f8b7bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:55.728 INFO 21062 --- [ocp-async-9,e661c1f4770c770b,2e84a1ee4624fde6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7d13b72a93431250,c01e227e1c13d4ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:53:57.717 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:53:57.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:53:57.729 WARN 21062 --- [ocp-async-2,26c5ca35f3cdc89a,cf0af2022e59bbfc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:57.729 INFO 21062 --- [ocp-async-5,647766f5fc8685d0,2415305faf8a5033] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:58.499 INFO 21062 --- [ocp-schedules-18,6693fe28bf2ab151,91efeef9464a0efc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:53:58.500 INFO 21062 --- [ocp-schedules-15,5560a9da29915512,17079a866b33da1d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:53:58.502 INFO 21062 --- [ocp-schedules-30,6ba72d7541909d16,9227730d2afe424f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:53:58.504 INFO 21062 --- [ocp-schedules-18,6693fe28bf2ab151,91efeef9464a0efc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:53:58.504 INFO 21062 --- [ocp-schedules-15,5560a9da29915512,17079a866b33da1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:53:58.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:53:58.505 INFO 21062 --- [ocp-schedules-3,6f3edde7f270ab6e,9367c2997f55c1f5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:53:58.509 INFO 21062 --- [ocp-schedules-27,86cdf26b0d3150bd,11d59cd736d14bb7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:53:58.509 INFO 21062 --- [ocp-schedules-27,86cdf26b0d3150bd,11d59cd736d14bb7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:53:58.509 INFO 21062 --- [ocp-schedules-20,89f9ccbdb5215881,fd83bc5e52e642da] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:53:58.509 INFO 21062 --- [ocp-schedules-20,89f9ccbdb5215881,fd83bc5e52e642da] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:53:58.510 INFO 21062 --- [ocp-schedules-25,4df35ea4becd01d5,6be3fd687d82c488] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:53:58.510 INFO 21062 --- [ocp-schedules-25,4df35ea4becd01d5,6be3fd687d82c488] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:53:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:53:58.588209+08:00, lastProcessTime=2024-11-04T15:53:48.589168+08:00 2024-11-04 15:53:58.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:53:58.588209+08:00 2024-11-04 15:53:58.830 INFO 21062 --- [ocp-async-4,6d39a584728e3042,efba7541b5278640] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:53:59.685 INFO 21062 --- [ocp-schedules-2,44b0e798cc9ce00d,0e6efcc547953f73] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:53:59.729 WARN 21062 --- [ocp-async-8,6433de575225e05c,d681d2add13fd175] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:53:59.729 INFO 21062 --- [ocp-async-7,e8b30df1ae6dad15,f612e613ebba86a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:53:59.983 INFO 21062 --- [prometheus-minute-schedule-1,eaf08ee5b0bd1bc4,ab2aff745f63b051] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706840, interval=60 2024-11-04 15:54:00.718 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:00.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:01.729 WARN 21062 --- [ocp-async-3,f9df451644c2cb69,8d71e4a35415ba78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:01.729 INFO 21062 --- [ocp-async-3,9b94ae27c8d0db25,2f0c74713e336230] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:02.173 INFO 21062 --- [ocp-schedules-8,d900042831ee379e,ff901bbe3286090e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:54:02.680 INFO 21062 --- [ocp-schedules-14,1117ba445c618fd0,30983ec6fae5f86b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:54:02.983 INFO 21062 --- [prometheus-minute-schedule-1,f5677b5371354b16,4978fa3fa6af1077] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:03.719 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:03.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:03.729 WARN 21062 --- [ocp-async-6,581f385e48aa8b4e,9edc351792645528] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:03.729 INFO 21062 --- [ocp-async-5,3e4205c076846754,2599ddacb0a39ffc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:05.729 WARN 21062 --- [ocp-async-8,4a6e66db225a9648,b6a2d80d1a619f2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:05.729 INFO 21062 --- [ocp-async-4,0c17c1c4bb5d1a22,75fc87fe1f5ecb31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:05.983 INFO 21062 --- [prometheus-minute-schedule-1,2ce8c57d4cfaf376,1235dd8a92160388] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:06.720 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:06.721 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:07.730 WARN 21062 --- [ocp-async-9,4b1381cd1c1ba6db,3ad8ace68ae5c6d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:07.730 INFO 21062 --- [ocp-async-9,381269713cd669f0,a6429b2c946ac2aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:08.505 INFO 21062 --- [ocp-schedules-18,6693fe28bf2ab151,91efeef9464a0efc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:54:08.505 INFO 21062 --- [ocp-schedules-15,5560a9da29915512,17079a866b33da1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:54:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:54:08.588249+08:00, lastProcessTime=2024-11-04T15:53:58.588209+08:00 2024-11-04 15:54:08.764 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:54:08.588249+08:00 2024-11-04 15:54:08.830 INFO 21062 --- [ocp-async-6,390bfdf362d2e337,5adedf07d9b96d95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:54:08.983 INFO 21062 --- [prometheus-minute-schedule-1,236b7417d9f39d90,3dc7e863d30555ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:09.440 INFO 21062 --- [ocp-schedules-30,6d03d676ef31c2ca,8f3a9cd93c7a10d7] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:54:09.444 INFO 21062 --- [ocp-schedules-30,6d03d676ef31c2ca,8f3a9cd93c7a10d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:09.445 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:54:09.721 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:09.722 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:09.730 WARN 21062 --- [ocp-async-1,324f0a6672a19b64,29ef6f55d1d81ee6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:09.730 INFO 21062 --- [ocp-async-2,e4ec7ead6be38032,912322d40c0fcca7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:10.003 INFO 21062 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:54:10.003 INFO 21062 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:54:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:54:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:54:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:54:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:54:10.322 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:54:10.322 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:54:11.730 WARN 21062 --- [ocp-async-5,aa40a01f3d9b5064,d1aae759513345eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:11.730 INFO 21062 --- [ocp-async-4,eb5be22350227f1f,62ea0ec0e43f80ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:11.967 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=2, affectRows=2 2024-11-04 15:54:11.967 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=2, validCount=2 2024-11-04 15:54:11.967 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=11, affectRows=11 2024-11-04 15:54:11.967 INFO 21062 --- [metric-archive-second-54,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=11, validCount=11 2024-11-04 15:54:11.968 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=25, affectRows=25 2024-11-04 15:54:11.968 INFO 21062 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=25, validCount=25 2024-11-04 15:54:11.968 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=14, affectRows=14 2024-11-04 15:54:11.968 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=14, validCount=14 2024-11-04 15:54:11.968 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=48, affectRows=48 2024-11-04 15:54:11.968 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=41, affectRows=41 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=20, affectRows=20 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-51,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=20, validCount=20 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-48,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=41, validCount=41 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-55,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=48, validCount=48 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=70, affectRows=70 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=69, validCount=69 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=105, affectRows=105 2024-11-04 15:54:11.969 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706780, nodeCount=88, affectRows=88 2024-11-04 15:54:11.970 INFO 21062 --- [metric-archive-second-43,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=104, validCount=104 2024-11-04 15:54:11.970 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706780, nodeCount=87, validCount=87 2024-11-04 15:54:11.983 INFO 21062 --- [prometheus-minute-schedule-1,b100a1346f5cba3d,a4444934f2babaab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:12.176 INFO 21062 --- [ocp-schedules-20,836eb5ec089cb774,3f0eaacf4a6650c1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:54:12.303 INFO 21062 --- [ocp-schedules-23,d7af155521c1c956,d92f093cae0c618a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:54:12.303 INFO 21062 --- [ocp-schedules-23,bf8b866e0453def6,17555139aecefdb7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:54:12.307 INFO 21062 --- [ocp-schedules-23,bf8b866e0453def6,17555139aecefdb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:12.337 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:54:12.337 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:54:12.704 INFO 21062 --- [ocp-schedules-28,b7cf7f8f1ef2487b,04a42dd92bfdb87f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:54:12.722 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:12.722 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:13.730 WARN 21062 --- [ocp-async-7,66bc77c2054972cb,a63a3db461ba2f93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:13.730 INFO 21062 --- [ocp-async-9,3536dff56db86cd5,b6e663093a9c0dac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:13.738 INFO 21062 --- [ocp-schedules-21,a425ad17ffe7ceee,274704966fc6fadf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:54:13.743 INFO 21062 --- [ocp-schedules-21,a425ad17ffe7ceee,274704966fc6fadf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:54:13.743 INFO 21062 --- [ocp-schedules-21,a425ad17ffe7ceee,274704966fc6fadf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:54:13.984 INFO 21062 --- [prometheus-cache-monitor1,2a4c9e4cf8c7e8b3,d6817ad63e9bca90] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:54:13.984 INFO 21062 --- [prometheus-cache-monitor1,2a4c9e4cf8c7e8b3,d6817ad63e9bca90] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:54:14.983 INFO 21062 --- [prometheus-minute-schedule-1,adbb75569470bfd6,02769ddd8aa85d1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:15.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:15.723 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:15.730 WARN 21062 --- [ocp-async-1,2f9bfc16d8ffd818,990ea45834377b48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:15.730 INFO 21062 --- [ocp-async-1,ccc9f16737141ce1,a0e201590ab730b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:17.730 WARN 21062 --- [ocp-async-5,b0469bcf8aa22bbd,ac42c25d08cbd065] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:17.731 INFO 21062 --- [ocp-async-8,58aafbc4a11480d7,c93b26f18b9888b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:17.983 INFO 21062 --- [prometheus-minute-schedule-1,bcafa9d520618188,d5c8defe9136eceb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:18.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:54:18.589484+08:00, lastProcessTime=2024-11-04T15:54:08.588249+08:00 2024-11-04 15:54:18.724 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:18.724 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:18.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:54:18.589484+08:00 2024-11-04 15:54:18.830 INFO 21062 --- [ocp-async-7,0589b943ef356c09,c2ccfbb7c1012669] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:54:19.445 INFO 21062 --- [ocp-schedules-30,6d03d676ef31c2ca,8f3a9cd93c7a10d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:54:19.731 WARN 21062 --- [ocp-async-10,52126434c288c504,7733832cc1ca6572] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:19.731 INFO 21062 --- [ocp-async-7,2f2bbed1ba3f1a89,627e1e20d638d721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:20.983 INFO 21062 --- [prometheus-minute-schedule-1,34649639c0d6761a,8bfa4cfbb15b639e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:21.724 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:21.725 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:21.731 WARN 21062 --- [ocp-async-3,652fe38633084644,885e2ec7a178812f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:21.731 INFO 21062 --- [ocp-async-1,50cfc708522f750c,e568a46fc0b35add] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:22.178 INFO 21062 --- [ocp-schedules-22,6d695126d6256287,9dff6b895a5cd639] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:54:22.753 INFO 21062 --- [ocp-schedules-11,a5d4674057289e4a,448bb145af2712b8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:54:23.731 WARN 21062 --- [ocp-async-2,34a6f36acb6004d6,d5c8f69f16e397f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:23.731 INFO 21062 --- [ocp-async-8,868a84886f20e120,83f797fc949aa4ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:23.983 INFO 21062 --- [prometheus-minute-schedule-1,fadc128d3f3068e0,32ded13be91fff14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:24.725 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:24.726 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:25.731 WARN 21062 --- [ocp-async-9,44e39257434baf84,19d1d5c054a7b3da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:25.731 INFO 21062 --- [ocp-async-10,e7c5b9e05895a0cc,1a6e75752bebeff1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:26.983 INFO 21062 --- [prometheus-minute-schedule-1,b5888e981967bbe3,75bbbe7290dad178] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:26.984 INFO 21062 --- [prometheus-hour-schedule-1,a445ea3e46840683,eed0a64dbaccb61d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:54:27.726 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:27.727 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:27.731 WARN 21062 --- [ocp-async-3,eae5a765b63733f1,ace05a1207383ff2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:27.731 INFO 21062 --- [ocp-async-3,3efc7d646ade9bb5,b0e182e5b543b361] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:28.498 INFO 21062 --- [ocp-schedules-6,94a86d370e2ba1f6,67287e5f6d218c7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:54:28.499 INFO 21062 --- [ocp-schedules-30,f601f1333bd9ea57,8a5186be6b50abcb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:54:28.499 INFO 21062 --- [ocp-schedules-10,ee5777673e694746,1b734138d0e24640] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:54:28.500 INFO 21062 --- [ocp-schedules-19,079da63b8f9949eb,ceffe8fbaa078b9a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:54:28.502 INFO 21062 --- [ocp-schedules-6,94a86d370e2ba1f6,67287e5f6d218c7a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:54:28.503 INFO 21062 --- [ocp-schedules-22,81391e5ea69c849f,5986e8de0f885b4d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:54:28.503 INFO 21062 --- [ocp-schedules-10,ee5777673e694746,1b734138d0e24640] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:28.503 INFO 21062 --- [ocp-schedules-19,079da63b8f9949eb,ceffe8fbaa078b9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:54:28.504 INFO 21062 --- [ocp-schedules-17,86d1c0dc10722525,554285e94c67e61b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:54:28.506 INFO 21062 --- [ocp-schedules-30,f601f1333bd9ea57,8a5186be6b50abcb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:54:28.510 INFO 21062 --- [ocp-schedules-14,8a424738dd0e41f5,70a5833016620875] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:54:28.510 INFO 21062 --- [ocp-schedules-1,7d34e52ae817d0a3,bdfcea8f5b67198b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:54:28.510 INFO 21062 --- [ocp-schedules-14,8a424738dd0e41f5,70a5833016620875] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:54:28.510 INFO 21062 --- [ocp-schedules-1,7d34e52ae817d0a3,bdfcea8f5b67198b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:54:28.510 INFO 21062 --- [ocp-schedules-5,d9edef24ffd7a987,e26dfe11e566faf3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:54:28.510 INFO 21062 --- [ocp-schedules-5,d9edef24ffd7a987,e26dfe11e566faf3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:54:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:54:28.589519+08:00, lastProcessTime=2024-11-04T15:54:18.589484+08:00 2024-11-04 15:54:28.595 INFO 21062 --- [ocp-schedules-14,0526139c7b64b8e9,b4d835c7583482ab] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":423,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:54:28.595 INFO 21062 --- [ocp-schedules-14,0526139c7b64b8e9,b4d835c7583482ab] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:54:28.635 INFO 21062 --- [ocp-schedules-25,e514f6d93ee81e31,7b3b3a91097f6a41] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-11-04T15:54:28.635+08:00, tenantSize=0 2024-11-04 15:54:28.683 INFO 21062 --- [ocp-schedules-9,64fc76b428865bf6,088352971f7daa10] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:54:28.684 INFO 21062 --- [ocp-schedules-9,64fc76b428865bf6,088352971f7daa10] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-11-04 15:54:28.754 INFO 21062 --- [ocp-schedules-16,dffdb19d3acb179c,bd12bd49bacd31d8] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2024-11-04 15:54:28.754 INFO 21062 --- [ocp-schedules-16,dffdb19d3acb179c,bd12bd49bacd31d8] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-11-04 15:54:28.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:54:28.589519+08:00 2024-11-04 15:54:28.831 INFO 21062 --- [ocp-async-8,b14425b3d0f699e4,948f35c076ca3c40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:54:29.052 INFO 21062 --- [ocp-schedules-8,19f84184e2e5f736,90e6b857543bb778] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2024-11-04 15:54:29.052 INFO 21062 --- [ocp-schedules-8,19f84184e2e5f736,90e6b857543bb778] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-11-04 15:54:29.066 INFO 21062 --- [ocp-schedules-1,b1ee74157cc7181b,b91fef40c0cfa8e4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:54:29.522 INFO 21062 --- [ocp-schedules-14,d01cd93351ba7f2d,c697052ced7595a1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:54:29.522 INFO 21062 --- [ocp-schedules-14,d01cd93351ba7f2d,c697052ced7595a1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:54:29.689 INFO 21062 --- [ocp-schedules-7,78fbb58207e2f292,9e2a360ef72cf9c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:54:29.697 INFO 21062 --- [ocp-schedules-28,e3c7103ed63f1793,c7126aa1abe49b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:54:29.703 INFO 21062 --- [ocp-schedules-28,e3c7103ed63f1793,c7126aa1abe49b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:54:29.703 INFO 21062 --- [ocp-schedules-28,e3c7103ed63f1793,c7126aa1abe49b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:54:29.732 WARN 21062 --- [ocp-async-4,a64683241e765217,4ce3777d7a7113c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:29.732 INFO 21062 --- [ocp-async-9,3b0c81cd9c3e8cf7,2889e2ba217e774c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:29.983 INFO 21062 --- [prometheus-minute-schedule-1,3c58daaaa6b82a24,5ec78e861ccbe0fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:30.252 INFO 21062 --- [ocp-schedules-27,7b6e2e86bdb82c86,9f0c87a7f4e55b68] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:54:30.252 INFO 21062 --- [ocp-schedules-27,7b6e2e86bdb82c86,9f0c87a7f4e55b68] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:54:30.253 INFO 21062 --- [ocp-schedules-27,7b6e2e86bdb82c86,9f0c87a7f4e55b68] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:54:30.253 INFO 21062 --- [ocp-schedules-27,7b6e2e86bdb82c86,9f0c87a7f4e55b68] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:54:30.254 INFO 21062 --- [ocp-schedules-27,7b6e2e86bdb82c86,9f0c87a7f4e55b68] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:54:30.254 INFO 21062 --- [ocp-schedules-27,7b6e2e86bdb82c86,9f0c87a7f4e55b68] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:54:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:54:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:54:30.727 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:30.727 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:31.732 WARN 21062 --- [ocp-async-7,596a3a71fe5845d9,0e51283e75ff2747] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:31.732 INFO 21062 --- [ocp-async-6,053d7bee3324f3af,439d9cecf1a104c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:32.182 INFO 21062 --- [ocp-schedules-22,5cf39210cd785250,28fa1ac5028476b8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:54:32.781 INFO 21062 --- [ocp-schedules-6,04153eac921ad841,e4bd9ee7f8d0ca89] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:54:32.983 INFO 21062 --- [prometheus-minute-schedule-1,3c048e0e79b952e7,f962fb56a8902643] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:33.727 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:33.728 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:33.732 WARN 21062 --- [ocp-async-1,1e5bb890a8aeeb1e,ca038dce5bc760fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:33.732 INFO 21062 --- [ocp-async-2,8bc106ac71197ef2,c04acc8f750c4192] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:33.744 INFO 21062 --- [ocp-schedules-2,2a4a5098324ada50,f5ec767a2f447bfc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:54:33.749 INFO 21062 --- [ocp-schedules-2,2a4a5098324ada50,f5ec767a2f447bfc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:54:33.749 INFO 21062 --- [ocp-schedules-2,2a4a5098324ada50,f5ec767a2f447bfc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:54:34.094 INFO 21062 --- [ocp-schedules-27,d9c342fd9f8e3a22,bee0ca914ec2a098] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:54:34.094 INFO 21062 --- [ocp-schedules-27,d9c342fd9f8e3a22,bee0ca914ec2a098] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:54:34.121 INFO 21062 --- [ocp-schedules-27,d9c342fd9f8e3a22,bee0ca914ec2a098] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:54:34.121 INFO 21062 --- [ocp-schedules-27,d9c342fd9f8e3a22,bee0ca914ec2a098] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:54:34.121 INFO 21062 --- [ocp-schedules-6,24da1695fc1ff0c7,c5cd3a91a84d8795] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:54:34.299 INFO 21062 --- [ocp-schedules-4,879afcdabd470ecc,490362cbff2f4e81] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:54:34.299+08:00, clusters=[1] 2024-11-04 15:54:34.301 INFO 21062 --- [ocp-schedules-4,879afcdabd470ecc,490362cbff2f4e81] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:54:34.691 INFO 21062 --- [ocp-schedules-25,c6b6e1df5dae9ba0,20cd6efb51cc2f4d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:54:34.691+08:00, clusters=[1] 2024-11-04 15:54:34.695 INFO 21062 --- [ocp-schedules-25,c6b6e1df5dae9ba0,20cd6efb51cc2f4d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:54:34.796 INFO 21062 --- [ocp-schedules-5,8d4b5e06453f663e,50c38cb735f2d776] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:54:34.799 INFO 21062 --- [ocp-schedules-5,8d4b5e06453f663e,50c38cb735f2d776] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:34.860 ERROR 21062 --- [ocp-schedules-5,8d4b5e06453f663e,50c38cb735f2d776] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:54:34.861 INFO 21062 --- [ocp-schedules-5,8d4b5e06453f663e,50c38cb735f2d776] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:54:35.732 WARN 21062 --- [ocp-async-4,3777b8cbf5f37502,447b73396b1e4136] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:35.732 INFO 21062 --- [ocp-async-4,f1cd05900bdd7134,27c5d1978b0c7677] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:35.983 INFO 21062 --- [prometheus-minute-schedule-1,bb4522963ce023b2,74f5b57b484d0a4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:36.729 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:36.729 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:36.904 INFO 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:54:36.904 INFO 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:54:36.904 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:54:36.904 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:54:36.914 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:54:36.915 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:54:36.915 INFO 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:54:36.916 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:54:36.917 WARN 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:54:36.917 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:54:36.918 INFO 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:54:36.918 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:54:36.918 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:54:36.918 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:54:36.919 WARN 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:54:36.920 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:54:36.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:54:36.926 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:54:36.926 INFO 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:54:36.938 ERROR 21062 --- [ocp-schedules-21,7fc685da6bc9d4c7,27d4c547cc813dbd] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:54:37.733 WARN 21062 --- [ocp-async-7,745c8a4a13ffe6b2,2beb9bb8f95825e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:37.733 INFO 21062 --- [ocp-async-10,a232f52960fe5d4f,0531d5218ffa0cc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:38.504 INFO 21062 --- [ocp-schedules-10,ee5777673e694746,1b734138d0e24640] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:54:38.504 INFO 21062 --- [ocp-schedules-19,079da63b8f9949eb,ceffe8fbaa078b9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:54:38.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:54:38.587186+08:00, lastProcessTime=2024-11-04T15:54:28.589519+08:00 2024-11-04 15:54:38.757 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:54:38.587186+08:00 2024-11-04 15:54:38.831 INFO 21062 --- [ocp-async-3,5e523aebbd6e10f9,2fc5b8869bb3f33e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:54:38.983 INFO 21062 --- [prometheus-minute-schedule-1,a5336c9bcd5f05c4,c3e75543d7822036] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:39.730 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:39.731 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:39.733 WARN 21062 --- [ocp-async-2,e5c1c5b11223c381,ec637b1852b6c62e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:39.733 INFO 21062 --- [ocp-async-8,94f596f75b9e5ac4,5559d620df09731a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:40.002 INFO 21062 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:54:40.003 INFO 21062 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:54:41.733 WARN 21062 --- [ocp-async-5,732099c3a3da7c6d,5d122464ccafec94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:41.733 INFO 21062 --- [ocp-async-4,6913a4f5b92632f3,45eb0277d9f2e6b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:41.828 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:54:41.829 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:41.830 INFO 21062 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:54:41.831 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-11-04 15:54:41.858 WARN 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster dba_test_ob(1), status=TAKINGOVER 2024-11-04 15:54:41.858 INFO 21062 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=27ms, period=120000ms 2024-11-04 15:54:41.983 INFO 21062 --- [prometheus-minute-schedule-1,52375975d9000824,53e821afd7a0cb24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:42.185 INFO 21062 --- [ocp-schedules-7,463fb0f03b9439a9,ece73250bf95d9d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:54:42.337 INFO 21062 --- [ocp-schedules-23,bf8b866e0453def6,17555139aecefdb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:54:42.338 INFO 21062 --- [ocp-schedules-23,f14fe3a12809437f,3f48c92cdac28956] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:54:42.341 INFO 21062 --- [ocp-schedules-23,f14fe3a12809437f,3f48c92cdac28956] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:42.369 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:54:42.369 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:54:42.731 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:42.732 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:42.823 INFO 21062 --- [ocp-schedules-15,bafa7186bc7acf3c,164520517919e065] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:54:43.733 WARN 21062 --- [ocp-async-9,73bacea252a12587,b851e60829f08339] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:43.733 INFO 21062 --- [ocp-async-10,e5ca20ff4e7709c0,32b141b64b6950d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:44.983 INFO 21062 --- [prometheus-minute-schedule-1,aba194922547f54e,23f64e77a825f690] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:45.732 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:45.732 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:45.733 WARN 21062 --- [ocp-async-3,a2b022d96359084b,549d6f883af1770f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:45.733 INFO 21062 --- [ocp-async-2,bbdd12fb39f19162,6f4af5bebd4b411f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:46.053 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-11-04 15:54:46.053 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-11-04 15:54:46.054 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.054 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:54:46.055 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.056 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.057 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.057 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-11-04 15:54:46.072 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.073 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.073 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.074 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-11-04 15:54:46.086 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.087 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.087 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.088 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-11-04 15:54:46.102 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.102 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.103 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.103 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-11-04 15:54:46.116 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.117 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.117 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.118 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-11-04 15:54:46.123 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.124 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.124 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.125 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-11-04 15:54:46.129 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.129 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.129 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.130 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-11-04 15:54:46.134 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.135 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.135 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.136 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-11-04 15:54:46.141 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.142 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.142 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.143 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-11-04 15:54:46.148 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.149 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.149 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.150 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-11-04 15:54:46.156 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.156 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.157 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.157 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-11-04 15:54:46.166 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.166 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.166 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.167 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-11-04 15:54:46.181 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.181 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.182 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.182 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-11-04 15:54:46.186 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.187 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.187 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.188 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-11-04 15:54:46.193 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.194 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.194 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.194 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-11-04 15:54:46.200 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.200 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.201 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.201 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-11-04 15:54:46.205 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.206 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.206 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.207 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-11-04 15:54:46.211 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.211 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.212 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.212 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-11-04 15:54:46.216 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.217 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.217 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.217 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-11-04 15:54:46.223 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.223 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.224 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.224 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-11-04 15:54:46.229 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.229 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.230 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.230 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-11-04 15:54:46.235 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.235 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.236 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.236 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-11-04 15:54:46.240 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.240 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.241 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.241 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-11-04 15:54:46.246 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.246 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.246 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.247 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-11-04 15:54:46.252 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.252 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.253 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.253 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-11-04 15:54:46.257 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.257 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.258 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.258 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-11-04 15:54:46.263 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.264 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.264 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.264 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-11-04 15:54:46.269 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.269 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.270 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.270 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-11-04 15:54:46.274 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.275 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.275 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.276 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-11-04 15:54:46.280 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.281 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.281 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.282 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-11-04 15:54:46.286 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.286 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.287 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.287 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-11-04 15:54:46.291 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.292 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.292 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.293 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-11-04 15:54:46.297 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.297 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.298 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.298 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-11-04 15:54:46.303 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.304 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.304 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.305 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-11-04 15:54:46.312 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.312 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.313 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.313 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-11-04 15:54:46.319 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.319 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.320 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.320 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-11-04 15:54:46.326 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.326 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.326 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.327 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-11-04 15:54:46.333 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.333 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.334 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.334 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-11-04 15:54:46.338 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.338 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.339 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.339 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-11-04 15:54:46.343 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.343 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.344 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.344 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-11-04 15:54:46.348 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.348 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.349 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.349 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-11-04 15:54:46.353 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.353 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.354 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.354 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-11-04 15:54:46.364 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.364 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.364 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.365 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-11-04 15:54:46.385 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.386 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.386 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.387 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-11-04 15:54:46.393 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.393 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.394 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.394 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-11-04 15:54:46.400 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.400 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.401 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.401 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-11-04 15:54:46.408 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.408 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.408 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.409 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-11-04 15:54:46.413 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.413 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.413 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.414 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-11-04 15:54:46.417 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.418 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.418 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.419 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-11-04 15:54:46.423 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-11-04 15:54:46.423 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-11-04 15:54:46.424 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.424 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-11-04 15:54:46.425 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.425 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.426 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.426 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-11-04 15:54:46.432 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.432 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-11-04 15:54:46.433 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-11-04 15:54:46.433 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-11-04 15:54:46.437 INFO 21062 --- [ocp-schedules-27,b70cc12974371ca4,392bd529710ef674] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-11-04 15:54:47.193 INFO 21062 --- [ocp-schedules-2,56dfbe634c522bc3,fc3d44ca9bceda83] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:54:47.197 INFO 21062 --- [ocp-schedules-2,56dfbe634c522bc3,fc3d44ca9bceda83] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:47.282 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:54:47.284 ERROR 21062 --- [ocp-schedules-2,56dfbe634c522bc3,fc3d44ca9bceda83] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:54:47.285 INFO 21062 --- [ocp-schedules-2,56dfbe634c522bc3,fc3d44ca9bceda83] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:54:47.733 WARN 21062 --- [ocp-async-7,5efb71e75368dfb3,6404cee17f16ca7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:47.734 INFO 21062 --- [ocp-async-5,1d0139b2a8e61d96,0139cc3018beaebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:47.983 INFO 21062 --- [prometheus-minute-schedule-1,8b1e4c6ee36f266b,a65273bca68ce76d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:54:48.590139+08:00, lastProcessTime=2024-11-04T15:54:38.587186+08:00 2024-11-04 15:54:48.768 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:48.769 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:48.803 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:54:48.590139+08:00 2024-11-04 15:54:48.831 INFO 21062 --- [ocp-async-9,8b3046fbb830a3fa,e86e9b004c919af3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:54:49.734 WARN 21062 --- [ocp-async-4,1f465b6ba158d656,601f720a4167e0de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:49.734 INFO 21062 --- [ocp-async-6,2cf480318e477e24,09bb549c887662ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:50.983 INFO 21062 --- [prometheus-minute-schedule-1,72eb8ce6704ed04d,b1a8826e9ce3d444] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:51.734 WARN 21062 --- [ocp-async-10,8caa53b6e85ce9b4,fb87c203dac81860] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:51.734 INFO 21062 --- [ocp-async-1,90ff993e5987b13a,c82184a0da83109c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:51.769 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:51.770 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:52.188 INFO 21062 --- [ocp-schedules-17,db63d0fe85be3860,a7a95aa482a24e66] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:54:52.847 INFO 21062 --- [ocp-schedules-12,349f237da1be6aa3,8b566260b1ecf112] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:54:53.734 WARN 21062 --- [ocp-async-2,bf4930ebc0a3efcd,ae539c94fd3d57f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:53.734 INFO 21062 --- [ocp-async-8,1f9b370f1223499c,8c818aeea6eefd41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:53.751 INFO 21062 --- [ocp-schedules-24,adafcb08d143aa79,0156319aa4ed4c5c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:54:53.757 INFO 21062 --- [ocp-schedules-24,adafcb08d143aa79,0156319aa4ed4c5c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:54:53.757 INFO 21062 --- [ocp-schedules-24,adafcb08d143aa79,0156319aa4ed4c5c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:54:53.983 INFO 21062 --- [prometheus-minute-schedule-1,247200142e9699a0,24a7a4de7cc33c5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:54.770 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:54.770 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:55.734 WARN 21062 --- [ocp-async-4,65993c1c52127466,6d3f20468bcb2d08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:55.734 INFO 21062 --- [ocp-async-4,8d912653e2f10784,876cdba8770812da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:56.983 INFO 21062 --- [prometheus-minute-schedule-1,c5dccec15a1d3dd2,5d6495588f9589c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:54:57.735 WARN 21062 --- [ocp-async-10,9dd1336c4c17191a,9046d1d908fc2de6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:57.735 INFO 21062 --- [ocp-async-3,d52d5839b41f2a15,bbc99037002767a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:57.770 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:54:57.771 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:54:58.499 INFO 21062 --- [ocp-schedules-6,f73c5891ad8b4ddd,8b9b1b9c9d0d409f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:54:58.500 INFO 21062 --- [ocp-schedules-22,8abfd091575ebfa7,7384acfa27544cb5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:54:58.502 INFO 21062 --- [ocp-schedules-18,b105393c7ac3cf75,f410d7a22b4f1dda] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:54:58.503 INFO 21062 --- [ocp-schedules-6,f73c5891ad8b4ddd,8b9b1b9c9d0d409f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:58.503 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:54:58.504 INFO 21062 --- [ocp-schedules-22,8abfd091575ebfa7,7384acfa27544cb5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:54:58.506 INFO 21062 --- [ocp-schedules-21,c5ec38b71f3da82f,a5dae7e4e59bc83f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:54:58.509 INFO 21062 --- [ocp-schedules-28,6628c70269d2a2a0,afdb4ef7f472c56a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:54:58.509 INFO 21062 --- [ocp-schedules-28,6628c70269d2a2a0,afdb4ef7f472c56a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:54:58.509 INFO 21062 --- [ocp-schedules-30,0eaeda8e564f5c8b,4fb2fa8b6790df1c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:54:58.510 INFO 21062 --- [ocp-schedules-30,0eaeda8e564f5c8b,4fb2fa8b6790df1c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:54:58.510 INFO 21062 --- [ocp-schedules-19,e492c2948e4b6ca4,8ce481bb73190d33] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:54:58.510 INFO 21062 --- [ocp-schedules-19,e492c2948e4b6ca4,8ce481bb73190d33] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:54:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:54:58.588571+08:00, lastProcessTime=2024-11-04T15:54:48.590139+08:00 2024-11-04 15:54:58.766 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:54:58.588571+08:00 2024-11-04 15:54:58.831 INFO 21062 --- [ocp-async-2,ba563fffb4c1fab8,daf0f6d9de8b2cae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:54:59.691 INFO 21062 --- [ocp-schedules-30,87836cf526d39921,e3b8546a516c8f6e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:54:59.735 WARN 21062 --- [ocp-async-7,de858c945bcad277,48bac94062af6be6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:54:59.735 INFO 21062 --- [ocp-async-8,3f410b44a0befd3c,c2487beb5f59d6a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:54:59.983 INFO 21062 --- [prometheus-minute-schedule-1,be45bcac039401aa,9607590d6fc063aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706900, interval=60 2024-11-04 15:55:00.250 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,294b64629b1071d2] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-11-04 15:50:00.0, nextRun=2024-11-04 15:55:00.0, currentDate=Mon Nov 04 15:55:00 CST 2024 2024-11-04 15:55:00.250 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,294b64629b1071d2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-11-04 15:50:00.0, nextRunTime=2024-11-04 15:55:00.0} 2024-11-04 15:55:00.277 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,294b64629b1071d2] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2024-11-04 15:55:00.281 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,294b64629b1071d2] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2024-11-04 15:55:00.281 INFO 21062 --- [task-coordinator2,dd193f8f67a5a423,294b64629b1071d2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-11-04 15:55:00.771 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:00.773 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:01.735 INFO 21062 --- [ocp-async-5,7536b2cb3a806d0a,282ffe37d7cfd77c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:01.735 WARN 21062 --- [ocp-async-4,e3794f01452b5f3f,cd6fd6ed19e56443] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:02.191 INFO 21062 --- [ocp-schedules-8,88651b98e1f853f1,0c679f5bb3bbf44f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:55:02.887 INFO 21062 --- [ocp-schedules-29,86ad4bd627fb2b4c,a25ad0158dcd633f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:55:02.983 INFO 21062 --- [prometheus-minute-schedule-1,a35d9646504a3576,a478968c6c3e7f27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:03.735 INFO 21062 --- [ocp-async-6,2a5fcd73e1611733,a4e9202dd2844efb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:03.735 WARN 21062 --- [ocp-async-3,80a7d99995cebc34,2f1c36c27354273a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:03.773 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:03.774 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:05.735 INFO 21062 --- [ocp-async-7,063975e31ad86538,cc122299081a8238] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:05.735 WARN 21062 --- [ocp-async-2,2533b5b9c0029de8,05abd210d141d11f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:05.983 INFO 21062 --- [prometheus-minute-schedule-1,85df6c0ea4cf695f,a152c3439b2dbd7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:06.774 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:06.775 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:07.736 INFO 21062 --- [ocp-async-4,ff84c523fc61b84d,5948aa767e00c5be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:07.736 WARN 21062 --- [ocp-async-9,63ace2ba91f7b9df,9b76da34bea432c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:08.504 INFO 21062 --- [ocp-schedules-6,f73c5891ad8b4ddd,8b9b1b9c9d0d409f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:55:08.505 INFO 21062 --- [ocp-schedules-22,8abfd091575ebfa7,7384acfa27544cb5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:55:08.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:55:08.589511+08:00, lastProcessTime=2024-11-04T15:54:58.588571+08:00 2024-11-04 15:55:08.767 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:55:08.589511+08:00 2024-11-04 15:55:08.831 INFO 21062 --- [ocp-async-3,315cde45812370a1,593f353fea359276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:55:08.983 INFO 21062 --- [prometheus-minute-schedule-1,2baf10716e729c37,56bdf65a176cb9b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:09.736 INFO 21062 --- [ocp-async-10,7692a1086f863a0b,8b217af57b006312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:09.736 WARN 21062 --- [ocp-async-6,d8d25f902aead6c4,34ea255fc46fc01e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:09.775 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:09.776 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:55:10.003 INFO 21062 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:55:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:55:10.298 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:55:10.298 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:55:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:55:10.323 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:55:10.323 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:55:11.736 INFO 21062 --- [ocp-async-1,d7df8191eb036d46,d5fe0cfea7a23f37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:11.736 WARN 21062 --- [ocp-async-7,114177d4caaffb60,af574b55980872ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:11.972 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=6, affectRows=6 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=6, validCount=6 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=19, affectRows=19 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=9, affectRows=9 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=15, affectRows=15 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=9, validCount=9 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=19, validCount=19 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=15, validCount=15 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=36, affectRows=36 2024-11-04 15:55:11.973 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=42, affectRows=42 2024-11-04 15:55:11.974 INFO 21062 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=35, validCount=35 2024-11-04 15:55:11.974 INFO 21062 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=42, validCount=42 2024-11-04 15:55:11.974 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=25, affectRows=25 2024-11-04 15:55:11.974 INFO 21062 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=25, validCount=25 2024-11-04 15:55:11.974 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=73, affectRows=73 2024-11-04 15:55:11.974 INFO 21062 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=73, validCount=73 2024-11-04 15:55:11.974 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=92, affectRows=92 2024-11-04 15:55:11.975 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=92, validCount=92 2024-11-04 15:55:11.975 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706840, nodeCount=107, affectRows=107 2024-11-04 15:55:11.976 INFO 21062 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706840, nodeCount=105, validCount=105 2024-11-04 15:55:11.983 INFO 21062 --- [prometheus-minute-schedule-1,9dc83b17ce87a435,88ae777488ada392] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:12.194 INFO 21062 --- [ocp-schedules-29,54c3a6a0047d4a5f,9f41b3d3f094858f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:55:12.369 INFO 21062 --- [ocp-schedules-23,f14fe3a12809437f,3f48c92cdac28956] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:55:12.369 INFO 21062 --- [ocp-schedules-23,98ecf70836d0b8a3,885f960574ed202d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:55:12.373 INFO 21062 --- [ocp-schedules-23,98ecf70836d0b8a3,885f960574ed202d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:12.403 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:55:12.404 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:55:12.776 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:12.777 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:12.912 INFO 21062 --- [ocp-schedules-17,469363938846b682,c352ea4165d18ee6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:55:13.736 INFO 21062 --- [ocp-async-9,34bcce6f1d420973,2251204a4a5ac4f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:13.736 WARN 21062 --- [ocp-async-4,b6c9530b91fbc4a1,b9991b815720a00b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:13.759 INFO 21062 --- [ocp-schedules-20,c33ed83ab257ebfe,14261761d6a442d4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:55:13.764 INFO 21062 --- [ocp-schedules-20,c33ed83ab257ebfe,14261761d6a442d4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:55:13.764 INFO 21062 --- [ocp-schedules-20,c33ed83ab257ebfe,14261761d6a442d4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:55:13.984 INFO 21062 --- [prometheus-cache-monitor1,8504fda12920eb87,004ca3829b124598] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:55:13.984 INFO 21062 --- [prometheus-cache-monitor1,8504fda12920eb87,004ca3829b124598] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:55:14.983 INFO 21062 --- [prometheus-minute-schedule-1,69403c1ceed175b7,87c88ae67de8e812] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:15.736 WARN 21062 --- [ocp-async-6,f2d1b903c8d42578,4eac10afc775bc71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:15.736 INFO 21062 --- [ocp-async-3,30a27fc7908f6fda,5a239eb9071e75ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:15.777 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:15.778 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:15.884 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-11-04T15:51:15.866, end=2024-11-04T15:53:15.881} 2024-11-04 15:55:15.887 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2024-11-04 15:55:15.887 INFO 21062 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-11-04 15:55:17.736 WARN 21062 --- [ocp-async-10,d8d0b5d393987fca,0f0aa08eb1f85d31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:17.737 INFO 21062 --- [ocp-async-10,a3df16f8dd72ed1a,fd638304263fc8ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:17.983 INFO 21062 --- [prometheus-minute-schedule-1,188e3c5ed56b34f4,d353831d8c213d15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:18.588 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:55:18.587584+08:00, lastProcessTime=2024-11-04T15:55:08.589511+08:00 2024-11-04 15:55:18.762 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:55:18.587584+08:00 2024-11-04 15:55:18.778 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:18.779 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:18.831 INFO 21062 --- [ocp-async-4,1a7738a7896704f7,b81637a41f2a2f3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:55:19.445 INFO 21062 --- [ocp-schedules-26,67cdacf7239d19ce,a175eccc07489f71] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:55:19.449 INFO 21062 --- [ocp-schedules-26,67cdacf7239d19ce,a175eccc07489f71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:19.450 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:55:19.737 WARN 21062 --- [ocp-async-8,817491e06c0eb420,6ef078c6bc58fa65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:19.737 INFO 21062 --- [ocp-async-8,86bc98da11857442,05769b24cc054691] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:20.983 INFO 21062 --- [prometheus-minute-schedule-1,ae7c340487a84af1,662b37a306aac6c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:21.737 WARN 21062 --- [ocp-async-5,f3292773a13cddf0,72c0c843e2b2fb43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:21.737 INFO 21062 --- [ocp-async-5,6b61c83a30432759,286585b27af1f1df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:21.779 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:21.780 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:22.198 INFO 21062 --- [ocp-schedules-7,87d1945b58ed5948,64b4d7ee7bc56fa8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:55:22.953 INFO 21062 --- [ocp-schedules-16,ae1670fdcb4dc5a0,ec85d6eb64996b4a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:55:23.737 WARN 21062 --- [ocp-async-2,ee1cfbef8dd0d13d,713425d781b7a7da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:23.737 INFO 21062 --- [ocp-async-2,a5498b37f7b822c5,ce019f8c47268107] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:23.983 INFO 21062 --- [prometheus-minute-schedule-1,fb39ab7f6977a7fe,145e0fd2a64638ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:24.780 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:24.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:25.737 WARN 21062 --- [ocp-async-4,5346423013b89f52,d8d92e0c0a89e039] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:25.737 INFO 21062 --- [ocp-async-9,403446859ce3095f,b63a7b2df6914202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:26.540 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8667c8a340cc88ff,9d4be52e35a07661] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=8667c8a340cc88ff, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 15:55:26.541 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,8667c8a340cc88ff,9d4be52e35a07661] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=8667c8a340cc88ff, duration=14 ms] 2024-11-04 15:55:26.983 INFO 21062 --- [prometheus-minute-schedule-1,0982b9e66db92e6e,590144d2c2b5666e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:26.984 INFO 21062 --- [prometheus-hour-schedule-1,4a4064c1cc7c4369,5d65b061f70b0c57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:55:27.737 INFO 21062 --- [ocp-async-6,d9d80566496b67ad,3155dba8cf04d932] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:27.737 WARN 21062 --- [ocp-async-3,6654a3776ab06ed5,118c87602e08d2ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:27.781 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:27.782 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:28.498 INFO 21062 --- [ocp-schedules-24,da8d483ced25cabe,9874380e4b7aa89b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:55:28.499 INFO 21062 --- [ocp-schedules-18,312efa702330afb8,631e2ba977b4bc8c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:55:28.500 INFO 21062 --- [ocp-schedules-14,eb3284bdd16a6c70,c5d7754b131905bc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:55:28.500 INFO 21062 --- [ocp-schedules-27,43824d26e6443de9,083eb94fe3c394cd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:55:28.502 INFO 21062 --- [ocp-schedules-12,5980cd208364e82e,420d8876d4e257c2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:55:28.503 INFO 21062 --- [ocp-schedules-24,da8d483ced25cabe,9874380e4b7aa89b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:55:28.503 INFO 21062 --- [ocp-schedules-18,312efa702330afb8,631e2ba977b4bc8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:28.504 INFO 21062 --- [ocp-schedules-27,43824d26e6443de9,083eb94fe3c394cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:55:28.504 INFO 21062 --- [ocp-schedules-11,dd8794996e9f3dbc,5c763d740f10e80b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:55:28.505 INFO 21062 --- [ocp-schedules-14,eb3284bdd16a6c70,c5d7754b131905bc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:55:28.509 INFO 21062 --- [ocp-schedules-10,3bb34e43ecd12719,be11c4637fc931c3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:55:28.509 INFO 21062 --- [ocp-schedules-10,3bb34e43ecd12719,be11c4637fc931c3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:55:28.509 INFO 21062 --- [ocp-schedules-30,cc76b8b008c1ce74,7dcd997e1e158108] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:55:28.509 INFO 21062 --- [ocp-schedules-30,cc76b8b008c1ce74,7dcd997e1e158108] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:55:28.509 INFO 21062 --- [ocp-schedules-22,0eafca5f27f55351,3c17661aa0c926a9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:55:28.510 INFO 21062 --- [ocp-schedules-22,0eafca5f27f55351,3c17661aa0c926a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:55:28.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:55:28.588815+08:00, lastProcessTime=2024-11-04T15:55:18.587584+08:00 2024-11-04 15:55:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:55:28.588815+08:00 2024-11-04 15:55:28.832 INFO 21062 --- [ocp-async-7,2fac9e658a1e60cd,ba2faee9149b0c7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:55:29.069 INFO 21062 --- [ocp-schedules-12,ba14f582bedc92c1,eab549c69d33ec71] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:55:29.450 INFO 21062 --- [ocp-schedules-26,67cdacf7239d19ce,a175eccc07489f71] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:55:29.526 INFO 21062 --- [ocp-schedules-4,f5eee9a7a85115f7,b468dc9e3fbffede] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:55:29.526 INFO 21062 --- [ocp-schedules-4,f5eee9a7a85115f7,b468dc9e3fbffede] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:55:29.694 INFO 21062 --- [ocp-schedules-20,1e8f8c570bfe28c1,0a698fb573ac9506] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:55:29.703 INFO 21062 --- [ocp-schedules-29,67e1a7efa29520d3,bdf6cb84d685d600] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:55:29.708 INFO 21062 --- [ocp-schedules-29,67e1a7efa29520d3,bdf6cb84d685d600] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:55:29.708 INFO 21062 --- [ocp-schedules-29,67e1a7efa29520d3,bdf6cb84d685d600] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:55:29.737 WARN 21062 --- [ocp-async-7,b9be02fb397821da,29546c83bdd093c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:29.737 INFO 21062 --- [ocp-async-4,72893cc51df917c1,a52065f621999378] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:29.983 INFO 21062 --- [prometheus-minute-schedule-1,d7db2870add47afa,b5c1d550cdf46dca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:30.067 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-11-04 15:55:30.067 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-11-04 15:55:30.076 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-11-04 15:55:30.076 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-11-04 15:55:30.076 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-11-04 15:55:30.086 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-11-04 15:55:30.086 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-11-04 15:55:30.086 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-11-04 15:55:30.090 INFO 21062 --- [ocp-schedules-21,a211f602c88bac01,1748745bfef50885] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-11-04 15:55:30.255 INFO 21062 --- [ocp-schedules-30,9b13657e9fc4702a,786ad6ecc3afd523] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:55:30.255 INFO 21062 --- [ocp-schedules-30,9b13657e9fc4702a,786ad6ecc3afd523] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:55:30.256 INFO 21062 --- [ocp-schedules-30,9b13657e9fc4702a,786ad6ecc3afd523] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:55:30.256 INFO 21062 --- [ocp-schedules-30,9b13657e9fc4702a,786ad6ecc3afd523] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:55:30.257 INFO 21062 --- [ocp-schedules-30,9b13657e9fc4702a,786ad6ecc3afd523] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:55:30.257 INFO 21062 --- [ocp-schedules-30,9b13657e9fc4702a,786ad6ecc3afd523] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:55:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:55:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:55:30.782 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:30.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:31.738 WARN 21062 --- [ocp-async-8,f0c7ddc4f3ee695b,01efa9b6d467b364] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:31.738 INFO 21062 --- [ocp-async-9,661673b8391f45ac,73454862dc772acd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:32.201 INFO 21062 --- [ocp-schedules-7,b00ebad96cd2f2d2,c16a979ec02f5a2d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:55:32.981 INFO 21062 --- [ocp-schedules-5,621315e67c8e3b50,91ef148e2aa30b9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:55:32.984 INFO 21062 --- [prometheus-minute-schedule-1,d661ba894aa49c93,03b3ef617ff60cc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:33.738 WARN 21062 --- [ocp-async-6,117dec9404bfb5ba,dd9a98485b1af8dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:33.738 INFO 21062 --- [ocp-async-5,a9fcdf96d056a4b5,ba3937bfcd758e55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:33.766 INFO 21062 --- [ocp-schedules-9,4fd92d3994e06098,40c44f10d5757924] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:55:33.772 INFO 21062 --- [ocp-schedules-9,4fd92d3994e06098,40c44f10d5757924] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:55:33.772 INFO 21062 --- [ocp-schedules-9,4fd92d3994e06098,40c44f10d5757924] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:55:33.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:33.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:34.121 INFO 21062 --- [ocp-schedules-3,f9e066e07e6a8d8e,e4467cd88273fa6e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:55:34.121 INFO 21062 --- [ocp-schedules-3,f9e066e07e6a8d8e,e4467cd88273fa6e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:55:34.149 INFO 21062 --- [ocp-schedules-3,f9e066e07e6a8d8e,e4467cd88273fa6e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:55:34.149 INFO 21062 --- [ocp-schedules-3,f9e066e07e6a8d8e,e4467cd88273fa6e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:55:34.149 INFO 21062 --- [ocp-schedules-24,07c0d801f5321fe3,63caeaaa588e21b8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:55:34.328 INFO 21062 --- [ocp-schedules-6,8ee7b0ed3413dba1,a92729be1b74352c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:55:34.328+08:00, clusters=[1] 2024-11-04 15:55:34.329 INFO 21062 --- [ocp-schedules-6,8ee7b0ed3413dba1,a92729be1b74352c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:55:34.722 INFO 21062 --- [ocp-schedules-13,f46f76ca41f738fd,2579a7f77bb503a5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:55:34.722+08:00, clusters=[1] 2024-11-04 15:55:34.726 INFO 21062 --- [ocp-schedules-13,f46f76ca41f738fd,2579a7f77bb503a5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:55:35.738 WARN 21062 --- [ocp-async-7,dd74d3371528bd10,44a58d9995a92d7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:35.738 INFO 21062 --- [ocp-async-1,0d4880f5800aadb1,4041353cf2e96346] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:35.983 INFO 21062 --- [prometheus-minute-schedule-1,554f13dfd53705a6,bf40c08510a6003c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:36.783 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:36.784 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:36.947 INFO 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.46.128.11, 10.46.128.24, 10.46.128.12] 2024-11-04 15:55:36.947 INFO 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-11-04 15:55:36.947 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-11-04 15:55:36.947 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-11-04 15:55:36.958 INFO 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:55:36.958 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:55:36.958 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:55:36.960 WARN 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:55:36.960 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:55:36.960 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:55:36.961 INFO 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-11-04 15:55:36.961 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:55:36.961 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:55:36.962 WARN 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2024-11-04 15:55:36.962 WARN 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:55:36.962 WARN 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:55:36.970 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.11:62889/api/v1/log/alarms, params:null 2024-11-04 15:55:36.970 INFO 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.24:62889/api/v1/log/alarms, params:null 2024-11-04 15:55:36.970 INFO 21062 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.46.128.12:62889/api/v1/log/alarms, params:null 2024-11-04 15:55:36.981 ERROR 21062 --- [ocp-schedules-29,d2f222a38fe1dcdb,27808d71dad24fab] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:498) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:82) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1086) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$1729bf75.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:217) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:489) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 10.46.128.12:62889 [/10.46.128.12] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2024-11-04 15:55:37.738 WARN 21062 --- [ocp-async-8,39ca2faa5ef9cf3f,dcab384e6597229e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:37.738 INFO 21062 --- [ocp-async-4,37692d1441333691,4377be2fe37cdaf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:38.504 INFO 21062 --- [ocp-schedules-27,43824d26e6443de9,083eb94fe3c394cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:55:38.504 INFO 21062 --- [ocp-schedules-18,312efa702330afb8,631e2ba977b4bc8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:55:38.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:55:38.588025+08:00, lastProcessTime=2024-11-04T15:55:28.588815+08:00 2024-11-04 15:55:38.761 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:55:38.588025+08:00 2024-11-04 15:55:38.832 INFO 21062 --- [ocp-async-9,46939512429e771f,2c69b34fdfdc6ab4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:55:38.983 INFO 21062 --- [prometheus-minute-schedule-1,55fa42d660ec10ce,eb800be88ead087a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:39.738 WARN 21062 --- [ocp-async-7,b853e84d4bf90f9a,bd1e1df6356c9138] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:39.738 INFO 21062 --- [ocp-async-10,4f7b1bbdebb74ab0,b99d4f545315cbe9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:39.784 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:39.785 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:40.002 INFO 21062 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:55:40.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:55:41.738 INFO 21062 --- [ocp-async-1,c265fdfb6d30d1b2,1ae0d59c21e8ea0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:41.739 WARN 21062 --- [ocp-async-8,5deba278f78669ca,ebcc99f2f5e6ca31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:41.983 INFO 21062 --- [prometheus-minute-schedule-1,d57cc7bb84d13065,78de5e98384dae2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:42.204 INFO 21062 --- [ocp-schedules-24,e2e6dda8321235b6,0cce04977365e619] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:55:42.404 INFO 21062 --- [ocp-schedules-23,98ecf70836d0b8a3,885f960574ed202d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:55:42.404 INFO 21062 --- [ocp-schedules-23,5e9f5e05a0ea09b8,9af5fa4a2fd856c8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:55:42.408 INFO 21062 --- [ocp-schedules-23,5e9f5e05a0ea09b8,9af5fa4a2fd856c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:42.438 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:55:42.438 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:55:42.785 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:42.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:43.022 INFO 21062 --- [ocp-schedules-4,ce77dcd657a49902,66197657d6ee1d25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:55:43.074 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-11-04 15:55:43.074 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-11-04 15:55:43.081 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:55:43.081 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:55:43.081 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:55:43.081 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:55:43.086 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-11-04 15:55:43.086 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : init spring 2024-11-04 15:55:43.086 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-11-04 15:55:43.086 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : starting 2024-11-04 15:55:43.122 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:55:43.130 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:55:43.130 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:55:43.130 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:55:43.130 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:55:43.130 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:55:43.138 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:55:43.659 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-11-04 15:55:43.750 WARN 21062 --- [ocp-async-3,18ea5d03e195e2a8,829ed78e06dbf26b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:43.750 INFO 21062 --- [ocp-async-2,461a1a23a00921fe,e73769ae36c528ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:44.102 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,afe43941fda22d59,74dd8a8bab9cac44] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.130.47.17, traceId afe43941fda22d59] 2024-11-04 15:55:44.106 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,afe43941fda22d59,74dd8a8bab9cac44] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.130.47.17, traceId=afe43941fda22d59, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2024-11-04 15:55:44.265 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.178581199S 2024-11-04 15:55:44.265 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-11-04 15:55:44.277 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,afe43941fda22d59,74dd8a8bab9cac44] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 13, current state: FAILED, new state: READY 2024-11-04 15:55:44.277 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,afe43941fda22d59,74dd8a8bab9cac44] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 42, current state: FAILED, new state: READY 2024-11-04 15:55:44.299 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,afe43941fda22d59,74dd8a8bab9cac44] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.46.128.24:2881, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=2, eventInfo={"param":{"address":"10.46.128.24","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":1046},"port":2881,"proxyroPassword":"******","rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://10.46.128.24:8090/api/v2/ob/clusters/takeOver, clientIp=10.130.47.17, serverIp=null, startTime=2024-11-04T15:55:44.109+08:00, endTime=2024-11-04T15:55:44.299+08:00, role=null, aliUid=null) 2024-11-04 15:55:44.312 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,afe43941fda22d59,74dd8a8bab9cac44] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.130.47.17, traceId=afe43941fda22d59, duration=214 ms] 2024-11-04 15:55:44.374 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-11-04 15:55:44.374 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-11-04 15:55:44.398 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-11-04 15:55:44.464 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-11-04 15:55:44.464 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-11-04 15:55:44.464 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-11-04 15:55:44.464 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-11-04 15:55:44.464 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-11-04 15:55:44.464 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-11-04 15:55:44.475 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-11-04 15:55:44.656 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] o.s.boot.SpringApplication : Started application in 1.575 seconds (JVM running for 12358.336) 2024-11-04 15:55:44.656 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.575483321S 2024-11-04 15:55:44.734 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,a11a47a009018a60,47af0ccd63ff9bd7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=a11a47a009018a60, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:44.762 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,a11a47a009018a60,47af0ccd63ff9bd7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=a11a47a009018a60, duration=36 ms] 2024-11-04 15:55:44.797 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,a6402b87fbe16a9e,4abde43d6332a65d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.130.47.17, traceId=a6402b87fbe16a9e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-11-04 15:55:44.798 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,a6402b87fbe16a9e,4abde43d6332a65d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.130.47.17, traceId=a6402b87fbe16a9e, duration=5 ms] 2024-11-04 15:55:44.908 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,8a0b9c36c447c054] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: RUNNING 2024-11-04 15:55:44.914 INFO 21062 --- [subtask-coordinator1,dd193f8f67a5a423,8a0b9c36c447c054] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: RUNNING 2024-11-04 15:55:44.927 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=13, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T15:55:44.900+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=13}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 15:55:44.930 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 15:55:44.932 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=42, context=Context{parallelIdx=2, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=1046, service_name=dba_test_ob:1730694047, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=10.46.128.24:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2024-11-04T15:55:44.912+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=42}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, executor=10.46.128.24 2024-11-04 15:55:44.938 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2024-11-04 15:55:44.936 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 15:55:44.941 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 15:55:44.943 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 15:55:44.946 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 15:55:44.963 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.11, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null) 2024-11-04 15:55:44.963 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - BY_APPLICATION 2024-11-04 15:55:44.964 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.12, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null) 2024-11-04 15:55:44.964 INFO 21062 --- [ForkJoinPool.commonPool-worker-53,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - BY_APPLICATION 2024-11-04 15:55:44.966 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 15:55:44.967 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-11-04 15:55:44.977 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 15:55:44.983 INFO 21062 --- [prometheus-minute-schedule-1,3c6149d32641eb0c,a82eb039d711b9e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:44.983 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-11-04 15:55:44.988 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3 2024-11-04 15:55:44.994 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3 2024-11-04 15:55:45.005 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.11:1046 success 2024-11-04 15:55:45.008 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.46.128.12:1046 success 2024-11-04 15:55:45.069 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.12 2024-11-04 15:55:45.073 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.12, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:4 2024-11-04 15:55:45.075 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PRIVATE KEY, host: dba@10.46.128.11 2024-11-04 15:55:45.078 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.46.128.11, httpPort=null, sshPort=1046, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=private key, passwordAuthConfig=null, privateKeyAuthConfig=PrivateKeyAuthConfig(username=dba))), proxy=null), estimated size:5 2024-11-04 15:55:45.120 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.126 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.126 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.134 INFO 21062 --- [ocp-schedules-15,d578386f6b2f00d7,26bd2b2a1f867961] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-11-04 15:55:45.134 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.168 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 15:55:45.180 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 15:55:45.194 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.197 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.209 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.212 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.240 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.246 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.255 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.260 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.288 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 15:55:45.307 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 15:55:45.314 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.317 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.338 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.340 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.356 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.362 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.384 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.389 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.404 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12 2024-11-04 15:55:45.431 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.432 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11 2024-11-04 15:55:45.434 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.439 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 15:55:45.463 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.466 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.468 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-11-04 15:55:45.476 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.481 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.507 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.513 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.528 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12 2024-11-04 15:55:45.555 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11 2024-11-04 15:55:45.572 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.575 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.577 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 15:55:45.580 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 15:55:45.582 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 15:55:45.600 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.603 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.606 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-11-04 15:55:45.608 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-11-04 15:55:45.611 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-11-04 15:55:45.620 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.625 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.644 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.649 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.672 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12 2024-11-04 15:55:45.692 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11 2024-11-04 15:55:45.716 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.719 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.722 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-11-04 15:55:45.724 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-11-04 15:55:45.727 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 15:55:45.729 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 15:55:45.732 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:55:45.737 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:55:45.738 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.742 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.2-20241012145836.el7, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.750 WARN 21062 --- [ocp-async-6,8bce67fb41a244cc,c32823ff22f2981a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:45.750 INFO 21062 --- [ocp-async-9,6bc1ead423c7462f,e3ddcf2c33de5391] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:45.757 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-11-04 15:55:45.761 WARN 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=3 2024-11-04 15:55:45.764 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 15:55:45.766 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 15:55:45.780 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.786 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:45.787 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:45.785 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.785 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.790 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.832 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/a1ec9ce0ef2d4e49 on 10.46.128.11 2024-11-04 15:55:45.836 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.838 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/a1ec9ce0ef2d4e49 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a1ec9ce0ef2d4e49, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.841 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.842 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/a1ec9ce0ef2d4e49 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/a1ec9ce0ef2d4e49, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.880 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.885 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.888 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 15:55:45.907 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,157abc2ce6a95486,744b8df1d2239dab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=157abc2ce6a95486, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 15:55:45.911 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.913 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,157abc2ce6a95486,744b8df1d2239dab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=157abc2ce6a95486, duration=18 ms] 2024-11-04 15:55:45.932 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && chmod +x /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh on 10.46.128.11 2024-11-04 15:55:45.939 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && chmod +x /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && chmod +x /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.941 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && chmod +x /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && chmod +x /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.952 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:45.957 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:45.984 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:45.989 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.004 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12 2024-11-04 15:55:46.027 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.029 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.036 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh on 10.46.128.11 2024-11-04 15:55:46.050 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 15:55:46.068 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:46.074 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.120 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12 2024-11-04 15:55:46.141 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,e018017e1e67aa47,8c21e9c7e8248bcd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.12, traceId=e018017e1e67aa47, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 15:55:46.146 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,e018017e1e67aa47,8c21e9c7e8248bcd] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 15:55:46.146 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,e018017e1e67aa47,8c21e9c7e8248bcd] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 15:55:46.147 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,e018017e1e67aa47,8c21e9c7e8248bcd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.12, traceId=e018017e1e67aa47, duration=8 ms] 2024-11-04 15:55:46.233 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/a1ec9ce0ef2d4e49/uninstall_package.sh, out=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 15:55:46.237 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/a1ec9ce0ef2d4e49/uninstall_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/a1ec9ce0ef2d4e49/uninstall_package.sh, out=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ..., err=dpkg: warning: while removing ocp-agent-ce, directory '/usr/lib/systemd/system' not empty so not removed dpkg: warning: while removing ocp-agent-ce, directory '/etc/systemd/system/multi-user.target.wants' not empty so not removed , extOut=null, exitStatus=0) 2024-11-04 15:55:46.238 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.243 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.280 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:46.280 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:46.285 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.288 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.328 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: rm -rf /tmp/a1ec9ce0ef2d4e49 on 10.46.128.11 2024-11-04 15:55:46.328 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12 2024-11-04 15:55:46.333 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: rm -rf /tmp/a1ec9ce0ef2d4e49 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=rm -rf /tmp/a1ec9ce0ef2d4e49, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.336 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: rm -rf /tmp/a1ec9ce0ef2d4e49 on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=rm -rf /tmp/a1ec9ce0ef2d4e49, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.338 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.ocp.executor.executor.SshExecutor : uninstall complete, rpmName=ocp-agent-ce, result=(Reading database ... 94005 files and directories currently installed.) Removing ocp-agent-ce (4.3.2-20241012145836.el7) ... Purging configuration files for ocp-agent-ce (4.3.2-20241012145836.el7) ... Processing triggers for libc-bin (2.31-13+deb11u10) ... 2024-11-04 15:55:46.341 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2024-11-04 15:55:46.343 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-11-04 15:55:46.345 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:55:46.350 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:55:46.353 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-11-04 15:55:46.356 WARN 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2024-11-04 15:55:46.359 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-11-04 15:55:46.362 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-11-04 15:55:46.380 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:46.386 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.431 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:46.437 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.479 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 15:55:46.480 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.485 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.488 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 15:55:46.503 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.506 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.524 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:46.529 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.552 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:46.557 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.576 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/e380bdae0e994e9a on 10.46.128.12 2024-11-04 15:55:46.582 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/e380bdae0e994e9a on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e380bdae0e994e9a, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.585 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/e380bdae0e994e9a on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e380bdae0e994e9a, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.603 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11 2024-11-04 15:55:46.624 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:46.627 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.629 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.630 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.651 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-11-04 15:55:46.672 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:46.676 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/e380bdae0e994e9a/cpio && chmod +x /tmp/e380bdae0e994e9a/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e380bdae0e994e9a/cpio on 10.46.128.12 2024-11-04 15:55:46.677 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.684 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/e380bdae0e994e9a/cpio && chmod +x /tmp/e380bdae0e994e9a/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e380bdae0e994e9a/cpio on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e380bdae0e994e9a/cpio && chmod +x /tmp/e380bdae0e994e9a/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e380bdae0e994e9a/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.686 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/e380bdae0e994e9a/cpio && chmod +x /tmp/e380bdae0e994e9a/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e380bdae0e994e9a/cpio on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e380bdae0e994e9a/cpio && chmod +x /tmp/e380bdae0e994e9a/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/e380bdae0e994e9a/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.724 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11 2024-11-04 15:55:46.728 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:46.733 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.746 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f213ad279cfffc8d,7037919f84cbd268] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm?null, client=10.46.128.11, traceId=f213ad279cfffc8d, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm,] 2024-11-04 15:55:46.750 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f213ad279cfffc8d,7037919f84cbd268] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, objectId=4 2024-11-04 15:55:46.751 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f213ad279cfffc8d,7037919f84cbd268] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm, sha1=f0454eadd8fd83b60b2496697e54501bed39fae5 2024-11-04 15:55:46.751 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,f213ad279cfffc8d,7037919f84cbd268] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm client=10.46.128.11, traceId=f213ad279cfffc8d, duration=7 ms] 2024-11-04 15:55:46.776 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/e380bdae0e994e9a on 10.46.128.12 2024-11-04 15:55:46.782 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/e380bdae0e994e9a on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e380bdae0e994e9a, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.785 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/e380bdae0e994e9a on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=mkdir -p /tmp/e380bdae0e994e9a, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.828 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:46.833 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.840 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.843 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' 'http://10.46.128.24:8090/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.880 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/e380bdae0e994e9a/install_package.sh && chmod +x /tmp/e380bdae0e994e9a/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e380bdae0e994e9a/install_package.sh on 10.46.128.12 2024-11-04 15:55:46.883 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:46.887 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/e380bdae0e994e9a/install_package.sh && chmod +x /tmp/e380bdae0e994e9a/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e380bdae0e994e9a/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e380bdae0e994e9a/install_package.sh && chmod +x /tmp/e380bdae0e994e9a/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e380bdae0e994e9a/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.889 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.890 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/e380bdae0e994e9a/install_package.sh && chmod +x /tmp/e380bdae0e994e9a/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e380bdae0e994e9a/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=touch /tmp/e380bdae0e994e9a/install_package.sh && chmod +x /tmp/e380bdae0e994e9a/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/e380bdae0e994e9a/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.932 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.12 2024-11-04 15:55:46.936 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11 2024-11-04 15:55:46.937 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:46.984 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/e380bdae0e994e9a/install_package.sh on 10.46.128.12 2024-11-04 15:55:47.041 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,81d123eb833c9e5a,aa10dae9bb4744d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=81d123eb833c9e5a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:47.068 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,81d123eb833c9e5a,aa10dae9bb4744d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=81d123eb833c9e5a, duration=33 ms] 2024-11-04 15:55:47.085 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.088 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }' on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm' | awk '{ print $1 }', out=f0454eadd8fd83b60b2496697e54501bed39fae5, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.090 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm 2024-11-04 15:55:47.128 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:47.133 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.180 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/7f9a42bbc96a4a4b on 10.46.128.11 2024-11-04 15:55:47.186 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/7f9a42bbc96a4a4b on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/7f9a42bbc96a4a4b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.189 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/7f9a42bbc96a4a4b on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/7f9a42bbc96a4a4b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.228 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:47.232 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.280 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/7f9a42bbc96a4a4b/cpio && chmod +x /tmp/7f9a42bbc96a4a4b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7f9a42bbc96a4a4b/cpio on 10.46.128.11 2024-11-04 15:55:47.285 INFO 21062 --- [ocp-schedules-27,80783c0f345a9661,bb6127d2fce26432] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-11-04 15:55:47.286 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/7f9a42bbc96a4a4b/cpio && chmod +x /tmp/7f9a42bbc96a4a4b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7f9a42bbc96a4a4b/cpio on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/7f9a42bbc96a4a4b/cpio && chmod +x /tmp/7f9a42bbc96a4a4b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7f9a42bbc96a4a4b/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.290 INFO 21062 --- [ocp-schedules-27,80783c0f345a9661,bb6127d2fce26432] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:47.289 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/7f9a42bbc96a4a4b/cpio && chmod +x /tmp/7f9a42bbc96a4a4b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7f9a42bbc96a4a4b/cpio on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/7f9a42bbc96a4a4b/cpio && chmod +x /tmp/7f9a42bbc96a4a4b/cpio && echo '#!/bin/bash ... eval $new_command' > /tmp/7f9a42bbc96a4a4b/cpio, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.332 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:47.337 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.373 ERROR 21062 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [dba_test_ob, sys] 2024-11-04 15:55:47.375 ERROR 21062 --- [ocp-schedules-27,80783c0f345a9661,bb6127d2fce26432] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:43) at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=dba_test_ob,sys at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:171) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:171) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$622e302.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:271) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:55:47.375 INFO 21062 --- [ocp-schedules-27,80783c0f345a9661,bb6127d2fce26432] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2024-11-04 15:55:47.383 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/7f9a42bbc96a4a4b on 10.46.128.11 2024-11-04 15:55:47.390 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/7f9a42bbc96a4a4b on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/7f9a42bbc96a4a4b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.392 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/7f9a42bbc96a4a4b on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=mkdir -p /tmp/7f9a42bbc96a4a4b, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.436 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:47.440 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.484 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/7f9a42bbc96a4a4b/install_package.sh && chmod +x /tmp/7f9a42bbc96a4a4b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7f9a42bbc96a4a4b/install_package.sh on 10.46.128.11 2024-11-04 15:55:47.491 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/7f9a42bbc96a4a4b/install_package.sh && chmod +x /tmp/7f9a42bbc96a4a4b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7f9a42bbc96a4a4b/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=touch /tmp/7f9a42bbc96a4a4b/install_package.sh && chmod +x /tmp/7f9a42bbc96a4a4b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7f9a42bbc96a4a4b/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.493 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/7f9a42bbc96a4a4b/install_package.sh && chmod +x /tmp/7f9a42bbc96a4a4b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7f9a42bbc96a4a4b/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=touch /tmp/7f9a42bbc96a4a4b/install_package.sh && chmod +x /tmp/7f9a42bbc96a4a4b/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.2-20241012145836.el7.x86_64.rpm'' > /tmp/7f9a42bbc96a4a4b/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.531 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.46.128.11 2024-11-04 15:55:47.536 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-11-04 15:55:47.584 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/7f9a42bbc96a4a4b/install_package.sh on 10.46.128.11 2024-11-04 15:55:47.750 WARN 21062 --- [ocp-async-10,bef9636e445d3bbe,e5a1652771134179] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:47.750 INFO 21062 --- [ocp-async-8,3ab793081a3a1615,754e3b45c4f7f6fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:47.983 INFO 21062 --- [prometheus-minute-schedule-1,682f70b8581bb885,34aa71484d4def05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:48.139 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b21604a6a1e9a57a,3b1252f1e567d788] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=b21604a6a1e9a57a, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 15:55:48.149 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,b21604a6a1e9a57a,3b1252f1e567d788] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=b21604a6a1e9a57a, duration=16 ms] 2024-11-04 15:55:48.591 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:55:48.590827+08:00, lastProcessTime=2024-11-04T15:55:38.588025+08:00 2024-11-04 15:55:48.763 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:55:48.590827+08:00 2024-11-04 15:55:48.787 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:48.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:48.832 INFO 21062 --- [ocp-async-3,311d7affdba7857c,c7a360da5c8e5e19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:55:49.276 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,4db2aa94315f820a,2d5272e4b12f09af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=4db2aa94315f820a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:49.302 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,4db2aa94315f820a,2d5272e4b12f09af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=4db2aa94315f820a, duration=32 ms] 2024-11-04 15:55:49.750 WARN 21062 --- [ocp-async-3,620a2ef883553e3e,c344c81610a00cc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:49.750 INFO 21062 --- [ocp-async-4,28b36de36ee63211,892e1dfeb9d1c381] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:50.983 INFO 21062 --- [prometheus-minute-schedule-1,ae7c3d7746c5a1e2,b6d096fe0a82e9a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:51.062 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,8db92fb82f547973,bbd33897738faae0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=8db92fb82f547973, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:51.088 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,8db92fb82f547973,bbd33897738faae0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=8db92fb82f547973, duration=32 ms] 2024-11-04 15:55:51.750 INFO 21062 --- [ocp-async-6,2366d5307aa5f43b,54cb7d00292e6a63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:51.751 WARN 21062 --- [ocp-async-9,f44ac3486553dab9,ac76d904d4e842f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:51.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:51.788 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:52.207 INFO 21062 --- [ocp-schedules-20,92b0926d8061764b,903e4b69020025be] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:55:52.612 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,24d87ee513ea684f,5a58fc6cd5f5b1e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=24d87ee513ea684f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:52.640 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,24d87ee513ea684f,5a58fc6cd5f5b1e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=24d87ee513ea684f, duration=35 ms] 2024-11-04 15:55:53.047 INFO 21062 --- [ocp-schedules-18,dae822cb1db6b3ea,9a81767f585b6211] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:55:53.751 WARN 21062 --- [ocp-async-8,e5a66afc8b765bf4,d9076fdf993136e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:53.751 INFO 21062 --- [ocp-async-10,83dcbc0e8a7d247c,94a7bc1fe8df7dd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:53.775 INFO 21062 --- [ocp-schedules-22,f9cb078ea28c04a9,5665b9ba8dc1f02a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:55:53.780 INFO 21062 --- [ocp-schedules-22,f9cb078ea28c04a9,5665b9ba8dc1f02a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:55:53.780 INFO 21062 --- [ocp-schedules-22,f9cb078ea28c04a9,5665b9ba8dc1f02a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:55:53.983 INFO 21062 --- [prometheus-minute-schedule-1,45b26d5940534abb,9db31bed25d3fe75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:54.159 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ef5bcc1255ecc73f,c72a58365e3e0779] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=ef5bcc1255ecc73f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:54.183 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,ef5bcc1255ecc73f,c72a58365e3e0779] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=ef5bcc1255ecc73f, duration=31 ms] 2024-11-04 15:55:54.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:54.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:55.751 WARN 21062 --- [ocp-async-2,c29d08adf6dee2b1,26e013816f437440] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:55.751 INFO 21062 --- [ocp-async-3,abaa449dd05c2145,46bbcd836970a520] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:55.858 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,55e4a0ee2ff773f5,56afdb9d9638533d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=55e4a0ee2ff773f5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:55.884 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,55e4a0ee2ff773f5,56afdb9d9638533d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=55e4a0ee2ff773f5, duration=32 ms] 2024-11-04 15:55:56.983 INFO 21062 --- [prometheus-minute-schedule-1,7ce0715362bf516e,044c459ddc956726] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:55:57.506 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,1235f8fe1eb132c1,d8bbe6e5b5a23771] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=1235f8fe1eb132c1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:57.531 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,1235f8fe1eb132c1,d8bbe6e5b5a23771] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=1235f8fe1eb132c1, duration=31 ms] 2024-11-04 15:55:57.751 WARN 21062 --- [ocp-async-9,a8fbf0df0ff11453,a72cc2d6bc01036f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:57.751 INFO 21062 --- [ocp-async-7,20745ca680d1dab0,55ae8ba018b250a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:57.789 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:55:57.790 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:55:58.499 INFO 21062 --- [ocp-schedules-22,82950fc5fbae4c8c,a76f1cb0611c4cd9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:55:58.500 INFO 21062 --- [ocp-schedules-9,7df0ac9a39fa04f0,7f00a08b16d6342f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:55:58.502 INFO 21062 --- [ocp-schedules-30,832e0bbe173466ee,45814122c4d23178] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:55:58.504 INFO 21062 --- [ocp-schedules-9,7df0ac9a39fa04f0,7f00a08b16d6342f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:58.504 INFO 21062 --- [ocp-schedules-22,82950fc5fbae4c8c,a76f1cb0611c4cd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:55:58.505 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:55:58.505 INFO 21062 --- [ocp-schedules-19,d608afde98ea64f3,e144a8e5d96fab35] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:55:58.509 INFO 21062 --- [ocp-schedules-26,cfcded16e2561ad7,d31aebbae968e4aa] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:55:58.509 INFO 21062 --- [ocp-schedules-5,be633eb35d827279,7e79279c238f5f01] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:55:58.509 INFO 21062 --- [ocp-schedules-26,cfcded16e2561ad7,d31aebbae968e4aa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:55:58.509 INFO 21062 --- [ocp-schedules-15,ecf8c5570d0ffcd7,3ff34cf1fa95f178] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:55:58.509 INFO 21062 --- [ocp-schedules-15,ecf8c5570d0ffcd7,3ff34cf1fa95f178] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:55:58.509 INFO 21062 --- [ocp-schedules-5,be633eb35d827279,7e79279c238f5f01] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:55:58.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:55:58.588890+08:00, lastProcessTime=2024-11-04T15:55:48.590827+08:00 2024-11-04 15:55:58.808 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:55:58.588890+08:00 2024-11-04 15:55:58.832 INFO 21062 --- [ocp-async-8,4d3cc6cc098deeb9,64972ba8b7616b28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:55:59.166 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,3b2aeec1b25728d0,177125078ea969e7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=3b2aeec1b25728d0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:55:59.194 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,3b2aeec1b25728d0,177125078ea969e7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=3b2aeec1b25728d0, duration=35 ms] 2024-11-04 15:55:59.697 INFO 21062 --- [ocp-schedules-26,66883fc53a43ac8d,8821227db4a1a86c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:55:59.751 WARN 21062 --- [ocp-async-5,534b12c73812311c,ed7993b5d743a50c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:55:59.751 INFO 21062 --- [ocp-async-10,d40d5359273b2296,d00cb19511056008] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:55:59.983 INFO 21062 --- [prometheus-minute-schedule-1,dda36f1c4c6710e7,6a9be029d7ad9642] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730706960, interval=60 2024-11-04 15:56:00.761 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,6c331a3ede47eaa5,d5dd4d2a572612c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=6c331a3ede47eaa5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:00.787 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,6c331a3ede47eaa5,d5dd4d2a572612c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=6c331a3ede47eaa5, duration=33 ms] 2024-11-04 15:56:00.790 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:00.791 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:01.751 INFO 21062 --- [ocp-async-1,7166433ed55262d6,c44dcd4ccf19e890] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:01.751 WARN 21062 --- [ocp-async-3,6ed781a03a6fb152,be57f07d71ebba0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:02.209 INFO 21062 --- [ocp-schedules-3,574446c90cbf5d09,26578e73e501c47c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:56:02.371 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,bda7ab8782edd6d4,1f4183be3caba1f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=bda7ab8782edd6d4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:02.399 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,bda7ab8782edd6d4,1f4183be3caba1f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=bda7ab8782edd6d4, duration=36 ms] 2024-11-04 15:56:02.983 INFO 21062 --- [prometheus-minute-schedule-1,28cbbf8944f25b67,7f46e294819d7d08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:03.087 INFO 21062 --- [ocp-schedules-15,c25ad36bd5fdfb5e,bb0b90631940057e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:56:03.751 INFO 21062 --- [ocp-async-4,21ff66ce6cf3449b,127aaeb1abc90ecb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:03.751 WARN 21062 --- [ocp-async-7,78a6ccb9b05f663d,80ab383db90bf8a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:03.791 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:03.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:03.995 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,1dd22f4964f7abdf,5fffbb7b31cacc4a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=1dd22f4964f7abdf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:04.019 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,1dd22f4964f7abdf,5fffbb7b31cacc4a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=1dd22f4964f7abdf, duration=31 ms] 2024-11-04 15:56:05.541 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,bee6390dcecd427f,23045c8d86ca3016] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=bee6390dcecd427f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:05.567 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,bee6390dcecd427f,23045c8d86ca3016] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=bee6390dcecd427f, duration=33 ms] 2024-11-04 15:56:05.751 INFO 21062 --- [ocp-async-5,f376b9559efefa44,52ec9cb33b51f6ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:05.751 WARN 21062 --- [ocp-async-8,d5a1d582b23d7a31,b2980d242107b5bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:05.983 INFO 21062 --- [prometheus-minute-schedule-1,cece14433071401b,052a544ead3e62d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:06.792 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:06.793 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:07.082 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1dfba7fa72016d47,ce17faf2c3e878d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=1dfba7fa72016d47, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:07.108 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,1dfba7fa72016d47,ce17faf2c3e878d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=1dfba7fa72016d47, duration=33 ms] 2024-11-04 15:56:07.752 INFO 21062 --- [ocp-async-3,34565076f12f24ee,cce6e1d08266a5b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:07.752 WARN 21062 --- [ocp-async-2,78d127e161969b03,d6a0580df249a53c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:08.505 INFO 21062 --- [ocp-schedules-9,7df0ac9a39fa04f0,7f00a08b16d6342f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2024-11-04 15:56:08.505 INFO 21062 --- [ocp-schedules-22,82950fc5fbae4c8c,a76f1cb0611c4cd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:56:08.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:56:08.588678+08:00, lastProcessTime=2024-11-04T15:55:58.588890+08:00 2024-11-04 15:56:08.686 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,605b29967d2eb999,9b5e9936b6878267] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=605b29967d2eb999, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:08.711 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,605b29967d2eb999,9b5e9936b6878267] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=605b29967d2eb999, duration=32 ms] 2024-11-04 15:56:08.760 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:56:08.588678+08:00 2024-11-04 15:56:08.833 INFO 21062 --- [ocp-async-7,da7f3b43b3fad061,ae0147a2a9f6059e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:56:08.983 INFO 21062 --- [prometheus-minute-schedule-1,d6d40cb4e5f70d40,81d0980fb4180e75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:09.752 INFO 21062 --- [ocp-async-9,4f79dce3c5781bf4,3cb6b61dd961e345] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:09.752 WARN 21062 --- [ocp-async-4,93b3be7a71d3181a,244520de499cebfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:09.793 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:09.794 INFO 21062 --- [pool-11-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:10.003 INFO 21062 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-11-04 15:56:10.003 INFO 21062 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-11-04 15:56:10.205 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1743127845cb695c,54a5825da1a103a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=1743127845cb695c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:10.231 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,1743127845cb695c,54a5825da1a103a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=1743127845cb695c, duration=33 ms] 2024-11-04 15:56:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:56:10.302 WARN 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:56:10.303 INFO 21062 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 1, ruleSize = 1. 2024-11-04 15:56:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = ob_memory_percent,ob_tenant_cpu_percent,ob_tenant_disk_usage_v4, label = app=OB, gbLabel = app, ob_cluster_name, ob_cluster_id, tenant_name 2024-11-04 15:56:10.324 WARN 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.s.ShardingRecordingRulesHelper : Cannot match label for rule. metrics = sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,sql_local_count,sql_remote_count,sql_distributed_count,sql_local_time,sql_remote_time,sql_distributed_time,active_session,all_session,wait_event_count,system_event_application_total_waits,system_event_configuration_total_waits,system_event_administrative_total_waits,system_event_concurrency_total_waits,system_event_commit_total_waits,system_event_network_total_waits,system_event_user_io_total_waits,system_event_system_io_total_waits,system_event_cluster_total_waits,system_event_other_total_waits,wait_event_rt,system_event_application_time_waited,system_event_configuration_time_waited,system_event_administrative_time_waited,system_event_concurrency_time_waited,system_event_commit_time_waited,system_event_network_time_waited,system_event_user_io_time_waited,system_event_system_io_time_waited,system_event_cluster_time_waited,system_event_other_time_waited,request_enqueue_count,request_dequeue_count,request_queue_time,ob_tenant_cpu_percent,ob_tenant_thread_percent,ob_memory_percent,memstore_percent,rpc_packet_in_rt,rpc_packet_out_rt,rpc_packet_in,rpc_packet_out,opened_cursors_count,max_full_clog_sync_delay_seconds,max_readonly_clog_sync_delay_seconds,max_logonly_clog_sync_delay_seconds,tps,transaction_commit_count,transaction_rollback_count,transaction_timeout_count,trans_sp_total_used_time,trans_distributed_total_used_time,tps_rt,transaction_commit_rt,transaction_rollback_rt,trans_commit_log_count,clog_trans_log_total_size,trans_commit_log_sync_rt,clog_write_disk_rt,memstore_write_lock_succ_count,memstore_write_lock_fail_count,memstore_write_lock_wait_time,transaction_single_partition_count,transaction_multi_partition_count,total_memstore_used,active_memstore_used,major_freeze_trigger,memstore_limit,io_read_count,io_write_count,io_read_rt,io_write_rt,io_read_size,io_writ 2024-11-04 15:56:10.324 INFO 21062 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 5, ruleSize = 5. 2024-11-04 15:56:11.741 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4422ed0b8cd35b00,e91236c99c9dfa95] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=4422ed0b8cd35b00, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:11.752 WARN 21062 --- [ocp-async-6,6cb26a2c2432232a,67c996410bea121f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:11.752 INFO 21062 --- [ocp-async-5,70eeb686fd3e51d4,5a393806da29afe7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:11.767 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,4422ed0b8cd35b00,e91236c99c9dfa95] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=4422ed0b8cd35b00, duration=35 ms] 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=5, affectRows=5 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-41,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=5, validCount=5 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=12, affectRows=12 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=9, affectRows=9 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=14, affectRows=14 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-45,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=12, validCount=12 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-42,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=9, validCount=9 2024-11-04 15:56:11.977 INFO 21062 --- [metric-archive-second-35,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=14, validCount=14 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=22, affectRows=22 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=38, affectRows=38 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=22, validCount=22 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=26, affectRows=26 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=38, validCount=38 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=52, affectRows=52 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=26, validCount=26 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=71, affectRows=71 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=52, validCount=52 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=43, affectRows=43 2024-11-04 15:56:11.978 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=41, affectRows=41 2024-11-04 15:56:11.979 INFO 21062 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=69, validCount=69 2024-11-04 15:56:11.979 INFO 21062 --- [metric-archive-second-34,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=43, validCount=43 2024-11-04 15:56:11.979 INFO 21062 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=41, validCount=41 2024-11-04 15:56:11.979 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=45, affectRows=45 2024-11-04 15:56:11.979 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1730706900, nodeCount=46, affectRows=46 2024-11-04 15:56:11.979 INFO 21062 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=45, validCount=45 2024-11-04 15:56:11.979 INFO 21062 --- [metric-archive-second-50,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1730706900, nodeCount=45, validCount=45 2024-11-04 15:56:11.983 INFO 21062 --- [prometheus-minute-schedule-1,22fc988a832cad68,6a4f1c4b85aef9cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:12.213 INFO 21062 --- [ocp-schedules-12,9177a4d4964b7e4a,5dd4e7265deb6ba7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:56:12.439 INFO 21062 --- [ocp-schedules-23,5e9f5e05a0ea09b8,9af5fa4a2fd856c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2024-11-04 15:56:12.439 INFO 21062 --- [ocp-schedules-23,0df3132c0f1df554,750032a305c41e73] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-11-04 15:56:12.442 INFO 21062 --- [ocp-schedules-23,0df3132c0f1df554,750032a305c41e73] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:56:12.472 INFO 21062 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2024-11-04 15:56:12.472 INFO 21062 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:56:12.794 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:12.795 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:13.112 INFO 21062 --- [ocp-schedules-1,a4d93bdb76314889,2315e0675438bda3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:56:13.278 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,128ecf58007b8c94,983447617578c5cf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=128ecf58007b8c94, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:13.304 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,128ecf58007b8c94,983447617578c5cf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=128ecf58007b8c94, duration=32 ms] 2024-11-04 15:56:13.752 INFO 21062 --- [ocp-async-2,ab499944482b54d8,d61558df5b2bc97d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:13.752 WARN 21062 --- [ocp-async-3,3b67333bd825ae0f,74eb501933dd415d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:13.783 INFO 21062 --- [ocp-schedules-5,a58bd09e62d332de,dd3fcbcdbd9893d7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:56:13.789 INFO 21062 --- [ocp-schedules-5,a58bd09e62d332de,dd3fcbcdbd9893d7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:56:13.789 INFO 21062 --- [ocp-schedules-5,a58bd09e62d332de,dd3fcbcdbd9893d7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:56:13.984 INFO 21062 --- [prometheus-cache-monitor1,e83d131b336a8bdd,4b1eff086c94528e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":0,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:56:13.984 INFO 21062 --- [prometheus-cache-monitor1,e83d131b336a8bdd,4b1eff086c94528e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:56:14.818 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8c96d64e22d286d1,a1c34a500a985ee0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=8c96d64e22d286d1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:14.847 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,8c96d64e22d286d1,a1c34a500a985ee0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=8c96d64e22d286d1, duration=35 ms] 2024-11-04 15:56:14.983 INFO 21062 --- [prometheus-minute-schedule-1,d56d5c0be5963c43,313d7e8044ea58d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:15.752 WARN 21062 --- [ocp-async-1,a13a5a5ca0c8ec47,65b8434a993378ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:15.752 INFO 21062 --- [ocp-async-4,a2c3ce48cf8389ba,404fb64480f0b7c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:15.795 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:15.795 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:16.354 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9914d9ff70b5c3b0,e32b71d4826db3fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9914d9ff70b5c3b0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:16.380 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,9914d9ff70b5c3b0,e32b71d4826db3fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9914d9ff70b5c3b0, duration=34 ms] 2024-11-04 15:56:17.752 WARN 21062 --- [ocp-async-9,9ee483c8289af446,ca4971b680ed5ff8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:17.752 INFO 21062 --- [ocp-async-6,fe9ccf7214d8be42,2dda7b19913d01ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:17.916 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,70d6726c095a9d48,b0e4b37aa807d32c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=70d6726c095a9d48, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:17.942 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,70d6726c095a9d48,b0e4b37aa807d32c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=70d6726c095a9d48, duration=33 ms] 2024-11-04 15:56:17.983 INFO 21062 --- [prometheus-minute-schedule-1,c1dc0e4610b4b5e9,8ef741bb6581280e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:18.589 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:56:18.588379+08:00, lastProcessTime=2024-11-04T15:56:08.588678+08:00 2024-11-04 15:56:18.771 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:56:18.588379+08:00 2024-11-04 15:56:18.795 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:18.796 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:18.833 INFO 21062 --- [ocp-async-3,2e28e888b8aa33dd,fea4f47cffc45aad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:56:19.492 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,74278ceddf57314d,4ddf331932877506] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=74278ceddf57314d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:19.519 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,74278ceddf57314d,4ddf331932877506] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=74278ceddf57314d, duration=35 ms] 2024-11-04 15:56:19.753 WARN 21062 --- [ocp-async-10,32f572ffa677c37b,81e7987dd2cec91e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:19.753 INFO 21062 --- [ocp-async-2,8ebb3c7d08471ec8,bc791f47397fc0fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:20.983 INFO 21062 --- [prometheus-minute-schedule-1,d860b844de702656,2ecee66fdafbd2b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:21.037 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,7c79c66c869dd2f3,a3c5651264be6ac6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=7c79c66c869dd2f3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:21.065 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-10,7c79c66c869dd2f3,a3c5651264be6ac6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=7c79c66c869dd2f3, duration=33 ms] 2024-11-04 15:56:21.753 WARN 21062 --- [ocp-async-7,3c8ccbe7e0f0110e,055843e6f93623a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:21.753 INFO 21062 --- [ocp-async-4,79127011ce21a625,f0425ee734170ea9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:21.797 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:21.798 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:22.216 INFO 21062 --- [ocp-schedules-22,ca7243bb6d6a1bb2,7ec0dc386f490fda] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:56:22.622 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,08afc7556eb7b4ac,c4d5a2ff889df97e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=08afc7556eb7b4ac, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:22.649 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-6,08afc7556eb7b4ac,c4d5a2ff889df97e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=08afc7556eb7b4ac, duration=35 ms] 2024-11-04 15:56:23.155 INFO 21062 --- [ocp-schedules-16,3b4f9d126f58e685,581a7235ab6284e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:56:23.753 INFO 21062 --- [ocp-async-9,c09b423f5284b9c7,015da38b19b34992] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:23.753 WARN 21062 --- [ocp-async-6,56be4681c19f9e86,fc97af0889a3c98b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:23.983 INFO 21062 --- [prometheus-minute-schedule-1,6192d624272cb907,8ebf4bb317deed79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:24.173 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,590a233850fd6b1f,b2aa63227505ae33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=590a233850fd6b1f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:24.199 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-4,590a233850fd6b1f,b2aa63227505ae33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=590a233850fd6b1f, duration=34 ms] 2024-11-04 15:56:24.798 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:24.799 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:25.718 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,9f5df90b6b109864,7d56f2230c37b2b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=9f5df90b6b109864, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:25.744 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-7,9f5df90b6b109864,7d56f2230c37b2b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=9f5df90b6b109864, duration=33 ms] 2024-11-04 15:56:25.753 WARN 21062 --- [ocp-async-5,53938d6325ad48e9,f3f568701c8ab5d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:25.753 INFO 21062 --- [ocp-async-3,21f8daee749bd287,fc479f198d6ef962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:26.465 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/e380bdae0e994e9a/install_package.sh on 10.46.128.12, result: SshResult(host=10.46.128.12, username=dba, command=/tmp/e380bdae0e994e9a/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 15:56:26.470 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/e380bdae0e994e9a/install_package.sh on 10.46.128.12,result:SshResult(host=10.46.128.12, username=dba, command=/tmp/e380bdae0e994e9a/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 15:56:26.472 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/e380bdae0e994e9a/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 15:56:26.475 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/e380bdae0e994e9a/install_package.sh, exit code: 2, cause:null 2024-11-04 15:56:26.478 ERROR 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/e380bdae0e994e9a/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/e380bdae0e994e9a/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:56:26.490 INFO 21062 --- [manual-subtask-executor16,e46f513af4706458,0e1b6350aa8bb00a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:RETRY, state: FAILED 2024-11-04 15:56:26.983 INFO 21062 --- [prometheus-minute-schedule-1,0963404398328c82,27cc165dc115fde3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:26.984 INFO 21062 --- [prometheus-hour-schedule-1,5ac7113b926571f0,8ede18f3fcd6d42a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707200, interval=3600 2024-11-04 15:56:27.324 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,05d00073b5a06fd2,91e4e91342832f67] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=05d00073b5a06fd2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:27.351 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-3,05d00073b5a06fd2,91e4e91342832f67] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=05d00073b5a06fd2, duration=34 ms] 2024-11-04 15:56:27.753 WARN 21062 --- [ocp-async-1,fc7d13c8663c0b5e,7bba738ff9b897e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:27.753 INFO 21062 --- [ocp-async-2,cc9f2ed8f0d3eb89,f4d29bbf7d20f581] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:27.799 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:27.800 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:28.498 INFO 21062 --- [ocp-schedules-7,9bdefe15f72ccdf5,b29ad7f733649dec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-11-04 15:56:28.499 INFO 21062 --- [ocp-schedules-26,0396241c77d762ba,00c9d6b7f4262ee9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-11-04 15:56:28.500 INFO 21062 --- [ocp-schedules-27,66ddb1c3cbb0731d,b867f05fe3245c50] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-11-04 15:56:28.500 INFO 21062 --- [ocp-schedules-17,846cd4fa909daf58,af9e06fe741a2e1e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-11-04 15:56:28.502 INFO 21062 --- [ocp-schedules-22,ee389cb7d6aed1e4,ae5acd7d2d80984f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-11-04 15:56:28.503 INFO 21062 --- [ocp-schedules-7,9bdefe15f72ccdf5,b29ad7f733649dec] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-11-04 15:56:28.503 INFO 21062 --- [ocp-schedules-26,0396241c77d762ba,00c9d6b7f4262ee9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:56:28.504 INFO 21062 --- [ocp-schedules-17,846cd4fa909daf58,af9e06fe741a2e1e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:56:28.504 INFO 21062 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=dba_test_ob, obClusterId=1730694047, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2024-11-04T12:32:38+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, tenantCount=0) 2024-11-04 15:56:28.505 INFO 21062 --- [ocp-schedules-10,0b0c217a17e3244c,579be25ea077a0ae] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-11-04 15:56:28.506 INFO 21062 --- [ocp-schedules-27,66ddb1c3cbb0731d,b867f05fe3245c50] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-11-04 15:56:28.509 INFO 21062 --- [ocp-schedules-8,06816a2938dae320,43adb20b28e6f2af] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2024-11-04 15:56:28.509 INFO 21062 --- [ocp-schedules-28,c5f2fd16fb442477,235c5d757101f2bd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2024-11-04 15:56:28.510 INFO 21062 --- [ocp-schedules-8,06816a2938dae320,43adb20b28e6f2af] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2024-11-04 15:56:28.510 INFO 21062 --- [ocp-schedules-28,c5f2fd16fb442477,235c5d757101f2bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2024-11-04 15:56:28.510 INFO 21062 --- [ocp-schedules-14,57f2cfed3ff8a811,9216ba7da64f6491] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2024-11-04 15:56:28.510 INFO 21062 --- [ocp-schedules-14,57f2cfed3ff8a811,9216ba7da64f6491] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2024-11-04 15:56:28.590 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-11-04T15:56:28.589653+08:00, lastProcessTime=2024-11-04T15:56:18.588379+08:00 2024-11-04 15:56:28.596 INFO 21062 --- [ocp-schedules-11,95fe5db7101cd447,a0d3740689e7fa62] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1696,"minuteWriteQueue":0,"minuteReadCacheSize":419,"secondWriteCacheSize":423,"seriesIdKeySize":424,"secondWriteQueue":0,"minuteWriteCacheSize":421} 2024-11-04 15:56:28.596 INFO 21062 --- [ocp-schedules-11,95fe5db7101cd447,a0d3740689e7fa62] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=67, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=131, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=67, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=131, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2024-11-04 15:56:28.759 INFO 21062 --- [alarm-task-1,4a9c660eabd3cfdf,5b21e9c8778a7d9b] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-11-04T15:56:28.589653+08:00 2024-11-04 15:56:28.833 INFO 21062 --- [ocp-async-8,454b77ef356430f2,80144cfe4b465601] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-11-04 15:56:28.933 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,3198767c8a613fda,a12cd95c8106d584] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=3198767c8a613fda, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:28.960 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-2,3198767c8a613fda,a12cd95c8106d584] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=3198767c8a613fda, duration=36 ms] 2024-11-04 15:56:29.073 INFO 21062 --- [ocp-schedules-18,d4e050465bc51ecd,fccd36072dad73ce] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-11-04 15:56:29.451 INFO 21062 --- [ocp-schedules-1,2e9ca7f5ee6536e3,6663cc1e27e66b20] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-11-04 15:56:29.455 INFO 21062 --- [ocp-schedules-1,2e9ca7f5ee6536e3,6663cc1e27e66b20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:56:29.455 INFO 21062 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-11-04 15:56:29.531 INFO 21062 --- [ocp-schedules-21,75233362ab8ba037,9ec15d00bbd87032] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-11-04 15:56:29.531 INFO 21062 --- [ocp-schedules-21,75233362ab8ba037,9ec15d00bbd87032] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-11-04 15:56:29.699 INFO 21062 --- [ocp-schedules-6,ba6e6a36c4c917ef,a107c78e57565844] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-11-04 15:56:29.710 INFO 21062 --- [ocp-schedules-13,2d6be087fb1f6003,9a1a10be95cffae4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-11-04 15:56:29.715 INFO 21062 --- [ocp-schedules-13,2d6be087fb1f6003,9a1a10be95cffae4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-11-04 15:56:29.715 INFO 21062 --- [ocp-schedules-13,2d6be087fb1f6003,9a1a10be95cffae4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2024-11-04 15:56:29.754 INFO 21062 --- [ocp-async-5,d1b9654837d514fb,0eda68e05ea47640] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:29.754 WARN 21062 --- [ocp-async-10,b73a9dc5f006b257,ce8cee3d78f88cf5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:29.983 INFO 21062 --- [prometheus-minute-schedule-1,6662afe31d5e01f7,a119d8208d0eaf04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:30.258 INFO 21062 --- [ocp-schedules-28,5a7d6e479bf2be32,aca2653d25d11579] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-11-04 15:56:30.258 INFO 21062 --- [ocp-schedules-28,5a7d6e479bf2be32,aca2653d25d11579] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2024-11-04 15:56:30.264 INFO 21062 --- [ocp-schedules-28,5a7d6e479bf2be32,aca2653d25d11579] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-11-04 15:56:30.264 INFO 21062 --- [ocp-schedules-28,5a7d6e479bf2be32,aca2653d25d11579] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709908318657990656 2024-11-04 15:56:30.269 INFO 21062 --- [ocp-schedules-28,5a7d6e479bf2be32,aca2653d25d11579] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-11-04 15:56:30.269 INFO 21062 --- [ocp-schedules-28,5a7d6e479bf2be32,aca2653d25d11579] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=709957217611436032 2024-11-04 15:56:30.282 INFO 21062 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:56:30.309 INFO 21062 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-11-04 15:56:30.445 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7f5d0e72efd849f3,a63b7102d920cba0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=7f5d0e72efd849f3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:30.473 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-5,7f5d0e72efd849f3,a63b7102d920cba0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=7f5d0e72efd849f3, duration=35 ms] 2024-11-04 15:56:30.800 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:30.801 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:31.241 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/7f9a42bbc96a4a4b/install_package.sh on 10.46.128.11, result: SshResult(host=10.46.128.11, username=dba, command=/tmp/7f9a42bbc96a4a4b/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 15:56:31.245 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/7f9a42bbc96a4a4b/install_package.sh on 10.46.128.11,result:SshResult(host=10.46.128.11, username=dba, command=/tmp/7f9a42bbc96a4a4b/install_package.sh, out=dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err=dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. , extOut=null, exitStatus=2) 2024-11-04 15:56:31.247 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:/tmp/7f9a42bbc96a4a4b/install_package.sh, return code:2, output:dpkg --no-force-overwrite -i ocp-agent-ce_4.3.2-20241012145836.el7_amd64.deb find ocp-agent-ce-4.3.2 -type d -exec chmod 755 {} ; rm -rf ocp-agent-ce-4.3.2, err:dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin Unable to install at /usr/share/perl5/Alien/Package/Deb.pm line 97. 2024-11-04 15:56:31.250 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: /tmp/7f9a42bbc96a4a4b/install_package.sh, exit code: 2, cause:null 2024-11-04 15:56:31.252 ERROR 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: /tmp/7f9a42bbc96a4a4b/install_package.sh, exit code: 2 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: /tmp/7f9a42bbc96a4a4b/install_package.sh, exit code: 2 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:415) at com.oceanbase.ocp.executor.executor.SshExecutor.execByShellFile(SshExecutor.java:144) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:384) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:218) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$82f39690.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:56:31.262 INFO 21062 --- [manual-subtask-executor14,e46f513af4706458,69ae6af86190093b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 13, operation:RETRY, state: FAILED 2024-11-04 15:56:31.754 INFO 21062 --- [ocp-async-7,f63e8b6d16ee212e,71daa8f04b236730] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:31.754 WARN 21062 --- [ocp-async-1,a7c07c1024530947,89c9a1be6da359ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:31.987 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,b64b3e5879adfef0,b9638ec3ddd404d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=b64b3e5879adfef0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:32.012 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-9,b64b3e5879adfef0,b9638ec3ddd404d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=b64b3e5879adfef0, duration=32 ms] 2024-11-04 15:56:32.025 INFO 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='10.46.128.24', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.8, cluster_name=dba_test_ob, ssh_port=1046, prohibit_rollback=false, service_name=dba_test_ob:1730694047, ob_connect_address=10.46.128.24:2881, cluster_type=PRIMARY, service_version=4.2.1.8, host_credential_id=1, ob_cluster_id=1730694047, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/data1/oceanbase/data1, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1, 2, 3], add_idc_ids=[1], all_host_ids=[1, 2, 3], add_host_ids=[1, 2, 3], host_ids=[1, 2, 3], zone_names=[zone2, zone1, zone3]}}, createTime=2024-11-04T12:32:39+08:00, startTime=2024-11-04T12:32:39.355+08:00, endTime=2024-11-04T14:52:19.316+08:00, subtasks=[SubtaskInstanceEntity{id=68, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=33, startTime=2024-11-04T14:11:16.789+08:00, endTime=2024-11-04T14:11:16.854+08:00}, SubtaskInstanceEntity{id=12, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=8, name=Sync cluster info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncClusterInfoTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=40, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=42, startTime=2024-11-04T12:32:53.452+08:00, endTime=2024-11-04T12:32:53.478+08:00}, SubtaskInstanceEntity{id=33, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busin 2024-11-04 15:56:32.034 WARN 21062 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-11-04 15:56:32.035 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2024-11-04 15:56:32.036 INFO 21062 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:2, status:FAILED 2024-11-04 15:56:32.221 INFO 21062 --- [ocp-schedules-27,6032e8cf0ba2a160,26068dfa8d1a443e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-11-04 15:56:32.729 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d31001736e1f791e,8fd880836b5aadb4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments?taskInstanceId=2&subtaskInstanceId=13, client=10.130.47.17, traceId=d31001736e1f791e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,13,] 2024-11-04 15:56:32.739 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-8,d31001736e1f791e,8fd880836b5aadb4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/13/logSegments client=10.130.47.17, traceId=d31001736e1f791e, duration=16 ms] 2024-11-04 15:56:32.983 INFO 21062 --- [prometheus-minute-schedule-1,f879ea27eefcdcce,f58a83e327a29a98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60 2024-11-04 15:56:33.183 INFO 21062 --- [ocp-schedules-12,44b6daa5a07590a0,e6c49894700e0530] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-11-04 15:56:33.754 WARN 21062 --- [ocp-async-2,a7b030b7581fb9c4,0297678913ac19e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:33.754 INFO 21062 --- [ocp-async-4,dba5f3635a155091,838c059dc3512811] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:33.791 INFO 21062 --- [ocp-schedules-8,e7f9f04504d2c5ef,5a96469d96752b0b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-11-04 15:56:33.796 INFO 21062 --- [ocp-schedules-8,e7f9f04504d2c5ef,5a96469d96752b0b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:56:33.796 INFO 21062 --- [ocp-schedules-8,e7f9f04504d2c5ef,5a96469d96752b0b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[354af545-dee8-4634-bc44-2ef0a637b63a] 2024-11-04 15:56:33.801 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.46.128.24:8090 refreshes policies... 2024-11-04 15:56:33.802 INFO 21062 --- [pool-11-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-11-04 15:56:33.956 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,572f4f3b7da14e32,fa77ee70e6e80492] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.130.47.17, traceId=572f4f3b7da14e32, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2024-11-04 15:56:33.984 INFO 21062 --- [http-nio-0.0.0.0-8090-exec-1,572f4f3b7da14e32,fa77ee70e6e80492] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.130.47.17, traceId=572f4f3b7da14e32, duration=34 ms] 2024-11-04 15:56:34.149 INFO 21062 --- [ocp-schedules-19,c4543cc9c333042e,f9b74253dcb83fed] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-11-04 15:56:34.149 INFO 21062 --- [ocp-schedules-19,c4543cc9c333042e,f9b74253dcb83fed] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-11-04 15:56:34.175 INFO 21062 --- [ocp-schedules-19,c4543cc9c333042e,f9b74253dcb83fed] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2024-11-04 15:56:34.175 INFO 21062 --- [ocp-schedules-19,c4543cc9c333042e,f9b74253dcb83fed] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-11-04 15:56:34.178 INFO 21062 --- [ocp-schedules-7,2bb5217bd04f3c73,28ae93f4b868ff38] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-11-04 15:56:34.355 INFO 21062 --- [ocp-schedules-20,d40f4ef9c0d48c20,13e64b7e3611f527] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-11-04T15:56:34.355+08:00, clusters=[1] 2024-11-04 15:56:34.357 INFO 21062 --- [ocp-schedules-20,d40f4ef9c0d48c20,13e64b7e3611f527] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster dba_test_ob, trans stat result is empty. 2024-11-04 15:56:34.753 INFO 21062 --- [ocp-schedules-15,f90b979586400653,3fae35773c3607f2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-11-04T15:56:34.753+08:00, clusters=[1] 2024-11-04 15:56:34.756 INFO 21062 --- [ocp-schedules-15,f90b979586400653,3fae35773c3607f2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster dba_test_ob for slowSqlStat, sql count is 0. 2024-11-04 15:56:34.861 INFO 21062 --- [ocp-schedules-16,7d891b316931150a,c7518253418b337f] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-11-04 15:56:34.865 INFO 21062 --- [ocp-schedules-16,7d891b316931150a,c7518253418b337f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2024-11-04 15:56:34.920 ERROR 21062 --- [ocp-schedules-16,7d891b316931150a,c7518253418b337f] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:85) at sun.reflect.GeneratedMethodAccessor810.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=dba_test_ob,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$77c1a84d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2024-11-04 15:56:34.920 INFO 21062 --- [ocp-schedules-16,7d891b316931150a,c7518253418b337f] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2024-11-04 15:56:35.754 WARN 21062 --- [ocp-async-4,25ec0b6a01e2973e,d6503e73a79472bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2024-11-04 15:56:35.754 INFO 21062 --- [ocp-async-4,2880af7fed2320b8,2d9b78703bad46eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-11-04 15:56:35.983 INFO 21062 --- [prometheus-minute-schedule-1,aa195c67800244a0,ad5730548607c029] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1730707020, interval=60